External dependencies placed in $art_home/etc/plugins/lib, like Jar files, are not replicated correctly, the file is usually a partial, much smaller size (about half) on secondary nodes.
Steps to reproduce:
1. On Primary Node, place Jar file into dir, (I did it using curl), file size should be normal.
2. HA will begin to sync the file over immediately to other nodes, in the corresponding directories ($second_node_art_home/etc/plugins/lib)
3. Check secondary node's $art_home/etc/plugins/lib, the file will be there, but a smaller size. Restarting the secondary, or deleting the file will only result in the partial file being downloaded to it again. No fix other than manually moving the file over to secondary nodes.
Note: this occurs more frequently than not, especially with files over the size of 10MB, and over slower connections. Was able to do reproduce with files as small as 57 KB.
This can also happen on the primary, if the customer is starting with a (mostly) clean Artifactory home when doing an upgrade (download new Art version -> copy old config files, but not plugins dep. into new home, start Art), so the node will pull the files from the DB, and end up in the bad state.
Leave the corrupt files intact and also place the same files with a different name on the node.