This issue can be reproduce with a 3-node cluster, a sample script (dl.sh) is provided by Gal to reproduce the deadlock
1) Upload a large file of 1gb to Node-1, it will be uploaded to eventual first.
2) While this upload is being done to S3, please stop the Node-1
3) Make concurrent requests for this file to Node-2 & Node-3
i) The redundancy is set to 2, each request to Node-2 & Node-3 will not find the binary in local cache and will get lock in ConcurrentReadCacheMissHandler.handleCacheMiss
ii) A deadlock is created when the request from Node-2 is waiting for the lock on Node-3 cache, and the request from Node-3 is waiting for the lock on Node-2 cache.