We have currently trying to transmit data from one server to another using RYSNC by synchronizing 2 folders. Occasionally, we noticed that the file stop when in the middle of transmission (we can see tmp file created on the receiving end) even through the file has completely written to the the folder on the sending side. After a long while (7-30 minutes), it will complete sending. We are not sure if the following is true, but if we start sending a file to the same folder the such issue happen, the "hung" file will resume sending and complete immediately. The message file that we sent are usually in the sizes between 1MB-4MB. For the time being, we are sending small-sized file of 1byte to "trigger" and "hung" transmission. Below is the arguments that we are using RSYNC_ARG="--log-file=$LOGFILE --chmod=a+rwx,g+rwx,o+rwx --exclude=*.tmp --remove-sent-files --stats --ignore-times --whole-file -qr" There is no data loss but we encountered a long delay in data transmission.
For the time being, we are sending small 1byte message periodically to "give a push" to the "appeared to be hung" files over. This seems to work temporarily and we are hopping to have a fix for this issue. Let us know if this is a configuration issue as well.
This sounds like some kind of networking issue given what you say about how you get things going again.