When transferring large files over a slow network, we interrupt rsync at the beginning of business hours leaving the transfer unfinished. The command used is: rsync -av --inplace --bwlimit=400 hostname::module /dest When restarting the transfer, a lot of time is "wasted" while first the local system is reading the partially transferred file and sends the checksums to the remote, which only then starts to read the source file until it finds something to transfer. So nothing happens until 2 times the time required to read the partial transfer from the disks! When the partial file is many many GB, this can take hours. Suggestions: 1. when the source is larger than the destination, immediately begin to transfer from the offset in the source equal to the size of the destination. it is already known that this part will have to be transferred. 2. try to do the reading of the partial file at the destination and the same part of the source in parallel (so the time is halved), and preferably also in parallel to 1. Of course these optimizations (at least #2) may actually decrease performance when the transfer is local (not over slow network) and the disk read rate is negatively affected by reading at two different places in parallel. So #2 should only be attempted when the transfer is over a network.
If you are sure the file has not been changed since it was partially copied, see --append.
Thanks, that helps a lot for this particular use case. (the files are backups)
Rsync is never going to assume that a file can be continued, as it doesn't know what the old data is compared to the source. You can tell rsync to assume that the early data is all fine by using --append, but that can cause you problems if any non-new files need an update that is not an append.
Ok you apparently did not understand what I proposed. However it is not that important as in our use case we can use --append.