Bug 5801 - Speed up delta-transfer of zeros in new files with --sparse
Speed up delta-transfer of zeros in new files with --sparse
Status: NEW
Product: rsync
Classification: Unclassified
Component: core
Other Linux
: P3 enhancement
: ---
Assigned To: Wayne Davison
Rsync QA Contact
: 7854 (view as bug list)
Depends on:
  Show dependency treegraph
Reported: 2008-09-29 14:56 UTC by Martin Scharrer
Modified: 2016-01-14 10:42 UTC (History)
2 users (show)

See Also:


Note You need to log in before you can comment on or make changes to this bug.
Description Martin Scharrer 2008-09-29 14:56:52 UTC
Excuse me, this seems to be a known issue, but I couldn't find any related existing bug for it.

The 'handle sparse files efficiently' option (--sparse/-S) doesn't work with new files, i.e. files with doesn't exist on the destination. In this case all sparse blocks are transmitted as zero blocks which slows down the sync process heavily in some application.

This issue was discussed already:

I detected this today when I was scripting something to sync some large video files in mkv format (around 1.1GB) with the last MB transfered first in order to watch the incomplete video while it was transfered. (BTW: An option in rsync for this would be awesome!!). To do this I created a sparse copy of the video with only the last MB holding real data using 'dd if=... of=.sparse/... count=0 bs=1M seek=$SIZE_IN_MB_MINUS_ONE' and then rsyncing the .sparse dir first using -S. To my surprise this was as slow as the rsyncing of the real file.
Comment 1 Matt McCutchen 2008-10-01 20:17:17 UTC
As currently designed, the --sparse option only affects writing of the destination file, not the transfer of the data, so this is an enhancement request.  -z may help.  It may be worth adding a special case for zero blocks to the delta-transfer algorithm, as suggested in the Debian bug report, to improve the behavior for sparse files without the full CPU-time penalty of compression.
Comment 2 Martin Scharrer 2009-10-05 09:16:15 UTC
I would like to repeat my interest in such a feature. I frequently copying large sparse files over a not-so-fast network connections. At the moment I have to pre-create the destination files as large sparse files using an extra script.
In my opinion this should be done by rsync itself.
Comment 3 Matt McCutchen 2009-11-21 00:00:04 UTC
Fedora has a similar RFE:


There, the further suggestion is made that the receiver should be able to skip long runs of zeros without reconstructing them in memory.  A simple approach would be to skip one delta-transfer block at a time.  Or, if the protocol is changed (rather than just having the generator fabricate an all-zero block), it would be easy to add a token representing a run of zeros of arbitrary length.
Comment 4 grarpamp 2012-03-10 22:53:06 UTC
Maybe related:
Comment 5 Björn Jacke 2016-01-14 10:42:34 UTC
*** Bug 7854 has been marked as a duplicate of this bug. ***