I created a small patch to get --inplace and --sparse to work together. The main problem with this is, as far as I can see, that the write code just skips blocks of zeroes instead of writing them, as is requested by the --sparse switch. If the target already exists and has data in places where the source has zeroes, this data will never get erased. My patch simply reads these regions first and makes sure they really are zero. If they are not, it clears them. It would be good if someone with a deeper knowledge of the implications could have a look at the patch to see if I missed something.
Created attachment 5440 [details] Patch as described
https://bugzilla.samba.org/show_bug.cgi?id=7778 might be related
Created attachment 6065 [details] updated patch This updated patch contains a small bugfix. With this version we have successfully copied many TB of data snapshot-by-snapshot and verified the result with gdiff.
An improvement like this seems good for 3.1.0. Your patch looks nice, but I do want to make sure that it is leveraging knowledge from the generator -- if we're doing a normal (not whole-file) transfer, then a seek over identical bytes doesn't need to read the bytes again. Does you patch take that into consideration? Also, there's already a change in the 3.1.0dev code in sparse_end() that replaces that write-a-0-at-the-end-of-the-file code with an ftruncate() call (if ftruncate() is available), which will avoid allocating an extra block at the end of the file to hold the 0.
I assume that when write_file is called, the receiver already has determined that the buffer has to be written to disk. In case of inplace, we cannot assume that the file contains zeros in this location, nor it's sparse. So we need to make sure it really contains what we require. I don't see how any information from the generator can help to avoid this step. Can you please elaborate on your question?
Is it ready to be pulled into 3.1, or do you need anything else?
Wayne?
Is there any update available on what's stopping this patch being merged? It would be very useful...
This was fixed in an earlier release.