Bug 13423 - Checksum option does not work as expected when append-verify is used
Summary: Checksum option does not work as expected when append-verify is used
Status: RESOLVED WONTFIX
Alias: None
Product: rsync
Classification: Unclassified
Component: core (show other bugs)
Version: 3.1.3
Hardware: All All
: P5 normal (vote)
Target Milestone: ---
Assignee: Wayne Davison
QA Contact: Rsync QA Contact
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-05-08 12:14 UTC by dariuszb
Modified: 2020-07-27 22:35 UTC (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description dariuszb 2018-05-08 12:14:45 UTC
Checksum option does not work as expected when append-verify is used. I can see that checksums are created for all files but they are not synced.




rsync sample files to remote site:

# rsync -azhcv --append-verify  --progress --stats .  rsyncclient@192.168.142.2::DBTest/DBtest/
sending incremental file list
./
Firefox 60.0.dmg
         55.37M 100%    8.40MB/s    0:00:06 (xfr#1, to-chk=3/6)
Skype-8.19.0.1.dmg
         92.68M 100%    8.35MB/s    0:00:10 (xfr#2, to-chk=2/6)
SuperDuper!-2.dmg
          6.09M 100%    5.16MB/s    0:00:01 (xfr#3, to-chk=1/6)
SuperDuper!.dmg
          6.09M 100%    7.20MB/s    0:00:00 (xfr#4, to-chk=0/6)

Number of files: 6 (reg: 5, dir: 1)
Number of created files: 4 (reg: 4)
Number of deleted files: 0
Number of regular files transferred: 4
Total file size: 160.23M bytes
Total transferred file size: 160.23M bytes
Literal data: 160.23M bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.388 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 159.83M
Total bytes received: 95

sent 159.83M bytes  received 95 bytes  8.20M bytes/sec
total size is 160.23M  speedup is 1.00


Now i use hexcurse and change one of the source files - just flip some bytes.

As -c option is present I would expect that rsync detects that one file has changed and updates it accordingly. I run rsync again - no files are synced - I can confirm it by manually checking files' hashes. 

# rsync -azhcv --append-verify  --progress --stats .  rsyncclient@192.168.142.2::DBTest/DBtest/
sending incremental file list

Number of files: 6 (reg: 5, dir: 1)
Number of created files: 0
Number of deleted files: 0
Number of regular files transferred: 0
Total file size: 160.23M bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.402 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 272
Total bytes received: 12

sent 272 bytes  received 12 bytes  81.14 bytes/sec
total size is 160.23M  speedup is 564,202.91



I remove --append-verify option and run rsync again

# rsync -azhcv  --progress --stats .  rsyncclient@192.168.142.2::DBTest/DBtest/
sending incremental file list
Firefox 60.0.dmg
         55.37M 100%  101.92MB/s    0:00:00 (xfr#1, to-chk=3/6)

Number of files: 6 (reg: 5, dir: 1)
Number of created files: 0
Number of deleted files: 0
Number of regular files transferred: 1
Total file size: 160.23M bytes
Total transferred file size: 55.37M bytes
Literal data: 7.44K bytes
Matched data: 55.36M bytes
File list size: 0
File list generation time: 0.390 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 7.44K
Total bytes received: 52.14K

sent 7.44K bytes  received 52.14K bytes  7.94K bytes/sec
total size is 160.23M  speedup is 2,689.70


This time files were synced as expected
Comment 1 Kevin Korb 2018-05-08 12:29:06 UTC
If the file has not grown then there is nothing for --append[-verify] to do.  Also, without --itemize-changes you have no reporting from --checksum.  You probably shouldn't have either option.
Comment 2 dariuszb 2018-05-08 12:32:48 UTC
Yes it is clear that --append-verify will not update the same size files.

But --checksum should check hashes of all files and trigger update if different.

What is happening here looks like append-verify overwrites checksum functionality.
Comment 3 Kevin Korb 2018-05-08 12:52:13 UTC
From man rsync --append:
> If a file needs to be transferred and its size on the receiver is the same or longer than the size on the sender, the file is skipped.

--append-verify does the verify AFTER an append happens so if --append won't copy the file neither will --append-verify.  Since --append essentially means only care if the source file is bigger the file is ignored.

The combination is why I said you probably don't want either option and especially not both options.  With both, all files on both ends are checksummed first (even files that are only on one side!)  Then files that are bigger on the source are appended to.  Then the files are verified.  Then if the verification fails rsync throws a warning and redoes the whole file --inplace style.

Normally I recommend against --checksum but I have no Mac experience and I don't really know how your disk image files work.  In many cases it is actually faster to use --ignore-times (delta-xfer everything regardless of timestamp) and maybe --inplace instead of --checksum.  Usually --checksum is used when you can't afford to write out a new file because of limited/slow writes or because you are retaining the old version and don't want a new version unless it is actually new.
Comment 4 dariuszb 2018-05-08 13:01:06 UTC
Thank you for clarification.
Comment 5 Wayne Davison 2020-07-27 22:35:40 UTC
Combining --append with --checksum doesn't really make any sense, since --append is a very special-case option for the transfer of growing files, and you should not be including non-growing files that have differing content in them within such a transfer.