Bug 11609 - Incorrect (or at least dangerous) behaviour of --append-verify
Summary: Incorrect (or at least dangerous) behaviour of --append-verify
Alias: None
Product: rsync
Classification: Unclassified
Component: core (show other bugs)
Version: 3.1.1
Hardware: All All
: P5 normal (vote)
Target Milestone: ---
Assignee: Wayne Davison
QA Contact: Rsync QA Contact
Depends on:
Reported: 2015-11-18 14:59 UTC by Kal Sze
Modified: 2020-07-27 23:46 UTC (History)
1 user (show)

See Also:


Note You need to log in before you can comment on or make changes to this bug.
Description Kal Sze 2015-11-18 14:59:10 UTC
This has been reported at the MacPorts bug tracker already (https://trac.macports.org/ticket/49657), as I first discovered this using rsync 3.1.1 from MacPorts, on OS X 10.11.1; but I'm told that this needs to be reported upstream.

As far as I can understand, one use of --append-verify is that in case of interruption, rsync will leave the destination file in its partial state so that it can be resumed at a later point.

Consider, for example:

source volume: "/Volume/DyingHDD"
destination volume: "/Volume/NewHDD"
Suppose I have a 2 GB file "/Volume/DyingHDD/00027.MTS" and my DyingHDD is really dying: its read speed has degraded to around 5 MB/s and it's so unstable that it can't read 10 GB without suddenly going offline (OS X would think that it has not been ejected cleanly).

So I try to salvage whatever is on DyingHDD by doing "rsync -av --append-verify --timeout=60 /Volume/DyingHDD/ /Volume/NewHDD/". (You may point out that ddrescue should be a better tool for the job, but that's beside the point.) Now, a few hundred MBs into 00027.MTS, DyingHDD suddenly disappears. I would expect rsync to leave "/Volume/NewHDD/00027.MTS" in its partial state, at the few hundred MB mark, and quit.

But that's not what happened. Instead, rsync proceeded to fill "/Volume/NewHDD/00027.MTS" all the way to 2 GB at the full speed of NewHDD (about 150 MB/s), and leave its timestamp at the current system time. Whatever rsync filled the end of the file with must have been garbage (there is no way the OS could have cached the whole source file in RAM).

So the next time I call "rsync -av --append-verify --timeout=60 /Volume/DyingHDD/ /Volume/NewHDD/", rsync just skips over 00027.MTS.

--partial has a similar behaviour

I have been able to reproduce the same behaviour in a Debian 8 VM in VirtualBox:
1. I put a read-only SD card on my Mac;
2. I shared it across to the Debian 8 VM using VirtualBox's shared folder function;
3. I then started rsync in the Debian 8 VM to copy some 2GB file from the SD card;
4. A few hundred MBs in, I yanked the SD card from my Mac;
5. rsync proceeded to fill the destination file with garbage;

Output from rsync:
root@debian:/media/sf_PRIVATE/AVCHD/BDMV/STREAM# rsync -av --progress --append-verify --timeout=60 00013.MTS /media/pycon/3d79a203-ca8f-4b89-98f2-fafa39898aa2/sending incremental file list
  2,125,037,568 100%  144.59MB/s    0:00:14 (xfr#1, to-chk=0/1)
rsync: read errors mapping "/media/sf_PRIVATE/AVCHD/BDMV/STREAM/00013.MTS": Protocol error (71)
WARNING: 00013.MTS failed verification -- update retained (will try again).
file has vanished: "/media/sf_PRIVATE/AVCHD/BDMV/STREAM/00013.MTS"

sent 2,125,556,484 bytes  received 922,216 bytes  94,510,164.44 bytes/sec
total size is 2,125,037,568  speedup is 1.00
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1183) [sender=3.1.1]

root@debian:/media/sf_PRIVATE/AVCHD/BDMV/STREAM# ls -l /media/pycon/3d79a203-ca8f-4b89-98f2-fafa39898aa2/00013.MTS 
-rwxrwx--- 1 root vboxsf 2125037568 Nov 18 22:51 /media/pycon/3d79a203-ca8f-4b89-98f2-fafa39898aa2/00013.MTS

And the next time I call rsync again:
root@debian:/media/sf_PRIVATE/AVCHD/BDMV/STREAM# rsync -av --progress --append-verify --timeout=60 00013.MTS /media/pycon/3d79a203-ca8f-4b89-98f2-fafa39898aa2/
sending incremental file list

sent 53 bytes  received 16 bytes  138.00 bytes/sec
total size is 2,125,037,568  speedup is 30,797,645.91

rsync just skipped over the file.
Comment 1 Wayne Davison 2020-07-27 23:46:03 UTC
The use of --append-verify is not generally recommended, as log files are better served by --append (when you know what you're transferring) and other files are better served by --inplace.  In your case, you should probably have just used --inplace.