The Samba-Bugzilla – Bug 11609
Incorrect (or at least dangerous) behaviour of --append-verify
Last modified: 2015-11-22 18:38:33 UTC
This has been reported at the MacPorts bug tracker already (https://trac.macports.org/ticket/49657), as I first discovered this using rsync 3.1.1 from MacPorts, on OS X 10.11.1; but I'm told that this needs to be reported upstream.
As far as I can understand, one use of --append-verify is that in case of interruption, rsync will leave the destination file in its partial state so that it can be resumed at a later point.
Consider, for example:
source volume: "/Volume/DyingHDD"
destination volume: "/Volume/NewHDD"
Suppose I have a 2 GB file "/Volume/DyingHDD/00027.MTS" and my DyingHDD is really dying: its read speed has degraded to around 5 MB/s and it's so unstable that it can't read 10 GB without suddenly going offline (OS X would think that it has not been ejected cleanly).
So I try to salvage whatever is on DyingHDD by doing "rsync -av --append-verify --timeout=60 /Volume/DyingHDD/ /Volume/NewHDD/". (You may point out that ddrescue should be a better tool for the job, but that's beside the point.) Now, a few hundred MBs into 00027.MTS, DyingHDD suddenly disappears. I would expect rsync to leave "/Volume/NewHDD/00027.MTS" in its partial state, at the few hundred MB mark, and quit.
But that's not what happened. Instead, rsync proceeded to fill "/Volume/NewHDD/00027.MTS" all the way to 2 GB at the full speed of NewHDD (about 150 MB/s), and leave its timestamp at the current system time. Whatever rsync filled the end of the file with must have been garbage (there is no way the OS could have cached the whole source file in RAM).
So the next time I call "rsync -av --append-verify --timeout=60 /Volume/DyingHDD/ /Volume/NewHDD/", rsync just skips over 00027.MTS.
--partial has a similar behaviour
I have been able to reproduce the same behaviour in a Debian 8 VM in VirtualBox:
1. I put a read-only SD card on my Mac;
2. I shared it across to the Debian 8 VM using VirtualBox's shared folder function;
3. I then started rsync in the Debian 8 VM to copy some 2GB file from the SD card;
4. A few hundred MBs in, I yanked the SD card from my Mac;
5. rsync proceeded to fill the destination file with garbage;
Output from rsync:
root@debian:/media/sf_PRIVATE/AVCHD/BDMV/STREAM# rsync -av --progress --append-verify --timeout=60 00013.MTS /media/pycon/3d79a203-ca8f-4b89-98f2-fafa39898aa2/sending incremental file list
2,125,037,568 100% 144.59MB/s 0:00:14 (xfr#1, to-chk=0/1)
rsync: read errors mapping "/media/sf_PRIVATE/AVCHD/BDMV/STREAM/00013.MTS": Protocol error (71)
WARNING: 00013.MTS failed verification -- update retained (will try again).
file has vanished: "/media/sf_PRIVATE/AVCHD/BDMV/STREAM/00013.MTS"
sent 2,125,556,484 bytes received 922,216 bytes 94,510,164.44 bytes/sec
total size is 2,125,037,568 speedup is 1.00
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1183) [sender=3.1.1]
root@debian:/media/sf_PRIVATE/AVCHD/BDMV/STREAM# ls -l /media/pycon/3d79a203-ca8f-4b89-98f2-fafa39898aa2/00013.MTS
-rwxrwx--- 1 root vboxsf 2125037568 Nov 18 22:51 /media/pycon/3d79a203-ca8f-4b89-98f2-fafa39898aa2/00013.MTS
And the next time I call rsync again:
root@debian:/media/sf_PRIVATE/AVCHD/BDMV/STREAM# rsync -av --progress --append-verify --timeout=60 00013.MTS /media/pycon/3d79a203-ca8f-4b89-98f2-fafa39898aa2/
sending incremental file list
sent 53 bytes received 16 bytes 138.00 bytes/sec
total size is 2,125,037,568 speedup is 30,797,645.91
rsync just skipped over the file.