Bug 1489 - Corrupt transfer with the fuzzy option
Corrupt transfer with the fuzzy option
Product: rsync
Classification: Unclassified
Component: core
All Linux
: P3 normal
: ---
Assigned To: Wayne Davison
Rsync QA Contact
Depends on:
  Show dependency treegraph
Reported: 2004-06-29 06:38 UTC by Egmont Koblinger
Modified: 2005-03-16 16:48 UTC (History)
0 users

See Also:


Note You need to log in before you can comment on or make changes to this bug.
Description Egmont Koblinger 2004-06-29 06:38:12 UTC
I've been using the rusty-fuzzy patch of rsync for a long time without problems,
but now I've found a special circumstance where this option of the patched
rsync 2.6.2 leads to data corruption.

How to reproduce:

Create a /tmp/foo1 directory with four files.
foobar-1.0.txt contains one megabyte of '1' and then one megabyte of '0' chars.
A possible way to create this file:
{ dd if=/dev/zero bs=1k count=1k | tr '\0' '1'; dd if=/dev/zero bs=1k count=1k | 
tr '\0' '0'; } > /tmp/foo1/foobar-1.0.txt
Then create foobar-2.0.txt which has a megabyte of '2' and then a meg of '0'.
Similarly create foobar-2.1.txt which has lots of '2' and then lots of '1'
digits. (So the contents of these three files reflect their filename.)

The md5sums are:
f6535bdc24b1074a704ef0166f93b4f0  foobar-1.0.txt
a1574b29877c570cb44436f7a404aa71  foobar-2.0.txt
80c3df2d255f201f842a89ebcb3f078c  foobar-2.1.txt

And let's create a zzz.txt with any content, it doesn't influence anything,
only makes the example better.

Create /tmp/foo2, cp -a /tmp/foo1/foobar-1.0.txt /tmp/foo2/  but other files
should not yet be copied.

Here we go (bwlimit is not important):
$ rsync -a --fuzzy --bwlimit=100 localhost:/tmp/foo1/ /tmp/foo2/
receiving file list ... done

wrote 46464 bytes  read 3155804 bytes  98531.32 bytes/sec
total size is 6291460  speedup is 1.96

As the 'screenshot' shows foobar-2.1.txt is transferred two times. If rsync
is not interrupted then at the end foobar-2.1.txt is okay.

However, after the first transfer its content is invalid (2096928 '2' followed
by 224 '1' chars, md5sum is 795cb4c484c711d902acd3011e70832e). The file size
and the timestamp are correct.

Hence if either rsync is interrupted or someone else mirrors us (using rsync
without -c) during this process, the result will be a file with incorrect
content but correct metadata and so further rsyncing will not repair it.

First note: it seems based on this example that rsync has some self-protecting
mechanism (a stupid program would most likely not even notice that the transfer
was incorrect and wouldn't restart it). However, this way this self-protecting
mechanism isn't really perfect. It should check whether the whole file is okay
before renaming it to its final name so that an interrupt cannot leave a corrupt
file on the disk. If it is not trivial to solve due to some technical
difficulties, then at least the time stamp should be set to some fake value to
force a recheck of this file if rsync gets interrupted.

Second note: I think rsync --fuzzy starts to misbehave when the closest filename 
changes during the operation. In my example, initially foobar-1.0.txt was the
closest filename to foobar-2.1.txt, however, during the operation,
foobar-2.0.txt has appeared which is even closer to foobar-2.1.txt. Somehow
I guess rsync cannot clearly decide which one of these two files to use as the
reference, and this might be the cause of the problem.
Comment 1 Wayne Davison 2004-06-29 09:37:07 UTC
The problem is not directly caused by the --fuzzy option, since you would
probably see the same result if you copied the file that --fuzzy chose to use as
the basis for the transfer of the foobar-2.1.txt file (I assume foobar-2.0.txt)
to the destination under the name foobar-2.1.txt and did a regular copy.

The problem is that rsync used to link a corrupt file into place when it failed
the CRC and needed to be re-transferred.  This is no longer the case in CVS, so
this is fixed.