Bug 7683 - New rsync deletes destination files it shouldn't
New rsync deletes destination files it shouldn't
Status: RESOLVED INVALID
Product: rsync
Classification: Unclassified
Component: core
3.0.7
x86 Linux
: P3 critical
: ---
Assigned To: Wayne Davison
Rsync QA Contact
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2010-09-16 09:42 UTC by Vaskor Basak
Modified: 2010-09-18 08:51 UTC (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Vaskor Basak 2010-09-16 09:42:39 UTC
I have just upgraded to Ubuntu 10.04.1 (from 8.04, I think). I previously used the rsync program regularly to backup files incrementally between locally mounted Windows type filesystems (hard disks and memory pen etc.) without problems. However, when I do a trial run using the -n parameter on the upgraded system, the computer deletes a large number of files on the destination filesystem (and does not evidently recopy these across).

I have copied and pasted some output below to demonstrate the issue. The first command shows the execution of rsync on a small sample of the original fileset and the output shows the files being deleted (in -n trial mode). The 'ls' command next shows the contents of the "New recovery" source folder to show that the roa.doc file does indeed exist and shouldn't have been deleted on the destination without being recopied. Finally, I have just run the rsync command on its own to show the version number.

Interestingly, I have copied an old version on the rsync program from a old Linux CD and it exhibits the same problem running under the new Ubuntu, so it may be an issue relating to the way rsync reads a VFAT filesystem.

vb@celery:~$ sudo rsync --modify-window=2 -r -u -t -n --delete-during --progress /media/sda1/My\ Documents/Vaskor/Tasks /media/4BEF-DCDA/Vaskor
sending incremental file list
Tasks/
deleting Tasks/cleanvir.bat
deleting Tasks/CPD Technical Updates 2011 - Topics nomination form.doc
Tasks/Moringa-Brochure-Anstiss.pdf
Tasks/Orange free SIM with £5 credit.htm
Tasks/Tasks.txt
Tasks/Time.txt
Tasks/attachments_18_07_2010.zip
Tasks/benefits.txt
Tasks/benefits.txt~
Tasks/check address.TXT
Tasks/chris-howard-bwb.pdf
Tasks/phone_usa.jsp.html
Tasks/ppp36.pdf
Tasks/preaching peace.pdf
Tasks/spam2.txt
Tasks/statistics.txt
Tasks/tci-guide-to-investments.pdf
Tasks/tiscali.ins
Tasks/trendexplosion.pdf
Tasks/tripletrend34 - bad.zip
Tasks/New recovery/
deleting Tasks/New recovery/roa.doc
deleting Tasks/New recovery/3-s098~1.jpg
deleting Tasks/New recovery/2/annual~1.doc
deleting Tasks/Taken on disk/Permanent/friends.txt

sent 20578 bytes  received 134 bytes  13808.00 bytes/sec
total size is 124405234  speedup is 6006.43 (DRY RUN)
vb@celery:~$ ls /media/sda1/My\ Documents/Vaskor/Tasks/New\ recovery
1  Careers.zip  desktop.ini  New Folder  Thumbs.db
2  CV.doc       linux        ROA.DOC     Vaskor.zip
vb@celery:~$ rsync
rsync  version 3.0.7  protocol version 30
Copyright (C) 1996-2009 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 32-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
    append, ACLs, xattrs, iconv, symtimes
Comment 1 Paul Slootman 2010-09-16 10:21:06 UTC
> ... The 'ls'
> command next shows the contents of the "New recovery" source folder to show
> that the roa.doc file does indeed exist ...

roa.doc != ROA.DOC
Comment 2 Vaskor Basak 2010-09-16 10:37:12 UTC
I agree, but shouldn't rsync attempt to copy across ROA.DOC after deleting roa.doc in that case?

(In reply to comment #1)
> > ... The 'ls'
> > command next shows the contents of the "New recovery" source folder to show
> > that the roa.doc file does indeed exist ...
> roa.doc != ROA.DOC

Comment 3 Paul Slootman 2010-09-16 10:43:57 UTC
I expect that rsync has also checked for "DOA.DOC" and has determined that it
exists and is identical to the source. After that a check for extraneous files
(by reading the directory listing) shows a "doa.doc", so that gets deleted.

Just make sure you use a case-preserving filesystem.
Comment 4 Matt McCutchen 2010-09-16 16:08:12 UTC
(In reply to comment #3)
> I expect that rsync has also checked for "DOA.DOC" and has determined that it
> exists and is identical to the source. After that a check for extraneous files
> (by reading the directory listing) shows a "doa.doc", so that gets deleted.

That analysis is almost correct.  The delete mode is --delete-during, so rsync considers "doa.doc" for deletion before it considers "DOA.DOC" for transfer.  The reason why a transfer of "DOA.DOC" is not itemized is that on a dry run, "doa.doc" is not actually deleted, and rsync has no way to anticipate that deleting it would cause "DOA.DOC" to no longer exist on the destination.  On a real run, rsync should transfer "DOA.DOC".

The main version of rsync currently does not claim to support case-insensitive filesystems (there is a maintained patch "ignore-case.diff" that adds the functionality), so this report is invalid.

Vaskor, the options I see for you are to make a one-time change to harmonize the filenames (with rsync or otherwise), use the ignore-case patch, or fiddle with the "shortname" mount option of the source and/or destination filesystem to try to make the discrepancy in the names go away.
Comment 5 Vaskor Basak 2010-09-18 08:51:22 UTC
(In reply to comment #4)

I actually prefer the ext3 filesystem, but the filesystems in question needed to be readable by Windows computers.

I assumed by DOA.DOC you were referring to ROA.DOC (or perhaps you were suggesting that the file was "Dead On Arrival"!)

Many thanks for the suggestions, which has resolved my issue - I wish commercial software was as well supported!