I use rsync to create daily/weekly/monthly backups to an external USB drive of my Gentoo Linux using this command line options : --archive --delete --delete-excluded --exclude-from=/exclude.list --link-dest=<old dest> --stats --verbose <src> <dest> This works like a charm for all files located on my ext3 Linux partition. However for the 2 files of a NTFS drive not. Here rsync always transfers an unchanged file too, even if <old dest> contains that file (from a backup made yesterday) and therefore it should better be hard-linked only instead of being transferred twice. What is really curious - at the beginning of using this backup solution (some weeks ago) the <dest> doesn't contain an older version of that file, and there hard-linking worked IIRC. The NTFS drive is mounted read-only: tfoerste@n22 ~ $ mount | grep C /dev/sda1 on /mnt/C type ntfs (ro,noexec,nosuid,nodev,noatime,uid=1000,gid=100,umask=0027) <src> is /mnt/C/notes/data/archive/ <dest> is /mnt/media/daily/f/ <old dest> is /mnt/media/weekly/b (given as ../../weekly/b at the command line) Here you can see that different hard links are used : tfoerste@n22 ~ $ ls -il /mnt/C/notes/data/archive/l_DE1448.nsf /mnt/media/daily/f/archive/l_DE1448.nsf /mnt/media/weekly/b/archive/l_DE1448.nsf 80719 -r-xr-x--- 1 tfoerste users 8126464 Sep 2 13:40 /mnt/C/notes/data/archive/l_DE1448.nsf 2293796 -r-xr-x--- 1 tfoerste users 8126464 Sep 2 13:40 /mnt/media/daily/f/archive/l_DE1448.nsf 2310874 -r-xr-x--- 1 tfoerste users 8126464 Sep 2 13:40 /mnt/media/weekly/b/archive/l_DE1448.nsf md5sum proves identical content, the file attributes shouldn't changed too (NTFS is always mounted read-only, never read-write) : tfoerste@n22 ~ $ md5sum /mnt/C/notes/data/archive/l_DE1448.nsf /mnt/media/daily/f/archive/l_DE1448.nsf /mnt/media/weekly/b/archive/l_DE1448.nsf 88ef0d516c5696bf120d749f4b1fc9ba /mnt/C/notes/data/archive/l_DE1448.nsf 88ef0d516c5696bf120d749f4b1fc9ba /mnt/media/daily/f/archive/l_DE1448.nsf 88ef0d516c5696bf120d749f4b1fc9ba /mnt/media/weekly/b/archive/l_DE1448.nsf OTOH does it make any difference whether the <dest> drive at the external USB drive is mounted as ext2 instead of ext3 ?
Try running with --itemize-changes. It will tell you why rsync thinks the file needs to be updated. (Sorry for the duplicate reply on the email list. I replied there before I realized this was from the bugzilla)
Ah - now the pciture becomes clear - I changed my script few days ago to use relative paths - which doesn't work: When I specify : target dir: /mnt/media/daily/a ref dir: ../../daily/f it doesn't work, however this works : target dir: /mnt/media/daily/a ref dir: /mnt/media/daily/f OTOH the relative path itself is ok : n22 ~ # ls -ld /mnt/media/daily/a drwxr-xr-x 17 root root 4096 Aug 12 18:28 /mnt/media/daily/a n22 ~ # ls -ld /mnt/media/daily/a/../../daily/f drwxr-xr-x 17 root root 4096 Sep 12 17:22 /mnt/media/daily/a/../../daily/f which let me thinkig, that rsync appends the relative path to <dest> in this way (which is not intended, but I cannot specify an additional slash "/" - then it would be an absolute path) : n22 ~ # ls -ld /mnt/media/daily/a../../daily/f ls: cannot access /mnt/media/daily/a../../daily/f: No such file or directory
FWIW : I ofc tried to append a "/" to "target dir" but this issue still do exist.
Well, beside the trailing "/" I can reproduce, that rsyncing from the NTFS drive doesn't work in the same manner like rsyncing from my linux partition. I made a backup few minutes ago from /mnt/C into /mnt/media/daily/a : n22 ~ # ls -il /mnt/C/notes/data/archive/ total 1288960 80717 -r-xr-x--- 1 tfoerste users 1311768576 Sep 2 09:43 a_DE1448.nsf 80719 -r-xr-x--- 1 tfoerste users 8126464 Sep 2 13:40 l_DE1448.nsf n22 ~ # ls -il /mnt/media/daily/a/archive/* 4769048 -r-xr-x--- 1 tfoerste users 1311768576 Sep 2 09:43 /mnt/media/daily/a/archive/a_DE1448.nsf 73737 -r-xr-x--- 1 tfoerste users 8126464 Sep 2 13:40 /mnt/media/daily/a/archive/l_DE1448.nsf After that dry-run shows that rsync wants to transfer the NTFS files again :
<continuing from #comment 4> n22 ~ # rsync --archive --delete --delete-excluded --exclude-from=/exclude.list --itemize-changes --link-dest=/mnt/medi/daily/a --stats --verbose --dry-run /mnt/C/notes/data/archive /mnt/media/daily/b sending incremental file list >f.st...... archive/a_DE1448.nsf >f..t...... archive/l_DE1448.nsf Number of files: 3 Number of files transferred: 2 Well, the files in destination dir "b" has to be replaced: n22 ~ # ls -il /mnt/media/daily/b/archive/* 9306235 -r-xr-x--- 1 tfoerste users 1308622848 Aug 29 11:43 /mnt/media/daily/b/archive/a_DE1448.nsf 65673 -r-xr-x--- 1 tfoerste users 8126464 Aug 29 11:43 /mnt/media/daily/b/archive/l_DE1448.nsf but rather with a hard-link from "/a" instead copying it from the NTFS drive, isn't it ?
The itemized output is telling you that rsync is re-copying the files because there is a time stamp difference. When rsync is used locally it runs with --whole-file so it will re-copy the entire file because of the time stamp difference. But even if it wasn't running in --whole-file mode it can't store the same file with two different timestamps so it still wouldn't link to the previous version.
(In reply to comment #6) > The itemized output is telling you that rsync is re-copying the files because > there is a time stamp difference. The file(at the read-only mounted NTFS file system) wasn't changed between the 2 backup attempts - I did not boot Windows in the mean while, I only mounted the NTFS file system itself for the rsync backup and unmounted it afterwards.
Rsync is clearly seeing a times tamp difference on both files and a file size difference on one of them so obviously something changed. Forget using ls -l. Try using the stat command on the same file in each of the 3 paths (source, target, and link-dest).
(In reply to comment #8) > Forget using ls -l. Try using the stat command ok, here are the results : n22 ~ # stat /mnt/media/daily/[an]/archive/l* /mnt/C/notes/data/archive/l* File: `/mnt/media/daily/a/archive/l_DE1448.nsf' Size: 8126464 Blocks: 15896 IO Block: 4096 regular file Device: fe00h/65024d Inode: 73737 Links: 1 Access: (0550/-r-xr-x---) Uid: ( 1000/tfoerste) Gid: ( 100/ users) Access: 2011-09-13 13:29:58.000000000 +0200 Modify: 2011-09-02 13:40:33.000000000 +0200 Change: 2011-09-13 13:06:59.000000000 +0200 Birth: - File: `/mnt/media/daily/n/archive/l_DE1448.nsf' Size: 8126464 Blocks: 15896 IO Block: 4096 regular file Device: fe00h/65024d Inode: 65707 Links: 7 Access: (0550/-r-xr-x---) Uid: ( 1000/tfoerste) Gid: ( 100/ users) Access: 2011-08-15 18:16:00.000000000 +0200 Modify: 2011-08-15 10:03:05.000000000 +0200 Change: 2011-08-23 16:26:08.000000000 +0200 Birth: - File: `/mnt/C/notes/data/archive/l_DE1448.nsf' Size: 8126464 Blocks: 15872 IO Block: 512 regular file Device: 801h/2049d Inode: 80719 Links: 1 Access: (0550/-r-xr-x---) Uid: ( 1000/tfoerste) Gid: ( 100/ users) Access: 2011-09-02 13:40:33.281250000 +0200 Modify: 2011-09-02 13:40:33.281250000 +0200 Change: 2011-09-02 13:40:33.281250000 +0200 Birth: - n22 ~ # md5sum /mnt/media/daily/[an]/archive/l* /mnt/C/notes/data/archive/l* 88ef0d516c5696bf120d749f4b1fc9ba /mnt/media/daily/a/archive/l_DE1448.nsf 9d9e723cbeb841adf6552d94daa115b7 /mnt/media/daily/n/archive/l_DE1448.nsf 88ef0d516c5696bf120d749f4b1fc9ba /mnt/C/notes/data/archive/l_DE1448.nsf And rsync tries to do this : Tue Sep 13 18:17:51 CEST 2011 target dir: /mnt/media/daily/n ref dir: /mnt/media/daily/a (==link-dest directory) sending incremental file list >f.st...... archive/a_DE1448.nsf >f..t...... archive/l_DE1448.nsf
Gaaah - if I mount the NTFS drive with the fuse driver ntfs-3g instead of the kernel ntfs driver then it works ! The kernel drivers yields into some odds like this (first rsync gives an warning, 2nd immediate call is fine, but rm doesn't work): tfoerste@n22 ~ $ rsync --archive --itemize-changes /mnt/C/adsmcfg/ /tmp/rsync_test/a .d..tp..... ./ >f+++++++++ GetNwBat.CMD >f+++++++++ LastMonthChecked.txt rsync: mkstemp "/tmp/rsync_test/a/.GetNwBat.CMD.qz3uut" failed: Permission denied (13) rsync: mkstemp "/tmp/rsync_test/a/.LastMonthChecked.txt.xy5sgj" failed: Permission denied (13) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1052) [sender=3.0.8] tfoerste@n22 ~ $ rsync --archive --itemize-changes /mnt/C/adsmcfg/ /tmp/rsync_test/a >f+++++++++ GetNwBat.CMD >f+++++++++ LastMonthChecked.txt foerste@n22 ~ $ rm -rf /tmp/rsync_test/a/ rm: cannot remove `/tmp/rsync_test/a/GetNwBat.CMD': Permission denied rm: cannot remove `/tmp/rsync_test/a/LastMonthChecked.txt': Permission denied tfoerste@n22 ~ $ stat /tmp/rsync_test/a/* File: `/tmp/rsync_test/a/GetNwBat.CMD' Size: 3799 Blocks: 8 IO Block: 4096 regular file Device: 807h/2055d Inode: 950629 Links: 1 Access: (0550/-r-xr-x---) Uid: ( 1000/tfoerste) Gid: ( 100/ users) Access: 2011-09-13 19:45:22.000000000 +0200 Modify: 2005-08-17 11:25:57.000000000 +0200 Change: 2011-09-13 19:45:22.000000000 +0200 Birth: - File: `/tmp/rsync_test/a/LastMonthChecked.txt' Size: 5 Blocks: 8 IO Block: 4096 regular file Device: 807h/2055d Inode: 950630 Links: 1 Access: (0550/-r-xr-x---) Uid: ( 1000/tfoerste) Gid: ( 100/ users) Access: 2011-09-13 19:45:22.000000000 +0200 Modify: 2011-09-02 09:38:38.000000000 +0200 Change: 2011-09-13 19:45:22.000000000 +0200 Sry guys to bother you for a bug where the culprit might be somewhere else. Should I blame the kernel guys, but due to the kernel.org mess their bugzilla isn't reachable - or does ntfs-3g doesn't follow the spec ?
*** Bug 8456 has been marked as a duplicate of this bug. ***
Now I'm convinced that it has nothing to do with NTFS. *** This bug has been marked as a duplicate of bug 8456 ***