Bug 8450 - --link-dest seems not to work mounted NTFS file systems
Summary: --link-dest seems not to work mounted NTFS file systems
Status: RESOLVED DUPLICATE of bug 8456
Alias: None
Product: rsync
Classification: Unclassified
Component: core (show other bugs)
Version: 3.0.8
Hardware: x86 Linux
: P5 normal (vote)
Target Milestone: ---
Assignee: Wayne Davison
QA Contact: Rsync QA Contact
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-09-12 16:06 UTC by Toralf Förster
Modified: 2011-09-14 13:17 UTC (History)
0 users

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Toralf Förster 2011-09-12 16:06:13 UTC
I use rsync to create daily/weekly/monthly backups to an external USB drive of my Gentoo Linux using this command line options :

--archive --delete --delete-excluded --exclude-from=/exclude.list --link-dest=<old dest> --stats --verbose <src> <dest>

This works like a charm for all files located on my ext3 Linux partition. However for the 2 files of a NTFS drive not.

Here rsync always transfers an unchanged file too, even if <old dest> contains that file (from a backup made yesterday) and therefore it should better be hard-linked only instead of being transferred twice.

What is really curious - at the beginning of using this backup solution (some weeks ago) the <dest> doesn't contain an older version of that file, and there hard-linking worked IIRC.

The NTFS drive is mounted read-only:

tfoerste@n22 ~ $ mount | grep C
/dev/sda1 on /mnt/C type ntfs (ro,noexec,nosuid,nodev,noatime,uid=1000,gid=100,umask=0027)

<src>      is /mnt/C/notes/data/archive/
<dest>     is /mnt/media/daily/f/
<old dest> is /mnt/media/weekly/b (given as ../../weekly/b at the command line)

Here you can see that different hard links are used :

tfoerste@n22 ~ $ ls -il /mnt/C/notes/data/archive/l_DE1448.nsf /mnt/media/daily/f/archive/l_DE1448.nsf  /mnt/media/weekly/b/archive/l_DE1448.nsf 
  80719 -r-xr-x--- 1 tfoerste users 8126464 Sep  2 13:40 /mnt/C/notes/data/archive/l_DE1448.nsf
2293796 -r-xr-x--- 1 tfoerste users 8126464 Sep  2 13:40 /mnt/media/daily/f/archive/l_DE1448.nsf
2310874 -r-xr-x--- 1 tfoerste users 8126464 Sep  2 13:40 /mnt/media/weekly/b/archive/l_DE1448.nsf

md5sum proves identical content, the file attributes shouldn't changed too (NTFS is always mounted read-only, never read-write) :

tfoerste@n22 ~ $ md5sum /mnt/C/notes/data/archive/l_DE1448.nsf /mnt/media/daily/f/archive/l_DE1448.nsf  /mnt/media/weekly/b/archive/l_DE1448.nsf 
88ef0d516c5696bf120d749f4b1fc9ba  /mnt/C/notes/data/archive/l_DE1448.nsf
88ef0d516c5696bf120d749f4b1fc9ba  /mnt/media/daily/f/archive/l_DE1448.nsf
88ef0d516c5696bf120d749f4b1fc9ba  /mnt/media/weekly/b/archive/l_DE1448.nsf

OTOH does it make any difference whether the <dest> drive at the external USB drive is mounted as ext2 instead of ext3 ?
Comment 1 Kevin Korb 2011-09-12 16:11:17 UTC
Try running with --itemize-changes.  It will tell you why rsync thinks the file needs to be updated.

(Sorry for the duplicate reply on the email list.  I replied there before I realized this was from the bugzilla)
Comment 2 Toralf Förster 2011-09-12 16:45:32 UTC
Ah - now the pciture becomes clear - I changed my script few days ago to use relative paths - which doesn't work:

When I specify :

target dir: /mnt/media/daily/a
   ref dir:      ../../daily/f

it doesn't work, however this works :

target dir: /mnt/media/daily/a
   ref dir:      /mnt/media/daily/f


OTOH the relative path itself is ok :

n22 ~ # ls -ld /mnt/media/daily/a
drwxr-xr-x 17 root root 4096 Aug 12 18:28 /mnt/media/daily/a

n22 ~ # ls -ld /mnt/media/daily/a/../../daily/f
drwxr-xr-x 17 root root 4096 Sep 12 17:22 /mnt/media/daily/a/../../daily/f

which let me thinkig, that rsync appends the relative path to <dest> in this way (which is not intended, but I cannot specify an additional slash "/" - then it would be an absolute path) :

n22 ~ # ls -ld /mnt/media/daily/a../../daily/f
ls: cannot access /mnt/media/daily/a../../daily/f: No such file or directory
Comment 3 Toralf Förster 2011-09-13 06:54:29 UTC
FWIW : I ofc tried to append a "/" to "target dir" but this issue still do exist.
Comment 4 Toralf Förster 2011-09-13 11:25:31 UTC
Well, beside the trailing "/" I can reproduce, that rsyncing from the NTFS drive doesn't work in the same manner like rsyncing from my linux partition.

I made a backup few minutes ago from /mnt/C into /mnt/media/daily/a :

n22 ~ # ls -il /mnt/C/notes/data/archive/
total 1288960                                                                                                           
80717 -r-xr-x--- 1 tfoerste users 1311768576 Sep  2 09:43 a_DE1448.nsf                                                  
80719 -r-xr-x--- 1 tfoerste users    8126464 Sep  2 13:40 l_DE1448.nsf                                                  

n22 ~ # ls -il /mnt/media/daily/a/archive/*
4769048 -r-xr-x--- 1 tfoerste users 1311768576 Sep  2 09:43 /mnt/media/daily/a/archive/a_DE1448.nsf                     
  73737 -r-xr-x--- 1 tfoerste users    8126464 Sep  2 13:40 /mnt/media/daily/a/archive/l_DE1448.nsf                     


After that dry-run shows that rsync wants to transfer the NTFS files again :
Comment 5 Toralf Förster 2011-09-13 11:27:31 UTC
<continuing from #comment 4>

n22 ~ # rsync --archive --delete --delete-excluded --exclude-from=/exclude.list --itemize-changes --link-dest=/mnt/medi/daily/a --stats --verbose --dry-run /mnt/C/notes/data/archive /mnt/media/daily/b                                      
sending incremental file list                                                                                          
>f.st...... archive/a_DE1448.nsf                                                                                       
>f..t...... archive/l_DE1448.nsf                                                                                       
                                                                                                                       
Number of files: 3
Number of files transferred: 2



Well, the files in destination dir "b" has to be replaced:

n22 ~ # ls -il /mnt/media/daily/b/archive/*
9306235 -r-xr-x--- 1 tfoerste users 1308622848 Aug 29 11:43 /mnt/media/daily/b/archive/a_DE1448.nsf
  65673 -r-xr-x--- 1 tfoerste users    8126464 Aug 29 11:43 /mnt/media/daily/b/archive/l_DE1448.nsf                    

but rather with a hard-link from "/a" instead copying it from the NTFS drive, isn't it ?
Comment 6 Kevin Korb 2011-09-13 12:21:21 UTC
The itemized output is telling you that rsync is re-copying the files because there is a time stamp difference.  When rsync is used locally it runs with --whole-file so it will re-copy the entire file because of the time stamp difference.  But even if it wasn't running in --whole-file mode it can't store the same file with two different timestamps so it still wouldn't link to the previous version.
Comment 7 Toralf Förster 2011-09-13 14:48:27 UTC
(In reply to comment #6)
> The itemized output is telling you that rsync is re-copying the files because
> there is a time stamp difference.
The file(at the read-only mounted NTFS file system)  wasn't changed between the 2 backup attempts - I did not boot Windows in the mean while, I only mounted the NTFS file system itself for the rsync backup and unmounted it afterwards.
Comment 8 Kevin Korb 2011-09-13 15:47:34 UTC
Rsync is clearly seeing a times tamp difference on both files and a file size difference on one of them so obviously something changed.

Forget using ls -l.  Try using the stat command on the same file in each of the 3 paths (source, target, and link-dest).
Comment 9 Toralf Förster 2011-09-13 16:23:57 UTC
(In reply to comment #8)
> Forget using ls -l.  Try using the stat command 

ok, here are the results :

n22 ~ # stat /mnt/media/daily/[an]/archive/l* /mnt/C/notes/data/archive/l*                                              
  File: `/mnt/media/daily/a/archive/l_DE1448.nsf'                                                                       
  Size: 8126464         Blocks: 15896      IO Block: 4096   regular file                                                
Device: fe00h/65024d    Inode: 73737       Links: 1                                                                     
Access: (0550/-r-xr-x---)  Uid: ( 1000/tfoerste)   Gid: (  100/   users)                                                
Access: 2011-09-13 13:29:58.000000000 +0200                                                                             
Modify: 2011-09-02 13:40:33.000000000 +0200                                                                             
Change: 2011-09-13 13:06:59.000000000 +0200                                                                             
 Birth: -                                                                                                               
  File: `/mnt/media/daily/n/archive/l_DE1448.nsf'                                                                       
  Size: 8126464         Blocks: 15896      IO Block: 4096   regular file                                                
Device: fe00h/65024d    Inode: 65707       Links: 7                                                                     
Access: (0550/-r-xr-x---)  Uid: ( 1000/tfoerste)   Gid: (  100/   users)                                                
Access: 2011-08-15 18:16:00.000000000 +0200                                                                             
Modify: 2011-08-15 10:03:05.000000000 +0200                                                                             
Change: 2011-08-23 16:26:08.000000000 +0200
 Birth: -
  File: `/mnt/C/notes/data/archive/l_DE1448.nsf'
  Size: 8126464         Blocks: 15872      IO Block: 512    regular file
Device: 801h/2049d      Inode: 80719       Links: 1
Access: (0550/-r-xr-x---)  Uid: ( 1000/tfoerste)   Gid: (  100/   users)
Access: 2011-09-02 13:40:33.281250000 +0200
Modify: 2011-09-02 13:40:33.281250000 +0200
Change: 2011-09-02 13:40:33.281250000 +0200
 Birth: -


n22 ~ # md5sum /mnt/media/daily/[an]/archive/l* /mnt/C/notes/data/archive/l*
88ef0d516c5696bf120d749f4b1fc9ba  /mnt/media/daily/a/archive/l_DE1448.nsf
9d9e723cbeb841adf6552d94daa115b7  /mnt/media/daily/n/archive/l_DE1448.nsf
88ef0d516c5696bf120d749f4b1fc9ba  /mnt/C/notes/data/archive/l_DE1448.nsf

And rsync tries to do this :

Tue Sep 13 18:17:51 CEST 2011
target dir: /mnt/media/daily/n
   ref dir: /mnt/media/daily/a    (==link-dest directory)

sending incremental file list
>f.st...... archive/a_DE1448.nsf
>f..t...... archive/l_DE1448.nsf
Comment 10 Toralf Förster 2011-09-13 18:02:13 UTC
Gaaah -

if I mount the NTFS drive with the fuse driver ntfs-3g instead of the kernel ntfs driver then it works !



The kernel drivers yields into some odds like this (first rsync gives an warning, 2nd immediate call is fine, but rm doesn't work):


tfoerste@n22 ~ $ rsync --archive --itemize-changes /mnt/C/adsmcfg/ /tmp/rsync_test/a
.d..tp..... ./
>f+++++++++ GetNwBat.CMD
>f+++++++++ LastMonthChecked.txt
rsync: mkstemp "/tmp/rsync_test/a/.GetNwBat.CMD.qz3uut" failed: Permission denied (13)
rsync: mkstemp "/tmp/rsync_test/a/.LastMonthChecked.txt.xy5sgj" failed: Permission denied (13)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1052) [sender=3.0.8]

tfoerste@n22 ~ $ rsync --archive --itemize-changes /mnt/C/adsmcfg/ /tmp/rsync_test/a
>f+++++++++ GetNwBat.CMD
>f+++++++++ LastMonthChecked.txt

foerste@n22 ~ $ rm -rf /tmp/rsync_test/a/
rm: cannot remove `/tmp/rsync_test/a/GetNwBat.CMD': Permission denied
rm: cannot remove `/tmp/rsync_test/a/LastMonthChecked.txt': Permission denied

tfoerste@n22 ~ $ stat /tmp/rsync_test/a/*
  File: `/tmp/rsync_test/a/GetNwBat.CMD'
  Size: 3799            Blocks: 8          IO Block: 4096   regular file
Device: 807h/2055d      Inode: 950629      Links: 1
Access: (0550/-r-xr-x---)  Uid: ( 1000/tfoerste)   Gid: (  100/   users)
Access: 2011-09-13 19:45:22.000000000 +0200
Modify: 2005-08-17 11:25:57.000000000 +0200
Change: 2011-09-13 19:45:22.000000000 +0200
 Birth: -
  File: `/tmp/rsync_test/a/LastMonthChecked.txt'
  Size: 5               Blocks: 8          IO Block: 4096   regular file
Device: 807h/2055d      Inode: 950630      Links: 1
Access: (0550/-r-xr-x---)  Uid: ( 1000/tfoerste)   Gid: (  100/   users)
Access: 2011-09-13 19:45:22.000000000 +0200
Modify: 2011-09-02 09:38:38.000000000 +0200
Change: 2011-09-13 19:45:22.000000000 +0200


Sry guys to bother you for a bug where the culprit might be somewhere else.
Should I blame the kernel guys, but due to the kernel.org mess their bugzilla isn't reachable - or does ntfs-3g doesn't follow the spec ?
Comment 11 Toralf Förster 2011-09-14 13:16:18 UTC
*** Bug 8456 has been marked as a duplicate of this bug. ***
Comment 12 Toralf Förster 2011-09-14 13:17:43 UTC
Now I'm convinced that it has nothing to do with NTFS.

*** This bug has been marked as a duplicate of bug 8456 ***