Bug 6276 - crtimes.patch does not preserve creation dates on Mac x86_64 only
Summary: crtimes.patch does not preserve creation dates on Mac x86_64 only
Status: RESOLVED FIXED
Alias: None
Product: rsync
Classification: Unclassified
Component: core (show other bugs)
Version: 3.0.6
Hardware: x86 Mac OS X
: P3 normal (vote)
Target Milestone: ---
Assignee: Wayne Davison
QA Contact: Rsync QA Contact
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2009-04-18 16:02 UTC by Pepi Zawodsky
Modified: 2010-02-11 12:19 UTC (History)
0 users

See Also:


Attachments
Output of rsync log when testing with backup bouncer 0.1.3 rsync 3.0.6 for crtimes (11.25 KB, text/plain)
2009-05-10 14:58 UTC, Pepi Zawodsky
no flags Details
Output of backup bouncer 0.1.3 when testing rsync 3.0.6 for crtimes problem (2.41 KB, text/plain)
2009-05-10 14:59 UTC, Pepi Zawodsky
no flags Details
patch to fix creation dates bug on Mac x86_64 (1011 bytes, patch)
2010-02-06 01:14 UTC, Steve Ortiz
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Pepi Zawodsky 2009-04-18 16:02:19 UTC
Summary:

rsync with fileflags and crtimes patches does not preserve creation date/time on Mac OS X 10.5.6 under x86_64 when compiled as x86_64.

I'm compiling rsync 3.0.6-pre1 (and 3.0.5) on Mac OS X Leopard 10.5.6 (9G2141) with gcc version 4.0.1 (Apple Inc. build 5490). I'm creating a 4 way universal binary for ppc, ppc64, i386 and x86_64.

crtimes patch works fine for 3.0.5 and 3.0.6-pre1 on ppc, ppc64 and i386.

However rsync does not preserve creation dates on Mac OS X when run as x86_64 binary. Running the same rsync with the same patch created as i386 (32 bit binary) DOES correctly preserve creation dates.


This is how I build rsync 3.0.6-pre1 as a 4 way universal binary:

patch -p1 <../patches/fileflags.diff
./prepare-source

patch -p1 <../patches/crtimes.diff

./configure --prefix=/usr/local/maclemon-beta --disable-debug CC="gcc -std=gnu99 -mmacosx-version-min=10.4 -isysroot /Developer/SDKs/MacOSX10.4u.sdk" CFLAGS="-g -fprefetch-loop-arrays -funroll-loops -O3 -DHAVE_CONFIG_H -Wall -W -I./popt -isysroot /Developer/SDKs/MacOSX10.4u.sdk -arch ppc -arch i386 -arch ppc64 -arch x86_64" LDFLAGS="-mmacosx-version-min=10.4 -arch ppc -arch i386 -arch ppc64 -arch x86_64"

make

Result:
MD5 (rsync) = 9cc18a386a256f6aecd9c46815088594
SHA1(rsync)= d4c9b32e088208e12821f96117b632f20750e08f
rsync: Mach-O universal binary with 4 architectures
rsync (for architecture ppc):	Mach-O executable ppc
rsync (for architecture i386):	Mach-O executable i386
rsync (for architecture ppc64):	Mach-O 64-bit executable ppc64
rsync (for architecture x86_64):	Mach-O 64-bit executable x86_64
-rwxr-xr-x  1 pepi  staff  2535144 18 Apr 22:49 rsync


When created this way, rsync running on an intel Core2Duo based Mac will not preserve creation date/time.
I've tested this with backupbouncer 0.1.3:

------------------ rsync3-maclemon-beta ------------------
This copier produced log output in:
   /Volumes/Dst/81-rsync3-maclemon-beta/log
Verifying:    basic-permissions ... ok (Critical)
Verifying:           timestamps ... ok (Critical)
Verifying:             symlinks ... ok (Critical)
Verifying:    symlink-ownership ... ok 
Verifying:            hardlinks ... ok (Important)
Verifying:       resource-forks ... 
   Sub-test:             on files ... ok (Critical)
   Sub-test:  on hardlinked files ... ok (Important)
Verifying:         finder-flags ... ok (Critical)
Verifying:         finder-locks ... ok 
Verifying:        creation-date ... FAIL           <-----------------FAILS here
Verifying:            bsd-flags ... ok 
Verifying:       extended-attrs ... 
   Sub-test:             on files ... ok (Important)
   Sub-test:       on directories ... ok (Important)
   Sub-test:          on symlinks ... ok 
Verifying: access-control-lists ... 
   Sub-test:             on files ... ok (Important)
   Sub-test:              on dirs ... ok (Important)
Verifying:                 fifo ... ok 
Verifying:              devices ... ok 
Verifying:          combo-tests ... 
   Sub-test:  xattrs + rsrc forks ... ok 
   Sub-test:     lots of metadata ... ok 







When compiled as i386 (32bit) only (doesn't matter if I also compile in the ppc and ppc64 versions)
The ONLY difference to the above build is the omission of -arch x86_64 in configure.


patch -p1 <../patches/fileflags.diff
./prepare-source

patch -p1 <../patches/crtimes.diff

./configure --prefix=/usr/local/maclemon-beta --disable-debug CC="gcc -std=gnu99 -mmacosx-version-min=10.4 -isysroot /Developer/SDKs/MacOSX10.4u.sdk" CFLAGS="-g -fprefetch-loop-arrays -funroll-loops -O3 -DHAVE_CONFIG_H -Wall -W -I./popt -isysroot /Developer/SDKs/MacOSX10.4u.sdk -arch ppc -arch i386 -arch ppc64" LDFLAGS="-mmacosx-version-min=10.4 -arch ppc -arch i386 -arch ppc64"

make


MD5 (rsync) = 85e347ea3cf8567b1b2f980c673d856f
SHA1(rsync)= 23c2c188a80db12ca268202fa28ce32c1a02e804
rsync: Mach-O universal binary with 3 architectures
rsync (for architecture ppc):	Mach-O executable ppc
rsync (for architecture i386):	Mach-O executable i386
rsync (for architecture ppc64):	Mach-O 64-bit executable ppc64
-rwxr-xr-x  1 pepi  staff  1888720 18 Apr 22:56 rsync


Now backup bouncer reports this:
------------------ rsync3-maclemon-beta ------------------
This copier produced log output in:
   /Volumes/Dst/81-rsync3-maclemon-beta/log
Verifying:    basic-permissions ... ok (Critical)
Verifying:           timestamps ... ok (Critical)
Verifying:             symlinks ... ok (Critical)
Verifying:    symlink-ownership ... ok 
Verifying:            hardlinks ... ok (Important)
Verifying:       resource-forks ... 
   Sub-test:             on files ... ok (Critical)
   Sub-test:  on hardlinked files ... ok (Important)
Verifying:         finder-flags ... ok (Critical)
Verifying:         finder-locks ... ok 
Verifying:        creation-date ... ok                                <-------------- WORKS!
Verifying:            bsd-flags ... ok 
Verifying:       extended-attrs ... 
   Sub-test:             on files ... ok (Important)
   Sub-test:       on directories ... ok (Important)
   Sub-test:          on symlinks ... ok 
Verifying: access-control-lists ... 
   Sub-test:             on files ... ok (Important)
   Sub-test:              on dirs ... ok (Important)
Verifying:                 fifo ... ok 
Verifying:              devices ... ok 
Verifying:          combo-tests ... 
   Sub-test:  xattrs + rsrc forks ... ok 
   Sub-test:     lots of metadata ... ok 



Conclusion:

crtimes patch does not work when rsync 3.0.6-pre1 (and 3.0.5) is compiled for x86_64 and run on Core2Duo in 64bit mode on Mac OS X 10.5.6. crtimes patch does only work when rsync is compiled in 32bit mode (i386) or for any PPC build.

Best regards
Pepi
Comment 1 Wayne Davison 2009-04-25 17:00:28 UTC
I compiled an x86_64 binary of rsync on OS X, and it worked fine to preserve create times.

I've seen other reports about combination binaries having issues, so I assume that this is an issue with the tools creating a combination binary, not in rsync.  If you discover otherwise, please feel free to let me know.
Comment 2 Matt McCutchen 2009-04-25 17:10:12 UTC
The recommended way to make a universal binary of rsync is to compile a separate binary for each architecture and then combine them using "lipo".  See:

http://lists.samba.org/archive/rsync/2008-March/020271.html
Comment 3 Pepi Zawodsky 2009-05-10 14:58:17 UTC
Created attachment 4136 [details]
Output of rsync log when testing with backup bouncer 0.1.3 rsync 3.0.6 for crtimes
Comment 4 Pepi Zawodsky 2009-05-10 14:59:05 UTC
Created attachment 4137 [details]
Output of backup bouncer 0.1.3 when testing rsync 3.0.6 for crtimes problem
Comment 5 Pepi Zawodsky 2009-05-10 15:02:44 UTC
Retested with rsync 3.0.6 release version, and x86_64 on Mac OS X still does NOT preserve creation dates with the crtimes.patch.

This time I've created a single (non-fat) binary for testing.

Apply patches:
patch -p1 <../patches/fileflags.diff
./prepare-source
patch -p1 <../patches/crtimes.diff


This is how I configure:
./configure --prefix=/usr/local/maclemon-beta --disable-debug CC="gcc -std=gnu99 -mmacosx-version-min=10.4 -isysroot /Developer/SDKs/MacOSX10.4u.sdk" CFLAGS="-g -fprefetch-loop-arrays -funroll-loops -O3 -DHAVE_CONFIG_H -Wall -W -I./popt -isysroot /Developer/SDKs/MacOSX10.4u.sdk -arch x86_64" LDFLAGS="-mmacosx-version-min=10.4 -arch x86_64"

MD5 (rsync) = 6aab05fca8e521e8a6afa2d2099e7f85
SHA1(rsync)= 89f0f988ee01cf96ce5da22a249142545d426008
rsync: Mach-O 64-bit executable x86_64
-rwxr-xr-x  1 pepi  staff  650544 10 Mai 21:44 rsync

When testing crtimes with "make test" the test does pass. Yet when copying a file with rsync x86_64 on 10.5.6 the creation time of the copied file is set to the modification date of the file.

Output of "make test"
$ sudo make test
pepi > root@marvelousmonolith.maclemon.lan Password: 
rsync_bin=`pwd`/rsync ./runtests.sh
============================================================
./runtests.sh running in /Users/pepi/unix/rsync/rsync-3.0.6/rsync-3.0.6
    rsync_bin=/Users/pepi/unix/rsync/rsync-3.0.6/rsync-3.0.6/rsync 
    srcdir=/Users/pepi/unix/rsync/rsync-3.0.6/rsync-3.0.6
    TLS_ARGS= -L
    testuser=root
    os=Darwin marvelousmonolith.maclemon.lan 9.6.2 Darwin Kernel Version 9.6.2: Tue Jan 13 20:42:22 PST 2009; root:xnu-1228.9.80~1/RELEASE_I386 i386
    preserve_scratch=no
    scratchbase=/Users/pepi/unix/rsync/rsync-3.0.6/rsync-3.0.6/testtmp
PASS    00-hello
PASS    acls
PASS    backup
PASS    batch-mode
PASS    chgrp
PASS    chmod-option
SKIP    chmod-temp-dir (Can't find a tmp dir on a different file system)
PASS    chmod
SKIP    chown-fake (Can't chown (probably need root))
PASS    chown
PASS    compare-dest
PASS    crtimes
PASS    daemon-gzip-download
PASS    daemon-gzip-upload
PASS    daemon
SKIP    default-acls (I don't know how to use your setfacl command)
PASS    delete
SKIP    devices-fake (Can't create char device node)
PASS    devices
SKIP    dir-sgid (Your filesystem doesn't use directory setgid; maybe it's BSD.)
PASS    duplicates
PASS    exclude
PASS    executability
PASS    files-from
PASS    fuzzy
PASS    hands
PASS    hardlinks
PASS    itemize
PASS    longdir
PASS    merge
PASS    missing
PASS    relative
PASS    ssh-basic
PASS    symlink-ignore
PASS    trimslash
PASS    unsafe-byname
PASS    unsafe-links
PASS    wildmatch
SKIP    xattrs (Unable to set an xattr)
------------------------------------------------------------
----- overall results:
      33 passed
      6 skipped
------------------------------------------------------------
overall result is 0




Testing with backoup bouncer 0.1.3:
This copier produced log output in:
   /Volumes/Dst/81-rsync3-maclemon-beta/log
Verifying:    basic-permissions ... ok (Critical)
Verifying:           timestamps ... ok (Critical)
Verifying:             symlinks ... ok (Critical)
Verifying:    symlink-ownership ... ok 
Verifying:            hardlinks ... ok (Important)
Verifying:       resource-forks ... 
   Sub-test:             on files ... ok (Critical)
   Sub-test:  on hardlinked files ... ok (Important)
Verifying:         finder-flags ... ok (Critical)
Verifying:         finder-locks ... ok 
Verifying:        creation-date ... FAIL 
Verifying:            bsd-flags ... ok 
Verifying:       extended-attrs ... 
   Sub-test:             on files ... ok (Important)
   Sub-test:       on directories ... ok (Important)
   Sub-test:          on symlinks ... ok 
Verifying: access-control-lists ... 
   Sub-test:             on files ... ok (Important)
   Sub-test:              on dirs ... ok (Important)
Verifying:                 fifo ... ok 
Verifying:              devices ... ok 
Verifying:          combo-tests ... 
   Sub-test:  xattrs + rsrc forks ... ok 
   Sub-test:     lots of metadata ... ok 

Logs from the backup bouncer and rsync output are attached.
The identical way to configure and build rsync as i386 (32bit) works fine for all creation timestamp tests. Be that with make test or by manually checking in the filesystem. x86_64 fails.

All tests were done as root.
Comment 6 Steve Ortiz 2010-02-06 01:14:42 UTC
Created attachment 5288 [details]
patch to fix creation dates bug on Mac x86_64

I ran into this same bug (6276) today, using the same tool (backup bouncer) to check how rsync would work on Mac OS X (Snow Leopard, 10.6.2).  rsync passed every test except for the "creation-date" test... this was surprising because I had installed the crtimes (and fileflags) patch.  I expected rsync to work flawlessly.

I looked into it and figured out the problem was in syscall.c, which effects both rsync and the testsuite.  Following is a summary of the problem, the fix, and steps to verify the problem and the fix.

Problem:

The get_create_time function calls the getattrlist system call, but the assumptions that worked on a 32-bit version don't hold for 64-bit.  Here's an excerpt from the man page for getattrlist...

=============================

The data returned in the buffer described by attrBuf and attrBufSize is formatted as follows.

1.   The first element of the buffer is a u_int32_t that contains the overall length, in bytes, of the attributes returned. This size includes the length field itself.

2.   Following the length field is a list of attributes.  Each attribute is represented by a field of its type, where the type is given as part of the attribute description (below).

3.   The attributes are placed into the attribute buffer in the order that they are described below.

4.   Each attribute is aligned to a 4-byte boundary (including 64-bit data types).

=============================

The key thing to note is that the data start 4 bytes into the buffer.  The current implementation uses the following struct as the attrBuf...

struct create_time {
	unsigned long length;
	struct timespec crtime;
};

Assuming the unsigned long is 4 bytes, and the crtime field continues immediately afterwards, this would be valid.  Unfortunately, what's happening is the crtime field starts 8 bytes into the struct, so it is 4 bytes offset, and the time returned is 0, which is also the value returned by get_create_time to indicate an error.  Since this is not really a problem with the system call, it's a problem with how the result is being extracted, the errno is 0 (Unknown error).

Unfortunately, the crtimes test in the test suite passes.  crtimes.test calls on checkit in rsync.fns, which calls on rsync_ls_lR in the same file, which calls on the tls tool, which uses the same get_create_time function that's causing rsync to fail.  tls produces the following stderr message with the --crtimes (or -N) option, but it results in a zero length output, which matches the other file listing it compares to "verify" that crtimes.test is working.  That's why the test appears to pass.

tls: get_create_time show-bug/from/foo: Unknown error: 0

Fix:

So, the fix is fairly straightforward... the get_create_time function needs to grab the right set of bytes.  My patch is attached... you can see I got rid of the struct create_time, and I use a char array to load results from the system call.  It works, but if you want to do it differently, I won't mind one bit (or 4 bytes).

Verify:

Here's what you can do to verify the problem and the fix.  Note, I think this problem is limited to Mac OS X (64-bit), so you'll need to test it accordingly.

tar xzf rsync-3.0.7.tar.gz 
tar xzf rsync-patches-3.0.7.tar.gz 
cd rsync-3.0.7
patch -p1 <patches/fileflags.diff 
patch -p1 <patches/crtimes.diff 
./configure
make
make test
# it should pass all the tests, so here's how you can manually see it is really broken...
mkdir -p show-bug/from
mkdir show-bug/to
touch -t 200111111111.11 show-bug/from/foo  # created date/time
touch -t 200212122222.22 show-bug/from/foo  # modified date/time
./rsync -rtg --crtimes show-bug/from/ show-bug/to
./tls -N show-bug/from/foo  # this is how the testsuite tries to verify it... broken test
GetFileInfo -d show-bug/from/foo  # shows created date
GetFileInfo -d show-bug/to/foo    # shows the modified date, but should show created... broken rsync

Now, you should be able to run my patch and go through the same steps... you'll see rsync applies the correct creation date now.

patch -p1 <../fix_get_create_time.patch   # apply my patch (wherever you saved it)
make
make test  # tests still pass, but tls works this time so it's more meaningful
rm show-bug/to/foo
./rsync -rtg --crtimes show-bug/from/ show-bug/to
./tls -N show-bug/from/foo  # shows correct output now, no errors
./tls -N show-bug/to/foo  # output matches above, good
GetFileInfo -d show-bug/from/foo  # shows created date
GetFileInfo -d show-bug/to/foo    # output matches above, good

More comments:

To be thorough, the test for crtime should probably be updated too, so it fails if the files are empty.  There are a lot of ways to do this... You may want to consider adding some output to stdout when tls fails... you could then grep for that text in the checkit function to see if any tls commands failed, and then you'd probably want to fail that test.  Otherwise, similar problems to this could silently pass.

Philosophically, maybe it would be better not to use tls in the testing, but I guess you've got to trust something for your tests.  This was probably a rare case, where the same bug was affecting both rsync and the testsuite.

Thanks to everyone who maintains rsync!  It's a great tool, and I wouldn't have bothered to write up this fix if I didn't find it useful.  Thanks to n8gray for backup bouncers, and Mike Bombich who has some nice rsync for Mac OS X help on his website.

Here are some odds and ends... the rsync version info and my backup bouncer 0.2.0 test results for my modified rsync (the untouched version 3.0.7 fails the creation-date test, but this fix will produces a version of rsync that passes everything).  If you need anything else to close out this bug, please let me know.  I am glad to help.

$ rsync --version
rsync  version 3.0.7  protocol version 30
Copyright (C) 1996-2009 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
    append, ACLs, xattrs, no iconv, symtimes, file-flags


$ sudo ./bbouncer copy -c 15-rsync-3.0.7 -d /Volumes/Src /Volumes/Dst
src = /Volumes/Src
dst = /Volumes/Dst
Enabling owners on src/dst disks
/dev/disk1s1 on /Volumes/Src (hfs, local, journaled)
/dev/disk2s1 on /Volumes/Dst (hfs, local, journaled)
Cleaning.
Copying with:          rsync-3.0.7 ... ok

------------------ rsync-3.0.7 ------------------
Verifying:    basic-permissions ... ok (Critical)
Verifying:           timestamps ... ok (Critical)
Verifying:             symlinks ... ok (Critical)
Verifying:    symlink-ownership ... ok 
Verifying:            hardlinks ... ok (Important)
Verifying:       resource-forks ... 
   Sub-test:             on files ... ok (Critical)
   Sub-test:  on hardlinked files ... ok (Important)
Verifying:         finder-flags ... ok (Critical)
Verifying:         finder-locks ... ok 
Verifying:        creation-date ... ok 
Verifying:            bsd-flags ... ok 
Verifying:       extended-attrs ... 
   Sub-test:             on files ... ok (Important)
   Sub-test:       on directories ... ok (Important)
   Sub-test:          on symlinks ... ok 
Verifying: access-control-lists ... 
   Sub-test:             on files ... ok (Important)
   Sub-test:              on dirs ... ok (Important)
Verifying:                 fifo ... ok 
Verifying:              devices ... ok 
Verifying:          combo-tests ... 
   Sub-test:  xattrs + rsrc forks ... ok 
   Sub-test:     lots of metadata ... ok
Comment 7 Wayne Davison 2010-02-06 15:57:01 UTC
Thanks for the detailed analysis!  I'm thinking that all we need to do is to change "unsigned long" to "uint32" in the create_time structure.  I've checked in that change into the patches repo (both branches).  Would you try that and see if it works?

 +struct create_time {
-+      unsigned long length;
++      uint32 length;
 +      struct timespec crtime;
 +};
Comment 8 Wayne Davison 2010-02-11 12:19:56 UTC
I have used Steve's suggested fix of wrapping the structure with an alignment pragma:

#pragma pack(push)
#pragma pack(4)
...create_time struct here...
#pragma pack(pop)

Since this is specific to OS X, I think that should be OK.  If that eventually turns out to be problematic, Steve's suggested fix to use a byte array will be used instead.