Bug 8566 - Spotlight comments (extended attributes) are not synced
Summary: Spotlight comments (extended attributes) are not synced
Status: ASSIGNED
Alias: None
Product: rsync
Classification: Unclassified
Component: core (show other bugs)
Version: 3.0.9
Hardware: All Mac OS X
: P5 normal (vote)
Target Milestone: ---
Assignee: Wayne Davison
QA Contact: Rsync QA Contact
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-11-02 22:08 UTC by Stefan Nowak
Modified: 2013-05-02 10:27 UTC (History)
0 users

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Stefan Nowak 2011-11-02 22:08:36 UTC
rsync 3.0.9 on Mac OS X 10.6.8

As an example for working extended attributes: File labels are always properly synced: Both during their first sync when the file + attributes is copied and also after you change their states and sync again, the file remains untouched, but the attributes are properly updated!

But Spotlight comments are not transferred, neither when a file is synced the first time nor when the comment is changed, and another sync is then run.


For completeness, this is the command line I ran:

rsync \
--protect-args \
\
--archive \
--crtimes \
--hard-links \
--acls \
--xattrs \
--fileflags \
--one-file-system \
--force-change \
\
--exclude=".*" \
--exclude="Icon*" \
\
--verbose \
--stats \
--progress \
--itemize-changes \
--human-readable \
\
--dry-run \
\
~/Temporary/ \
/Volumes/ExternalDisk/All/
Comment 1 Stefan Nowak 2011-12-01 21:31:18 UTC
1) Could someone please answer how to successfully sync Spotlight comments? Thanks!

2) As of 3.0.9 this functionality seems not to be integrated in the main distribution, as my testing in my previous comment shows, but through patching it seems to be possible

User Gnarlodious reports at:
https://discussions.apple.com/message/12142363#12142363
Downloaded 3.0.7 and applied the patch "fileflags.diff" (among others) and then comments seem to get properly synced.
Comment 2 Stefan Nowak 2011-12-01 21:58:28 UTC
I downloaded 3.0.9 source + patches, patched in the 2 patches:
patch -p1 <patches/fileflags.diff
patch -p1 <patches/crtimes.diff
And although running with the appropriate arguments (as in the original report), the comments still do not get synced!

Help please!
Comment 3 Wayne Davison 2011-12-24 18:28:18 UTC
Hmm, stock 3.0.7 with official fileflags & crtimes patches was working and 3.0.9 with those patches is not?  That's weird.  I don't have access to an OS X box at the moment, but should be able to look into this in the future.  If anyone wants to dive in and debug, feel free.
Comment 4 Stefan Nowak 2012-03-23 13:21:24 UTC
I'd appreciate some help!
I'm still unable to correctly rsync Finder / Spotlight comments.
Comment 5 Stefan Nowak 2012-03-23 13:47:44 UTC
(In reply to comment #3)
> Hmm, stock 3.0.7 with official fileflags & crtimes patches was working and 3.0.9 with those patches is not?  That's weird.

I now also downloaded source 3.0.7, patched it with 3.0.7, and built it, and it doesn't work as well.
Comment 6 Mike Bombich 2012-03-23 15:25:19 UTC
My own build of rsync preserves Spotlight comments just fine. Those comments are stored as extended attributes, so as long as you preserve xattrs, your Spotlight comments should be preserved. Before you make the conclusion that your Spotlight comments have *not* been preserved, verify whether the "com.apple.metadata:kMDItemFinderComment" exists on the destination file. Also, you should verify that Spotlight is enabled on the destination.

mdutil -s /Volumes/Target

[bombich:/Volumes/Source] touch test
[bombich:/Volumes/Source] xattr --list test

<add a Spotlight comment in Finder>

[bombich:/Volumes/Source] xattr --list test
com.apple.metadata:kMDItemFinderComment

[bombich:/Volumes/Source] xattr --get com.apple.metadata:kMDItemFinderComment test
com.apple.metadata:kMDItemFinderComment	bplist00^This is a tes   

[bombich:/Volumes/Source] ls -la /Volumes/Target/test
ls: /Volumes/Target/test: No such file or directory

<rsync Source to Target, preserving xattrs -- I used 3.0.6 + some various bug fixes from 3.0.7-3.0.9>

[bombich:/Volumes/Source] ls -la /Volumes/Target/test
-rw-r--r--@ 1 bombich  wheel  0 Mar 23 11:18 /Volumes/Target/test

[bombich:/Volumes/Source] xattr --list /Volumes/Target/test 
com.apple.metadata:kMDItemFinderComment

[bombich:/Volumes/Source] xattr --get com.apple.metadata:kMDItemFinderComment /Volumes/Target/test 
com.apple.metadata:kMDItemFinderComment	bplist00^This is a tes   

[Note: I use a custom build of the "xattr" tool so the arguments may be different on your system]

Lastly, you should note that Finder has some display problems with Spotlight comments. I don't know if the problem resides within Finder or Spotlight, but I've seen odd delays associated with the proper information appearing the the Spotlight comments field. For example, the Spotlight comments appear in the xattr on the destination file, but they do not appear immediately in the Finder. Additionally, if you delete the source file, then re-touch it and Get Info, the old Spotlight comments are still there, even though the xattr is empty. So don't use the Finder as a measure of whether these comments are preserved, use the xattr contents instead.
Comment 7 Stefan Nowak 2012-03-24 14:02:49 UTC
Dear Mike, thanks for your effort!
Some questions remain, maybe you can help?

I performed the tests as advised, and can now confirm:
What I experience(d) all the time, is indeed a heavy Finder bug, rather than an rsync bug!

xattr proofed, that after every rsync, all comments were properly synced!

It is the buggy Finder, which hides the proper comment info:
* Finder does not only conceal Finder comments of freshly synced files (0-10 seconds ago), but also from files, which were synced many months (!) and sessions ago!
* Even if I select "Show Info" (CMD-I) for a destination file, which was recently or long ago rsynced, I get no info, even though xattr prooves, that it is there!
* "Odd delay" is an understatement! This is a large scale bug.

Is there any way to force Finder to update its comment display?
(Or a Spotlight Update/Re-Indexing of the target volume?)

The methods below did not accomplish comment display update:
* Refresh a Finder window to force it to update: http://hints.macworld.com/article.php?story=2009091413423819
* Restart Finder
* Log out & in again
* Restart Mac
Comment 8 Stefan Nowak 2012-03-24 14:15:36 UTC
Interesting: I now performed a .DS_Store removal, according to:
http://www.finetunedmac.com/forums/ubbthreads.php?ubb=showflat&Number=11266#Post11266

cd /Volumes/Destination/Folder
osascript -e 'tell application "Finder" to quit'
rm .DS_Store
rm ../.DS_Store
osascript -e 'tell application "Finder" to activate'

After that ALL comments are gone (in the Finder view).
Before that, at least those which I added at the destination were shown.
xattr -rl   prooves that all comments are still there, only now NONE shown in Finder.
Comment 9 Stefan Nowak 2012-03-24 14:45:18 UTC
mdutil -E /Volumes/Destination

Re-indexing the destination volume did not help either.
xattr -rl showed that all comment data is still there.
Comment 10 Stefan Nowak 2013-04-05 06:56:45 UTC
User V.K. posted a script here: https://discussions.apple.com/message/10549961#10549961

I ran V.K.'s script over the rsync backup'ed files and then the comments were restored! Also after disconnecting, un-powering, re-powering, re-connecting the external harddisk, the restored comments persisted.

My post: https://discussions.apple.com/message/21704242#21704242

On my rsync backup destination the files had shown NO comments in the Finder but
mdls -name kMDItemFinderComment <YourFile> had revealed that they were stored (at least somewhere).

For test purposes I created the folder "_1" with comment "1" on the source with Mac OS X Finder 10.6.8, ran rsync 3.0.7 (compiled with xattrs and others), and on the destination again no comment was shown for folder "_1". After running the comment-restore-applescript over "_1" the comment was there! I created folder "_2" with comment "22" on the source, and copied it via Finder. The comment appeared on the destination as well.

CONCLUSION:
It seems rsync syncs the Finder comments to a deprecated storage place?!
Or Finder has a bug, ignoring info on valid storage positions.
Comment 11 Stefan Nowak 2013-05-02 10:27:05 UTC
Apple Developer Bug Reporting Team replied today:

Engineering has determined that this issue behaves as intended based on the following information:
Comments are stored primarily in the .DS_Store files and mirrored to Spotlight for searching. xattrs are not the primary storage.
This is the effect of attributes being stored externally, and command line tools not being aware of them.

---

Now I realize why my rsync failed:
I had used --exclude=".*" which excluded .DS_Store from being synced!


Finder's metadata handling being only .DS_Store-aware and mostly ignoring xattrs poses the following problems:

PROBLEM 1: It is neither fish nor fowl

a) The labels ARE considered from xattrs, more precisely at least ALSO considered from xattrs.
Proof: rsync xattr sync results immediately visible in Finder.

b) Whereas the comments are considered SOLELY from .DS_store.
Proof: Rsync results not visible in Finder.
Workaround: See comment 10

PROBLEM 2: Partial file+metadata syncs are IMPOSSIBLE by design without .DS_Store merging capabilities

Even if I would include .DS_Store files in my sync, synching a folder partially MUST fail by design. Why?

If metadata would be per file, it is transported correctly on whatever selection of files I sync over.
If metadata is per one .DS_Store per folder for multiple files, and you sync only a few files plus the .DS_Store of that folder, you OVERWRITE the .DS_Store on the destination.
Then the metadata for the files being both in the source and destination are correct, but those only at the destination LOOSE their metadata. So a sync process must be .DS_Store aware and MERGE .DS_Store from source and destination, which practically EXCLUDES MOST 3rd PARTY apps from properly syncing metadata!!!

What is my application: My entire projects are too large for my internal disk. Therefore I eventually keep them on an external disk. The actual projects are always on my internal disk, and I rsync them over additively (without the option "delete on destination") to my external disk every here and then. If eventually done, I sync a last time, and then erase on my internal disk. Therefore my source (internal HD) is a subset of my external disk. And syncing only parts over with a .DS_Store unaware tool would copy over only the synced files' metadata but ERASE the metadata of all other files on the destination (external HD).

EXAMPLE:
Source: 3, 4 -- all with metadata
Destination: 1, 2, 3, 7, 8 -- all with metadata
Outcome: If I sync over 3+4 with a DS_Store unaware sync tool, then 3 gets its metadata updated, 4 gets copied freshly, and all others (1, 2, 7, 8) loose their metadata!