I've recently been burned a couple of times by trying to copy filesystems with --numeric-id and forgetting to do it as root. The transfer succeeds just fine, but of course the owner/group etc info can't be set, and this was -particularly- easy to overlook because the same users (mostly) existed at both ends, but with different UIDs, so it's easy to miss that something went wrong because mostly you see what looks like "correct" output from ls & friends, but is actually wrong because some of the UIDs weren't set, etc. (This was actually what I was mostly doing with your recent file-attr-restore script...) Even though I was running with -v, I don't recall seeing any errors emitted when this happened. I'm wondering whether it would be a good idea to create an option to make this much more obvious, so the cautious among us can set it by default in aliases & scripts & whatnot and hence be warned when we're about to make an annoying-to-fix mistake; it would also make it less critical to carefully scan for these hard-to-notice failures---after all, rsync is supposed to -save- us from that sort of thing... Just enabling it with excessive verbosity is probably not the right thing, since it'd get lost in the noise, and most people probably don't run at high verbosities if they're not -already- debugging something. I'm of mixed opinions about whether the behavior should be a default, since it could be irritating to see thousands of "couldn't set uid/gid" messages if you're okay with not running as root. If the option is enabled, a warning should definitely be emitted at the end of the run (where it's not buried) that something went wrong (and probably the exit status should change). And perhaps there should be another option that emits a warning for each file? Or something like --warn-me=none,end,all or something like that. There are probably other things besides uid/gid that might benefit from this (tiemstamps? ACLs?), but I fear either a huge number of warnings if they're all enabled, or the hair of a warning bitmask ("tell me about UID/GID, but not about problems setting permissions") or something. If this looks like the sort of thing that might be better hashed out on the list, I can forward this there instead; let me know. Thanks!
I think that this problem is limited to user and group settings when not run as root. The complication is that -a implies -o and -g, but we don't want errors to be returned when running as a normal user for a normal copy. One possible solution is to add some kind of an extra mode where rsync knows that it should definitely try setting the user and/or group and report a failure if it cannot. There is a diff in the patches dir named owner-group-mod.diff that chooses to force the user to specify -oo and/or -gg to tell rsync to trying setting all users and/or all groups no matter what. I have debated whether -aog should be enough to also trigger these new actions or if (as the patch used to be implemented) -oogg is still required even if -a was also specified. Any thoughts?
I think what I'd like to see is a single option called --strict or something (surely there's a better name) which I can either set all the time (e.g., by calling rsync via an alias that sets it), or a corresponding environment variable I can set in my init. (And, of course, either/both would have to take --nostrict so I can turn it off on a case-by-case basis if necessary.) If I have to remember to set a separate option, I'll probably forget, and that's exactly what I'm trying to avoid---I typically wind up typing (or scripting, even worse) some rsync command and then realizing days or weeks later that I -meant- to get owners/groups set but didn't happen to run the command as root, by which point it's way too late. And this seems conceptually simpler to me than the peculiar other ways of specifying owner/group you mention below--and extensible to other cases where the user may want to know that rsync wasn't quite able to fulfill the request (e.g., ACL's or timestamps maybe). I'm not sure whether it should complain on every file or just issue a warning at the end, though. The former might produce a lot of output, but OTOH, it'd sure be hard to miss, and you'd only get it if you said --strict in the first place...
Instead of treating -o and -g specially, I think it is time rsync had separate archive options for "running as a normal user for a normal copy", i.e. using default destination security settings, and preserving all security settings. Since there are tons of existing backup scripts that rely on -a/--archive preserving all security settings, I think -a should continue to behave as it does now, except that inability to set user or group always causes an error message. A new option can modify -a so that it does not imply -pog. I recommend calling this option -U/--def-security; -U, mnemonic for "use umask", is probably the best letter that is not already taken. For good measure, one could also add an option -s/--security that preserves all security settings (-pog). That way, people can start to use "rsync -as ..." in their backup scripts, and it might eventually be practical to change the behavior of unqualified -a to something like the tar-esque "-rltD, plus -s if run as root". Once the two modes are available, I think rsync should take all options seriously (i.e. -ogD) whether or not it is root. I also think it should issue a warning about every failure (i.e. every file) because all three of these options can succeed on some files and fail on others and it's important that the user gets the message. A concise output format that coalesces preservations that failed for the same reason might be good: rsync: "/foo/badfile": failed to preserve time, owner, group: Operation not permitted (1) rsync: "/foo/badfile": failed to preserve ACLs: Operation not supported (2) I might try to make some of these changes in my custom rsync.
The CVS version now has an option named --super that tells rsync to attempt to perform super-user activities even if it is not being run by the super-user (set file ownership, preserve all groups instead of just the current user's groups, and to create devices (which are now a separate class from special files). This both enables warnings on an rsync run where you're expecting all those things to work, and also makes it possible for rsync to work on a system where it doesn't need a UID of 0 to perform those operations. Also added was the --no-super option that allows root access to the non-super-user version of the -g option. If the user wants to make "-a --super" easier to type, consider a popt alias in your ~/.popt file to define this as the -s option: rsync alias -s -a --super (Perhaps an official -s option should be added that means that...)
I like the idea of --super and --no-super overriding rsync's own detection of whether it is running as root. But at the risk of annoying Wayne, I still think rsync's behavior with respect to file security settings should be reconsidered. Right now, rsync offers two behaviors when -a is given: (1) Preserve permissions but not ownership or devices. Preserve groups to which the user running rsync belongs. (2) Preserve permissions, owners, groups, and devices, producing an error every time it is not possible to do so. Rsync currently chooses #1 or #2 based on whether it is root, and this choice can now be overridden with --super or --no-super. Many people (including me and Lenny) use rsync for backups, and the stricter checking of #2 is obviously preferred for backups. Presumably #1 is for a "normal user for a normal copy", and indeed, I use rsync as a normal user for a normal copy several times a day. I definitely want to recurse and preserve links, modtimes, and (if there are any) special files. -a would be perfect except for one thing: I often copy files between areas with different "ambient" permissions (which I have configured with default ACLs). When I copy files from my home directory (700) to my Web site (755) or to a shared group workspace (2770), I expect them to take on permissions appropriate for the destination! When they don't, I fix them manually with chmod; if I forget to do so, annoyance can result. For example, visitors to my school's Web site encountered a 403 Forbidden error on a PDF file, which surprised me because I had personally configured the site with a default ACL of 775. It turned out that another administrator had uploaded the file, which for some reason had come out of the PDF creator with 600 permissions, using a program that preserved permissions. I think rsync can do better. I want this behavior: (3) Preserve everything not security related, but use appropriate security settings (permissions, group) for the destination. -a incorrectly second-guesses my intent; even -p ANDs with source permissions. I have been using the following incantation, which I named cp2: rsync -rltE --chmod=ugo=rwX ... (-E is my custom rsync's --executability; ignore it throughout my comment unless the official rsync adopts this feature.) I guess I could make a personal popt alias for -rltE --chmod=ugo=rwX, but if -a --super is to have an official alias, could I please have one for behavior #3? I venture to say that it is at least as useful as #1. In fact, would anyone object if the non-root meaning of -a changed to behavior #3?
(In reply to comment #5) > In fact, would anyone object if the non-root meaning of -a changed > to behavior #3? Yes -- I would. I do a lot of copying where I want the permissions and groups copied exactly from the source. > rsync -rltE --chmod=ugo=rwX ... I believe you could change the (custom) -E option in that option set into a -p and it would behave the same way (due to how X works in the chmod option and the fact that you are forcing the read and write bits on).
(In reply to comment #6) > Yes -- I would. I do a lot of copying where I want the permissions and groups > copied exactly from the source. If you care about the permissions and the groups, I bet you would want to be warned if there were an owner or a group rsync couldn't preserve. If so, ask for behavior #2 by using -s. > I believe you could change the (custom) -E option in that option set into a -p > and it would behave the same way (due to how X works in the chmod option and > the fact that you are forcing the read and write bits on). I want permissions for existing files unchanged, and I want permissions for new files masked by destination default permissions; those are the opposites of the two behaviors that -p activates. The sole purpose of the --chmod=ugo=rwX is to render ineffectual the AND-ing with source permissions for new files. The files come over the wire with 666 or 777 permissions, and the receiver ANDs that with destination default permissions; that way, a file copied from a private area to a public one can gain permissions. For exactly the same reason, I create my tar packages with embedded 666 or 777 permissions using tar --mode=ugo=rwX. I see it as discourteous to embed other permissions because that second-guesses the intent of the person extracting the archive. I ranted about this on the bug-tar list a while ago, and then the embedded permissions in kernel source tarballs mysteriously changed from 644/755 to 666/777; I speculate that I may have been the reason. Enough digression. In the rare event that an existing file changes executability on the source, I want the same to happen on the destination. That is the purpose of my custom -E, which affects only existing files. Subversion set a precedent of considering a file's executability to be as essential a counterpart to its logical data as its modtime, even though executability is (unfortunately) encoded in file security settings.
This is not really the place to hold a tangent conversatation, but no, I don't want behavior #2, as I'm not running as root (but I do have multiple groups associated with my user). Thanks for the other permissions clarifications.
I'm sorry for abusing your Bugzilla. Let the discussion continue on the mailing list if necessary. Never mind changing the behavior of -a, but an official alias for -rltE --chmod=ugo=rwX would still be nice; I'll request it as an enhancement on Bugzilla if that would be appropriate.