Bug 13147 - inconsistent behaviour regaring vanished files information
inconsistent behaviour regaring vanished files information
Status: NEW
Product: rsync
Classification: Unclassified
Component: core
3.1.2
x64 Mac OS X
: P5 minor
: ---
Assigned To: Wayne Davison
Rsync QA Contact
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2017-11-16 16:14 UTC by roland
Modified: 2017-11-28 14:55 UTC (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description roland 2017-11-16 16:14:52 UTC
i found that for our osx host "gonzo" , vanished files (not the warning message itself) appear in stdout - for linux hosts they _both_ appear in stderr. no information about vanished files appears in stdout (in my backup script, i´m redirecting stdout to rsync.log and stderr to rsync.err)

what explains this difference in behaviour? 

is this a bug ? or is it related to rsynv version ?

backupvm1->spacewalk is 3.0.9->3.0.9 and backupvm1->gonzo is 3.0.9->3.1.2 (installed from brew repo) 

[root@backupvm1 backup]# grep -i vanished spacewalk/rsync.*
spacewalk/rsync.err:file has vanished: "/var/lib/rhn/search/indexes/server/_80a2.cfs"
spacewalk/rsync.err:file has vanished: "/var/lib/rhn/search/indexes/server/_80a2_6.del"
spacewalk/rsync.err:file has vanished: "/var/lib/rhn/search/indexes/server/_80a3.cfs"
spacewalk/rsync.err:file has vanished: "/var/lib/rhn/search/indexes/server/_80a4.cfs"
spacewalk/rsync.err:file has vanished: "/var/lib/rhn/search/indexes/server/_80a5.cfs"
spacewalk/rsync.err:file has vanished: "/var/lib/rhn/search/indexes/server/segments_n801"
spacewalk/rsync.err:rsync warning: some files vanished before they could be transferred (code 24) at main.c(1518) [generator=3.0.9]

[root@backupvm1 backup]# grep -i vanished gonzo/rsync.*
gonzo/rsync.err:rsync warning: some files vanished before they could be transferred (code 24) at main.c(1518) [generator=3.0.9]
gonzo/rsync.log:file has vanished: "/Library/Server/Mail/Data/spool/maildrop/AFD811E2D07C"
gonzo/rsync.log:file has vanished: "/Library/Server/Wiki/Database.xpg/Cluster.pg/pg_xlog/000000010000004F00000025"
gonzo/rsync.log:file has vanished: "/Library/Server/Wiki/Database.xpg/Cluster.pg/pg_xlog/archive_status/000000010000004F00000025.done"
gonzo/rsync.log:file has vanished: "/Library/Server/Wiki/Database.xpg/backup/000000010000004F00000026.partial"
Comment 1 roland 2017-11-25 10:06:55 UTC
i searched into this a little bit and it`s not a problem with differing rsync versions, it also happens with rsync 3.1.2 (linux) -> rsync 3.1.2 (osx)

what i found is, that rsync 3.1.2 behaves correctly when run on osx locally, i.e. if i rsync -av /tmp/localdir/rapidly-appearing+disappearing-files-inside  /tmp/localdir2  on osx locally, i`m getting all vanished files messages including which files are missing on stdout. 

if i do the same from linux/remote, then the information which files are vanished goes to stdout instead of sterr.
the message "rsync warning: some files vanished before they could be transferred (code 24) at main.c(1650) [generator=3.1.2]"  still goes to sterr (as expected)
Comment 2 roland 2017-11-25 10:34:05 UTC
it`s also no linux->osx interacting issue, the problem also appears when doing rsync on osx via ssh/localhost.


but why ?

ssh & osx does handle stdout and stdout correctly, as shown here:


gonzo:tmp root# ssh root@localhost "echo stdout; echo stderr >&2" >/dev/null
stderr
gonzo:tmp root# ssh root@localhost "echo stdout; echo stderr >&2" 2>/dev/null
stdout
Comment 3 roland 2017-11-25 11:26:54 UTC
apparently, ssh as a transport is the culprit, as it doesn`t happen with rsync in daemon mode.

i found some osx related ticket on "stderr/stdout merge" ( https://github.com/ansible/ansible/issues/14960 ) 

after further searching, i found some infos on pty/tty allocation:
https://stackoverflow.com/questions/7114990/pseudo-terminal-will-not-be-allocated-because-stdin-is-not-a-terminal  (see comment "Optional background information:")

 and...


# ssh root@localhost "echo stdout; echo stderr >&2" >/dev/null
stderr

# ssh root@localhost "echo stdout; echo stderr >&2" 2>/dev/null
stdout

# ssh -t root@localhost "echo stdout; echo stderr >&2" 2>/dev/null
stdout
stderr

gotcha !

now the question is how to work around the problem...
Comment 4 roland 2017-11-28 14:55:47 UTC
mhh, if the error is not a warning but an error, the offending file is shown (for the warning of vanished files, the warning itself goes to stderr but the vanished filename goes to stdout). but only on OSX

why that difference ?

rsync: readlink "/private/var/folders/7p/mcsl60wn0djct6d9_kpzy1nc0000gp/0/com.apple.LaunchServices.dv" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1518) [generator=3.0.9]