When using --delete-missing-args and --ignore-errors with a a file list it terminates anyway if one directory, that is not a sub of root, is not existing. See: ~$ mkdir -p src/a trg/a ~$ echo "/a/b/c" > list ~$ /usr/local/bin/rsync -slt --ignore-errors --force --ignore-missing-args --delete-missing-args --files-from=list --rsync-path=/usr/local/bin/rsync src localhost:`pwd`/trg file has vanished: "/home/axel/src/a/b" ABORTING due to invalid path from sender: a/b/c rsync error: protocol incompatibility (code 2) at generator.c(1270) [generator=3.1.2] ~$ /usr/local/bin/rsync --version rsync version 3.1.2 protocol version 31 Copyright (C) 1996-2015 by Andrew Tridgell, Wayne Davison, and others. Web site: http://rsync.samba.org/ Capabilities: 64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints, socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace, append, no ACLs, xattrs, iconv, symtimes, prealloc rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions. See the GNU General Public Licence for details. Following patch fixes it for me, albeit it ignores this kind of error now always: --- rsync-3.1.2-org/generator.c 2015-12-05 20:10:24.000000000 +0100 +++ rsync-3.1.2/generator.c 2017-02-07 10:58:33.768453242 +0100 @@ -1264,10 +1264,11 @@ && (*dn != '.' || dn[1]) /* Avoid an issue with --relative and the "." dir. */ && (prior_dir_file && strcmp(dn, f_name(prior_dir_file, NULL)) != 0) && flist_find_name(cur_flist, dn, 1) < 0) { + io_error |= IOERR_VANISHED; rprintf(FERROR, - "ABORTING due to invalid path from sender: %s/%s\n", + "WARNING: invalid path from sender: %s/%s\n", dn, file->basename); - exit_cleanup(RERR_PROTOCOL); + return; } if (relative_paths && !implied_dirs && do_stat(dn, &sx.st) < 0) { Now the exit code is 24, partial transfer, as would be expected.
I'd like to point out that this change is a changed behavior that breaks some scripts depending on this behavior. Can you consider to change it to the original behavior, or add a new parameter, that causes missing directories to be ignored.
shouldn't "--ignore-errors" already be that parameter? It uses --force also. --ignore-errors --force --i-really-mean-it? :)
that is my understanding too! And this was true before the last release. Basic tools like rsync should not break their behaviour!
My bugreport at mageia: https://bugs.mageia.org/show_bug.cgi?id=21395
I think you need to add "--no-implied-dirs" to get the behaviour you want. The issue is that the contents list contains /a/b/c, so problems with that specific file are suppressed by "--ignore-missing-args", but /a/b is not a missing argument, it's a missing implied directory. Adding the extra flag means that it won't be (implicitly) transferred, but would instead be created on the destination if necessary. Then you just get a warning about the file that's vanished: $ rsync -slt --delete-missing-args --files-from=list src trg file has vanished: ".../src/a/b" ABORTING due to invalid path from sender: a/b/c rsync error: protocol incompatibility (code 2) at generator.c(1270) [generator=3.1.1] $ rsync -slt --delete-missing-args --no-implied-dirs --files-from=list src trg file has vanished: ".../src/a/b" rsync warning: some files vanished before they could be transferred (code 24) at main.c(1183) [sender=3.1.1] HTH, .Dave.
ups, didn't get a notice from your reply. Thanks for your explanation. This was not obvious to me. It should be documented, the behaviour has changed. You can close this one, thanks.
No please don't close. Still not the behavior I'd expect: """ ~$ mkdir test ~$ cd test test$ mkdir -p src/a trg/a test$ echo "/a/b/c" > list test$ /usr/bin/rsync -slt --ignore-errors --force --ignore-missing-args --delete-missing-args --files-from=list --no-implied-dirs src localhost:`pwd`/trg file has vanished: "/home/axel/test/src/a/b" rsync warning: some files vanished before they could be transferred (code 24) at main.c(1196) [sender=3.1.2] test$ diff -r src/ trg/ Only in trg/a: b """ Unless I misunderstand something. It still should not create b at the target, if it isn't in the source.
@Axel: you're right. This is not what we want. Even the output sync warning: some files vanished before they could be transferred is not desireable if the parameter is called "ignore" there should not be any output. And the return code of this transfer should not indicate any errors. I assume, you use this for automatic synchronisation between servers as well? I don't understand why they changed the behaviour, before this change, it worked as desired.
@Marc, indeed. I'm the author of Lsyncd. https://github.com/axkibe/lsyncd If this could work properly, it would simplify things a lot, also improve perfomance a good deal. Due to this bug I had to drop the use of --files-from feature and keep creating include/exclude filters on the fly, which also means rsync has to inspect way more files, which is a quite a penalty with directories and a lot files.
@Axel: cool, I've played a bit with your tool, but for my needs with many directories inotify was the pitfall. I'm coauthor on sfs (https://github.com/mokraemer/sfs) which uses fuse for signaling. And then, as you do, rsync for synchronization. So we have both the same problem here :(
Created attachment 14775 [details] proposed patch This patch is based on the patch posted earlier, but checks the options --delete-missing-args and --ignore-errors. I see no reason why this shouldn't be patched into the main rsync sources.
This check was added to try to prevent an untrusted sender from sneaking through some bad file-list data. However, the --delete-missing-args option does seem to have the ability to send these incongruent lists. I'm thinking that the best fix is probably to have the sender fudge up the missing parent dirs if they can be marked as missing like the missing args. In the meantime, I'm checking in a slightly different version of Paul's patch that makes the receiver just complain about the missing parent dir info and continue on (it does not return or a valid missing arg won't be deleted).
Hi Wayne, There is a report on this in RHEL too [0], and i reproduced it also in Fedora. Did you make any progress? I can help with implementation of fix if you want. Thanks for any information you can provide. [0] - https://bugzilla.redhat.com/show_bug.cgi?id=1727093
As I mentioned above, I've already fixed the issue and left this open as a reminder to look into an even better solution in the future.
Ah. Sorry about that. I misunderstood your previous comment. Thanks for clarification.
I'm not sure if this is still the same issue. If the directory vanishes before transfer, e.g. issuing the command: rsync -e ssh -d --no-r -ltpDcuhROHigoz --exclude /.sfs.conf --exclude /.sfs.mounted --no-implied-dirs --exclude-from=/etc/sysconfig/sfs/excludes.conf --delete-after --files-from=input.txt leonato:/mnt/bricks/www/ /tmp/x --delete-missing-args --ignore-errors input.txt: /test17/21/ff/zz /test17/21/ff /test17/21 /test17 I get: file has vanished: "/mnt/bricks/www/test17" file has vanished: "/mnt/bricks/www/test17/21" file has vanished: "/mnt/bricks/www/test17/21/ff" *deleting test17/21/ff/ *deleting test17/21/ *deleting test17/ rsync warning: some files vanished before they could be transferred (code 24) at main.c(1637) [generator=3.1.1] but on target the directory is still present!
tested release 3.2.2 bug is still present: $ cd /tmp $ mkdir x y $ mkdir -p x/test17/21/ff/zz $ echo "/test17/21/ff/zz /test17/21/ff /test17/21 /test17 " > input.txt $ cd y $ rsync --no-r -ltpDcuhROHigoz --no-implied-dirs --delete --files-from=../input.txt /tmp/x/ /tmp/y/ --delete-missing-args --ignore-errors $ ls test17/ $ rm -rf ../x/test17 $ rsync --no-r -ltpDcuhROHigoz --no-implied-dirs --delete --files-from=../input.txt /tmp/x/ /tmp/y/ --delete-missing-args --ignore-errors *deleting test17/21/ff/ *deleting test17/21/ *deleting test17/ WARNING: parent dir is absent in the file list: test17 WARNING: parent dir is absent in the file list: test17/21 WARNING: parent dir is absent in the file list: test17/21/ff rsync warning: some files vanished before they could be transferred (code 24) at main.c(1287) [sender=3.2.2] $ ls test17/ so this bug is still not fixed!
What in that sequence of events are you claiming is a bug?
Oops, I missed the "ls" at the end. I'll give it a look.
Get rid of --no-implied-dirs and --ignore-errors is also unneeded.
The --no-implied-dirs code was recreating the directories after they had been deleted. I changed it to avoid that if the entry it is processing is marked as missing. I also silenced the parent-dir warning for a missing entry that has a parent entry that is also marked as missing.
@Wayne: thanks, you're right. I've overlooked this option and didn't expect it to be harmful. Removing it really solves the issue.