Bug 12569 - Missing directory errors not ignored
Missing directory errors not ignored
Status: NEW
Product: rsync
Classification: Unclassified
Component: core
3.1.2
All Linux
: P5 normal
: ---
Assigned To: Wayne Davison
Rsync QA Contact
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2017-02-07 11:31 UTC by Axel Kittenberger
Modified: 2018-10-09 23:17 UTC (History)
0 users

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Axel Kittenberger 2017-02-07 11:31:43 UTC
When using --delete-missing-args and --ignore-errors with a a file list it terminates anyway if one directory, that is not a sub of root, is not existing.

See:

~$ mkdir -p src/a trg/a
~$ echo "/a/b/c" > list
~$ /usr/local/bin/rsync -slt --ignore-errors --force --ignore-missing-args --delete-missing-args --files-from=list --rsync-path=/usr/local/bin/rsync src localhost:`pwd`/trg
file has vanished: "/home/axel/src/a/b"
ABORTING due to invalid path from sender: a/b/c
rsync error: protocol incompatibility (code 2) at generator.c(1270) [generator=3.1.2]
~$ /usr/local/bin/rsync --version
rsync  version 3.1.2  protocol version 31
Copyright (C) 1996-2015 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
    append, no ACLs, xattrs, iconv, symtimes, prealloc

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.


Following patch fixes it for me, albeit it ignores this kind of error now always:
--- rsync-3.1.2-org/generator.c	2015-12-05 20:10:24.000000000 +0100
+++ rsync-3.1.2/generator.c	2017-02-07 10:58:33.768453242 +0100
@@ -1264,10 +1264,11 @@
 			 && (*dn != '.' || dn[1]) /* Avoid an issue with --relative and the "." dir. */
 			 && (prior_dir_file && strcmp(dn, f_name(prior_dir_file, NULL)) != 0)
 			 && flist_find_name(cur_flist, dn, 1) < 0) {
+				io_error |= IOERR_VANISHED;
 				rprintf(FERROR,
-					"ABORTING due to invalid path from sender: %s/%s\n",
+					"WARNING: invalid path from sender: %s/%s\n",
 					dn, file->basename);
-				exit_cleanup(RERR_PROTOCOL);
+				return;
 			}
 			if (relative_paths && !implied_dirs
 			 && do_stat(dn, &sx.st) < 0) {

Now the exit code is 24, partial transfer, as would be expected.
Comment 1 Marc Krämer 2018-03-14 19:28:47 UTC
I'd like to point out that this change is a changed behavior that breaks some scripts depending on this behavior.
Can you consider to change it to the original behavior, or add a new parameter, that causes missing directories to be ignored.
Comment 2 Axel Kittenberger 2018-05-25 06:56:03 UTC
shouldn't "--ignore-errors" already be that parameter?
It uses --force also.

--ignore-errors --force --i-really-mean-it? :)
Comment 3 Marc Krämer 2018-06-08 14:59:49 UTC
that is my understanding too! And this was true before the last release.

Basic tools like rsync should not break their behaviour!
Comment 4 Marc Krämer 2018-06-08 15:02:03 UTC
My bugreport at mageia: https://bugs.mageia.org/show_bug.cgi?id=21395
Comment 5 Dave Gordon 2018-08-15 20:20:14 UTC
I think you need to add "--no-implied-dirs" to get the behaviour you want.

The issue is that the contents list contains /a/b/c, so problems with that
specific file are suppressed by "--ignore-missing-args", but /a/b is not
a missing argument, it's a missing implied directory. Adding the extra flag
means that it won't be (implicitly) transferred, but would instead be created
on the destination if necessary. Then you just get a warning about the file
that's vanished:

$ rsync -slt --delete-missing-args --files-from=list src trg
file has vanished: ".../src/a/b"
ABORTING due to invalid path from sender: a/b/c
rsync error: protocol incompatibility (code 2) at generator.c(1270) [generator=3.1.1]

$ rsync -slt --delete-missing-args --no-implied-dirs --files-from=list src trg
file has vanished: ".../src/a/b"
rsync warning: some files vanished before they could be transferred (code 24) at main.c(1183) [sender=3.1.1]

HTH,
.Dave.
Comment 6 Marc Krämer 2018-10-07 11:16:22 UTC
ups, didn't get a notice from your reply.

Thanks for your explanation. This was not obvious to me. It should be documented, the behaviour has changed.

You can close this one, thanks.
Comment 7 Axel Kittenberger 2018-10-07 13:40:07 UTC
No please don't close. Still not the behavior I'd expect:

"""
~$ mkdir test
~$ cd test
test$ mkdir -p src/a trg/a
test$ echo "/a/b/c" > list
test$ /usr/bin/rsync -slt --ignore-errors --force --ignore-missing-args --delete-missing-args --files-from=list --no-implied-dirs src localhost:`pwd`/trg
file has vanished: "/home/axel/test/src/a/b"
rsync warning: some files vanished before they could be transferred (code 24) at main.c(1196) [sender=3.1.2]
test$ diff -r src/ trg/
Only in trg/a: b
"""

Unless I misunderstand something. It still should not create b at the target, if it isn't in the source.
Comment 8 Marc Krämer 2018-10-09 16:46:15 UTC
@Axel: you're right. This is not what we want. Even the output 
sync warning: some files vanished before they could be transferred
is not desireable if the parameter is called "ignore" there should not be any output.
And the return code of this transfer should not indicate any errors.

I assume, you use this for automatic synchronisation between servers as well? I don't understand why they changed the behaviour, before this change, it worked as desired.
Comment 9 Axel Kittenberger 2018-10-09 17:06:02 UTC
@Marc, indeed. I'm the author of Lsyncd.

https://github.com/axkibe/lsyncd

If this could work properly, it would simplify things a lot, also improve perfomance a good deal. Due to this bug I had to drop the use of --files-from feature and keep creating include/exclude filters on the fly, which also means rsync has to inspect way more files, which is a quite a penalty with directories and a lot files.
Comment 10 Marc Krämer 2018-10-09 23:17:30 UTC
@Axel: cool, I've played a bit with your tool, but for my needs with many directories inotify was the pitfall.

I'm coauthor on sfs (https://github.com/mokraemer/sfs) which uses fuse for signaling. And then, as you do, rsync for synchronization. So we have both the same problem here :(