Bug 3653 - Reduce the need for the "vanished files" warning
Reduce the need for the "vanished files" warning
Status: RESOLVED FIXED
Product: rsync
Classification: Unclassified
Component: core
2.6.8
All All
: P3 enhancement
: ---
Assigned To: Wayne Davison
Rsync QA Contact
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2006-04-03 07:05 UTC by Andreas Kotes
Modified: 2016-03-20 16:41 UTC (History)
10 users (show)

See Also:


Attachments
fixed rsync-no-vanished wrapper script (256 bytes, text/plain)
2006-06-12 09:02 UTC, Matt McCutchen
no flags Details
Demonstration of two nasty vanishing cases (365 bytes, application/x-shellscript)
2008-06-04 20:24 UTC, Matt McCutchen
no flags Details
updated patch in #c1 against rsync-3.0.7 (1.85 KB, patch)
2011-01-25 13:55 UTC, Elan Ruusamäe
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Andreas Kotes 2006-04-03 07:05:08 UTC
Please add an option to disable the 'vanished files' error messages and return code. This way, running rsync from crontab/meep wouldn't send mails for this rather unproblematic case when you already know about it.
Comment 1 Andreas Kotes 2006-04-03 07:13:20 UTC
diff -u -r1.332 options.c
--- options.c   28 Mar 2006 23:09:36 -0000      1.332
+++ options.c   3 Apr 2006 12:12:44 -0000
@@ -113,6 +113,7 @@
 int checksum_seed = 0;
 int inplace = 0;
 int delay_updates = 0;
+int ignore_vanished = 0;
 long block_size = 0; /* "long" because popt can't set an int32. */


@@ -491,6 +492,7 @@
   {"no-partial",       0,  POPT_ARG_VAL,    &keep_partial, 0, 0, 0 },
   {"partial-dir",      0,  POPT_ARG_STRING, &partial_dir, 0, 0, 0 },
   {"delay-updates",    0,  POPT_ARG_NONE,   &delay_updates, 0, 0, 0 },
+  {"ignore-vanished",  0,  POPT_ARG_NONE,   &ignore_vanished, 0, 0, 0 },
   {"prune-empty-dirs",'m', POPT_ARG_NONE,   &prune_empty_dirs, 0, 0, 0 },
   {"log-format",       0,  POPT_ARG_STRING, &log_format, 0, 0, 0 },
   {"itemize-changes", 'i', POPT_ARG_NONE,   0, 'i', 0, 0 },
diff -u -r1.92 sender.c
--- sender.c    14 Jan 2006 20:26:24 -0000      1.92
+++ sender.c    3 Apr 2006 12:12:44 -0000
@@ -41,6 +41,7 @@
 extern struct stats stats;
 extern struct file_list *the_file_list;
 extern char *log_format;
+extern int ignore_vanished;


 /**
@@ -296,12 +297,14 @@
                fd = do_open(fname, O_RDONLY, 0);
                if (fd == -1) {
                        if (errno == ENOENT) {
-                               enum logcode c = am_daemon
+                               if (!ignore_vanished) {
+                                       enum logcode c = am_daemon
                                    && protocol_version < 28 ? FERROR
                                                             : FINFO;
-                               io_error |= IOERR_VANISHED;
-                               rprintf(c, "file has vanished: %s\n",
-                                       full_fname(fname));
+                                       io_error |= IOERR_VANISHED;
+                                       rprintf(c, "file has vanished: %s\n",
+                                                       full_fname(fname));
+                               }
                        } else {
                                io_error |= IOERR_GENERAL;
                                rsyserr(FERROR, errno,
Comment 2 Matt McCutchen 2006-04-03 14:59:02 UTC
I think filtering error messages should be the job of the cron script invoking rsync, not of rsync itself; otherwise we'll find ourselves adding options to disable each of the myriad errors rsync can produce.  Do "rsync <...> 2>&1 | grep -v 'file has vanished'" .
Comment 3 Andreas Kotes 2006-06-07 09:20:41 UTC
(In reply to comment #2)
> I think filtering error messages should be the job of the cron script invoking
> rsync, not of rsync itself; otherwise we'll find ourselves adding options to
> disable each of the myriad errors rsync can produce.  Do "rsync <...> 2>&1 |
> grep -v 'file has vanished'" .

the problem here is that this isn't really an error, but basically a warning/information.

when I'm calling rsync from meep (http://freshmeat.net/projects/meep/) I can't filter the error messages or change the error code before meep gets them.

I really think this condition should be silenceable. No other error is comparable (at least to me).
Comment 4 Matt McCutchen 2006-06-07 17:41:39 UTC
(In reply to comment #3)
> when I'm calling rsync from meep (http://freshmeat.net/projects/meep/) I can't
> filter the error messages or change the error code before meep gets them.

Yes you can!  Write an rsync wrapper script like this:

rsync-no-vanished:
#!/bin/bash
(rsync "$@"; if [ $? == 24 ]; then exit 0; else exit $?; fi) 2>&1 \
    | grep -v 'vanished'

Then edit CMD_RSYNC in meep to refer to rsync-no-vanished instead of rsync.  I have tested this script, and it blocks all vanished-related messages and exit codes while letting everything else through.
Comment 5 Andreas Kotes 2006-06-09 10:28:11 UTC
works, thank you. I'd love the patch to be included nonetheless, so I don't have to spawn yet another shell every time :-/
Comment 6 Matt McCutchen 2006-06-09 19:17:24 UTC
I still think that incorporating a patch into the official rsync for something done so easily outside of rsync would be foolish.  Of course you are welcome to patch your own copy of rsync if you prefer that to a shell script.
Comment 7 Andreas Kotes 2006-06-12 03:55:20 UTC
Patching my copy would bar me from Debian (security) updates.

... btw, the script DOES NOT work as expected :(

try:

mkdir a ; mkdir b ; rsync -r a b ; echo $? ; rm -rf a b
vs.
mkdir a ; mkdir b ; rsync-no-vanished -r a b ; echo $? ; rm -rf a b

... I found no way to fix that, and it breaks my whole backup scheme.

Thus, I would definitely say it's NOT easily fixed outside rsync, and would still like the patch to be included - it's the only warning condition rsync is noisy about, AFAIK :(
Comment 8 Matt McCutchen 2006-06-12 09:02:28 UTC
Created attachment 1957 [details]
fixed rsync-no-vanished wrapper script

You have a point about security updates.  I wonder if anyone has a system that will automatically apply local customizations to software updates.

OK, so I made a mistake and my script mangled the exit code.  Please try the attached improved version of the script.  As for "not easy to fix": maybe it's not easy for you to come up with a fix, but if my new script works, there will be a fix that is easy to _use_ and that's what counts.
Comment 9 Andreas Kotes 2006-06-14 08:19:14 UTC
Well, easy-to-use would be --ignore-vanished ;) .. as rsync is the one emitting the warning.

"pipefail" is a bash 3.x feature, which hasn't made it into all distributions (yet) .. and which I didn't know of, until today :)

... I still thinks this wrapper is a kludge which shouldn't be needed at all. Do you know ANY other kind of message/warning/exit code for rsync which would demand a similiar kind of treatment?
Comment 10 Guy Baconniere 2006-10-13 03:54:54 UTC
I have a theory that vanised files are caused 
by clock sync of ntpdate or hwclock. 

Please could you check if the vanished files apears only
when a cron job of ntpdate is done ?

Patching rsync to hide vanished files is just a work around
and this is very dangerous if you use "ignore-errors" with "delete"
options. Rsync will start to delete files who still present on
the source because of reported as "vanished" !

https://bugzilla.samba.org/show_bug.cgi?id=4168

Best Regards,
Guy Baconniere
Comment 11 Matt McCutchen 2006-10-14 19:01:00 UTC
Andreas, you have a point about the special nature of the "file has vanished" message.  The message seems to be an artifact of rsync's implementation that isn't meaningful at the level of rsync's purpose.  I now believe the sender should treat a vanished file V as if it never existed in the first place.  That means:

(1) If V was a command-line argument or --files-from entry, the sender should issue the same error it would have issued if V did not exist during file-list building.

(2) If the generator has itemized the transfer, the sender needs to print an informational message stating that the transfer didn't actually happen so that programs reading the output know to forget about it.  This message is neither an error nor a warning.

(3) If --delete is on and V exists in the destination, the generator needs to consider deleting it.  To this end, the sender uses a protocol extension to tell the receiver that V vanished.  The receiver passes the news to the generator, which removes V from its file list.  If the generator has already deleted in V's containing directory, it deletes V immediately modulo protect filters.  If the receiving side is not modern enough to support this process, the sender prints a warning to that effect and sets IOERR_VANISHED.

To a sender containing my proposed changes, a vanishing is no longer cause for a warning or an exit code unless the receiving side is old and #3 applies.  And I anticipate that #3 will rarely happen because most source files that vanish are short-lived temporary files that haven't existed for long enough to be copied to the destination by a previous rsync run.
Comment 12 Matt McCutchen 2008-06-04 20:24:55 UTC
Created attachment 3335 [details]
Demonstration of two nasty vanishing cases

Even with the changes I proposed in comment 11, there are two cases in which rsync's handling of a vanishing could be considered incorrect and thus merit a warning and code 24.  They are both demonstrated in the attached script.

4. A source file causes rsync to delete a non-regular destination file (which could even be a nonempty directory with --force) and then vanishes.  The end result is that rsync deleted something from the destination without replacing it, which shouldn't happen without --delete.

5. With --prune-empty-dirs, a source file preventing the pruning of its parent directory and then vanishes.  The end result is that rsync creates an empty directory, which isn't supposed to happen with --prune-empty-dirs.

Both of these seem rare and a pain to fix, and there are other other concurrent modifications that cause code 23, such as replacing a regular file with a directory, which can't be read(2).  I think making rsync completely correct and warning-free in the face of concurrent modifications to the source is a losing battle.  I propose that we focus on reducing the inconvenience in the common case by suppressing the warning and code 24 for "safe" vanishings, namely new source files in unprunable directories, and perhaps handling #3.  The need to issue a code 23 when a mandatory file vanishes (#1) is a separate issue.
Comment 13 Omry Yadan 2008-07-14 16:24:25 UTC
Guys, I also find this warning annoying.
I get a pretty much daily email about it from my backup 'failing'.

/etc/cron.daily/snapback2:
rsync warning: some files vanished before they could be transferred (code 24) at main.c(1515) [generator=3.0.2]

I am not even sure which files are vanishing.
I suspect the files are from a mail directory, where some spam messages are moved after rsync builds the files list.

I vote for including the patch into rsync to make it more useful for this use case.
Comment 14 Hervé Eychenne 2009-04-22 03:39:19 UTC
I also find this warning annoying, I really vote for this patch.
Comment 15 Omry Yadan 2009-04-22 03:51:26 UTC
I ended up using the following script for snapback2 (which exits with the same error codes as rsync if rsync fails:

# cat /etc/cron.daily/snapback2
#! /bin/sh
OUT=`/usr/bin/snapback2 2>&1`
RET=$?
if [ "$RET" != "23" -a "$RET" != "0" -a "$RET" != 24 ]; then
    echo "$OUT"
    exit $RET
fi

this is a workaround, it would still be great to be able to tell rsync to ignore vanished files.
Comment 16 Ross Golder 2009-05-18 07:59:00 UTC
This is a 'me too' for '--ignore-vanished-files' flag. Lots of legitimate cases where files could vanish, so having the option to treat this as normal (i.e. prevent rsync from producing this warning and ensure a zero exit value) should be simple enough, and would prevent lots of people (me included) from needing to use wrapper scripts, hacks and/or just give it to receiving the annoying cron messages it causes :)
Comment 17 Jamie Zawinski 2010-10-01 11:27:46 UTC
"Me too."

Doing nightly backups from cron is an *extremely* common use-case of rsync, and telling each and every user of rsync that they should either A) write their own shell script to filter and discard error messages, or B) play a constant game of whack-a-mole chasing down new --exclude options to add, is just silly.  It pushes work downstream that could be done upstream much more efficiently.  Why make each user write the same code over and over when the developer could write it just once, and get it right?

Option A is bad because everyone gets to introduce differently-subtle bugs in their script (wrong error code? accidentally ignoring too many error messages? who knows. It's fragile.)

Option B is bad because I don't *want* to have a bunch of --exclude options for my backup: disk and bandwidth are cheap, so I want my backup disk to be an exact mirror.  That way, when the disk holding / dies, I can just drop the backup in and boot it, being confident that everything is exactly the same.
Comment 18 Jordan Russell 2010-11-20 12:47:15 UTC
+1 for --ignore-vanished. Filtering out the warnings when stdout and stderr are redirected to different places isn't so trivial.

What's also annoying is that a "vanished file" warning triggers:

"IO error encountered -- skipping file deletion"

which if I'm not mistaken stops *all* files from being deleted on the receiving end, not just the ones that have vanished.
Comment 19 Matt McCutchen 2010-11-21 17:57:39 UTC
(In reply to comment #18)
> What's also annoying is that a "vanished file" warning triggers:
> 
> "IO error encountered -- skipping file deletion"
> 
> which if I'm not mistaken stops *all* files from being deleted on the receiving
> end, not just the ones that have vanished.

You're right that the IOERR_GENERAL flag (which corresponds to that message) stops all destination files from being deleted.  But it is only supposed be set when there is an error that might lead to the omission of existing source files from the file list.  A run-of-the-mill vanished file should not set the flag.  If you have a reproducible case in which it does, please file a separate bug.
Comment 20 Jordan Russell 2010-11-21 19:43:34 UTC
(In reply to comment #19)
> You're right that the IOERR_GENERAL flag (which corresponds to that message)
> stops all destination files from being deleted.  But it is only supposed be set
> when there is an error that might lead to the omission of existing source files
> from the file list.  A run-of-the-mill vanished file should not set the flag. 

AFAICS, delete_in_dir() suppresses deletion when /any/ flag is set in io_error:

        if (io_error && !ignore_errors) {
Comment 21 Matt McCutchen 2010-11-21 21:12:22 UTC
(In reply to comment #20)
> AFAICS, delete_in_dir() suppresses deletion when /any/ flag is set in io_error:
> 
>         if (io_error && !ignore_errors) {

Indeed, that is a bug.  I entered bug 7809.
Comment 22 Elan Ruusamäe 2011-01-25 13:55:07 UTC
Created attachment 6228 [details]
updated patch in #c1 against rsync-3.0.7
Comment 23 Elan Ruusamäe 2011-02-11 09:41:59 UTC
i updated patch again as found similar pattern in code

http://cvs.pld-linux.org/cgi-bin/cvsweb.cgi/packages/rsync/ignore-vanished.patch?rev=1.2

but seems the problem is reported on server side, not client side and it still does not work, because the --ignore-vanished apparently is not "transported" to rsync server, and response generated by rsync server would still include those vanished error messages.

totally different approach must be taken perhaps.

to me the rsync exit code ignoring is not a solution, because my problem looks like:

i rsync large disk with Maildir files, rsync alone calculating filelist takes 1.5 hour, so when the sync finishes there are certainly some files moved around. and as for me the delete is also skipped, meaning backup space from old removed files is never released, meaning my backup grows and grows. what would the alternative fix here?


file has VANISHED: "/Maildir/new/1297421485.M458439P3915V000000000000FE04I00000000003EDB8D_0.localhost,S=8354" (in vmail), protocol=30, code=4
delfi.ee/ylle/Maildir/tmp/
IO error encountered -- skipping file deletion

sent 7900103 bytes  received 31665465290 bytes  1978039.99 bytes/sec
total size is 281386128794  speedup is 8.88
rsync warning: some files vanished before they could be transferred (code 24) at main.c(1508) [generator=3.0.7]
Comment 24 Nick Hibma 2012-11-01 08:41:35 UTC
Interesting to see how a simple request is actually a rather intricate little problem in the bigger scheme of things. I'm very interested to see how this is resolved (as I get back up warnings once in a while because of this).

By the way: Suggesting that removing warnings from the error log is silly. Creating a script like this on more than 5 different machines gets old very quickly.
Comment 25 Jakob Unterwurzacher 2014-01-01 15:04:58 UTC
What remains to be done is to make the the rsync-no-vanished script accessible to users. I created bug 10356 about including it in the tarball.
Comment 26 Wayne Davison 2014-01-01 18:38:33 UTC
I should note that the issue with deletions being skipped has been fixed (the file-has-vanished errors were changed into warnings).

The script support/rsync-no-vanished that will be in the next release.
Comment 27 samba 2016-03-20 16:41:38 UTC
Still an issue on rsync-3.1.1-3 as found on Debian 8