Bug 8990 - It might be nice to make --append-verify also transfer non-appending files
Summary: It might be nice to make --append-verify also transfer non-appending files
Status: RESOLVED WONTFIX
Alias: None
Product: rsync
Classification: Unclassified
Component: core (show other bugs)
Version: 3.0.9
Hardware: All All
: P5 enhancement (vote)
Target Milestone: ---
Assignee: Wayne Davison
QA Contact: Rsync QA Contact
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2012-06-08 09:25 UTC by Matt A
Modified: 2020-07-27 23:43 UTC (History)
0 users

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Matt A 2012-06-08 09:25:25 UTC
Example:

$ echo "hello" > test1.txt

$ echo "world" > test2.txt

$ rsync --append test1.txt test2.txt

$ diff test1.txt test2.txt
--- test1.txt   2012-06-08 10:23:31.059215000 +0100
+++ test2.txt   2012-06-08 10:23:36.891225000 +0100
@@ -1 +1 @@
-hello
+world

$ rsync --append-verify test1.txt test2.txt

$ diff test1.txt test2.txt
--- test1.txt   2012-06-08 10:23:31.059215000 +0100
+++ test2.txt   2012-06-08 10:23:36.891225000 +0100
@@ -1 +1 @@
-hello
+world

$ rsync --append-verify --checksum test1.txt test2.txt

$ diff test1.txt test2.txt
--- test1.txt   2012-06-08 10:23:31.059215000 +0100
+++ test2.txt   2012-06-08 10:23:36.891225000 +0100
@@ -1 +1 @@
-hello
+world

$ rsync --append-verify --checksum --ignore-times test1.txt test2.txt

$ diff test1.txt test2.txt
--- test1.txt   2012-06-08 10:23:31.059215000 +0100
+++ test2.txt   2012-06-08 10:23:36.891225000 +0100
@@ -1 +1 @@
-hello
+world

Bug confirmed in rsync-3.0.6 and 3.0.9. This syncs correctly with rsync-2.6.9 (using only --append).
Comment 1 Wayne Davison 2012-06-16 17:03:30 UTC
The way append currently works is that it only transfers files that have data to append.  From the manpage:

"If a file needs to be transferred and its size on the receiver is
the same or longer than the size on the sender, the file is skipped."

Thus, it is assumed that if you are using --append (or --append-verify) that you have narrowed down your transfer list to just files that need to be appended and that you know have unchanged data prior to the append bits.

It might be nice to allow rsync to do a non-appending transfer for any files it finds that have a non-longer length (possibly only with --append-verify).  Such a feature would require some way to alert the sender what kind of file is currently being processed (possibly using the sum data fields), and that is not currently possible.  I'm thinking that we don't want to do that with --append because it is already dangerous if your transfer list is not limited to just the files that are known good besides needing some appending.  The --append-verify option is already telling rsync that you aren't sure about the files, so it might be nice if that option also mode did more updating work.  However, such a change would be incompatible with older rsyncs and is not currently planned.

I'll change this into an enhancement request for future consideration.
Comment 2 Matt A 2012-06-26 08:07:39 UTC
(In reply to comment #1)
> Thus, it is assumed that if you are using --append (or --append-verify) that
> you have narrowed down your transfer list to just files that need to be
> appended and that you know have unchanged data prior to the append bits.

I can see that that is the intended behaviour for --append, but then what does --append-verify do differently? Prior to version 3.0.0, --append did check that the files matched after attempting an append, and the way the documentation is written gives the impression that this behaviour is now implemented by --append-verify. To quote:

  --append-verify
              This  works  just  like the --append option, but the existing data on the receiving side is included in the full-file checksum verification step, which will cause a file to be resent if the final verification step fails (rsync uses a normal, non-appending --inplace transfer for the resend).

              Note: prior to rsync 3.0.0, the --append option worked like --append-verify, so if you are interacting with an older rsync (or the transfer is using a protocol prior  to  30), specifying either append option will initiate an --append-verify transfer.

I'm pretty sure the current behaviour is a bug.
Comment 3 Andrey Bond 2016-01-26 09:40:49 UTC
Hello,
I make backup users data. There is the problem with file that are reduced.

For example there is the file outlook.pst with size more 2GB. I use the --inplace and --append-verify options. When this file become larger the sync is working properly. But if this file become smaller, because the user erase some messages, the file will not be syncroniced.

Now I start rsync twice for backup of snapshot with --append-verify after whithout --append-verify. If I start rsync without --append-verify first the rsync copy the whole file (instead of copying the changes) if size is larger then distenation file.

I think the need to check all files with modified date by checksum. May be with aditional option.
Comment 4 Wayne Davison 2020-07-27 23:43:41 UTC
The --append* options should be targetted at just log files that are known to be good except for some missing trailing data.  The man page has been updated to warn more about its use.