Bug 10724 - rsync 3.1.1 creates extra dirs inside --backup-dir
rsync 3.1.1 creates extra dirs inside --backup-dir
Status: RESOLVED FIXED
Product: rsync
Classification: Unclassified
Component: core
3.1.1
x86 All
: P5 normal
: ---
Assigned To: Wayne Davison
Rsync QA Contact
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2014-07-18 03:20 UTC by chdiza
Modified: 2015-08-30 15:38 UTC (History)
3 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description chdiza 2014-07-18 03:20:05 UTC
On MacOSX 10.9.4, new to rsync 3.1.1, I observe the incorrect creation of empty dirs inside a --backup-dir.

To reproduce:

1. mkdir foo
2. mkdir bar
3. cd foo; touch fun
4. cd ..
5. rsync -trlvhp foo bar

Everything is as it should be: inside bar there is a clone of foo.  But now continue (from the same working dir as from setp 5):

6. mkdir foo/newdir
7. cd foo; mv fun newdir; cd ..
8. rsync -trlvhpb --delete --backup-dir=mybackups --suffix=~ foo bar

Expected result: the directory bar should look like this:

bar/
bar/foo/
bar/foo/newdir/fun
bar/mybackups/
bar/mybackups/foo/
bar/mybackups/foo/fun~

Actual result: rsync 3.1.1 has created an empty dir "newdir" inside
mybackups.  That is, this is what bar looks like:

bar/
bar/foo/
bar/foo/newdir/fun
bar/mybackups/
bar/mybackups/foo/
bar/mybackups/foo/fun~
bar/mybackups/foo/newdir/

As you can see there is magically a "newdir" there.  It shouldn't be
there, because there never was a "newdir" inside bar/foo prior to step 8.
Therefore, there was no "newdir" there to be "backed up" into mybackups.

This behavior is new to 3.1.1 as best as I can tell.  I'm positive it never occurred
on any prior version of rsync that I've ever used (2.6-ish?).

The above was just a toy example designed to illustrate the issue.  In practice, rsync 3.1.1 left dozens of such "ghost" directories inside my --backup-dir.
Comment 1 chdiza 2014-07-18 03:28:52 UTC
My apologies, this bug is also present on rsync 3.1.0.

I still think it's a bug.  Also, I was almost right; it didn't work like this on 3.0.9 and earlier.
Comment 2 chdiza 2014-07-18 04:26:08 UTC
For fun I cloned the git repo and bisected.  The first commit that exhibits the aforementioned buggy behavior is: 3696674bc62b0c1250027dbeedafdd7ebafdcf8b

which carries the description "More backup improvements".
Comment 3 David 2014-08-10 12:25:58 UTC
I wonder if that commit was trying to fix another bug that exists in at least 3.1.0, and possibly other versions.  When an empty directory is deleted from source but exists in target, and the --delete option is enabled, the empty directory is not placed in --backup-dir.  Going to see if I can figure out what exactly the commit changed.
Comment 4 David 2014-08-10 12:35:52 UTC
(In reply to comment #3)
> I wonder if that commit was trying to fix another bug that exists in at least
> 3.1.0, and possibly other versions.  When an empty directory is deleted from
> source but exists in target, and the --delete option is enabled, the empty
> directory is not placed in --backup-dir.  Going to see if I can figure out what
> exactly the commit changed.

Sorry for the self-reply, but I just verified that the bug I noticed exists in 3.0.9, so the commit 3696674bc62b0c1250027dbeedafdd7ebafdcf8b is not related.  Looks like this needs a new bug report.
Comment 5 Wayne Davison 2015-08-02 17:17:34 UTC
*** Bug 11423 has been marked as a duplicate of this bug. ***
Comment 6 Wayne Davison 2015-08-02 17:22:53 UTC
In rsync 3.1.x the backup logic was improved to ensure that older, conflicting files/symlinks couldn't cause problems in a new backup.  For instance, if a file changed to a directory (and was backed up) and then later a file needed to be backed up in the new path, the lingering file that now needs to be a directory caused rsync to die.  The new logic is to ensure that every dir that exists in the transfer is now a directory in the backup area.  This logic could be amended to instead ensure that any conflicts are removed, and the dir creation delayed until needed.
Comment 7 Wayne Davison 2015-08-08 19:20:44 UTC
This is now fixed in the git source, and will be in the next release.
Comment 8 adsh 2015-08-13 01:31:37 UTC
Bug is NOT fixed:

rsync --version (on host and client machines)
rsync  version 3.1.2pre1  protocol version 31
...

rsync -btrhv -e "ssh -i /root/.ssh/id_ecdsa_rsync_server" --delete-after \
--backup-dir=/var/backups/server.ua/pkg/old/`date +%F` \
root@server.ua:/home/user/soft/packages/ /var/backups/server.ua/pkg/data

receiving file list ... done
Created backup_dir /var/backups/server.ua/pkg/old/2015-08-13/

sent 19 bytes  received 7.06K bytes  4.72K bytes/sec
total size is 188.73M  speedup is 26,664.10

ls -l /var/backups/server.ua/pkg/old/2015-08-13/
total 0

---

rsync --version (on host and client machines)
rsync  version 3.0.9  protocol version 30
...

rmdir /var/backups/server.ua/pkg/old/2015-08-13/

rsync -btrhv -e "ssh -i /root/.ssh/id_ecdsa_rsync_server" --delete-after \
--backup-dir=/var/backups/server.ua/pkg/old/`date +%F` \
root@server.ua:/home/user/soft/packages/ /var/backups/server.ua/pkg/data

receiving file list ... done

sent 11 bytes  received 7.06K bytes  14.13K bytes/sec
total size is 188.73M  speedup is 26705.61
Comment 9 adsh 2015-08-13 01:44:11 UTC
Please read https://bugzilla.samba.org/show_bug.cgi?id=10724 carefully.
Comment 10 adsh 2015-08-13 01:46:55 UTC
Sorry - read this link: https://bugzilla.samba.org/show_bug.cgi?id=11423
Comment 11 Wayne Davison 2015-08-13 22:29:21 UTC
Yes, the top-level backup dir is created at the start of the transfer.  I don't plan to change that.
Comment 12 chdiza 2015-08-30 15:38:40 UTC
I can confirm that on rsync 3.1.2pre, the reproduction example I gave at the start of this thread no longer replicates the issue.  That is, 3.1.2pre appears at first blush to have fixed the problem.

Thanks!