Bug 11086 - rsync --delete has stopped to delete destination files that don't exist at the source
Summary: rsync --delete has stopped to delete destination files that don't exist at th...
Status: RESOLVED INVALID
Alias: None
Product: rsync
Classification: Unclassified
Component: core (show other bugs)
Version: 3.1.1
Hardware: x64 Linux
: P5 major (vote)
Target Milestone: ---
Assignee: Wayne Davison
QA Contact: Rsync QA Contact
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-01-31 14:32 UTC by ddd
Modified: 2015-07-13 04:33 UTC (History)
0 users

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description ddd 2015-01-31 14:32:44 UTC
I have a system Linux root partition wich I backup to another disk periodically. I use the following script for that action:

#!/bin/bash
NICE=15
PH="/mnt/b"
cd /

nice -n $NICE rm -v /root/.xsession-errors
nice -n $NICE rm -v /root/dead.letter

nice -n $NICE /usr/bin/rsync -avhHxq --progress --stats --force --delete --delete-before --log-file='/var/log/backup2diskrsync.log' --exclude='/usr/portage/distfiles/' --exclude='/home/ftp/' --exclude='/lost+found/' --exclude='/var/tmp/ccache/' --exclude='/root/.ccache/' --exclude='/var/cache/edb/' --exclude='/home/myuser/.cache/chromium/' --exclude='/home/myuser/.opera/cache/' --exclude='/home/myuser/.opera/opcache/' --exclude='/home/mysearch/search/fsearch/swapfile' --exclude='/home/myuser/workspace/mysearch/swapfile' / $PH/

The problem is that I don't see any "delete" messages in log file anymore. For example, I have the following in /usr/src/ directory on backup disk:
# du -sh /mnt/b/usr/src/*
0       /mnt/b/usr/src/linux
944M    /mnt/b/usr/src/linux-3.14.14-gentoo
961M    /mnt/b/usr/src/linux-3.16.5-gentoo
964M    /mnt/b/usr/src/linux-3.17.7-gentoo
972M    /mnt/b/usr/src/linux-3.18.3-gentoo
971M    /mnt/b/usr/src/linux-3.18.5-gentoo
But I don't have this outdated long time ago directories on my root partition:
# du -sh /usr/src/*
0       /usr/src/linux
943M    /usr/src/linux-3.18.5-gentoo

And that's just a one of examples of outdated files and directories. My backup storage partition will become full soon.

The rsync log (after # /bin/bash backup2diskwithrsync.sh under root):
2015/01/31 17:25:56 [12091] building file list
2015/01/31 17:25:58 [12091] rsync: readlink_stat("/home/myuser/.gvfs") failed: Permission denied (13)
2015/01/31 17:25:59 [12091] 877346 files to consider
2015/01/31 17:26:01 [12091] >f..t...... etc/openvpn/openvpn-status.log
2015/01/31 17:26:01 [12091] >f..t...... home/incoming/scripts/enp2s0-upd.err
2015/01/31 17:26:01 [12091] >f..t...... home/incoming/scripts/enp2s0-upd.info
2015/01/31 17:26:01 [12091] >f..t...... home/incoming/scripts/ppp0-upd.err
2015/01/31 17:26:01 [12091] >f..t...... home/incoming/scripts/ppp0-upd.info
2015/01/31 17:26:03 [12091] .d..t...... var/log/
2015/01/31 17:26:03 [12091] >f.st...... var/log/backup2diskrsync.log
2015/01/31 17:26:03 [12091] >f.st...... var/log/messages
2015/01/31 17:26:03 [12091] >f.st...... var/log/net-snmpd.log
2015/01/31 17:26:04 [12091] >f.st...... var/log/samba/log.nmbd
2015/01/31 17:26:05 [12091] >f..t...... var/tmp/kdecache-root/icon-cache.kcache
2015/01/31 17:26:05 [12091] >f..t...... var/tmp/kdecache-myuser/plasma_theme_G-Remix-T_v5.3.kcache
2015/01/31 17:26:06 [12091] >f..t...... var/tmp/kdecache-myuser/plasma_theme_internal-system-colors.kcache
2015/01/31 17:26:06 [12091] .d..t...... var/www/localhost/htdocs/rutorrent/share/users/myuserutorrent/settings/erasedata/
2015/01/31 17:26:07 [12091] Number of files: 877,346 (reg: 758,168, dir: 67,562, link: 45,877, dev: 5,678, special: 61)
2015/01/31 17:26:07 [12091] Number of created files: 0
2015/01/31 17:26:07 [12091] Number of deleted files: 0
2015/01/31 17:26:07 [12091] Number of regular files transferred: 12
2015/01/31 17:26:07 [12091] Total file size: 25.31G bytes
2015/01/31 17:26:07 [12091] Total transferred file size: 379.83M bytes
2015/01/31 17:26:07 [12091] Literal data: 379.83M bytes
2015/01/31 17:26:07 [12091] Matched data: 0 bytes
2015/01/31 17:26:07 [12091] File list size: 24.44M
2015/01/31 17:26:07 [12091] File list generation time: 3.099 seconds
2015/01/31 17:26:07 [12091] File list transfer time: 0.000 seconds
2015/01/31 17:26:07 [12091] Total bytes sent: 404.40M
2015/01/31 17:26:07 [12091] Total bytes received: 291
2015/01/31 17:26:07 [12091] sent 404.40M bytes  received 291 bytes  35.17M bytes/sec
2015/01/31 17:26:07 [12091] total size is 25.31G  speedup is 62.60
2015/01/31 17:26:07 [12091] rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1165) [sender=3.1.1]

Error log in the console (some files it doesn't like probably)
rsync: readlink_stat("/home/xor/.gvfs") failed: Permission denied (13)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1165) [sender=3.1.1]

And still there are a lot of garbage on backup partition wich had to be deleted a looong time ago:
# du -sh /mnt/b/usr/src/*
0       /mnt/b/usr/src/linux
944M    /mnt/b/usr/src/linux-3.14.14-gentoo
961M    /mnt/b/usr/src/linux-3.16.5-gentoo
964M    /mnt/b/usr/src/linux-3.17.7-gentoo
972M    /mnt/b/usr/src/linux-3.18.3-gentoo
971M    /mnt/b/usr/src/linux-3.18.5-gentoo
Comment 1 Kevin Korb 2015-01-31 17:45:28 UTC
This is your problem:
> 2015/01/31 17:25:58 [12091] rsync: readlink_stat("/home/myuser/.gvfs") failed: Permission denied (13)

If rsync hits any error it aborts the deletion process which it would have said without -q.  You can override that with --ignore-errors but better to --exclude=.gvfs since it isn't a real file/dir/filesystem.

Also, if you are making backups you should really look into rsync --link-dest or an rsync based backup system if you don't feel like scripting it.

Also, --delete-before is much slower than --delete-during which is the default version of --delete.
Comment 2 ddd 2015-01-31 22:24:29 UTC
Thank you a lot! I added --exclude='.gvfs' parameter and everything works now. Thumbs up!

I was trying to use --delete-before because of low space on the partition. Now it's really enough space and I use default --delete option.

> Also, if you are making backups you should really look into rsync --link-dest or an rsync based backup system if you don't feel like scripting it.
Hmm, what --link-dest stands for? Is it somehow related to hard-links on the same filesystem? I usually try to exclude hardlinks like --exclude='/home/ftp/' wich usually works for me.

--link-dest=DIR         hardlink to files in DIR when unchanged
I don't understand what that option is doing.
I use rsync for easy backups of the system partition from the primary disk to a backup one 2 times a week. It's very useful when I delete smth by accident like one time I did "crontab -r" instead of "crontab -e". ) And in case 1st system drive fails I change SATA interface immediately and start OS from the backup one. Rsync based backup system? I've seen some of those. They usually don't do what I need (like they backup to files, snapshots with dates in them instead of simple backup to another disk wich consumes too much space and you'll have to use liveDVD to start from such snapshot wich is time-consuming) or too overburdened with options, configs etc. My scripts is very simple and does exactly what I need.
Comment 3 Kevin Korb 2015-01-31 22:32:52 UTC
What you have is a mirror not a backup system.  Sure, it can protect you from accidental deletions or disk failures but it wouldn't protect you from intentional data destruction (think virus or hacker).

Yes, I was referring to date+time based "snapshots".  That is what rsync with --link-dest and some scripting can do and that is what most of the rsync based backup systems use.

I prefer to have many backups in a system.  That way I can restore data from the past if I need to (which I have).  It is pretty trivial to boot from something like SystemRescueCD and restore your whole system from such a backup.  In fact, if your backups are on a different system and you setup a network boot environment you can boot from the backup over the network.  I did that once when my desktop became unstable and I was trying to figure out if it was a hardware problem or if it was caused by recent updates.  When I could duplicate a several day old problem on your computer with the software it was running a month ago I knew it was the hardware that was bad.  Turned out there was a leaking capacitor behind the heatsink on the video card.
Comment 4 ddd 2015-01-31 23:03:48 UTC
Well, maybe I should look into those rsync-based snapshot systems when I'll buy another 3 TB HDD. ) Can you recommend any of those backup systems?
Comment 5 Kevin Korb 2015-01-31 23:48:01 UTC
Not really.  I wrote my own.  IIRC BackupPC is the most up to date one.