I have a system Linux root partition wich I backup to another disk periodically. I use the following script for that action: #!/bin/bash NICE=15 PH="/mnt/b" cd / nice -n $NICE rm -v /root/.xsession-errors nice -n $NICE rm -v /root/dead.letter nice -n $NICE /usr/bin/rsync -avhHxq --progress --stats --force --delete --delete-before --log-file='/var/log/backup2diskrsync.log' --exclude='/usr/portage/distfiles/' --exclude='/home/ftp/' --exclude='/lost+found/' --exclude='/var/tmp/ccache/' --exclude='/root/.ccache/' --exclude='/var/cache/edb/' --exclude='/home/myuser/.cache/chromium/' --exclude='/home/myuser/.opera/cache/' --exclude='/home/myuser/.opera/opcache/' --exclude='/home/mysearch/search/fsearch/swapfile' --exclude='/home/myuser/workspace/mysearch/swapfile' / $PH/ The problem is that I don't see any "delete" messages in log file anymore. For example, I have the following in /usr/src/ directory on backup disk: # du -sh /mnt/b/usr/src/* 0 /mnt/b/usr/src/linux 944M /mnt/b/usr/src/linux-3.14.14-gentoo 961M /mnt/b/usr/src/linux-3.16.5-gentoo 964M /mnt/b/usr/src/linux-3.17.7-gentoo 972M /mnt/b/usr/src/linux-3.18.3-gentoo 971M /mnt/b/usr/src/linux-3.18.5-gentoo But I don't have this outdated long time ago directories on my root partition: # du -sh /usr/src/* 0 /usr/src/linux 943M /usr/src/linux-3.18.5-gentoo And that's just a one of examples of outdated files and directories. My backup storage partition will become full soon. The rsync log (after # /bin/bash backup2diskwithrsync.sh under root): 2015/01/31 17:25:56 [12091] building file list 2015/01/31 17:25:58 [12091] rsync: readlink_stat("/home/myuser/.gvfs") failed: Permission denied (13) 2015/01/31 17:25:59 [12091] 877346 files to consider 2015/01/31 17:26:01 [12091] >f..t...... etc/openvpn/openvpn-status.log 2015/01/31 17:26:01 [12091] >f..t...... home/incoming/scripts/enp2s0-upd.err 2015/01/31 17:26:01 [12091] >f..t...... home/incoming/scripts/enp2s0-upd.info 2015/01/31 17:26:01 [12091] >f..t...... home/incoming/scripts/ppp0-upd.err 2015/01/31 17:26:01 [12091] >f..t...... home/incoming/scripts/ppp0-upd.info 2015/01/31 17:26:03 [12091] .d..t...... var/log/ 2015/01/31 17:26:03 [12091] >f.st...... var/log/backup2diskrsync.log 2015/01/31 17:26:03 [12091] >f.st...... var/log/messages 2015/01/31 17:26:03 [12091] >f.st...... var/log/net-snmpd.log 2015/01/31 17:26:04 [12091] >f.st...... var/log/samba/log.nmbd 2015/01/31 17:26:05 [12091] >f..t...... var/tmp/kdecache-root/icon-cache.kcache 2015/01/31 17:26:05 [12091] >f..t...... var/tmp/kdecache-myuser/plasma_theme_G-Remix-T_v5.3.kcache 2015/01/31 17:26:06 [12091] >f..t...... var/tmp/kdecache-myuser/plasma_theme_internal-system-colors.kcache 2015/01/31 17:26:06 [12091] .d..t...... var/www/localhost/htdocs/rutorrent/share/users/myuserutorrent/settings/erasedata/ 2015/01/31 17:26:07 [12091] Number of files: 877,346 (reg: 758,168, dir: 67,562, link: 45,877, dev: 5,678, special: 61) 2015/01/31 17:26:07 [12091] Number of created files: 0 2015/01/31 17:26:07 [12091] Number of deleted files: 0 2015/01/31 17:26:07 [12091] Number of regular files transferred: 12 2015/01/31 17:26:07 [12091] Total file size: 25.31G bytes 2015/01/31 17:26:07 [12091] Total transferred file size: 379.83M bytes 2015/01/31 17:26:07 [12091] Literal data: 379.83M bytes 2015/01/31 17:26:07 [12091] Matched data: 0 bytes 2015/01/31 17:26:07 [12091] File list size: 24.44M 2015/01/31 17:26:07 [12091] File list generation time: 3.099 seconds 2015/01/31 17:26:07 [12091] File list transfer time: 0.000 seconds 2015/01/31 17:26:07 [12091] Total bytes sent: 404.40M 2015/01/31 17:26:07 [12091] Total bytes received: 291 2015/01/31 17:26:07 [12091] sent 404.40M bytes received 291 bytes 35.17M bytes/sec 2015/01/31 17:26:07 [12091] total size is 25.31G speedup is 62.60 2015/01/31 17:26:07 [12091] rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1165) [sender=3.1.1] Error log in the console (some files it doesn't like probably) rsync: readlink_stat("/home/xor/.gvfs") failed: Permission denied (13) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1165) [sender=3.1.1] And still there are a lot of garbage on backup partition wich had to be deleted a looong time ago: # du -sh /mnt/b/usr/src/* 0 /mnt/b/usr/src/linux 944M /mnt/b/usr/src/linux-3.14.14-gentoo 961M /mnt/b/usr/src/linux-3.16.5-gentoo 964M /mnt/b/usr/src/linux-3.17.7-gentoo 972M /mnt/b/usr/src/linux-3.18.3-gentoo 971M /mnt/b/usr/src/linux-3.18.5-gentoo
This is your problem: > 2015/01/31 17:25:58 [12091] rsync: readlink_stat("/home/myuser/.gvfs") failed: Permission denied (13) If rsync hits any error it aborts the deletion process which it would have said without -q. You can override that with --ignore-errors but better to --exclude=.gvfs since it isn't a real file/dir/filesystem. Also, if you are making backups you should really look into rsync --link-dest or an rsync based backup system if you don't feel like scripting it. Also, --delete-before is much slower than --delete-during which is the default version of --delete.
Thank you a lot! I added --exclude='.gvfs' parameter and everything works now. Thumbs up! I was trying to use --delete-before because of low space on the partition. Now it's really enough space and I use default --delete option. > Also, if you are making backups you should really look into rsync --link-dest or an rsync based backup system if you don't feel like scripting it. Hmm, what --link-dest stands for? Is it somehow related to hard-links on the same filesystem? I usually try to exclude hardlinks like --exclude='/home/ftp/' wich usually works for me. --link-dest=DIR hardlink to files in DIR when unchanged I don't understand what that option is doing. I use rsync for easy backups of the system partition from the primary disk to a backup one 2 times a week. It's very useful when I delete smth by accident like one time I did "crontab -r" instead of "crontab -e". ) And in case 1st system drive fails I change SATA interface immediately and start OS from the backup one. Rsync based backup system? I've seen some of those. They usually don't do what I need (like they backup to files, snapshots with dates in them instead of simple backup to another disk wich consumes too much space and you'll have to use liveDVD to start from such snapshot wich is time-consuming) or too overburdened with options, configs etc. My scripts is very simple and does exactly what I need.
What you have is a mirror not a backup system. Sure, it can protect you from accidental deletions or disk failures but it wouldn't protect you from intentional data destruction (think virus or hacker). Yes, I was referring to date+time based "snapshots". That is what rsync with --link-dest and some scripting can do and that is what most of the rsync based backup systems use. I prefer to have many backups in a system. That way I can restore data from the past if I need to (which I have). It is pretty trivial to boot from something like SystemRescueCD and restore your whole system from such a backup. In fact, if your backups are on a different system and you setup a network boot environment you can boot from the backup over the network. I did that once when my desktop became unstable and I was trying to figure out if it was a hardware problem or if it was caused by recent updates. When I could duplicate a several day old problem on your computer with the software it was running a month ago I knew it was the hardware that was bad. Turned out there was a leaking capacitor behind the heatsink on the video card.
Well, maybe I should look into those rsync-based snapshot systems when I'll buy another 3 TB HDD. ) Can you recommend any of those backup systems?
Not really. I wrote my own. IIRC BackupPC is the most up to date one.