Bug 2734 - --delete doesn't work
--delete doesn't work
Status: CLOSED INVALID
Product: rsync
Classification: Unclassified
Component: core
2.6.5
All Linux
: P3 major
: ---
Assigned To: Wayne Davison
Rsync QA Contact
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2005-05-21 16:25 UTC by Martin Jeppesen
Modified: 2005-06-03 01:39 UTC (History)
0 users

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Martin Jeppesen 2005-05-21 16:25:34 UTC
When I use this 

rsync -arRWxn --delete --delete-excluded --log-format='%i %f      %l'
--exclude-from=/home/maj/.rsync-exclude-bootlist / /mnt/backupboot

files one the destianation side are not being deleted, even if they are not the
source side.

--delete ought to delete files on the destination side, that are not on the
source side.
Comment 1 John Van Essen 2005-05-21 22:15:13 UTC
Your rsync options (-arRWxn) include -n, so nothing is actually going to get
deleted on the destination, even though rsync reports that they are being
deleted (which is what -n is for - i.e. to preview what will happen).
Comment 2 Wayne Davison 2005-05-21 23:13:24 UTC
Closing due to use of -n (--dry-run).
Comment 3 Martin Jeppesen 2005-05-22 01:50:12 UTC
The problem is that rsync doesn't report any files to be deleted, and it should,
as the source side is 3,6GB and the dest side is 3,8GB.
Comment 4 Wayne Davison 2005-05-22 10:45:06 UTC
I've tried the command you specified (supplying my own list of exclusions, since
you didn't mention any) and it reports deletions as expected.  Keep in mind two
things:

1. When log-format specifies %i, deletes are logged using the log-format you
specified with "*delete" in the %i field.

2. When -x is specified, any mounted directories will be excluded from the
deletions (that would be any dirs are mounted inside /mnt/backupboot, not if
that dir itself is mounted).

If that doesn't help you, I'll need more information on what is going wrong. 
You might also try removing the -R option (which shouldn't make any difference,
but it's not needed with a source path of "/").
Comment 5 Wayne Davison 2005-05-22 15:18:26 UTC
One other possibility occurred to me:  if rsync encounted some kind of an error
reading the source files (such as a directory that it couldn't open), it will
output this warning "IO error encountered -- skipping file deletion".  Perhaps
that is the root of the missing deletes?  If so, you can specify --ignore-errors
to tell rsync to proceed with the deletions anyway.
Comment 6 Martin Jeppesen 2005-05-23 04:47:47 UTC
Replying to your two latest posts.

When I use the log-format with the %i I do see some files to be deleted, but not
enough.

I don't tend to backup across mounted partitions, so I use -x inorder to not
ending up then an everending backup, as the dest can be reached from /, which is
from where I do the back up.

I have not removed the -R. So my command now looks like:
rsync -avrWxn --delete --delete-excluded --log-format='%i %f'
--exclude-from=/home/maj/.rsync-exclude-bootlist / /mnt/backupboot

The files that are not being deleted seams to be in
/var
/usr
/lib


This is from the source side:
/# du -hsm lib/ usr/ var/
666     lib/
3183    usr/
102     var/
/#


This is from the dest side:
/home/maj/.bin# du -hsm /mnt/backupboot/*
6       /mnt/backupboot/bin
32      /mnt/backupboot/boot
1       /mnt/backupboot/dev
57      /mnt/backupboot/etc
1       /mnt/backupboot/home
1       /mnt/backupboot/initrd
952     /mnt/backupboot/lib
1       /mnt/backupboot/lost+found
1       /mnt/backupboot/media
1       /mnt/backupboot/misc
1       /mnt/backupboot/mnt
1       /mnt/backupboot/opt
13      /mnt/backupboot/root
16      /mnt/backupboot/sbin
1       /mnt/backupboot/selinux
1       /mnt/backupboot/srv
1       /mnt/backupboot/sys
8       /mnt/backupboot/tmp
3208    /mnt/backupboot/usr
113     /mnt/backupboot/var
/home/maj/.bin#


--ignore-errors didn't help.

If I look closer in /lib I see

~# du -hsm /lib/modules/
643     /lib/modules/
~# du -hsm /lib/modules/*
5       /lib/modules/2.6.10
101     /lib/modules/2.6.10-1.737_FC3
102     /lib/modules/2.6.10-1.741_FC3
103     /lib/modules/2.6.10-1.760_FC3
103     /lib/modules/2.6.10-1.766_FC3
103     /lib/modules/2.6.10-1.770_FC3
107     /lib/modules/2.6.11-1.14_FC3
5       /lib/modules/2.6.7
5       /lib/modules/2.6.9
97      /lib/modules/2.6.9-1.667
98      /lib/modules/2.6.9-1.681_FC3
98      /lib/modules/2.6.9-1.724_FC3
5       /lib/modules/2.6.9-prep
~# 

Notice the 643MB and that the dir. sizes are 932MB


On the dest side:
~# du -hsm /mnt/backupboot/lib/modules/
928     /mnt/backupboot/lib/modules/
~# du -hsm /mnt/backupboot/lib/modules/*
5       /mnt/backupboot/lib/modules/2.6.10
101     /mnt/backupboot/lib/modules/2.6.10-1.737_FC3
102     /mnt/backupboot/lib/modules/2.6.10-1.741_FC3
103     /mnt/backupboot/lib/modules/2.6.10-1.760_FC3
103     /mnt/backupboot/lib/modules/2.6.10-1.766_FC3
103     /mnt/backupboot/lib/modules/2.6.10-1.770_FC3
107     /mnt/backupboot/lib/modules/2.6.11-1.14_FC3
5       /mnt/backupboot/lib/modules/2.6.7
5       /mnt/backupboot/lib/modules/2.6.9
97      /mnt/backupboot/lib/modules/2.6.9-1.667
98      /mnt/backupboot/lib/modules/2.6.9-1.681_FC3
98      /mnt/backupboot/lib/modules/2.6.9-1.724_FC3
5       /mnt/backupboot/lib/modules/2.6.9-prep
~# 

Notice the 928MB and that the dir. sizes are 932MB


This doesn't make sense to me, how the dest side can be larger than the source
side, and that the dir. sises are the same.

The problems seamed to start when I switched from rsync 2.6.3.
Comment 7 Wayne Davison 2005-05-23 13:06:15 UTC
1. It's quite easy for a different filesystem to have a different block size, so
du often produces very different results for an identical set of files on two
different filesystems.

2. You're still using -n, so running du after _not_updating_ the destination
means that the filesystems haven't been synchronized, and thus cannot be
expected to match in size even if the block sizes of the filesystems are the same.

Since you didn't mention any extra files that should have been deleted, I'm left
to assume that there are none.
Comment 8 John Van Essen 2005-05-23 14:06:20 UTC
You really need to compare the contents of the directories.  Try this:

cd /lib
ls -lAR >/tmp/ls1
cd /mnt/backupboot/lib
ls -lAR >/tmp/ls2
cd /tmp
diff ls1 ls2

That should tell you exactly what is different (assuming you are not excluding
anything within /lib in rsync).
Comment 9 Martin Jeppesen 2005-05-24 13:57:03 UTC
[different blocksizes]
Can this be an issue, when I messure enverthing i megabytes with "du -hsm"?

[-n doesn't update]
Now I udnerstand what you mean=)

I have forgot to mention a rather essential information. Before the rsync
command with -n, have I just run the same command without -n.

As the disks ought to be sync'ed, I use the -n just for safety reasons.

[compare with ls -lAR]
When I now have rsync'ed, my destination disk have run out of space.

cL+++++++ usr/lib/wine/user.exe.so
>f+++++++ usr/lib/wine/user32.dll.so
rsync: writefd_unbuffered failed to write 4 bytes: phase "unknown" [sender]:
Broken pipe (32)
rsync: write failed on "/mnt/backupboot/usr/lib/wine/user32.dll.so": No space
left on device (28)
rsync error: error in file IO (code 11) at receiver.c(291)
rsync: connection unexpectedly closed (5023511 bytes received so far) [generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(434)
rsync: connection unexpectedly closed (17556 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(434)
#

Which to my understanding should not be the case:
# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/hda2             3,8G  3,4G  175M  96% /
none                  189M     0  189M   0% /dev/shm
/dev/hda3             109G  105G  4,5G  96% /home
172.20.1.1:/home/ftp/pub/userspace
                      132G   28G   98G  22% /mnt/userspace
172.20.1.1:/home/maj  132G   28G   98G  22% /mnt/maj
/dev/hdd2             3,7G  3,7G     0 100% /mnt/backupboot
/home/maj/.bin#

Notice /dev/hda2 (source) have 3,4GB used of 3,8GB and
       /dev/hdd2 (dest.)  have 3,7GB used of 3,7GB.

The source side is 300MB smaller then the destination side, so there ought to be
space.

rsync starts be deleting files, so at this point I still ought to use your lib
compare script.

# ls -lAR /lib/ > /tmp/ls1
# ls -lAR /mnt/backupboot/lib/ > /tmp/ls2
# diff /tmp/ls1 /tmp/ls2
# diff /tmp/ls1 /tmp/ls2 > /tmp/ls-diff
# wc -l /tmp/ls-diff
205894 /tmp/ls-diff
#

# ll /tmp/ls-diff
-rw-r--r--  1 root root 8,6M May 24 22:49 /tmp/ls-diff
#

As you can see, it is a rather large file, so I have put it up at
http://www.heko.dk/~maj/ls-diff.bz2

Now only 600kb as bz2.

An example from the file:
---
> /mnt/backupboot/lib/modules/2.6.10-1.737_FC3/build/arch/um/sys-i386/util:
3227c3227
< -rw-r--r--  8 root root 211 Jan 10 20:00 Makefile
---
> -rw-r--r--  1 root root 211 Jan 10 20:00 Makefile
3229c3229
< /lib/modules/2.6.10-1.737_FC3/build/arch/um/sys-ia64:
---
> /mnt/backupboot/lib/modules/2.6.10-1.737_FC3/build/arch/um/sys-ia64:
3231c3231
< -rw-r--r--  8 root root 152 Jan 10 20:00 Makefile
---
> -rw-r--r--  1 root root 152 Jan 10 20:00 Makefile
3233c3233
< /lib/modules/2.6.10-1.737_FC3/build/arch/um/sys-ppc:
---
> /mnt/backupboot/lib/modules/2.6.10-1.737_FC3/build/arch/um/sys-ppc:
3235c3235
< -rw-r--r--  8 root root 1,8k Jan 10 20:00 Makefile
---
> -rw-r--r--  1 root root 1,8k Jan 10 20:00 Makefile
3237c3237
< /lib/modules/2.6.10-1.737_FC3/build/arch/um/util:
---

Notice how the checksum always differ by 2.
Comment 10 Wayne Davison 2005-05-24 16:09:43 UTC
You'll note that the differences mention a change in the link count.  Since
you're not specifying -H (--hard-links), that can result in a lot of extra space
being taken up on the destination due to files being duplicated instead of
hard-linked together.
Comment 11 Martin Jeppesen 2005-05-24 17:38:08 UTC
You are right! =)

Now it works! =)

Sorry for this false bug report, and thank you very much for your help!
Comment 12 Wayne Davison 2005-05-24 17:59:34 UTC
Closing...