1. Everything is under one "/" filesystem. 2. When I run this: # rsync -av --delete /apps50/backup/./home/ape/ /backup01/ape.5.20040702010001/ I got this. Please note the error at the end. The target directory contains the old backup 1 week ago, it is refreshed with the current files. building file list ... done apps01/tg/jobs/log/download_reply/ apps01/tg/jobs/log/download_reply/download_reply.log.2004-06-29_09:48 apps01/tg/jobs/log/download_reply/download_reply.log.2004-06-29_09:58 apps01/tg/jobs/log/download_reply/download_reply.log.2004-06-29_10:08 apps01/tg/jobs/log/download_reply/download_reply.log.2004-06-29_10:18 apps01/tg/jobs/log/download_reply/download_reply.log.2004-06-29_10:28 apps01/tg/jobs/log/download_reply/download_reply.log.2004-06-29_10:38 apps01/tg/jobs/log/download_reply/download_reply.log.2004-06-29_10:48 apps01/tg/jobs/log/download_reply/download_reply.log.2004-06-29_10:58 apps01/tg/jobs/log/download_reply/download_reply.log.2004-06-29_11:08 apps01/tg/jobs/log/download_reply/download_reply.log.2004-06-29_11:18 apps01/tg/jobs/log/download_reply/download_reply.log.2004-06-29_11:28 apps01/tg/jobs/log/download_reply/download_reply.log.2004-06-29_11:38 apps01/tg/jobs/log/download_reply/download_reply.log.2004-06-29_11:48 apps01/tg/jobs/log/download_reply/download_reply.log.2004-06-29_11:58 apps01/tg/jobs/log/download_reply/download_reply.log.2004-06-29_12:08 apps01/tg/jobs/log/download_reply/download_reply.log.2004-06-29_12:18 apps01/tg/jobs/log/download_reply/download_reply.log.2004-06-29_12:28 apps01/tg/jobs/log/download_reply/download_reply.log.2004-06-29_12:38 apps01/tg/jobs/log/download_reply/download_reply.log.2004-06-29_12:48 apps01/tg/jobs/log/download_reply/download_reply.log.2004-06-29_12:58 apps01/tg/jobs/log/download_reply/download_reply.log.2004-06-29_13:08 apps01/tg/jobs/log/download_reply/download_reply.log.2004-06-29_13:18 apps01/tg/jobs/log/download_reply/download_reply.log.2004-06-29_13:28 apps01/tg/jobs/log/download_reply/download_reply.log.2004-06-29_13:38 rsync: writefd_unbuffered failed to write 4 bytes: phase "unknown": Broken pipe rsync error: error in rsync protocol data stream (code 12) at io.c(836) 3. Do I have enough space? # du -sk /apps50/backup/./home/ape/ /backup01/ape.5.20040702010001/ 2919456 /apps50/backup/./home/ape 2348284 /backup01/ape.5.20040702010001 # df -k . Filesystem 1K-blocks Used Available Use% Mounted on /dev/hda3 37262796 26873832 8496108 76% / So I have enough space. 4. However, if I delete all files from the destination directory: # cd /backup01/ape.5.20040702010001 # ls apps01 # \rm -rf apps01/ And rerun the command. It is fine: ... wrote 2907157423 bytes read 534660 bytes 4568251.51 bytes/sec total size is 2905200920 speedup is 1.00 This happened twice. The process is run on daily basis. I am sure it will happen again. So whatever debugging commands that I need to run, I can. Thanks a lot.
The rsync debug output shows: [pid 5246] write(3, "\200\0\331f\17\v\0\20\230\0050#PLD20040625\200\0\331g\17". .., 4096) = -1 ENOSPC (No space left on device) However, this is not true. The system log shows: Jul 4 07:03:31 ram kernel: EXT3-fs error (device ide0(3,3)): ext3_new_block: block(9465252) >= blocks count(9464293) - block_group = 288, es == c17f5400 Rebooted the server, manually run fsck, the problem is resolved.