Bug 5567 - 3.0.3pre3 dies with "failed to open XXX, continuing: Too many open files"
3.0.3pre3 dies with "failed to open XXX, continuing: Too many open files"
Status: CLOSED FIXED
Product: rsync
Classification: Unclassified
Component: core
3.0.3
Other Linux
: P3 normal
: ---
Assigned To: Wayne Davison
Rsync QA Contact
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2008-06-26 02:29 UTC by Timo Lindfors
Modified: 2008-07-28 20:46 UTC (History)
0 users

See Also:


Attachments
Close fd in the generator for empty files (550 bytes, patch)
2008-06-26 10:14 UTC, Wayne Davison
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Timo Lindfors 2008-06-26 02:29:07 UTC
I started getting a lot of errors like

rsync: failed to open "/tftpboot/moniscreen/usr/src/linux-headers-2.6.24-19-generic/include/config/fs/posix/acl.h", continuing: Too many open files (24)
rsync: failed to open "/tftpboot/moniscreen/usr/src/linux-headers-2.6.24-19-generic/include/config/fujitsu/laptop.h", continuing: Too many open files (24)
rsync: failed to open "/tftpboot/moniscreen/usr/src/linux-headers-2.6.24-19-generic/include/config/fuse/fs.h", continuing: Too many open files (24)
rsync: failed to open "/tftpboot/moniscreen/usr/src/linux-headers-2.6.24-19-generic/include/config/fusion/ctl.h", continuing: Too many open files (24)
rsync: failed to open "/tftpboot/moniscreen/usr/src/linux-headers-2.6.24-19-generic/include/config/fusion/fc.h", continuing: Too many open files (24)
rsync: failed to open "/tftpboot/moniscreen/usr/src/linux-headers-2.6.24-19-generic/include/config/fusion/lan.h", continuing: Too many open files (24)
rsync: failed to open "/tftpboot/moniscreen/usr/src/linux-headers-2.6.24-19-generic/include/config/fusion/logging.h", continuing: Too many open files (24)
rsync: failed to open "/tftpboot/moniscreen/usr/src/linux-headers-2.6.24-19-generic/include/config/fusion/sas.h", continuing: Too many open files (24)

after I upgraded from rsync 3.0.3pre2 to 3.0.0pre3. I invoke rsync on a debian stable system on x86 with the following arguments/environment:

execve("/usr/bin/rsync", ["rsync", "--numeric-ids", "--delete", "--delete-after", "-ax", "root@moniscreen:/", "/tftpboot/moniscreen"], ["SHELL=/bin/bash", "TERM
=screen", "USER=root", "http_proxy=http://proxy.kurp.hut.fi:3128/", "SUDO_USER=lindi", "SUDO_UID=548", "ftp_proxy=http://proxy.kurp.hut.fi:3128/", "PATH=/usr/local/sb
in:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/X11R6/bin"..., "PWD=/root", "EDITOR=/home/lindi/bin/editor", "SHLVL=1", "SUDO_COMMAND=/bin/bash backup2 --stdin",
 "HOME=/home/lindi", "LESSCHARSET=latin1", "LOGNAME=root", "LC_CTYPE=fi_FI", "SUDO_GID=500", "_=/usr/bin/strace"])

and with the following ulimits:

core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) 1004140
max nice                        (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) unlimited
max locked memory       (kbytes, -l) 1004140
max memory size         (kbytes, -m) 1004140
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) unlimited
max rt priority                 (-r) 0
stack size              (kbytes, -s) 4096
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) 1004140
file locks                      (-x) unlimited

I could raise ulimit of open files but shouldn't rsync cope just fine with the default limit?

The diff between pre2 and pre3 is quite small but I was unable to figure out anything obvious. What extra info should I provide? The whole strace output is 96 megabytes and contains sensitive data. I was unable to reproduce the issue with a simple

mkdir a
for i in `seq 1 4096`; do echo > a/$i; done
rsync --numeric-ids --delete --delete-after -ax a/ b
Comment 1 Timo Lindfors 2008-06-26 06:22:31 UTC
Steps to reproduce:
1) cd /tmp
2) wget http://iki.fi/lindi/rsync/bug5567.tar.gz
3) sudo tar xzf bug5567.tar.gz
4) sudo rsync --numeric-ids --delete --delete-after -ax localhost:/tmp/bug5567/source/ /tmp/bug5567/target

Expected results:
4) rsync does not give error messages

Actual results:
4) rsync complains loudly with

...
rsync: failed to open "/tmp/bug5567/target/scripts/mod/elfconfig.h", continuing: Too many open files (24)
rsync: failed to open "/tmp/bug5567/target/scripts/mod/empty.o", continuing: Too many open files (24)
rsync: failed to open "/tmp/bug5567/target/scripts/mod/file2alias.o", continuing: Too many open files (24)
rsync: failed to open "/tmp/bug5567/target/scripts/mod/mk_elfconfig", continuing: Too many open files (24)
rsync: failed to open "/tmp/bug5567/target/scripts/mod/modpost", continuing: Too many open files (24)
rsync: failed to open "/tmp/bug5567/target/scripts/mod/modpost.o", continuing: Too many open files (24)
rsync: failed to open "/tmp/bug5567/target/scripts/mod/sumversion.o", continuing: Too many open files (24)
rsync: opendir "/tmp/bug5567/target/." failed: Too many open files (24)
IO error encountered -- skipping file deletion
rsync error: some files could not be transferred (code 23) at main.c(1506) [generator=3.0.3pre3]

Please let me know if you are unable to reproduce this I am happy to provide more info.
Comment 2 Wayne Davison 2008-06-26 09:55:35 UTC
I can run rsync on the tar you specified with "ulimit -n 16" without problem.  Your tar file didn't contain any xattrs, though -- are there xattrs involved?  Are any patches applied to the default source?  Is selinux involved?  Can you run "lsof -c rsync" while rsync is running to see if it has a lot of open files?
Comment 3 Timo Lindfors 2008-06-26 10:14:35 UTC
xattrs are probably not involved since I too can reproduce the problem by just untarring bug5567.tar.gz (which does not have them) to /tmp (which should be just regular ext3 without any special setup). No special patches have been applied, this was compiled from

8b1eb7e42b36744cf580a2fba6224087  rsync-3.0.3pre3.tar.gz

I also don't think there is any selinux setup on the debian stable system  (at least I have never configured any). Finally, I started

sudo sh -c 'while true; do lsof |grep rsync|wc; done'

before running rsync and it printed

   17     153    1533
   1065    9591  126779
   1065    9587  126745
   1065    9587  126745
   1065    9587  126745
   1065    9587  126745
   1065    9587  126745
   1065    9587  126745
   1065    9587  126745
   1065    9587  126745
   1065    9587  126745
   1065    9587  126745
   1065    9587  126745
    178    1626   20262
      0       0       0

so rsync quite rapidly opened all the files it could and then started to give out errors.

If you still can't reproduce I could prepare a clean qemu image where this can be reproduced, would that help?

Comment 4 Wayne Davison 2008-06-26 10:14:56 UTC
Created attachment 3372 [details]
Close fd in the generator for empty files

I looked at the code after my reply, and found the problem.  If I add --no-whole-file to the command options, I too get the errors, and the attached patch fixes this.
Comment 5 Wayne Davison 2008-06-26 10:15:26 UTC
This is now fixed in the git repository.

Thanks for your report!