Bug 11338 - Rsync Crash - Segmentation fault
Summary: Rsync Crash - Segmentation fault
Status: RESOLVED FIXED
Alias: None
Product: rsync
Classification: Unclassified
Component: core (show other bugs)
Version: 3.1.1
Hardware: x64 Linux
: P5 major (vote)
Target Milestone: ---
Assignee: Wayne Davison
QA Contact: Rsync QA Contact
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-06-17 12:35 UTC by Balveer Singh
Modified: 2019-03-18 10:55 UTC (History)
2 users (show)

See Also:


Attachments
Core file generated at crash (28.76 KB, application/x-rar)
2015-06-17 12:35 UTC, Balveer Singh
no flags Details
Patch for the rwrite function (652 bytes, patch)
2019-02-27 10:00 UTC, Michal Ruprich
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Balveer Singh 2015-06-17 12:35:00 UTC
Created attachment 11170 [details]
Core file generated at crash

Dear Team,

We are facing a crash in rsync on one of our production servers. We have updated rsync to the latest version 3.1.1 but still crash is observed. Below call stack is generated by the crash:
(gdb)
#0  0x0000003ad7e25433 in __gconv_transform_ascii_internal () from /lib64/libc.so.6
#1  0x0000003ad7e20264 in __gconv () from /lib64/libc.so.6
#2  0x0000003ad7e1f83c in iconv () from /lib64/libc.so.6
#3  0x000000000040d252 in iconvbufs ()
#4  0x0000000000423bce in rwrite ()
#5  0x0000000000423505 in rprintf ()
#6  0x000000000042b911 in whine_about_eof ()
#7  0x000000000042f5e2 in read_buf ()
#8  0x000000000042fde1 in read_int ()
#9  0x000000000043055d in setup_protocol ()
#10 0x000000000041ffb8 in client_run ()
#11 0x00000000004209c6 in main ()

Please let us know if any additional information is needed for debugging.

Regards,
Balveer Singh
Comment 1 Wayne Davison 2015-06-24 16:47:12 UTC
You need to compile rsync with debug symbols (or load an rpm/deb/etc that provides debug symbols) to get useful info in your core's backtrace.  You should be figuring out why the call to iconvbufs is crashing by checking its args (change level to rwrite() and look at the vars). Rsync is exiting due to an EOF in the protocol stream (the whine_about_eof() call), but the message that it outputs shouldn't cause the iconv code any issues.
Comment 2 James Stevenson 2017-10-10 13:57:06 UTC
I seem to see the same crash. Here is some additional details with symbols. It seems to happen during network timeouts.

#0  0x00007fcc2e167b05 in utf8_internal_loop (step=0x560710d73f90, step_data=0x560710d740c0, irreversible=0x7ffccf327d28, outend=0x560710d7dad0 "", outptrp=<synthetic pointer>, 
    inend=0x7ffccf3287f8 "\n", inptrp=0x7ffccf327e80) at ../iconv/loop.c:325
#1  __gconv_transform_utf8_internal (step=0x560710d73f90, data=data@entry=0x560710d740c0, inptrp=inptrp@entry=0x7ffccf327e80, inend=inend@entry=0x7ffccf3287f8 "\n", 
    outbufstart=outbufstart@entry=0x0, irreversible=irreversible@entry=0x7ffccf327df8, do_flush=do_flush@entry=0, consume_incomplete=consume_incomplete@entry=0)
    at ../iconv/skeleton.c:613
#2  0x00007fcc2e162722 in __gconv (cd=0x560710d740b0, inbuf=inbuf@entry=0x7ffccf327e80, inbufend=0x7ffccf3287f8 "\n", outbuf=outbuf@entry=0x7ffccf327e98, outbufend=<optimized out>, 
    irreversible=irreversible@entry=0x7ffccf327df8) at gconv.c:79
#3  0x00007fcc2e161fa1 in iconv (cd=cd@entry=0x560710d740b0, inbuf=inbuf@entry=0x7ffccf327e80, inbytesleft=inbytesleft@entry=0x7ffccf327e88, outbuf=outbuf@entry=0x7ffccf327e98, 
    outbytesleft=outbytesleft@entry=0x7ffccf327e90) at iconv.c:52
#4  0x000056070fa1954f in iconvbufs (ic=ic@entry=0x560710d740b0, in=in@entry=0x7ffccf327f20, out=out@entry=0x7ffccf327f00, flags=0) at rsync.c:208
#5  0x000056070fa30191 in rwrite (code=<optimized out>, code@entry=FERROR, buf=<optimized out>, 
    buf@entry=0x7ffccf3287b0 "rsync: connection unexpectedly closed (0 bytes received so far) [sender]\n", len=<optimized out>, is_utf8=<optimized out>, is_utf8@entry=0) at log.c:374
#6  0x000056070fa307e3 in rprintf (code=code@entry=FERROR, format=format@entry=0x56070fa677b0 "rsync: connection unexpectedly closed (%s bytes received so far) [%s]\n") at log.c:433
#7  0x000056070fa39064 in whine_about_eof (allow_kluge=allow_kluge@entry=0) at io.c:231
#8  0x000056070fa3c992 in read_buf (f=f@entry=5, buf=buf@entry=0x7ffccf329cd0 "\037", len=len@entry=4) at io.c:1834
#9  0x000056070fa3caa1 in read_int (f=f@entry=5) at io.c:1729
#10 0x000056070fa3e48c in setup_protocol (f_out=4, f_in=5) at compat.c:158
#11 0x000056070fa2d4b0 in client_run (f_in=5, f_out=4, pid=9022, argc=1, argv=0x560710d74760) at main.c:1110
#12 0x000056070fa109a1 in start_client (argv=0x560710d74760, argc=1) at main.c:1420
#13 main (argc=2, argv=0x560710d74760) at main.c:1648
(gdb) f 4
#4  0x000056070fa1954f in iconvbufs (ic=ic@entry=0x560710d740b0, in=in@entry=0x7ffccf327f20, out=out@entry=0x7ffccf327f00, flags=0) at rsync.c:208
208	in rsync.c
(gdb) p flags
$15 = 0
(gdb) p xbuf
Attempt to use a type name as an expression
(gdb) p in
$16 = (xbuf *) 0x7ffccf327f20
(gdb) p *in
$17 = {buf = 0x7ffccf3287b0 "rsync: connection unexpectedly closed (0 bytes received so far) [sender]\n", pos = 18889, len = 18446744073709532799, size = 18446744073709551615}
(gdb) p *out
$18 = {buf = 0x7ffccf327f40 "\"\023", pos = 0, len = 0, size = 1024}
Comment 3 Ralph Corderoy 2018-01-04 11:04:54 UTC
Hi Wayne,

I don't know if James's SEGV is the same root cause as Balveer's that
was two years earlier, but I'm seeing SEGV like James's.  Both times, my
network connection was down when rsync was run by cron;  sounds similar
to James's `network timeouts'.

This is on Arch Linux with rsync 3.1.2-8 that depends on acl 2.2.52-4,
attr 2.4.47-3, glibc 2.26-8, and popt 1.16-9.
Comment 4 Michal Ruprich 2019-02-27 09:59:28 UTC
Hi,

we have encountered similar segfault a couple of years ago. As you can see here in comment #2, the p *in shows that the input buffer's size and len are unreasonably huge - len = 18446744073709532799, size = 18446744073709551615, which leads to an assumption that there was a buffer overflow here. We were never able to create a reliable reproducer for this but after some analysis, we think that the problem is in the rwrite function:

#ifdef ICONV_CONST
    if (ic != (iconv_t)-1) {
        xbuf outbuf, inbuf;
        char convbuf[1024];
        int ierrno;

        INIT_CONST_XBUF(outbuf, convbuf);
        INIT_XBUF(inbuf, (char*)buf, len, (size_t)-1);

        while (inbuf.len) {
            iconvbufs(ic, &inbuf, &outbuf, inbuf.pos ? 0 : ICB_INIT);
            ierrno = errno;
            if (outbuf.len) {
                filtered_fwrite(f, convbuf, outbuf.len, 0); 
                outbuf.len = 0;
            }
            if (!ierrno || ierrno == E2BIG)
                continue;
            fprintf(f, "\\#%03o", CVAL(inbuf.buf, inbuf.pos++));
            inbuf.len--;     <============== no check for the buffer length
        }
    } else
#endif

The problem is(probably) that the input buffer length is changed in the iconvbufs function. And the rwrite function is dependent on the iconvbufs return values. I know that there have been changes in the iconvbufs code over the years and without a reproducer it is hard to test this but we came up with a patch to make the rsync code in rwite a little bit more robust. Adding the patch.
Comment 5 Michal Ruprich 2019-02-27 10:00:05 UTC
Created attachment 14872 [details]
Patch for the rwrite function
Comment 6 Michal Ruprich 2019-02-28 11:55:27 UTC
This has just freshly arrived yesterday. Same bug as before and as described here. 

https://bugzilla.redhat.com/show_bug.cgi?id=1683737
Comment 7 Wayne Davison 2019-03-16 18:54:46 UTC
Thanks for the patch, Michal!  That looks quite reasonable to me, so I have committed it to git.  Reopen this bug if someone can still get the issue to trigger with the patched source.
Comment 8 Michal Ruprich 2019-03-18 10:55:36 UTC
Great, thanks Wayne.