Created attachment 11170 [details] Core file generated at crash Dear Team, We are facing a crash in rsync on one of our production servers. We have updated rsync to the latest version 3.1.1 but still crash is observed. Below call stack is generated by the crash: (gdb) #0 0x0000003ad7e25433 in __gconv_transform_ascii_internal () from /lib64/libc.so.6 #1 0x0000003ad7e20264 in __gconv () from /lib64/libc.so.6 #2 0x0000003ad7e1f83c in iconv () from /lib64/libc.so.6 #3 0x000000000040d252 in iconvbufs () #4 0x0000000000423bce in rwrite () #5 0x0000000000423505 in rprintf () #6 0x000000000042b911 in whine_about_eof () #7 0x000000000042f5e2 in read_buf () #8 0x000000000042fde1 in read_int () #9 0x000000000043055d in setup_protocol () #10 0x000000000041ffb8 in client_run () #11 0x00000000004209c6 in main () Please let us know if any additional information is needed for debugging. Regards, Balveer Singh
You need to compile rsync with debug symbols (or load an rpm/deb/etc that provides debug symbols) to get useful info in your core's backtrace. You should be figuring out why the call to iconvbufs is crashing by checking its args (change level to rwrite() and look at the vars). Rsync is exiting due to an EOF in the protocol stream (the whine_about_eof() call), but the message that it outputs shouldn't cause the iconv code any issues.
I seem to see the same crash. Here is some additional details with symbols. It seems to happen during network timeouts. #0 0x00007fcc2e167b05 in utf8_internal_loop (step=0x560710d73f90, step_data=0x560710d740c0, irreversible=0x7ffccf327d28, outend=0x560710d7dad0 "", outptrp=<synthetic pointer>, inend=0x7ffccf3287f8 "\n", inptrp=0x7ffccf327e80) at ../iconv/loop.c:325 #1 __gconv_transform_utf8_internal (step=0x560710d73f90, data=data@entry=0x560710d740c0, inptrp=inptrp@entry=0x7ffccf327e80, inend=inend@entry=0x7ffccf3287f8 "\n", outbufstart=outbufstart@entry=0x0, irreversible=irreversible@entry=0x7ffccf327df8, do_flush=do_flush@entry=0, consume_incomplete=consume_incomplete@entry=0) at ../iconv/skeleton.c:613 #2 0x00007fcc2e162722 in __gconv (cd=0x560710d740b0, inbuf=inbuf@entry=0x7ffccf327e80, inbufend=0x7ffccf3287f8 "\n", outbuf=outbuf@entry=0x7ffccf327e98, outbufend=<optimized out>, irreversible=irreversible@entry=0x7ffccf327df8) at gconv.c:79 #3 0x00007fcc2e161fa1 in iconv (cd=cd@entry=0x560710d740b0, inbuf=inbuf@entry=0x7ffccf327e80, inbytesleft=inbytesleft@entry=0x7ffccf327e88, outbuf=outbuf@entry=0x7ffccf327e98, outbytesleft=outbytesleft@entry=0x7ffccf327e90) at iconv.c:52 #4 0x000056070fa1954f in iconvbufs (ic=ic@entry=0x560710d740b0, in=in@entry=0x7ffccf327f20, out=out@entry=0x7ffccf327f00, flags=0) at rsync.c:208 #5 0x000056070fa30191 in rwrite (code=<optimized out>, code@entry=FERROR, buf=<optimized out>, buf@entry=0x7ffccf3287b0 "rsync: connection unexpectedly closed (0 bytes received so far) [sender]\n", len=<optimized out>, is_utf8=<optimized out>, is_utf8@entry=0) at log.c:374 #6 0x000056070fa307e3 in rprintf (code=code@entry=FERROR, format=format@entry=0x56070fa677b0 "rsync: connection unexpectedly closed (%s bytes received so far) [%s]\n") at log.c:433 #7 0x000056070fa39064 in whine_about_eof (allow_kluge=allow_kluge@entry=0) at io.c:231 #8 0x000056070fa3c992 in read_buf (f=f@entry=5, buf=buf@entry=0x7ffccf329cd0 "\037", len=len@entry=4) at io.c:1834 #9 0x000056070fa3caa1 in read_int (f=f@entry=5) at io.c:1729 #10 0x000056070fa3e48c in setup_protocol (f_out=4, f_in=5) at compat.c:158 #11 0x000056070fa2d4b0 in client_run (f_in=5, f_out=4, pid=9022, argc=1, argv=0x560710d74760) at main.c:1110 #12 0x000056070fa109a1 in start_client (argv=0x560710d74760, argc=1) at main.c:1420 #13 main (argc=2, argv=0x560710d74760) at main.c:1648 (gdb) f 4 #4 0x000056070fa1954f in iconvbufs (ic=ic@entry=0x560710d740b0, in=in@entry=0x7ffccf327f20, out=out@entry=0x7ffccf327f00, flags=0) at rsync.c:208 208 in rsync.c (gdb) p flags $15 = 0 (gdb) p xbuf Attempt to use a type name as an expression (gdb) p in $16 = (xbuf *) 0x7ffccf327f20 (gdb) p *in $17 = {buf = 0x7ffccf3287b0 "rsync: connection unexpectedly closed (0 bytes received so far) [sender]\n", pos = 18889, len = 18446744073709532799, size = 18446744073709551615} (gdb) p *out $18 = {buf = 0x7ffccf327f40 "\"\023", pos = 0, len = 0, size = 1024}
Hi Wayne, I don't know if James's SEGV is the same root cause as Balveer's that was two years earlier, but I'm seeing SEGV like James's. Both times, my network connection was down when rsync was run by cron; sounds similar to James's `network timeouts'. This is on Arch Linux with rsync 3.1.2-8 that depends on acl 2.2.52-4, attr 2.4.47-3, glibc 2.26-8, and popt 1.16-9.
Hi, we have encountered similar segfault a couple of years ago. As you can see here in comment #2, the p *in shows that the input buffer's size and len are unreasonably huge - len = 18446744073709532799, size = 18446744073709551615, which leads to an assumption that there was a buffer overflow here. We were never able to create a reliable reproducer for this but after some analysis, we think that the problem is in the rwrite function: #ifdef ICONV_CONST if (ic != (iconv_t)-1) { xbuf outbuf, inbuf; char convbuf[1024]; int ierrno; INIT_CONST_XBUF(outbuf, convbuf); INIT_XBUF(inbuf, (char*)buf, len, (size_t)-1); while (inbuf.len) { iconvbufs(ic, &inbuf, &outbuf, inbuf.pos ? 0 : ICB_INIT); ierrno = errno; if (outbuf.len) { filtered_fwrite(f, convbuf, outbuf.len, 0); outbuf.len = 0; } if (!ierrno || ierrno == E2BIG) continue; fprintf(f, "\\#%03o", CVAL(inbuf.buf, inbuf.pos++)); inbuf.len--; <============== no check for the buffer length } } else #endif The problem is(probably) that the input buffer length is changed in the iconvbufs function. And the rwrite function is dependent on the iconvbufs return values. I know that there have been changes in the iconvbufs code over the years and without a reproducer it is hard to test this but we came up with a patch to make the rsync code in rwite a little bit more robust. Adding the patch.
Created attachment 14872 [details] Patch for the rwrite function
This has just freshly arrived yesterday. Same bug as before and as described here. https://bugzilla.redhat.com/show_bug.cgi?id=1683737
Thanks for the patch, Michal! That looks quite reasonable to me, so I have committed it to git. Reopen this bug if someone can still get the issue to trigger with the patched source.
Great, thanks Wayne.