Bug 9798 - rsync crash with SIGSEGV when read time out happens
Summary: rsync crash with SIGSEGV when read time out happens
Status: RESOLVED FIXED
Alias: None
Product: rsync
Classification: Unclassified
Component: core (show other bugs)
Version: 3.0.2
Hardware: All All
: P5 critical (vote)
Target Milestone: ---
Assignee: Wayne Davison
QA Contact: Rsync QA Contact
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-04-16 13:48 UTC by Vijay Nag
Modified: 2013-05-27 18:41 UTC (History)
1 user (show)

See Also:


Attachments
core file (288.00 KB, application/octet-stream)
2013-04-16 13:48 UTC, Vijay Nag
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Vijay Nag 2013-04-16 13:48:17 UTC
Created attachment 8774 [details]
core file

rsync crashed with the following backtrace when read timeout happened.

(gdb) bt
#0  ascii_internal_loop (step=0x80a2ee8, step_data=0x80a3d24, inptrp=0xbfffb05c, inend=0xbfffbd48 "\n", outptrp=0xbfffaf84, outend=0x80a5018 "", irreversible=0xbfffaf88) at loop.c:282
#1  0xb7f9cbb2 in __gconv_transform_ascii_internal (step=0x80a2ee8, data=0x80a2fe8, inptrp=0xbfffb05c, inend=0xbfffbd48 "\n", outbufstart=0x0, irreversible=0xbfffb018, do_flush=0, 
    consume_incomplete=0) at skeleton.c:483
#2  0xb7f99ec7 in __gconv (cd=0x80a2fe0, inbuf=0xbfffb05c, inbufend=0xbfffbd48 "\n", outbuf=0xbfffb064, outbufend=0x80a3d24 "", irreversible=0xbfffb018) at gconv.c:63
#3  0xb7f996fc in iconv (cd=0x80a2fe0, inbuf=0xbfffb05c, inbytesleft=0xbfffb060, outbuf=0xbfffb064, outbytesleft=0xbfffb068) at iconv.c:53
#4  0x08050f6d in iconvbufs (ic=0x80a2fe0, in=0xbfffb4a0, out=0xbfffb4b0, flags=0) at rsync.c:175
#5  0x080630e6 in rwrite (code=3086884128, buf=0xbfffbcf0 "rsync error: error in rsync protocol data stream (code 12) at io.c(635) [receiver=3.0.2]\n", len=-1073761120, is_utf8=0) at log.c:339
#6  0x080632fc in rprintf (code=FERROR, format=0x8083c20 "rsync error: %s (code %d) at %s(%d) [%s=%s]\n") at log.c:398
#7  0x08064107 in log_exit (code=12, file=0x80895e0 "io.c", line=134780384) at log.c:829
#8  0x08058b1d in _exit_cleanup (code=12, file=0x80895e0 "io.c", line=635) at cleanup.c:191
#9  0x08068cc1 in whine_about_eof (fd=134888740) at io.c:635
#10 0x08068f21 in read_timeout (fd=5, buf=0xbfffe984 "\036", len=4) at io.c:783
#11 0x08069933 in readfd_unbuffered (fd=5, buf=0xbfffe984 "\036", len=4) at io.c:1041
#12 0x08069e38 in readfd (fd=5, buffer=0xbfffe984 "\036", N=4) at io.c:1187
#13 0x08069ea5 in read_int (f=5) at io.c:1215
#14 0x0806b2cd in setup_protocol (f_out=4, f_in=5) at compat.c:150
#15 0x08060374 in client_run (f_in=5, f_out=4, pid=25865, argc=1, argv=0x80a21a4) at main.c:983
#16 0x0806097d in start_client (argc=1, argv=0x80a21a4) at main.c:1266
#17 0x08061033 in main (argc=2, argv=0x80a21a0) at main.c:1493

The most recent errno before the crash was EILSEQ.

gdb)frame 4
gdb) p *in
$49 = {buf = 0xbfffbcf0 "rsync error: error in rsync protocol data stream (code 12) at io.c(635) [receiver=3.0.2]\n", pos = 16342, len = 4294951042, size = 4294967295}
(gdb) p icnt 
$50 = 4294951042  -- This doesn’t look a good number. It is causing out of buffer array access.
(gdb)
Icnt was initially 89 and probably it wrapped around ?
Comment 1 Paul Slootman 2013-04-22 11:22:30 UTC
Please retry with a current version of rsync. Your 3.0.2 version is 5 years old.
Comment 2 Vijay Nag 2013-04-22 15:26:41 UTC
(In reply to comment #1)
> Please retry with a current version of rsync. Your 3.0.2 version is 5 years
> old.

Crash is observed in the latest rsync. I checked the diff with the latest one and
with 3.0.2

I think rsync is not handling EILSEQ error returned by iconv(3). Here are couple of observations.

1>
 while (icnt) {
    retVal = -1;
    while (iconv(ic, &ibuf, &icnt, &obuf, &ocnt) == (size_t)-1 || retVal == -1) {
      errno = EILSEQ;
      if (errno == EINTR)
        continue;
      if (errno == EINVAL) {
        if (!(flags & ICB_INCLUDE_INCOMPLETE))
          goto finish;
      } else if (errno == EILSEQ) {
        if (!(flags & ICB_INCLUDE_BAD))
          goto finish;
      } else {
        size_t opos = obuf - out->buf;
        if (!(flags & ICB_EXPAND_OUT)) {
          errno = E2BIG;
          goto finish;
        }
        realloc_xbuf(out, out->size + 1024);
        obuf = out->buf + opos;
        ocnt += 1024;
        continue;
      }
      *obuf++ = *ibuf++;
      ocnt--, icnt--;
    }
  }

The condition checking "while(icnt)" is bad since icnt is size_t and there were cases where icnt had wrapped causing it to crash.

2>
   while (inbuf.len) {                      
      iconvbufs(ic, &inbuf, &outbuf, 0);                 
      ierrno = errno;
      if (outbuf.len) {  
        filtered_fwrite(f, convbuf, outbuf.len, 0);
        outbuf.len = 0;
      }              
      if (!ierrno || ierrno == E2BIG)
        continue;                          
      fprintf(f, "\\#%03o", CVAL(inbuf.buf, inbuf.pos++));
      inbuf.len--;        
      save_len = inbuf.len;   
    }                           


Lets assume iconvbufs is returning EILSEQ continuously there are
chances of "inbuf.len"  wrapping again when inbuf.len reaches zero.
so again the "while(inbuf.len)" starts iterating from (size_t)-1 which is
0xffffff. This can lead to a crash again.

Best way to do this check is to save the (size_t)len to an integer and have
the while check "0 < save_len". This will make sure that inbuf.len can never
wrap. Let me know if it is plausible so that i can give you a patch. I have tested it and it works.

+  int save_len = len;                                                     
                                                                             
    INIT_CONST_XBUF(outbuf, convbuf);                                        
    INIT_XBUF(inbuf, (char*)buf, len, -1);                                 

 -while(inbuf.len)                                                                           
+    while (0 < save_len) {                      
      iconvbufs(ic, &inbuf, &outbuf, 0);                 
      ierrno = errno;
      if (outbuf.len) {  
        filtered_fwrite(f, convbuf, outbuf.len, 0);
        outbuf.len = 0;
      }              
      if (!ierrno || ierrno == E2BIG)
        continue;                          
      fprintf(f, "\\#%03o", CVAL(inbuf.buf, inbuf.pos++));
      inbuf.len--;        
      save_len = inbuf.len;   
    }
Comment 3 Wayne Davison 2013-05-26 23:50:21 UTC
(In reply to comment #2)
> I think rsync is not handling EILSEQ error returned by iconv(3).

No, but it did expect an E2BIG in preference to an EILSEQ, which might not be the case.  I've fixed that in the latest git (for 3.1.0).

> Lets assume iconvbufs is returning EILSEQ continuously there are
> chances of "inbuf.len" wrapping again when inbuf.len reaches zero.

I would hope that iconv() would not return EILSEQ if inbuf.len was zero, but I went ahead and made the code avoid the extra iconv() call if the input buffer goes empty.

Thanks for sharing your thoughts -- the code should be a good bit safer now.

Can you test if the latest rsync fixes the crash for you?
Comment 4 Vijay Nag 2013-05-27 01:28:35 UTC
(In reply to comment #3)
> (In reply to comment #2)
> > I think rsync is not handling EILSEQ error returned by iconv(3).
> 
> No, but it did expect an E2BIG in preference to an EILSEQ, which might not be
> the case.  I've fixed that in the latest git (for 3.1.0).
> 
> > Lets assume iconvbufs is returning EILSEQ continuously there are
> > chances of "inbuf.len" wrapping again when inbuf.len reaches zero.
> 
> I would hope that iconv() would not return EILSEQ if inbuf.len was zero, but I
> went ahead and made the code avoid the extra iconv() call if the input buffer
> goes empty.
> 
> Thanks for sharing your thoughts -- the code should be a good bit safer now.
> 
> Can you test if the latest rsync fixes the crash for you

I have already tested EILSEQ case and it works fine. Did you add any code by the way ? Do you need any test reports to mark this bug verified ?
Comment 5 Wayne Davison 2013-05-27 18:35:56 UTC
I asked you to test the fix.  Either grab the git version or the latest nightly tar file.
Comment 6 Vijay Nag 2013-05-27 18:41:28 UTC
(In reply to comment #5)
> I asked you to test the fix.  Either grab the git version or the latest nightly
> tar file.

Tested ok.