Bug 11774 - Use after free in vfs_glusterfs AIO code.
Use after free in vfs_glusterfs AIO code.
Status: ASSIGNED
Product: Samba 4.1 and newer
Classification: Unclassified
Component: VFS Modules
4.4.0rc3
All All
: P5 major
: ---
Assigned To: Ira Cooper
Samba QA Contact
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2016-03-04 23:09 UTC by Ira Cooper
Modified: 2016-03-14 09:02 UTC (History)
4 users (show)

See Also:


Attachments
patch for 4.2, 4.3 and 4.4. (1.94 KB, patch)
2016-03-07 22:57 UTC, Ira Cooper
jra: review+
obnox: review+
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Ira Cooper 2016-03-04 23:09:43 UTC
There is currently a use after free error in the vfs_glusterfs AIO code, that can cause a crash, under high load.

Stack shown, patch will be enclosed, as soon as it is accepted into master.

#0  0x00007fbc695fe0f9 in __libc_waitpid (pid=29491, stat_loc=stat_loc@entry=0x7ffffb0424a0, options=options@entry=0) at ../sysdeps/unix/sysv/linux/waitpid.c:40                                                     │······················
#1  0x00007fbc69583092 in do_system (line=line@entry=0x7fbc6f1d6b70 "/usr/bin/sleep 9999999999") at ../sysdeps/posix/system.c:148                                                                                    │······················
#2  0x00007fbc69583441 in __libc_system (line=line@entry=0x7fbc6f1d6b70 "/usr/bin/sleep 9999999999") at ../sysdeps/posix/system.c:189                                                                                │······················
#3  0x00007fbc6cf3fff5 in system (line=line@entry=0x7fbc6f1d6b70 "/usr/bin/sleep 9999999999") at pt-system.c:28                                                                                                      │······················
#4  0x00007fbc6aec7e81 in smb_panic_s3 (why=<optimized out>) at ../source3/lib/util.c:801                                                                                                                            │······················
#5  0x00007fbc6cd178cf in smb_panic (why=why@entry=0x7fbc6cd248aa "internal error") at ../lib/util/fault.c:166                                                                                                       │······················
#6  0x00007fbc6cd17ae6 in fault_report (sig=<optimized out>) at ../lib/util/fault.c:83                                                                                                                               │······················
#7  sig_fault (sig=<optimized out>) at ../lib/util/fault.c:94                                                                                                                                                        │······················
#8  <signal handler called>                                                                                                                                                                                          │······················
#9  0x00007fbc527e2ee3 in aio_wrapper_destructor (wrap=wrap@entry=0x7fbc6f228270) at ../source3/modules/vfs_glusterfs.c:505                                                                                          │······················
#10 0x00007fbc69b13928 in _talloc_free_internal (location=<optimized out>, ptr=<optimized out>) at ../talloc.c:993                                                                                                   │······················
#11 _talloc_free (ptr=0x7fbc6f228270, location=0x7fbc6990bc91 "../tevent_req.c:247") at ../talloc.c:1594                                                                                                             │······················
#12 0x00007fbc69907370 in tevent_req_received (req=req@entry=0x7fbc6f2280e0) at ../tevent_req.c:247                                                                                                                  │······················
#13 0x00007fbc699073a9 in tevent_req_destructor (req=req@entry=0x7fbc6f2280e0) at ../tevent_req.c:99                                                                                                                 │······················
#14 0x00007fbc69b13928 in _talloc_free_internal (location=<optimized out>, ptr=<optimized out>) at ../talloc.c:993                                                                                                   │······················
#15 _talloc_free (ptr=0x7fbc6f2280e0, location=0x7fbc6ca1ef14 "../source3/smbd/vfs.c:1701") at ../talloc.c:1594                                                                                                      │······················
#16 0x00007fbc6c8d977f in smb_vfs_call_pread_done (subreq=0x7fbc6f2280e0) at ../source3/smbd/vfs.c:1701                                                                                                              │······················
#17 0x00007fbc527e39f7 in aio_tevent_fd_done (event_ctx=<optimized out>, fde=<optimized out>, flags=<optimized out>, data=<optimized out>) at ../source3/modules/vfs_glusterfs.c:589                                 │······················
#18 0x00007fbc6aeddbfc in run_events_poll (ev=0x7fbc6efdbc40, pollrtn=<optimized out>, pfds=0x7fbc6efe7ee0, num_pfds=6) at ../source3/lib/events.c:257                                                               │······················
#19 0x00007fbc6aedde50 in s3_event_loop_once (ev=0x7fbc6efdbc40, location=<optimized out>) at ../source3/lib/events.c:326                                                                                            │······················
#20 0x00007fbc699060fd in _tevent_loop_once (ev=ev@entry=0x7fbc6efdbc40, location=location@entry=0x7fbc6ca28cb0 "../source3/smbd/process.c:3997") at ../tevent.c:533                                                 │······················
#21 0x00007fbc6990629b in tevent_common_loop_wait (ev=0x7fbc6efdbc40, location=0x7fbc6ca28cb0 "../source3/smbd/process.c:3997") at ../tevent.c:637                                                                   │······················
#22 0x00007fbc6c8efd09 in smbd_process (ev_ctx=ev_ctx@entry=0x7fbc6efdbc40, msg_ctx=msg_ctx@entry=0x7fbc6efdbd30, sock_fd=sock_fd@entry=39, interactive=interactive@entry=false) at ../source3/smbd/process.c:3997   │······················
#23 0x00007fbc6d37be44 in smbd_accept_connection (ev=0x7fbc6efdbc40, fde=<optimized out>, flags=<optimized out>, private_data=<optimized out>) at ../source3/smbd/server.c:627
#24 0x00007fbc6aeddbfc in run_events_poll (ev=0x7fbc6efdbc40, pollrtn=<optimized out>, pfds=0x7fbc6efe7ee0, num_pfds=7) at ../source3/lib/events.c:257                                                               │······················
#25 0x00007fbc6aedde50 in s3_event_loop_once (ev=0x7fbc6efdbc40, location=<optimized out>) at ../source3/lib/events.c:326                                                                                            │······················
#26 0x00007fbc699060fd in _tevent_loop_once (ev=ev@entry=0x7fbc6efdbc40, location=location@entry=0x7fbc6d37e847 "../source3/smbd/server.c:985") at ../tevent.c:533                                                   │······················
#27 0x00007fbc6990629b in tevent_common_loop_wait (ev=0x7fbc6efdbc40, location=0x7fbc6d37e847 "../source3/smbd/server.c:985") at ../tevent.c:637                                                                     │······················
#28 0x00007fbc6d378361 in smbd_parent_loop (parent=<optimized out>, ev_ctx=0x7fbc6efdbc40) at ../source3/smbd/server.c:985                                                                                           │······················
#29 main (argc=<optimized out>, argv=<optimized out>) at ../source3/smbd/server.c:1626
messages
Mar 04 09:55:18 GFSNode1 smbd[18154]: [2016/03/04 09:55:18.853690,  0] ../source3/modules/vfs_glusterfs.c:257(vfs_gluster_connect)
Mar 04 09:55:18 GFSNode1 smbd[18154]:   glusterfs: Initialized volume from server localhost
Mar 04 09:55:18 GFSNode1 smbd[18186]: [2016/03/04 09:55:18.854484,  0] ../source3/modules/vfs_glusterfs.c:257(vfs_gluster_connect)
Mar 04 09:55:18 GFSNode1 smbd[18186]:   glusterfs: Initialized volume from server localhost
Mar 04 09:55:18 GFSNode1 smbd[18295]: [2016/03/04 09:55:18.856751,  0] ../source3/modules/vfs_glusterfs.c:257(vfs_gluster_connect)
Mar 04 09:55:18 GFSNode1 smbd[18295]:   glusterfs: Initialized volume from server localhost
Mar 04 09:55:18 GFSNode1 smbd[18204]: [2016/03/04 09:55:18.887609,  0] ../source3/modules/vfs_glusterfs.c:257(vfs_gluster_connect)
Mar 04 09:55:18 GFSNode1 smbd[18204]:   glusterfs: Initialized volume from server localhost
Mar 04 09:55:18 GFSNode1 smbd[18197]: [2016/03/04 09:55:18.900225,  0] ../source3/modules/vfs_glusterfs.c:257(vfs_gluster_connect)
Mar 04 09:55:18 GFSNode1 smbd[18197]:   glusterfs: Initialized volume from server localhost
Mar 04 09:55:18 GFSNode1 smbd[20574]: [2016/03/04 09:55:18.957515,  0] ../source3/modules/vfs_glusterfs.c:257(vfs_gluster_connect)
Mar 04 09:55:18 GFSNode1 smbd[20574]:   glusterfs: Initialized volume from server localhost
Mar 04 09:55:19 GFSNode1 smbd[18337]: [2016/03/04 09:55:19.617540,  0] ../lib/util/fault.c:78(fault_report)
Mar 04 09:55:19 GFSNode1 smbd[18337]:   ===============================================================
Mar 04 09:55:19 GFSNode1 smbd[18337]: [2016/03/04 09:55:19.617608,  0] ../lib/util/fault.c:79(fault_report)
Mar 04 09:55:19 GFSNode1 smbd[18337]:   INTERNAL ERROR: Signal 11 in pid 18337 (4.2.4)
Mar 04 09:55:19 GFSNode1 smbd[18337]:   Please read the Trouble-Shooting section of the Samba HOWTO
Mar 04 09:55:19 GFSNode1 smbd[18337]: [2016/03/04 09:55:19.617633,  0] ../lib/util/fault.c:81(fault_report)
Mar 04 09:55:19 GFSNode1 smbd[18337]:   ===============================================================
Mar 04 09:55:19 GFSNode1 smbd[18337]: [2016/03/04 09:55:19.617666,  0] ../source3/lib/util.c:788(smb_panic_s3)
Mar 04 09:55:19 GFSNode1 smbd[18337]:   PANIC (pid 18337): internal error
Mar 04 09:55:19 GFSNode1 smbd[18337]: [2016/03/04 09:55:19.628078,  0] ../source3/lib/util.c:899(log_stack_trace)
Mar 04 09:55:19 GFSNode1 smbd[18337]:   BACKTRACE: 25 stack frames:
Mar 04 09:55:19 GFSNode1 smbd[18337]:    #0 /lib64/libsmbconf.so.0(log_stack_trace+0x1a) [0x7fbc6aec7d5a]
Mar 04 09:55:19 GFSNode1 smbd[18337]:    #1 /lib64/libsmbconf.so.0(smb_panic_s3+0x20) [0x7fbc6aec7e30]
Mar 04 09:55:19 GFSNode1 smbd[18337]:    #2 /lib64/libsamba-util.so.0(smb_panic+0x2f) [0x7fbc6cd178cf]
Mar 04 09:55:19 GFSNode1 smbd[18337]:    #3 /lib64/libsamba-util.so.0(+0x1aae6) [0x7fbc6cd17ae6]
Mar 04 09:55:19 GFSNode1 smbd[18337]:    #4 /lib64/libpthread.so.0(+0xf100) [0x7fbc6cf40100]
Mar 04 09:55:19 GFSNode1 smbd[18337]:    #5 /usr/lib64/samba/vfs/glusterfs.so(+0x2ee3) [0x7fbc527e2ee3]
Mar 04 09:55:19 GFSNode1 smbd[18337]:    #6 /lib64/libtalloc.so.2(_talloc_free+0x3f8) [0x7fbc69b13928]
Mar 04 09:55:19 GFSNode1 smbd[18337]:    #7 /lib64/libtevent.so.0(tevent_req_received+0x80) [0x7fbc69907370]
Mar 04 09:55:19 GFSNode1 smbd[18337]:    #8 /lib64/libtevent.so.0(+0x53a9) [0x7fbc699073a9]
Mar 04 09:55:19 GFSNode1 smbd[18337]:    #9 /lib64/libtalloc.so.2(_talloc_free+0x3f8) [0x7fbc69b13928]
Mar 04 09:55:19 GFSNode1 smbd[18337]:    #10 /usr/lib64/samba/libsmbd-base-samba4.so(+0x11a77f) [0x7fbc6c8d977f]
Mar 04 09:55:19 GFSNode1 smbd[18337]:    #11 /usr/lib64/samba/vfs/glusterfs.so(+0x39f7) [0x7fbc527e39f7]
Mar 04 09:55:19 GFSNode1 smbd[18337]:    #12 /lib64/libsmbconf.so.0(run_events_poll+0x16c) [0x7fbc6aeddbfc]
Mar 04 09:55:19 GFSNode1 smbd[18337]:    #13 /lib64/libsmbconf.so.0(+0x36e50) [0x7fbc6aedde50]
Mar 04 09:55:19 GFSNode1 smbd[18337]:    #14 /lib64/libtevent.so.0(_tevent_loop_once+0x8d) [0x7fbc699060fd]
Mar 04 09:55:19 GFSNode1 smbd[18337]:    #15 /lib64/libtevent.so.0(tevent_common_loop_wait+0x1b) [0x7fbc6990629b]
Mar 04 09:55:19 GFSNode1 smbd[18337]:    #16 /usr/lib64/samba/libsmbd-base-samba4.so(smbd_process+0x6d9) [0x7fbc6c8efd09]
Mar 04 09:55:19 GFSNode1 smbd[18337]:    #17 /usr/sbin/smbd(+0xae44) [0x7fbc6d37be44]
Mar 04 09:55:19 GFSNode1 smbd[18337]:    #18 /lib64/libsmbconf.so.0(run_events_poll+0x16c) [0x7fbc6aeddbfc]
Mar 04 09:55:19 GFSNode1 smbd[18337]:    #19 /lib64/libsmbconf.so.0(+0x36e50) [0x7fbc6aedde50]
Mar 04 09:55:19 GFSNode1 smbd[18337]:    #20 /lib64/libtevent.so.0(_tevent_loop_once+0x8d) [0x7fbc699060fd]
Mar 04 09:55:19 GFSNode1 smbd[18337]:    #21 /lib64/libtevent.so.0(tevent_common_loop_wait+0x1b) [0x7fbc6990629b]
Mar 04 09:55:19 GFSNode1 smbd[18337]:    #22 /usr/sbin/smbd(main+0x1501) [0x7fbc6d378361]
Mar 04 09:55:19 GFSNode1 smbd[18337]:    #23 /lib64/libc.so.6(__libc_start_main+0xf5) [0x7fbc69562b15]
Mar 04 09:55:19 GFSNode1 smbd[18337]:    #24 /usr/sbin/smbd(+0x7765) [0x7fbc6d378765]
Mar 04 09:55:19 GFSNode1 smbd[18337]: [2016/03/04 09:55:19.632898,  0] ../source3/lib/util.c:800(smb_panic_s3)
Mar 04 09:55:19 GFSNode1 smbd[18337]:   smb_panic(): calling panic action [/usr/bin/sleep 9999999999]
Comment 1 Ira Cooper 2016-03-07 22:57:42 UTC
Created attachment 11902 [details]
patch for 4.2, 4.3 and 4.4.

Patch for 4.2, 4.3, and 4.4.
Comment 2 Michael Adam 2016-03-08 12:05:08 UTC
Comment on attachment 11902 [details]
patch for 4.2, 4.3 and 4.4.

We usually want the (cherry-picked from ...) line in the patch

i.e. do the cherry-pick with 'git cherry-pick -x ...'.

LGTM apart from that cosmetic complaint.

Cheers - Michael
Comment 3 Ira Cooper 2016-03-08 13:43:11 UTC
(In reply to Michael Adam from comment #2)

It is not a cherry pick.
Comment 4 Michael Adam 2016-03-08 13:47:41 UTC
(In reply to Ira Cooper from comment #3)
> (In reply to Michael Adam from comment #2)
> 
> It is not a cherry pick.

Oops. in that case --> RB+ :-)
Comment 5 Ira Cooper 2016-03-08 19:17:33 UTC
Karolin,

Could you please merge the above patch into 4.2, 4.3 and 4.4?

Thanks,
Comment 6 Karolin Seeger 2016-03-14 09:02:00 UTC
(In reply to Ira Cooper from comment #5)
Pushed to autobuild-v4-4-test.
Patch does not apply on current v4-3-test and v4-2-test.
Re-assigning to Ira.