There is currently a use after free error in the vfs_glusterfs AIO code, that can cause a crash, under high load. Stack shown, patch will be enclosed, as soon as it is accepted into master. #0 0x00007fbc695fe0f9 in __libc_waitpid (pid=29491, stat_loc=stat_loc@entry=0x7ffffb0424a0, options=options@entry=0) at ../sysdeps/unix/sysv/linux/waitpid.c:40 │······················ #1 0x00007fbc69583092 in do_system (line=line@entry=0x7fbc6f1d6b70 "/usr/bin/sleep 9999999999") at ../sysdeps/posix/system.c:148 │······················ #2 0x00007fbc69583441 in __libc_system (line=line@entry=0x7fbc6f1d6b70 "/usr/bin/sleep 9999999999") at ../sysdeps/posix/system.c:189 │······················ #3 0x00007fbc6cf3fff5 in system (line=line@entry=0x7fbc6f1d6b70 "/usr/bin/sleep 9999999999") at pt-system.c:28 │······················ #4 0x00007fbc6aec7e81 in smb_panic_s3 (why=<optimized out>) at ../source3/lib/util.c:801 │······················ #5 0x00007fbc6cd178cf in smb_panic (why=why@entry=0x7fbc6cd248aa "internal error") at ../lib/util/fault.c:166 │······················ #6 0x00007fbc6cd17ae6 in fault_report (sig=<optimized out>) at ../lib/util/fault.c:83 │······················ #7 sig_fault (sig=<optimized out>) at ../lib/util/fault.c:94 │······················ #8 <signal handler called> │······················ #9 0x00007fbc527e2ee3 in aio_wrapper_destructor (wrap=wrap@entry=0x7fbc6f228270) at ../source3/modules/vfs_glusterfs.c:505 │······················ #10 0x00007fbc69b13928 in _talloc_free_internal (location=<optimized out>, ptr=<optimized out>) at ../talloc.c:993 │······················ #11 _talloc_free (ptr=0x7fbc6f228270, location=0x7fbc6990bc91 "../tevent_req.c:247") at ../talloc.c:1594 │······················ #12 0x00007fbc69907370 in tevent_req_received (req=req@entry=0x7fbc6f2280e0) at ../tevent_req.c:247 │······················ #13 0x00007fbc699073a9 in tevent_req_destructor (req=req@entry=0x7fbc6f2280e0) at ../tevent_req.c:99 │······················ #14 0x00007fbc69b13928 in _talloc_free_internal (location=<optimized out>, ptr=<optimized out>) at ../talloc.c:993 │······················ #15 _talloc_free (ptr=0x7fbc6f2280e0, location=0x7fbc6ca1ef14 "../source3/smbd/vfs.c:1701") at ../talloc.c:1594 │······················ #16 0x00007fbc6c8d977f in smb_vfs_call_pread_done (subreq=0x7fbc6f2280e0) at ../source3/smbd/vfs.c:1701 │······················ #17 0x00007fbc527e39f7 in aio_tevent_fd_done (event_ctx=<optimized out>, fde=<optimized out>, flags=<optimized out>, data=<optimized out>) at ../source3/modules/vfs_glusterfs.c:589 │······················ #18 0x00007fbc6aeddbfc in run_events_poll (ev=0x7fbc6efdbc40, pollrtn=<optimized out>, pfds=0x7fbc6efe7ee0, num_pfds=6) at ../source3/lib/events.c:257 │······················ #19 0x00007fbc6aedde50 in s3_event_loop_once (ev=0x7fbc6efdbc40, location=<optimized out>) at ../source3/lib/events.c:326 │······················ #20 0x00007fbc699060fd in _tevent_loop_once (ev=ev@entry=0x7fbc6efdbc40, location=location@entry=0x7fbc6ca28cb0 "../source3/smbd/process.c:3997") at ../tevent.c:533 │······················ #21 0x00007fbc6990629b in tevent_common_loop_wait (ev=0x7fbc6efdbc40, location=0x7fbc6ca28cb0 "../source3/smbd/process.c:3997") at ../tevent.c:637 │······················ #22 0x00007fbc6c8efd09 in smbd_process (ev_ctx=ev_ctx@entry=0x7fbc6efdbc40, msg_ctx=msg_ctx@entry=0x7fbc6efdbd30, sock_fd=sock_fd@entry=39, interactive=interactive@entry=false) at ../source3/smbd/process.c:3997 │······················ #23 0x00007fbc6d37be44 in smbd_accept_connection (ev=0x7fbc6efdbc40, fde=<optimized out>, flags=<optimized out>, private_data=<optimized out>) at ../source3/smbd/server.c:627 #24 0x00007fbc6aeddbfc in run_events_poll (ev=0x7fbc6efdbc40, pollrtn=<optimized out>, pfds=0x7fbc6efe7ee0, num_pfds=7) at ../source3/lib/events.c:257 │······················ #25 0x00007fbc6aedde50 in s3_event_loop_once (ev=0x7fbc6efdbc40, location=<optimized out>) at ../source3/lib/events.c:326 │······················ #26 0x00007fbc699060fd in _tevent_loop_once (ev=ev@entry=0x7fbc6efdbc40, location=location@entry=0x7fbc6d37e847 "../source3/smbd/server.c:985") at ../tevent.c:533 │······················ #27 0x00007fbc6990629b in tevent_common_loop_wait (ev=0x7fbc6efdbc40, location=0x7fbc6d37e847 "../source3/smbd/server.c:985") at ../tevent.c:637 │······················ #28 0x00007fbc6d378361 in smbd_parent_loop (parent=<optimized out>, ev_ctx=0x7fbc6efdbc40) at ../source3/smbd/server.c:985 │······················ #29 main (argc=<optimized out>, argv=<optimized out>) at ../source3/smbd/server.c:1626 messages Mar 04 09:55:18 GFSNode1 smbd[18154]: [2016/03/04 09:55:18.853690, 0] ../source3/modules/vfs_glusterfs.c:257(vfs_gluster_connect) Mar 04 09:55:18 GFSNode1 smbd[18154]: glusterfs: Initialized volume from server localhost Mar 04 09:55:18 GFSNode1 smbd[18186]: [2016/03/04 09:55:18.854484, 0] ../source3/modules/vfs_glusterfs.c:257(vfs_gluster_connect) Mar 04 09:55:18 GFSNode1 smbd[18186]: glusterfs: Initialized volume from server localhost Mar 04 09:55:18 GFSNode1 smbd[18295]: [2016/03/04 09:55:18.856751, 0] ../source3/modules/vfs_glusterfs.c:257(vfs_gluster_connect) Mar 04 09:55:18 GFSNode1 smbd[18295]: glusterfs: Initialized volume from server localhost Mar 04 09:55:18 GFSNode1 smbd[18204]: [2016/03/04 09:55:18.887609, 0] ../source3/modules/vfs_glusterfs.c:257(vfs_gluster_connect) Mar 04 09:55:18 GFSNode1 smbd[18204]: glusterfs: Initialized volume from server localhost Mar 04 09:55:18 GFSNode1 smbd[18197]: [2016/03/04 09:55:18.900225, 0] ../source3/modules/vfs_glusterfs.c:257(vfs_gluster_connect) Mar 04 09:55:18 GFSNode1 smbd[18197]: glusterfs: Initialized volume from server localhost Mar 04 09:55:18 GFSNode1 smbd[20574]: [2016/03/04 09:55:18.957515, 0] ../source3/modules/vfs_glusterfs.c:257(vfs_gluster_connect) Mar 04 09:55:18 GFSNode1 smbd[20574]: glusterfs: Initialized volume from server localhost Mar 04 09:55:19 GFSNode1 smbd[18337]: [2016/03/04 09:55:19.617540, 0] ../lib/util/fault.c:78(fault_report) Mar 04 09:55:19 GFSNode1 smbd[18337]: =============================================================== Mar 04 09:55:19 GFSNode1 smbd[18337]: [2016/03/04 09:55:19.617608, 0] ../lib/util/fault.c:79(fault_report) Mar 04 09:55:19 GFSNode1 smbd[18337]: INTERNAL ERROR: Signal 11 in pid 18337 (4.2.4) Mar 04 09:55:19 GFSNode1 smbd[18337]: Please read the Trouble-Shooting section of the Samba HOWTO Mar 04 09:55:19 GFSNode1 smbd[18337]: [2016/03/04 09:55:19.617633, 0] ../lib/util/fault.c:81(fault_report) Mar 04 09:55:19 GFSNode1 smbd[18337]: =============================================================== Mar 04 09:55:19 GFSNode1 smbd[18337]: [2016/03/04 09:55:19.617666, 0] ../source3/lib/util.c:788(smb_panic_s3) Mar 04 09:55:19 GFSNode1 smbd[18337]: PANIC (pid 18337): internal error Mar 04 09:55:19 GFSNode1 smbd[18337]: [2016/03/04 09:55:19.628078, 0] ../source3/lib/util.c:899(log_stack_trace) Mar 04 09:55:19 GFSNode1 smbd[18337]: BACKTRACE: 25 stack frames: Mar 04 09:55:19 GFSNode1 smbd[18337]: #0 /lib64/libsmbconf.so.0(log_stack_trace+0x1a) [0x7fbc6aec7d5a] Mar 04 09:55:19 GFSNode1 smbd[18337]: #1 /lib64/libsmbconf.so.0(smb_panic_s3+0x20) [0x7fbc6aec7e30] Mar 04 09:55:19 GFSNode1 smbd[18337]: #2 /lib64/libsamba-util.so.0(smb_panic+0x2f) [0x7fbc6cd178cf] Mar 04 09:55:19 GFSNode1 smbd[18337]: #3 /lib64/libsamba-util.so.0(+0x1aae6) [0x7fbc6cd17ae6] Mar 04 09:55:19 GFSNode1 smbd[18337]: #4 /lib64/libpthread.so.0(+0xf100) [0x7fbc6cf40100] Mar 04 09:55:19 GFSNode1 smbd[18337]: #5 /usr/lib64/samba/vfs/glusterfs.so(+0x2ee3) [0x7fbc527e2ee3] Mar 04 09:55:19 GFSNode1 smbd[18337]: #6 /lib64/libtalloc.so.2(_talloc_free+0x3f8) [0x7fbc69b13928] Mar 04 09:55:19 GFSNode1 smbd[18337]: #7 /lib64/libtevent.so.0(tevent_req_received+0x80) [0x7fbc69907370] Mar 04 09:55:19 GFSNode1 smbd[18337]: #8 /lib64/libtevent.so.0(+0x53a9) [0x7fbc699073a9] Mar 04 09:55:19 GFSNode1 smbd[18337]: #9 /lib64/libtalloc.so.2(_talloc_free+0x3f8) [0x7fbc69b13928] Mar 04 09:55:19 GFSNode1 smbd[18337]: #10 /usr/lib64/samba/libsmbd-base-samba4.so(+0x11a77f) [0x7fbc6c8d977f] Mar 04 09:55:19 GFSNode1 smbd[18337]: #11 /usr/lib64/samba/vfs/glusterfs.so(+0x39f7) [0x7fbc527e39f7] Mar 04 09:55:19 GFSNode1 smbd[18337]: #12 /lib64/libsmbconf.so.0(run_events_poll+0x16c) [0x7fbc6aeddbfc] Mar 04 09:55:19 GFSNode1 smbd[18337]: #13 /lib64/libsmbconf.so.0(+0x36e50) [0x7fbc6aedde50] Mar 04 09:55:19 GFSNode1 smbd[18337]: #14 /lib64/libtevent.so.0(_tevent_loop_once+0x8d) [0x7fbc699060fd] Mar 04 09:55:19 GFSNode1 smbd[18337]: #15 /lib64/libtevent.so.0(tevent_common_loop_wait+0x1b) [0x7fbc6990629b] Mar 04 09:55:19 GFSNode1 smbd[18337]: #16 /usr/lib64/samba/libsmbd-base-samba4.so(smbd_process+0x6d9) [0x7fbc6c8efd09] Mar 04 09:55:19 GFSNode1 smbd[18337]: #17 /usr/sbin/smbd(+0xae44) [0x7fbc6d37be44] Mar 04 09:55:19 GFSNode1 smbd[18337]: #18 /lib64/libsmbconf.so.0(run_events_poll+0x16c) [0x7fbc6aeddbfc] Mar 04 09:55:19 GFSNode1 smbd[18337]: #19 /lib64/libsmbconf.so.0(+0x36e50) [0x7fbc6aedde50] Mar 04 09:55:19 GFSNode1 smbd[18337]: #20 /lib64/libtevent.so.0(_tevent_loop_once+0x8d) [0x7fbc699060fd] Mar 04 09:55:19 GFSNode1 smbd[18337]: #21 /lib64/libtevent.so.0(tevent_common_loop_wait+0x1b) [0x7fbc6990629b] Mar 04 09:55:19 GFSNode1 smbd[18337]: #22 /usr/sbin/smbd(main+0x1501) [0x7fbc6d378361] Mar 04 09:55:19 GFSNode1 smbd[18337]: #23 /lib64/libc.so.6(__libc_start_main+0xf5) [0x7fbc69562b15] Mar 04 09:55:19 GFSNode1 smbd[18337]: #24 /usr/sbin/smbd(+0x7765) [0x7fbc6d378765] Mar 04 09:55:19 GFSNode1 smbd[18337]: [2016/03/04 09:55:19.632898, 0] ../source3/lib/util.c:800(smb_panic_s3) Mar 04 09:55:19 GFSNode1 smbd[18337]: smb_panic(): calling panic action [/usr/bin/sleep 9999999999]
Created attachment 11902 [details] patch for 4.2, 4.3 and 4.4. Patch for 4.2, 4.3, and 4.4.
Comment on attachment 11902 [details] patch for 4.2, 4.3 and 4.4. We usually want the (cherry-picked from ...) line in the patch i.e. do the cherry-pick with 'git cherry-pick -x ...'. LGTM apart from that cosmetic complaint. Cheers - Michael
(In reply to Michael Adam from comment #2) It is not a cherry pick.
(In reply to Ira Cooper from comment #3) > (In reply to Michael Adam from comment #2) > > It is not a cherry pick. Oops. in that case --> RB+ :-)
Karolin, Could you please merge the above patch into 4.2, 4.3 and 4.4? Thanks,
(In reply to Ira Cooper from comment #5) Pushed to autobuild-v4-4-test. Patch does not apply on current v4-3-test and v4-2-test. Re-assigning to Ira.
this was fixed in master for 4.5. Left unfixed for older release branches.