Bug 7859 - double free in case of composite call in error/timeout
double free in case of composite call in error/timeout
Status: CLOSED FIXED
Product: Samba 4.0
Classification: Unclassified
Component: DCE-RPCs and pipes
unspecified
Other Linux
: P3 critical
: ---
Assigned To: Andrew Bartlett
samba4-qa@samba.org
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2010-12-10 04:12 UTC by Matthieu Patou
Modified: 2014-08-27 18:19 UTC (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Matthieu Patou 2010-12-10 04:12:10 UTC
On a couple of non x86/x64 machines we have double free in case of timeout or error in a composite call.

For instance on opensolaris:

gdb_backtrace: Trying to use /opt/sunstudio12.1/bin/dbx on /export/home/tridge/build_farm/samba_4_0_test/source4/bin/default/source4/smbd/samba on PID 29904
dbx: warning: extraneous argument "29904"
dbx: Cannot open "where;dump;kill;quit" -- No such file or directory
Warning: Low number of available file descriptors (250).  If the
program has too many threads, you might need to set the
mt_scalable option, or increase the hard limit on the number of
file descriptors.  See `help dbxenv' under mt_scalable.
For information about new features see `help changes'
To remove this message, put `dbxenv suppress_startup_message 7.7' in your .dbxrc

smb_panic(): action returned status 0
PANIC: Bad talloc magic value - double free
BACKTRACE: 21 stack frames:
 #0 /export/home/tridge/build_farm/samba_4_0_test/source4/bin/default/lib/util/libsamba-util.so'call_backtrace+0x27 [0xfedba7c6]
 #1 /export/home/tridge/build_farm/samba_4_0_test/source4/bin/default/lib/util/libsamba-util.so'smb_panic+0x1f1 [0xfedbaa89]
 #2 /export/home/tridge/build_farm/samba_4_0_test/source4/bin/default/lib/talloc/libtalloc-samba4.so'talloc_abort+0x44 [0xfe8920a2]
 #3 /export/home/tridge/build_farm/samba_4_0_test/source4/bin/default/lib/talloc/libtalloc-samba4.so'talloc_abort_double_free+0x22 [0xfe892141]
 #4 /export/home/tridge/build_farm/samba_4_0_test/source4/bin/default/lib/talloc/libtalloc-samba4.so'talloc_chunk_from_ptr+0x8c [0xfe89227b]
 #5 /export/home/tridge/build_farm/samba_4_0_test/source4/bin/default/lib/talloc/libtalloc-samba4.so'_talloc_steal_internal+0x168 [0xfe892962]
 #6 /export/home/tridge/build_farm/samba_4_0_test/source4/bin/default/lib/talloc/libtalloc-samba4.so'_talloc_free_internal+0x2f2 [0xfe89309e]
samba: EOF on stdin - terminating
./bin/samba exited with no error
samba: EOF on stdin - terminating
 #7 /export/home/tridge/build_farm/samba_4_0_test/source4/bin/default/lib/talloc/libtalloc-samba4.so'_talloc_free+0xda [0xfe8937f6]
 #8 /export/home/tridge/build_farm/samba_4_0_test/source4/bin/default/source4/librpc/libdcerpc.so'dcerpc_pipe_connect_b_recv+0x7f [0xfec839df]
 #9 /export/home/tridge/build_farm/samba_4_0_test/source4/bin/default/source4/winbind/libservice-winbind.so'init_domain_recv_netlogonpipe+0x50 [0xfce6de66]
 #10 /export/home/tridge/build_farm/samba_4_0_test/source4/bin/default/lib/libsamba-sockets-samba4.so'composite_error+0xa2 [0xfe7f8ec0]
 #11 /export/home/tridge/build_farm/samba_4_0_test/source4/bin/default/source4/librpc/libdcerpc.so'dcerpc_connect_timeout_handler+0x40 [0xfec836f0]
 #12 /export/home/tridge/build_farm/samba_4_0_test/source4/bin/default/lib/tevent/libtevent-samba4.so'tevent_common_loop_timer_delay+0x18e [0xfeaa70a7]
 #13 /export/home/tridge/build_farm/samba_4_0_test/source4/bin/default/lib/tevent/libtevent-samba4.so'std_event_loop_select+0x204 [0xfeaa6775]
 #14 /export/home/tridge/build_farm/samba_4_0_test/source4/bin/default/lib/tevent/libtevent-samba4.so'std_event_loop_once+0xb3 [0xfeaa6944]
 #15 /export/home/tridge/build_farm/samba_4_0_test/source4/bin/default/lib/tevent/libtevent-samba4.so'_tevent_loop_once+0xb7 [0xfeaa3869]
 #16 /export/home/tridge/build_farm/samba_4_0_test/source4/bin/default/lib/tevent/libtevent-samba4.so'tevent_common_loop_wait+0x47 [0xfeaa3a48]
 #17 /export/home/tridge/build_farm/samba_4_0_test/source4/bin/default/lib/tevent/libtevent-samba4.so'_tevent_loop_wait+0x19 [0xfeaa3ac6]
 #18 /export/home/tridge/build_farm/samba_4_0_test/source4/bin/default/source4/smbd/samba'binary_smbd_main+0x9e3 [0x805affe]
 #19 /export/home/tridge/build_farm/samba_4_0_test/source4/bin/default/source4/smbd/samba'main+0x3e [0x805b06f]
 #20 /export/home/tridge/build_farm/samba_4_0_test/source4/bin/default/source4/smbd/samba'_start+0x83 [0x8055693]


this can happen also in "clients" like smbtorture like on builder72 (freebsd 7.2):

#4  0x0000000803a9316f in talloc_abort (reason=0x803a96b30 "Bad talloc magic value - double free") at ../../lib/talloc/talloc.c:213
#5  0x0000000803a93200 in talloc_abort_double_free () at ../../lib/talloc/talloc.c:229
#6  0x0000000803a93336 in talloc_chunk_from_ptr (ptr=0x807045820) at ../../lib/talloc/talloc.c:250
#7  0x0000000803a94995 in talloc_get_name (ptr=0x807045820) at ../../lib/talloc/talloc.c:975
#8  0x0000000803a94a33 in talloc_check_name (ptr=0x807045820, name=0x802c8fbc0 "struct composite_context") at ../../lib/talloc/talloc.c:994
#9  0x0000000802c25120 in continue_smb_connect (ctx=0x8070458b0) at ../librpc/rpc/dcerpc_connect.c:68
#10 0x00000008032ffe7f in composite_error (ctx=0x8070458b0, status={v = 3221225787}) at ../libcli/composite/composite.c:114
#11 0x00000008032fff09 in composite_is_ok (ctx=0x8070458b0) at ../libcli/composite/composite.c:132
#12 0x0000000802c56124 in state_handler (c=0x8070458b0) at ../libcli/smb_composite/connect.c:398
#13 0x0000000802c56160 in request_handler (req=0x806f323b0) at ../libcli/smb_composite/connect.c:410
#14 0x0000000802c64784 in smbcli_transport_dead (transport=0x807002580, status={v = 3221225787}) at ../libcli/raw/clitransport.c:151
#15 0x0000000802c6428f in transport_destructor (transport=0x807002580) at ../libcli/raw/clitransport.c:56
#16 0x0000000803a94275 in _talloc_free_internal (ptr=0x807002580, location=0x802c8ffa0 "../librpc/rpc/dcerpc_connect.c:810")
    at ../../lib/talloc/talloc.c:621
#17 0x0000000803a94458 in _talloc_free_internal (ptr=0x80702c830, location=0x802c8ffa0 "../librpc/rpc/dcerpc_connect.c:810")
    at ../../lib/talloc/talloc.c:652
#18 0x0000000803a94458 in _talloc_free_internal (ptr=0x8070458b0, location=0x802c8ffa0 "../librpc/rpc/dcerpc_connect.c:810")
    at ../../lib/talloc/talloc.c:652
#19 0x0000000803a94458 in _talloc_free_internal (ptr=0x806fbf350, location=0x802c8ffa0 "../librpc/rpc/dcerpc_connect.c:810")
    at ../../lib/talloc/talloc.c:652
#20 0x0000000803a94458 in _talloc_free_internal (ptr=0x80702c5b0, location=0x802c8ffa0 "../librpc/rpc/dcerpc_connect.c:810")
    at ../../lib/talloc/talloc.c:652
#21 0x0000000803a94458 in _talloc_free_internal (ptr=0x807045790, location=0x802c8ffa0 "../librpc/rpc/dcerpc_connect.c:810")
    at ../../lib/talloc/talloc.c:652
#22 0x0000000803a94fc4 in _talloc_free (ptr=0x807045790, location=0x802c8ffa0 "../librpc/rpc/dcerpc_connect.c:810")
    at ../../lib/talloc/talloc.c:1171
#23 0x0000000802c267d1 in dcerpc_pipe_connect_b_recv (c=0x807045790, mem_ctx=0x807029a60, p=0x7fffffffa790) at ../librpc/rpc/dcerpc_connect.c:810
#24 0x0000000802c26835 in dcerpc_pipe_connect_b (parent_ctx=0x807029a60, pp=0x7fffffffa790, binding=0x80702c0b0, table=0x8042ed5a0,
    credentials=0x806f32070, ev=0x806f14590, lp_ctx=0x806f0a850) at ../librpc/rpc/dcerpc_connect.c:831
#25 0x000000000068d87a in torture_rpc_connection (tctx=0x807029a60, p=0x7fffffffa790, table=0x8042ed5a0) at ../torture/rpc/rpc.c:84
#26 0x000000000068f22d in test_handles_lsa (torture=0x807029a60) at ../torture/rpc/handles.c:52
#27 0x0000000802867bba in wrap_simple_test (torture_ctx=0x807029a60, tcase=0x806f847f0, test=0x806f8a240) at ../../lib/torture/torture.c:628
#28 0x0000000802867341 in internal_torture_run_test (context=0x807029a60, tcase=0x806f847f0, test=0x806f8a240, already_setup=true, restricted=0x0)
    at ../../lib/torture/torture.c:439
#29 0x0000000802867588 in torture_run_tcase_restricted (context=0x807029a60, tcase=0x806f847f0, restricted=0x0) at ../../lib/torture/torture.c:502
#30 0x0000000802866f6b in torture_run_suite_restricted (context=0x807029a60, suite=0x806f86dd0, restricted=0x0) at ../../lib/torture/torture.c:354
#31 0x0000000802866ed2 in torture_run_suite (context=0x807029a60, suite=0x806f86dd0) at ../../lib/torture/torture.c:336
#32 0x0000000000532730 in run_matching (torture=0x807029a60, prefix=0x80703adb0 "RPC", expr=0x7fffffffb9e2 "RPC-HANDLES", restricted=0x0,
    suite=0x806f78250, matched=0x7fffffffaade) at ../torture/smbtorture.c:64
#33 0x000000000053276d in run_matching (torture=0x807029a60, prefix=0x0, expr=0x7fffffffb9e2 "RPC-HANDLES", restricted=0x0, suite=0x806f2d350,
Comment 1 Matthias Dieter Wallnöfer 2011-01-18 14:52:10 UTC
Metze has interest in fixing this.
Comment 2 Stefan Metzmacher 2011-01-19 00:47:25 UTC
I'll do the real fix, but that will take a few month.

So I'm fine if someone adds a short term hack.
Comment 3 Andreas Schneider 2011-08-12 14:23:09 UTC
==4530== Invalid read of size 4
==4530==    at 0xED5E8B: talloc_chunk_from_ptr (talloc.c:349)
==4530==    by 0xED7343: _talloc_steal_internal (talloc.c:915)
==4530==    by 0xED7FA1: _talloc_free_children_internal (talloc.c:1261)
==4530==    by 0xED6ED3: _talloc_free_internal (talloc.c:846)
==4530==    by 0xED8349: _talloc_free (talloc.c:1371)
==4530==    by 0xB426F7: dcerpc_pipe_connect_b_recv (dcerpc_connect.c:813)
==4530==    by 0x613CEF: continue_pipe_connect (libnet_rpc.c:152)
==4530==    by 0x140B86A: composite_error (composite.c:114)
==4530==    by 0xB42365: dcerpc_connect_timeout_handler (dcerpc_connect.c:719)
==4530==    by 0x1095F51: tevent_common_loop_timer_delay (tevent_timed.c:254)
==4530==    by 0x1094CFD: epoll_event_loop (tevent_standard.c:296)
==4530==    by 0x10955C7: std_event_loop_once (tevent_standard.c:565)
==4530==    by 0x10907B1: _tevent_loop_once (tevent.c:505)
==4530==    by 0x140B6DC: composite_wait (composite.c:58)
==4530==    by 0x615514: libnet_RpcConnectDCInfo_recv (libnet_rpc.c:887)
==4530==    by 0x61591E: libnet_RpcConnect_recv (libnet_rpc.c:989)
==4530==    by 0x615996: libnet_RpcConnect (libnet_rpc.c:1013)
==4530==    by 0x616C35: libnet_JoinDomain (libnet_join.c:505)
==4530==    by 0xE07072: torture_join_domain (testjoin.c:452)
==4530==    by 0xE0864F: test_schannel (schannel.c:278)
==4530==  Address 0x80faae0 is 64 bytes inside a block of size 136 free'd
==4530==    at 0x4C2599C: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==4530==    by 0xED715C: _talloc_free_internal (talloc.c:876)
==4530==    by 0xED8349: _talloc_free (talloc.c:1371)
==4530==    by 0x613EEF: libnet_RpcConnectSrv_recv (libnet_rpc.c:217)
==4530==    by 0x6158E4: libnet_RpcConnect_recv (libnet_rpc.c:982)
==4530==    by 0x6147ED: continue_dci_rpc_connect (libnet_rpc.c:543)
==4530==    by 0x140B9A9: composite_done (composite.c:143)
==4530==    by 0x613D70: continue_pipe_connect (libnet_rpc.c:172)
==4530==    by 0x140B86A: composite_error (composite.c:114)
==4530==    by 0x140B8EA: composite_is_ok (composite.c:132)
==4530==    by 0xB42012: continue_pipe_connect_ncacn_np_smb (dcerpc_connect.c:618)
==4530==    by 0x140B86A: composite_error (composite.c:114)
==4530==    by 0x140B8EA: composite_is_ok (composite.c:132)
==4530==    by 0xB40DC6: continue_pipe_open_smb (dcerpc_connect.c:56)
==4530==    by 0x140B86A: composite_error (composite.c:114)
==4530==    by 0x140B8EA: composite_is_ok (composite.c:132)
==4530==    by 0xB3E698: pipe_open_recv (dcerpc_smb.c:523)
==4530==    by 0x6A9564: smbcli_transport_dead (clitransport.c:151)
==4530==    by 0x6A9094: transport_destructor (clitransport.c:56)
==4530==    by 0xED6D2A: _talloc_free_internal (talloc.c:826)
Comment 4 Matthias Dieter Wallnöfer 2012-03-15 08:58:18 UTC
metze,

shouldn't this have been fixed by your recent rpc library rework?
Comment 5 Stefan Metzmacher 2012-03-15 15:41:36 UTC
I hope so
Comment 6 Stefan Metzmacher 2012-03-15 15:46:32 UTC
I hope so
Comment 7 Matthias Dieter Wallnöfer 2012-03-15 20:40:56 UTC
Should have been fixed. Thanks, metze
Comment 8 Matthieu Patou 2012-03-15 22:54:57 UTC
No it's not I think.

I can reproduce this very easily:


Bad talloc magic value - access after free
==29425== Invalid read of size 4
==29425==    at 0x47D15B0: gensec_spnego_create_negTokenInit (spnego.c:654)
==29425==    by 0x47D1E03: gensec_spnego_update (spnego.c:844)
==29425==    by 0x47D3444: gensec_spnego_update_wrapper (spnego.c:1314)
==29425==    by 0x47D5DD4: gensec_update (gensec.c:220)
==29425==    by 0x41EE1F2: dcerpc_bind_auth_send (dcerpc_auth.c:336)
==29425==    by 0x41F05BB: dcerpc_pipe_auth_send (dcerpc_util.c:621)
==29425==    by 0x41F5C92: continue_pipe_connect (dcerpc_connect.c:689)
==29425==    by 0x41F5ABE: continue_pipe_connect_ncacn_ip_tcp (dcerpc_connect.c:637)
==29425==    by 0x497837B: composite_done (composite.c:143)
==29425==    by 0x41F4FA0: continue_pipe_open_ncacn_ip_tcp (dcerpc_connect.c:300)
==29425==    by 0x497837B: composite_done (composite.c:143)
==29425==    by 0x41F3F87: continue_ip_open_socket (dcerpc_sock.c:441)
==29425==  Address 0x541b3f8 is 56 bytes inside a block of size 104 free'd
==29425==    at 0x4024B3A: free (vg_replace_malloc.c:366)
==29425==    by 0x4281BF4: _talloc_free_internal (talloc.c:876)
==29425==    by 0x428268D: _talloc_free_children_internal (talloc.c:1254)
==29425==    by 0x4281AB7: _talloc_free_internal (talloc.c:846)
==29425==    by 0x428268D: _talloc_free_children_internal (talloc.c:1254)
==29425==    by 0x4281AB7: _talloc_free_internal (talloc.c:846)
==29425==    by 0x428268D: _talloc_free_children_internal (talloc.c:1254)
==29425==    by 0x4281AB7: _talloc_free_internal (talloc.c:846)
==29425==    by 0x42829E2: _talloc_free (talloc.c:1369)
==29425==    by 0x41F60F0: dcerpc_pipe_connect_b_recv (dcerpc_connect.c:813)
==29425==    by 0x41F63DE: continue_pipe_connect_b (dcerpc_connect.c:904)
==29425==    by 0x4978247: composite_error (composite.c:114)
Comment 9 Andrew Bartlett 2012-08-23 04:29:33 UTC
I'm pretty sure I've fixed this one by delaying the free of the DCE/RPC pipe during a kerberos timeout.

Please confirm,
Comment 10 Partha 2014-08-26 22:49:13 UTC
(In reply to comment #9)
> I'm pretty sure I've fixed this one by delaying the free of the DCE/RPC pipe
> during a kerberos timeout.
> 
> Please confirm,

Andrew,

Looks like we also hitting the similar double free issue, but the stack is different. 

Please do let us know is there a patch availble for this double free issue.

The version of SAMBA using is 3.6.12

#0  0x0000000802cf4ffc in thr_kill () from /lib/libc.so.7
#1  0x0000000802d9058b in abort () from /lib/libc.so.7
#2  0x0000000000792711 in dump_core () at lib/fault.c:414
#3  0x00000000007a25af in smb_panic (why=<optimized out>) at lib/util.c:1133
#4  0x0000000000792db2 in fault_report (sig=<optimized out>) at lib/fault.c:53
#5  sig_fault (sig=10) at lib/fault.c:76
#6  <signal handler called>
#7  0x0000000802d1d55a in free () from /lib/libc.so.7
#8  0x000000080285ccd3 in _talloc_free_internal (ptr=0x803eb86f0, location=0xa1678f "smbd/trans2.c:2196") at ../lib/talloc/talloc.c:876
#9  0x000000080285cc03 in _talloc_free_children_internal (location=<optimized out>, ptr=<optimized out>, tc=<optimized out>) at ../lib/talloc/talloc.c:1256
#10 _talloc_free_internal (ptr=<optimized out>, location=0xa1678f "smbd/trans2.c:2196") at ../lib/talloc/talloc.c:846
#11 0x00000000004f459c in smbd_dirptr_lanman2_entry (ctx=0x805075b00, conn=0x10, dirptr=0x804f69830, flags2=49217, path_mask=<optimized out>, dirtype=<optimized out>, info_level=262, requires_resume_key=0, do
nt_descend=<optimized out>, ask_sharemode=<optimized out>, align=8 '\b', do_pad=false, ppdata=0x7fffffffe330, base_data=<optimized out>, end_data=0x8052e9047 "", space_remaining=63580, out_of_space=0x7fffffff
e356, got_exact_match=0x7fffffffe357, _last_entry_off=0x7fffffffe350, name_list=0x0) at smbd/trans2.c:2196
#12 0x000000000053d453 in smbd_smb2_find_send (in_file_name=0x8052d887c "", in_output_buffer_length=<optimized out>, in_file_index=<optimized out>, in_flags=<optimized out>, in_file_info_class=<optimized out>
, fsp=<optimized out>, smb2req=<optimized out>, ev=0x80370e110, mem_ctx=<optimized out>) at smbd/smb2_find.c:398
#13 smbd_smb2_request_process_find (req=0x805075110) at smbd/smb2_find.c:124
#14 0x000000000053056d in smbd_smb2_request_dispatch (req=0x805075110) at smbd/smb2_server.c:1661
#15 0x000000000053146e in smbd_smb2_request_incoming (subreq=0x80504f850) at smbd/smb2_server.c:2661
#16 0x000000000052f06c in smbd_smb2_request_read_done (subreq=0x803ea2890) at smbd/smb2_server.c:2504
#17 0x00000000005c7231 in tstream_readv_pdu_queue_done (subreq=0x8050e11d0) at ../lib/tsocket/tsocket_helpers.c:423
#18 0x00000000005c75e3 in tstream_readv_pdu_readv_done (subreq=0x803ea42d0) at ../lib/tsocket/tsocket_helpers.c:316
#19 0x00000000005c6662 in tstream_readv_done (subreq=0x803776150) at ../lib/tsocket/tsocket.c:604
#20 0x00000000007b2130 in tevent_common_loop_immediate (ev=0x80370e110) at ../lib/tevent/tevent_immediate.c:139
#21 0x00000000007b03f5 in run_events_poll (ev=0x80370e110, pollrtn=0, pfds=0x0, num_pfds=0) at lib/events.c:197
#22 0x0000000000521a6d in smbd_server_connection_loop_once (conn=<optimized out>) at smbd/process.c:999
#23 smbd_process (sconn=0x803711350) at smbd/process.c:3172
#24 0x0000000000a00626 in smbd_accept_connection (ev=<optimized out>, fde=<optimized out>, flags=<optimized out>, private_data=<optimized out>) at smbd/server.c:664
#25 0x00000000007b0711 in run_events_poll (ev=0x80370e110, pollrtn=<optimized out>, pfds=0x80370ffd0, num_pfds=7) at lib/events.c:286
#26 0x00000000007b0b7f in s3_event_loop_once (ev=0x80370e110, location=<optimized out>) at lib/events.c:349
#27 0x00000000007b0f31 in _tevent_loop_once (ev=0x80370e110, location=0xc0a809 "smbd/server.c:970") at ../lib/tevent/tevent.c:494
#28 0x0000000000a0234e in smbd_parent_loop (parent=<optimized out>) at smbd/server.c:970
#29 main (argc=<optimized out>, argv=<optimized out>) at smbd/server.c:1464
Comment 11 Andrew Bartlett 2014-08-26 22:54:40 UTC
(In reply to comment #10)
> (In reply to comment #9)
> > I'm pretty sure I've fixed this one by delaying the free of the DCE/RPC pipe
> > during a kerberos timeout.
> > 
> > Please confirm,
> 
> Andrew,
> 
> Looks like we also hitting the similar double free issue, but the stack is
> different. 
> 
> Please do let us know is there a patch availble for this double free issue.
> 
> The version of SAMBA using is 3.6.12

This is not the same issue (this bug is about code only in Samba 4.0 and above), and Samba 3.6 is now in security-only fix mode.  That the stack is different is a very good clue that this is not the same issue.

As regards the original issue, I'm pretty sure this is fixed, at least as well as it can be before we get a full async gensec.
Comment 12 Partha 2014-08-27 18:19:36 UTC
(In reply to comment #11)
> (In reply to comment #10)
> > (In reply to comment #9)
> > > I'm pretty sure I've fixed this one by delaying the free of the DCE/RPC pipe
> > > during a kerberos timeout.
> > > 
> > > Please confirm,
> > 
> > Andrew,
> > 
> > Looks like we also hitting the similar double free issue, but the stack is
> > different. 
> > 
> > Please do let us know is there a patch availble for this double free issue.
> > 
> > The version of SAMBA using is 3.6.12
> 
> This is not the same issue (this bug is about code only in Samba 4.0 and
> above), and Samba 3.6 is now in security-only fix mode.  That the stack is
> different is a very good clue that this is not the same issue.
> 
> As regards the original issue, I'm pretty sure this is fixed, at least as well
> as it can be before we get a full async gensec.

Thanks Andrew. As I mentioned earlier, we are using 3.6.12 and fewer picked patches till 3.6.24. Also please let me know Do you want me to file a separate bug for this issue mentioned at comment10  against SAMBA 4.0