Bug 8349 - Openchange aborts connections with core dump when sending more than 500 connections due to samba authentication issue
Openchange aborts connections with core dump when sending more than 500 conne...
Status: RESOLVED FIXED
Product: Samba 4.0
Classification: Unclassified
Component: DCE-RPCs and pipes
unspecified
x64 Linux
: P1 regression
: ---
Assigned To: Andrew Bartlett
samba4-qa@samba.org
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2011-08-04 06:50 UTC by radhakrishnan
Modified: 2012-11-14 21:31 UTC (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description radhakrishnan 2011-08-04 06:50:38 UTC
We are using openchange 0.11 version. When sending more than 500 concurrent connections from openchange client with Kerberos authentication getting below error with core dump. Have discussed with openchange guys and they conformed the issue was from samba authentication side. For find out the root cause I have attached the network traces with this bug. If u needs any other information let me know.
Error seeing in client:
./sendMail.sh: line 6: 13027 Aborted (core dumped) ./openchangeclient --profile=oc$1 --to="oc$1" --subject="openchange" --body="this is the body" --sendmail --attachments Messages/attach$1.msg
...............................................................................................................................................................................................................................................Waiting for sendMail to finish
...................We are about to write 1048576 bytes in the stream
...................................................................................................................................talloc: access after free error - first free may be at ../source4/librpc/rpc/dcerpc_connect.c:813
Bad talloc magic value - access after free
Core dump back traces:
gdb) bt
#0 0x00000030dce352d5 in raise () from /lib64/libc.so.6
#1 0x00000030dce36beb in abort () from /lib64/libc.so.6
#2 0x00007f0cc269027f in talloc_abort (reason=0x7f0cc26944b0 "Bad talloc magic value - unknown value") at ../talloc.c:317
#3 0x00007f0cc269031d in talloc_abort_unknown_value () at ../talloc.c:341
#4 0x00007f0cc2690394 in talloc_chunk_from_ptr (ptr=0x11fd310) at ../talloc.c:360
#5 0x00007f0cc26905e3 in __talloc (context=0x11fd310, size=2593) at ../talloc.c:555
#6 0x00007f0cc26909ca in _talloc_named_const (context=0x11fd310, size=2593,
name=0x7f0cc2acf740 "../lib/util/data_blob.c:52") at ../talloc.c:669
#7 0x00007f0cc269304f in _talloc_memdup (t=0x11fd310, p=0x1200e80, size=2593,
name=0x7f0cc2acf740 "../lib/util/data_blob.c:52") at ../talloc.c:1910
#8 0x00007f0cc2ab2685 in data_blob_talloc_named (mem_ctx=0x11fd310, p=0x1200e80, length=2593,
name=0x7f0cc02fb960 "DATA_BLOB: ../source4/auth/gensec/gensec_gssapi.c:522") at ../lib/util/data_blob.c:52
#9 0x00007f0cc02f36db in gensec_gssapi_update (gensec_security=0x11fde80, out_mem_ctx=0x11fd310, in=..., out=0x7fff3f65a0b0)
at ../source4/auth/gensec/gensec_gssapi.c:522
#10 0x00007f0cc02eb4d8 in gensec_update (gensec_security=0x11fde80, out_mem_ctx=0x11fd310, in=..., out=0x7fff3f65a0b0)
at ../source4/auth/gensec/gensec.c:982
#11 0x00007f0cc02f05b3 in gensec_spnego_create_negTokenInit (gensec_security=0x11e6bb0, spnego_state=0x11fce60,
out_mem_ctx=0x11fd310, in=..., out=0x11fd318) at ../source4/auth/gensec/spnego.c:619
#12 0x00007f0cc02f0faf in gensec_spnego_update (gensec_security=0x11e6bb0, out_mem_ctx=0x11fd310, in=..., out=0x11fd318)
at ../source4/auth/gensec/spnego.c:816
#13 0x00007f0cc02eb4d8 in gensec_update (gensec_security=0x11e6bb0, out_mem_ctx=0x11fd310, in=..., out=0x11fd318)
at ../source4/auth/gensec/gensec.c:982
#14 0x00007f0cc3129534 in dcerpc_bind_auth_send (mem_ctx=0x11fbac0, p=0x11f2c30, table=0x7f0cc38b78c0,
credentials=0x11c6640, gensec_settings=0x11fd400, auth_type=9 '\t', auth_level=2 '\002',
service=0x7f0cc3677a0a "exchangeMDB") at ../source4/librpc/rpc/dcerpc_auth.c:336
#15 0x00007f0cc312b686 in dcerpc_pipe_auth_send (p=0x11f2c30, binding=0x11e72f0, table=0x7f0cc38b78c0,
credentials=0x11c6640, lp_ctx=0x11b7b00) at ../source4/librpc/rpc/dcerpc_util.c:621
#16 0x00007f0cc3130876 in continue_pipe_connect (c=0x11f6930, s=0x11e5490) at ../source4/librpc/rpc/dcerpc_connect.c:689
#17 0x00007f0cc31306cb in continue_pipe_connect_ncacn_ip_tcp (ctx=0x11fcdf0) at ../source4/librpc/rpc/dcerpc_connect.c:637
#18 0x00007f0cbea15a49 in composite_done (ctx=0x11fcdf0) at ../source4/libcli/composite/composite.c:143
#19 0x00007f0cc312fc4b in continue_pipe_open_ncacn_ip_tcp (ctx=0x11e6bb0) at ../source4/librpc/rpc/dcerpc_connect.c:300
#20 0x00007f0cbea15a49 in composite_done (ctx=0x11e6bb0) at ../source4/libcli/composite/composite.c:143
#21 0x00007f0cc312ecc1 in continue_ip_open_socket (ctx=0x11fd5d0) at ../source4/librpc/rpc/dcerpc_sock.c:423
#22 0x00007f0cbea15a49 in composite_done (ctx=0x11fd5d0) at ../source4/libcli/composite/composite.c:143
#23 0x00007f0cc312e7b3 in continue_socket_connect (ctx=0x11fbac0) at ../source4/librpc/rpc/dcerpc_sock.c:304
#24 0x00007f0cbea15a49 in composite_done (ctx=0x11fbac0) at ../source4/libcli/composite/composite.c:143
#25 0x00007f0cbea14a40 in socket_connect_handler (ev=0x11fa140, fde=0x11fdb20, flags=2, private_data=0x11fbac0)
at ../source4/lib/socket/connect.c:131
#26 0x00007f0cc289f169 in epoll_event_loop (std_ev=0x11fbde0, tvalp=0x7fff3f65a870) at ../tevent_standard.c:309
#27 0x00007f0cc289f8af in std_event_loop_once (ev=0x11fa140,
location=0x7f0cbea1fd30 "../source4/libcli/composite/composite.c:58") at ../tevent_standard.c:548
#28 0x00007f0cc289adad in _tevent_loop_once (ev=0x11fa140,
location=0x7f0cbea1fd30 "../source4/libcli/composite/composite.c:58") at ../tevent.c:494
#29 0x00007f0cbea15789 in composite_wait (c=0x11f6620) at ../source4/libcli/composite/composite.c:58
---Type <return> to continue, or q <return> to quit---
#30 0x00007f0cc3131015 in dcerpc_pipe_connect_recv (c=0x11f6620, mem_ctx=0x11e6eb0, pp=0x7fff3f65aa18)
at ../source4/librpc/rpc/dcerpc_connect.c:923
#31 0x00007f0cc31310e5 in dcerpc_pipe_connect (parent_ctx=0x11e6eb0, pp=0x7fff3f65aa18,
binding=0x11fb680 "ncacn_ip_tcp:vmpc-2001.mtp2k7.local[]", table=0x7f0cc38b78c0, credentials=0x11c6640, ev=0x11fa140,
lp_ctx=0x11b7b00) at ../source4/librpc/rpc/dcerpc_connect.c:948
#32 0x00007f0cc35b37eb in provider_rpc_connection (parent_ctx=0x11e6eb0, p=0x7fff3f65aa18,
binding=0x11fb680 "ncacn_ip_tcp:vmpc-2001.mtp2k7.local[]", credentials=0x11c6640, table=0x7f0cc38b78c0, lp_ctx=0x11b7b00)
at libmapi/IMSProvider.c:60
#33 0x00007f0cc35b3d15 in Logon (session=0x11c5e70, provider=0x11e6eb0, provider_id=<optimized out>)
at libmapi/IMSProvider.c:262
#34 0x00007f0cc35b97bf in MapiLogonProvider (mapi_ctx=0x11b7a80, session=0x7fff3f65aac0, profname=<optimized out>,
password=<optimized out>, provider=<optimized out>) at libmapi/cdo_mapi.c:171
#35 0x00007f0cc35b9aa7 in MapiLogonEx (mapi_ctx=0x11b7a80, session=0x7fff3f65b878, profname=0x11b75b0 "oc1303", password=0x0)
at libmapi/cdo_mapi.c:70
#36 0x0000000000403b86 in main (argc=<optimized out>, argv=<optimized out>) at utils/openchangeclient.c:3189
(gdb)
Comment 1 radhakrishnan 2011-08-04 07:09:42 UTC
Operating system & current  configurations

Client operating system - fedora 15 64 bit
Server operating system - windows server (2008 and 200r2)
Samba installed version - samba-4.0.0alpha16.tar.gz
Client side configuration file

Smb.conf file 



global]
        workgroup = mtp2k7
        security = ads
        realm = MTP2k7.LOCAL
        client use spnego = no
        server signing = auto
        netbios name = VMPC-657
        winbind use default domain = yes
        #removes the domain prefix from usernames
        winbind separator = +
        #this is the seperator used to separate domain from username.
        encrypt passwords = yes

        password server = MTP2k7.LOCAL
        template shell = /bin/bash
[test]
        comment = Test Share using Active Directory
        path = /data
        valid users = @"Mtp2k7\Users"
        writeable = yes
        browseable = yes

Krb5.config


[logging]
 default = FILE:/var/log/krb5libs.log
 kdc = FILE:/var/log/krb5kdc.log
 admin_server = FILE:/var/log/kadmind.log

[libdefaults]
 default_realm = MTP2K7.LOCAL
 dns_lookup_realm = true
 dns_lookup_kdc = true
 ticket_lifetime = 24h
 renew_lifetime = 7d
 forwardable = true

[realms]
 MTP2K71.LOCAL = {
  kdc = vmpc-2001.mtp2k7.local:88
  admin_server = vmpc-2001.mtp2k7.local:749
  default_domain = mtp2k7.local
 }

[domain_realm]
 .mtp2k7.local = MTP2K7.LOCAL
  mtp2k7.local = MTP2K7.LOCAL
Comment 2 Matthias Dieter Wallnöfer 2011-09-14 10:51:59 UTC
The GENSEC auth backend has seen some restructuring in the last months. Please get a newer s4 release (GIT checkout if possible) and retest!
Comment 3 Matthias Dieter Wallnöfer 2011-10-11 07:11:09 UTC
I'm closing this with "INVALID" since in the meantime the issue could have been fixed. Please retry with new versions of both OpenChange and s4 and feel free to REOPEN if the problem persists.
Comment 4 Andrew Bartlett 2011-10-11 07:53:09 UTC
I'm sorry, but I'm pretty sure this is still very real.  We need to sort out the event loops here, and that's trickier than it looks.

The work that will fix this will be metze's new dce/rpc client lib, and then other fixes in addition.
Comment 5 Matthias Dieter Wallnöfer 2012-03-15 09:06:26 UTC
metze,

shouldn't this have been fixed by your recent rpc library rework?
Comment 6 Stefan Metzmacher 2012-03-15 15:42:38 UTC
I'm not sure, I fixed the the rpc layer not the gensec layer.
Comment 7 Andrew Bartlett 2012-11-14 21:31:00 UTC
I'm pretty sure this is fixed now, because we avoid tearing down the connection while talking to the KDC.