Bug 8570 - talloc_stackframe() is an evil with multiple RPC requests on two connections
Summary: talloc_stackframe() is an evil with multiple RPC requests on two connections
Status: NEW
Alias: None
Product: Samba 4.0
Classification: Unclassified
Component: DCE-RPCs and pipes (show other bugs)
Version: unspecified
Hardware: All All
: P5 normal (vote)
Target Milestone: ---
Assignee: Andrew Bartlett
QA Contact: samba4-qa@samba.org
URL:
Keywords:
Depends on:
Blocks: 6358
  Show dependency treegraph
 
Reported: 2011-11-03 10:51 UTC by Milan Crha
Modified: 2012-02-24 15:54 UTC (History)
2 users (show)

See Also:


Attachments
proposed samba4 patch (836 bytes, patch)
2011-11-03 11:05 UTC, Milan Crha
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Milan Crha 2011-11-03 10:51:43 UTC
Using talloc_stackframe() in a multi-threaded environment with simultaneous RPC requests being done on two independent connections from two threads leads to a crash. See the valgrind claim below. Using talloc_new() acquired to passed-in TALLOC_CTX pointer fixes the issue and I'm able to do multiple requests in the same time without any issue (I had opened about 4 different connections with no noticed problem).

The idea behind this is that if I make sure that there will be only one RPC call on each independent connection, which I am able to do, then I can do independent calls on independent connections in multiple threads. Thus if I'm able to distinguish them from outside, then it cannot crash from inside.

talloc information passed in just before the crash:

   talloc: access after free error - first free may be at
     ../lib/util/talloc_stack.c:103
   Bad talloc magic value - access after free


Relevant part of the valgrind report:

==14661== Invalid read of size 4
==14661==    at 0x1EAAB659: tevent_req_is_in_progress (tevent_req.c:174)
==14661==    by 0x1EAAB73D: tevent_req_poll (tevent_req.c:195)
==14661==    by 0x22314DC8: dcerpc_binding_handle_call (binding_handle.c:542)
==14661==    by 0x1E5EB999: dcerpc_RfrGetNewDSA_r (ndr_exchange_c.c:659)
==14661==    by 0x1E563A26: RfrGetNewDSA (IMSProvider.c:171)
==14661==    by 0x1E563ED2: Logon (IMSProvider.c:299)
==14661==    by 0x1E569856: MapiLogonProvider (cdo_mapi.c:179)
==14661==    by 0x1E569B59: MapiLogonEx (cdo_mapi.c:67)
==14661==    by 0x1E2D9715: mapi_profile_load (e-mapi-connection.c:4445)
==14661==    by 0x1E2C807F: e_mapi_connection_new (e-mapi-connection.c:378)
==14661==    by 0x2A26BC05: mapi_authenticate_sync (camel-mapi-store.c:1823)
==14661==    by 0x6FD7A3D: camel_service_authenticate_sync (camel-service.c:1320)
==14661==    by 0x12ADCBE8: mail_session_authenticate_sync (e-mail-session.c:1248)
==14661==    by 0x6FDBB95: camel_session_authenticate_sync (camel-session.c:1621)
==14661==    by 0x2A26B7EA: mapi_connect_sync (camel-mapi-store.c:1722)
==14661==    by 0x6FD715D: camel_service_connect_sync (camel-service.c:1131)
==14661==    by 0x2A269776: mapi_update_folder_info_cb (camel-mapi-store.c:951)
==14661==    by 0x6FD8395: session_do_job_cb (camel-session.c:165)
==14661==    by 0x3571C67C0B: run_in_thread (gsimpleasyncresult.c:843)
==14661==    by 0x3571C59495: io_job_thread (gioscheduler.c:180)
==14661==    by 0x356F86C757: g_thread_pool_thread_proxy (gthreadpool.c:319)
==14661==    by 0x356F86A235: g_thread_create_proxy (gthread.c:1962)
==14661==    by 0x356DC07D30: start_thread (in /lib64/libpthread-2.14.90.so)
==14661==    by 0x356D8EFDFC: clone (in /lib64/libc-2.14.90.so)
==14661==  Address 0x100e28e8 is 152 bytes inside a block of size 184 free'd
==14661==    at 0x4A0662E: free (vg_replace_malloc.c:366)
==14661==    by 0x1F0EC16D: _talloc_free_internal (talloc.c:875)
==14661==    by 0x1F0ECF14: _talloc_free_children_internal (talloc.c:1268)
==14661==    by 0x1F0EC01A: _talloc_free_internal (talloc.c:845)
==14661==    by 0x1F0ED134: _talloc_free (talloc.c:1360)
==14661==    by 0x1F2FD2F7: talloc_pop (talloc_stack.c:103)
==14661==    by 0x1F0EBE91: _talloc_free_internal (talloc.c:826)
==14661==    by 0x1F0ED134: _talloc_free (talloc.c:1360)
==14661==    by 0x22314E42: dcerpc_binding_handle_call (binding_handle.c:554)
==14661==    by 0x1E5FC1CC: dcerpc_EcDoRpc_r (ndr_exchange_c.c:12090)
==14661==    by 0x1E54C86E: emsmdb_transaction (emsmdb.c:441)
==14661==    by 0x1E54CC8F: emsmdb_transaction_wrapper (emsmdb.c:593)
==14661==    by 0x1E55B333: QueryRows (IMAPITable.c:296)
==14661==    by 0x1E2CD395: foreach_tablerow (e-mapi-connection.c:1696)
==14661==    by 0x1E2D6575: get_child_folders (e-mapi-connection.c:3796)
==14661==    by 0x1E2D7134: e_mapi_connection_get_folders_list (e-mapi-connection.c:4067)
==14661==    by 0x2A267FA9: mapi_folders_sync (camel-mapi-store.c:398)
==14661==    by 0x2A2697B9: mapi_update_folder_info_cb (camel-mapi-store.c:958)
==14661==    by 0x6FD8395: session_do_job_cb (camel-session.c:165)
==14661==    by 0x3571C67C0B: run_in_thread (gsimpleasyncresult.c:843)
==14661==    by 0x3571C59495: io_job_thread (gioscheduler.c:180)
==14661==    by 0x356F86C757: g_thread_pool_thread_proxy (gthreadpool.c:319)
==14661==    by 0x356F86A235: g_thread_create_proxy (gthread.c:1962)
==14661==    by 0x356DC07D30: start_thread (in /lib64/libpthread-2.14.90.so)
==14661==    by 0x356D8EFDFC: clone (in /lib64/libc-2.14.90.so)
Comment 1 Volker Lendecke 2011-11-03 10:54:31 UTC
Well, we need to create a stackframe per thread I think.
Comment 2 Milan Crha 2011-11-03 11:05:12 UTC
Created attachment 7058 [details]
proposed samba4 patch

for samba4;

With this simple patch there are no valgrind claims and no crashes when doing RPC calls from multiple threads on separate connections.

I see the talloc_stackframe() being used on many places in the code, maybe you'll consider replacing it there as well, though I'm interested only in these two, which are (mostly?) relevant to OpenChange and for me to evolution-mapi.
Comment 3 Stefan Metzmacher 2011-11-03 11:37:43 UTC
We already have code to have a stackframe per thread.

But the application needs to provide thread hooks.

Here's an example:
http://gitweb.samba.org/?p=samba.git;a=blob;f=source3/libsmb/libsmb_thread_posix.c

But I'm not sure if we export the needs function (smb_thread_set_functions) yet.
Comment 4 Andrew Bartlett 2011-11-03 12:00:15 UTC
This is very unfortunate, as when we discussed talloc_stackframe(), it was not clear to me that our library callers would need to be changed.  I would prefer that users of the dcerpc API did not need to know about our internal threading handling in order to use the API.  

Similarly, we should ensure that this is detected by some clearer failure path.

We only recently started to use talloc_stackframe() in the common code (and essentially never use talloc_tos()), and so if we chose too, it would not be a very large set of patches to remove the remaining users.
Comment 5 Volker Lendecke 2011-11-03 12:17:42 UTC
The much bigger question to me is -- are our libraries (both s3 and s4 and all the base code both depend on) designed to be used in a multi-threaded environment?
Comment 6 Milan Crha 2011-11-03 13:19:00 UTC
Hrm, I seem to be too quick, as I see evolution crashing when run without valgrind, in ndr_push_string and convert_string_talloc, like in this backtrace:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffcbdd3700 (LWP 9998)]
0x000000356d822755 in __gconv () from /lib64/libc.so.6

(gdb) t a a bt

Thread 15 (Thread 0x7fffc9dcf700 (LWP 10013)):
#0  0x000000356d8f03e3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fffe64c6f7c in epoll_event_loop (std_ev=0x7fffc402c440, tvalp=0x7fffc9dce350) at ../tevent_standard.c:264
#2  0x00007fffe64c7897 in std_event_loop_once (ev=0x7fffc402e8a0, location=0x7fffe64c997d "../tevent_req.c:198") at ../tevent_standard.c:548
#3  0x00007fffe64c2dad in _tevent_loop_once (ev=0x7fffc402e8a0, location=0x7fffe64c997d "../tevent_req.c:198") at ../tevent.c:494
#4  0x00007fffe64c4722 in tevent_req_poll (req=0x7fffb40140f0, ev=0x7fffc402e8a0) at ../tevent_req.c:198
#5  0x00007fffe2c6dd9c in dcerpc_binding_handle_call (h=0x7fffc40292c0, object=0x0, table=0x7fffe6c5c9c0, opnum=2, r_mem=0x7fffb4014710, r_ptr=0x7fffc9dce490) at ../librpc/rpc/binding_handle.c:542
#6  0x00007fffe69f11cd in dcerpc_EcDoRpc_r (h=<optimized out>, mem_ctx=<optimized out>, r=<optimized out>) at gen_ndr/ndr_exchange_c.c:12090
#7  0x00007fffe694186f in emsmdb_transaction (emsmdb_ctx=0x7fffc402e610, mem_ctx=0x7fffb4014710, req=0x7fffb4014520, repl=0x7fffc9dce530) at libmapi/emsmdb.c:441
#8  0x00007fffe6941c90 in emsmdb_transaction_wrapper (session=<optimized out>, mem_ctx=<optimized out>, req=<optimized out>, repl=<optimized out>) at libmapi/emsmdb.c:593
#9  0x00007fffe69581a9 in Release (obj=0x7fffc9dce5e0) at libmapi/IUnknown.c:149
#10 0x00007fffe695ffd4 in mapi_object_release (obj=0x7fffc9dce5e0) at libmapi/mapi_object.c:97
#11 0x00007fffe6c89234 in e_mapi_connection_resolve_named_prop (conn=0x64b6c0 [EMapiConnection], fid=18245770940338143233, pidlid_propid=2149318687, cancellable=0x0, perror=0x0) at e-mapi-connection.c:3030
#12 0x00007fffe6c95152 in e_mapi_util_find_array_namedid (properties=0x7fffc9dce950, conn=0x64b6c0 [EMapiConnection], fid=18245770940338143233, namedid=2149318687) at e-mapi-utils.c:256
#13 0x00007fffe6ed1e71 in mapi_book_utils_contact_from_props (conn=0x64b6c0 [EMapiConnection], fid=18245770940338143233, book_uri=0x7fffc40010c0 "mapi://user@server/;Contacts", 
    mapi_properties=0x7fffc9dce950, aRow=0x0) at e-book-backend-mapi.c:1870
#14 0x00007fffe6ed54fa in fetch_contacts_cb (item_data=0x7fffb40038a0, data=0x7fffc9dceb00, cancellable=0x0, perror=0x7fffc9dceaf8) at e-book-backend-mapi-contacts.c:643
#15 0x00007fffe6c8508f in e_mapi_connection_fetch_items (conn=0x64b6c0 [EMapiConnection], fid=18245770940338143233, build_rs_cb=0x7fffe6eccb24 <ebbm_build_cache_update_restriction>, build_rs_cb_data=0x756a10, 
    sort_order=0x0, build_props=0x7fffe6ed1639 <mapi_book_utils_get_prop_list>, brp_data=0x1, cb=0x7fffe6ed5410 <fetch_contacts_cb>, data=0x7fffc9dceb00, options=15, cancellable=0x0, perror=0x7fffc9dceaf8)
    at e-mapi-connection.c:2119
#16 0x00007fffe6ed8146 in ebbm_contacts_fetch_contacts (ebma=0x756a10 [EBookBackendMAPIContacts], build_rs_cb=0x7fffe6eccb24 <ebbm_build_cache_update_restriction>, build_rs_cb_data=0x756a10, book_view=0x0, 
    notify_contact_data=0x7fffc9dceb90, error=0x7fffc9dcebd0) at e-book-backend-mapi-contacts.c:1162
#17 0x00007fffe6eccab1 in ebbm_fetch_contacts (ebma=0x756a10 [EBookBackendMAPIContacts], build_rs_cb=0x7fffe6eccb24 <ebbm_build_cache_update_restriction>, build_rs_cb_data=0x756a10, book_view=0x0, 
    last_modification_secs=0x7fffc9dcebd8, error=0x7fffc9dcebd0) at e-book-backend-mapi.c:251
#18 0x00007fffe6eccff5 in ebbm_update_cache_cb (data=0x756a10) at e-book-backend-mapi.c:342
#19 0x000000356f86a236 in g_thread_create_proxy (data=0x7fffc402a920) at gthread.c:1962
#20 0x000000356dc07d31 in start_thread () from /lib64/libpthread.so.0
#21 0x000000356d8efdfd in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7fffddc09700 (LWP 10008)):
#0  0x000000356d8e2865 in _xstat () from /lib64/libc.so.6
#1  0x00007fffe5a20aa0 in ldb_modules_load_path (path=0x7fffd801cbf0 "/build/local/samba/modules/ldb", version=0x7fffe5a43e47 "1.1.0") at ../common/ldb_modules.c:867
#2  0x00007fffe5a212d9 in ldb_modules_load (modules_path=0x7fffe5a43e28 "/build/local/samba/modules/ldb", version=0x7fffe5a43e47 "1.1.0") at ../common/ldb_modules.c:1073
#3  0x00007fffe5a3af1f in ldb_init (mem_ctx=0x7fffd801c820, ev_ctx=0x7fffd801de90) at ../common/ldb.c:101
#4  0x00007fffe69444ed in OpenProfileStore (mem_ctx=0x7fffd801c820, ldb_ctx=0x7fffd801c828, profiledb=0x7fffd8002c00 "/home/user/.local/share/evolution/mapi-profiles.ldb") at libmapi/IProfAdmin.c:559
#5  0x00007fffe695ec69 in MAPIInitialize (_mapi_ctx=0x7fffddc08968, profiledb=0x7fffd8002c00 "/home/user/.local/share/evolution/mapi-profiles.ldb") at libmapi/cdo_mapi.c:247
#6  0x00007fffe6c97218 in e_mapi_utils_create_mapi_context (mapi_ctx=0x7fffddc08968, perror=0x7fffddc089d8) at e-mapi-utils.c:1232
#7  0x00007fffe6c7e053 in e_mapi_connection_new (profile=0x7fffd8007ce0 "user@server2", password=0x7fffd8018120 "<pass>", cancellable=0x7fffd80016a0 [GCancellable], perror=
    0x7fffddc089d8) at e-mapi-connection.c:375
#8  0x00007fffe6ecd2ef in ebbm_connect_user (ebma=0x756dd0 [EBookBackendMAPIContacts], cancellable=0x7fffd80016a0 [GCancellable], password=0x7fffd8018120 "<pass>", error=0x7fffddc08ab0)
    at e-book-backend-mapi.c:414
#9  0x00007fffe6ecde15 in ebbm_authenticate_user (ebma=0x756dd0 [EBookBackendMAPIContacts], cancellable=0x7fffd80016a0 [GCancellable], credentials=0x7fffb8001180, error=0x7fffddc08ab0)
    at e-book-backend-mapi.c:606
#10 0x00007fffe6ecf0f4 in ebbm_operation_cb (op=0x7fffb8001150, cancelled=0, backend=0x756dd0 [EBookBackendMAPIContacts]) at e-book-backend-mapi.c:1036
#11 0x00007fffe6cac0b0 in thread_func_cb (data=0x7fffb8001d10, pqueue=0x7fffc4016940) at e-mapi-operation-queue.c:144
#12 0x000000356f86c758 in g_thread_pool_thread_proxy (data=<optimized out>) at gthreadpool.c:319
#13 0x000000356f86a236 in g_thread_create_proxy (data=0x7fffb8001100) at gthread.c:1962
#14 0x000000356dc07d31 in start_thread () from /lib64/libpthread.so.0
#15 0x000000356d8efdfd in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7fffc95ce700 (LWP 10006)):
#0  0x000000356dc0e2dd in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x000000356dc09f27 in _L_lock_863 () from /lib64/libpthread.so.0
#2  0x000000356dc09d7b in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000000356f86a648 in g_static_rec_mutex_lock (mutex=0x7fffe6ec6c20) at gthread.c:1450
#4  0x00007fffe6c970e3 in manage_global_lock (lock=1) at e-mapi-utils.c:1188
#5  0x00007fffe6c97101 in e_mapi_utils_global_lock () at e-mapi-utils.c:1196
#6  0x00007fffe6c97153 in e_mapi_utils_create_mapi_context (mapi_ctx=0x7fffc95cd968, perror=0x7fffc95cd9d8) at e-mapi-utils.c:1214
#7  0x00007fffe6c7e053 in e_mapi_connection_new (profile=0x7fffac001110 "user@server2", password=0x7fffac00fac0 "<pass>", cancellable=0x7fffb4015d60 [GCancellable], perror=
    0x7fffc95cd9d8) at e-mapi-connection.c:375
#8  0x00007fffe6ecd2ef in ebbm_connect_user (ebma=0x756ce0 [EBookBackendMAPIGAL], cancellable=0x7fffb4015d60 [GCancellable], password=0x7fffac00fac0 "<pass>", error=0x7fffc95cdab0)
    at e-book-backend-mapi.c:414
#9  0x00007fffe6ecde15 in ebbm_authenticate_user (ebma=0x756ce0 [EBookBackendMAPIGAL], cancellable=0x7fffb4015d60 [GCancellable], credentials=0x7fffd8007950, error=0x7fffc95cdab0) at e-book-backend-mapi.c:606
#10 0x00007fffe6ecf0f4 in ebbm_operation_cb (op=0x7fffd8007300, cancelled=0, backend=0x756ce0 [EBookBackendMAPIGAL]) at e-book-backend-mapi.c:1036
#11 0x00007fffe6cac0b0 in thread_func_cb (data=0x7fffd8007ca0, pqueue=0x7a3320) at e-mapi-operation-queue.c:144
#12 0x000000356f86c758 in g_thread_pool_thread_proxy (data=<optimized out>) at gthreadpool.c:319
#13 0x000000356f86a236 in g_thread_create_proxy (data=0x7fffb40043b0) at gthread.c:1962
#14 0x000000356dc07d31 in start_thread () from /lib64/libpthread.so.0
#15 0x000000356d8efdfd in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7fffca5d0700 (LWP 10002)):
#0  0x000000356dc0bddf in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007ffff7181445 in g_cond_timed_wait_posix_impl (cond=<optimized out>, entered_mutex=<optimized out>, abs_time=<optimized out>) at gthread-posix.c:242
#2  0x000000356f81968f in g_async_queue_pop_intern_unlocked (queue=0x767b50, try=0, end_time=0x7fffca5cfbf0) at gasyncqueue.c:415
#3  0x000000356f819ee1 in g_async_queue_timed_pop (queue=0x767b50, end_time=<optimized out>) at gasyncqueue.c:536
#4  0x000000356f86c677 in g_thread_pool_wait_for_new_pool () at gthreadpool.c:175
#5  g_thread_pool_thread_proxy (data=<optimized out>) at gthreadpool.c:378
#6  0x000000356f86a236 in g_thread_create_proxy (data=0x7fffb4000990) at gthread.c:1962
#7  0x000000356dc07d31 in start_thread () from /lib64/libpthread.so.0
#8  0x000000356d8efdfd in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7fffcb5d2700 (LWP 10000)):
#0  0x000000356dc0e2dd in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x000000356dc09f27 in _L_lock_863 () from /lib64/libpthread.so.0
#2  0x000000356dc09d7b in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000000356f86a648 in g_static_rec_mutex_lock (mutex=0x7fffe6ec6c20) at gthread.c:1450
#4  0x00007fffe6c970e3 in manage_global_lock (lock=1) at e-mapi-utils.c:1188
#5  0x00007fffe6c97101 in e_mapi_utils_global_lock () at e-mapi-utils.c:1196
#6  0x00007fffe6c97153 in e_mapi_utils_create_mapi_context (mapi_ctx=0x7fffcb5d1968, perror=0x7fffcb5d19d8) at e-mapi-utils.c:1214
#7  0x00007fffe6c7e053 in e_mapi_connection_new (profile=0x7fffc0001100 "user@server2", password=0x7fffc00119c0 "<pass>", cancellable=0x7fffc0010580 [GCancellable], perror=
    0x7fffcb5d19d8) at e-mapi-connection.c:375
#8  0x00007fffe6ecd2ef in ebbm_connect_user (ebma=0x756bf0 [EBookBackendMAPIContacts], cancellable=0x7fffc0010580 [GCancellable], password=0x7fffc00119c0 "<pass>", error=0x7fffcb5d1ab0)
    at e-book-backend-mapi.c:414
#9  0x00007fffe6ecde15 in ebbm_authenticate_user (ebma=0x756bf0 [EBookBackendMAPIContacts], cancellable=0x7fffc0010580 [GCancellable], credentials=0x7fffd8003770, error=0x7fffcb5d1ab0)
    at e-book-backend-mapi.c:606
#10 0x00007fffe6ecf0f4 in ebbm_operation_cb (op=0x7fffd8003740, cancelled=0, backend=0x756bf0 [EBookBackendMAPIContacts]) at e-book-backend-mapi.c:1036
#11 0x00007fffe6cac0b0 in thread_func_cb (data=0x7fffd8003cc0, pqueue=0x7fffd0016c00) at e-mapi-operation-queue.c:144
#12 0x000000356f86c758 in g_thread_pool_thread_proxy (data=<optimized out>) at gthreadpool.c:319
#13 0x000000356f86a236 in g_thread_create_proxy (data=0x7fffd8003840) at gthread.c:1962
#14 0x000000356dc07d31 in start_thread () from /lib64/libpthread.so.0
#15 0x000000356d8efdfd in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7fffcbdd3700 (LWP 9998)):
#0  0x000000356d822755 in __gconv () from /lib64/libc.so.6
#1  0x000000356d821dba in iconv () from /lib64/libc.so.6
#2  0x00007fffe5c727c7 in sys_iconv (cd=0x7fffc401d200, inbuf=0x7fffcbdd23b0, inbytesleft=0x7fffcbdd23c0, outbuf=0x7fffcbdd2420, outbytesleft=0x7fffcbdd2430) at ../lib/util/charset/iconv.c:150
#3  0x00007fffe5c7293c in smb_iconv (cd=0x7fffc402ad60, inbuf=0x7fffcbdd2428, inbytesleft=0x7fffcbdd2438, outbuf=0x7fffcbdd2420, outbytesleft=0x7fffcbdd2430) at ../lib/util/charset/iconv.c:210
#4  0x00007fffe5c6fe75 in convert_string_talloc_handle (ctx=0x7fffbc032660, ic=0x7fffd001a3e0, from=CH_UNIX, to=CH_DOS, src=0x7fffbc0325a0, srclen=96, dst=0x7fffcbdd2518, converted_size=0x7fffcbdd2520)
    at ../lib/util/charset/convert_string.c:417
#5  0x00007fffe5c7020d in convert_string_talloc (ctx=0x7fffbc032660, from=CH_UNIX, to=CH_DOS, src=0x7fffbc0325a0, srclen=96, dest=0x7fffcbdd2518, converted_size=0x7fffcbdd2520)
    at ../lib/util/charset/convert_string.c:535
#6  0x00007fffe62ad325 in ndr_push_string (ndr=0x7fffbc032660, ndr_flags=1, s=0x7fffbc0325a0 "/o=Exchange/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=user")
    at ../librpc/ndr/ndr_string.c:235
#7  0x00007fffe696f09d in ndr_push_Logon_req (ndr=0x7fffbc032660, ndr_flags=<optimized out>, r=0x7fffbc033c68) at ndr_mapi.c:1602
#8  0x00007fffe69c8dfd in ndr_push_EcDoRpc_MAPI_REQ_UNION (ndr=0x7fffbc032660, ndr_flags=1, r=0x7fffbc033c68) at gen_ndr/ndr_exchange.c:37170
#9  0x00007fffe69d9337 in ndr_push_EcDoRpc_MAPI_REQ (ndr=0x7fffbc032660, ndr_flags=1, r=0x7fffbc033c60) at gen_ndr/ndr_exchange.c:41626
#10 0x00007fffe696a46b in ndr_push_mapi_request (ndr=0x7fffbc032660, ndr_flags=<optimized out>, r=0x7fffbc031890) at ndr_mapi.c:536
#11 0x00007fffe696be6c in ndr_push_EcDoRpc (ndr=0x7fffbc032e20, flags=1, r=0x7fffcbdd2800) at ndr_mapi.c:890
#12 0x00007fffe2c6d7be in dcerpc_binding_handle_call_send (mem_ctx=0x7fffbc031dd0, ev=0x7fffbc02ce20, h=0x7fffbc02c2d0, object=0x0, table=0x7fffe6c5c9c0, opnum=2, r_mem=0x7fffbc01c790, r_ptr=0x7fffcbdd2800)
    at ../librpc/rpc/binding_handle.c:387
#13 0x00007fffe2c6dd61 in dcerpc_binding_handle_call (h=0x7fffbc02c2d0, object=0x0, table=0x7fffe6c5c9c0, opnum=2, r_mem=0x7fffbc01c790, r_ptr=0x7fffcbdd2800) at ../librpc/rpc/binding_handle.c:534
#14 0x00007fffe69f11cd in dcerpc_EcDoRpc_r (h=<optimized out>, mem_ctx=<optimized out>, r=<optimized out>) at gen_ndr/ndr_exchange_c.c:12090
#15 0x00007fffe694186f in emsmdb_transaction (emsmdb_ctx=0x7fffbc0328c0, mem_ctx=0x7fffbc01c790, req=0x7fffbc031890, repl=0x7fffcbdd28c0) at libmapi/emsmdb.c:441
#16 0x00007fffe6941c90 in emsmdb_transaction_wrapper (session=<optimized out>, mem_ctx=<optimized out>, req=<optimized out>, repl=<optimized out>) at libmapi/emsmdb.c:593
#17 0x00007fffe694d3c0 in OpenUserMailbox (session=0x7fffbc019570, username=0x7fffbc015e50 "user", obj_store=0x64ba98) at libmapi/IMAPISession.c:372
#18 0x00007fffe694d6e3 in OpenMsgStore (session=0x7fffbc019570, obj_store=0x64ba98) at libmapi/IMAPISession.c:275
#19 0x00007fffe6c7e1d8 in e_mapi_connection_new (profile=0x7fffbc001100 "user@server2", password=0x7fffbc0013a0 "<pass>", cancellable=0x7606f0 [GCancellable], perror=
    0x7fffcbdd29d8) at e-mapi-connection.c:394
#20 0x00007fffe6ecd2ef in ebbm_connect_user (ebma=0x756b00 [EBookBackendMAPIContacts], cancellable=0x7606f0 [GCancellable], password=0x7fffbc0013a0 "<pass>", error=0x7fffcbdd2ab0)
    at e-book-backend-mapi.c:414
#21 0x00007fffe6ecde15 in ebbm_authenticate_user (ebma=0x756b00 [EBookBackendMAPIContacts], cancellable=0x7606f0 [GCancellable], credentials=0x7fffd80031d0, error=0x7fffcbdd2ab0) at e-book-backend-mapi.c:606
#22 0x00007fffe6ecf0f4 in ebbm_operation_cb (op=0x7fffd80031a0, cancelled=0, backend=0x756b00 [EBookBackendMAPIContacts]) at e-book-backend-mapi.c:1036
#23 0x00007fffe6cac0b0 in thread_func_cb (data=0x7fffd8003720, pqueue=0x7fffe8004000) at e-mapi-operation-queue.c:144
#24 0x000000356f86c758 in g_thread_pool_thread_proxy (data=<optimized out>) at gthreadpool.c:319
#25 0x000000356f86a236 in g_thread_create_proxy (data=0x7fffd80032a0) at gthread.c:1962
#26 0x000000356dc07d31 in start_thread () from /lib64/libpthread.so.0
#27 0x000000356d8efdfd in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7fffeea35700 (LWP 9982)):
#0  0x000000356d8e7343 in poll () from /lib64/libc.so.6
#1  0x000000356f844fe8 in g_main_context_poll (n_fds=3, fds=0x7fffe8001150, priority=<optimized out>, timeout=-1, context=0x754660) at gmain.c:3402
#2  g_main_context_iterate (context=0x754660, block=<optimized out>, dispatch=1, self=<optimized out>) at gmain.c:3084
#3  0x000000356f845825 in g_main_loop_run (loop=0x75d160) at gmain.c:3297
#4  0x0000003571cbe626 in gdbus_shared_thread_func (user_data=0x75d180) at gdbusprivate.c:276
#5  0x000000356f86a236 in g_thread_create_proxy (data=0x75d1b0) at gthread.c:1962
#6  0x000000356dc07d31 in start_thread () from /lib64/libpthread.so.0
#7  0x000000356d8efdfd in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7fffef236700 (LWP 9981)):
#0  0x000000356dc0e5bd in read () from /lib64/libpthread.so.0
#1  0x000000356f840ba3 in read (__nbytes=128, __buf=0x7fffef235b70, __fd=<optimized out>) at /usr/include/bits/unistd.h:45
#2  unix_signal_helper_thread (data=<optimized out>) at gmain.c:4567
#3  0x000000356f86a236 in g_thread_create_proxy (data=0x749a60) at gthread.c:1962
#4  0x000000356dc07d31 in start_thread () from /lib64/libpthread.so.0
#5  0x000000356d8efdfd in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7ffff6d3a980 (LWP 9977)):
#0  0x000000356d8e7343 in poll () from /lib64/libc.so.6
#1  0x000000356f844fe8 in g_main_context_poll (n_fds=4, fds=0x76a570, priority=<optimized out>, timeout=-1, context=0x650590) at gmain.c:3402
#2  g_main_context_iterate (context=0x650590, block=<optimized out>, dispatch=1, self=<optimized out>) at gmain.c:3084
#3  0x000000356f845825 in g_main_loop_run (loop=0x649d00) at gmain.c:3297
#4  0x00007ffff75e6827 in e_dbus_server_run (server=0x74a010 [EDataBookFactory]) at e-dbus-server.c:248
#5  0x00000000004015e8 in main (argc=1, argv=0x7fffffffdeb8) at evolution-addressbook-factory.c:131
Comment 7 Jeremy Allison 2011-11-03 17:39:48 UTC
abartlett wrote:

"This is very unfortunate, as when we discussed talloc_stackframe(), it was not
clear to me that our library callers would need to be changed.  I would prefer
that users of the dcerpc API did not need to know about our internal threading
handling in order to use the API."

*WHAT* internal threading handling ???

These libraries are *NOT* thread safe right now, end of argument. We could just close this with a WONTFIX as it's using the libraries in a way that is has never been designed for.

The thread handling in talloc_stackframe() was added so if we decide to bite the bullet and make everything thread safe that talloc_stackframe() was already done. I was testing the water to see how hard this would be so to speak.

The only fix for this is for the caller to mutex around the calls into this library. It isn't thread safe. AT ALL.

Jeremy.
Comment 8 Andrew Bartlett 2011-11-03 20:41:48 UTC
We have two distinct bugs here.  Please open a separate bug about the iconv issue, as I think this can be handled by ensuring that we open a new iconv handle when we cannot be sure that the handle is private to this operation (which for efficiency could be a whole incoming RPC, if we put the handle on the ndr structures). 

The Samba4 client libraries (such as the ones you are using) were intended to be thread safe, as long as no context pointers passed in were common between those threads.  Sadly we have slipped backwards on this however, and have not had tests for this.
Comment 9 Milan Crha 2011-11-04 08:09:52 UTC
(In reply to comment #8)
> We have two distinct bugs here.  Please open a separate bug about the iconv
> issue, as I think this can be handled by ensuring that we open a new iconv
> handle when we cannot be sure that the handle is private to this operation
> (which for efficiency could be a whole incoming RPC, if we put the handle on
> the ndr structures). 

Here you are, bug #8572

> The Samba4 client libraries (such as the ones you are using) were intended to
> be thread safe, as long as no context pointers passed in were common between
> those threads.  Sadly we have slipped backwards on this however, and have not
> had tests for this.

Thanks. It would not make any sense to have RPC calls processing in one thread only, on two totally independent and separated connections. I create connections on separated talloc memory contexts, if you mean these contexts. Or, maybe, your 'context' is what I call 'connection' here? You know, there are too many contexts in samba4.