Reported by ab@samba.org on the samba-technical mailing list. From the mail: https://lists.samba.org/archive/samba-technical/2020-June/135443.html Right now libldb does not handle LDAPI connection to an LDAP server like 389-ds (openldap libraries do support that without any problem) on Fedora 32. In fact, that doesn't work for any LDAP/LDAPI url for non-AD LDAP compatible LDAP server because ldb doesn't have checks for missing attributes and crashes inside its initialization code. ldbsearch on Fedora 32 with Samba 4.12, ldb 2.1.3: # ldbsearch -H ldapi://%2Fvar%2Frun%2Fslapd-IPA-TEST.socket Segmentation fault (core dumped) # coredumpctl debug PID: 14312 (ldbsearch) UID: 0 (root) GID: 0 (root) Signal: 11 (SEGV) Timestamp: Tue 2020-06-16 08:03:57 UTC (8min ago) Command Line: ldbsearch -H ldapi://%2Fvar%2Frun%2Fslapd-IPA-TEST.socket Executable: /usr/bin/ldbsearch Control Group: /user.slice/user-0.slice/session-1.scope Unit: session-1.scope Slice: user-0.slice Session: 1 Owner UID: 0 (root) Boot ID: 30c1480b4a66445695260249204cdcee Machine ID: 23515da731974563b0dbd8618404271c Hostname: master.ipa.test Storage: /var/lib/systemd/coredump/core.ldbsearch.0.30c1480b4a66445695260249204cdcee.14312.1592294637000000000000.lz4 Message: Process 14312 (ldbsearch) of user 0 dumped core. Stack trace of thread 14312: #0 0x00007f5b62a46f39 __strcasecmp_l_avx (libc.so.6 + 0x15ef39) #1 0x00007f5b62b2215e ldb_msg_find_element (libldb.so.2 + 0x1115e) #2 0x00007f5b62b22dbd ldb_msg_find_ldb_val (libldb.so.2 + 0x11dbd) #3 0x00007f5b62b2331a ldb_msg_find_attr_as_dn (libldb.so.2 + 0x1231a) #4 0x00007f5b62b36a7d ldb_set_default_dns (libldb.so.2 + 0x25a7d) #5 0x00007f5b62b36bfe ldb_connect (libldb.so.2 + 0x25bfe) #6 0x00007f5b62b0bef4 ldb_cmdline_process_internal (libldb-cmdline.so + 0x2ef4) #7 0x000055e2ae72e484 main (ldbsearch + 0x1484) #8 0x00007f5b6290f042 __libc_start_main (libc.so.6 + 0x27042) #9 0x000055e2ae72e62e _start (ldbsearch + 0x162e) GNU gdb (GDB) Fedora 9.1-5.fc32 Copyright (C) 2020 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/bin/ldbsearch... Reading symbols from /usr/lib/debug/usr/bin/ldbsearch-2.1.3-1.fc32.x86_64.debug... [New LWP 14312] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `ldbsearch -H ldapi://%2Fvar%2Frun%2Fslapd-IPA-TEST.socket'. Program terminated with signal SIGSEGV, Segmentation fault. #0 __strcasecmp_l_avx () at ../sysdeps/x86_64/multiarch/strcmp-sse42.S:199 199 movdqu (%rdi), %xmm1 (gdb) bt full #0 __strcasecmp_l_avx () at ../sysdeps/x86_64/multiarch/strcmp-sse42.S:199 No locals. #1 0x00007f5b62b2215e in ldb_msg_find_element (msg=<optimized out>, attr_name=attr_name at entry=0x7f5b62b3d74f "rootDomainNamingContext") at ../../common/ldb_msg.c:52 i = <optimized out> #2 0x00007f5b62b22dbd in ldb_msg_find_ldb_val (msg=<optimized out>, attr_name=attr_name at entry=0x7f5b62b3d74f "rootDomainNamingContext") at ../../common/ldb_msg.c:612 el = <optimized out> #3 0x00007f5b62b2331a in ldb_msg_find_attr_as_dn (ldb=ldb at entry=0x55e2af0343b0, mem_ctx=mem_ctx at entry=0x55e2af0343b0, msg=<optimized out>, attr_name=attr_name at entry=0x7f5b62b3d74f "rootDomainNamingContext") at ../../common/ldb_msg.c:815 res_dn = <optimized out> v = <optimized out> #4 0x00007f5b62b36a7d in ldb_set_default_dns (ldb=ldb at entry=0x55e2af0343b0) at ../../common/ldb.c:180 tmp_ctx = 0x55e2af03c470 ret = <optimized out> res = 0x55e2af03c550 tmp_dn = 0x0 attrs = {0x7f5b62b3d74f "rootDomainNamingContext", 0x7f5b62b3d767 "configurationNamingContext", 0x7f5b62b3d782 "schemaNamingContext", 0x7f5b62b3d796 "defaultNamingContext", 0x0} #5 0x00007f5b62b36bfe in ldb_connect (ldb=ldb at entry=0x55e2af0343b0, url=0x55e2af034d90 "ldapi://%2Fvar%2Frun%2Fslapd-IPA-TEST.socket", flags=flags at entry=64, options=0x0) at ../../common/ldb.c:283 ret = 0 url2 = <optimized out> #6 0x00007f5b62b0bef4 in ldb_cmdline_process_internal (ldb=0x55e2af0343b0, argc=<optimized out>, argv=<optimized out>, usage=0x55e2ae72e6f0 <usage>, search=<optimized out>) at ../../tools/cmdline.c:305 ret = 0x55e2af034920 pc = 0x55e2af0349a0 num_options = 0 opt = <optimized out> flags = 64 rc = <optimized out> popt_options = <optimized out> #7 0x000055e2ae72e484 in main (argc=3, argv=0x7ffd32e24568) at ../../tools/ldbsearch.c:307 ldb = 0x55e2af0343b0 basedn = 0x0 attrs = 0x0 options = <optimized out> ret = -1 expression = 0x55e2ae72f120 "(|(objectClass=*)(distinguishedName=*))" mem_ctx = 0x55e2af021300 The crash actually happens in a basic initialize part of ldb and is exactly same if I specify an LDAP url too: # ldbsearch -H ldap://`hostname` Segmentation fault (core dumped) In the LDAP server log I can see the searches: LDAPI search: [16/Jun/2020:08:13:27.110825313 +0000] conn=139 fd=79 slot=79 connection from local to /var/run/slapd-IPA-TEST.socket [16/Jun/2020:08:13:27.112187352 +0000] conn=139 op=0 SRCH base="" scope=0 filter="(objectClass=*)" attrs="rootdomainnamingcontext configurationnamingcontext schemanamingcontext defaultnamingcontext" [16/Jun/2020:08:13:27.114176714 +0000] conn=139 op=0 RESULT err=0 tag=101 nentries=1 etime=0.002617861 [16/Jun/2020:08:13:27.525247748 +0000] conn=139 op=-1 fd=79 closed - B1 LDAP search: [16/Jun/2020:08:15:28.407027866 +0000] conn=140 fd=79 slot=79 connection from IP.AD.DR.ES to IP.AD.DR.ES [16/Jun/2020:08:15:28.408473254 +0000] conn=140 op=0 SRCH base="" scope=0 filter="(objectClass=*)" attrs="rootdomainnamingcontext configurationnamingcontext schemanamingcontext defaultnamingcontext" [16/Jun/2020:08:15:28.410683616 +0000] conn=140 op=0 RESULT err=0 tag=101 nentries=1 etime=0.003439788 [16/Jun/2020:08:15:28.787188218 +0000] conn=140 op=-1 fd=79 closed - B1
(gdb) print *(struct ldb_message *) 0x55c055c979f0 $15 = {dn = 0x55c055c97a90, num_elements = 2, elements = 0x55c055c97c70} (gdb) print ((struct ldb_message *) 0x55c055c979f0)->elements[0] $16 = {flags = 0, name = 0x55c055c97d00 "defaultNamingContext", num_values = 1, values = 0x55c055c97d80} (gdb) print ((struct ldb_message *) 0x55c055c979f0)->elements[1] $17 = {flags = 0, name = 0x81 <error: Cannot access memory at address 0x81>, num_values = 2308730688, values = 0x0}
Somehow, num_elements is set to 2 while there is only one element in the list.
From 389-ds access log when performing [root@master ~]# ldbsearch -v -H ldapi://%2Fvar%2Frun%2Fslapd-IPA-TEST.socket Segmentation fault (core dumped) [18/Jun/2020:07:22:43.772271855 +0000] conn=333 fd=95 slot=95 connection from local to /var/run/slapd-IPA-TEST.socket [18/Jun/2020:07:22:43.772607823 +0000] conn=333 op=0 SRCH base="" scope=0 filter="(objectClass=*)" attrs="rootdomainnamingcontext configurationnamingcontext schemanamingcontext defaultnamingcontext" [18/Jun/2020:07:22:43.773930828 +0000] conn=333 op=0 RESULT err=0 tag=101 nentries=1 etime=0.001574250 [18/Jun/2020:07:22:44.160966346 +0000] conn=333 op=-1 fd=95 closed - B1 We can see there is one entry returned and if I'd do the same search with ldapsearch: [root@master ~]# ldapsearch -Y EXTERNAL -H ldapi://%2Fvar%2Frun%2Fslapd-IPA-TEST.socket -b '' -s base rootdomainnamingcontext configurationnamingcontext schemanamingcontext defaultnamingcontext SASL/EXTERNAL authentication started SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth SASL SSF: 0 # extended LDIF # # LDAPv3 # base <> with scope baseObject # filter: (objectclass=*) # requesting: rootdomainnamingcontext configurationnamingcontext schemanamingcontext defaultnamingcontext # # dn: defaultnamingcontext: dc=ipa,dc=test # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1
Breakpoint 1, lldb_parse_result (result=0x5555555783d0, ac=0x5555555746d0) at ../../ldb_ldap/ldb_ldap.c:526 526 ldap_memfree(dn); (gdb) n 528 for (attr=ldap_first_attribute(lldb->ldap, msg, &berptr); (gdb) n 529 attr; (gdb) n 532 bval = ldap_get_values_len(lldb->ldap, msg, attr); (gdb) n 535 lldb_add_msg_attr(ldb, ldbmsg, attr, bval); (gdb) print *ldbmsg $4 = {dn = 0x555555578510, num_elements = 0, elements = 0x0} (gdb) n 536 ldap_value_free_len(bval); (gdb) n 530 attr=ldap_next_attribute(lldb->ldap, msg, berptr)) { (gdb) print *ldbmsg $5 = {dn = 0x555555578510, num_elements = 2, elements = 0x5555555786f0} (gdb) So after adding a single attribute with lldb_add_msg_attr() we have two elements while there was only one. If I'd a breakpoint in lldb_add_msg_attr() it is clear that offending line is the last one in that function: msg->num_elements++; because ret = ldb_msg_add_empty(msg, attr, 0, &el); already increased num_elements by one. Breakpoint 1, lldb_add_msg_attr (ldb=0x55555556c3b0, bval=0x555555578630, attr=0x555555578490 "defaultNamingContext", msg=0x555555578470) at ../../ldb_ldap/ldb_ldap.c:535 535 lldb_add_msg_attr(ldb, ldbmsg, attr, bval); Missing separate debuginfos, use: dnf debuginfo-install lmdb-libs-0.9.24-1.fc32.x86_64 sssd-common-2.3.0-1.fc32.x86_64 (gdb) bt full #0 lldb_add_msg_attr (ldb=0x55555556c3b0, bval=0x555555578630, attr=0x555555578490 "defaultNamingContext", msg=0x555555578470) at ../../ldb_ldap/ldb_ldap.c:535 count = <optimized out> i = <optimized out> ret = <optimized out> el = 0x7ffff7f34070 count = <optimized out> i = <optimized out> ret = <optimized out> el = <optimized out> #1 lldb_parse_result (result=0x5555555783d0, ac=0x5555555746d0) at ../../ldb_ldap/ldb_ldap.c:535 bval = 0x555555578630 berptr = 0x5555555785d0 attr = 0x555555578490 "defaultNamingContext" dn = <optimized out> msg = <optimized out> i = <optimized out> ret = -135053200 ldb = 0x55555556c3b0 referral = <optimized out> lret = <optimized out> lldb = 0x555555559490 referralsp = 0x0 matcheddnp = 0x0 type = <optimized out> callback_failed = false request_done = false serverctrlsp = 0x0 errmsgp = 0x0 ldbmsg = 0x555555578470 ldb = <optimized out> lldb = <optimized out> serverctrlsp = <optimized out> referralsp = <optimized out> matcheddnp = <optimized out> errmsgp = <optimized out> msg = <optimized out> type = <optimized out> ldbmsg = <optimized out> referral = <optimized out> callback_failed = <optimized out> (gdb) print *msg $1 = {dn = 0x555555578510, num_elements = 0, elements = 0x0} (gdb) n 149 if (count <= 0) { (gdb) print count $2 = 1 (gdb) n 153 ret = ldb_msg_add_empty(msg, attr, 0, &el); (gdb) n 154 if (ret != LDB_SUCCESS) { (gdb) print *msg $3 = {dn = 0x555555578510, num_elements = 1, elements = 0x5555555786f0} (gdb) n 159 el->values = talloc_array(msg->elements, struct ldb_val, count); (gdb) n 160 if (!el->values) { (gdb) n 168 el->values[i].data = talloc_size(el->values, bval[i]->bv_len+1); (gdb) n 169 if (!el->values[i].data) { (gdb) n 173 memcpy(el->values[i].data, bval[i]->bv_val, bval[i]->bv_len); (gdb) n 34 return __builtin___memcpy_chk (__dest, __src, __len, __bos0 (__dest)); (gdb) n 174 el->values[i].data[bval[i]->bv_len] = 0; (gdb) n 175 el->values[i].length = bval[i]->bv_len; (gdb) n 176 el->num_values++; (gdb) n 165 for (i=0;i<count;i++) { (gdb) n 179 msg->num_elements++; (gdb) n lldb_parse_result (result=0x5555555783d0, ac=0x5555555746d0) at ../../ldb_ldap/ldb_ldap.c:536 536 ldap_value_free_len(bval); (gdb) print *ldbmsg $4 = {dn = 0x555555578510, num_elements = 2, elements = 0x5555555786f0} (gdb)
This happens because ldb_msg_add_empty() calls _ldb_msg_add_el() which already increments msg->num_elements by one. This is a regression introduced with commit ce2bf5c72b6423fff680b3d6a9042103a6cdda55 Author: Douglas Bagnall <douglas.bagnall@catalyst.net.nz> Date: Tue Apr 9 20:29:59 2019 +1200 ldb_ldap: use ldb_msg API to add elements Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz> Reviewed-by: Andrew Bartlett <abartlet@samba.org> MR: https://gitlab.com/samba-team/samba/-/merge_requests/1404
Great debugging Alexander, thanks a *lot* for looking into this !
Created attachment 16061 [details] backport to v4.12 and v4.11 Fix is in master now. Attaching backports.
Comment on attachment 16061 [details] backport to v4.12 and v4.11 LGTM
Karolin, could you please apply the patchset to the relevant branches? Thanks.
(In reply to Andreas Schneider from comment #9) Pushed to autobuild-v4-{11,12}-test.
(In reply to Karolin Seeger from comment #10) Pushed to both branches. Closing out bug report. Thanks!