Bug 14413 - ldapi search to FreeIPA crashes.
Summary: ldapi search to FreeIPA crashes.
Status: RESOLVED FIXED
Alias: None
Product: Samba 4.1 and newer
Classification: Unclassified
Component: AD: LDB/DSDB/SAMDB (show other bugs)
Version: 4.12.2
Hardware: All All
: P5 normal (vote)
Target Milestone: ---
Assignee: Karolin Seeger
QA Contact: Samba QA Contact
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-06-17 20:26 UTC by Jeremy Allison
Modified: 2020-06-25 07:20 UTC (History)
3 users (show)

See Also:


Attachments
backport to v4.12 and v4.11 (5.76 KB, patch)
2020-06-19 10:26 UTC, Alexander Bokovoy
asn: review+
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Jeremy Allison 2020-06-17 20:26:19 UTC
Reported by ab@samba.org on the samba-technical mailing list.

 From the mail:

https://lists.samba.org/archive/samba-technical/2020-June/135443.html

Right now libldb does not handle LDAPI connection to an LDAP server like
389-ds (openldap libraries do support that without any problem) on
Fedora 32. In fact, that doesn't work for any LDAP/LDAPI url for non-AD
LDAP compatible LDAP server because ldb doesn't have checks for missing
attributes and crashes inside its initialization code.

ldbsearch on Fedora 32 with Samba 4.12, ldb 2.1.3:

# ldbsearch -H ldapi://%2Fvar%2Frun%2Fslapd-IPA-TEST.socket
Segmentation fault (core dumped)

# coredumpctl debug
           PID: 14312 (ldbsearch)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 11 (SEGV)
     Timestamp: Tue 2020-06-16 08:03:57 UTC (8min ago)
  Command Line: ldbsearch -H ldapi://%2Fvar%2Frun%2Fslapd-IPA-TEST.socket
    Executable: /usr/bin/ldbsearch
 Control Group: /user.slice/user-0.slice/session-1.scope
          Unit: session-1.scope
         Slice: user-0.slice
       Session: 1
     Owner UID: 0 (root)
       Boot ID: 30c1480b4a66445695260249204cdcee
    Machine ID: 23515da731974563b0dbd8618404271c
      Hostname: master.ipa.test
       Storage: /var/lib/systemd/coredump/core.ldbsearch.0.30c1480b4a66445695260249204cdcee.14312.1592294637000000000000.lz4
       Message: Process 14312 (ldbsearch) of user 0 dumped core.
                
                Stack trace of thread 14312:
                #0  0x00007f5b62a46f39 __strcasecmp_l_avx (libc.so.6 + 0x15ef39)
                #1  0x00007f5b62b2215e ldb_msg_find_element (libldb.so.2 + 0x1115e)
                #2  0x00007f5b62b22dbd ldb_msg_find_ldb_val (libldb.so.2 + 0x11dbd)
                #3  0x00007f5b62b2331a ldb_msg_find_attr_as_dn (libldb.so.2 + 0x1231a)
                #4  0x00007f5b62b36a7d ldb_set_default_dns (libldb.so.2 + 0x25a7d)
                #5  0x00007f5b62b36bfe ldb_connect (libldb.so.2 + 0x25bfe)
                #6  0x00007f5b62b0bef4 ldb_cmdline_process_internal (libldb-cmdline.so + 0x2ef4)
                #7  0x000055e2ae72e484 main (ldbsearch + 0x1484)
                #8  0x00007f5b6290f042 __libc_start_main (libc.so.6 + 0x27042)
                #9  0x000055e2ae72e62e _start (ldbsearch + 0x162e)

GNU gdb (GDB) Fedora 9.1-5.fc32
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/ldbsearch...
Reading symbols from /usr/lib/debug/usr/bin/ldbsearch-2.1.3-1.fc32.x86_64.debug...
[New LWP 14312]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `ldbsearch -H ldapi://%2Fvar%2Frun%2Fslapd-IPA-TEST.socket'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  __strcasecmp_l_avx () at ../sysdeps/x86_64/multiarch/strcmp-sse42.S:199
199		movdqu	(%rdi), %xmm1
(gdb) bt full
#0  __strcasecmp_l_avx () at ../sysdeps/x86_64/multiarch/strcmp-sse42.S:199
No locals.
#1  0x00007f5b62b2215e in ldb_msg_find_element (msg=<optimized out>, attr_name=attr_name at entry=0x7f5b62b3d74f "rootDomainNamingContext") at ../../common/ldb_msg.c:52
        i = <optimized out>
#2  0x00007f5b62b22dbd in ldb_msg_find_ldb_val (msg=<optimized out>, attr_name=attr_name at entry=0x7f5b62b3d74f "rootDomainNamingContext") at ../../common/ldb_msg.c:612
        el = <optimized out>
#3  0x00007f5b62b2331a in ldb_msg_find_attr_as_dn (ldb=ldb at entry=0x55e2af0343b0, mem_ctx=mem_ctx at entry=0x55e2af0343b0, msg=<optimized out>, 
    attr_name=attr_name at entry=0x7f5b62b3d74f "rootDomainNamingContext") at ../../common/ldb_msg.c:815
        res_dn = <optimized out>
        v = <optimized out>
#4  0x00007f5b62b36a7d in ldb_set_default_dns (ldb=ldb at entry=0x55e2af0343b0) at ../../common/ldb.c:180
        tmp_ctx = 0x55e2af03c470
        ret = <optimized out>
        res = 0x55e2af03c550
        tmp_dn = 0x0
        attrs = {0x7f5b62b3d74f "rootDomainNamingContext", 0x7f5b62b3d767 "configurationNamingContext", 0x7f5b62b3d782 "schemaNamingContext", 0x7f5b62b3d796 "defaultNamingContext", 0x0}
#5  0x00007f5b62b36bfe in ldb_connect (ldb=ldb at entry=0x55e2af0343b0, url=0x55e2af034d90 "ldapi://%2Fvar%2Frun%2Fslapd-IPA-TEST.socket", flags=flags at entry=64, options=0x0)
    at ../../common/ldb.c:283
        ret = 0
        url2 = <optimized out>
#6  0x00007f5b62b0bef4 in ldb_cmdline_process_internal (ldb=0x55e2af0343b0, argc=<optimized out>, argv=<optimized out>, usage=0x55e2ae72e6f0 <usage>, search=<optimized out>)
    at ../../tools/cmdline.c:305
        ret = 0x55e2af034920
        pc = 0x55e2af0349a0
        num_options = 0
        opt = <optimized out>
        flags = 64
        rc = <optimized out>
        popt_options = <optimized out>
#7  0x000055e2ae72e484 in main (argc=3, argv=0x7ffd32e24568) at ../../tools/ldbsearch.c:307
        ldb = 0x55e2af0343b0
        basedn = 0x0
        attrs = 0x0
        options = <optimized out>
        ret = -1
        expression = 0x55e2ae72f120 "(|(objectClass=*)(distinguishedName=*))"
        mem_ctx = 0x55e2af021300

The crash actually happens in a basic initialize part of ldb and is
exactly same if I specify an LDAP url too:

# ldbsearch -H ldap://`hostname`
Segmentation fault (core dumped)

In the LDAP server log I can see the searches:

LDAPI search:
[16/Jun/2020:08:13:27.110825313 +0000] conn=139 fd=79 slot=79 connection from local to /var/run/slapd-IPA-TEST.socket
[16/Jun/2020:08:13:27.112187352 +0000] conn=139 op=0 SRCH base="" scope=0 filter="(objectClass=*)" attrs="rootdomainnamingcontext configurationnamingcontext schemanamingcontext defaultnamingcontext"
[16/Jun/2020:08:13:27.114176714 +0000] conn=139 op=0 RESULT err=0 tag=101 nentries=1 etime=0.002617861
[16/Jun/2020:08:13:27.525247748 +0000] conn=139 op=-1 fd=79 closed - B1

LDAP search:
[16/Jun/2020:08:15:28.407027866 +0000] conn=140 fd=79 slot=79 connection from IP.AD.DR.ES to IP.AD.DR.ES
[16/Jun/2020:08:15:28.408473254 +0000] conn=140 op=0 SRCH base="" scope=0 filter="(objectClass=*)" attrs="rootdomainnamingcontext configurationnamingcontext schemanamingcontext defaultnamingcontext"
[16/Jun/2020:08:15:28.410683616 +0000] conn=140 op=0 RESULT err=0 tag=101 nentries=1 etime=0.003439788
[16/Jun/2020:08:15:28.787188218 +0000] conn=140 op=-1 fd=79 closed - B1
Comment 1 Alexander Bokovoy 2020-06-18 07:20:23 UTC
(gdb) print *(struct ldb_message *) 0x55c055c979f0
$15 = {dn = 0x55c055c97a90, num_elements = 2, elements = 0x55c055c97c70}
(gdb) print ((struct ldb_message *) 0x55c055c979f0)->elements[0]
$16 = {flags = 0, name = 0x55c055c97d00 "defaultNamingContext", num_values = 1, values = 0x55c055c97d80}
(gdb) print ((struct ldb_message *) 0x55c055c979f0)->elements[1]
$17 = {flags = 0, name = 0x81 <error: Cannot access memory at address 0x81>, num_values = 2308730688, values = 0x0}
Comment 2 Alexander Bokovoy 2020-06-18 07:20:59 UTC
Somehow, num_elements is set to 2 while there is only one element in the list.
Comment 3 Alexander Bokovoy 2020-06-18 07:25:58 UTC
From 389-ds access log when performing

[root@master ~]# ldbsearch -v -H ldapi://%2Fvar%2Frun%2Fslapd-IPA-TEST.socket
Segmentation fault (core dumped)


[18/Jun/2020:07:22:43.772271855 +0000] conn=333 fd=95 slot=95 connection from local to /var/run/slapd-IPA-TEST.socket
[18/Jun/2020:07:22:43.772607823 +0000] conn=333 op=0 SRCH base="" scope=0 filter="(objectClass=*)" attrs="rootdomainnamingcontext configurationnamingcontext schemanamingcontext defaultnamingcontext"
[18/Jun/2020:07:22:43.773930828 +0000] conn=333 op=0 RESULT err=0 tag=101 nentries=1 etime=0.001574250
[18/Jun/2020:07:22:44.160966346 +0000] conn=333 op=-1 fd=95 closed - B1

We can see there is one entry returned and if I'd do the same search with ldapsearch:

[root@master ~]# ldapsearch -Y EXTERNAL -H ldapi://%2Fvar%2Frun%2Fslapd-IPA-TEST.socket -b '' -s base rootdomainnamingcontext configurationnamingcontext schemanamingcontext defaultnamingcontext
SASL/EXTERNAL authentication started
SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
SASL SSF: 0
# extended LDIF
#
# LDAPv3
# base <> with scope baseObject
# filter: (objectclass=*)
# requesting: rootdomainnamingcontext configurationnamingcontext schemanamingcontext defaultnamingcontext 
#

#
dn:
defaultnamingcontext: dc=ipa,dc=test

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1
Comment 4 Alexander Bokovoy 2020-06-18 07:44:07 UTC
Breakpoint 1, lldb_parse_result (result=0x5555555783d0, ac=0x5555555746d0) at ../../ldb_ldap/ldb_ldap.c:526
526				ldap_memfree(dn);
(gdb) n
528				for (attr=ldap_first_attribute(lldb->ldap, msg, &berptr);
(gdb) n
529				     attr;
(gdb) n
532					bval = ldap_get_values_len(lldb->ldap, msg, attr);
(gdb) n
535						lldb_add_msg_attr(ldb, ldbmsg, attr, bval);
(gdb) print *ldbmsg
$4 = {dn = 0x555555578510, num_elements = 0, elements = 0x0}
(gdb) n
536						ldap_value_free_len(bval);
(gdb) n
530				     attr=ldap_next_attribute(lldb->ldap, msg, berptr)) {
(gdb) print *ldbmsg
$5 = {dn = 0x555555578510, num_elements = 2, elements = 0x5555555786f0}
(gdb) 

So after adding a single attribute with lldb_add_msg_attr() we have two elements while there was only one.
If I'd a breakpoint in lldb_add_msg_attr() it is clear that offending line is the last one in that function:

        msg->num_elements++;

because 

        ret = ldb_msg_add_empty(msg, attr, 0, &el);

already increased num_elements by one.

Breakpoint 1, lldb_add_msg_attr (ldb=0x55555556c3b0, bval=0x555555578630, attr=0x555555578490 "defaultNamingContext", msg=0x555555578470) at ../../ldb_ldap/ldb_ldap.c:535
535						lldb_add_msg_attr(ldb, ldbmsg, attr, bval);
Missing separate debuginfos, use: dnf debuginfo-install lmdb-libs-0.9.24-1.fc32.x86_64 sssd-common-2.3.0-1.fc32.x86_64
(gdb) bt full
#0  lldb_add_msg_attr (ldb=0x55555556c3b0, bval=0x555555578630, attr=0x555555578490 "defaultNamingContext", msg=0x555555578470) at ../../ldb_ldap/ldb_ldap.c:535
        count = <optimized out>
        i = <optimized out>
        ret = <optimized out>
        el = 0x7ffff7f34070
        count = <optimized out>
        i = <optimized out>
        ret = <optimized out>
        el = <optimized out>
#1  lldb_parse_result (result=0x5555555783d0, ac=0x5555555746d0) at ../../ldb_ldap/ldb_ldap.c:535
        bval = 0x555555578630
        berptr = 0x5555555785d0
        attr = 0x555555578490 "defaultNamingContext"
        dn = <optimized out>
        msg = <optimized out>
        i = <optimized out>
        ret = -135053200
        ldb = 0x55555556c3b0
        referral = <optimized out>
        lret = <optimized out>
        lldb = 0x555555559490
        referralsp = 0x0
        matcheddnp = 0x0
        type = <optimized out>
        callback_failed = false
        request_done = false
        serverctrlsp = 0x0
        errmsgp = 0x0
        ldbmsg = 0x555555578470
        ldb = <optimized out>
        lldb = <optimized out>
        serverctrlsp = <optimized out>
        referralsp = <optimized out>
        matcheddnp = <optimized out>
        errmsgp = <optimized out>
        msg = <optimized out>
        type = <optimized out>
        ldbmsg = <optimized out>
        referral = <optimized out>
        callback_failed = <optimized out>
(gdb) print *msg
$1 = {dn = 0x555555578510, num_elements = 0, elements = 0x0}
(gdb) n
149		if (count <= 0) {
(gdb) print count
$2 = 1
(gdb) n
153		ret = ldb_msg_add_empty(msg, attr, 0, &el);
(gdb) n
154		if (ret != LDB_SUCCESS) {
(gdb) print *msg
$3 = {dn = 0x555555578510, num_elements = 1, elements = 0x5555555786f0}
(gdb) n
159		el->values = talloc_array(msg->elements, struct ldb_val, count);
(gdb) n
160		if (!el->values) {
(gdb) n
168			el->values[i].data = talloc_size(el->values, bval[i]->bv_len+1);
(gdb) n
169			if (!el->values[i].data) {
(gdb) n
173			memcpy(el->values[i].data, bval[i]->bv_val, bval[i]->bv_len);
(gdb) n
34	  return __builtin___memcpy_chk (__dest, __src, __len, __bos0 (__dest));
(gdb) n
174			el->values[i].data[bval[i]->bv_len] = 0;
(gdb) n
175			el->values[i].length = bval[i]->bv_len;
(gdb) n
176			el->num_values++;
(gdb) n
165		for (i=0;i<count;i++) {
(gdb) n
179		msg->num_elements++;
(gdb) n
lldb_parse_result (result=0x5555555783d0, ac=0x5555555746d0) at ../../ldb_ldap/ldb_ldap.c:536
536						ldap_value_free_len(bval);
(gdb) print *ldbmsg
$4 = {dn = 0x555555578510, num_elements = 2, elements = 0x5555555786f0}
(gdb)
Comment 5 Alexander Bokovoy 2020-06-18 07:57:26 UTC
This happens because ldb_msg_add_empty() calls _ldb_msg_add_el() which already increments msg->num_elements by one.

This is a regression introduced with 

commit ce2bf5c72b6423fff680b3d6a9042103a6cdda55
Author: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Date:   Tue Apr 9 20:29:59 2019 +1200

    ldb_ldap: use ldb_msg API to add elements
    
    Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
    Reviewed-by: Andrew Bartlett <abartlet@samba.org>



MR: https://gitlab.com/samba-team/samba/-/merge_requests/1404
Comment 6 Jeremy Allison 2020-06-18 17:47:36 UTC
Great debugging Alexander, thanks a *lot* for looking into this !
Comment 7 Alexander Bokovoy 2020-06-19 10:26:28 UTC
Created attachment 16061 [details]
backport to v4.12 and v4.11

Fix is in master now. Attaching backports.
Comment 8 Andreas Schneider 2020-06-19 13:45:49 UTC
Comment on attachment 16061 [details]
backport to v4.12 and v4.11

LGTM
Comment 9 Andreas Schneider 2020-06-19 13:46:18 UTC
Karolin, could you please apply the patchset to the relevant branches? Thanks.
Comment 10 Karolin Seeger 2020-06-24 09:57:54 UTC
(In reply to Andreas Schneider from comment #9)
Pushed to autobuild-v4-{11,12}-test.
Comment 11 Karolin Seeger 2020-06-25 07:20:11 UTC
(In reply to Karolin Seeger from comment #10)
Pushed to both branches.
Closing out bug report.

Thanks!