winbindd dies with a SEGV when running "wbinfo -u" to retrieve the domain user list. Please note that the domain contains _lots_ of entries. The domain controller is beyond my control. # net rpc info Domain Name: WORK Domain SID: S-1-5-21-1454472166-527247240-691004440 Sequence number: 1 Num users: 48368 Num domain groups: 628078 Num local groups: 46710 Reproducible: always Steps to reproduce: Follow Samba Howto Chapter 2 "Fast Start: Domain Member Server": Step 8 will fail. OS: opensuse 10.1/x86 using the 3.0.25a-8.1.63 RPMs from http://ftp.suse.com//pub/projects/samba/3.0/10.1/i386 Note: also failed on SLES 10 with 3.0.22-13.30 RPMs. Relevant(?) smb.conf settings (tell me if full more is needed): [global] security = domain password server = * netbios name = server3 workgroup = WORK wins server = 192.168.5.1 os level = 0 domain master = no local master = no preferred master = no dns proxy = no winbind separator = / winbind use default domain = yes idmap uid = 1000000-7000000 idmap gid = 1000000-7000000 winbind enum users = yes winbind enum groups = yes The winbindd.log shows: From: http://www.samba.org/samba/docs/Samba3-HOWTO.pdf [2007/06/14 12:39:30, 0] lib/fault.c:fault_report(45) =============================================================== [2007/06/14 12:39:30, 0] lib/util.c:smb_panic(1632) PANIC (pid 29485): internal error [2007/06/14 12:39:30, 0] lib/util.c:log_stack_trace(1736) BACKTRACE: 26 stack frames: #0 /usr/sbin/winbindd(log_stack_trace+0x2d) [0x800d288d] #1 /usr/sbin/winbindd(smb_panic+0x5d) [0x800d29bd] #2 /usr/sbin/winbindd [0x800bda1a] #3 [0xffffe420] #4 /usr/lib/libkrb5.so.3(krb5_free_principal+0x63) [0xb7e70e33] #5 /usr/lib/libkrb5.so.3(krb5_free_cred_contents+0x2d) [0xb7e7215d] #6 /usr/lib/libkrb5.so.3(krb5_free_creds+0x29) [0xb7e72249] #7 /usr/lib/libkrb5.so.3(krb5_free_tgt_creds+0x2e) [0xb7e7228e] #8 /usr/lib/libkrb5.so.3(krb5_get_credentials+0x1dc) [0xb7e6cc9c] #9 /usr/sbin/winbindd(cli_krb5_get_ticket+0x4b9) [0x800fe1d9] #10 /usr/sbin/winbindd(spnego_gen_negTokenTarg+0x69) [0x800ff229] #11 /usr/sbin/winbindd [0x801ea5a0] #12 /usr/sbin/winbindd [0x801eafd1] #13 /usr/sbin/winbindd(ads_sasl_bind+0x13a) [0x801e9bca] #14 /usr/sbin/winbindd(ads_connect+0x84f) [0x801e86af] #15 /usr/sbin/winbindd [0x800664fc] #16 /usr/sbin/winbindd [0x80066792] #17 /usr/sbin/winbindd [0x8004c24e] #18 /usr/sbin/winbindd [0x8004c929] #19 /usr/sbin/winbindd [0x8004eb58] #20 /usr/sbin/winbindd(winbindd_list_users+0x107) [0x80040ce7] #21 /usr/sbin/winbindd [0x8003f3d7] #22 /usr/sbin/winbindd [0x8003f858] #23 /usr/sbin/winbindd(main+0x92d) [0x8004028d] #24 /lib/libc.so.6(__libc_start_main+0xdc) [0xb7cb587c] #25 /usr/sbin/winbindd [0x8003e5e1] [2007/06/14 12:39:30, 0] lib/fault.c:dump_core(181) dumping core in /var/log/samba/cores/winbindd
Created attachment 2756 [details] winbindd core file from the initial bug report
Lars, Are there any non-standard patches patches applied to those 3.0.25a RPMs?
The RPM spec files and other files to build are available from http://ftp.suse.com//pub/projects/samba/3.0/src/ There is no spec for 10.1, though. I guess they used samba.spec-10.0 for 10.1 too.
http://ftp.suse.com//pub/projects/samba/README explains in detail which spec file is used to create which packages (see section 'Spec files and SUSE abbreviations'). In addition it also includes a pointer to http://en.openSUSE.org/Samba Jerry: The packages include some samba.org revisions but none for winbindd. Walter: Are you able to install the debuginfo packages from the same location and run winbindd with gdb? In the upcoming week some changes from the current Samba development tree will be added to the 3.0.25a packages to address large domain environments.
There are no debuginfo packages in http://ftp.suse.com//pub/projects/samba/3.0/10.1/i386/ Therefore I've built samba 3.0.25a from plain source (./configure --enable-debug && make). Configure and build log is available as well as installed RPM list required to build from source. As expected, no changes. Winbindd still SEGVs, see below. If you want me to build from the Suse source RPMs with debugging enabled and test those too, please tell me. Running 'wbinfo -u' while while winbindd is run from gdb: (gdb) run -F -S -s /etc/samba/smb.conf Starting program: /opt/samba/sbin/winbindd -F -S -s /etc/samba/smb.conf winbindd version 3.0.25a started. Copyright Andrew Tridgell and the Samba Team 1992-2007 Program received signal SIGSEGV, Segmentation fault. 0xb7ecc60f in krb5_copy_principal () from /usr/lib/libkrb5.so.3 (gdb) bt #0 0xb7ecc60f in krb5_copy_principal () from /usr/lib/libkrb5.so.3 #1 0xb7ecc144 in krb5_copy_creds () from /usr/lib/libkrb5.so.3 #2 0xb7ec307b in krb5_get_notification_message () from /usr/lib/libkrb5.so.3 #3 0xb7ec3be0 in krb5_cc_store_cred () from /usr/lib/libkrb5.so.3 #4 0xb7ecfc83 in krb5_get_credentials () from /usr/lib/libkrb5.so.3 #5 0x80118b53 in ads_krb5_mk_req (context=0x803bc230, auth_context=0xbfa5c414, ap_req_options=1, principal=0x803bc210 "mchp7u3a$@WW200.SIEMENS.NET", ccache=0x803bc2d0, outbuf=0xbfa5c420, expire_time=0x802d7178) at libsmb/clikrb5.c:599 #6 0x8011918a in cli_krb5_get_ticket ( principal=0x803bc210 "mchp7u3a$@WW200.SIEMENS.NET", time_offset=0, ticket=0xbfa5c4b0, session_key_krb5=0xbfa5c4fc, extra_ap_opts=0, ccname=0x0, tgs_expire=0x802d7178) at libsmb/clikrb5.c:696 #7 0x8011b13b in spnego_gen_negTokenTarg ( principal=0x803bc210 "mchp7u3a$@WW200.SIEMENS.NET", time_offset=0, targ=0xbfa5c514, session_key_krb5=0xbfa5c4fc, extra_ap_opts=0, expire_time=0x802d7178) at libsmb/clispnego.c:354 #8 0x80234e44 in ads_sasl_spnego_krb5_bind (ads=0x802d7138, principal=0x803bc210 "mchp7u3a$@WW200.SIEMENS.NET") at libads/sasl.c:150 #9 0x8023552e in ads_sasl_spnego_bind (ads=0x802d7138) at libads/sasl.c:267 #10 0x80236062 in ads_sasl_bind (ads=0x802d7138) at libads/sasl.c:522 #11 0x8022c89c in ads_connect (ads=0x802d7138) at libads/ldap.c:466 #12 0x8006a469 in ads_cached_connection (domain=0x8038b130) at nsswitch/winbindd_ads.c:125 #13 0x8006d785 in sequence_number (domain=0x8038b130, seq=0x8038b5b4) at nsswitch/winbindd_ads.c:1014 #14 0x8004cbc5 in refresh_sequence_number (domain=0x8038b130, force=0) at nsswitch/winbindd_cache.c:479 #15 0x8004d213 in wcache_fetch (cache=0x80331878, domain=0x8038b130, format=0x80264997 "UL/%s") at nsswitch/winbindd_cache.c:601 #16 0x8004e7b2 in query_user_list (domain=0x8038b130, mem_ctx=0x80332c40, num_entries=0xbfa5cc54, info=0xbfa5cc58) at nsswitch/winbindd_cache.c:1082 #17 0x800437b3 in winbindd_list_users (state=0x803581f0) at nsswitch/winbindd_user.c:777 #18 0x8003ef67 in process_request (state=0x803581f0) at nsswitch/winbindd.c:312 #19 0x8003fc9f in request_recv (private_data=0x803581f0, success=1) at nsswitch/winbindd.c:602 #20 0x8003fa9a in request_main_recv (private_data=0x803581f0, success=1) at nsswitch/winbindd.c:563 #21 0x8003f36e in rw_callback (event=0x803581fc, flags=1) at nsswitch/winbindd.c:395 #22 0x800403d7 in process_loop () at nsswitch/winbindd.c:832 #23 0x800410da in main (argc=5, argv=0xbfa5d4b4, envp=0xbfa5d4cc) at nsswitch/winbindd.c:1100 (gdb) Anything else I can do?
Perhaps a bug in kerberos? Downgraded to OpenSuse 10.1 RPM krb5-1.4.3-19 to install associated krb5-debuginfo-1.4.3-19.i586.rpm. Now the backtrace shows: #0 0xb7cd7f49 in free () from /lib/libc.so.6 #1 0xb7e41e46 in krb5_free_principal (context=0x803bc118, val=0x803bfa20) at kfree.c:394 #2 0xb7e4315d in krb5_free_cred_contents (context=0x803bc118, val=0x803bfa58) at kfree.c:155 #3 0xb7e43249 in krb5_free_creds (context=0x803bc118, val=0x803bfa58) at kfree.c:220 #4 0xb7e4328e in krb5_free_tgt_creds (context=0x803bc118, tgts=0x803bc8f0) at kfree.c:493 #5 0xb7e3dc9c in krb5_get_credentials (context=0x803bc118, options=<value optimized out>, ccache=0x803bc1b8, in_creds=0xbfe83f58, out_creds=0xbfe83fac) at get_creds.c:159 #6 0x80118b53 in ads_krb5_mk_req (context=0x803bc118, auth_context=0xbfe84034, ap_req_options=1, principal=0x803bc0f8 "mchp7u3a$@WW200.SIEMENS.NET", ccache=0x803bc1b8, outbuf=0xbfe84040, expire_time=0x803b2dc8) at libsmb/clikrb5.c:599 #7 0x8011918a in cli_krb5_get_ticket ( principal=0x803bc0f8 "mchp7u3a$@WW200.SIEMENS.NET", time_offset=0, ticket=0xbfe840d0, session_key_krb5=0xbfe8411c, extra_ap_opts=0, ccname=0x0, tgs_expire=0x803b2dc8) at libsmb/clikrb5.c:696 #8 0x8011b13b in spnego_gen_negTokenTarg ( principal=0x803bc0f8 "mchp7u3a$@WW200.SIEMENS.NET", time_offset=0, targ=0xbfe84134, session_key_krb5=0xbfe8411c, extra_ap_opts=0, expire_time=0x803b2dc8) at libsmb/clispnego.c:354 #9 0x80234e44 in ads_sasl_spnego_krb5_bind (ads=0x803b2d88, principal=0x803bc0f8 "mchp7u3a$@WW200.SIEMENS.NET") at libads/sasl.c:150 #10 0x8023552e in ads_sasl_spnego_bind (ads=0x803b2d88) at libads/sasl.c:267 #11 0x80236062 in ads_sasl_bind (ads=0x803b2d88) at libads/sasl.c:522 #12 0x8022c89c in ads_connect (ads=0x803b2d88) at libads/ldap.c:466 #13 0x8006a469 in ads_cached_connection (domain=0x8038b130) at nsswitch/winbindd_ads.c:125 #14 0x8006d785 in sequence_number (domain=0x8038b130, seq=0x8038b5b4) at nsswitch/winbindd_ads.c:1014 #15 0x8004cbc5 in refresh_sequence_number (domain=0x8038b130, force=0) at nsswitch/winbindd_cache.c:479 #16 0x8004d213 in wcache_fetch (cache=0x80331878, domain=0x8038b130, format=0x80264997 "UL/%s") at nsswitch/winbindd_cache.c:601 #17 0x8004e7b2 in query_user_list (domain=0x8038b130, mem_ctx=0x80332c40, num_entries=0xbfe84874, info=0xbfe84878) at nsswitch/winbindd_cache.c:1082 #18 0x800437b3 in winbindd_list_users (state=0x803581f0) at nsswitch/winbindd_user.c:777 #19 0x8003ef67 in process_request (state=0x803581f0) at nsswitch/winbindd.c:312 #20 0x8003fc9f in request_recv (private_data=0x803581f0, success=1) at nsswitch/winbindd.c:602 #21 0x8003fa9a in request_main_recv (private_data=0x803581f0, success=1) at nsswitch/winbindd.c:563 #22 0x8003f36e in rw_callback (event=0x803581fc, flags=1) at nsswitch/winbindd.c:395 #23 0x800403d7 in process_loop () at nsswitch/winbindd.c:832 #24 0x800410da in main (argc=5, argv=0xbfe850d4, envp=0xbfe850ec) at nsswitch/winbindd.c:1100 (gdb)
not a known issue in recent samba versions.