Bug 11081 - Winbind INTERNAL ERROR Signal 11 periodically (not constantly)
Summary: Winbind INTERNAL ERROR Signal 11 periodically (not constantly)
Status: RESOLVED FIXED
Alias: None
Product: Samba 4.1 and newer
Classification: Unclassified
Component: Winbind (show other bugs)
Version: 4.1.16
Hardware: x64 Linux
: P5 normal (vote)
Target Milestone: 4.3
Assignee: Karolin Seeger
QA Contact: Samba QA Contact
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-01-28 20:21 UTC by John Douglass
Modified: 2021-02-11 14:10 UTC (History)
6 users (show)

See Also:


Attachments
Core file created by the above error (4.30 MB, application/x-core)
2015-01-28 20:21 UTC, John Douglass
no flags Details
Core dump suffering from the error referred to in comment #4 (1.83 MB, application/octet-stream)
2015-06-24 08:19 UTC, Orlando Richards
no flags Details
test patch for 4.1.16 (2.15 KB, patch)
2015-06-24 09:50 UTC, Stefan Metzmacher
no flags Details
Patches for v4-1-test (2.28 KB, patch)
2015-06-24 09:59 UTC, Stefan Metzmacher
gd: review+
Details
Cleanup patch for master (1.45 KB, patch)
2015-06-24 10:29 UTC, Stefan Metzmacher
gd: review+
Details

Note You need to log in before you can comment on or make changes to this bug.
Description John Douglass 2015-01-28 20:21:24 UTC
Created attachment 10681 [details]
Core file created by the above error

When using ntlm_auth at the command line, I am seeing a few authentications sometimes fail with: NT_STATUS_CONNECTION_DISCONNECTED reported by the ntlm_process.

Looking for issues within the samba logs at the same time, I came across a mention of:

Jan 28 14:02:03 radius5-vm winbindd[57540]:   INTERNAL ERROR: Signal 11 in pid 57540 (4.1.16-SerNet-RedHat-10.el6)

Attached are my configuration, a larger chunk of the debugging logs, and the core file created.

smb.conf

[global]

	workgroup = AD
	server string = Samba Server Version %v

	netbios name = RADIUS05
	log file = /var/log/samba/log.%m
	# maximum size of 50KB per log file, then rotate:
	max log size = 50
        log level = 4
        syslog = 6
	security = ads
	passdb backend = tdbsam
	realm = AD.GATECH.EDU

	#password server = gtad02.ad.gatech.edu gtad05.ad.gatech.edu gtad06.ad.gatech.edu
	password server = plusdc01.ad.gatech.edu

# ------------ winbind configuration -----------------------------------------

   #winbind max clients = 8096
   winbind max clients = 16192
   winbind max domain connections = 128
   # Default is 60 
   winbind request timeout = 30
   winbind reconnect delay = 5

	load printers = yes
	cups options = raw


[homes]
	comment = Home Directories
	browseable = no
	writable = yes

[printers]
	comment = All Printers
	path = /var/spool/samba
	browseable = no
	guest ok = no
	writable = no
	printable = yes


Error message in /var/log/samba:

Jan 28 14:02:02 radius5-vm winbindd[57540]:   Successfully contacted LDAP server 143.215.14.165
Jan 28 14:02:02 radius5-vm winbindd[57540]: [2015/01/28 14:02:02.831542,  3] ../source3/libsmb/namequery.c:3102(get_dc_list)
Jan 28 14:02:02 radius5-vm winbindd[57540]:   get_dc_list: preferred server list: ", plusdc01.ad.gatech.edu"
Jan 28 14:02:02 radius5-vm winbindd[57540]: [2015/01/28 14:02:02.831832,  4] ../source3/libsmb/namequery.c:3239(get_dc_list)
Jan 28 14:02:02 radius5-vm winbindd[57540]:   get_dc_list: returning 1 ip addresses in an ordered list
Jan 28 14:02:02 radius5-vm winbindd[57540]: [2015/01/28 14:02:02.834843,  4] ../source3/libsmb/namequery.c:3240(get_dc_list)
Jan 28 14:02:02 radius5-vm winbindd[57540]:   get_dc_list: 143.215.14.165:389 
Jan 28 14:02:02 radius5-vm winbindd[57540]: [2015/01/28 14:02:02.848249,  3] ../source3/libsmb/cliconnect.c:1781(cli_session_setup_spnego_send)
Jan 28 14:02:02 radius5-vm winbindd[57540]:   Doing spnego session setup (blob length=120)
Jan 28 14:02:02 radius5-vm winbindd[57540]: [2015/01/28 14:02:02.848423,  3] ../source3/libsmb/cliconnect.c:1808(cli_session_setup_spnego_send)
Jan 28 14:02:02 radius5-vm winbindd[57540]:   got OID=1.3.6.1.4.1.311.2.2.30
Jan 28 14:02:02 radius5-vm winbindd[57540]:   got OID=1.2.840.48018.1.2.2
Jan 28 14:02:02 radius5-vm winbindd[57540]:   got OID=1.2.840.113554.1.2.2
Jan 28 14:02:02 radius5-vm winbindd[57540]:   got OID=1.2.840.113554.1.2.2.3
Jan 28 14:02:02 radius5-vm winbindd[57540]:   got OID=1.3.6.1.4.1.311.2.2.10
Jan 28 14:02:02 radius5-vm winbindd[57540]: [2015/01/28 14:02:02.848550,  3] ../source3/libsmb/cliconnect.c:1818(cli_session_setup_spnego_send)
Jan 28 14:02:02 radius5-vm winbindd[57540]:   got principal=not_defined_in_RFC4178@please_ignore
Jan 28 14:02:02 radius5-vm winbindd[57540]: [2015/01/28 14:02:02.948506,  3] ../source3/libsmb/cliconnect.c:1687(cli_session_setup_get_principal)
Jan 28 14:02:02 radius5-vm winbindd[57540]:   cli_session_setup_spnego: using target hostname not SPNEGO principal
Jan 28 14:02:02 radius5-vm winbindd[57540]: [2015/01/28 14:02:02.948837,  3] ../source3/libsmb/cliconnect.c:1702(cli_session_setup_get_principal)
Jan 28 14:02:02 radius5-vm winbindd[57540]:   cli_session_setup_spnego: guessed server principal=cifs/plusdc01.ad.gatech.edu@AD.GATECH.EDU
Jan 28 14:02:02 radius5-vm winbindd[57540]: [2015/01/28 14:02:02.949064,  2] ../source3/libsmb/cliconnect.c:1312(cli_session_setup_kerberos_send)
Jan 28 14:02:02 radius5-vm winbindd[57540]:   Doing kerberos session setup
Jan 28 14:02:02 radius5-vm winbindd[57540]: [2015/01/28 14:02:02.974234,  3] ../lib/krb5_wrap/krb5_samba.c:266(ads_cleanup_expired_creds)
Jan 28 14:02:02 radius5-vm winbindd[57540]:   ads_cleanup_expired_creds: Ticket in ccache[MEMORY:cliconnect] expiration Thu, 29 Jan 2015 00:02:02 EST
Jan 28 14:02:03 radius5-vm winbindd[57540]: [2015/01/28 14:02:03.046893,  0] ../lib/util/fault.c:72(fault_report)
Jan 28 14:02:03 radius5-vm winbindd[57540]:   ===============================================================
Jan 28 14:02:03 radius5-vm winbindd[57540]: [2015/01/28 14:02:03.046987,  0] ../lib/util/fault.c:73(fault_report)
Jan 28 14:02:03 radius5-vm winbindd[57540]:   INTERNAL ERROR: Signal 11 in pid 57540 (4.1.16-SerNet-RedHat-10.el6)
Jan 28 14:02:03 radius5-vm winbindd[57540]:   Please read the Trouble-Shooting section of the Samba HOWTO
Jan 28 14:02:03 radius5-vm winbindd[57540]: [2015/01/28 14:02:03.047053,  0] ../lib/util/fault.c:75(fault_report)
Jan 28 14:02:03 radius5-vm winbindd[57540]:   ===============================================================
Jan 28 14:02:03 radius5-vm winbindd[57540]: [2015/01/28 14:02:03.047101,  0] ../source3/lib/util.c:785(smb_panic_s3)
Jan 28 14:02:03 radius5-vm winbindd[57540]:   PANIC (pid 57540): internal error
Jan 28 14:02:03 radius5-vm winbindd[57540]: [2015/01/28 14:02:03.047727,  0] ../source3/lib/util.c:896(log_stack_trace)
Jan 28 14:02:03 radius5-vm winbindd[57540]:   BACKTRACE: 22 stack frames:
Jan 28 14:02:03 radius5-vm winbindd[57540]:    #0 /usr/lib64/samba/libsmbconf.so.0(log_stack_trace+0x1c) [0x7f9bb16869b7]
Jan 28 14:02:03 radius5-vm winbindd[57540]:    #1 /usr/lib64/samba/libsmbconf.so.0(smb_panic_s3+0x55) [0x7f9bb1686ab9]
Jan 28 14:02:03 radius5-vm winbindd[57540]:    #2 /usr/lib64/samba/libsamba-util.so.0(smb_panic+0x2d) [0x7f9bb67e6bae]
Jan 28 14:02:03 radius5-vm winbindd[57540]:    #3 /usr/lib64/samba/libsamba-util.so.0(+0x17ec7) [0x7f9bb67e6ec7]
Jan 28 14:02:03 radius5-vm winbindd[57540]:    #4 /lib64/libpthread.so.0(+0x3fa480f710) [0x7f9bb6c16710]
Jan 28 14:02:03 radius5-vm winbindd[57540]:    #5 /usr/lib64/samba/liblibcli_netlogon3.so(rpccli_netlogon_sam_network_logon+0x16e) [0x7f9bb38603c7]
Jan 28 14:02:03 radius5-vm winbindd[57540]:    #6 /usr/sbin/winbindd(+0x3ed04) [0x7f9bb7084d04]
Jan 28 14:02:03 radius5-vm winbindd[57540]:    #7 /usr/sbin/winbindd(winbindd_dual_pam_auth_crap+0x2d6) [0x7f9bb70870f7]
Jan 28 14:02:03 radius5-vm winbindd[57540]:    #8 /usr/sbin/winbindd(+0x5691a) [0x7f9bb709c91a]
Jan 28 14:02:03 radius5-vm winbindd[57540]:    #9 /usr/lib64/samba/libtevent.so.0(+0x8a9c) [0x7f9bb5f8fa9c]
Jan 28 14:02:03 radius5-vm winbindd[57540]:    #10 /usr/lib64/samba/libtevent.so.0(+0x6d26) [0x7f9bb5f8dd26]
Jan 28 14:02:03 radius5-vm winbindd[57540]:    #11 /usr/lib64/samba/libtevent.so.0(_tevent_loop_once+0x92) [0x7f9bb5f8a407]
Jan 28 14:02:03 radius5-vm winbindd[57540]:    #12 /usr/sbin/winbindd(+0x57e79) [0x7f9bb709de79]
Jan 28 14:02:03 radius5-vm winbindd[57540]:    #13 /usr/sbin/winbindd(+0x57ffc) [0x7f9bb709dffc]
Jan 28 14:02:03 radius5-vm winbindd[57540]:    #14 /usr/lib64/samba/libtevent.so.0(+0x430b) [0x7f9bb5f8b30b]
Jan 28 14:02:03 radius5-vm winbindd[57540]:    #15 /usr/lib64/samba/libtevent.so.0(tevent_common_loop_immediate+0x128) [0x7f9bb5f8af48]
Jan 28 14:02:03 radius5-vm winbindd[57540]:    #16 /usr/lib64/samba/libtevent.so.0(+0x8816) [0x7f9bb5f8f816]
Jan 28 14:02:03 radius5-vm winbindd[57540]:    #17 /usr/lib64/samba/libtevent.so.0(+0x6d26) [0x7f9bb5f8dd26]
Jan 28 14:02:03 radius5-vm winbindd[57540]:    #18 /usr/lib64/samba/libtevent.so.0(_tevent_loop_once+0x92) [0x7f9bb5f8a407]
Jan 28 14:02:03 radius5-vm winbindd[57540]:    #19 /usr/sbin/winbindd(main+0x113e) [0x7f9bb707721a]
Jan 28 14:02:03 radius5-vm winbindd[57540]:    #20 /lib64/libc.so.6(__libc_start_main+0xfd) [0x7f9baff6cd5d]
Jan 28 14:02:03 radius5-vm winbindd[57540]:    #21 /usr/sbin/winbindd(+0x226d9) [0x7f9bb70686d9]
Jan 28 14:02:03 radius5-vm winbindd[57540]: [2015/01/28 14:02:03.049425,  0] ../source3/lib/dumpcore.c:317(dump_core)
Jan 28 14:02:03 radius5-vm winbindd[57540]:   dumping core in /var/log/samba/cores/winbindd
Jan 28 14:02:03 radius5-vm winbindd[57540]: 
Jan 28 14:02:03 radius5-vm abrtd: Directory 'ccpp-2015-01-28-14:02:03-57540' creation detected
Jan 28 14:02:03 radius5-vm abrt[57635]: Saved core dump of pid 57540 (/usr/sbin/winbindd) to /var/spool/abrt/ccpp-2015-01-28-14:02:03-57540 (4505600 bytes)
Jan 28 14:02:03 radius5-vm abrtd: Package 'sernet-samba-winbind' isn't signed with proper key
Jan 28 14:02:03 radius5-vm abrtd: 'post-create' on '/var/spool/abrt/ccpp-2015-01-28-14:02:03-57540' exited with 1
Jan 28 14:02:03 radius5-vm abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2015-01-28-14:02:03-57540'
Jan 28 14:02:05 radius5-vm winbindd[7328]: [2015/01/28 14:02:05.656042,  3] ../source3/winbindd/winbindd_misc.c:395(winbindd_interface_version)
Jan 28 14:02:05 radius5-vm winbindd[7328]:   [57865]: request interface version
Jan 28 14:02:05 radius5-vm winbindd[7328]: [2015/01/28 14:02:05.656452,  3] ../source3/winbindd/winbindd_misc.c:428(winbindd_priv_pipe_dir)
Jan 28 14:02:05 radius5-vm winbindd[7328]:   [57865]: request location of privileged pipe
Jan 28 14:02:05 radius5-vm winbindd[7328]: [2015/01/28 14:02:05.657537,  3] ../source3/winbindd/winbindd_misc.c:405(winbindd_domain_name)
Jan 28 14:02:05 radius5-vm winbindd[7328]:   [57865]: request domain name
Jan 28 14:02:05 radius5-vm winbindd[7328]: [2015/01/28 14:02:05.657663,  3] ../source3/winbindd/winbindd_pam_auth_crap.c:73(winbindd_pam_auth_crap_send)
Jan 28 14:02:05 radius5-vm winbindd[7328]:   [57865]: pam auth crap domain: [AD] user: dp57
Jan 28 14:02:05 radius5-vm winbindd[7332]: [2015/01/28 14:02:05.657788,  4] ../source3/winbindd/winbindd_dual.c:1338(child_handler)
Jan 28 14:02:05 radius5-vm winbindd[7332]:   child daemon request 14
Jan 28 14:02:05 radius5-vm winbindd[7332]: [2015/01/28 14:02:05.657874,  3] ../source3/winbindd/winbindd_pam.c:1896(winbindd_dual_pam_auth_crap)
Jan 28 14:02:05 radius5-vm winbindd[7332]:   [ 7328]: pam auth crap domain: AD user: dp57
Jan 28 14:02:05 radius5-vm winbindd[7332]: [2015/01/28 14:02:05.687608,  2] ../source3/winbindd/winbindd_pam.c:2003(winbindd_dual_pam_auth_crap)
Jan 28 14:02:05 radius5-vm winbindd[7332]:   NTLM CRAP authentication for user [AD]\[dp57] returned NT_STATUS_LOGON_FAILURE (PAM: 7)
Jan 28 14:02:05 radius5-vm winbindd[7332]: [2015/01/28 14:02:05.687688,  4] ../source3/winbindd/winbindd_dual.c:1346(child_handler)
Jan 28 14:02:05 radius5-vm winbindd[7332]:   Finished processing child request 14
Jan 28 14:02:07 radius5-vm winbindd[7328]: [2015/01/28 14:02:07.828169,  3] ../source3/winbindd/winbindd_misc.c:395(winbindd_interface_version)
Jan 28 14:02:07 radius5-vm winbindd[7328]:   [57867]: request interface version
Jan 28 14:02:07 radius5-vm winbindd[7328]: [2015/01/28 14:02:07.828312,  3] ../source3/winbindd/winbindd_misc.c:428(winbindd_priv_pipe_dir)
Jan 28 14:02:07 radius5-vm winbindd[7328]:   [57867]: request location of privileged pipe
Jan 28 14:02:07 radius5-vm winbindd[7328]: [2015/01/28 14:02:07.828539,  3] ../source3/winbindd/winbindd_misc.c:405(winbindd_domain_name)
Jan 28 14:02:07 radius5-vm winbindd[7328]:   [57867]: request domain name
Jan 28 14:02:07 radius5-vm winbindd[7328]: [2015/01/28 14:02:07.828669,  3] ../source3/winbindd/winbindd_pam_auth_crap.c:73(winbindd_pam_auth_crap_send)
Jan 28 14:02:07 radius5-vm winbindd[7328]:   [57867]: pam auth crap domain: [AD] user: llenhardt3
Jan 28 14:02:07 radius5-vm winbindd[7332]: [2015/01/28 14:02:07.828804,  4] ../source3/winbindd/winbindd_dual.c:1338(child_handler)
Jan 28 14:02:07 radius5-vm winbindd[7332]:   child daemon request 14
Jan 28 14:02:07 radius5-vm winbindd[7332]: [2015/01/28 14:02:07.828897,  3] ../source3/winbindd/winbindd_pam.c:1896(winbindd_dual_pam_auth_crap)
Jan 28 14:02:07 radius5-vm winbindd[7332]:   [ 7328]: pam auth crap domain: AD user: llenhardt3
Jan 28 14:02:07 radius5-vm winbindd[7332]: [2015/01/28 14:02:07.835702,  4] ../source3/winbindd/winbindd_dual.c:1346(child_handler)
Jan 28 14:02:07 radius5-vm winbindd[7332]:   Finished processing child request 14
Comment 1 John Douglass 2015-01-28 20:35:39 UTC
This internal error signal 11 occurs periodically through the day and causes a failed authentication.
Comment 2 Warren 2015-06-18 03:26:02 UTC
We hit this too in Samba 4.1.12 when the domain controller goes offline. It might have similar AD connection issue in the original report at comment #0.
Comment 3 Warren 2015-06-18 14:48:07 UTC
#0  0x00007ff1451ad625 in raise () from /lib64/libc.so.6
#1  0x00007ff1451aee05 in abort () from /lib64/libc.so.6
#2  0x00007ff146cd6151 in dump_core () at ../source3/lib/dumpcore.c:336
#3  0x00007ff146cc9080 in smb_panic_s3 (why=<optimized out>) at ../source3/lib/util.c:808
#4  0x00007ff14b7fa358 in smb_panic (why=0x7ff14b8093a5 "internal error") at ../lib/util/fault.c:159
#5  0x00007ff14b7fa412 in fault_report (sig=11) at ../lib/util/fault.c:77
#6  sig_fault (sig=11) at ../lib/util/fault.c:88
#7  <signal handler called>
#8  rpccli_netlogon_sam_network_logon (cli=0x7ff14d549a30, mem_ctx=0x7ff14d54ba40, logon_parameters=2080, server=0x0, username=0x7fff74f88b74
...

It looks like winbindd tried to access 0x0 pointer "server" in some case.

405 |_______netlogon_creds_client_authenticator(cli->dc, &clnt_creds);
406
407 |_______if (server[0] != '\\' && server[1] != '\\') {  <=== 
408 |_______|_______server_name_slash = talloc_asprintf(mem_ctx, "\\\\%s", server);
409 |_______} else {
410 |_______|_______server_name_slash = server;
411 |_______}
Comment 4 Orlando Richards 2015-06-24 08:04:34 UTC
We also see something very similar with high winbind load, particularly where there is a large number of new connections to winbind (for example - when restarting smbd). This only occurs when we have "winbind max domain connections" set to anything greater than 1.

I got this from gdb:
(gdb) bt
#0  0x00007f78277f7625 in raise () from /lib64/libc.so.6
#1  0x00007f78277f8e05 in abort () from /lib64/libc.so.6
#2  0x00007f7828f0fb9a in dump_core () at ../source3/lib/dumpcore.c:336
#3  0x00007f7828efa914 in smb_panic_s3 (why=<value optimized out>) at ../source3/lib/util.c:808
#4  0x00007f782e054b8e in smb_panic (why=0x7f782e062dca "internal error") at ../lib/util/fault.c:159
#5  0x00007f782e054ea7 in fault_report (sig=11) at ../lib/util/fault.c:77
#6  sig_fault (sig=11) at ../lib/util/fault.c:88
#7  <signal handler called>


And we get the following in syslog:

Jun 16 13:43:15 rdm-server04 winbindd[4058]:   STATUS=daemon 'winbindd' finished starting up and ready to serve connections===============================================================
Jun 16 13:43:15 rdm-server04 winbindd[4058]: [2015/06/16 13:43:15.919874,  0] 
Jun 16 13:43:15 rdm-server04 winbindd[4058]:   INTERNAL ERROR: Signal 11 in pid 4058 (4.1.17-SerNet-RedHat-10.el6)
Jun 16 13:43:15 rdm-server04 winbindd[4058]:   Please read the Trouble-Shooting section of the Samba HOWTO
Jun 16 13:43:15 rdm-server04 winbindd[4058]: [2015/06/16 13:43:15.920009,  0] 
Jun 16 13:43:15 rdm-server04 winbindd[4058]:   ===============================================================
Jun 16 13:43:15 rdm-server04 winbindd[4058]: [2015/06/16 13:43:15.920063,  0] 
Jun 16 13:43:15 rdm-server04 winbindd[4058]:   PANIC (pid 4058): internal error
Jun 16 13:43:15 rdm-server04 winbindd[4058]: [2015/06/16 13:43:15.920488,  0] 
Jun 16 13:43:15 rdm-server04 winbindd[4058]:   BACKTRACE: 22 stack frames:
Jun 16 13:43:15 rdm-server04 winbindd[4058]:    #0 /usr/lib64/samba/libsmbconf.so.0(log_stack_trace+0x1c) [0x7f7828efa6f7]
Jun 16 13:43:15 rdm-server04 winbindd[4058]:    #1 /usr/lib64/samba/libsmbconf.so.0(smb_panic_s3+0x55) [0x7f7828efa7f9]
Jun 16 13:43:15 rdm-server04 winbindd[4058]:    #2 /usr/lib64/samba/libsamba-util.so.0(smb_panic+0x2d) [0x7f782e054b8e]
Jun 16 13:43:15 rdm-server04 winbindd[4058]:    #3 /usr/lib64/samba/libsamba-util.so.0(+0x17ea7) [0x7f782e054ea7]
Jun 16 13:43:15 rdm-server04 winbindd[4058]:    #4 /lib64/libpthread.so.0(+0xf710) [0x7f782e484710]
Jun 16 13:43:15 rdm-server04 winbindd[4058]:    #5 /usr/lib64/samba/liblibcli_netlogon3.so(rpccli_netlogon_sam_network_logon+0x16e) [0x7f782b0d23c7]
Jun 16 13:43:15 rdm-server04 winbindd[4058]:    #6 /usr/sbin/winbindd(+0x3ed14) [0x7f782e8f2d14]
Jun 16 13:43:15 rdm-server04 winbindd[4058]:    #7 /usr/sbin/winbindd(winbindd_dual_pam_auth_crap+0x2d6) [0x7f782e8f5107]
Jun 16 13:43:15 rdm-server04 winbindd[4058]:    #8 /usr/sbin/winbindd(+0x5695a) [0x7f782e90a95a]
Jun 16 13:43:15 rdm-server04 winbindd[4058]:    #9 /usr/lib64/samba/libtevent.so.0(+0x8a9c) [0x7f782d7fda9c]
Jun 16 13:43:15 rdm-server04 winbindd[4058]:    #10 /usr/lib64/samba/libtevent.so.0(+0x6d26) [0x7f782d7fbd26]
Jun 16 13:43:15 rdm-server04 winbindd[4058]:    #11 /usr/lib64/samba/libtevent.so.0(_tevent_loop_once+0x92) [0x7f782d7f8407]
Jun 16 13:43:15 rdm-server04 winbindd[4058]:    #12 /usr/sbin/winbindd(+0x57eb9) [0x7f782e90beb9]
Jun 16 13:43:15 rdm-server04 winbindd[4058]:    #13 /usr/sbin/winbindd(+0x5803c) [0x7f782e90c03c]
Jun 16 13:43:15 rdm-server04 winbindd[4058]:    #14 /usr/lib64/samba/libtevent.so.0(+0x430b) [0x7f782d7f930b]
Jun 16 13:43:15 rdm-server04 winbindd[4058]:    #15 /usr/lib64/samba/libtevent.so.0(tevent_common_loop_immediate+0x128) [0x7f782d7f8f48]
Jun 16 13:43:15 rdm-server04 winbindd[4058]:    #16 /usr/lib64/samba/libtevent.so.0(+0x8816) [0x7f782d7fd816]
Jun 16 13:43:15 rdm-server04 winbindd[4058]:    #17 /usr/lib64/samba/libtevent.so.0(+0x6d26) [0x7f782d7fbd26]
Jun 16 13:43:15 rdm-server04 winbindd[4058]:    #18 /usr/lib64/samba/libtevent.so.0(_tevent_loop_once+0x92) [0x7f782d7f8407]
Jun 16 13:43:15 rdm-server04 winbindd[4058]:    #19 /usr/sbin/winbindd(main+0x113e) [0x7f782e8e521a]
Jun 16 13:43:15 rdm-server04 winbindd[4058]:    #20 /lib64/libc.so.6(__libc_start_main+0xfd) [0x7f78277e3d5d]
Jun 16 13:43:15 rdm-server04 winbindd[4058]:    #21 /usr/sbin/winbindd(+0x226d9) [0x7f782e8d66d9]
Jun 16 13:43:15 rdm-server04 winbindd[4058]: [2015/06/16 13:43:15.921067,  0] 
Jun 16 13:43:15 rdm-server04 winbindd[4058]:   dumping core in /var/log/cores/winbindd
Jun 16 13:43:15 rdm-server04 winbindd[4058]:
Comment 5 Orlando Richards 2015-06-24 08:19:57 UTC
Created attachment 11188 [details]
Core dump suffering from the error referred to in comment #4

(gdb) bt
#0  0x00007f78277f7625 in raise () from /lib64/libc.so.6
#1  0x00007f78277f8e05 in abort () from /lib64/libc.so.6
#2  0x00007f7828f0fb9a in dump_core () at ../source3/lib/dumpcore.c:336
#3  0x00007f7828efa914 in smb_panic_s3 (why=<value optimized out>) at ../source3/lib/util.c:808
#4  0x00007f782e054b8e in smb_panic (why=0x7f782e062dca "internal error") at ../lib/util/fault.c:159
#5  0x00007f782e054ea7 in fault_report (sig=11) at ../lib/util/fault.c:77
#6  sig_fault (sig=11) at ../lib/util/fault.c:88
#7  <signal handler called>
#8  rpccli_netlogon_sam_network_logon (cli=0x7f7830f9c5c0, mem_ctx=0x7f7830fb1d00, logon_parameters=2080, server=0x0, username=0x7fff9c4653f4 "cekmekci", 
    domain=0x7fff9c4654f4 "ED", workstation=0x7fff9c4657fc "IS-LAC-2046", chal=0x7fff9c4653e8 "\005\060ky\233\313\060\270 \b", validation_level=3, 
    lm_response=..., nt_response=..., info3=0x7fff9c4642e8) at ../source3/rpc_client/cli_netlogon.c:407
#9  0x00007f782e8f2d14 in winbind_samlogon_retry_loop (domain=0x7f7830bb4150, mem_ctx=0x7f7830fb1d00, logon_parameters=2080, server=0x0, 
    username=0x7fff9c4653f4 "cekmekci", domainname=0x7fff9c4654f4 "ED", workstation=0x7fff9c4657fc "IS-LAC-2046", 
    chal=0x7fff9c4653e8 "\005\060ky\233\313\060\270 \b", lm_response=..., nt_response=..., info3=0x7fff9c4642e8)
    at ../source3/winbindd/winbindd_pam.c:1349
#10 0x00007f782e8f5107 in winbindd_dual_pam_auth_crap (domain=0x7f7830bb4150, state=0x7fff9c464468) at ../source3/winbindd/winbindd_pam.c:1936
#11 0x00007f782e90a95a in child_process_request (ev=<value optimized out>, fde=<value optimized out>, flags=<value optimized out>, 
    private_data=0x7fff9c464460) at ../source3/winbindd/winbindd_dual.c:459
#12 child_handler (ev=<value optimized out>, fde=<value optimized out>, flags=<value optimized out>, private_data=0x7fff9c464460)
    at ../source3/winbindd/winbindd_dual.c:1343
#13 0x00007f782d7fda9c in epoll_event_loop (ev=<value optimized out>, location=<value optimized out>) at ../lib/tevent/tevent_epoll.c:736
#14 epoll_event_loop_once (ev=<value optimized out>, location=<value optimized out>) at ../lib/tevent/tevent_epoll.c:931
#15 0x00007f782d7fbd26 in std_event_loop_once (ev=0x7f7830bac4a0, location=0x7f782e96edd0 "../source3/winbindd/winbindd_dual.c:1558")
    at ../lib/tevent/tevent_standard.c:112
#16 0x00007f782d7f8407 in _tevent_loop_once (ev=0x7f7830bac4a0, location=0x7f782e96edd0 "../source3/winbindd/winbindd_dual.c:1558")
    at ../lib/tevent/tevent.c:530
#17 0x00007f782e90beb9 in fork_domain_child (child=0x7f7830bb43e8) at ../source3/winbindd/winbindd_dual.c:1558
#18 0x00007f782e90c03c in wb_child_request_trigger (req=0x7f7830fb4ea0, private_data=<value optimized out>) at ../source3/winbindd/winbindd_dual.c:146
#19 0x00007f782d7f930b in tevent_queue_immediate_trigger (ev=<value optimized out>, im=<value optimized out>, private_data=<value optimized out>)
    at ../lib/tevent/tevent_queue.c:148
#20 0x00007f782d7f8f48 in tevent_common_loop_immediate (ev=0x7f7830bac4a0) at ../lib/tevent/tevent_immediate.c:135
#21 0x00007f782d7fd816 in epoll_event_loop_once (ev=0x7f7830bac4a0, location=<value optimized out>) at ../lib/tevent/tevent_epoll.c:911
#22 0x00007f782d7fbd26 in std_event_loop_once (ev=0x7f7830bac4a0, location=0x7f782e95add0 "../source3/winbindd/winbindd.c:1625")
    at ../lib/tevent/tevent_standard.c:112
#23 0x00007f782d7f8407 in _tevent_loop_once (ev=0x7f7830bac4a0, location=0x7f782e95add0 "../source3/winbindd/winbindd.c:1625")
    at ../lib/tevent/tevent.c:530
#24 0x00007f782e8e521a in main (argc=<value optimized out>, argv=<value optimized out>, envp=<value optimized out>) at ../source3/winbindd/winbindd.c:1625
Comment 6 Stefan Metzmacher 2015-06-24 09:12:05 UTC
(In reply to Orlando Richards from comment #5)

can you paste the output of the following commands in gdb?

set height 0
bt full
up 9
p *domain
Comment 7 Orlando Richards 2015-06-24 09:19:18 UTC
(gdb) set height 0
(gdb) bt full
#0  0x00007f78277f7625 in raise () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007f78277f8e05 in abort () from /lib64/libc.so.6
No symbol table info available.
#2  0x00007f7828f0fb9a in dump_core () at ../source3/lib/dumpcore.c:336
        called = true
        __FUNCTION__ = "dump_core"
#3  0x00007f7828efa914 in smb_panic_s3 (why=<value optimized out>) at ../source3/lib/util.c:808
        cmd = <value optimized out>
        result = <value optimized out>
        __FUNCTION__ = "smb_panic_s3"
#4  0x00007f782e054b8e in smb_panic (why=0x7f782e062dca "internal error") at ../lib/util/fault.c:159
No locals.
#5  0x00007f782e054ea7 in fault_report (sig=11) at ../lib/util/fault.c:77
        counter = 1
#6  sig_fault (sig=11) at ../lib/util/fault.c:88
No locals.
#7  <signal handler called>
No symbol table info available.
#8  rpccli_netlogon_sam_network_logon (cli=0x7f7830f9c5c0, mem_ctx=0x7f7830fb1d00, logon_parameters=2080, server=0x0, username=0x7fff9c4653f4 "cekmekci", domain=0x7fff9c4654f4 "ED", 
    workstation=0x7fff9c4657fc "IS-LAC-2046", chal=0x7fff9c4653e8 "\005\060ky\233\313\060\270 \b", validation_level=3, lm_response=..., nt_response=..., info3=0x7fff9c4642e8) at ../source3/rpc_client/cli_netlogon.c:407
        result = {v = 3221225473}
        status = <value optimized out>
        workstation_name_slash = <value optimized out>
        server_name_slash = <value optimized out>
        clnt_creds = {cred = {data = "1\354=\262;\265\244", <incomplete sequence \305>}, timestamp = 1434458588}
        ret_creds = {cred = {data = "\000\000\000\000\000\000\000"}, timestamp = 0}
        logon = 0x7f7830f2cce0
        network_info = 0x7f7830fa1140
        authoritative = 0 '\000'
        validation = {sam2 = 0x0, sam3 = 0x0, pac = 0x0, generic = 0x0, sam6 = 0x0}
        lm = {length = 0, size = 0, data = 0x0}
        nt = {length = 0, size = 0, data = 0x0}
        b = 0x7f7830fa6cd0
        __FUNCTION__ = "rpccli_netlogon_sam_network_logon"
#9  0x00007f782e8f2d14 in winbind_samlogon_retry_loop (domain=0x7f7830bb4150, mem_ctx=0x7f7830fb1d00, logon_parameters=2080, server=0x0, username=0x7fff9c4653f4 "cekmekci", domainname=0x7fff9c4654f4 "ED", 
    workstation=0x7fff9c4657fc "IS-LAC-2046", chal=0x7fff9c4653e8 "\005\060ky\233\313\060\270 \b", lm_response=..., nt_response=..., info3=0x7fff9c4642e8) at ../source3/winbindd/winbindd_pam.c:1349
        netlogon_pipe = 0x7f7830f9c5c0
        auth = <value optimized out>
        neg_flags = <value optimized out>
        attempts = 0
        netr_attempts = 0
        retry = false
        result = <value optimized out>
        __FUNCTION__ = "winbind_samlogon_retry_loop"
#10 0x00007f782e8f5107 in winbindd_dual_pam_auth_crap (domain=0x7f7830bb4150, state=0x7fff9c464468) at ../source3/winbindd/winbindd_pam.c:1936
        result = <value optimized out>
        info3 = 0x0
        name_user = 0x7fff9c4653f4 "cekmekci"
        name_domain = 0x7fff9c4654f4 "ED"
        workstation = 0x7fff9c4657fc "IS-LAC-2046"
        lm_resp = {data = 0x7f7830eb19b0 "", length = 24}
        nt_resp = {data = 0x7f7830f9a9c0 "\260\035k\200\357\331Z$\204\211V\022e\334[\277\001\001", length = 300}
        __FUNCTION__ = "winbindd_dual_pam_auth_crap"
#11 0x00007f782e90a95a in child_process_request (ev=<value optimized out>, fde=<value optimized out>, flags=<value optimized out>, private_data=0x7fff9c464460) at ../source3/winbindd/winbindd_dual.c:459
        domain = 0x7f7830bb4150
        table = 0x7f782ebaa1c0
#12 child_handler (ev=<value optimized out>, fde=<value optimized out>, flags=<value optimized out>, private_data=0x7fff9c464460) at ../source3/winbindd/winbindd_dual.c:1343
        state = 0x7fff9c464460
        iov = {{iov_base = 0x4, iov_len = 999856}, {iov_base = 0x4, iov_len = 999856}}
        iov_count = <value optimized out>
        __FUNCTION__ = "child_handler"
#13 0x00007f782d7fda9c in epoll_event_loop (ev=<value optimized out>, location=<value optimized out>) at ../lib/tevent/tevent_epoll.c:736
        fde = 0x7f7830f03520
        flags = <value optimized out>
        mpx_fde = <value optimized out>
        ret = 1
        i = <value optimized out>
        timeout = <value optimized out>
        wait_errno = <value optimized out>
        events = {{events = 1, data = {ptr = 0x7f7830f03520, fd = 821048608, u32 = 821048608, u64 = 140154193851680}}}
#14 epoll_event_loop_once (ev=<value optimized out>, location=<value optimized out>) at ../lib/tevent/tevent_epoll.c:931
        epoll_ev = 0x7f7830ed71d0
        tval = {tv_sec = 4, tv_usec = 999856}
        panic_triggered = false
#15 0x00007f782d7fbd26 in std_event_loop_once (ev=0x7f7830bac4a0, location=0x7f782e96edd0 "../source3/winbindd/winbindd_dual.c:1558") at ../lib/tevent/tevent_standard.c:112
        glue_ptr = <value optimized out>
        glue = 0x7f7830bac590
        ret = <value optimized out>
#16 0x00007f782d7f8407 in _tevent_loop_once (ev=0x7f7830bac4a0, location=0x7f782e96edd0 "../source3/winbindd/winbindd_dual.c:1558") at ../lib/tevent/tevent.c:530
        ret = <value optimized out>
        nesting_stack_ptr = 0x0
#17 0x00007f782e90beb9 in fork_domain_child (child=0x7f7830bb43e8) at ../source3/winbindd/winbindd_dual.c:1558
        ret = <value optimized out>
        frame = 0x7f7830fb1d00
        fdpair = {497, 499}
        state = {child = 0x7f7830bb43e8, cli = {prev = 0x0, next = 0x0, sock = 497, pid = 31719, last_access = 0, privileged = false, mem_ctx = 0x7f7830fb1d00, cmd_name = 0x0, recv_fn = 0, request = 0x7fff9c4652d0, 
            out_queue = 0x0, response = 0x7fff9c464520, getpwent_initialized = false, getgrent_initialized = false, pwent_state = 0x0, grent_state = 0x0}}
        request = {length = 2096, cmd = WINBINDD_PAM_AUTH_CRAP, original_cmd = WINBINDD_INTERFACE_VERSION, pid = 32738, wb_flags = 0, flags = 65550, domain_name = '\000' <repeats 255 times>, data = {
            winsreq = "\005\060ky\233\313\060\270 \b\000\000cekmekci", '\000' <repeats 235 times>, username = "\005\060ky\233\313\060\270 \b\000\000cekmekci", '\000' <repeats 235 times>, 
            groupname = "\005\060ky\233\313\060\270 \b\000\000cekmekci", '\000' <repeats 235 times>, uid = 2037067781, gid = 2037067781, ndrcmd = 2037067781, auth = {
              user = "\005\060ky\233\313\060\270 \b\000\000cekmekci", '\000' <repeats 235 times>, pass = '\000' <repeats 12 times>, "ED", '\000' <repeats 241 times>, 
              require_membership_of_sid = '\000' <repeats 268 times>, "\030", '\000' <repeats 259 times>, ",\001\000\000IS-LAC-2046", '\000' <repeats 480 times>, krb5_cc_type = '\000' <repeats 255 times>, uid = 0}, 
            auth_crap = {chal = "\005\060ky\233\313\060\270", logon_parameters = 2080, user = "cekmekci", '\000' <repeats 247 times>, domain = "ED", '\000' <repeats 253 times>, lm_resp = '\000' <repeats 255 times>, 
              lm_resp_len = 24, nt_resp = '\000' <repeats 255 times>, nt_resp_len = 300, workstation = "IS-LAC-2046", '\000' <repeats 244 times>, require_membership_of_sid = '\000' <repeats 255 times>}, chauthtok = {
              user = "\005\060ky\233\313\060\270 \b\000\000cekmekci", '\000' <repeats 235 times>, oldpass = '\000' <repeats 12 times>, "ED", '\000' <repeats 241 times>, newpass = '\000' <repeats 255 times>}, 
            chng_pswd_auth_crap = {user = "\005\060ky\233\313\060\270 \b\000\000cekmekci", '\000' <repeats 235 times>, domain = '\000' <repeats 12 times>, "ED", '\000' <repeats 241 times>, 
              new_nt_pswd = '\000' <repeats 268 times>, "\030", '\000' <repeats 246 times>, new_nt_pswd_len = 0, old_nt_hash_enc = "\000\000\000\000\000\000\000\000\000\000,\001\000\000IS", old_nt_hash_enc_len = 19501, 
              new_lm_pswd = "AC-2046", '\000' <repeats 508 times>, new_lm_pswd_len = 0, old_lm_hash_enc = '\000' <repeats 15 times>, old_lm_hash_enc_len = 0}, logoff = {
              user = "\005\060ky\233\313\060\270 \b\000\000cekmekci", '\000' <repeats 235 times>, krb5ccname = '\000' <repeats 12 times>, "ED", '\000' <repeats 241 times>, uid = 0}, 
            sid = "\005\060ky\233\313\060\270 \b\000\000cekmekci", '\000' <repeats 235 times>, name = {dom_name = "\005\060ky\233\313\060\270 \b\000\000cekmekci", '\000' <repeats 235 times>, 
              name = '\000' <repeats 12 times>, "ED", '\000' <repeats 241 times>}, num_entries = 2037067781, acct_mgt = {username = "\005\060ky\233\313\060\270 \b\000\000cekmekci", '\000' <repeats 235 times>, 
              groupname = '\000' <repeats 12 times>, "ED", '\000' <repeats 241 times>}, init_conn = {is_primary = 5, dcname = "0ky\233\313\060\270 \b\000\000cekmekci", '\000' <repeats 236 times>}, dual_sid2id = {
              sid = "\005\060ky\233\313\060\270 \b\000\000cekmekci", '\000' <repeats 235 times>, name = '\000' <repeats 12 times>, "ED", '\000' <repeats 241 times>}, dual_idmapset = {
              sid = "\005\060ky\233\313\060\270 \b\000\000cekmekci", '\000' <repeats 235 times>, type = 0, id = 0}, list_all_domains = 5, ccache_ntlm_auth = {uid = 2037067781, 
              user = "\233\313\060\270 \b\000\000cekmekci", '\000' <repeats 239 times>, initial_blob_len = 0, challenge_blob_len = 0}, ccache_save = {uid = 2037067781, 
              user = "\233\313\060\270 \b\000\000cekmekci", '\000' <repeats 239 times>, pass = "\000\000\000\000\000\000\000\000ED", '\000' <repeats 245 times>}, dsgetdcname = {
              domain_name = "\005\060ky\233\313\060\270 \b\000\000cekmekci", '\000' <repeats 235 times>, domain_guid = '\000' <repeats 12 times>, "ED", '\000' <repeats 241 times>, site_name = '\000' <repeats 255 times>, 
              flags = 0}, 
            padding = "\005\060ky\233\313\060\270 \b\000\000cekmekci", '\000' <repeats 248 times>, "ED", '\000' <repeats 510 times>, "\030", '\000' <repeats 259 times>, ",\001\000\000IS-LAC-2046", '\000' <repeats 744 times>}, extra_data = {padding = 140154194472016, data = 0x7f7830f9ac50 "\260\035k\200\357\331Z$\204\211V\022e\334[\277\001\001"}, extra_len = 300, null_term = 0 '\000'}
        response = {length = 3496, result = WINBINDD_ERROR, data = {interface_version = 0, winsresp = '\000' <repeats 255 times>, pw = {pw_name = '\000' <repeats 255 times>, pw_passwd = '\000' <repeats 255 times>, 
              pw_uid = 0, pw_gid = 0, pw_gecos = '\000' <repeats 255 times>, pw_dir = '\000' <repeats 255 times>, pw_shell = '\000' <repeats 255 times>}, gr = {gr_name = '\000' <repeats 255 times>, 
              gr_passwd = '\000' <repeats 255 times>, gr_gid = 0, num_gr_mem = 0, gr_mem_ofs = 0}, num_entries = 0, sid = {sid = '\000' <repeats 255 times>, type = 0}, name = {dom_name = '\000' <repeats 255 times>, 
              name = '\000' <repeats 255 times>, type = 0}, uid = 0, gid = 0, info = {winbind_separator = 0 '\000', samba_version = '\000' <repeats 255 times>}, domain_name = '\000' <repeats 255 times>, 
            netbios_name = '\000' <repeats 255 times>, dc_name = '\000' <repeats 255 times>, auth = {nt_status = 0, nt_status_string = '\000' <repeats 255 times>, error_string = '\000' <repeats 255 times>, pam_error = 0, 
              user_session_key = '\000' <repeats 15 times>, first_8_lm_hash = "\000\000\000\000\000\000\000", krb5ccname = '\000' <repeats 255 times>, reject_reason = 0, padding = 0, policy = {min_length_password = 0, 
                password_history = 0, password_properties = 0, padding = 0, expire = 0, min_passwordage = 0}, info3 = {logon_time = 0, logoff_time = 0, kickoff_time = 0, pass_last_set_time = 0, pass_can_change_time = 0, 
                pass_must_change_time = 0, logon_count = 0, bad_pw_count = 0, user_rid = 0, group_rid = 0, num_groups = 0, user_flgs = 0, acct_flags = 0, num_other_sids = 0, dom_sid = '\000' <repeats 255 times>, 
                user_name = '\000' <repeats 255 times>, full_name = '\000' <repeats 255 times>, logon_script = '\000' <repeats 255 times>, profile_path = '\000' <repeats 255 times>, home_dir = '\000' <repeats 255 times>, 
                dir_drive = '\000' <repeats 255 times>, logon_srv = '\000' <repeats 255 times>, logon_dom = '\000' <repeats 255 times>}, unix_username = '\000' <repeats 255 times>}, domain_info = {
              name = '\000' <repeats 255 times>, alt_name = '\000' <repeats 255 times>, sid = '\000' <repeats 255 times>, native_mode = false, active_directory = false, primary = false}, sequence_number = 0, user_info = {
              acct_name = '\000' <repeats 255 times>, full_name = '\000' <repeats 255 times>, homedir = '\000' <repeats 255 times>, shell = '\000' <repeats 255 times>, primary_gid = 0, group_rid = 0}, ccache_ntlm_auth = {
              session_key = '\000' <repeats 15 times>, auth_blob_len = 0}, dsgetdcname = {dc_unc = '\000' <repeats 255 times>, dc_address = '\000' <repeats 255 times>, dc_address_type = 0, 
              domain_guid = '\000' <repeats 255 times>, domain_name = '\000' <repeats 255 times>, forest_name = '\000' <repeats 255 times>, dc_flags = 0, dc_site_name = '\000' <repeats 255 times>, 
              client_site_name = '\000' <repeats 255 times>}}, extra_data = {padding = 0, data = 0x0}}
        primary_domain = <value optimized out>
        status = {v = 0}
        nwritten = <value optimized out>
        fde = <value optimized out>
        __FUNCTION__ = "fork_domain_child"
#18 0x00007f782e90c03c in wb_child_request_trigger (req=0x7f7830fb4ea0, private_data=<value optimized out>) at ../source3/winbindd/winbindd_dual.c:146
        state = 0x7f7830fb6710
        subreq = <value optimized out>
#19 0x00007f782d7f930b in tevent_queue_immediate_trigger (ev=<value optimized out>, im=<value optimized out>, private_data=<value optimized out>) at ../lib/tevent/tevent_queue.c:148
        q = <value optimized out>
#20 0x00007f782d7f8f48 in tevent_common_loop_immediate (ev=0x7f7830bac4a0) at ../lib/tevent/tevent_immediate.c:135
        im = 0x7f7830bb4ad0
        handler = 0x7f782d7f92d6 <tevent_queue_immediate_trigger>
        private_data = 0x7f7830bb49b0
#21 0x00007f782d7fd816 in epoll_event_loop_once (ev=0x7f7830bac4a0, location=<value optimized out>) at ../lib/tevent/tevent_epoll.c:911
        epoll_ev = 0x7f7830badd00
        tval = {tv_sec = 0, tv_usec = 140154144951136}
        panic_triggered = false
#22 0x00007f782d7fbd26 in std_event_loop_once (ev=0x7f7830bac4a0, location=0x7f782e95add0 "../source3/winbindd/winbindd.c:1625") at ../lib/tevent/tevent_standard.c:112
        glue_ptr = <value optimized out>
        glue = 0x7f7830bac590
        ret = <value optimized out>
#23 0x00007f782d7f8407 in _tevent_loop_once (ev=0x7f7830bac4a0, location=0x7f782e95add0 "../source3/winbindd/winbindd.c:1625") at ../lib/tevent/tevent.c:530
        ret = <value optimized out>
        nesting_stack_ptr = 0x0
#24 0x00007f782e8e521a in main (argc=<value optimized out>, argv=<value optimized out>, envp=<value optimized out>) at ../source3/winbindd/winbindd.c:1625
        is_daemon = true
        Fork = true
        log_stdout = false
        no_process_group = false
        long_options = {{longName = 0x0, shortName = 0 '\000', argInfo = 4, arg = 0x7f782bf81cc0, val = 0, descrip = 0x7f782e95b058 "Help options:", argDescrip = 0x0}, {longName = 0x7f782e95b06d "stdout", 
            shortName = 83 'S', argInfo = 0, arg = 0x0, val = 1003, descrip = 0x7f782e95b066 "Log to stdout", argDescrip = 0x0}, {longName = 0x7f782e95b074 "foreground", shortName = 70 'F', argInfo = 0, arg = 0x0, 
            val = 1001, descrip = 0x7f782e95b07f "Daemon in foreground mode", argDescrip = 0x0}, {longName = 0x7f782e95b099 "no-process-group", shortName = 0 '\000', argInfo = 0, arg = 0x0, val = 1002, 
            descrip = 0x7f782e95a4e0 "Don't create a new process group", argDescrip = 0x0}, {longName = 0x7f782e99ace5 "daemon", shortName = 68 'D', argInfo = 0, arg = 0x0, val = 1000, 
            descrip = 0x7f782e95b0aa "Become a daemon (default)", argDescrip = 0x0}, {longName = 0x7f782e95b0c4 "interactive", shortName = 105 'i', argInfo = 0, arg = 0x0, val = 105, 
            descrip = 0x7f782e95b0d0 "Interactive mode", argDescrip = 0x0}, {longName = 0x7f782e95b0e1 "no-caching", shortName = 110 'n', argInfo = 0, arg = 0x0, val = 110, descrip = 0x7f782e95b0ec "Disable caching", 
            argDescrip = 0x0}, {longName = 0x0, shortName = 0 '\000', argInfo = 4, arg = 0x7f782e474860, val = 0, descrip = 0x7f782e95b0fc "Common samba options:", argDescrip = 0x0}, {longName = 0x0, shortName = 0 '\000', 
            argInfo = 4, arg = 0x7f782e474040, val = 0, descrip = 0x7f782e95a508 "Build-time configuration overrides:", argDescrip = 0x0}, {longName = 0x0, shortName = 0 '\000', argInfo = 0, arg = 0x0, val = 0, 
            descrip = 0x0, argDescrip = 0x0}}
        pc = <value optimized out>
        opt = <value optimized out>
        frame = 0x7f7830ba81a0
        status = <value optimized out>
        ok = <value optimized out>
        __FUNCTION__ = "main"
(gdb) up 9
#9  0x00007f782e8f2d14 in winbind_samlogon_retry_loop (domain=0x7f7830bb4150, mem_ctx=0x7f7830fb1d00, logon_parameters=2080, server=0x0, username=0x7fff9c4653f4 "cekmekci", domainname=0x7fff9c4654f4 "ED", 
    workstation=0x7fff9c4657fc "IS-LAC-2046", chal=0x7fff9c4653e8 "\005\060ky\233\313\060\270 \b", lm_response=..., nt_response=..., info3=0x7fff9c4642e8) at ../source3/winbindd/winbindd_pam.c:1349
1349	../source3/winbindd/winbindd_pam.c: No such file or directory.
	in ../source3/winbindd/winbindd_pam.c
(gdb) p *domain
$1 = {name = 0x7f7830badd90 "ED", alt_name = 0x7f7830bb4050 "ed.ac.uk", forest_name = 0x0, sid = {sid_rev_num = 1 '\001', num_auths = 4 '\004', id_auth = "\000\000\000\000\000\005", sub_auths = {21, 861567501, 1417001333, 
      682003330, 0 <repeats 11 times>}}, domain_flags = 29, domain_type = 2, domain_trust_attribs = 0, initialized = true, native_mode = true, active_directory = true, primary = true, internal = false, online = true, 
  startup_time = 29043666, startup = false, can_do_samlogon_ex = true, can_do_ncacn_ip_tcp = false, can_do_validation6 = false, methods = 0x7f782ebad020, backend = 0x0, private_data = 0x0, have_idmap_config = true, 
  id_range_low = 1001, id_range_high = 20000000, dc_probe_pid = 32059, dcname = 0x7f7830fa1ba0 "kelso.ed.ac.uk", dcaddr = {ss_family = 2, __ss_align = 0, __ss_padding = '\000' <repeats 111 times>}, last_seq_check = 0, 
  sequence_number = 4294967295, last_status = {v = 0}, conn = {cli = 0x7f7830fa2560, samr_pipe = 0x0, sam_connect_handle = {handle_type = 0, uuid = {time_low = 0, time_mid = 0, time_hi_and_version = 0, clock_seq = "\000", 
        node = "\000\000\000\000\000"}}, sam_domain_handle = {handle_type = 0, uuid = {time_low = 0, time_mid = 0, time_hi_and_version = 0, clock_seq = "\000", node = "\000\000\000\000\000"}}, lsa_pipe = 0x0, 
    lsa_pipe_tcp = 0x0, lsa_policy = {handle_type = 0, uuid = {time_low = 0, time_mid = 0, time_hi_and_version = 0, clock_seq = "\000", node = "\000\000\000\000\000"}}, netlogon_pipe = 0x7f7830f9c5c0}, 
  children = 0x7f7830bb4390, check_online_timeout = 0, check_online_event = 0x0, prev = 0x7f7830bb3340, next = 0x7f7830bbaa60}
Comment 8 Stefan Metzmacher 2015-06-24 09:49:10 UTC
(In reply to Orlando Richards from comment #7)

Thanks!

It's really strange that the argument 'server' of winbind_samlogon_retry_loop()
is NULL, while domain->dcname is valid. All callers just pass domain->dcname
into the server argument...
Comment 9 Stefan Metzmacher 2015-06-24 09:50:10 UTC
Created attachment 11189 [details]
test patch for 4.1.16

As a wild guess can you try the following patch?
Comment 10 Stefan Metzmacher 2015-06-24 09:52:05 UTC
(In reply to Stefan (metze) Metzmacher from comment #8)

Ah, maybe this.

domain->dcname is filled via cm_connect_netlogon() within
winbind_samlogon_retry_loop(), while it was NULL when calling
winbind_samlogon_retry_loop().
Comment 11 Stefan Metzmacher 2015-06-24 09:59:22 UTC
Created attachment 11190 [details]
Patches for v4-1-test

I really think this is exactly the bug here.

The 'server' argument is unused in master and v4-2-test,
so we only need a v4-1 patch here.
Comment 12 Stefan Metzmacher 2015-06-24 10:29:41 UTC
Created attachment 11191 [details]
Cleanup patch for master
Comment 13 Guenther Deschner 2015-06-24 11:20:31 UTC
Comment on attachment 11190 [details]
Patches for v4-1-test

LGTM
Comment 14 Guenther Deschner 2015-06-24 11:20:45 UTC
Comment on attachment 11191 [details]
Cleanup patch for master

LGTM
Comment 15 Andreas Schneider 2015-06-26 08:44:24 UTC
Karolin, please add the patch to the 4.1 branch, thanks!
Comment 16 Karolin Seeger 2015-06-29 20:09:17 UTC
Pushed to autobuild-v4-1-test.
Comment 17 Karolin Seeger 2015-07-05 19:00:03 UTC
(In reply to Karolin Seeger from comment #16)
Pushed to v4-1-test.
Closing out bug report.

Thanks!