Bug 15110 - Segmentation fault in krb5_plugin_register since samba-4.16rc1 on x86 (32 bit), regression from samba-4.15
Summary: Segmentation fault in krb5_plugin_register since samba-4.16rc1 on x86 (32 bit...
Status: RESOLVED FIXED
Alias: None
Product: Samba 4.1 and newer
Classification: Unclassified
Component: File services (show other bugs)
Version: 4.16.2
Hardware: x86 Linux
: P5 regression with 11 votes (vote)
Target Milestone: ---
Assignee: Jule Anger
QA Contact: Samba QA Contact
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2022-06-28 16:00 UTC by Krzysztof Olędzki
Modified: 2022-12-15 16:36 UTC (History)
7 users (show)

See Also:

Attachments
Tar Gz archive with gdb_backtrace logs (7.61 KB, application/x-compressed)
2022-06-29 09:01 UTC, Krzysztof Olędzki 		no flags Details
kdc: fix Segmentation fault due to struct mismatch (4.03 KB, patch)
2022-09-20 15:54 UTC, Krzysztof Olędzki 		no flags Details
patch for Samba 4.16 (18.40 KB, patch)
2022-11-03 00:56 UTC, Jo Sutton 		abartlet: review+
jsutton: ci-passed+
Details
patch for Samba 4.17 (18.40 KB, patch)
2022-11-03 01:58 UTC, Jo Sutton 		abartlet: review+
jsutton: ci-passed+
Details
View All Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Krzysztof Olędzki 2022-06-28 16:00:08 UTC 
Starting samba-4.16rc1 (so far the oldest version I have tested) samba crashes during start on to similar systems.

While this is a little bit less standard setup (x86_64 - 64 bit kernel, x86 - 32 bit userspace, including samba itself) everything worked well over the last many years.

I'm mentioning this as on a very similar, native x86_64 system everything works fine. Note that I have not tested 32 bit kernel, but can try if needed - perhaps this is a problem with a 32 bit binary?

More about my system:
 server role = active directory domain controller
 server services = s3fs, rpc, nbt, wrepl, ldap, cldap, kdc, drepl, winbindd, ntp_signd, kcc, dnsupdate

Crash log sorted by pid:
Jun 28 17:46:03 server samba[17718]: [2022/06/28 17:46:03.707209,  0] ../../lib/util/fault.c:172(smb_panic_log)
Jun 28 17:46:03 server samba[17718]:   ===============================================================
Jun 28 17:46:03 server samba[17718]: [2022/06/28 17:46:03.707283,  0] ../../lib/util/fault.c:173(smb_panic_log)
Jun 28 17:46:03 server samba[17718]:   INTERNAL ERROR: Signal 11: Segmentation fault in pid 17718 (4.16.2)
Jun 28 17:46:03 server samba[17718]: [2022/06/28 17:46:03.707313,  0] ../../lib/util/fault.c:177(smb_panic_log)
Jun 28 17:46:03 server samba[17718]:   If you are running a recent Samba version, and if you think this problem is not yet fixed in the latest versions, please consider reporting this bug, see https://wiki.samba.org/index.php/Bug_Reporting
Jun 28 17:46:03 server samba[17718]: [2022/06/28 17:46:03.707340,  0] ../../lib/util/fault.c:182(smb_panic_log)
Jun 28 17:46:03 server samba[17718]:   ===============================================================
Jun 28 17:46:03 server samba[17718]: [2022/06/28 17:46:03.707361,  0] ../../lib/util/fault.c:183(smb_panic_log)
Jun 28 17:46:03 server samba[17718]:   PANIC (pid 17718): Signal 11: Segmentation fault in 4.16.2
Jun 28 17:46:03 server samba[17718]: [2022/06/28 17:46:03.707518,  0] ../../lib/util/fault.c:245(log_stack_trace)
Jun 28 17:46:03 server samba[17718]:   BACKTRACE:
Jun 28 17:46:03 server samba[17718]:    #0 log_stack_trace + 0x36 [ip=0xf7ec6226] [sp=0xffffbbc0]
Jun 28 17:46:04 server samba[17718]:    #1 smb_panic_log + 0x76 [ip=0xf7ec6526] [sp=0xffffc090]
Jun 28 17:46:04 server samba[17718]:    #2 smb_panic + 0x1a [ip=0xf7ec66ca] [sp=0xffffc0a0]
Jun 28 17:46:04 server samba[17718]:    #3 smb_panic + 0xc0 [ip=0xf7ec6770] [sp=0xffffc0c0]
Jun 28 17:46:04 server samba[17718]:    #4 <unknown symbol> [ip=0xf7fc6930] [sp=0xffffc170]
Jun 28 17:46:05 server samba[17714]:   prefork_child_pipe_handler: Parent 17714, Child 17718 terminated with signal 6
Jun 28 17:46:03 server samba[17722]: [2022/06/28 17:46:03.710452,  0] ../../lib/util/fault.c:172(smb_panic_log)
Jun 28 17:46:03 server samba[17722]:   ===============================================================
Jun 28 17:46:04 server samba[17722]: [2022/06/28 17:46:04.130017,  0] ../../lib/util/fault.c:173(smb_panic_log)
Jun 28 17:46:04 server samba[17722]:   INTERNAL ERROR: Signal 11: Segmentation fault in pid 17722 (4.16.2)
Jun 28 17:46:04 server samba[17722]: [2022/06/28 17:46:04.649636,  0] ../../lib/util/fault.c:177(smb_panic_log)
Jun 28 17:46:04 server samba[17722]:   If you are running a recent Samba version, and if you think this problem is not yet fixed in the latest versions, please consider reporting this bug, see https://wiki.samba.org/index.php/Bug_Reporting
Jun 28 17:46:04 server samba[17722]: [2022/06/28 17:46:04.985653,  0] ../../lib/util/fault.c:182(smb_panic_log)
Jun 28 17:46:05 server samba[17722]:   ===============================================================
Jun 28 17:46:05 server samba[17722]: [2022/06/28 17:46:05.738297,  0] ../../lib/util/fault.c:183(smb_panic_log)
Jun 28 17:46:05 server samba[17722]:   PANIC (pid 17722): Signal 11: Segmentation fault in 4.16.2
Jun 28 17:46:05 server samba[17722]: [2022/06/28 17:46:05.948046,  0] ../../lib/util/fault.c:245(log_stack_trace)
Jun 28 17:46:06 server samba[17722]:   BACKTRACE:
Jun 28 17:46:06 server samba[17722]:    #0 log_stack_trace + 0x36 [ip=0xf7ec6226] [sp=0xffffbbc0]
Jun 28 17:46:06 server samba[17722]:    #1 smb_panic_log + 0x76 [ip=0xf7ec6526] [sp=0xffffc090]
Jun 28 17:46:06 server samba[17722]:    #2 smb_panic + 0x1a [ip=0xf7ec66ca] [sp=0xffffc0a0]
Jun 28 17:46:06 server samba[17722]:    #3 smb_panic + 0xc0 [ip=0xf7ec6770] [sp=0xffffc0c0]
Jun 28 17:46:06 server samba[17722]:    #4 <unknown symbol> [ip=0xf7fc6930] [sp=0xffffc170]
Jun 28 17:46:07 server samba[17714]:   prefork_child_pipe_handler: Parent 17714, Child 17722 terminated with signal 6
Jun 28 17:46:03 server samba[17731]: [2022/06/28 17:46:03.719536,  0] ../../lib/util/fault.c:172(smb_panic_log)
Jun 28 17:46:03 server samba[17731]:   ===============================================================
Jun 28 17:46:04 server samba[17731]: [2022/06/28 17:46:04.171919,  0] ../../lib/util/fault.c:173(smb_panic_log)
Jun 28 17:46:04 server samba[17731]:   INTERNAL ERROR: Signal 11: Segmentation fault in pid 17731 (4.16.2)
Jun 28 17:46:04 server samba[17731]: [2022/06/28 17:46:04.691855,  0] ../../lib/util/fault.c:177(smb_panic_log)
Jun 28 17:46:04 server samba[17731]:   If you are running a recent Samba version, and if you think this problem is not yet fixed in the latest versions, please consider reporting this bug, see https://wiki.samba.org/index.php/Bug_Reporting
Jun 28 17:46:05 server samba[17731]: [2022/06/28 17:46:05.078133,  0] ../../lib/util/fault.c:182(smb_panic_log)
Jun 28 17:46:05 server samba[17731]:   ===============================================================
Jun 28 17:46:05 server samba[17731]: [2022/06/28 17:46:05.763484,  0] ../../lib/util/fault.c:183(smb_panic_log)
Jun 28 17:46:05 server samba[17731]:   PANIC (pid 17731): Signal 11: Segmentation fault in 4.16.2
Jun 28 17:46:05 server samba[17731]: [2022/06/28 17:46:05.964936,  0] ../../lib/util/fault.c:245(log_stack_trace)
Jun 28 17:46:06 server samba[17731]:   BACKTRACE:
Jun 28 17:46:06 server samba[17731]:    #0 log_stack_trace + 0x36 [ip=0xf7ec6226] [sp=0xffffbbc0]
Jun 28 17:46:06 server samba[17731]:    #1 smb_panic_log + 0x76 [ip=0xf7ec6526] [sp=0xffffc090]
Jun 28 17:46:06 server samba[17731]:    #2 smb_panic + 0x1a [ip=0xf7ec66ca] [sp=0xffffc0a0]
Jun 28 17:46:06 server samba[17731]:    #3 smb_panic + 0xc0 [ip=0xf7ec6770] [sp=0xffffc0c0]
Jun 28 17:46:06 server samba[17731]:    #4 <unknown symbol> [ip=0xf7fc6930] [sp=0xffffc170]
Jun 28 17:46:07 server samba[17714]:   prefork_child_pipe_handler: Parent 17714, Child 17731 terminated with signal 6
Jun 28 17:46:03 server samba[17734]: [2022/06/28 17:46:03.721018,  0] ../../lib/util/fault.c:172(smb_panic_log)
Jun 28 17:46:03 server samba[17734]:   ===============================================================
Jun 28 17:46:04 server samba[17734]: [2022/06/28 17:46:04.205509,  0] ../../lib/util/fault.c:173(smb_panic_log)
Jun 28 17:46:04 server samba[17734]:   INTERNAL ERROR: Signal 11: Segmentation fault in pid 17734 (4.16.2)
Jun 28 17:46:04 server samba[17734]: [2022/06/28 17:46:04.708589,  0] ../../lib/util/fault.c:177(smb_panic_log)
Jun 28 17:46:04 server samba[17734]:   If you are running a recent Samba version, and if you think this problem is not yet fixed in the latest versions, please consider reporting this bug, see https://wiki.samba.org/index.php/Bug_Reporting
Jun 28 17:46:05 server samba[17734]: [2022/06/28 17:46:05.111732,  0] ../../lib/util/fault.c:182(smb_panic_log)
Jun 28 17:46:05 server samba[17734]:   ===============================================================
Jun 28 17:46:05 server samba[17734]: [2022/06/28 17:46:05.788658,  0] ../../lib/util/fault.c:183(smb_panic_log)
Jun 28 17:46:05 server samba[17734]:   PANIC (pid 17734): Signal 11: Segmentation fault in 4.16.2
Jun 28 17:46:05 server samba[17734]: [2022/06/28 17:46:05.981807,  0] ../../lib/util/fault.c:245(log_stack_trace)
Jun 28 17:46:06 server samba[17734]:   BACKTRACE:
Jun 28 17:46:06 server samba[17734]:    #0 log_stack_trace + 0x36 [ip=0xf7ec6226] [sp=0xffffbbc0]
Jun 28 17:46:06 server samba[17734]:    #1 smb_panic_log + 0x76 [ip=0xf7ec6526] [sp=0xffffc090]
Jun 28 17:46:06 server samba[17734]:    #2 smb_panic + 0x1a [ip=0xf7ec66ca] [sp=0xffffc0a0]
Jun 28 17:46:06 server samba[17734]:    #3 smb_panic + 0xc0 [ip=0xf7ec6770] [sp=0xffffc0c0]
Jun 28 17:46:06 server samba[17734]:    #4 <unknown symbol> [ip=0xf7fc6930] [sp=0xffffc170]
Jun 28 17:46:08 server samba[17714]:   prefork_child_pipe_handler: Parent 17714, Child 17734 terminated with signal 6
Jun 28 17:46:05 server samba[17761]: [2022/06/28 17:46:05.939204,  0] ../../lib/util/fault.c:172(smb_panic_log)
Jun 28 17:46:05 server samba[17761]:   ===============================================================
Jun 28 17:46:06 server samba[17761]: [2022/06/28 17:46:06.065688,  0] ../../lib/util/fault.c:173(smb_panic_log)
Jun 28 17:46:06 server samba[17761]:   INTERNAL ERROR: Signal 11: Segmentation fault in pid 17761 (4.16.2)
Jun 28 17:46:06 server samba[17761]: [2022/06/28 17:46:06.260165,  0] ../../lib/util/fault.c:177(smb_panic_log)
Jun 28 17:46:06 server samba[17761]:   If you are running a recent Samba version, and if you think this problem is not yet fixed in the latest versions, please consider reporting this bug, see https://wiki.samba.org/index.php/Bug_Reporting
Jun 28 17:46:06 server samba[17761]: [2022/06/28 17:46:06.804294,  0] ../../lib/util/fault.c:182(smb_panic_log)
Jun 28 17:46:07 server samba[17761]:   ===============================================================
Jun 28 17:46:07 server samba[17761]: [2022/06/28 17:46:07.243201,  0] ../../lib/util/fault.c:183(smb_panic_log)
Jun 28 17:46:07 server samba[17761]:   PANIC (pid 17761): Signal 11: Segmentation fault in 4.16.2
Jun 28 17:46:07 server samba[17761]: [2022/06/28 17:46:07.503917,  0] ../../lib/util/fault.c:245(log_stack_trace)
Jun 28 17:46:07 server samba[17761]:   BACKTRACE:
Jun 28 17:46:07 server samba[17761]:    #0 log_stack_trace + 0x36 [ip=0xf7ec6226] [sp=0xffffb9c0]
Jun 28 17:46:07 server samba[17761]:    #1 smb_panic_log + 0x76 [ip=0xf7ec6526] [sp=0xffffbe90]
Jun 28 17:46:08 server samba[17761]:    #2 smb_panic + 0x1a [ip=0xf7ec66ca] [sp=0xffffbea0]
Jun 28 17:46:08 server samba[17761]:    #3 smb_panic + 0xc0 [ip=0xf7ec6770] [sp=0xffffbec0]
Jun 28 17:46:08 server samba[17761]:    #4 <unknown symbol> [ip=0xf7fc6930] [sp=0xffffbf70]
Jun 28 17:46:08 server samba[17714]:   prefork_child_pipe_handler: Parent 17714, Child 17761 terminated with signal 6
Jun 28 17:46:07 server samba[17763]: [2022/06/28 17:46:07.696804,  0] ../../lib/util/fault.c:172(smb_panic_log)
Jun 28 17:46:07 server samba[17763]:   ===============================================================
Jun 28 17:46:07 server samba[17763]: [2022/06/28 17:46:07.950095,  0] ../../lib/util/fault.c:173(smb_panic_log)
Jun 28 17:46:08 server samba[17763]:   INTERNAL ERROR: Signal 11: Segmentation fault in pid 17763 (4.16.2)
Jun 28 17:46:08 server samba[17763]: [2022/06/28 17:46:08.201961,  0] ../../lib/util/fault.c:177(smb_panic_log)
Jun 28 17:46:08 server samba[17763]:   If you are running a recent Samba version, and if you think this problem is not yet fixed in the latest versions, please consider reporting this bug, see https://wiki.samba.org/index.php/Bug_Reporting
Jun 28 17:46:08 server samba[17763]: [2022/06/28 17:46:08.336334,  0] ../../lib/util/fault.c:182(smb_panic_log)
Jun 28 17:46:08 server samba[17763]:   ===============================================================
Jun 28 17:46:08 server samba[17763]: [2022/06/28 17:46:08.528943,  0] ../../lib/util/fault.c:183(smb_panic_log)
Jun 28 17:46:08 server samba[17763]:   PANIC (pid 17763): Signal 11: Segmentation fault in 4.16.2
Jun 28 17:46:08 server samba[17763]: [2022/06/28 17:46:08.680039,  0] ../../lib/util/fault.c:245(log_stack_trace)
Jun 28 17:46:08 server samba[17763]:   BACKTRACE:
Jun 28 17:46:08 server samba[17763]:    #0 log_stack_trace + 0x36 [ip=0xf7ec6226] [sp=0xffffb9c0]
Jun 28 17:46:08 server samba[17763]:    #1 smb_panic_log + 0x76 [ip=0xf7ec6526] [sp=0xffffbe90]
Jun 28 17:46:08 server samba[17763]:    #2 smb_panic + 0x1a [ip=0xf7ec66ca] [sp=0xffffbea0]
Jun 28 17:46:09 server samba[17763]:    #3 smb_panic + 0xc0 [ip=0xf7ec6770] [sp=0xffffbec0]
Jun 28 17:46:09 server samba[17763]:    #4 <unknown symbol> [ip=0xf7fc6930] [sp=0xffffbf70]
Jun 28 17:46:09 server samba[17714]:   prefork_child_pipe_handler: Parent 17714, Child 17763 terminated with signal 6
Jun 28 17:46:08 server samba[17765]: [2022/06/28 17:46:08.193266,  0] ../../lib/util/fault.c:172(smb_panic_log)
Jun 28 17:46:08 server samba[17765]:   ===============================================================
Jun 28 17:46:08 server samba[17765]: [2022/06/28 17:46:08.319538,  0] ../../lib/util/fault.c:173(smb_panic_log)
Jun 28 17:46:08 server samba[17765]:   INTERNAL ERROR: Signal 11: Segmentation fault in pid 17765 (4.16.2)
Jun 28 17:46:08 server samba[17765]: [2022/06/28 17:46:08.512017,  0] ../../lib/util/fault.c:177(smb_panic_log)
Jun 28 17:46:08 server samba[17765]:   If you are running a recent Samba version, and if you think this problem is not yet fixed in the latest versions, please consider reporting this bug, see https://wiki.samba.org/index.php/Bug_Reporting
Jun 28 17:46:08 server samba[17765]: [2022/06/28 17:46:08.654865,  0] ../../lib/util/fault.c:182(smb_panic_log)
Jun 28 17:46:08 server samba[17765]:   ===============================================================
Jun 28 17:46:08 server samba[17765]: [2022/06/28 17:46:08.805916,  0] ../../lib/util/fault.c:183(smb_panic_log)
Jun 28 17:46:08 server samba[17765]:   PANIC (pid 17765): Signal 11: Segmentation fault in 4.16.2
Jun 28 17:46:08 server samba[17765]: [2022/06/28 17:46:08.940097,  0] ../../lib/util/fault.c:245(log_stack_trace)
Jun 28 17:46:09 server samba[17765]:   BACKTRACE:
Jun 28 17:46:09 server samba[17765]:    #0 log_stack_trace + 0x36 [ip=0xf7ec6226] [sp=0xffffb9c0]
Jun 28 17:46:09 server samba[17765]:    #1 smb_panic_log + 0x76 [ip=0xf7ec6526] [sp=0xffffbe90]
Jun 28 17:46:09 server samba[17765]:    #2 smb_panic + 0x1a [ip=0xf7ec66ca] [sp=0xffffbea0]
Jun 28 17:46:09 server samba[17765]:    #3 smb_panic + 0xc0 [ip=0xf7ec6770] [sp=0xffffbec0]
Jun 28 17:46:09 server samba[17765]:    #4 <unknown symbol> [ip=0xf7fc6930] [sp=0xffffbf70]
Jun 28 17:46:10 server samba[17714]:   prefork_child_pipe_handler: Parent 17714, Child 17765 terminated with signal 6
Jun 28 17:46:08 server samba[17767]: [2022/06/28 17:46:08.502632,  0] ../../lib/util/fault.c:172(smb_panic_log)
Jun 28 17:46:08 server samba[17767]:   ===============================================================
Jun 28 17:46:08 server samba[17767]: [2022/06/28 17:46:08.638296,  0] ../../lib/util/fault.c:173(smb_panic_log)
Jun 28 17:46:08 server samba[17767]:   INTERNAL ERROR: Signal 11: Segmentation fault in pid 17767 (4.16.2)
Jun 28 17:46:08 server samba[17767]: [2022/06/28 17:46:08.789042,  0] ../../lib/util/fault.c:177(smb_panic_log)
Jun 28 17:46:08 server samba[17767]:   If you are running a recent Samba version, and if you think this problem is not yet fixed in the latest versions, please consider reporting this bug, see https://wiki.samba.org/index.php/Bug_Reporting
Jun 28 17:46:08 server samba[17767]: [2022/06/28 17:46:08.923172,  0] ../../lib/util/fault.c:182(smb_panic_log)
Jun 28 17:46:09 server samba[17767]:   ===============================================================
Jun 28 17:46:09 server samba[17767]: [2022/06/28 17:46:09.107919,  0] ../../lib/util/fault.c:183(smb_panic_log)
Jun 28 17:46:09 server samba[17767]:   PANIC (pid 17767): Signal 11: Segmentation fault in 4.16.2
Jun 28 17:46:09 server samba[17767]: [2022/06/28 17:46:09.317480,  0] ../../lib/util/fault.c:245(log_stack_trace)
Jun 28 17:46:09 server samba[17767]:   BACKTRACE:
Jun 28 17:46:09 server samba[17767]:    #0 log_stack_trace + 0x36 [ip=0xf7ec6226] [sp=0xffffb9c0]
Jun 28 17:46:09 server samba[17767]:    #1 smb_panic_log + 0x76 [ip=0xf7ec6526] [sp=0xffffbe90]
Jun 28 17:46:10 server samba[17767]:    #2 smb_panic + 0x1a [ip=0xf7ec66ca] [sp=0xffffbea0]
Jun 28 17:46:10 server samba[17767]:    #3 smb_panic + 0xc0 [ip=0xf7ec6770] [sp=0xffffbec0]
Jun 28 17:46:10 server samba[17767]:    #4 <unknown symbol> [ip=0xf7fc6930] [sp=0xffffbf70]
Jun 28 17:46:10 server samba[17714]:   prefork_child_pipe_handler: Parent 17714, Child 17767 terminated with signal 6


samba and most of the libraries are built with debug symbols, but there is not much in the backtrace.

(gdb) bt 0xf7fc6930
#0  0xf7fc6919 in __kernel_vsyscall ()
#1  0xf79e7653 in epoll_wait () from /lib/libc.so.6
#2  0xf7b071e8 in epoll_event_loop (tvalp=0xffffca08, epoll_ev=0x565f9b20) at ../../tevent_epoll.c:650
#3  epoll_event_loop_once (ev=0x565f99f0, location=0x5655be28 "../../source4/samba/server.c:965") at ../../tevent_epoll.c:937
#4  0xf7b05322 in std_event_loop_once (ev=0x565f99f0, location=0x5655be28 "../../source4/samba/server.c:965") at ../../tevent_standard.c:110
#5  0xf7aff964 in _tevent_loop_once (ev=0x565f99f0, location=0x5655be28 "../../source4/samba/server.c:965") at ../../tevent.c:790
#6  0xf7affc2a in tevent_common_loop_wait (ev=0x565f99f0, location=0x5655be28 "../../source4/samba/server.c:965") at ../../tevent.c:913
#7  0xf7b052b2 in std_event_loop_wait (ev=0x565f99f0, location=0x5655be28 "../../source4/samba/server.c:965") at ../../tevent_standard.c:141
#8  0x56559e37 in binary_smbd_main (mem_ctx=0x56573460, binary_name=0x5655c06b "samba", argc=2, argv=0xffffce54, binary_name=0x5655c06b "samba") at ../../source4/samba/server.c:965
#9  0x56558821 in main (argc=2, argv=0xffffce54) at ../../source4/samba/server.c:986
(gdb) bt 0xffffbf70
#0  0xf7fc6919 in __kernel_vsyscall ()
#1  0xf79e7653 in epoll_wait () from /lib/libc.so.6
#2  0xf7b071e8 in epoll_event_loop (tvalp=0xffffca08, epoll_ev=0x565f9b20) at ../../tevent_epoll.c:650
#3  epoll_event_loop_once (ev=0x565f99f0, location=0x5655be28 "../../source4/samba/server.c:965") at ../../tevent_epoll.c:937
#4  0xf7b05322 in std_event_loop_once (ev=0x565f99f0, location=0x5655be28 "../../source4/samba/server.c:965") at ../../tevent_standard.c:110
#5  0xf7aff964 in _tevent_loop_once (ev=0x565f99f0, location=0x5655be28 "../../source4/samba/server.c:965") at ../../tevent.c:790
#6  0xf7affc2a in tevent_common_loop_wait (ev=0x565f99f0, location=0x5655be28 "../../source4/samba/server.c:965") at ../../tevent.c:913
#7  0xf7b052b2 in std_event_loop_wait (ev=0x565f99f0, location=0x5655be28 "../../source4/samba/server.c:965") at ../../tevent_standard.c:141
#8  0x56559e37 in binary_smbd_main (mem_ctx=0x56573460, binary_name=0x5655c06b "samba", argc=2, argv=0xffffce54, binary_name=0x5655c06b "samba") at ../../source4/samba/server.c:965
#9  0x56558821 in main (argc=2, argv=0xffffce54) at ../../source4/samba/server.c:986
Comment 1 Douglas Bagnall 2022-06-28 22:36:42 UTC 
If you add 

   panic action = /usr/local/bin/gdb_backtrace %d

to the smb.conf, you will get more[*]. Or use 'bt full' in gdb. How much it helps in this case is another question.

*  if /usr/local/bin is the wrong path, change it!
Comment 2 Krzysztof Olędzki 2022-06-29 09:01:19 UTC 
Created attachment 17400 [details]
Tar Gz archive with gdb_backtrace logs
Comment 3 Krzysztof Olędzki 2022-06-29 09:04:57 UTC 
Thanks, this is very useful! 

I had to modify the script, as I'm getting multiple crashes at the same time, but it worked well overall. I have attached an archive with the gdb_backtrace logs.

Also, here is a sample Valgrind output from a process that crashes:

--2863-- REDIR: 0x4dda720 (libc.so.6:__memchr_sse2_bsf) redirected to 0x484bb00 (memchr)
--2863-- REDIR: 0x4ddc3e0 (libc.so.6:__GI_stpcpy) redirected to 0x484dcc0 (__GI_stpcpy)
==2863== Jump to the invalid address stated on the next line
==2863==    at 0x0: ???
==2863==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
==2863==
--2863-- REDIR: 0x4dc30d0 (libc.so.6:stpcpy) redirected to 0x483b1f0 (_vgnU_ifunc_wrapper)
--2863-- REDIR: 0x4dcd930 (libc.so.6:__stpcpy_ssse3) redirected to 0x484dc00 (stpcpy)
==2863==
==2863== Process terminating with default action of signal 6 (SIGABRT)
==2863==    at 0x4DB03E7: __pthread_kill_implementation (pthread_kill.c:44)
==2863==    by 0x4D60A40: raise (raise.c:26)
==2863==    by 0x4D4A271: abort (abort.c:79)
==2863==    by 0x48EEFC1: smb_panic_default.constprop.0 (fault.c:167)
==2863==    by 0x48EF6ED: smb_panic (fault.c:200)
==2863==    by 0x48EF76F: fault_report (fault.c:81)
==2863==    by 0x48EF76F: sig_fault (fault.c:92)
==2863==    by 0x4D60AF7: ??? (in /lib/libc.so.6)
Comment 4 Krzysztof Olędzki 2022-06-29 09:10:51 UTC 
I think all the traces look the same, and the crash happens around:

#11 0x00000000 in ?? ()
No symbol table info available.
#12 0xf6185b8e in heim_plugin_register (context=0x58331420, pcontext=0x58e19910, module=0xf6bc643a "krb5", name=0xf2ff8a07 "hdb_samba4_interface", ftable=0xf2fe9004 <hdb_samba4_interface>) at ../../third_party/heimdal/lib/base/plugin.c:235
        pl = 0x58b06a80
        ret = 0
        plugins = 0x58ca31d0
        hname = 0x58ca3190
        dso = 0x5860db30
        ctx = {symbol = 0xf2fe9004 <hdb_samba4_interface>, is_dup = 0}

plugin.c:235 is:
  ret = pl->ftable->init(pcontext, &pl->ctx);
Comment 5 Krzysztof Olędzki 2022-06-29 10:34:06 UTC 
With the following patch:

--- a/third_party/heimdal/lib/base/plugin.c     2022-06-29 11:28:04.196263495 +0200
+++ b/third_party/heimdal/lib/base/plugin.c     2022-06-29 12:06:07.955431592 +0200
@@ -232,12 +232,29 @@
             ret = heim_enomem(context);
         } else {
             pl->ftable = ftable;
-            ret = pl->ftable->init(pcontext, &pl->ctx);
-            if (ret == 0) {
-                heim_array_append_value(plugins, pl);
-                heim_debug(context, 5, "Registered %s plugin", name);
+
+            printf("Plugin %s, pl: %u\n", name, pl);
+            printf("Plugin %s, pl->ftable: %u\n", name, pl->ftable);
+            printf("Plugin %s, pl->ftable->init: %u\n", name, pl->ftable->init);
+
+            heim_debug(context, 0, "Plugin %s, pl: %u", name, pl);
+            heim_debug(context, 0, "Plugin %s, pl->ftable: %u", name, pl->ftable);
+            heim_debug(context, 0, "Plugin %s, pl->ftable->init: %u", name, pl->ftable->init);
+
+            heim_debug(context, 200, "Plugin %s, pl: %u", name, pl);
+            heim_debug(context, 200, "Plugin %s, pl->ftable: %u", name, pl->ftable);
+            heim_debug(context, 200, "Plugin %s, pl->ftable->init: %u", name, pl->ftable->init);
+
+            if (!pl->ftable->init) {
+              ret = heim_enomem(context);
+            } else {
+              ret = pl->ftable->init(pcontext, &pl->ctx);
+              if (ret == 0) {
+                  heim_array_append_value(plugins, pl);
+                  heim_debug(context, 5, "Registered %s plugin", name);
+              }
+              heim_release(pl);
             }
-            heim_release(pl);
         }
     }

I ended up with the following output:

Plugin send_to_kdc, pl: 1483976624
Plugin send_to_kdc, pl->ftable: 4148170772
Plugin send_to_kdc, pl->ftable->init: 4148086000
Plugin hdb_samba4_interface, pl: 1487279248
Plugin hdb_samba4_interface, pl->ftable: 4076302340
Plugin hdb_samba4_interface, pl->ftable->init: 0
Plugin send_to_kdc, pl: 1485860480
Plugin send_to_kdc, pl->ftable: 4148170772
Plugin send_to_kdc, pl->ftable->init: 4148086000
Plugin hdb_samba4_interface, pl: 1483976624
Plugin hdb_samba4_interface, pl->ftable: 4076302340
Plugin hdb_samba4_interface, pl->ftable->init: 0
Plugin send_to_kdc, pl: 1483976624
Plugin send_to_kdc, pl->ftable: 4148170772
Plugin send_to_kdc, pl->ftable->init: 4148086000
Plugin hdb_samba4_interface, pl: 1487279248
Plugin hdb_samba4_interface, pl->ftable: 4076302340
Plugin hdb_samba4_interface, pl->ftable->init: 0
Plugin send_to_kdc, pl: 1487279248
Plugin send_to_kdc, pl->ftable: 4148170772
Plugin send_to_kdc, pl->ftable->init: 4148086000
Plugin hdb_samba4_interface, pl: 1487628800
Plugin hdb_samba4_interface, pl->ftable: 4076302340
Plugin hdb_samba4_interface, pl->ftable->init: 0

... and error:
task_server_terminate: task_server_terminate: [kdc: failed to register hdb plugin]

[Yes, I have been unable to make heim_debug working]

So, pl->ftable->init is NULL when called with hdb_samba4_interface. Now the question why...
Comment 6 Krzysztof Olędzki 2022-06-29 16:14:13 UTC 
Maybe I'm incorrectly reading the code, but it seems like we have struct mismatch?

struct heim_plugin_common_ftable_desc {
    int                 version;
    int                 (HEIM_LIB_CALL *init)(heim_pcontext, void **);
    void                (HEIM_LIB_CALL *fini)(void *);
};


struct hdb_method {
    int                 version;
    unsigned int        is_file_based:1;
    unsigned int        can_taste:1;
    krb5_error_code     (*init)(krb5_context, void **);
    void                (*fini)(void *);
    const char *prefix;
    krb5_error_code (*create)(krb5_context, HDB **, const char *filename);
};

... but on x86-64, init/finit happen to be aligned to the same address?
Comment 7 Krzysztof Olędzki 2022-06-29 18:11:02 UTC 
Seems line nobody uses samba4+AD on 32 bit these days... :/

This patch, while I'm sure 100% incorrect, fixes the crash:

diff -Nur samba-4.16.2-orig/source4/kdc/hdb-samba4-plugin.c samba-4.16.2/source4/kdc/hdb-samba4-plugin.c
--- samba-4.16.2-orig/source4/kdc/hdb-samba4-plugin.c	2022-01-24 11:26:59.333308700 +0100
+++ samba-4.16.2/source4/kdc/hdb-samba4-plugin.c	2022-06-29 19:48:01.264927045 +0200
@@ -95,12 +95,8 @@
  *
  * The <address> is the string form of a pointer to a talloced struct hdb_samba_context
  */
-struct hdb_method hdb_samba4_interface = {
+struct heim_plugin_common_ftable_desc hdb_samba4_interface = {
 	HDB_INTERFACE_VERSION,
-#if HDB_INTERFACE_VERSION >= 8
 	.init = hdb_samba4_init,
 	.fini = hdb_samba4_fini,
-#endif
-	.prefix = "samba4",
-	.create = hdb_samba4_create
 };
diff -Nur samba-4.16.2-orig/source4/kdc/samba_kdc.h samba-4.16.2/source4/kdc/samba_kdc.h
--- samba-4.16.2-orig/source4/kdc/samba_kdc.h	2022-03-21 13:13:09.604498100 +0100
+++ samba-4.16.2/source4/kdc/samba_kdc.h	2022-06-29 19:38:05.492037885 +0200
@@ -21,6 +21,8 @@
    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 */
 
+#include "third_party/heimdal/lib/base/common_plugin.h"
+
 #ifndef _SAMBA_KDC_H_
 #define _SAMBA_KDC_H_
 
@@ -64,6 +66,6 @@
 	NTSTATUS reject_status;
 };
 
-extern struct hdb_method hdb_samba4_interface;
+extern struct heim_plugin_common_ftable_desc hdb_samba4_interface;
 
 #endif /* _SAMBA_KDC_H_ */
Comment 8 Douglas Bagnall 2022-06-29 22:06:07 UTC 
(In reply to Krzysztof Olędzki from comment #7)
> Seems line nobody uses samba4+AD on 32 bit these days... :/

There is one person! No automated testing though.

Your analysis looks good, and I think we found other bugs related to struct packing around the 4.16 Heimdal upgrade.

The best thing to do now would be to 

1. Read https://www.samba.org/samba/devel/copyright-policy.html, decide whether the work is on behalf of an employer, and if so, send an email as described there.

2. Commit your fix with a signed off by and BUG: reference, as per https://wiki.samba.org/index.php/Contribute (you can skim much of it)

3. attach the patch here, using `git format-patch HEAD^` or similar.

then I will push a merge request to gitlab with it, and the people who know about the Heimdal code can review it, the 64 bit tests will run, etc.

If you'd rather push to gitlab yourself, that is good, but will involve more work.
Comment 9 Michael Saxl 2022-09-03 19:55:15 UTC 
(In reply to Krzysztof Olędzki from comment #6)
just updated to ubuntu 22.10 (it is best to update one machine to the next ubuntu versions 2 month prior to release, there are always issues)

same here (although armhf)

came to the same conclusion. too bad that I found this report after I already gdb'd and saw what happend.

Don't know however why there is a struct mismatch and why there are two different ones..

In the mean time I will make a new package with your patch (thank you) in my ppa together with another fix that is not backported to 4.16 yet.
Comment 10 Krzysztof Olędzki 2022-09-03 20:14:41 UTC 
Oh, right...

I was recently traveling, next week I'll try to follow the described steps to get the fix integrated into the mainline.
Comment 11 Krzysztof Olędzki 2022-09-20 15:54:56 UTC 
Created attachment 17523 [details]
kdc: fix Segmentation fault due to struct mismatch

I have attached the patch. In addition to "master" I think it should go to v4-17 and v4-16 (where it was tested).

I also checked with my employer and confirmed I am free to provide a patch to fix this bug.
Comment 12 Douglas Bagnall 2022-10-14 02:29:06 UTC 
(In reply to Krzysztof Olędzki from comment #11)

pushed to a gitlab MR here:

https://gitlab.com/samba-team/samba/-/merge_requests/2751

The next step is for two team members to review it.
Comment 13 Douglas Bagnall 2022-10-14 02:32:27 UTC 
(In reply to Douglas Bagnall from comment #12)
oh, and the build is unhappy at https://gitlab.com/samba-team/devel/samba/-/pipelines/666505316


[3080/4821] Compiling source4/kdc/hdb-samba4-plugin.c
==> /builds/samba-team/devel/samba/samba-def-build.stderr <==
../../source4/kdc/hdb-samba4-plugin.c:77:10: error: initialization from incompatible pointer type [-Werror=incompatible-pointer-types]
  .init = hdb_samba4_init,
          ^~~~~~~~~~~~~~~
../../source4/kdc/hdb-samba4-plugin.c:77:10: note: (near initialization for ‘hdb_samba4_interface.init’)
../../source4/kdc/hdb-samba4-plugin.c:26:24: error: ‘hdb_samba4_create’ defined but not used [-Werror=unused-function]
 static krb5_error_code hdb_samba4_create(krb5_context context, struct HDB **db, const char *arg)
                        ^~~~~~~~~~~~~~~~~
cc1: all warnings being treated as errors
==> /builds/samba-team/devel/samba/samba-def-build.stdout <==
Waf: Leaving directory `/tmp/samba-testbase/samba-def-build/bin/default'
==> /builds/samba-team/devel/samba/samba-def-build.stderr <==
Build failed
 -> task in 'HDB_SAMBA4.objlist' failed with exit status 1 (run with -v to display more information)
==> /builds/samba-team/devel/samba/samba-def-build.stdout <==
Comment 14 Douglas Bagnall 2022-10-14 02:41:25 UTC 
(In reply to Krzysztof Olędzki from comment #11)

> I also checked with my employer and confirmed I am free to provide a patch to fix this bug.

We still need the "Samba Developer's Declaration" sent to contributing@samba.org (as found on https://www.samba.org/samba/devel/copyright-policy.html). Sorry, I think I (and perhaps that page) misled you on that point.

A gitlab.com login would be useful too, as the review will take place there.
Comment 15 Samba QA Contact 2022-11-02 05:22:12 UTC 
This bug was referenced in samba master:

074e92849715ed3485703cfbba3771d405e4e78a
Comment 16 Jo Sutton 2022-11-03 00:56:28 UTC 
Created attachment 17614 [details]
patch for Samba 4.16
Comment 17 Jo Sutton 2022-11-03 01:58:23 UTC 
Created attachment 17616 [details]
patch for Samba 4.17
Comment 18 Andrew Bartlett 2022-11-08 07:27:02 UTC 
Assigning to Jule for Samba 4.16 and 4.17
Comment 19 Jule Anger 2022-11-08 08:07:21 UTC 
Pushed to autobuild-v4-{17,16}-test.
Comment 20 Samba QA Contact 2022-11-08 09:24:12 UTC 
This bug was referenced in samba v4-17-test:

159054c3bb760eb8f7a199591d95e79e99fa6eb0
Comment 21 Samba QA Contact 2022-11-08 14:10:03 UTC 
This bug was referenced in samba v4-16-test:

eeea6587e92daf792c5ca382d7c03c40e6ccd621
Comment 22 Jule Anger 2022-11-08 17:14:38 UTC 
Closing out bug report.

Thanks!
Comment 23 Samba QA Contact 2022-12-15 16:34:39 UTC 
This bug was referenced in samba v4-17-stable (Release samba-4.17.4):

159054c3bb760eb8f7a199591d95e79e99fa6eb0
Comment 24 Samba QA Contact 2022-12-15 16:36:35 UTC 
This bug was referenced in samba v4-16-stable (Release samba-4.16.8):

eeea6587e92daf792c5ca382d7c03c40e6ccd621