Bug 15110 - Segmentation fault in krb5_plugin_register since samba-4.16rc1 on x86 (32 bit), regression from samba-4.15
Summary: Segmentation fault in krb5_plugin_register since samba-4.16rc1 on x86 (32 bit...
Status: NEW
Alias: None
Product: Samba 4.1 and newer
Classification: Unclassified
Component: File services (show other bugs)
Version: 4.16.2
Hardware: x86 Linux
: P5 regression with 15 votes (vote)
Target Milestone: ---
Assignee: Samba QA Contact
QA Contact: Samba QA Contact
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2022-06-28 16:00 UTC by Krzysztof Olędzki
Modified: 2022-06-29 22:06 UTC (History)
4 users (show)

See Also:


Attachments
Tar Gz archive with gdb_backtrace logs (7.61 KB, application/x-compressed)
2022-06-29 09:01 UTC, Krzysztof Olędzki
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Krzysztof Olędzki 2022-06-28 16:00:08 UTC
Starting samba-4.16rc1 (so far the oldest version I have tested) samba crashes during start on to similar systems.

While this is a little bit less standard setup (x86_64 - 64 bit kernel, x86 - 32 bit userspace, including samba itself) everything worked well over the last many years.

I'm mentioning this as on a very similar, native x86_64 system everything works fine. Note that I have not tested 32 bit kernel, but can try if needed - perhaps this is a problem with a 32 bit binary?

More about my system:
 server role = active directory domain controller
 server services = s3fs, rpc, nbt, wrepl, ldap, cldap, kdc, drepl, winbindd, ntp_signd, kcc, dnsupdate

Crash log sorted by pid:
Jun 28 17:46:03 server samba[17718]: [2022/06/28 17:46:03.707209,  0] ../../lib/util/fault.c:172(smb_panic_log)
Jun 28 17:46:03 server samba[17718]:   ===============================================================
Jun 28 17:46:03 server samba[17718]: [2022/06/28 17:46:03.707283,  0] ../../lib/util/fault.c:173(smb_panic_log)
Jun 28 17:46:03 server samba[17718]:   INTERNAL ERROR: Signal 11: Segmentation fault in pid 17718 (4.16.2)
Jun 28 17:46:03 server samba[17718]: [2022/06/28 17:46:03.707313,  0] ../../lib/util/fault.c:177(smb_panic_log)
Jun 28 17:46:03 server samba[17718]:   If you are running a recent Samba version, and if you think this problem is not yet fixed in the latest versions, please consider reporting this bug, see https://wiki.samba.org/index.php/Bug_Reporting
Jun 28 17:46:03 server samba[17718]: [2022/06/28 17:46:03.707340,  0] ../../lib/util/fault.c:182(smb_panic_log)
Jun 28 17:46:03 server samba[17718]:   ===============================================================
Jun 28 17:46:03 server samba[17718]: [2022/06/28 17:46:03.707361,  0] ../../lib/util/fault.c:183(smb_panic_log)
Jun 28 17:46:03 server samba[17718]:   PANIC (pid 17718): Signal 11: Segmentation fault in 4.16.2
Jun 28 17:46:03 server samba[17718]: [2022/06/28 17:46:03.707518,  0] ../../lib/util/fault.c:245(log_stack_trace)
Jun 28 17:46:03 server samba[17718]:   BACKTRACE:
Jun 28 17:46:03 server samba[17718]:    #0 log_stack_trace + 0x36 [ip=0xf7ec6226] [sp=0xffffbbc0]
Jun 28 17:46:04 server samba[17718]:    #1 smb_panic_log + 0x76 [ip=0xf7ec6526] [sp=0xffffc090]
Jun 28 17:46:04 server samba[17718]:    #2 smb_panic + 0x1a [ip=0xf7ec66ca] [sp=0xffffc0a0]
Jun 28 17:46:04 server samba[17718]:    #3 smb_panic + 0xc0 [ip=0xf7ec6770] [sp=0xffffc0c0]
Jun 28 17:46:04 server samba[17718]:    #4 <unknown symbol> [ip=0xf7fc6930] [sp=0xffffc170]
Jun 28 17:46:05 server samba[17714]:   prefork_child_pipe_handler: Parent 17714, Child 17718 terminated with signal 6
Jun 28 17:46:03 server samba[17722]: [2022/06/28 17:46:03.710452,  0] ../../lib/util/fault.c:172(smb_panic_log)
Jun 28 17:46:03 server samba[17722]:   ===============================================================
Jun 28 17:46:04 server samba[17722]: [2022/06/28 17:46:04.130017,  0] ../../lib/util/fault.c:173(smb_panic_log)
Jun 28 17:46:04 server samba[17722]:   INTERNAL ERROR: Signal 11: Segmentation fault in pid 17722 (4.16.2)
Jun 28 17:46:04 server samba[17722]: [2022/06/28 17:46:04.649636,  0] ../../lib/util/fault.c:177(smb_panic_log)
Jun 28 17:46:04 server samba[17722]:   If you are running a recent Samba version, and if you think this problem is not yet fixed in the latest versions, please consider reporting this bug, see https://wiki.samba.org/index.php/Bug_Reporting
Jun 28 17:46:04 server samba[17722]: [2022/06/28 17:46:04.985653,  0] ../../lib/util/fault.c:182(smb_panic_log)
Jun 28 17:46:05 server samba[17722]:   ===============================================================
Jun 28 17:46:05 server samba[17722]: [2022/06/28 17:46:05.738297,  0] ../../lib/util/fault.c:183(smb_panic_log)
Jun 28 17:46:05 server samba[17722]:   PANIC (pid 17722): Signal 11: Segmentation fault in 4.16.2
Jun 28 17:46:05 server samba[17722]: [2022/06/28 17:46:05.948046,  0] ../../lib/util/fault.c:245(log_stack_trace)
Jun 28 17:46:06 server samba[17722]:   BACKTRACE:
Jun 28 17:46:06 server samba[17722]:    #0 log_stack_trace + 0x36 [ip=0xf7ec6226] [sp=0xffffbbc0]
Jun 28 17:46:06 server samba[17722]:    #1 smb_panic_log + 0x76 [ip=0xf7ec6526] [sp=0xffffc090]
Jun 28 17:46:06 server samba[17722]:    #2 smb_panic + 0x1a [ip=0xf7ec66ca] [sp=0xffffc0a0]
Jun 28 17:46:06 server samba[17722]:    #3 smb_panic + 0xc0 [ip=0xf7ec6770] [sp=0xffffc0c0]
Jun 28 17:46:06 server samba[17722]:    #4 <unknown symbol> [ip=0xf7fc6930] [sp=0xffffc170]
Jun 28 17:46:07 server samba[17714]:   prefork_child_pipe_handler: Parent 17714, Child 17722 terminated with signal 6
Jun 28 17:46:03 server samba[17731]: [2022/06/28 17:46:03.719536,  0] ../../lib/util/fault.c:172(smb_panic_log)
Jun 28 17:46:03 server samba[17731]:   ===============================================================
Jun 28 17:46:04 server samba[17731]: [2022/06/28 17:46:04.171919,  0] ../../lib/util/fault.c:173(smb_panic_log)
Jun 28 17:46:04 server samba[17731]:   INTERNAL ERROR: Signal 11: Segmentation fault in pid 17731 (4.16.2)
Jun 28 17:46:04 server samba[17731]: [2022/06/28 17:46:04.691855,  0] ../../lib/util/fault.c:177(smb_panic_log)
Jun 28 17:46:04 server samba[17731]:   If you are running a recent Samba version, and if you think this problem is not yet fixed in the latest versions, please consider reporting this bug, see https://wiki.samba.org/index.php/Bug_Reporting
Jun 28 17:46:05 server samba[17731]: [2022/06/28 17:46:05.078133,  0] ../../lib/util/fault.c:182(smb_panic_log)
Jun 28 17:46:05 server samba[17731]:   ===============================================================
Jun 28 17:46:05 server samba[17731]: [2022/06/28 17:46:05.763484,  0] ../../lib/util/fault.c:183(smb_panic_log)
Jun 28 17:46:05 server samba[17731]:   PANIC (pid 17731): Signal 11: Segmentation fault in 4.16.2
Jun 28 17:46:05 server samba[17731]: [2022/06/28 17:46:05.964936,  0] ../../lib/util/fault.c:245(log_stack_trace)
Jun 28 17:46:06 server samba[17731]:   BACKTRACE:
Jun 28 17:46:06 server samba[17731]:    #0 log_stack_trace + 0x36 [ip=0xf7ec6226] [sp=0xffffbbc0]
Jun 28 17:46:06 server samba[17731]:    #1 smb_panic_log + 0x76 [ip=0xf7ec6526] [sp=0xffffc090]
Jun 28 17:46:06 server samba[17731]:    #2 smb_panic + 0x1a [ip=0xf7ec66ca] [sp=0xffffc0a0]
Jun 28 17:46:06 server samba[17731]:    #3 smb_panic + 0xc0 [ip=0xf7ec6770] [sp=0xffffc0c0]
Jun 28 17:46:06 server samba[17731]:    #4 <unknown symbol> [ip=0xf7fc6930] [sp=0xffffc170]
Jun 28 17:46:07 server samba[17714]:   prefork_child_pipe_handler: Parent 17714, Child 17731 terminated with signal 6
Jun 28 17:46:03 server samba[17734]: [2022/06/28 17:46:03.721018,  0] ../../lib/util/fault.c:172(smb_panic_log)
Jun 28 17:46:03 server samba[17734]:   ===============================================================
Jun 28 17:46:04 server samba[17734]: [2022/06/28 17:46:04.205509,  0] ../../lib/util/fault.c:173(smb_panic_log)
Jun 28 17:46:04 server samba[17734]:   INTERNAL ERROR: Signal 11: Segmentation fault in pid 17734 (4.16.2)
Jun 28 17:46:04 server samba[17734]: [2022/06/28 17:46:04.708589,  0] ../../lib/util/fault.c:177(smb_panic_log)
Jun 28 17:46:04 server samba[17734]:   If you are running a recent Samba version, and if you think this problem is not yet fixed in the latest versions, please consider reporting this bug, see https://wiki.samba.org/index.php/Bug_Reporting
Jun 28 17:46:05 server samba[17734]: [2022/06/28 17:46:05.111732,  0] ../../lib/util/fault.c:182(smb_panic_log)
Jun 28 17:46:05 server samba[17734]:   ===============================================================
Jun 28 17:46:05 server samba[17734]: [2022/06/28 17:46:05.788658,  0] ../../lib/util/fault.c:183(smb_panic_log)
Jun 28 17:46:05 server samba[17734]:   PANIC (pid 17734): Signal 11: Segmentation fault in 4.16.2
Jun 28 17:46:05 server samba[17734]: [2022/06/28 17:46:05.981807,  0] ../../lib/util/fault.c:245(log_stack_trace)
Jun 28 17:46:06 server samba[17734]:   BACKTRACE:
Jun 28 17:46:06 server samba[17734]:    #0 log_stack_trace + 0x36 [ip=0xf7ec6226] [sp=0xffffbbc0]
Jun 28 17:46:06 server samba[17734]:    #1 smb_panic_log + 0x76 [ip=0xf7ec6526] [sp=0xffffc090]
Jun 28 17:46:06 server samba[17734]:    #2 smb_panic + 0x1a [ip=0xf7ec66ca] [sp=0xffffc0a0]
Jun 28 17:46:06 server samba[17734]:    #3 smb_panic + 0xc0 [ip=0xf7ec6770] [sp=0xffffc0c0]
Jun 28 17:46:06 server samba[17734]:    #4 <unknown symbol> [ip=0xf7fc6930] [sp=0xffffc170]
Jun 28 17:46:08 server samba[17714]:   prefork_child_pipe_handler: Parent 17714, Child 17734 terminated with signal 6
Jun 28 17:46:05 server samba[17761]: [2022/06/28 17:46:05.939204,  0] ../../lib/util/fault.c:172(smb_panic_log)
Jun 28 17:46:05 server samba[17761]:   ===============================================================
Jun 28 17:46:06 server samba[17761]: [2022/06/28 17:46:06.065688,  0] ../../lib/util/fault.c:173(smb_panic_log)
Jun 28 17:46:06 server samba[17761]:   INTERNAL ERROR: Signal 11: Segmentation fault in pid 17761 (4.16.2)
Jun 28 17:46:06 server samba[17761]: [2022/06/28 17:46:06.260165,  0] ../../lib/util/fault.c:177(smb_panic_log)
Jun 28 17:46:06 server samba[17761]:   If you are running a recent Samba version, and if you think this problem is not yet fixed in the latest versions, please consider reporting this bug, see https://wiki.samba.org/index.php/Bug_Reporting
Jun 28 17:46:06 server samba[17761]: [2022/06/28 17:46:06.804294,  0] ../../lib/util/fault.c:182(smb_panic_log)
Jun 28 17:46:07 server samba[17761]:   ===============================================================
Jun 28 17:46:07 server samba[17761]: [2022/06/28 17:46:07.243201,  0] ../../lib/util/fault.c:183(smb_panic_log)
Jun 28 17:46:07 server samba[17761]:   PANIC (pid 17761): Signal 11: Segmentation fault in 4.16.2
Jun 28 17:46:07 server samba[17761]: [2022/06/28 17:46:07.503917,  0] ../../lib/util/fault.c:245(log_stack_trace)
Jun 28 17:46:07 server samba[17761]:   BACKTRACE:
Jun 28 17:46:07 server samba[17761]:    #0 log_stack_trace + 0x36 [ip=0xf7ec6226] [sp=0xffffb9c0]
Jun 28 17:46:07 server samba[17761]:    #1 smb_panic_log + 0x76 [ip=0xf7ec6526] [sp=0xffffbe90]
Jun 28 17:46:08 server samba[17761]:    #2 smb_panic + 0x1a [ip=0xf7ec66ca] [sp=0xffffbea0]
Jun 28 17:46:08 server samba[17761]:    #3 smb_panic + 0xc0 [ip=0xf7ec6770] [sp=0xffffbec0]
Jun 28 17:46:08 server samba[17761]:    #4 <unknown symbol> [ip=0xf7fc6930] [sp=0xffffbf70]
Jun 28 17:46:08 server samba[17714]:   prefork_child_pipe_handler: Parent 17714, Child 17761 terminated with signal 6
Jun 28 17:46:07 server samba[17763]: [2022/06/28 17:46:07.696804,  0] ../../lib/util/fault.c:172(smb_panic_log)
Jun 28 17:46:07 server samba[17763]:   ===============================================================
Jun 28 17:46:07 server samba[17763]: [2022/06/28 17:46:07.950095,  0] ../../lib/util/fault.c:173(smb_panic_log)
Jun 28 17:46:08 server samba[17763]:   INTERNAL ERROR: Signal 11: Segmentation fault in pid 17763 (4.16.2)
Jun 28 17:46:08 server samba[17763]: [2022/06/28 17:46:08.201961,  0] ../../lib/util/fault.c:177(smb_panic_log)
Jun 28 17:46:08 server samba[17763]:   If you are running a recent Samba version, and if you think this problem is not yet fixed in the latest versions, please consider reporting this bug, see https://wiki.samba.org/index.php/Bug_Reporting
Jun 28 17:46:08 server samba[17763]: [2022/06/28 17:46:08.336334,  0] ../../lib/util/fault.c:182(smb_panic_log)
Jun 28 17:46:08 server samba[17763]:   ===============================================================
Jun 28 17:46:08 server samba[17763]: [2022/06/28 17:46:08.528943,  0] ../../lib/util/fault.c:183(smb_panic_log)
Jun 28 17:46:08 server samba[17763]:   PANIC (pid 17763): Signal 11: Segmentation fault in 4.16.2
Jun 28 17:46:08 server samba[17763]: [2022/06/28 17:46:08.680039,  0] ../../lib/util/fault.c:245(log_stack_trace)
Jun 28 17:46:08 server samba[17763]:   BACKTRACE:
Jun 28 17:46:08 server samba[17763]:    #0 log_stack_trace + 0x36 [ip=0xf7ec6226] [sp=0xffffb9c0]
Jun 28 17:46:08 server samba[17763]:    #1 smb_panic_log + 0x76 [ip=0xf7ec6526] [sp=0xffffbe90]
Jun 28 17:46:08 server samba[17763]:    #2 smb_panic + 0x1a [ip=0xf7ec66ca] [sp=0xffffbea0]
Jun 28 17:46:09 server samba[17763]:    #3 smb_panic + 0xc0 [ip=0xf7ec6770] [sp=0xffffbec0]
Jun 28 17:46:09 server samba[17763]:    #4 <unknown symbol> [ip=0xf7fc6930] [sp=0xffffbf70]
Jun 28 17:46:09 server samba[17714]:   prefork_child_pipe_handler: Parent 17714, Child 17763 terminated with signal 6
Jun 28 17:46:08 server samba[17765]: [2022/06/28 17:46:08.193266,  0] ../../lib/util/fault.c:172(smb_panic_log)
Jun 28 17:46:08 server samba[17765]:   ===============================================================
Jun 28 17:46:08 server samba[17765]: [2022/06/28 17:46:08.319538,  0] ../../lib/util/fault.c:173(smb_panic_log)
Jun 28 17:46:08 server samba[17765]:   INTERNAL ERROR: Signal 11: Segmentation fault in pid 17765 (4.16.2)
Jun 28 17:46:08 server samba[17765]: [2022/06/28 17:46:08.512017,  0] ../../lib/util/fault.c:177(smb_panic_log)
Jun 28 17:46:08 server samba[17765]:   If you are running a recent Samba version, and if you think this problem is not yet fixed in the latest versions, please consider reporting this bug, see https://wiki.samba.org/index.php/Bug_Reporting
Jun 28 17:46:08 server samba[17765]: [2022/06/28 17:46:08.654865,  0] ../../lib/util/fault.c:182(smb_panic_log)
Jun 28 17:46:08 server samba[17765]:   ===============================================================
Jun 28 17:46:08 server samba[17765]: [2022/06/28 17:46:08.805916,  0] ../../lib/util/fault.c:183(smb_panic_log)
Jun 28 17:46:08 server samba[17765]:   PANIC (pid 17765): Signal 11: Segmentation fault in 4.16.2
Jun 28 17:46:08 server samba[17765]: [2022/06/28 17:46:08.940097,  0] ../../lib/util/fault.c:245(log_stack_trace)
Jun 28 17:46:09 server samba[17765]:   BACKTRACE:
Jun 28 17:46:09 server samba[17765]:    #0 log_stack_trace + 0x36 [ip=0xf7ec6226] [sp=0xffffb9c0]
Jun 28 17:46:09 server samba[17765]:    #1 smb_panic_log + 0x76 [ip=0xf7ec6526] [sp=0xffffbe90]
Jun 28 17:46:09 server samba[17765]:    #2 smb_panic + 0x1a [ip=0xf7ec66ca] [sp=0xffffbea0]
Jun 28 17:46:09 server samba[17765]:    #3 smb_panic + 0xc0 [ip=0xf7ec6770] [sp=0xffffbec0]
Jun 28 17:46:09 server samba[17765]:    #4 <unknown symbol> [ip=0xf7fc6930] [sp=0xffffbf70]
Jun 28 17:46:10 server samba[17714]:   prefork_child_pipe_handler: Parent 17714, Child 17765 terminated with signal 6
Jun 28 17:46:08 server samba[17767]: [2022/06/28 17:46:08.502632,  0] ../../lib/util/fault.c:172(smb_panic_log)
Jun 28 17:46:08 server samba[17767]:   ===============================================================
Jun 28 17:46:08 server samba[17767]: [2022/06/28 17:46:08.638296,  0] ../../lib/util/fault.c:173(smb_panic_log)
Jun 28 17:46:08 server samba[17767]:   INTERNAL ERROR: Signal 11: Segmentation fault in pid 17767 (4.16.2)
Jun 28 17:46:08 server samba[17767]: [2022/06/28 17:46:08.789042,  0] ../../lib/util/fault.c:177(smb_panic_log)
Jun 28 17:46:08 server samba[17767]:   If you are running a recent Samba version, and if you think this problem is not yet fixed in the latest versions, please consider reporting this bug, see https://wiki.samba.org/index.php/Bug_Reporting
Jun 28 17:46:08 server samba[17767]: [2022/06/28 17:46:08.923172,  0] ../../lib/util/fault.c:182(smb_panic_log)
Jun 28 17:46:09 server samba[17767]:   ===============================================================
Jun 28 17:46:09 server samba[17767]: [2022/06/28 17:46:09.107919,  0] ../../lib/util/fault.c:183(smb_panic_log)
Jun 28 17:46:09 server samba[17767]:   PANIC (pid 17767): Signal 11: Segmentation fault in 4.16.2
Jun 28 17:46:09 server samba[17767]: [2022/06/28 17:46:09.317480,  0] ../../lib/util/fault.c:245(log_stack_trace)
Jun 28 17:46:09 server samba[17767]:   BACKTRACE:
Jun 28 17:46:09 server samba[17767]:    #0 log_stack_trace + 0x36 [ip=0xf7ec6226] [sp=0xffffb9c0]
Jun 28 17:46:09 server samba[17767]:    #1 smb_panic_log + 0x76 [ip=0xf7ec6526] [sp=0xffffbe90]
Jun 28 17:46:10 server samba[17767]:    #2 smb_panic + 0x1a [ip=0xf7ec66ca] [sp=0xffffbea0]
Jun 28 17:46:10 server samba[17767]:    #3 smb_panic + 0xc0 [ip=0xf7ec6770] [sp=0xffffbec0]
Jun 28 17:46:10 server samba[17767]:    #4 <unknown symbol> [ip=0xf7fc6930] [sp=0xffffbf70]
Jun 28 17:46:10 server samba[17714]:   prefork_child_pipe_handler: Parent 17714, Child 17767 terminated with signal 6


samba and most of the libraries are built with debug symbols, but there is not much in the backtrace.

(gdb) bt 0xf7fc6930
#0  0xf7fc6919 in __kernel_vsyscall ()
#1  0xf79e7653 in epoll_wait () from /lib/libc.so.6
#2  0xf7b071e8 in epoll_event_loop (tvalp=0xffffca08, epoll_ev=0x565f9b20) at ../../tevent_epoll.c:650
#3  epoll_event_loop_once (ev=0x565f99f0, location=0x5655be28 "../../source4/samba/server.c:965") at ../../tevent_epoll.c:937
#4  0xf7b05322 in std_event_loop_once (ev=0x565f99f0, location=0x5655be28 "../../source4/samba/server.c:965") at ../../tevent_standard.c:110
#5  0xf7aff964 in _tevent_loop_once (ev=0x565f99f0, location=0x5655be28 "../../source4/samba/server.c:965") at ../../tevent.c:790
#6  0xf7affc2a in tevent_common_loop_wait (ev=0x565f99f0, location=0x5655be28 "../../source4/samba/server.c:965") at ../../tevent.c:913
#7  0xf7b052b2 in std_event_loop_wait (ev=0x565f99f0, location=0x5655be28 "../../source4/samba/server.c:965") at ../../tevent_standard.c:141
#8  0x56559e37 in binary_smbd_main (mem_ctx=0x56573460, binary_name=0x5655c06b "samba", argc=2, argv=0xffffce54, binary_name=0x5655c06b "samba") at ../../source4/samba/server.c:965
#9  0x56558821 in main (argc=2, argv=0xffffce54) at ../../source4/samba/server.c:986
(gdb) bt 0xffffbf70
#0  0xf7fc6919 in __kernel_vsyscall ()
#1  0xf79e7653 in epoll_wait () from /lib/libc.so.6
#2  0xf7b071e8 in epoll_event_loop (tvalp=0xffffca08, epoll_ev=0x565f9b20) at ../../tevent_epoll.c:650
#3  epoll_event_loop_once (ev=0x565f99f0, location=0x5655be28 "../../source4/samba/server.c:965") at ../../tevent_epoll.c:937
#4  0xf7b05322 in std_event_loop_once (ev=0x565f99f0, location=0x5655be28 "../../source4/samba/server.c:965") at ../../tevent_standard.c:110
#5  0xf7aff964 in _tevent_loop_once (ev=0x565f99f0, location=0x5655be28 "../../source4/samba/server.c:965") at ../../tevent.c:790
#6  0xf7affc2a in tevent_common_loop_wait (ev=0x565f99f0, location=0x5655be28 "../../source4/samba/server.c:965") at ../../tevent.c:913
#7  0xf7b052b2 in std_event_loop_wait (ev=0x565f99f0, location=0x5655be28 "../../source4/samba/server.c:965") at ../../tevent_standard.c:141
#8  0x56559e37 in binary_smbd_main (mem_ctx=0x56573460, binary_name=0x5655c06b "samba", argc=2, argv=0xffffce54, binary_name=0x5655c06b "samba") at ../../source4/samba/server.c:965
#9  0x56558821 in main (argc=2, argv=0xffffce54) at ../../source4/samba/server.c:986
Comment 1 Douglas Bagnall 2022-06-28 22:36:42 UTC
If you add 

   panic action = /usr/local/bin/gdb_backtrace %d

to the smb.conf, you will get more[*]. Or use 'bt full' in gdb. How much it helps in this case is another question.

*  if /usr/local/bin is the wrong path, change it!
Comment 2 Krzysztof Olędzki 2022-06-29 09:01:19 UTC
Created attachment 17400 [details]
Tar Gz archive with gdb_backtrace logs
Comment 3 Krzysztof Olędzki 2022-06-29 09:04:57 UTC
Thanks, this is very useful! 

I had to modify the script, as I'm getting multiple crashes at the same time, but it worked well overall. I have attached an archive with the gdb_backtrace logs.

Also, here is a sample Valgrind output from a process that crashes:

--2863-- REDIR: 0x4dda720 (libc.so.6:__memchr_sse2_bsf) redirected to 0x484bb00 (memchr)
--2863-- REDIR: 0x4ddc3e0 (libc.so.6:__GI_stpcpy) redirected to 0x484dcc0 (__GI_stpcpy)
==2863== Jump to the invalid address stated on the next line
==2863==    at 0x0: ???
==2863==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
==2863==
--2863-- REDIR: 0x4dc30d0 (libc.so.6:stpcpy) redirected to 0x483b1f0 (_vgnU_ifunc_wrapper)
--2863-- REDIR: 0x4dcd930 (libc.so.6:__stpcpy_ssse3) redirected to 0x484dc00 (stpcpy)
==2863==
==2863== Process terminating with default action of signal 6 (SIGABRT)
==2863==    at 0x4DB03E7: __pthread_kill_implementation (pthread_kill.c:44)
==2863==    by 0x4D60A40: raise (raise.c:26)
==2863==    by 0x4D4A271: abort (abort.c:79)
==2863==    by 0x48EEFC1: smb_panic_default.constprop.0 (fault.c:167)
==2863==    by 0x48EF6ED: smb_panic (fault.c:200)
==2863==    by 0x48EF76F: fault_report (fault.c:81)
==2863==    by 0x48EF76F: sig_fault (fault.c:92)
==2863==    by 0x4D60AF7: ??? (in /lib/libc.so.6)
Comment 4 Krzysztof Olędzki 2022-06-29 09:10:51 UTC
I think all the traces look the same, and the crash happens around:

#11 0x00000000 in ?? ()
No symbol table info available.
#12 0xf6185b8e in heim_plugin_register (context=0x58331420, pcontext=0x58e19910, module=0xf6bc643a "krb5", name=0xf2ff8a07 "hdb_samba4_interface", ftable=0xf2fe9004 <hdb_samba4_interface>) at ../../third_party/heimdal/lib/base/plugin.c:235
        pl = 0x58b06a80
        ret = 0
        plugins = 0x58ca31d0
        hname = 0x58ca3190
        dso = 0x5860db30
        ctx = {symbol = 0xf2fe9004 <hdb_samba4_interface>, is_dup = 0}

plugin.c:235 is:
  ret = pl->ftable->init(pcontext, &pl->ctx);
Comment 5 Krzysztof Olędzki 2022-06-29 10:34:06 UTC
With the following patch:

--- a/third_party/heimdal/lib/base/plugin.c     2022-06-29 11:28:04.196263495 +0200
+++ b/third_party/heimdal/lib/base/plugin.c     2022-06-29 12:06:07.955431592 +0200
@@ -232,12 +232,29 @@
             ret = heim_enomem(context);
         } else {
             pl->ftable = ftable;
-            ret = pl->ftable->init(pcontext, &pl->ctx);
-            if (ret == 0) {
-                heim_array_append_value(plugins, pl);
-                heim_debug(context, 5, "Registered %s plugin", name);
+
+            printf("Plugin %s, pl: %u\n", name, pl);
+            printf("Plugin %s, pl->ftable: %u\n", name, pl->ftable);
+            printf("Plugin %s, pl->ftable->init: %u\n", name, pl->ftable->init);
+
+            heim_debug(context, 0, "Plugin %s, pl: %u", name, pl);
+            heim_debug(context, 0, "Plugin %s, pl->ftable: %u", name, pl->ftable);
+            heim_debug(context, 0, "Plugin %s, pl->ftable->init: %u", name, pl->ftable->init);
+
+            heim_debug(context, 200, "Plugin %s, pl: %u", name, pl);
+            heim_debug(context, 200, "Plugin %s, pl->ftable: %u", name, pl->ftable);
+            heim_debug(context, 200, "Plugin %s, pl->ftable->init: %u", name, pl->ftable->init);
+
+            if (!pl->ftable->init) {
+              ret = heim_enomem(context);
+            } else {
+              ret = pl->ftable->init(pcontext, &pl->ctx);
+              if (ret == 0) {
+                  heim_array_append_value(plugins, pl);
+                  heim_debug(context, 5, "Registered %s plugin", name);
+              }
+              heim_release(pl);
             }
-            heim_release(pl);
         }
     }

I ended up with the following output:

Plugin send_to_kdc, pl: 1483976624
Plugin send_to_kdc, pl->ftable: 4148170772
Plugin send_to_kdc, pl->ftable->init: 4148086000
Plugin hdb_samba4_interface, pl: 1487279248
Plugin hdb_samba4_interface, pl->ftable: 4076302340
Plugin hdb_samba4_interface, pl->ftable->init: 0
Plugin send_to_kdc, pl: 1485860480
Plugin send_to_kdc, pl->ftable: 4148170772
Plugin send_to_kdc, pl->ftable->init: 4148086000
Plugin hdb_samba4_interface, pl: 1483976624
Plugin hdb_samba4_interface, pl->ftable: 4076302340
Plugin hdb_samba4_interface, pl->ftable->init: 0
Plugin send_to_kdc, pl: 1483976624
Plugin send_to_kdc, pl->ftable: 4148170772
Plugin send_to_kdc, pl->ftable->init: 4148086000
Plugin hdb_samba4_interface, pl: 1487279248
Plugin hdb_samba4_interface, pl->ftable: 4076302340
Plugin hdb_samba4_interface, pl->ftable->init: 0
Plugin send_to_kdc, pl: 1487279248
Plugin send_to_kdc, pl->ftable: 4148170772
Plugin send_to_kdc, pl->ftable->init: 4148086000
Plugin hdb_samba4_interface, pl: 1487628800
Plugin hdb_samba4_interface, pl->ftable: 4076302340
Plugin hdb_samba4_interface, pl->ftable->init: 0

... and error:
task_server_terminate: task_server_terminate: [kdc: failed to register hdb plugin]

[Yes, I have been unable to make heim_debug working]

So, pl->ftable->init is NULL when called with hdb_samba4_interface. Now the question why...
Comment 6 Krzysztof Olędzki 2022-06-29 16:14:13 UTC
Maybe I'm incorrectly reading the code, but it seems like we have struct mismatch?

struct heim_plugin_common_ftable_desc {
    int                 version;
    int                 (HEIM_LIB_CALL *init)(heim_pcontext, void **);
    void                (HEIM_LIB_CALL *fini)(void *);
};


struct hdb_method {
    int                 version;
    unsigned int        is_file_based:1;
    unsigned int        can_taste:1;
    krb5_error_code     (*init)(krb5_context, void **);
    void                (*fini)(void *);
    const char *prefix;
    krb5_error_code (*create)(krb5_context, HDB **, const char *filename);
};

... but on x86-64, init/finit happen to be aligned to the same address?
Comment 7 Krzysztof Olędzki 2022-06-29 18:11:02 UTC
Seems line nobody uses samba4+AD on 32 bit these days... :/

This patch, while I'm sure 100% incorrect, fixes the crash:

diff -Nur samba-4.16.2-orig/source4/kdc/hdb-samba4-plugin.c samba-4.16.2/source4/kdc/hdb-samba4-plugin.c
--- samba-4.16.2-orig/source4/kdc/hdb-samba4-plugin.c	2022-01-24 11:26:59.333308700 +0100
+++ samba-4.16.2/source4/kdc/hdb-samba4-plugin.c	2022-06-29 19:48:01.264927045 +0200
@@ -95,12 +95,8 @@
  *
  * The <address> is the string form of a pointer to a talloced struct hdb_samba_context
  */
-struct hdb_method hdb_samba4_interface = {
+struct heim_plugin_common_ftable_desc hdb_samba4_interface = {
 	HDB_INTERFACE_VERSION,
-#if HDB_INTERFACE_VERSION >= 8
 	.init = hdb_samba4_init,
 	.fini = hdb_samba4_fini,
-#endif
-	.prefix = "samba4",
-	.create = hdb_samba4_create
 };
diff -Nur samba-4.16.2-orig/source4/kdc/samba_kdc.h samba-4.16.2/source4/kdc/samba_kdc.h
--- samba-4.16.2-orig/source4/kdc/samba_kdc.h	2022-03-21 13:13:09.604498100 +0100
+++ samba-4.16.2/source4/kdc/samba_kdc.h	2022-06-29 19:38:05.492037885 +0200
@@ -21,6 +21,8 @@
    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 */
 
+#include "third_party/heimdal/lib/base/common_plugin.h"
+
 #ifndef _SAMBA_KDC_H_
 #define _SAMBA_KDC_H_
 
@@ -64,6 +66,6 @@
 	NTSTATUS reject_status;
 };
 
-extern struct hdb_method hdb_samba4_interface;
+extern struct heim_plugin_common_ftable_desc hdb_samba4_interface;
 
 #endif /* _SAMBA_KDC_H_ */
Comment 8 Douglas Bagnall 2022-06-29 22:06:07 UTC
(In reply to Krzysztof Olędzki from comment #7)
> Seems line nobody uses samba4+AD on 32 bit these days... :/

There is one person! No automated testing though.

Your analysis looks good, and I think we found other bugs related to struct packing around the 4.16 Heimdal upgrade.

The best thing to do now would be to 

1. Read https://www.samba.org/samba/devel/copyright-policy.html, decide whether the work is on behalf of an employer, and if so, send an email as described there.

2. Commit your fix with a signed off by and BUG: reference, as per https://wiki.samba.org/index.php/Contribute (you can skim much of it)

3. attach the patch here, using `git format-patch HEAD^` or similar.

then I will push a merge request to gitlab with it, and the people who know about the Heimdal code can review it, the 64 bit tests will run, etc.

If you'd rather push to gitlab yourself, that is good, but will involve more work.