Bug 14329 - smbd panic in zhandle_get_dataset
Summary: smbd panic in zhandle_get_dataset
Status: RESOLVED INVALID
Alias: None
Product: Samba 4.1 and newer
Classification: Unclassified
Component: File services (show other bugs)
Version: 4.12.0
Hardware: x64 FreeBSD
: P5 critical (vote)
Target Milestone: ---
Assignee: Samba QA Contact
QA Contact: Samba QA Contact
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-03-28 23:58 UTC by Thilo
Modified: 2020-03-30 09:42 UTC (History)
0 users

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Thilo 2020-03-28 23:58:09 UTC
I have a panic in smbd which is repeatable when using a macbook current release and connecting through the finder to the server.
It seems the server immediately crashes (reproducible)
I couldn’t find anything in the bugzilla database.


Please let know if I can provide anything else.


My Server OS is  (uname -a)
FreeBSD freenas.nispuk.com 12.1-STABLE FreeBSD 12.1-STABLE 13af4b2776b(freenas/12-stable) TRUENAS  amd64

The smbd log:

 smbd version 4.12.0 started.
 Copyright Andrew Tridgell and the Samba Team 1992-2020
[2020/03/28 23:28:32.616460,  1] ../../source3/profile/profile_dummy.c:30(set_profile_level)
 INFO: Profiling support unavailable in this build.
[2020/03/28 23:28:32.913060,  1] ../../source3/smbd/files.c:240(file_init_global)
 file_init_global: Information only: requested 469197 open files, 59392 are available.
[2020/03/28 23:28:32.916620,  0] ../../lib/util/become_daemon.c:136(daemon_ready)
 daemon_ready: daemon 'smbd' finished starting up and ready to serve connections
[2020/03/28 23:28:54.626563,  1] ../../source3/printing/printer_list.c:234(printer_list_get_last_refresh)
 Failed to fetch record!
[2020/03/28 23:28:54.626641,  1] ../../source3/smbd/server_reload.c:66(delete_and_reload_printers)
 pcap cache not loaded
[2020/03/28 23:28:57.807993,  0] ../../source3/modules/smb_libzfs.c:704(zhandle_get_dataset)
 zhandle_get_dataset: Failed to get mountpoint for Pool1/kali: Cannot allocate memory
[2020/03/28 23:28:57.808047,  0] ../../lib/util/fault.c:79(fault_report)
 ===============================================================
[2020/03/28 23:28:57.808067,  0] ../../lib/util/fault.c:80(fault_report)
 INTERNAL ERROR: Signal 11 in pid 8509 (4.12.0)
 If you are running a recent Samba version, and if you think this problem is not yet fixed in the latest versions, please consider reporting this bug, see https://wiki.samba.org/index.php/Bug_Reporting
[2020/03/28 23:28:57.808096,  0] ../../lib/util/fault.c:86(fault_report)
 ===============================================================
[2020/03/28 23:28:57.808116,  0] ../../source3/lib/util.c:830(smb_panic_s3)
 PANIC (pid 8509): internal error
[2020/03/28 23:28:57.810483,  0] ../../lib/util/fault.c:265(log_stack_trace)
 BACKTRACE: 6 stack frames:
  #0 0x801b398c7 <log_stack_trace+0x37> at /usr/local/lib/samba4/libsamba-util.so.0
  #1 0x802e9f096 <smb_panic_s3+0x56> at /usr/local/lib/samba4/libsmbconf.so.0
  #2 0x801b396b7 <smb_panic+0x17> at /usr/local/lib/samba4/libsamba-util.so.0
  #3 0x801b39a9e <log_stack_trace+0x20e> at /usr/local/lib/samba4/libsamba-util.so.0
  #4 0x801b39699 <fault_setup+0x59> at /usr/local/lib/samba4/libsamba-util.so.0
  #5 0x80fd123b0 <_pthread_sigmask+0x530> at /lib/libthr.so.3
[2020/03/28 23:28:57.810581,  0] ../../source3/lib/dumpcore.c:315(dump_core)
 dumping core in /var/db/system/cores
[2020/03/28 23:28:57.948931,  0] ../../source3/modules/smb_libzfs.c:704(zhandle_get_dataset)
 zhandle_get_dataset: Failed to get mountpoint for Pool1/kali: Cannot allocate memory
[2020/03/28 23:28:57.948983,  0] ../../lib/util/fault.c:79(fault_report)
 ===============================================================
[2020/03/28 23:28:57.949003,  0] ../../lib/util/fault.c:80(fault_report)
 INTERNAL ERROR: Signal 11 in pid 8511 (4.12.0)
 If you are running a recent Samba version, and if you think this problem is not yet fixed in the latest versions, please consider reporting this bug, see https://wiki.samba.org/index.php/Bug_Reporting
[2020/03/28 23:28:57.949032,  0] ../../lib/util/fault.c:86(fault_report)
 ===============================================================
[2020/03/28 23:28:57.949051,  0] ../../source3/lib/util.c:830(smb_panic_s3)
 PANIC (pid 8511): internal error


The config
#
# SMB.CONF(5)           The configuration file for the Samba suite 
# $FreeBSD$
#


[global]
       dns proxy = No
       aio max threads = 2
       max log size = 51200
       load printers = No
       printing = bsd
       disable spoolss = Yes
       dos filemode = Yes
       kernel change notify = No
       directory name cache size = 0
       nsupdate command = /usr/local/bin/samba-nsupdate -g
       unix charset = UTF-8
       log level = 1
       obey pam restrictions = True
       enable web service discovery = True
       logging = file
       server min protocol = SMB2_02
       unix extensions = No
       map to guest = Bad User
       server string = FreeNAS Server
       fruit:nfs_aces = No
       interfaces = 127.0.0.1 192.168.0.9
       bind interfaces only = Yes
       netbios name = freenas
       netbios aliases = 
       server role = standalone
       workgroup = WORKGROUP
       idmap config *: backend = tdb
       idmap config *: range = 90000001-100000000
       allow insecure wide links =  yes
       registry shares = yes
       include = registry

[homes]
       path = /mnt/Pool1/Home/%U
       read only = no
       guest ok = no
       kernel oplocks = no
       kernel share modes = no
       posix locking = no
       nfs4:chown = true
       ea support = false
       vfs objects = aio_fbsd fruit streams_xattr shadow_copy_zfs noacl
       fruit:metadata = stream
       fruit:resource = stream

[Movies]
       path = /mnt/Pool1/Movies
       read only = no
       guest ok = yes
       kernel oplocks = no
       kernel share modes = no
       posix locking = no
       nfs4:chown = true
       ea support = false
       vfs objects = aio_fbsd fruit streams_xattr shadow_copy_zfs ixnas
       fruit:metadata = stream
       fruit:resource = stream
Comment 1 Volker Lendecke 2020-03-29 09:30:50 UTC
The function zhandle_get_dataset does not exist in Samba master. It seems to come from the FreeNAS patches:

https://github.com/freenas/ports/blob/freenas/master/net/samba/files/0001-add-ix-custom-vfs-modules.patch

has function definitions for that. I don't have FreeNAS set up locally, so I can't really test this. Can you do a debug build (build with ./configure.developer) of Samba in FreeNAS and run smbd under valgrind, so that we can help the FreeNAS people with a more informative error report?
Comment 2 Thilo 2020-03-29 14:37:50 UTC
Thank you for pointing to the source, I have forwarded this info to the committer of the file.

I did some further research:

There is a core dump created:

Reading symbols from /usr/local/sbin/smbd...
(No debugging symbols found in /usr/local/sbin/smbd)
[New LWP 102381]

warning: Unexpected size of section `.reg-xstate/102381' in core file.
Core was generated by `/usr/local/sbin/smbd --daemon'.
Program terminated with signal SIGABRT, Aborted.

warning: Unexpected size of section `.reg-xstate/102381' in core file.
#0  0x000000080fee305a in thr_kill () from /lib/libc.so.7
(gdb) bt
#0  0x000000080fee305a in thr_kill () from /lib/libc.so.7
#1  0x000000080fee1494 in raise () from /lib/libc.so.7
#2  0x000000080fe56859 in abort () from /lib/libc.so.7
#3  0x0000000802ea8afc in dump_core () from /usr/local/lib/samba4/libsmbconf.so.0
#4  0x0000000802e9f187 in smb_panic_s3 () from /usr/local/lib/samba4/libsmbconf.so.0
#5  0x0000000801b396b7 in smb_panic () from /usr/local/lib/samba4/libsamba-util.so.0
#6  0x0000000801b39a9e in ?? () from /usr/local/lib/samba4/libsamba-util.so.0
#7  0x0000000801b39699 in ?? () from /usr/local/lib/samba4/libsamba-util.so.0
#8  0x000000080fd123b0 in ?? () from /lib/libthr.so.3
#9  0x000000080fd1197f in ?? () from /lib/libthr.so.3
#10 <signal handler called>
#11 0x0000000801d9b170 in zhandle_get_dataset () from /usr/local/lib/samba4/private/libsmb-libzfs-samba4.so
#12 0x0000000801d9cb68 in ?? () from /usr/local/lib/samba4/private/libsmb-libzfs-samba4.so
#13 0x000000081018b652 in zfs_iter_filesystems () from /usr/local/lib/libzfs.so.4
#14 0x0000000801d9ca56 in zhandle_list_children () from /usr/local/lib/samba4/private/libsmb-libzfs-samba4.so
#15 0x0000000801d9cd36 in cache_zhandle_list_children () from /usr/local/lib/samba4/private/libsmb-libzfs-samba4.so
#16 0x0000000801580ce3 in ?? () from /usr/local/lib/samba4/private/libsmbd-base-samba4.so
#17 0x000000081b2e3e7b in ?? () from /usr/local/lib/shared-modules/vfs/ixnas.so
#18 0x000000081b4f96bb in ?? () from /usr/local/lib/shared-modules/vfs/shadow_copy_zfs.so
#19 0x000000081b706872 in ?? () from /usr/local/lib/shared-modules/vfs/streams_xattr.so
#20 0x000000081b918f43 in ?? () from /usr/local/lib/shared-modules/vfs/fruit.so
#21 0x000000081bb2eeb8 in ?? () from /usr/local/lib/shared-modules/vfs/aio_fbsd.so
#22 0x00000008014fe0d9 in ?? () from /usr/local/lib/samba4/private/libsmbd-base-samba4.so
#23 0x00000008014fd914 in make_connection_smb2 () from /usr/local/lib/samba4/private/libsmbd-base-samba4.so
#24 0x0000000801512b2d in smbd_smb2_request_process_tcon () from /usr/local/lib/samba4/private/libsmbd-base-samba4.so
#25 0x000000080150ab4b in smbd_smb2_request_dispatch () from /usr/local/lib/samba4/private/libsmbd-base-samba4.so
#26 0x000000080150ddd9 in ?? () from /usr/local/lib/samba4/private/libsmbd-base-samba4.so
#27 0x00000008021ab68c in tevent_common_invoke_fd_handler () from /usr/local/lib/samba4/private/libtevent.so.0
#28 0x00000008021ae4d3 in ?? () from /usr/local/lib/samba4/private/libtevent.so.0
#29 0x00000008021aa8b1 in _tevent_loop_once () from /usr/local/lib/samba4/private/libtevent.so.0
#30 0x00000008021aab12 in tevent_common_loop_wait () from /usr/local/lib/samba4/private/libtevent.so.0
#31 0x00000008014fa08c in smbd_process () from /usr/local/lib/samba4/private/libsmbd-base-samba4.so
#32 0x000000000102f04f in ?? ()
#33 0x00000008021ab68c in tevent_common_invoke_fd_handler () from /usr/local/lib/samba4/private/libtevent.so.0
#34 0x00000008021ae4d3 in ?? () from /usr/local/lib/samba4/private/libtevent.so.0
#35 0x00000008021aa8b1 in _tevent_loop_once () from /usr/local/lib/samba4/private/libtevent.so.0
#36 0x00000008021aab12 in tevent_common_loop_wait () from /usr/local/lib/samba4/private/libtevent.so.0
#37 0x000000000102d59f in ?? ()
#38 0x000000000102ca60 in main ()
(gdb) frame 11
#11 0x0000000801d9b170 in zhandle_get_dataset () from /usr/local/lib/samba4/private/libsmb-libzfs-samba4.so
(gdb) x/8i $pc-8
   0x801d9b168 <zhandle_get_dataset+248>:	callq  0x801d98e20 <_talloc_zero@plt>
   0x801d9b16d <zhandle_get_dataset+253>:	mov    %rax,%r15
=> 0x801d9b170 <zhandle_get_dataset+256>:	mov    %rax,0x20(%r12)
   0x801d9b175 <zhandle_get_dataset+261>:	mov    %r14,%rdi
   0x801d9b178 <zhandle_get_dataset+264>:	callq  0x801d99550
   0x801d9b17d <zhandle_get_dataset+269>:	test   %rax,%rax
   0x801d9b180 <zhandle_get_dataset+272>:	je     0x801d9b1ca <zhandle_get_dataset+346>
   0x801d9b182 <zhandle_get_dataset+274>:	mov    %rax,%rbx
(gdb) i reg r12
r12            0x0                 0
(gdb) 

Looking at the source:

+	dsout = talloc_zero(mem_ctx, struct zfs_dataset);
+	dsout->mountpoint = talloc_zero_size(dsout, PATH_MAX);
+	dsout->zhandle = zfsp_ext;
+	dsout->dataset_name = talloc_strdup(dsout, zfs_get_name(zfsp));

The assumption is it's a null dereference from a failed talloc_zero.

Either case this is not correct on the freenas port.

The printed "Pool1/kali" is a ZFS volume(not filesystem) that is not mounted.


@samba-team: Thanks for the pointers. I guess you can close this.






I would read that as
Comment 3 Volker Lendecke 2020-03-30 08:40:55 UTC
Failing talloc_zero is really extreme. I'm sure that under such an extreme memory pressure Samba has thousands of potential segfaults. Nevertheless, closing this so far. FreeNAS people, feel free to re-open if you detect something that needs fixing in upstream Samba.
Comment 4 Andrew Walker 2020-03-30 09:42:23 UTC
Apologies, 12.0 is currently a nightly development snapshot and a little (or very) rough around the edges. Our bugtracker is at jira.ixsystems.com. I'll open a ticket for this issue there.