We tried to analyze the issue with the core dumps we got using gdb:

the 'share_entries_exist' variable seems not to be present, as in in the previous line, the 'key' variable doesn't seem to exist either.

We have available the full core dump if you would like to have the full information. In our case this was the most disruptive core dump we have had so far, so we are mostly interested in finding the cause of this problem.

(gdb) up 5
#5  0x00007f6b971700ba in share_mode_data_store (rec=0x555fb57a6b00, d=0x555fb57b55a0) at ../../source3/locking/share_mode_lock.c:448
448            SMB_ASSERT(!share_entries_exist);
(gdb) l
443
444        if (!d->have_share_modes) {
445            TDB_DATA key = dbwrap_record_get_key(rec);
446            bool share_entries_exist;
447            share_entries_exist = dbwrap_exists(share_entries_db, key);
448            SMB_ASSERT(!share_entries_exist);
449
450            TALLOC_FREE(d->delete_tokens);
451            d->num_delete_tokens = 0;
452
(gdb) p share_entries_db
$2 = (struct db_context *) 0x555fb5779240
(gdb) p *share_entries_db
$3 = {fetch_locked = 0x7f6b96e38d90 <db_ctdb_fetch_locked>, try_fetch_locked = 0x7f6b96e38ce0 <db_ctdb_try_fetch_locked>,
  traverse = 0x7f6b96e37350 <db_ctdb_traverse>, traverse_read = 0x7f6b96e37290 <db_ctdb_traverse_read>, get_seqnum = 0x7f6b96e36fc0 <db_ctdb_get_seqnum>,
  transaction_start = 0x7f6b96e36c60 <db_ctdb_transaction_start>, transaction_start_nonblock = 0x0,
  transaction_commit = 0x7f6b96e39330 <db_ctdb_transaction_commit>, transaction_cancel = 0x7f6b96e36770 <db_ctdb_transaction_cancel>,
  parse_record = 0x7f6b96e377d0 <db_ctdb_parse_record>, parse_record_send = 0x7f6b96e37560 <db_ctdb_parse_record_send>,
  parse_record_recv = 0x7f6b96e37550 <db_ctdb_parse_record_recv>, do_locked = 0x0, exists = 0x0, wipe = 0x0, check = 0x0, id = 0x7f6b96e368e0 <db_ctdb_id>,
  name = 0x555fb5779350 "share_entries.tdb", private_data = 0x555fb5777a30, lock_order = DBWRAP_LOCK_ORDER_3, persistent = false}
(gdb) p key
$4 = <optimized out>
(gdb) p share_entries_exist
$5 = <optimized out>

I dump the backtrace we got form the coredump we got in abrt with gdb and debug symbols enabled, which will be more meaningful:

#0  0x00007f6b95e8470f in raise () from /lib64/libc.so.6
#1  0x00007f6b95e6eb25 in abort () from /lib64/libc.so.6
#2  0x00007f6b9708a054 in dump_core () at ../../source3/lib/dumpcore.c:338
#3  0x00007f6b9709965c in smb_panic_s3 (
    why=0x7f6b9733fd58 "assert failed: !share_entries_exist")
    at ../../source3/lib/util.c:853
#4  0x00007f6b9763f631 in smb_panic (
    why=why@entry=0x7f6b9733fd58 "assert failed: !share_entries_exist")
    at ../../lib/util/fault.c:174
#5  0x00007f6b971700ba in share_mode_data_store (rec=0x555fb57a6b00, d=0x555fb57b55a0)
    at ../../source3/locking/share_mode_lock.c:448
#6  share_mode_lock_destructor (lck=<optimized out>)
    at ../../source3/locking/share_mode_lock.c:686
#7  share_mode_lock_destructor (lck=<optimized out>)
    at ../../source3/locking/share_mode_lock.c:675
#8  0x00007f6b968573a4 in _talloc_free () from /lib64/libtalloc.so.2
#9  0x00007f6b972ab9ef in open_directory (conn=conn@entry=0x555fb5753520, 
    req=req@entry=0x555fb57a4000, smb_dname=smb_dname@entry=0x555fb574ed70, 
    access_mask=<optimized out>, access_mask@entry=1048705, 
    share_access=share_access@entry=7, create_disposition=create_disposition@entry=1, 
    create_options=1, file_attributes=16, pinfo=0x7ffec433af54, result=0x7ffec433af60)
    at ../../source3/smbd/open.c:4602
#10 0x00007f6b972aec76 in create_file_unixpath (conn=conn@entry=0x555fb5753520, 
    try=0x555fb57a4000, smb_fname=smb_fname@entry=0x555fb574ed70, access_mask=access_mask@entry=1048705, share_access=share_access@entry=7, create_disposition=create_disposition@entry=1, create_options=1, file_attributes=0, oplock_request=0, lease=<optimized out>, allocation_size=0, private_flags=0, sd=0x0, ea_list=0x0, result=0x7ffec433b118, pinfo=0x7ffec433b114) at ../../source3/smbd/open.c:5593
#11 0x00007f6b972b0ae6 in create_file_default (conn=0x555fb5753520, req=0x555fb57a4000, root_dir_fid=<optimized out>, smb_fname=0x555fb574ed70, access_mask=1048705, share_access=7, create_disposition=1, create_options=1, file_attributes=0, oplock_request=0, lease=0x0, allocation_size=0, private_flags=0, sd=0x0, ea_list=0x0, result=0x555fb57a5dc8, pinfo=0x555fb57a5ddc, in_context_blobs=0x7ffec433b260, out_context_blobs=0x555fb57b5430) at ../../source3/smbd/open.c:6045
#12 0x00007f6b972e7c91 in smbd_smb2_create_send (in_context_blobs=..., in_name=<optimized out>, in_create_options=<optimized out>, in_create_disposition=<optimized out>, in_share_access=<optimized out>, in_file_attributes=<optimized out>, in_desired_access=<optimized out>, in_impersonation_level=<optimized out>, in_oplock_level=<optimized out>, smb2req=0x555fb579efe0, ev=<optimized out>, mem_ctx=0x555fb579efe0) at ../../source3/smbd/smb2_create.c:983
#13 smbd_smb2_request_process_create (smb2req=smb2req@entry=0x555fb579efe0) at ../../source3/smbd/smb2_create.c:268
#14 0x00007f6b972de7cb in smbd_smb2_request_dispatch (req=req@entry=0x555fb579efe0) at ../../source3/smbd/smb2_server.c:2709
#15 0x00007f6b972dff10 in smbd_smb2_io_handler (fde_flags=<optimized out>, xconn=0x555fb5795ba0) at ../../source3/smbd/smb2_server.c:4060
#16 smbd_smb2_connection_handler (ev=<optimized out>, fde=<optimized out>, flags=<optimized out>, private_data=<optimized out>) at ../../source3/smbd/smb2_server.c:4098
#17 0x00007f6b96436443 in tevent_common_invoke_fd_handler () from /lib64/libtevent.so.0
#18 0x00007f6b9643c9bf in epoll_event_loop_once () from /lib64/libtevent.so.0
#19 0x00007f6b9643a99b in std_event_loop_once () from /lib64/libtevent.so.0
#20 0x00007f6b96435b15 in _tevent_loop_once () from /lib64/libtevent.so.0
#21 0x00007f6b96435dbb in tevent_common_loop_wait () from /lib64/libtevent.so.0
#22 0x00007f6b9643a92b in std_event_loop_wait () from /lib64/libtevent.so.0
#23 0x00007f6b972ce09f in smbd_process (ev_ctx=0x555fb5752440, msg_ctx=<optimized out>, sock_fd=42, interactive=<optimized out>) at ../../source3/smbd/process.c:4170
#24 0x0000555fb3d50d00 in smbd_accept_connection (ev=0x555fb5752440, fde=<optimized out>, flags=<optimized out>, private_data=<optimized out>) at ../../source3/smbd/server.c:1012
#25 0x00007f6b96436443 in tevent_common_invoke_fd_handler () from /lib64/libtevent.so.0
#26 0x00007f6b9643c9bf in epoll_event_loop_once () from /lib64/libtevent.so.0
#27 0x00007f6b9643a99b in std_event_loop_once () from /lib64/libtevent.so.0
#28 0x00007f6b96435b15 in _tevent_loop_once () from /lib64/libtevent.so.0
#29 0x00007f6b96435dbb in tevent_common_loop_wait () from /lib64/libtevent.so.0
#30 0x00007f6b9643a92b in std_event_loop_wait () from /lib64/libtevent.so.0
#31 0x0000555fb3d4b2a2 in smbd_parent_loop (parent=0x555fb576a140, ev_ctx=0x555fb5752440) at ../../source3/smbd/server.c:1359
#32 main (argc=<optimized out>, argv=<optimized out>) at ../../source3/smbd/server.c:2197

Any update on this? Is there any more input needed for investigating it?
As mentioned, this issue was quite disruptive in our machines, the service was completely unusable during the time it happened. We tested the version 4.12.7 for a couple of weeks and everything was ready, but when we hit the peak of users was when it started happenning, so on our opinion it's quite important to have a look to this issue, and fix it for newer versions.

(In reply to Aritz Brosa from comment #3)
If this is urgent and you don't have the time to wait for someone to volunteer and look into this, you might want to consider dedicated support from you OS vendor or from one of the companies listed here: https://www.samba.org/samba/support/globalsupport.html.

(In reply to Ralph Böhme from comment #4)
We understand and appreciate the support is based on community effort, nevertheless it seemed that the failed assert was quite visible. At the same time, looking at how the code has evolved, version 4.13 got an important refactoring of the problematic part:
https://github.com/samba-team/samba/commit/49951b283d98d81a2201341624b4bcdd3eba9ec4 

Therefore an eventual fix for this issue could only have targeted version 4.12, not future ones. We will test version 4.13 to see if it reproduces a similar issue around the share_entries struct, and will update this bug report once we have a conclusion on that for the latest version.