Bug 13157 - Winbindd refuses new logins after "Bad SMB2 signature for message"
Summary: Winbindd refuses new logins after "Bad SMB2 signature for message"
Status: NEW
Alias: None
Product: Samba 4.1 and newer
Classification: Unclassified
Component: Winbind (show other bugs)
Version: 4.7.2
Hardware: x64 FreeBSD
: P5 normal (vote)
Target Milestone: ---
Assignee: Samba QA Contact
QA Contact: Samba QA Contact
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-11-21 15:48 UTC by Peter Eriksson
Modified: 2020-05-21 10:48 UTC (History)
1 user (show)

See Also:


Attachments
smb.conf file (2.86 KB, text/plain)
2017-11-21 15:48 UTC, Peter Eriksson
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Peter Eriksson 2017-11-21 15:48:09 UTC
Created attachment 13799 [details]
smb.conf file

Every now and then the Winbindd processes on our pretty busy six Samba fileservers (around 200-400 users per server) seems to stop responding - causing Samba to refuse new SMB connections. Today we saw it happen on two different servers at 12:09, and then one a third on at 14:47...

The timing seems to happen at the same time as we in the "log.smbd" file see the following errors:

> # egrep -A5 'signing' /var/samba/logs/log.smbd
> [2017/11/21 14:47:22.282388,  0] ../libcli/smb/smb2_signing.c:171(smb2_signing_check_pdu)
>  Bad SMB2 signature for message
> [2017/11/21 14:47:22.282480,  0] ../lib/util/util.c:515(dump_data)
>   [0000] 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00   ........ ........
> [2017/11/21 14:47:22.282522,  0] ../lib/util/util.c:515(dump_data)
>   [0000] C8 AA BD 98 1C CD D4 F7   47 B8 79 B6 EF 90 6D AF   ........ G.y...m.

Killing and restarting winbindd seems to allow the smbd processes to allow new connections again. Refuses both username+password & Kerberos-authenticated connections.

Operating System: 
  FreeBSD 11.1

Hardware: 
  Dell PowerEdge 730xd with 256GB RAM, 10Gbps Ethernet and ~140TB of ZFS storage

Joined with a Windows 2012 AD domain (6 AD servers) with around 100k users and many groups. winbind users & groups enumeration is disabled.

Attaching our smb.conf file. Had a quick look into the libcli/smb/smb2_signing.c file but could really see anything obviously wrong...
Comment 1 Björn Jacke 2020-05-21 00:52:59 UTC
is this still an issue?
Comment 2 Peter Eriksson 2020-05-21 10:48:23 UTC
Well, sort of. We are now running Samba 4.11-series and while we don't see that exact error message anymore we still experience regular winbindd freezes on our busy servers.

Every 10:th hour after we restart the winbindd processes the busy ("busy is in "many users connected, not necessarily doing a lot") ones tend to freeze up. I'm suspecting something going wrong at the same time as the AD Kerberos service ticket expires and is supposed to be renewed. It normally doesn't happen on less busy servers.

(We have a workaround though - we run a cron job that (around the time of when we know this is happening) runs a series of quick tests and if it fails it just restarts winbindd (kill, and if that doesn't help (sometimes) then "kill -9").

(We restart winbindd at 07:00 every morning so it won't affect our users untill 17:00 - so we can avoid disrupting users during prime office hours). (And then at 03:00 but that is not really a problem :-)

- Peter