My customer is deploying SuSE Linux Enterprise Desktop 10 SP1 into a large Active Directory environment.

SSH access to the SLED10 machines is restricted to members of one AD group by an AllowGroups restriction in /etc/ssh/sshd_config:

     AllowGroups  DOMAIN1\desktop-techs

When attempting to SSH to a SLED10 as a member of this group, there is a crippling delay of about 3 minutes after entering the username before the password prompt appears.  Sometimes we receive an error message:

   "server unexpectedly closed connection"

and are unable to obtain the password prompt and cannot login via SSH.

With sshd in debug mode, the last thing we saw on screen before the delay began was a message about reverse mapping the incoming host, however nslookups by hostname or IP address were near instantaneous.

A network capture revealed tremendous traffic back to the PC's domain controller, which lead us to root of the issue in the log.winbindd logfile.

Samba seems to be downloading the entire load of domain groups and their group memberships.  Tens of thousands of users.

Confirmation of a Samba problem comes from directory listings of /tmp that takes more than a minute to return output.

Listings of /var or /opt or /etc are normal and fast, but listing the /tmp folder when there are files owned by domain users in it results in this stall.

Again, the Samba logs show the retrieval of almost 31,000 usernames from AD just from performing a directory listing of /tmp.

This issue seems to have been noted on the Samba mailing lists, this thread in particular reviews the slow directory listings issue:

    http://lists.zerezo.com/samba/msg30977.html

and this response:

    http://lists.zerezo.com/samba/msg30981.html

confirms the root cause of the problem, indicating the issue is addressed in Samba 3.2.0:

  Hi Ralf,

  I assume that you are using "security = ads" and I assume that
  your AD setup has groups with lots of members?

  This is a known problem then that has been fixed in current 
  samba (SAMBA_3_2 as of today): The ads version of the function 
  lookup_groupmem (used to retrieve the members of a given group) 
  showed poor performance on large groups. I recently improved
  the performance of this call (starting with svn revisions r23070
  and r23072). This is in SAMBA_3_2 and in SAMBA_3_2_0, so it will 
  be in the next release (3.2.0). 

  There is no way to improve the performance significantly with
  3.0.24 (except patching). So I suggest that you grab the latest 
  sources with svn (see http://www.samba.org/samba/devel/), you 
  can also get the upcoming release branch SAMBA_3_2_0 here) or 
  get the unpacked sources with rsync like so:
  "rsync -avSH samba.org::ftp/pub/unpacked/samba_3_2/ ./samba_3_2"
  and then compile it yourself.

  The reason why lookup_groupmem gets used in "ls -l" at all is
  that the getgrgid library call is used to resolve the gids into
  names, and this call returns not only the name but the whole 
  group structure, including the list of members.

  So to confirm my assumptions above, you could compare the
  runtime of "ls -l" to that of "ls -ln": The latter should be
  much faster! 

  Cheers, Michael

I searched the Samba bugzilla and didn't find anything like this noted, so I thought I'd ensure you had an entry for tracking this even though it's possible this may already be fixed in 3.2.

I have Samba logs from an SSH login attempt, however it would be very difficult to sanitize them to the point the customer would allow them to be shared.

  -Randy Palmer