on solaris getent group DOMAIN\\somegroup hangs when the group is too large, e.g. there are too many group members. Solaris seems to have a limit for the members of each group (can be determined with the _SC_GETGR_R_SIZE_MAX sysconf(3C) parameter). when this limit is exceeded the getent group call hangs. The question is if the hanging is a bug of Solaris or if the nss winbind module needs to handle that case of oversized groups differently.
Alas, I don't have access to a Solaris box (or Illumos box!) right now. My feeling is: Hanging is broken. Now, which side do we fix it on... Probably both.
Created attachment 10659 [details] Patch for large groups in solaris nss The problem with large groups on Solaris/Illumos in the the NSS winbind module is Solaris wants the return value to be NSS_UNAVAIL if the buffer given is too small for getgrnam_r. The current code return NSS_TRYAGAIN which causes Solaris/Illumos to loop without trying to resize the buffer. This patch checks for the case of ERANGE and NSS_STATUS_TRYAGAIN and changes the return value to NSS_STATUS_UNAVAIL. It only covers the group database which is where I and I expect most other people see this issue. It could also be extended to the other databases. I have tested this on OmniOS 151006 which has the same NSS code as Illumos-gate and I assume this is still the same in Solaris proper.
this is fixed in master with d3e51b9cfe3d56530253571e020af72da1877044 Ralf: can you review+ for the cherry-pick to the release branches?
Pushed to autobuild-v4-[2|3]-test.
(In reply to Karolin Seeger from comment #4) Pushed to both branches. Closing out bug report. Thanks!