I have to do mkuser <user> to make it work (which i think may not be required).

# mkuser testsamba
# /opt/freeware/bin/smbclient -L localhost -U testsamba
Enter IN\testsamba's password:

        Sharename       Type      Comment
        ---------       ----      -------
        samba_share     Disk
        IPC$            IPC       IPC Service (Samba 4.10.11)
Reconnecting with SMB1 for workgroup listing.

        Server               Comment
        ---------            -------
        AIXOSS2              Samba 4.10.11

        Workgroup            Master
        ---------            -------
        IN                   AIXOSS2

So i guess there could be some configuration issue or build issue in my case.
Any hint will be greatly helpful.

I am experiencing this exact error as well, and I'm running 4.12.10 as built by IBM, so I can confirm that this is still an issue in at least that version.

For some additional information, here is my [global] section from smb.conf:

[global]
lock dir = /var/locks
pid directory = /var/locks
force user = root
read only = no
#log level = 1
log file = /var/log/samba/log.smbd
max log size = 5000k
disable netbios = yes
workgroup = MY
security = ADS
realm = MY.AD.DOMAIN
vfs objects = acl_xattr
map acl inherit = yes
winbind use default domain = yes
winbind nested groups = yes
winbind enum users = yes
winbind enum groups = yes
winbind normalize names = yes
#map to guest = bad uid
guest account = root
idmap config * : backend = tdb
idmap config * : range = 3000-7999
idmap config NSI : backend = rid
idmap config NSI : range = 10000-999999
template shell = /bin/ksh
template homedir = /home/%U
local master = no
username map = /etc/samba/user.map


And here is my krb5.conf:

[libdefaults]
        default_realm = MY.AD.DOMAIN
        default_keytab_name = FILE:/etc/krb5/krb5.keytab
        default_tkt_enctypes = des3-cbc-sha1 aes256-cts aes128-cts arcfour-hmac des-cbc-md5 des-cbc-crc
        default_tgs_enctypes = des3-cbc-sha1 aes256-cts aes128-cts arcfour-hmac des-cbc-md5 des-cbc-crc
        dns_lookup_realm = true
        dns_lookup_kdc = true
        forwardable = true

[realms]
        MY.AD.DOMAIN = {
                kdc = my.ad.domain:88
                admin_server = my.ad.domain:749
                default_domain = my.ad.domain
        }

[domain_realm]
        .my.ad.domain = MY.AD.DOMAIN
        my.ad.domain = MY.AD.DOMAIN

[logging]
        kdc = FILE:/var/krb5/log/krb5kdc.log
        admin_server = FILE:/var/krb5/log/kadmin.log
        kadmin_local = FILE:/var/krb5/log/kadmin_local.log
        default = FILE:/var/krb5/log/krb5lib.log


And my /etc/methods.cfg has this stanza:

WINBIND:
        program = /opt/freeware/lib/WINBIND.so
        options = debug



And my /etc/security/user has the following:

default:
        ...
        SYSTEM = "compat OR WINBIND"
        ...
        REGISTRY = WINBIND


And my /etc/security/login.cfg was updated as follows:

usw:
        ...
        mkhomeatlogin = true


Using these configurations, AD users can log into the AIX system via SSH.  However, connecting to \\<aix host name> results in an error message on the Windows system unless that user is defined locally.  Of course, this defeats the whole purpose of winbind.  I get the same log output as Ayappan, indicating that the authentication check against AD is successful (NT_STATUS_OK), but then when we get a NULL back from getgrset, we fail the session setup with NT_STATUS_UNSUCCESSFUL.  

I also concede that this may be a bug in AIX's LAM system or libc.  But I don't have enough information to confirm or debunk that.

I propose this bug be increased in severity to a critical severity and a higher importance, as this bug effectively indicates that winbind is completely broken on AIX.

Also, I missed a few references on my config file whereby I was trying to not list my internal configuration details.  In my [global] section please read all references to 'NSI' as 'MY'.  Should be:

idmap config MY : backend = rid
idmap config MY : range = 10000-999999

Thank you and sorry for the confusion.

This issue is also present in 4.12.15.

This issue is still present in 4.14.4.

Something isn't right here, we (Samba) shouldn't be looking up the user group list via nsswitch (or the AIX equivalent), we should be using the Kerberos PAC or the 'info3' reply.

So the issue is further up the stack I think.