I'm the current maintainer of the Samba 3.6 package for Solaris 10 as part of the OpenCSW project. 

I'm now trying to get the pam modules to work. I've used pam_winbind successfully in the past, and I've got reports it currently works fine. I've only recently found out about pam_smbpass. There's very little documentation about this one for Solaris, so there's a possibility I'm missing something, but at this point, I don't see it.

The module is configured like this in /etc/pam.conf, at the bottom of the "other" heap:
other   password required       pam_smbpass_csw.so debug nullok use_authtok try_first_pass

(the module name is changed to ensure no conflict with the system's)

To start the test, I make sure passwords are already in sync:
passwd user
smbpasswd user

Then I check it works:
su - user
smbclient \\\\server\\share

Both succeed, so far, all good.

Now I try to change it using passwd, first as user:
$ passwd
Enter existing login password:
New Password:
Permission denied

The logs show:
Jul 10 19:56:01 server passwd[5522]: [ID 272032 auth.debug] (pam_smbpass) username [user] obtained
Jul 10 19:56:01 server passwd[5522]: [ID 869136 auth.debug] (pam_smbpass) Located account for user
Jul 10 19:56:01 server passwd[5522]: [ID 871885 auth.notice] (pam_smbpass) failed auth request by user for service passwd as user
Jul 10 19:56:01 server passwd[5522]: [ID 507756 auth.notice] (pam_smbpass) failed auth request by user for service passwd as user(-18956203)
Jul 10 19:56:01 server passwd[5522]: [ID 965784 auth.notice] (pam_smbpass) 1 authentication failure from user for service passwd as user(50005)

Note the weird negative uid value above, no idea where that comes from. 

If I try as root:
# passwd user
New Password:
Re-enter new Password:
passwd: password successfully changed for user

su works with the new password:
su - user

Samba fails:
$ smbclient \\\\server\\share
Enter user's password:
session setup failed: NT_STATUS_LOGON_FAILURE

The logs show:
Jul 10 20:15:17 server passwd[5993]: [ID 272032 auth.debug] (pam_smbpass) username [user] obtained
Jul 10 20:15:17 server passwd[5993]: [ID 869136 auth.debug] (pam_smbpass) Located account for user
Jul 10 20:15:17 server passwd[5993]: [ID 632017 auth.notice] (pam_smbpass) password for (user/50005) changed by (root/0)

I used tdbdump to compare the content of passdb.tdb, and it seems wrong.

Here it is the line created when I change the password with smbpasswd (it's consistent if I replay it with the same password, only the "%\97" changes)

data(206) = "\00\00\00\00\7F\A9T|\7F\A9T|\00\00\00\00%\97\D6Q\00\00\00\00\7F\A9T|\09\00\00\00user\00\09\00\00\00SERVER\00\01\00\00\00\00\01\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\01\00\00\00\00\01\00\00\00\00\01\00\00\00\00\01\00\00\00\00\EA\03\00\00\01\02\00\00\10\00\00\00bJ\ACA7\95\CD\C1\FF\176_\AF\1F\FE\89\10\00\00\00;\1BG\E4.\04c'n=\EDl\EF4\9F\93\00\00\00\00\10\00\00\00\A8\00\15\00\00\00 \00\00\00\FF\FF\FF\FF\FF\FF\FF\FF\FF\FF\FF\FF\FF\FF\FF\FF\FF\FF\FF\FF\FF\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\EC\04\00\00"

The same line after modification via pam_smbpass, forcing the same password by root, the content is noticeably different, whatever is stored there is not the same password:

data(206) = "\00\00\00\00\7F\A9T|\7F\A9T|\00\00\00\00\9D\97\D6Q\00\00\00\00\FF\FF\FF\7F\09\00\00\00user\00\09\00\00\00SERVER\00\01\00\00\00\00\01\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\01\00\00\00\00\01\00\00\00\00\01\00\00\00\00\01\00\00\00\00\EA\03\00\00\01\02\00\00\10\00\00\00bJ\ACA7\95\CD\C1\FF\176_\AF\1F\FE\89\10\00\00\00\1B\A3Z\A9\D1\9D\B8\E7\0C9\AE\C1\BC\F2BB\00\00\00\00\10\00\00\00\A8\00\15\00\00\00 \00\00\00\FF\FF\FF\FF\FF\FF\FF\FF\FF\FF\FF\FF\FF\FF\FF\FF\FF\FF\FF\FF\FF\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\EC\04\00\00"