i compiled samba 3.0.0rc1 with ./configure --prefix=/usr/local --with-acl- support that all worked nice. but after that i wanted to modify groups with smbgroupedit "Failed to open group mapping database". after playign a bit i found out that the file group_mapping.tdb was generated in /usr/local/var/locks (as it correct with the prefix) but smbgroupedit wanted to open it in /usr/local/samba/var/locks. afte creating that it gived me a a few of the groups but i think it is not working this way. i even tried to link does two dirs to each other and i got the "error enum_group_mapping: could not allocate memory for privileges"
smbgroupedit does not exists any more. Sounds like you are using a binary from an older release. Group mappings are managed by 'net groupmap'. This change was made before beta 1 IIRC.
originally reported against one of the 3.0.0rc[1-4] releases. Cleaning up non-production versions.