The functions aixacl_sys_acl_get_file and aixacl_sys_acl_get_fd are not declared in any header file. This results in:

Compiling lib/sysacls.c
"/tmp/samba-3.5.1/source3/lib/sysacls.c", line 396.39: 1506-068 (W) Operation between types "struct smb_acl_t*" and "int" is not allowed.
"/tmp/samba-3.5.1/source3/lib/sysacls.c", line 401.37: 1506-068 (W) Operation between types "struct smb_acl_t*" and "int" is not allowed.

CC=/usr/vac/bin/cc, gcc gives a similar warning.

Since I'm compiling for 64-bit, the 64 bit long pointer struct smb_acl_t* gets truncated to a 32 bit int. This results in SIGSEGV, when the pointer later is used.

As a quick test, I put these two lines into lib/sysacl.c:

SMB_ACL_T aixacl_sys_acl_get_file(vfs_handle_struct*, const char*, SMB_ACL_TYPE_T);
SMB_ACL_T aixacl_sys_acl_get_fd(vfs_handle_struct *, files_struct *);

This fixes the bug.

The complete patch is to create and include modules/vfs_aixacl.h as already existing for other OS's.