Starting nmbd with "-F -d 2 -S" reveals the below output.

First we get:

added interface en2 ip=10.0.17.14 bcast=10.0.17.255 netmask=

But then later we get:

added interface en2 ip=10.0.17.14 bcast=10.0.17.191 netmask=

Here's the top part of the log:

nmbd version 3.6.5 started.
Copyright Andrew Tridgell and the Samba Team 1992-2011
Registered MSG_REQ_POOL_USAGE
Registered MSG_REQ_DMALLOC_MARK and LOG_CHANGED
standard input is not a socket, assuming -D option
Becoming a daemon.
added interface en2 ip=10.0.17.14 bcast=10.0.17.255 netmask=
added interface lo0 ip=127.0.0.1 bcast=127.255.255.255 netmask=
create_subnets: Ignoring loopback interface.
making subnet name:10.0.17.14 Broadcast address:10.0.17.255 Subnet mask:255.255.255.0
making subnet name:UNICAST_SUBNET Broadcast address:0.0.0.0 Subnet mask:0.0.0.0
making subnet name:REMOTE_BROADCAST_SUBNET Broadcast address:0.0.0.0 Subnet mask:0.0.0.0
load_lmhosts_file: Can't open lmhosts file /opt/local/samba/etc/samba/lmhosts. Error was No such file or directory
added interface en2 ip=10.0.17.14 bcast=10.0.17.191 netmask=
added interface lo0 ip=127.0.0.1 bcast=127.255.255.255 netmask=
Found new interface 10.0.17.14
bind failed on port 137 socket_addr = 10.0.17.191.
Error = Can't assign requested address
nmbd_subnetdb:make_subnet()
  Failed to open nmb bcast socket on interface 10.0.17.191 for port 137.  Error was Can't assign requested address
reload_interfaces: Ignoring loopback interface 127.0.0.1
Deleting dead interface 10.0.17.14

It seems everything is fine until we run the first "send_host_announcement" call. 

Using -d 3 on an AIX 5.3 system (where this works fine):

send_host_announcement: type 819b03 for host OPTAIX53TS on subnet 10.0.17.16 for workgroup CSWG
added interface en2 ip=10.0.17.16 bcast=10.0.17.255 netmask=

Using -d 3 on an AIX 6.1 system (where the bcast address is later miscalculated):

send_host_announcement: type 819b03 for host OPTAIXNIM01 on subnet 10.0.17.14 for workgroup CSWG
added interface en2 ip=10.0.17.14 bcast=10.0.17.191 netmask=

Hi, Ben.

Can you try the patch posted in
http://lists.samba.org/archive/samba-technical/2011-August/078954.html ?
I cannot try this patch because I cannot reproduce this bug.

See also Bug 6970 – getifaddrs() issues on AIX
https://bugzilla.samba.org/show_bug.cgi?id=6970

I changed the source like this:

--- lib/replace/getifaddrs.c	2011-01-07 13:24:23.000000000 +0100
+++ lib/replace/getifaddrs.c	2011-01-07 18:24:13.000000000 +0100
@@ -58,7 +58,7 @@
 {
 	struct sockaddr *ret;
 	socklen_t socklen;
-#ifdef HAVE_SOCKADDR_SA_LEN
+#if defined(HAVE_SOCKADDR_SA_LEN) && !defined(AIX)
 	socklen = sa->sa_len;
 #else
 	socklen = sizeof(struct sockaddr_storage);

and recompiled successfully. I am testing on more machines, but so far, this seems to work better.

I take it back, I am again getting this error:

[2012/09/20 16:54:37,  0] nmbd/nmbd.c:860(main)
  nmbd version 3.6.5 started.
  Copyright Andrew Tridgell and the Samba Team 1992-2011
[2012/09/20 16:54:37,  0] nmbd/nmbd.c:888(main)
  standard input is not a socket, assuming -D option
[2012/09/20 16:54:37,  0] lib/util_sock.c:667(open_socket_in)
  bind failed on port 137 socket_addr = 10.0.17.239.
  Error = Can't assign requested address
[2012/09/20 16:54:37,  0] nmbd/nmbd_subnetdb.c:127(make_subnet)
  nmbd_subnetdb:make_subnet()
    Failed to open nmb bcast socket on interface 10.0.17.239 for port 137.  Error was Can't assign requested address
[2012/09/20 16:54:37,  0] nmbd/nmbd.c:974(main)
  ERROR: Failed when creating subnet lists. Exiting.

Ugh.. never mind. The patch I applied was in reverse... fail. A fresh rebuild and I again have good results on two AIX systems, one on 5.3 and one on 6.1.

Can we maybe get this fixed integrated into the next possible release?

Thanks.

What should I do?

Please help get the following patch:

--- lib/replace/getifaddrs.c    2011-01-07 13:24:23.000000000 +0100
+++ lib/replace/getifaddrs.c    2011-01-07 18:24:13.000000000 +0100
@@ -58,7 +58,7 @@
 {
     struct sockaddr *ret;
     socklen_t socklen;
-#ifdef HAVE_SOCKADDR_SA_LEN
+#if defined(HAVE_SOCKADDR_SA_LEN) && !defined(AIX)
     socklen = sa->sa_len;
 #else
     socklen = sizeof(struct sockaddr_storage);

Applied into the mainstream releases.

Created attachment 8356 [details]
Patch for master and v3-6-test

I've created a patch by git format-patch.

I hate adding #ifdef AIX's for what looks like an AIX bug in their sa_len return.

Is there some way we can detect this in configure ?

Jeremy.

Jeremy -
I have support for the affected system. I can open a PMR with IBM assuming you are able to prove a defect in one of the system functions.

Let me know if that's something you're able to prove/disprove and I will move forward with triaging the IBM support process.

Well not counting the trailing zeros in the IP address just seems completely broken to me. But we can't wait for IBM to fix, that would take forever. Easier to just add a configure test to detect this problem and fix it inside Samba IMHO.

Jeremy.

I would at least try to open a PMR at the IBM support to ask if they think this is by design or an error.
There have been corrections to ioctls in the past, so maybe they'll accept it as a problem.

I am currently upgrading my AIX machine to the latest AIX6.1 level and see if the problem exists in that as well. More about that next week.

Still fighting to a get a build on my AIX machine due to Bug 9557

Just an FYI, I'm on AIX 6100-06-05-1115 using samba 3.6.4 (precompiled by thewrittenword guys for AIX) and had similar problems with the netmask.
Command: /opt/TWWfsw/samba364/sbin/nmbd -F -d 5 -S
- Non-working configuration and debug output from nmbd (nmbd bin explicit required to get nmbd to start)
 Relevant Line(s) in smb.conf
  ---cut---
  bind interfaces only = yes
  nmbd bind explicit broadcast = no
  interfaces = en1
  ---cut---

 Debug output from nmbd - badly calculated netmask
  ---cut---
  dump workgroup on subnet       10.0.9.79: netmask= 255.255.255.16:
  ---cut---
 
- Working configuration and debug output from nmbd
 Relevant Line(s) in smb.conf (nmbd bin explicit no longer required)
  ---cut---
  bind interfaces only = yes
  interfaces = 10.0.9.79/255.255.255.0 127.0.0.1
  ---cut---

 Debug output from nmbd - correct netmask
  ---cut---
  dump workgroup on subnet       10.0.9.79: netmask=  255.255.255.0:
  ---cut---

Christian, precompiled may help you escape from Bug 9557 indirectly as you shouldn't need gcc installed to run things.

*** Bug 9294 has been marked as a duplicate of this bug. ***