Support for dynamic interfaces in smbd uses a netlink socket to respond to IP address addition and removal. However, binding to newly added addresses sometimes fails with EADDRNOTAVAIL. A bit of debugging suggests that when CTDB adds an IP address via the "ip addr add ..." command, this generates an ADDRCHANGE_ADD type message, which is what smbd monitors the netlink socket for. This suggests the netlink socket acts like a bus and the ADDRCHANGE_ADD message is being processed by the kernel and smbd at the same time. Usually the kernel wins, but sometimes smbd wins and preemptively attempts to bind. From memory the time interval between logging the receipt of an ADDRCHANGE_ADD message and: * a successful bind was ~16 microseconds * a failed bind was ~9 microseconds So, this race is very tight. Adding a retry after 1ms or 10ms should be ample. Note that we could attempt to bind for a while until the address becomes available... and then do the intended bind. However, we might as well delay and attempt a single retry.
I've been working on a solution...
Correction: "ip addr add ..." sends a RTM_NEWADDR message, which is the same message that source3/lib/addrchange.c responds to. However, looking at Linux kernel source: * The kernel IP address handling also sends such messages for both IPv4 and IPv6 * For IPv6 they are sent from places including addrconf_dad_begin() and addrconf_dad_completed() This means we see initial noise about failing to open the socket. However, the socket is eventually successfully opened. I can't recreate the bug. It was only seen on a customer's machine. I'm now wondering if we need to fix this or whether it was a temporary kernel bug. We should find out exactly what kernel version our customer was using.
This bug was referenced in samba master: 9a6eb2d26fdfd16728116837d10517a8e0e3060e 51fe0eb702c545d89e672642c82857ddf69eddbf f3c50e7561de9cb95b34f844ea23602d29043c63 b700437c10f9734b184b66f41755727d195ce437 bd63602ebc7ca25fb48a4cfb9ebe3f3554cd7230 a4dc320c004bfe6394a65c598a06759b72cf3fd7 b62ef3401d3f557a224f9caaf9e307e18ebba3fd c69a1d21b3682727c54f560dbdb689907d143d15 2a4e95ccee8d11240d6c2594fef861e611ea07e2 1d2ffe9968913612afa8338f99a0b4accd5a4bcb