We are seeing the same problem with samba 3.4.5 on RHEL.
We have two AD domain controllers, if any of the two goes down, winbind hangs until it is restarted, at the exact place, it is unable to read /tmp/.winbindd/pipe.
with an lsof/strace, it looks like none of the winbind processes are looking at the right FD.

Can you try the patches attached to bug 7295?
https://bugzilla.samba.org/show_bug.cgi?id=7295

Or use the current v3-5-test branch

I just tested the build from branch v3-5-test (3.5.2-GIT-9620f5f-test) and reproduced on my lab.

It is a little bit better, though not perfect. That is:
- under the same test conditions, winbind remains stable and eventually produces an answer
- however, the answer is not correct because is does not try to connect to the other ActiveDirectory servers.

Here is the detail of lab
cat /var/run/samba/smb_krb5/krb5.conf.PREPROD
[libdefaults]
        default_realm = PREPROD.xxxx.xx
        default_tgs_enctypes = RC4-HMAC DES-CBC-CRC DES-CBC-MD5
        default_tkt_enctypes = RC4-HMAC DES-CBC-CRC DES-CBC-MD5
        preferred_enctypes = RC4-HMAC DES-CBC-CRC DES-CBC-MD5

[realms]
        PREPROD.xxxx.xx = {
                kdc = 172.29.129.22
        kdc = 172.29.129.21
        kdc = 172.29.129.21
        }

The test conditions are reproduced by

iptables -D OUTPUT -p tcp -d 172.29.129.22 -j REJECT

And in such conditions, winbind does not switch to the other available ActiveDirectory node and tries to use the already established connections

Can you rerun the test also blocking UDP?

The dc discovery uses UDP to detect if a dc is reachable.

We still should fallback if only TCP is blocked, but it should only cause delays
or temporary failures. If it doesn't recovery at all in this case we have an
additional bug to fix.

I checked out 3.5.2-GIT-UNKNOWN-test yesterday and installed it on two servers in my dev environment. I have two AD servers in my lab. I made the linux servers leave the domain, and then re-join the domain. I did not login to the servers for an hour after doing so to make sure nothing got cached. I then rebooted the one AD server, waited 15 minutes after it came up, then rebooted the other AD server.  I then waited another hour before logging in to the linux servers.
the one server with 3.5.2-GIT has been working like a champ. no issues at all.
The other gives this from the logs:
[2010/04/02 17:59:08.331357,  1] winbindd/winbindd_ads.c:126(ads_cached_connection)
  ads_connect for domain AWESOME failed: No logon servers
[2010/04/02 17:59:14.411718,  1] winbindd/winbindd_ads.c:126(ads_cached_connection)
  ads_connect for domain AWESOME failed: No logon servers
[2010/04/02 18:01:08.946911,  1] winbindd/winbindd_ads.c:126(ads_cached_connection)
  ads_connect for domain AWESOME failed: No logon servers

On further investigation, it appeared the ip it was trying to contact the AD server on was unreachable (I accidentally had the secondary AD server register a "private" ip on its other NIC into DNS, which neither linux server can hit). However it seemed as if winbind would not try to look at any other ips for that domain, like it was stuck on checking that ip until it was reachable, or winbind got restarted (which fixed the problem). I let that server sit for about an hour before I tried logging in, so it was trying to reach that IP for about an hour. The other ips advertised for the domain were reachable the entire time. 
Is this default behavior or a bug? Should winbind keep trying the ips listed from dns for the domain until it can talk to one instead of zoning in on one ip, or am I misunderstanding how its supposed to work?

After fixing the dns issue I decided to try this again, with all ips returned for the domain being reachable. 
one server worked great (this time the one that had the problem below is working great and the other one that worked fine before is having weird problems).
Here is the problem:
ssh logins hang, as it is trying to do group lookups. getent groups/passwd hangs. However, if I were to strace -p <pid> of the main winbind process, group lookups begin to work so long as I am attached that pid (I thought I was crazy at first but after 15 minutes of nothing but that I have completely verified that is the behavior I am seeing). I would be more than happy to take a video capture to show this. I wanted to get an strace to see where it was hanging, and everytime I attach to the processes, it immediately returns correct values. 
If I do not attach to the process via strace, getent group/passwd only returns local users. 

strace here: http://pastebin.com/dg08uzdp

logs:

  winbindd_getgrent_done: done with 17 groups
[2010/04/02 19:54:17.120958, 10] winbindd/winbindd.c:662(wb_request_done)
  wb_request_done[20109:GETGRENT]: NT_STATUS_OK
[2010/04/02 19:54:17.122203, 10] winbindd/winbindd.c:723(winbind_client_response_written)
  winbind_client_response_written[20109:GETGRENT]: deliverd response to client
[2010/04/02 19:54:17.124385, 10] winbindd/winbindd.c:600(process_request)
  process_request: Handling async request 20109:GETGRENT
[2010/04/02 19:54:17.125035,  3] winbindd/winbindd_getgrent.c:51(winbindd_getgrent_send)
  [20109]: getgrent
[2010/04/02 19:54:17.125669,  5] winbindd/winbindd_getgrent.c:149(winbindd_getgrent_recv)
  getgrent failed: NT_STATUS_NO_MORE_ENTRIES
[2010/04/02 19:54:17.126308, 10] winbindd/winbindd.c:662(wb_request_done)
  wb_request_done[20109:GETGRENT]: NT_STATUS_NO_MORE_ENTRIES
[2010/04/02 19:54:17.127456, 10] winbindd/winbindd.c:723(winbind_client_response_written)
  winbind_client_response_written[20109:GETGRENT]: deliverd response to client
[2010/04/02 19:54:17.129106, 10] winbindd/winbindd.c:600(process_request)
  process_request: Handling async request 20109:ENDGRENT
[2010/04/02 19:54:17.129893, 10] winbindd/winbindd.c:662(wb_request_done)
  wb_request_done[20109:ENDGRENT]: NT_STATUS_OK
[2010/04/02 19:54:17.131047, 10] winbindd/winbindd.c:723(winbind_client_response_written)
  winbind_client_response_written[20109:ENDGRENT]: deliverd response to client
[2010/04/02 19:54:17.132178,  6] winbindd/winbindd.c:823(winbind_client_request_read)
  closing socket 32, client exited

oh a restart of winbind fixes the issue

sorry for the many comments. that last issue seems to be a fluke. things have been working great using 3.5.2-test.
However the issue before that can be reproduced (if one of those ips is unreachable winbind seems to hone in on just that ip and not try any other ips), I would love to know if that is a bug or not. Again sorry for the many comments, we have planning on using this for a large (1200+ server) deployment and this is the only bug holding us back. Thanks!

I confirm that on version 3.5.2-GIT-9620f5f-test, and blocking udp as well as tcp (as it were accidentally on my production site), it works as expected (switching to other active directory). 

So is this one fixed in 3.5.2? I'd like to close this bug report.

I confirm it is ok in 3.5.2