Bug 9184 - DNS server is blocked
DNS server is blocked
Status: RESOLVED FIXED
Product: Samba 4.0
Classification: Unclassified
Component: DNS server
4.0.0rc3
All All
: P5 normal
: ---
Assigned To: Karolin Seeger
Samba QA Contact
:
Depends on:
Blocks: 8622
  Show dependency treegraph
 
Reported: 2012-09-20 02:50 UTC by Matthieu Patou
Modified: 2012-10-29 10:55 UTC (History)
1 user (show)

See Also:


Attachments
Tcpdump trace with first packet malformed (482 bytes, application/octet-stream)
2012-09-20 16:28 UTC, Matthieu Patou
no flags Details
similar problem with trailing null bytes (408 bytes, application/octet-stream)
2012-09-20 16:46 UTC, Matthieu Patou
no flags Details
Patches for v4-0-test (3.94 KB, patch)
2012-09-22 23:21 UTC, Stefan Metzmacher
metze: review?
kai: review+
Details
Additional patch for v4-0-test to fix 100% CPU usage on freebsd (1.65 KB, patch)
2012-10-23 13:36 UTC, Stefan Metzmacher
abartlet: review+
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Matthieu Patou 2012-09-20 02:50:25 UTC
After receiving a bogus dns packet the DNS server is blocked, we managed to reproduce the problem several time and in each time the receive queue is not empty:

root@dc2:/usr/local/samba/var# netstat -anp | grep 53
tcp        0      0 10.73.100.84:53         0.0.0.0:*               LISTEN      1290/samba      
tcp6       0      0 fe80::a00:27ff:fe60::53 :::*                    LISTEN      1290/samba      
udp     1800      0 10.73.100.84:53         0.0.0.0:*                           1290/samba      
udp6       0      0 fe80::a00:27ff:fe60::53 :::*                                1290/samba      
root@dc2:/usr/local/samba/var# netstat -anp | head

And the backtrace is the following one

(gdb) bt
#0  0x00007f2ed59736f7 in ioctl () from /lib/libc.so.6
#1  0x00007f2ed2bf3895 in swrap_ioctl (s=26, r=21531, p=0x7fff8b073d14) at ../lib/socket_wrapper/socket_wrapper.c:1928
#2  0x00007f2ed3007d19 in tsocket_bsd_pending (fd=26) at ../lib/tsocket/tsocket_bsd.c:156
#3  0x00007f2ed3008f72 in tdgram_bsd_recvfrom_handler (private_data=0x26efe60) at ../lib/tsocket/tsocket_bsd.c:878
#4  0x00007f2ed3008904 in tdgram_bsd_fde_handler (ev=0x1a75e90, fde=0x2307c50, flags=1, private_data=0x28275e0) at ../lib/tsocket/tsocket_bsd.c:681
#5  0x00007f2ed821332b in epoll_event_loop (std_ev=0x25ddef0, tvalp=0x7fff8b073e60) at ../lib/tevent/tevent_standard.c:328
#6  0x00007f2ed8213a88 in std_event_loop_once (ev=0x1a75e90, location=0x7f2ecd420950 "../source4/smbd/process_standard.c:199") at ../lib/tevent/tevent_standard.c:567
#7  0x00007f2ed820e314 in _tevent_loop_once (ev=0x1a75e90, location=0x7f2ecd420950 "../source4/smbd/process_standard.c:199") at ../lib/tevent/tevent.c:507
#8  0x00007f2ed820e551 in tevent_common_loop_wait (ev=0x1a75e90, location=0x7f2ecd420950 "../source4/smbd/process_standard.c:199") at ../lib/tevent/tevent.c:608
#9  0x00007f2ed820e61c in _tevent_loop_wait (ev=0x1a75e90, location=0x7f2ecd420950 "../source4/smbd/process_standard.c:199") at ../lib/tevent/tevent.c:627
#10 0x00007f2ecd4205a1 in standard_new_task (ev=0x1a75e90, lp_ctx=0x1a61820, service_name=0x7f2ecc4cdabf "dns", new_task=0x7f2ed8a84800 <task_server_callback>, 
    private_data=0x1e982d0) at ../source4/smbd/process_standard.c:199
#11 0x00007f2ed8a849af in task_server_startup (event_ctx=0x1a75e90, lp_ctx=0x1a61820, service_name=0x7f2ecc4cdabf "dns", model_ops=0x7f2ecd620c20, 
    task_init=0x7f2ecc4c6958 <dns_task_init>) at ../source4/smbd/service_task.c:110
#12 0x00007f2ed8a82fae in server_service_init (name=0x1a66ce0 "dns", event_context=0x1a75e90, lp_ctx=0x1a61820, model_ops=0x7f2ecd620c20)
    at ../source4/smbd/service.c:63
#13 0x00007f2ed8a830ef in server_service_startup (event_ctx=0x1a75e90, lp_ctx=0x1a61820, model=0x40fc63 "standard", server_services=0x1a66e70)
    at ../source4/smbd/service.c:95
#14 0x000000000040bea7 in binary_smbd_main (binary_name=0x40facb "samba", argc=2, argv=0x7fff8b074448) at ../source4/smbd/server.c:477
#15 0x000000000040bf75 in main (argc=2, argv=0x7fff8b074448) at ../source4/smbd/server.c:497


(gdb) bt
#0  0x00007f13b9eb2f03 in epoll_wait () from /lib/libc.so.6
#1  0x00007f13bc74b11f in epoll_event_loop (std_ev=0x31e2e50, tvalp=0x7fff29608e60) at ../lib/tevent/tevent_standard.c:282
#2  0x00007f13bc74ba88 in std_event_loop_once (ev=0x25e8f00, location=0x7f13b1958950 "../source4/smbd/process_standard.c:199") at ../lib/tevent/tevent_standard.c:567
#3  0x00007f13bc746314 in _tevent_loop_once (ev=0x25e8f00, location=0x7f13b1958950 "../source4/smbd/process_standard.c:199") at ../lib/tevent/tevent.c:507
#4  0x00007f13bc746551 in tevent_common_loop_wait (ev=0x25e8f00, location=0x7f13b1958950 "../source4/smbd/process_standard.c:199") at ../lib/tevent/tevent.c:608
#5  0x00007f13bc74661c in _tevent_loop_wait (ev=0x25e8f00, location=0x7f13b1958950 "../source4/smbd/process_standard.c:199") at ../lib/tevent/tevent.c:627
#6  0x00007f13b19585a1 in standard_new_task (ev=0x25e8f00, lp_ctx=0x25d4820, service_name=0x7f13b0a05abf "dns", new_task=0x7f13bcfbc800 <task_server_callback>, 
    private_data=0x2cc5e50) at ../source4/smbd/process_standard.c:199
#7  0x00007f13bcfbc9af in task_server_startup (event_ctx=0x25e8f00, lp_ctx=0x25d4820, service_name=0x7f13b0a05abf "dns", model_ops=0x7f13b1b58c20, 
    task_init=0x7f13b09fe958 <dns_task_init>) at ../source4/smbd/service_task.c:110
#8  0x00007f13bcfbafae in server_service_init (name=0x25d9ce0 "dns", event_context=0x25e8f00, lp_ctx=0x25d4820, model_ops=0x7f13b1b58c20)
    at ../source4/smbd/service.c:63
#9  0x00007f13bcfbb0ef in server_service_startup (event_ctx=0x25e8f00, lp_ctx=0x25d4820, model=0x40fc63 "standard", server_services=0x25d9e70)
    at ../source4/smbd/service.c:95
#10 0x000000000040bea7 in binary_smbd_main (binary_name=0x40facb "samba", argc=2, argv=0x7fff29609448) at ../source4/smbd/server.c:477
#11 0x000000000040bf75 in main (argc=2, argv=0x7fff29609448) at ../source4/smbd/server.c:497
(gdb) c
Continuing.
^C
Program received signal SIGINT, Interrupt.
0x00007f13b9eb361a in getsockopt () from /lib/libc.so.6
(gdb) bt
#0  0x00007f13b9eb361a in getsockopt () from /lib/libc.so.6
#1  0x00007f13b712b76b in swrap_getsockopt (s=26, level=1, optname=4, optval=0x7fff29608d18, optlen=0x7fff29608d10) at ../lib/socket_wrapper/socket_wrapper.c:1885
#2  0x00007f13b753fd84 in tsocket_bsd_pending (fd=26) at ../lib/tsocket/tsocket_bsd.c:179
#3  0x00007f13b7540f72 in tdgram_bsd_recvfrom_handler (private_data=0x36d3cd0) at ../lib/tsocket/tsocket_bsd.c:878
#4  0x00007f13b7540904 in tdgram_bsd_fde_handler (ev=0x25e8f00, fde=0x284cc60, flags=1, private_data=0x28f2df0) at ../lib/tsocket/tsocket_bsd.c:681
#5  0x00007f13bc74b32b in epoll_event_loop (std_ev=0x31e2e50, tvalp=0x7fff29608e60) at ../lib/tevent/tevent_standard.c:328
#6  0x00007f13bc74ba88 in std_event_loop_once (ev=0x25e8f00, location=0x7f13b1958950 "../source4/smbd/process_standard.c:199") at ../lib/tevent/tevent_standard.c:567
#7  0x00007f13bc746314 in _tevent_loop_once (ev=0x25e8f00, location=0x7f13b1958950 "../source4/smbd/process_standard.c:199") at ../lib/tevent/tevent.c:507
#8  0x00007f13bc746551 in tevent_common_loop_wait (ev=0x25e8f00, location=0x7f13b1958950 "../source4/smbd/process_standard.c:199") at ../lib/tevent/tevent.c:608
#9  0x00007f13bc74661c in _tevent_loop_wait (ev=0x25e8f00, location=0x7f13b1958950 "../source4/smbd/process_standard.c:199") at ../lib/tevent/tevent.c:627
#10 0x00007f13b19585a1 in standard_new_task (ev=0x25e8f00, lp_ctx=0x25d4820, service_name=0x7f13b0a05abf "dns", new_task=0x7f13bcfbc800 <task_server_callback>, 
    private_data=0x2cc5e50) at ../source4/smbd/process_standard.c:199
#11 0x00007f13bcfbc9af in task_server_startup (event_ctx=0x25e8f00, lp_ctx=0x25d4820, service_name=0x7f13b0a05abf "dns", model_ops=0x7f13b1b58c20, 
    task_init=0x7f13b09fe958 <dns_task_init>) at ../source4/smbd/service_task.c:110
#12 0x00007f13bcfbafae in server_service_init (name=0x25d9ce0 "dns", event_context=0x25e8f00, lp_ctx=0x25d4820, model_ops=0x7f13b1b58c20)
    at ../source4/smbd/service.c:63
#13 0x00007f13bcfbb0ef in server_service_startup (event_ctx=0x25e8f00, lp_ctx=0x25d4820, model=0x40fc63 "standard", server_services=0x25d9e70)
    at ../source4/smbd/service.c:95
#14 0x000000000040bea7 in binary_smbd_main (binary_name=0x40facb "samba", argc=2, argv=0x7fff29609448) at ../source4/smbd/server.c:477
#15 0x000000000040bf75 in main (argc=2, argv=0x7fff29609448) at ../source4/smbd/server.c:497
Comment 1 Matthieu Patou 2012-09-20 16:28:07 UTC
Created attachment 7914 [details]
Tcpdump trace with first packet malformed

After the first malformed packet the server is blocked
Comment 2 Matthieu Patou 2012-09-20 16:46:40 UTC
Created attachment 7915 [details]
similar problem with trailing null bytes
Comment 3 Matthieu Patou 2012-09-22 08:55:49 UTC
Easy repro:
in scapy (apt-get install scapy)

>>> ip=IP(src="192.168.1.71", dst="192.168.1.81",len=28)
>>> udp=UDP(sport=12345,dport=53,len=8)
>>> send(ip/udp/"\x00\x00\x00\x00\x00")
.
Sent 1 packets.

Then in the server:nslookup <dcname>
Comment 4 Stefan Metzmacher 2012-09-22 23:21:41 UTC
Created attachment 7918 [details]
Patches for v4-0-test
Comment 5 Matthieu Patou 2012-09-22 23:26:24 UTC
Comment on attachment 7918 [details]
Patches for v4-0-test

Tested the patch on the server where we had the problem and since then I'm unable to reproduce
Comment 6 Stefan Metzmacher 2012-09-22 23:30:58 UTC
(In reply to comment #5)
> Comment on attachment 7918 [details]
> Patches for v4-0-test
> 
> Tested the patch on the server where we had the problem and since then I'm
> unable to reproduce

Then please set the review flag to '+' and assign the bug to
Karolin (ks@sernet.de)
Comment 7 Matthieu Patou 2012-09-23 00:18:18 UTC
(In reply to comment #6)
> (In reply to comment #5)
> > Comment on attachment 7918 [details] [details]
> > Patches for v4-0-test
> > 
> > Tested the patch on the server where we had the problem and since then I'm
> > unable to reproduce
> 
> Then please set the review flag to '+' and assign the bug to
> Karolin (ks@sernet.de)

For some reason I can't put the review flag to +.
Comment 8 Kai Blin 2012-09-23 19:43:48 UTC
Comment on attachment 7918 [details]
Patches for v4-0-test

Tested the reproducer before and after, fixes the problem for me. Thanks a lot.
Comment 9 Kai Blin 2012-09-23 19:48:59 UTC
Karolin, please pick for 4.0-test
Comment 10 Karolin Seeger 2012-09-28 07:15:05 UTC
Pushed to autobuild-v4-0-test.
Closing out bug report.

Thanks!
Comment 11 Stefan Metzmacher 2012-10-23 13:36:51 UTC
Created attachment 8105 [details]
Additional patch for v4-0-test to fix 100% CPU usage on freebsd
Comment 12 Karolin Seeger 2012-10-29 10:55:01 UTC
(In reply to comment #11)
> Created attachment 8105 [details]
> Additional patch for v4-0-test to fix 100% CPU usage on freebsd

Pushed to autobuild-v4-0-test.
Closing out bug report.

Thanks!