the nmbd process is locking up our server due to rewriting /var/lib/samba/unexpected.tdb # vmstat 1 10 procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 1 14 140 14248 465956 149828 0 0 4 3 14 0 2 3 9 86 0 14 140 14232 465956 149832 0 0 0 6236 486 182 0 2 0 98 0 14 140 14232 465956 149832 0 0 0 6352 481 180 0 0 0 100 # ps -el| grep get_re 1 D 0 576 2 0 75 -5 - 0 get_re ? 00:00:14 kjournald 1 D 0 1364 2 0 75 -5 - 0 get_re ? 00:19:43 kjournald 5 D 0 10575 1 0 80 0 - 25541 get_re ? 00:43:47 nmbd So the system I/O is locked down. killing nmbd and restart fixes temporary. This is a 3.0 GHz P4 without HT Compiled by the openSUSE team. It is on openSUSE 11.0
This is the second report of this. I'll try to fix it. Volker
Okay, got it reproduced on a test box. It might take a bit, but consider it fixed. Reproducable bugs are *always* fixable :-) Thanks for reporting it. Volker
Created attachment 3468 [details] proposed patch Can you try the attached patch? I've not yet checked it in upstream, it needs to be reviewed by Tridge and Jeremy first. tdb code is tricky :-) Volker
Just reviewed this - looks a very clever fix, thanks ! Just to confirm my understanding of the logic, on first pass over this record in a traverse, just mark it dead. On second time we see the record in a traverse, really delete it. If my understanding is correct, looks good to go for me - please commit to all branches. Jeremy.
This does not really fix the problem: after 2 days the file is -rw-r--r-- 1 root root 47M 15. Aug 09:55 unexpected.tdb and the system starts to be waiting on IO on this file.
So it did not change anything at all? Not even the speed of growth? Volker
It looks so! I'm watching it with: while ( true ); do (date;ls -l /var/locks/unexpected.tdb;vmstat 1 2) >> nmbd-porblem-fixed.log ;sleep 600;done and can align the sizes nearly perfectly!
Weird. The patch did fix the problem which I could easily reproduce. Just double-checking: You are certain that you're running the patched nmbd? If you confirm this, we need to try different approaches. Volker
I'm really sure, as I changed the pathes for the tdb file by mistake. Now I have an old and a new tdb file I only changed the nmbd, no other files or libs.
Ah, the fix is in libtdb.so. So you need to exchange that as well... Cheers - Michael
o.k. an LD_LIBRARY_PATH helped. Now lsof shows: nmbd 8201 root txt REG 8,1 2203005 3341277 /usr/src/packages/SOURCES/samba-3.2.0rc1/source/bin/nmbd nmbd 8201 root mem REG 8,1 9936 1190206 /usr/lib/gconv/UTF-16.so nmbd 8201 root mem REG 8,1 1410320 2897547 /lib/libc-2.8.so nmbd 8201 root mem REG 8,1 56267 3341275 /usr/src/packages/SOURCES/samba-3.2.0rc1/source/bin/libtdb.so nmbd 8201 root mem REG 8,1 37196 3341271 /usr/src/packages/SOURCES/samba-3.2.0rc1/source/bin/libtalloc.so nmbd 8201 root mem REG 8,1 11060 2897553 /lib/libdl-2.8.so nmbd 8201 root mem REG 8,1 83232 2901678 /lib/libnsl-2.8.so nmbd 8201 root mem REG 8,1 69304 2897638 /lib/libresolv-2.8.so nmbd 8201 root mem REG 8,1 56276 2897540 /lib/libcrypt-2.8.so nmbd 8201 root mem REG 8,1 9924 1190334 /usr/lib/gconv/IBM850.so So I'm watching again.
If possible, can you give feedback pretty soon, because if the patch really does not help I need to find other workarounds over the weekend to keep up the monday 3.2.2 release? You should see growth or non-growth within a few hours I think. Even if the growth would be considerably reduced I would be happy... Volker
o.k. It looks better!!! [2008/08/13 11:53:56, 0] nmbd/nmbd.c:main(849) nmbd version 3.2.0rc1 started. and [2008/08/15 10:41:12, 0] nmbd/nmbd.c:main(849) nmbd version 3.2.0rc1 started. sdiff old new 225280 13. Aug 11:59 /var/locks/unexpected.tdb | 225280 15. Aug 12:01 /var/locks/unexpected.tdb 561152 13. Aug 12:09 /var/locks/unexpected.tdb | 225280 15. Aug 12:11 /var/locks/unexpected.tdb 704512 13. Aug 12:19 /var/locks/unexpected.tdb | 225280 15. Aug 12:21 /var/locks/unexpected.tdb 880640 13. Aug 12:29 /var/locks/unexpected.tdb | 225280 15. Aug 12:31 /var/locks/unexpected.tdb 1380352 13. Aug 12:39 /var/locks/unexpected.tdb | 225280 15. Aug 12:41 /var/locks/unexpected.tdb 1380352 13. Aug 12:49 /var/locks/unexpected.tdb <
Phew... Thanks for the feedback! :-) Volker
o.k. I watch out till Monday... Is it enough to replace the libtdb.so with the fixed version? As I have the suse-tuned samba here, and do not want to setup again in their way
If you got the original source-rpms, you applied the patch in the BUILD subdirectory (or wherever the SuSE-patched source ended up for you), changing libtdb.so is sufficient. Volker
*** Bug 5643 has been marked as a duplicate of this bug. ***
O.K. it IS stable: 2:-rw-r--r-- 1 root root 28672 15. Aug 10:51 /var/locks/unexpected.tdb 8:-rw-r--r-- 1 root root 180224 15. Aug 11:01 /var/locks/unexpected.tdb 14:-rw-r--r-- 1 root root 180224 15. Aug 11:11 /var/locks/unexpected.tdb 20:-rw-r--r-- 1 root root 180224 15. Aug 11:21 /var/locks/unexpected.tdb 26:-rw-r--r-- 1 root root 225280 15. Aug 11:31 /var/locks/unexpected.tdb ... 2522:-rw-r--r-- 1 root root 225280 18. Aug 08:58 /var/locks/unexpected.tdb 2528:-rw-r--r-- 1 root root 225280 18. Aug 09:08 /var/locks/unexpected.tdb 2534:-rw-r--r-- 1 root root 225280 18. Aug 09:18 /var/locks/unexpected.tdb