Bug 5684 - /var/lib/samba/unexpected.tdb grows infinite
Summary: /var/lib/samba/unexpected.tdb grows infinite
Status: RESOLVED FIXED
Alias: None
Product: Samba 3.2
Classification: Unclassified
Component: Nmbd (show other bugs)
Version: 3.2.0
Hardware: x86 Linux
: P3 normal
Target Milestone: ---
Assignee: Samba Bugzilla Account
QA Contact: Samba QA Contact
URL:
Keywords:
: 5643 (view as bug list)
Depends on:
Blocks:
 
Reported: 2008-08-12 07:35 UTC by Joachim Reichelt
Modified: 2008-08-18 02:32 UTC (History)
2 users (show)

See Also:


Attachments
proposed patch (1.22 KB, patch)
2008-08-12 15:36 UTC, Volker Lendecke
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Joachim Reichelt 2008-08-12 07:35:29 UTC
the nmbd process is locking up our server due to rewriting /var/lib/samba/unexpected.tdb

 # vmstat 1 10
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 1 14    140  14248 465956 149828    0    0     4     3   14    0  2  3  9 86
 0 14    140  14232 465956 149832    0    0     0  6236  486  182  0  2  0 98
 0 14    140  14232 465956 149832    0    0     0  6352  481  180  0  0  0 100 

# ps -el| grep get_re
1 D     0   576     2  0  75  -5 -     0 get_re ?        00:00:14 kjournald
1 D     0  1364     2  0  75  -5 -     0 get_re ?        00:19:43 kjournald
5 D     0 10575     1  0  80   0 - 25541 get_re ?        00:43:47 nmbd 

So the system I/O is locked down.
killing nmbd and restart fixes temporary.
This is a 3.0 GHz P4 without HT

Compiled by the openSUSE team.
It is on openSUSE 11.0
Comment 1 Volker Lendecke 2008-08-12 09:42:19 UTC
This is the second report of this. I'll try to fix it.

Volker
Comment 2 Volker Lendecke 2008-08-12 09:51:05 UTC
Okay, got it reproduced on a test box. It might take a bit, but consider it fixed. Reproducable bugs are *always* fixable :-)

Thanks for reporting it.

Volker
Comment 3 Volker Lendecke 2008-08-12 15:36:50 UTC
Created attachment 3468 [details]
proposed patch

Can you try the attached patch?

I've not yet checked it in upstream, it needs to be reviewed by Tridge and Jeremy first. tdb code is tricky :-)

Volker
Comment 4 Jeremy Allison 2008-08-12 18:44:00 UTC
Just reviewed this - looks a very clever fix, thanks ! Just to confirm my understanding of the logic, on first pass over this record in a traverse, just mark it dead. On second time we see the record in a traverse, really delete it.
If my understanding is correct, looks good to go for me - please commit to all branches.
Jeremy.
Comment 5 Joachim Reichelt 2008-08-15 02:57:20 UTC
This does not really fix the problem:
after 2 days the file is 
-rw-r--r--  1 root root  47M 15. Aug 09:55 unexpected.tdb
and the system starts to be waiting on IO on this file.
Comment 6 Volker Lendecke 2008-08-15 03:02:53 UTC
So it did not change anything at all? Not even the speed of growth?

Volker
Comment 7 Joachim Reichelt 2008-08-15 03:06:24 UTC
It looks so!
I'm watching it with:
while ( true ); do (date;ls -l /var/locks/unexpected.tdb;vmstat 1 2) >> nmbd-porblem-fixed.log ;sleep 600;done

and can align the sizes nearly perfectly!
Comment 8 Volker Lendecke 2008-08-15 03:29:52 UTC
Weird. The patch did fix the problem which I could easily reproduce. Just double-checking: You are certain that you're running the patched nmbd?

If you confirm this, we need to try different approaches.

Volker
Comment 9 Joachim Reichelt 2008-08-15 03:32:05 UTC
I'm really sure, as I changed the pathes for the tdb file by mistake. Now I have an old and a new tdb file
I only changed the nmbd, no other files or libs.
Comment 10 Michael Adam 2008-08-15 03:34:39 UTC
Ah, the fix is in libtdb.so. So you need to exchange that as well...

Cheers - Michael
Comment 11 Joachim Reichelt 2008-08-15 03:53:47 UTC
o.k.
an LD_LIBRARY_PATH helped. Now lsof shows:

nmbd    8201 root  txt    REG     8,1  2203005 3341277 /usr/src/packages/SOURCES/samba-3.2.0rc1/source/bin/nmbd
nmbd    8201 root  mem    REG     8,1     9936 1190206 /usr/lib/gconv/UTF-16.so
nmbd    8201 root  mem    REG     8,1  1410320 2897547 /lib/libc-2.8.so
nmbd    8201 root  mem    REG     8,1    56267 3341275 /usr/src/packages/SOURCES/samba-3.2.0rc1/source/bin/libtdb.so
nmbd    8201 root  mem    REG     8,1    37196 3341271 /usr/src/packages/SOURCES/samba-3.2.0rc1/source/bin/libtalloc.so
nmbd    8201 root  mem    REG     8,1    11060 2897553 /lib/libdl-2.8.so
nmbd    8201 root  mem    REG     8,1    83232 2901678 /lib/libnsl-2.8.so
nmbd    8201 root  mem    REG     8,1    69304 2897638 /lib/libresolv-2.8.so
nmbd    8201 root  mem    REG     8,1    56276 2897540 /lib/libcrypt-2.8.so
nmbd    8201 root  mem    REG     8,1     9924 1190334 /usr/lib/gconv/IBM850.so

So I'm watching again.
Comment 12 Volker Lendecke 2008-08-15 04:02:52 UTC
If possible, can you give feedback pretty soon, because if the patch really does not help I need to find other workarounds over the weekend to keep up the monday 3.2.2 release? You should see growth or non-growth within a few hours I think.

Even if the growth would be considerably reduced I would be happy...

Volker
Comment 13 Joachim Reichelt 2008-08-15 05:52:26 UTC
o.k.
It looks better!!!


[2008/08/13 11:53:56,  0] nmbd/nmbd.c:main(849)
  nmbd version 3.2.0rc1 started.

and

[2008/08/15 10:41:12,  0] nmbd/nmbd.c:main(849)
  nmbd version 3.2.0rc1 started.


sdiff old new
225280 13. Aug 11:59 /var/locks/unexpected.tdb                | 225280 15. Aug 12:01 /var/locks/unexpected.tdb
561152 13. Aug 12:09 /var/locks/unexpected.tdb                | 225280 15. Aug 12:11 /var/locks/unexpected.tdb
704512 13. Aug 12:19 /var/locks/unexpected.tdb                | 225280 15. Aug 12:21 /var/locks/unexpected.tdb
880640 13. Aug 12:29 /var/locks/unexpected.tdb                | 225280 15. Aug 12:31 /var/locks/unexpected.tdb
1380352 13. Aug 12:39 /var/locks/unexpected.tdb               | 225280 15. Aug 12:41 /var/locks/unexpected.tdb
1380352 13. Aug 12:49 /var/locks/unexpected.tdb               <
Comment 14 Volker Lendecke 2008-08-15 05:54:56 UTC
Phew... Thanks for the feedback! :-)

Volker
Comment 15 Joachim Reichelt 2008-08-15 06:00:00 UTC
o.k.
I watch out till Monday...

Is it enough to replace the libtdb.so with the fixed version?
As I have the suse-tuned samba here, and do not want to setup again in their way
Comment 16 Volker Lendecke 2008-08-15 06:03:58 UTC
If you got the original source-rpms, you applied the patch in the BUILD subdirectory (or wherever the SuSE-patched source ended up for you), changing libtdb.so is sufficient.

Volker
Comment 17 Volker Lendecke 2008-08-17 03:38:06 UTC
*** Bug 5643 has been marked as a duplicate of this bug. ***
Comment 18 Joachim Reichelt 2008-08-18 02:32:55 UTC
O.K. it IS stable:
2:-rw-r--r-- 1 root root 28672 15. Aug 10:51 /var/locks/unexpected.tdb
8:-rw-r--r-- 1 root root 180224 15. Aug 11:01 /var/locks/unexpected.tdb
14:-rw-r--r-- 1 root root 180224 15. Aug 11:11 /var/locks/unexpected.tdb
20:-rw-r--r-- 1 root root 180224 15. Aug 11:21 /var/locks/unexpected.tdb
26:-rw-r--r-- 1 root root 225280 15. Aug 11:31 /var/locks/unexpected.tdb
...
2522:-rw-r--r-- 1 root root 225280 18. Aug 08:58 /var/locks/unexpected.tdb
2528:-rw-r--r-- 1 root root 225280 18. Aug 09:08 /var/locks/unexpected.tdb
2534:-rw-r--r-- 1 root root 225280 18. Aug 09:18 /var/locks/unexpected.tdb