Bug 11985 - ctdb memory leak version 2.5.3 , 2.5.6 and 4.4.4
ctdb memory leak version 2.5.3 , 2.5.6 and 4.4.4
Status: ASSIGNED
Product: CTDB 2.5.x or older
Classification: Unclassified
Component: ctdb
4.4.x
All All
: P3 critical
: ---
Assigned To: Amitay Isaacs
Samba QA Contact
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2016-06-21 10:11 UTC by James
Modified: 2016-07-13 16:41 UTC (History)
2 users (show)

See Also:


Attachments
dumpmemory ,rddumpmemory (96.42 KB, text/plain)
2016-06-21 11:10 UTC, James
no flags Details
smaps (18.20 KB, text/plain)
2016-06-29 13:21 UTC, James
no flags Details
dumpmemory Wed Jun 29 21:18:51 (45.66 KB, text/plain)
2016-06-29 13:22 UTC, James
no flags Details
ps ctdb per second (296.69 KB, application/x-rar-compressed)
2016-06-29 13:23 UTC, James
no flags Details
ctdb dumpmemory Thu Jun 30 09:18:34 CST 2016 (45.66 KB, text/plain)
2016-06-30 01:22 UTC, James
no flags Details
cat /proc/16798/smaps Thu Jun 30 09:23:45 CST 2016 (18.76 KB, text/plain)
2016-06-30 01:27 UTC, James
no flags Details
ctdb info from 07-04-1444 to 07-05-0926 (10.46 KB, application/x-rar-compressed)
2016-07-05 02:30 UTC, James
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description James 2016-06-21 10:11:06 UTC
Hi all
  Ctdb was found to have a persistent memory leak, with the increase in memory usage per 4K size;
In order to simplify the problem analysis, close the samba service, or continue to leak;
Use ctdb  dumpmemory rddumpmemory  analysis, there is no memory leak found the structure
Related information is as follows:
[root@vx1600-cjf35 ~]# ctdb status
Number of nodes:5
pnn:0 192.168.77.91    OK
pnn:1 192.168.77.50    OK
pnn:2 192.168.77.35    OK (THIS NODE)
pnn:3 192.168.77.26    OK
pnn:4 192.168.77.60    OK
Generation:1795168275
Size:5
hash:0 lmaster:0
hash:1 lmaster:1
hash:2 lmaster:2
hash:3 lmaster:3
hash:4 lmaster:4
Recovery mode:NORMAL (0)
Recovery master:0
[root@vx1600-cjf35 ~]# ctdb ip
Public IPs on node 2
173.16.10.1 0
173.16.10.2 3
173.16.10.3 1
173.16.10.4 3
173.16.10.5 2
173.16.10.6 4
173.16.10.7 0
173.16.10.8 1
173.16.10.9 0
173.16.10.10 2
Memory leak,continue 4k,
root     19817  0.3  0.1  33468 21184 ?        SLs  Jun17   6:38 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
root     20178  0.0  0.0  22084  5016 ?        S    Jun17   0:31 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
Sat Jun 18 23:40:42 CST 2016
root     19817  0.3  0.1  33468 21184 ?        SLs  Jun17   6:38 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
root     20178  0.0  0.0  22084  5016 ?        S    Jun17   0:31 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
Sat Jun 18 23:40:43 CST 2016
from 21184 increase to 21188 ,4K increase
root     19817  0.3  0.1  33468 21188 ?        SLs  Jun17   6:38 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
root     20178  0.0  0.0  22084  5016 ?        S    Jun17   0:31 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
Sat Jun 18 23:40:44 CST 2016
root     19817  0.3  0.1  33468 21188 ?        SLs  Jun17   6:38 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
root     20178  0.0  0.0  22084  5016 ?        S    Jun17   0:31 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
Sat Jun 18 23:40:45 CST 2016
root     19817  0.3  0.1  33468 21188 ?        SLs  Jun17   6:38 /usr/sbin/ctdbd --rec
persistent memory leak,Leaked 4M memory every day
Sun Jun 19 02:18:23 CST 2016
root     19817  0.3  0.1  34008 21760 ?        SLs  Jun17   7:12 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
root     20178  0.0  0.0  22084  5016 ?        S    Jun17   0:34 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
Sun Jun 19 02:18:24 CST 2016
root     19817  0.3  0.1  34008 21764 ?        SLs  Jun17   7:12 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
root     20178  0.0  0.0  22084  5016 ?        S    Jun17   0:34 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
Sun Jun 19 02:18:26 CST 2016

Best Regards
James
Comment 1 James 2016-06-21 11:10:48 UTC
Created attachment 12193 [details]
dumpmemory ,rddumpmemory

Running a few months, ctdb memory increased to 700M
Comment 2 Amitay Isaacs 2016-06-21 15:43:31 UTC
(In reply to James from comment #1)

Can you provide the output of following commands:

  cat /proc/19817/maps
  cat /proc/19817/smaps
  ls -l /proc/19817/fd

assuming 19817 is the ctdb daemon pid or use "ctdb getpid".

Amitay.
Comment 3 James 2016-06-22 01:55:58 UTC
The problem can be reproduced:
191.168.17.91 ctdb info:
[root@vx1600-cjf91 ~]# ctdb getpid
Pid:25775
[root@vx1600-cjf91 ~]# ps aux | grep ctdb
root     25775  0.1  0.1  29768 17396 ?        SLs  Jun21   1:19 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
root     26050  0.0  0.0  22080  5028 ?        S    Jun21   0:16 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
root     26997  0.0  0.0   7960   892 pts/0    S+   09:48   0:00 grep ctdb
[root@vx1600-cjf91 ~]# 
00400000-004c5000 r-xp 00000000 08:02 3369                               /usr/sbin/ctdbd
006c5000-006c7000 rw-p 000c5000 08:02 3369                               /usr/sbin/ctdbd
00ef0000-00f11000 rw-p 00000000 00:00 0 
00f11000-00f61000 rw-p 00000000 00:00 0 
00f61000-01260000 rw-p 00000000 00:00 0 
3547400000-3547420000 r-xp 00000000 08:02 11428                          /lib64/ld-2.12.so
354761f000-3547620000 r--p 0001f000 08:02 11428                          /lib64/ld-2.12.so
3547620000-3547621000 rw-p 00020000 08:02 11428                          /lib64/ld-2.12.so
3547621000-3547622000 rw-p 00000000 00:00 0 
7eff2f51b000-7eff2f65b000 rw-s 00000000 08:02 14406                      /var/lib/ctdb/printer_list.tdb.0
7eff2f65b000-7eff2f781000 rw-s 00000000 08:02 14404                      /var/lib/ctdb/brlock.tdb.0
7eff2f781000-7eff2f8a7000 rw-s 00000000 08:02 14403                      /var/lib/ctdb/connections.tdb.0
7eff2f8a7000-7eff2f9cd000 rw-s 00000000 08:02 14401                      /var/lib/ctdb/g_lock.tdb.0
7eff2f9cd000-7eff2faf3000 rw-s 00000000 08:02 14400                      /var/lib/ctdb/serverid.tdb.0
7eff2faf3000-7eff2fc19000 rw-s 00000000 08:02 14402                      /var/lib/ctdb/sessionid.tdb.0
7eff2fc19000-7eff2fd3f000 rw-s 00000000 08:02 14405                      /var/lib/ctdb/locking.tdb.0
7eff2fd3f000-7eff2fe7f000 rw-s 00000000 08:02 14393                      /var/lib/ctdb/persistent/ctdb.tdb.0
7eff2fe7f000-7eff2ffbf000 rw-s 00000000 08:02 14394                      /var/lib/ctdb/persistent/group_mapping.tdb.0
7eff2ffbf000-7eff300ff000 rw-s 00000000 08:02 14399                      /var/lib/ctdb/persistent/passdb.tdb.0
7eff300ff000-7eff3023f000 rw-s 00000000 08:02 14397                      /var/lib/ctdb/persistent/account_policy.tdb.0
7eff3023f000-7eff3037f000 rw-s 00000000 08:02 14395                      /var/lib/ctdb/persistent/registry.tdb.0
7eff3037f000-7eff304bf000 rw-s 00000000 08:02 14398                      /var/lib/ctdb/persistent/share_info.tdb.0
7eff304bf000-7eff305ff000 rw-s 00000000 08:02 14396                      /var/lib/ctdb/persistent/secrets.tdb.0
7eff305ff000-7eff3060b000 r-xp 00000000 08:02 124                        /lib64/libnss_files-2.12.so
7eff3060b000-7eff3080b000 ---p 0000c000 08:02 124                        /lib64/libnss_files-2.12.so
7eff3080b000-7eff3080c000 r--p 0000c000 08:02 124                        /lib64/libnss_files-2.12.so
7eff3080c000-7eff3080d000 rw-p 0000d000 08:02 124                        /lib64/libnss_files-2.12.so
7eff3080d000-7eff30810000 rw-p 00000000 00:00 0 
7eff30810000-7eff309a7000 r-xp 00000000 08:02 86                         /lib64/libc-2.12.so
7eff309a7000-7eff30ba7000 ---p 00197000 08:02 86                         /lib64/libc-2.12.so
7eff30ba7000-7eff30bab000 r--p 00197000 08:02 86                         /lib64/libc-2.12.so
7eff30bab000-7eff30bac000 rw-p 0019b000 08:02 86                         /lib64/libc-2.12.so
7eff30bac000-7eff30bb1000 rw-p 00000000 00:00 0 
7eff30bb1000-7eff30bb9000 r-xp 00000000 08:02 29                         /lib64/libpopt.so.0.0.0
7eff30bb9000-7eff30db9000 ---p 00008000 08:02 29                         /lib64/libpopt.so.0.0.0
7eff30db9000-7eff30dba000 rw-p 00008000 08:02 29                         /lib64/libpopt.so.0.0.0
7eff30dbc000-7eff30dbd000 rw-s 00000000 08:02 13887                      /var/lib/ctdb/state/persistent_health.tdb.0
7eff30dbd000-7eff30dbe000 rw-p 00000000 00:00 0 
7fff32703000-7fff32718000 rw-p 00000000 00:00 0                          [stack]
7fff327ff000-7fff32800000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
00400000-004c5000 r-xp 00000000 08:02 3369                               /usr/sbin/ctdbd
Size:                788 kB
Rss:                 788 kB
Pss:                 636 kB
Shared_Clean:        304 kB
Shared_Dirty:          0 kB
Private_Clean:       484 kB
Private_Dirty:         0 kB
Referenced:          788 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
006c5000-006c7000 rw-p 000c5000 08:02 3369                               /usr/sbin/ctdbd
Size:                  8 kB
Rss:                   8 kB
Pss:                   8 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         8 kB
Referenced:            8 kB
Anonymous:             8 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
00ef0000-00f11000 rw-p 00000000 00:00 0 
Size:                132 kB
Rss:                 132 kB
Pss:                 122 kB
Shared_Clean:          0 kB
Shared_Dirty:         20 kB
Private_Clean:         0 kB
Private_Dirty:       112 kB
Referenced:          132 kB
Anonymous:           132 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
00f11000-00f61000 rw-p 00000000 00:00 0 
Size:                320 kB
Rss:                 320 kB
Pss:                 312 kB
Shared_Clean:          0 kB
Shared_Dirty:         16 kB
Private_Clean:         0 kB
Private_Dirty:       304 kB
Referenced:          320 kB
Anonymous:           320 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
00f61000-01260000 rw-p 00000000 00:00 0 
Size:               3068 kB
Rss:                3012 kB
Pss:                3012 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:      3012 kB
Referenced:         3012 kB
Anonymous:          3012 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
3547400000-3547420000 r-xp 00000000 08:02 11428                          /lib64/ld-2.12.so
Size:                128 kB
Rss:                 128 kB
Pss:                  14 kB
Shared_Clean:        116 kB
Shared_Dirty:          0 kB
Private_Clean:        12 kB
Private_Dirty:         0 kB
Referenced:          128 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
354761f000-3547620000 r--p 0001f000 08:02 11428                          /lib64/ld-2.12.so
Size:                  4 kB
Rss:                   4 kB
Pss:                   2 kB
Shared_Clean:          0 kB
Shared_Dirty:          4 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            4 kB
Anonymous:             4 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
3547620000-3547621000 rw-p 00020000 08:02 11428                          /lib64/ld-2.12.so
Size:                  4 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         4 kB
Referenced:            4 kB
Anonymous:             4 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
3547621000-3547622000 rw-p 00000000 00:00 0 
Size:                  4 kB
Rss:                   4 kB
Pss:                   2 kB
Shared_Clean:          0 kB
Shared_Dirty:          4 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            4 kB
Anonymous:             4 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff2f51b000-7eff2f65b000 rw-s 00000000 08:02 14406                      /var/lib/ctdb/printer_list.tdb.0
Size:               1280 kB
Rss:                 796 kB
Pss:                 796 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:       796 kB
Private_Dirty:         0 kB
Referenced:          796 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff2f65b000-7eff2f781000 rw-s 00000000 08:02 14404                      /var/lib/ctdb/brlock.tdb.0
Size:               1176 kB
Rss:                 784 kB
Pss:                 784 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:       784 kB
Private_Dirty:         0 kB
Referenced:          784 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff2f781000-7eff2f8a7000 rw-s 00000000 08:02 14403                      /var/lib/ctdb/connections.tdb.0
Size:               1176 kB
Rss:                 784 kB
Pss:                 784 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:       784 kB
Private_Dirty:         0 kB
Referenced:          784 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff2f8a7000-7eff2f9cd000 rw-s 00000000 08:02 14401                      /var/lib/ctdb/g_lock.tdb.0
Size:               1176 kB
Rss:                 784 kB
Pss:                 784 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:       784 kB
Private_Dirty:         0 kB
Referenced:          784 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff2f9cd000-7eff2faf3000 rw-s 00000000 08:02 14400                      /var/lib/ctdb/serverid.tdb.0
Size:               1176 kB
Rss:                 784 kB
Pss:                 784 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:       784 kB
Private_Dirty:         0 kB
Referenced:          784 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff2faf3000-7eff2fc19000 rw-s 00000000 08:02 14402                      /var/lib/ctdb/sessionid.tdb.0
Size:               1176 kB
Rss:                 784 kB
Pss:                 784 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:       784 kB
Private_Dirty:         0 kB
Referenced:          784 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff2fc19000-7eff2fd3f000 rw-s 00000000 08:02 14405                      /var/lib/ctdb/locking.tdb.0
Size:               1176 kB
Rss:                 784 kB
Pss:                 784 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:       784 kB
Private_Dirty:         0 kB
Referenced:          784 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff2fd3f000-7eff2fe7f000 rw-s 00000000 08:02 14393                      /var/lib/ctdb/persistent/ctdb.tdb.0
Size:               1280 kB
Rss:                 796 kB
Pss:                 796 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:       796 kB
Private_Dirty:         0 kB
Referenced:          796 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff2fe7f000-7eff2ffbf000 rw-s 00000000 08:02 14394                      /var/lib/ctdb/persistent/group_mapping.tdb.0
Size:               1280 kB
Rss:                 796 kB
Pss:                 796 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:       796 kB
Private_Dirty:         0 kB
Referenced:          796 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff2ffbf000-7eff300ff000 rw-s 00000000 08:02 14399                      /var/lib/ctdb/persistent/passdb.tdb.0
Size:               1280 kB
Rss:                 796 kB
Pss:                 796 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:       796 kB
Private_Dirty:         0 kB
Referenced:          796 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff300ff000-7eff3023f000 rw-s 00000000 08:02 14397                      /var/lib/ctdb/persistent/account_policy.tdb.0
Size:               1280 kB
Rss:                 796 kB
Pss:                 796 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:       796 kB
Private_Dirty:         0 kB
Referenced:          796 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff3023f000-7eff3037f000 rw-s 00000000 08:02 14395                      /var/lib/ctdb/persistent/registry.tdb.0
Size:               1280 kB
Rss:                 836 kB
Pss:                 836 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:       836 kB
Private_Dirty:         0 kB
Referenced:          836 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff3037f000-7eff304bf000 rw-s 00000000 08:02 14398                      /var/lib/ctdb/persistent/share_info.tdb.0
Size:               1280 kB
Rss:                 796 kB
Pss:                 796 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:       796 kB
Private_Dirty:         0 kB
Referenced:          796 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff304bf000-7eff305ff000 rw-s 00000000 08:02 14396                      /var/lib/ctdb/persistent/secrets.tdb.0
Size:               1280 kB
Rss:                 796 kB
Pss:                 796 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:       796 kB
Private_Dirty:         0 kB
Referenced:          796 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff305ff000-7eff3060b000 r-xp 00000000 08:02 124                        /lib64/libnss_files-2.12.so
Size:                 48 kB
Rss:                  48 kB
Pss:                  11 kB
Shared_Clean:         40 kB
Shared_Dirty:          0 kB
Private_Clean:         8 kB
Private_Dirty:         0 kB
Referenced:           48 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff3060b000-7eff3080b000 ---p 0000c000 08:02 124                        /lib64/libnss_files-2.12.so
Size:               2048 kB
Rss:                   0 kB
Pss:                   0 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            0 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff3080b000-7eff3080c000 r--p 0000c000 08:02 124                        /lib64/libnss_files-2.12.so
Size:                  4 kB
Rss:                   4 kB
Pss:                   2 kB
Shared_Clean:          0 kB
Shared_Dirty:          4 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            4 kB
Anonymous:             4 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff3080c000-7eff3080d000 rw-p 0000d000 08:02 124                        /lib64/libnss_files-2.12.so
Size:                  4 kB
Rss:                   4 kB
Pss:                   2 kB
Shared_Clean:          0 kB
Shared_Dirty:          4 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            4 kB
Anonymous:             4 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff3080d000-7eff30810000 rw-p 00000000 00:00 0 
Size:                 12 kB
Rss:                  12 kB
Pss:                   8 kB
Shared_Clean:          0 kB
Shared_Dirty:          8 kB
Private_Clean:         0 kB
Private_Dirty:         4 kB
Referenced:           12 kB
Anonymous:            12 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff30810000-7eff309a7000 r-xp 00000000 08:02 86                         /lib64/libc-2.12.so
Size:               1628 kB
Rss:                1628 kB
Pss:                 598 kB
Shared_Clean:       1096 kB
Shared_Dirty:          0 kB
Private_Clean:       532 kB
Private_Dirty:         0 kB
Referenced:         1628 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff309a7000-7eff30ba7000 ---p 00197000 08:02 86                         /lib64/libc-2.12.so
Size:               2048 kB
Rss:                   0 kB
Pss:                   0 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            0 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff30ba7000-7eff30bab000 r--p 00197000 08:02 86                         /lib64/libc-2.12.so
Size:                 16 kB
Rss:                  16 kB
Pss:                   8 kB
Shared_Clean:          0 kB
Shared_Dirty:         16 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:           16 kB
Anonymous:            16 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff30bab000-7eff30bac000 rw-p 0019b000 08:02 86                         /lib64/libc-2.12.so
Size:                  4 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         4 kB
Referenced:            4 kB
Anonymous:             4 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff30bac000-7eff30bb1000 rw-p 00000000 00:00 0 
Size:                 20 kB
Rss:                  20 kB
Pss:                  18 kB
Shared_Clean:          0 kB
Shared_Dirty:          4 kB
Private_Clean:         0 kB
Private_Dirty:        16 kB
Referenced:           20 kB
Anonymous:            20 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff30bb1000-7eff30bb9000 r-xp 00000000 08:02 29                         /lib64/libpopt.so.0.0.0
Size:                 32 kB
Rss:                  32 kB
Pss:                  29 kB
Shared_Clean:          4 kB
Shared_Dirty:          0 kB
Private_Clean:        28 kB
Private_Dirty:         0 kB
Referenced:           32 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff30bb9000-7eff30db9000 ---p 00008000 08:02 29                         /lib64/libpopt.so.0.0.0
Size:               2048 kB
Rss:                   0 kB
Pss:                   0 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            0 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff30db9000-7eff30dba000 rw-p 00008000 08:02 29                         /lib64/libpopt.so.0.0.0
Size:                  4 kB
Rss:                   4 kB
Pss:                   2 kB
Shared_Clean:          0 kB
Shared_Dirty:          4 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            4 kB
Anonymous:             4 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff30dbc000-7eff30dbd000 rw-s 00000000 08:02 13887                      /var/lib/ctdb/state/persistent_health.tdb.0
Size:                  4 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         4 kB
Private_Dirty:         0 kB
Referenced:            4 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff30dbd000-7eff30dbe000 rw-p 00000000 00:00 0 
Size:                  4 kB
Rss:                   4 kB
Pss:                   2 kB
Shared_Clean:          0 kB
Shared_Dirty:          4 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            4 kB
Anonymous:             4 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7fff32703000-7fff32718000 rw-p 00000000 00:00 0                          [stack]
Size:                 88 kB
Rss:                  84 kB
Pss:                  50 kB
Shared_Clean:          0 kB
Shared_Dirty:         68 kB
Private_Clean:         0 kB
Private_Dirty:        16 kB
Referenced:           84 kB
Anonymous:            84 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7fff327ff000-7fff32800000 r-xp 00000000 00:00 0                          [vdso]
Size:                  4 kB
Rss:                   4 kB
Pss:                   0 kB
Shared_Clean:          4 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            4 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Size:                  4 kB
Rss:                   0 kB
Pss:                   0 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            0 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
total 0
lr-x------. 1 root root 64 Jun 22 09:43 0 -> /dev/null
l-wx------. 1 root root 64 Jun 22 09:43 1 -> pipe:[1580347174]
lrwx------. 1 root root 64 Jun 22 09:43 10 -> /var/lib/ctdb/state/persistent_health.tdb.0
lrwx------. 1 root root 64 Jun 22 09:43 11 -> socket:[1580385812]
lrwx------. 1 root root 64 Jun 22 09:43 12 -> socket:[1597056131]
lrwx------. 1 root root 64 Jun 22 09:43 13 -> /var/lib/ctdb/persistent/secrets.tdb.0
lrwx------. 1 root root 64 Jun 22 09:43 14 -> /var/lib/ctdb/persistent/share_info.tdb.0
lrwx------. 1 root root 64 Jun 22 09:43 15 -> /var/lib/ctdb/persistent/registry.tdb.0
lrwx------. 1 root root 64 Jun 22 09:43 16 -> /var/lib/ctdb/persistent/account_policy.tdb.0
lrwx------. 1 root root 64 Jun 22 09:43 17 -> /var/lib/ctdb/persistent/passdb.tdb.0
lrwx------. 1 root root 64 Jun 22 09:43 18 -> /var/lib/ctdb/persistent/group_mapping.tdb.0
lrwx------. 1 root root 64 Jun 22 09:43 19 -> /var/lib/ctdb/persistent/ctdb.tdb.0
l-wx------. 1 root root 64 Jun 22 09:43 2 -> pipe:[1580347174]
lrwx------. 1 root root 64 Jun 22 09:43 20 -> socket:[1580400159]
lrwx------. 1 root root 64 Jun 22 09:43 21 -> socket:[1597091707]
lrwx------. 1 root root 64 Jun 22 09:43 23 -> socket:[1580450766]
lrwx------. 1 root root 64 Jun 22 09:43 24 -> socket:[1580425862]
lrwx------. 1 root root 64 Jun 22 09:43 25 -> socket:[1580425970]
lrwx------. 1 root root 64 Jun 22 09:43 27 -> socket:[1580425874]
lrwx------. 1 root root 64 Jun 22 09:43 28 -> socket:[1580425881]
lrwx------. 1 root root 64 Jun 22 09:43 29 -> socket:[1580444029]
lrwx------. 1 root root 64 Jun 22 09:43 3 -> anon_inode:[eventpoll]
l-wx------. 1 root root 64 Jun 22 09:43 31 -> pipe:[1580450764]
lrwx------. 1 root root 64 Jun 22 09:43 32 -> /var/lib/ctdb/printer_list.tdb.0
lrwx------. 1 root root 64 Jun 22 09:43 33 -> /var/lib/ctdb/locking.tdb.0
lrwx------. 1 root root 64 Jun 22 09:43 34 -> /var/lib/ctdb/brlock.tdb.0
lrwx------. 1 root root 64 Jun 22 09:43 35 -> /var/lib/ctdb/connections.tdb.0
lrwx------. 1 root root 64 Jun 22 09:43 36 -> /var/lib/ctdb/sessionid.tdb.0
lrwx------. 1 root root 64 Jun 22 09:43 37 -> /var/lib/ctdb/g_lock.tdb.0
lrwx------. 1 root root 64 Jun 22 09:43 38 -> /var/lib/ctdb/serverid.tdb.0
l-wx------. 1 root root 64 Jun 22 09:43 4 -> /var/log/log.ctdb
lrwx------. 1 root root 64 Jun 22 09:43 5 -> socket:[1580347021]
lrwx------. 1 root root 64 Jun 22 09:43 6 -> anon_inode:[eventpoll]
lr-x------. 1 root root 64 Jun 22 09:43 7 -> pipe:[1580347169]
l-wx------. 1 root root 64 Jun 22 09:43 8 -> pipe:[1580347169]
lr-x------. 1 root root 64 Jun 22 09:43 9 -> pipe:[1580347174]

Tue Jun 21 20:16:52 CST 2016
root     25775  0.2  0.0  26796 14464 ?        SLs  20:06   0:01 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
root     26050  0.0  0.0  22080  5028 ?        S    20:06   0:00 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
Tue Jun 21 20:16:53 CST 2016

root     25775  0.1  0.0  26796 14464 ?        SLs  20:06   0:02 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
root     26050  0.0  0.0  22080  5028 ?        S    20:06   0:00 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
Tue Jun 21 20:30:52 CST 2016
root     25775  0.1  0.0  26928 14468 ?        SLs  20:06   0:02 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
root     26050  0.0  0.0  22080  5028 ?        S    20:06   0:00 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
Tue Jun 21 20:30:53 CST 2016
continue memory leak,
Wed Jun 22 09:39:42 CST 2016
root     25775  0.1  0.1  29768 17360 ?        SLs  Jun21   1:18 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
root     26050  0.0  0.0  22080  5028 ?        S    Jun21   0:15 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
Wed Jun 22 09:39:43 CST 2016
root     25775  0.1  0.1  29768 17364 ?        SLs  Jun21   1:18 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
root     26050  0.0  0.0  22080  5028 ?        S    Jun21   0:15 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
Wed Jun 22 09:39:44 CST 2016
Comment 4 James 2016-06-22 02:37:45 UTC
An hour or so, again to collect information :
[root@vx1600-cjf91 ~]# date
Wed Jun 22 10:32:13 CST 2016
[root@vx1600-cjf91 ~]# 


[root@vx1600-cjf91 ~]# ps aux | grep ctdb
root     18553  0.0  0.0   7956   872 pts/0    S+   10:23   0:00 grep ctdb
root     25775  0.1  0.1  29900 17524 ?        SLs  Jun21   1:22 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
root     26050  0.0  0.0  22080  5028 ?        S    Jun21   0:16 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
[root@vx1600-cjf91 ~]# cat /proc/25775/maps >> b.txt
[root@vx1600-cjf91 ~]# cat /proc/25775/smaps >> b.txt
[root@vx1600-cjf91 ~]# ls -l  /proc/25775/fd >> b.txt
[root@vx1600-cjf91 ~]# 



00400000-004c5000 r-xp 00000000 08:02 3369                               /usr/sbin/ctdbd
006c5000-006c7000 rw-p 000c5000 08:02 3369                               /usr/sbin/ctdbd
00ef0000-00f11000 rw-p 00000000 00:00 0 
00f11000-00f61000 rw-p 00000000 00:00 0 
00f61000-01281000 rw-p 00000000 00:00 0 
3547400000-3547420000 r-xp 00000000 08:02 11428                          /lib64/ld-2.12.so
354761f000-3547620000 r--p 0001f000 08:02 11428                          /lib64/ld-2.12.so
3547620000-3547621000 rw-p 00020000 08:02 11428                          /lib64/ld-2.12.so
3547621000-3547622000 rw-p 00000000 00:00 0 
7eff2f51b000-7eff2f65b000 rw-s 00000000 08:02 14406                      /var/lib/ctdb/printer_list.tdb.0
7eff2f65b000-7eff2f781000 rw-s 00000000 08:02 14404                      /var/lib/ctdb/brlock.tdb.0
7eff2f781000-7eff2f8a7000 rw-s 00000000 08:02 14403                      /var/lib/ctdb/connections.tdb.0
7eff2f8a7000-7eff2f9cd000 rw-s 00000000 08:02 14401                      /var/lib/ctdb/g_lock.tdb.0
7eff2f9cd000-7eff2faf3000 rw-s 00000000 08:02 14400                      /var/lib/ctdb/serverid.tdb.0
7eff2faf3000-7eff2fc19000 rw-s 00000000 08:02 14402                      /var/lib/ctdb/sessionid.tdb.0
7eff2fc19000-7eff2fd3f000 rw-s 00000000 08:02 14405                      /var/lib/ctdb/locking.tdb.0
7eff2fd3f000-7eff2fe7f000 rw-s 00000000 08:02 14393                      /var/lib/ctdb/persistent/ctdb.tdb.0
7eff2fe7f000-7eff2ffbf000 rw-s 00000000 08:02 14394                      /var/lib/ctdb/persistent/group_mapping.tdb.0
7eff2ffbf000-7eff300ff000 rw-s 00000000 08:02 14399                      /var/lib/ctdb/persistent/passdb.tdb.0
7eff300ff000-7eff3023f000 rw-s 00000000 08:02 14397                      /var/lib/ctdb/persistent/account_policy.tdb.0
7eff3023f000-7eff3037f000 rw-s 00000000 08:02 14395                      /var/lib/ctdb/persistent/registry.tdb.0
7eff3037f000-7eff304bf000 rw-s 00000000 08:02 14398                      /var/lib/ctdb/persistent/share_info.tdb.0
7eff304bf000-7eff305ff000 rw-s 00000000 08:02 14396                      /var/lib/ctdb/persistent/secrets.tdb.0
7eff305ff000-7eff3060b000 r-xp 00000000 08:02 124                        /lib64/libnss_files-2.12.so
7eff3060b000-7eff3080b000 ---p 0000c000 08:02 124                        /lib64/libnss_files-2.12.so
7eff3080b000-7eff3080c000 r--p 0000c000 08:02 124                        /lib64/libnss_files-2.12.so
7eff3080c000-7eff3080d000 rw-p 0000d000 08:02 124                        /lib64/libnss_files-2.12.so
7eff3080d000-7eff30810000 rw-p 00000000 00:00 0 
7eff30810000-7eff309a7000 r-xp 00000000 08:02 86                         /lib64/libc-2.12.so
7eff309a7000-7eff30ba7000 ---p 00197000 08:02 86                         /lib64/libc-2.12.so
7eff30ba7000-7eff30bab000 r--p 00197000 08:02 86                         /lib64/libc-2.12.so
7eff30bab000-7eff30bac000 rw-p 0019b000 08:02 86                         /lib64/libc-2.12.so
7eff30bac000-7eff30bb1000 rw-p 00000000 00:00 0 
7eff30bb1000-7eff30bb9000 r-xp 00000000 08:02 29                         /lib64/libpopt.so.0.0.0
7eff30bb9000-7eff30db9000 ---p 00008000 08:02 29                         /lib64/libpopt.so.0.0.0
7eff30db9000-7eff30dba000 rw-p 00008000 08:02 29                         /lib64/libpopt.so.0.0.0
7eff30dbc000-7eff30dbd000 rw-s 00000000 08:02 13887                      /var/lib/ctdb/state/persistent_health.tdb.0
7eff30dbd000-7eff30dbe000 rw-p 00000000 00:00 0 
7fff32703000-7fff32718000 rw-p 00000000 00:00 0                          [stack]
7fff327ff000-7fff32800000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
00400000-004c5000 r-xp 00000000 08:02 3369                               /usr/sbin/ctdbd
Size:                788 kB
Rss:                 788 kB
Pss:                 636 kB
Shared_Clean:        304 kB
Shared_Dirty:          0 kB
Private_Clean:       484 kB
Private_Dirty:         0 kB
Referenced:          788 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
006c5000-006c7000 rw-p 000c5000 08:02 3369                               /usr/sbin/ctdbd
Size:                  8 kB
Rss:                   8 kB
Pss:                   8 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         8 kB
Referenced:            8 kB
Anonymous:             8 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
00ef0000-00f11000 rw-p 00000000 00:00 0 
Size:                132 kB
Rss:                 132 kB
Pss:                 122 kB
Shared_Clean:          0 kB
Shared_Dirty:         20 kB
Private_Clean:         0 kB
Private_Dirty:       112 kB
Referenced:          132 kB
Anonymous:           132 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
00f11000-00f61000 rw-p 00000000 00:00 0 
Size:                320 kB
Rss:                 320 kB
Pss:                 312 kB
Shared_Clean:          0 kB
Shared_Dirty:         16 kB
Private_Clean:         0 kB
Private_Dirty:       304 kB
Referenced:          320 kB
Anonymous:           320 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
00f61000-01281000 rw-p 00000000 00:00 0 
Size:               3200 kB
Rss:                3156 kB
Pss:                3156 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:      3156 kB
Referenced:         3156 kB
Anonymous:          3156 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
3547400000-3547420000 r-xp 00000000 08:02 11428                          /lib64/ld-2.12.so
Size:                128 kB
Rss:                 128 kB
Pss:                  14 kB
Shared_Clean:        116 kB
Shared_Dirty:          0 kB
Private_Clean:        12 kB
Private_Dirty:         0 kB
Referenced:          128 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
354761f000-3547620000 r--p 0001f000 08:02 11428                          /lib64/ld-2.12.so
Size:                  4 kB
Rss:                   4 kB
Pss:                   2 kB
Shared_Clean:          0 kB
Shared_Dirty:          4 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            4 kB
Anonymous:             4 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
3547620000-3547621000 rw-p 00020000 08:02 11428                          /lib64/ld-2.12.so
Size:                  4 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         4 kB
Referenced:            4 kB
Anonymous:             4 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
3547621000-3547622000 rw-p 00000000 00:00 0 
Size:                  4 kB
Rss:                   4 kB
Pss:                   2 kB
Shared_Clean:          0 kB
Shared_Dirty:          4 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            4 kB
Anonymous:             4 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff2f51b000-7eff2f65b000 rw-s 00000000 08:02 14406                      /var/lib/ctdb/printer_list.tdb.0
Size:               1280 kB
Rss:                 796 kB
Pss:                 796 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:       796 kB
Private_Dirty:         0 kB
Referenced:          796 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff2f65b000-7eff2f781000 rw-s 00000000 08:02 14404                      /var/lib/ctdb/brlock.tdb.0
Size:               1176 kB
Rss:                 784 kB
Pss:                 784 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:       784 kB
Private_Dirty:         0 kB
Referenced:          784 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff2f781000-7eff2f8a7000 rw-s 00000000 08:02 14403                      /var/lib/ctdb/connections.tdb.0
Size:               1176 kB
Rss:                 784 kB
Pss:                 784 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:       784 kB
Private_Dirty:         0 kB
Referenced:          784 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff2f8a7000-7eff2f9cd000 rw-s 00000000 08:02 14401                      /var/lib/ctdb/g_lock.tdb.0
Size:               1176 kB
Rss:                 784 kB
Pss:                 784 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:       784 kB
Private_Dirty:         0 kB
Referenced:          784 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff2f9cd000-7eff2faf3000 rw-s 00000000 08:02 14400                      /var/lib/ctdb/serverid.tdb.0
Size:               1176 kB
Rss:                 784 kB
Pss:                 784 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:       784 kB
Private_Dirty:         0 kB
Referenced:          784 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff2faf3000-7eff2fc19000 rw-s 00000000 08:02 14402                      /var/lib/ctdb/sessionid.tdb.0
Size:               1176 kB
Rss:                 784 kB
Pss:                 784 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:       784 kB
Private_Dirty:         0 kB
Referenced:          784 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff2fc19000-7eff2fd3f000 rw-s 00000000 08:02 14405                      /var/lib/ctdb/locking.tdb.0
Size:               1176 kB
Rss:                 784 kB
Pss:                 784 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:       784 kB
Private_Dirty:         0 kB
Referenced:          784 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff2fd3f000-7eff2fe7f000 rw-s 00000000 08:02 14393                      /var/lib/ctdb/persistent/ctdb.tdb.0
Size:               1280 kB
Rss:                 796 kB
Pss:                 796 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:       796 kB
Private_Dirty:         0 kB
Referenced:          796 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff2fe7f000-7eff2ffbf000 rw-s 00000000 08:02 14394                      /var/lib/ctdb/persistent/group_mapping.tdb.0
Size:               1280 kB
Rss:                 796 kB
Pss:                 796 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:       796 kB
Private_Dirty:         0 kB
Referenced:          796 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff2ffbf000-7eff300ff000 rw-s 00000000 08:02 14399                      /var/lib/ctdb/persistent/passdb.tdb.0
Size:               1280 kB
Rss:                 796 kB
Pss:                 796 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:       796 kB
Private_Dirty:         0 kB
Referenced:          796 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff300ff000-7eff3023f000 rw-s 00000000 08:02 14397                      /var/lib/ctdb/persistent/account_policy.tdb.0
Size:               1280 kB
Rss:                 796 kB
Pss:                 796 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:       796 kB
Private_Dirty:         0 kB
Referenced:          796 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff3023f000-7eff3037f000 rw-s 00000000 08:02 14395                      /var/lib/ctdb/persistent/registry.tdb.0
Size:               1280 kB
Rss:                 836 kB
Pss:                 836 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:       836 kB
Private_Dirty:         0 kB
Referenced:          836 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff3037f000-7eff304bf000 rw-s 00000000 08:02 14398                      /var/lib/ctdb/persistent/share_info.tdb.0
Size:               1280 kB
Rss:                 796 kB
Pss:                 796 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:       796 kB
Private_Dirty:         0 kB
Referenced:          796 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff304bf000-7eff305ff000 rw-s 00000000 08:02 14396                      /var/lib/ctdb/persistent/secrets.tdb.0
Size:               1280 kB
Rss:                 796 kB
Pss:                 796 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:       796 kB
Private_Dirty:         0 kB
Referenced:          796 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff305ff000-7eff3060b000 r-xp 00000000 08:02 124                        /lib64/libnss_files-2.12.so
Size:                 48 kB
Rss:                  48 kB
Pss:                  11 kB
Shared_Clean:         40 kB
Shared_Dirty:          0 kB
Private_Clean:         8 kB
Private_Dirty:         0 kB
Referenced:           48 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff3060b000-7eff3080b000 ---p 0000c000 08:02 124                        /lib64/libnss_files-2.12.so
Size:               2048 kB
Rss:                   0 kB
Pss:                   0 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            0 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff3080b000-7eff3080c000 r--p 0000c000 08:02 124                        /lib64/libnss_files-2.12.so
Size:                  4 kB
Rss:                   4 kB
Pss:                   2 kB
Shared_Clean:          0 kB
Shared_Dirty:          4 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            4 kB
Anonymous:             4 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff3080c000-7eff3080d000 rw-p 0000d000 08:02 124                        /lib64/libnss_files-2.12.so
Size:                  4 kB
Rss:                   4 kB
Pss:                   2 kB
Shared_Clean:          0 kB
Shared_Dirty:          4 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            4 kB
Anonymous:             4 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff3080d000-7eff30810000 rw-p 00000000 00:00 0 
Size:                 12 kB
Rss:                  12 kB
Pss:                   8 kB
Shared_Clean:          0 kB
Shared_Dirty:          8 kB
Private_Clean:         0 kB
Private_Dirty:         4 kB
Referenced:           12 kB
Anonymous:            12 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff30810000-7eff309a7000 r-xp 00000000 08:02 86                         /lib64/libc-2.12.so
Size:               1628 kB
Rss:                1628 kB
Pss:                 595 kB
Shared_Clean:       1100 kB
Shared_Dirty:          0 kB
Private_Clean:       528 kB
Private_Dirty:         0 kB
Referenced:         1628 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff309a7000-7eff30ba7000 ---p 00197000 08:02 86                         /lib64/libc-2.12.so
Size:               2048 kB
Rss:                   0 kB
Pss:                   0 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            0 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff30ba7000-7eff30bab000 r--p 00197000 08:02 86                         /lib64/libc-2.12.so
Size:                 16 kB
Rss:                  16 kB
Pss:                   8 kB
Shared_Clean:          0 kB
Shared_Dirty:         16 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:           16 kB
Anonymous:            16 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff30bab000-7eff30bac000 rw-p 0019b000 08:02 86                         /lib64/libc-2.12.so
Size:                  4 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         4 kB
Referenced:            4 kB
Anonymous:             4 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff30bac000-7eff30bb1000 rw-p 00000000 00:00 0 
Size:                 20 kB
Rss:                  20 kB
Pss:                  18 kB
Shared_Clean:          0 kB
Shared_Dirty:          4 kB
Private_Clean:         0 kB
Private_Dirty:        16 kB
Referenced:           20 kB
Anonymous:            20 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff30bb1000-7eff30bb9000 r-xp 00000000 08:02 29                         /lib64/libpopt.so.0.0.0
Size:                 32 kB
Rss:                  32 kB
Pss:                  29 kB
Shared_Clean:          4 kB
Shared_Dirty:          0 kB
Private_Clean:        28 kB
Private_Dirty:         0 kB
Referenced:           32 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff30bb9000-7eff30db9000 ---p 00008000 08:02 29                         /lib64/libpopt.so.0.0.0
Size:               2048 kB
Rss:                   0 kB
Pss:                   0 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            0 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff30db9000-7eff30dba000 rw-p 00008000 08:02 29                         /lib64/libpopt.so.0.0.0
Size:                  4 kB
Rss:                   4 kB
Pss:                   2 kB
Shared_Clean:          0 kB
Shared_Dirty:          4 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            4 kB
Anonymous:             4 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff30dbc000-7eff30dbd000 rw-s 00000000 08:02 13887                      /var/lib/ctdb/state/persistent_health.tdb.0
Size:                  4 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         4 kB
Private_Dirty:         0 kB
Referenced:            4 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7eff30dbd000-7eff30dbe000 rw-p 00000000 00:00 0 
Size:                  4 kB
Rss:                   4 kB
Pss:                   2 kB
Shared_Clean:          0 kB
Shared_Dirty:          4 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            4 kB
Anonymous:             4 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7fff32703000-7fff32718000 rw-p 00000000 00:00 0                          [stack]
Size:                 88 kB
Rss:                  84 kB
Pss:                  50 kB
Shared_Clean:          0 kB
Shared_Dirty:         68 kB
Private_Clean:         0 kB
Private_Dirty:        16 kB
Referenced:           84 kB
Anonymous:            84 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
7fff327ff000-7fff32800000 r-xp 00000000 00:00 0                          [vdso]
Size:                  4 kB
Rss:                   4 kB
Pss:                   0 kB
Shared_Clean:          4 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            4 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Size:                  4 kB
Rss:                   0 kB
Pss:                   0 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            0 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
total 0
lr-x------. 1 root root 64 Jun 22 09:43 0 -> /dev/null
l-wx------. 1 root root 64 Jun 22 09:43 1 -> pipe:[1580347174]
lrwx------. 1 root root 64 Jun 22 09:43 10 -> /var/lib/ctdb/state/persistent_health.tdb.0
lrwx------. 1 root root 64 Jun 22 09:43 11 -> socket:[1580385812]
lrwx------. 1 root root 64 Jun 22 09:43 12 -> socket:[1597056131]
lrwx------. 1 root root 64 Jun 22 09:43 13 -> /var/lib/ctdb/persistent/secrets.tdb.0
lrwx------. 1 root root 64 Jun 22 09:43 14 -> /var/lib/ctdb/persistent/share_info.tdb.0
lrwx------. 1 root root 64 Jun 22 09:43 15 -> /var/lib/ctdb/persistent/registry.tdb.0
lrwx------. 1 root root 64 Jun 22 09:43 16 -> /var/lib/ctdb/persistent/account_policy.tdb.0
lrwx------. 1 root root 64 Jun 22 09:43 17 -> /var/lib/ctdb/persistent/passdb.tdb.0
lrwx------. 1 root root 64 Jun 22 09:43 18 -> /var/lib/ctdb/persistent/group_mapping.tdb.0
lrwx------. 1 root root 64 Jun 22 09:43 19 -> /var/lib/ctdb/persistent/ctdb.tdb.0
l-wx------. 1 root root 64 Jun 22 09:43 2 -> pipe:[1580347174]
lrwx------. 1 root root 64 Jun 22 09:43 20 -> socket:[1580400159]
lrwx------. 1 root root 64 Jun 22 09:43 21 -> socket:[1597091707]
lrwx------. 1 root root 64 Jun 22 10:24 22 -> socket:[486250748]
lrwx------. 1 root root 64 Jun 22 09:43 23 -> socket:[1580450766]
lrwx------. 1 root root 64 Jun 22 09:43 24 -> socket:[1580425862]
lrwx------. 1 root root 64 Jun 22 09:43 25 -> socket:[1580425970]
lrwx------. 1 root root 64 Jun 22 09:43 27 -> socket:[1580425874]
lrwx------. 1 root root 64 Jun 22 09:43 28 -> socket:[1580425881]
lrwx------. 1 root root 64 Jun 22 09:43 29 -> socket:[1580444029]
lrwx------. 1 root root 64 Jun 22 09:43 3 -> anon_inode:[eventpoll]
l-wx------. 1 root root 64 Jun 22 09:43 31 -> pipe:[1580450764]
lrwx------. 1 root root 64 Jun 22 09:43 32 -> /var/lib/ctdb/printer_list.tdb.0
lrwx------. 1 root root 64 Jun 22 09:43 33 -> /var/lib/ctdb/locking.tdb.0
lrwx------. 1 root root 64 Jun 22 09:43 34 -> /var/lib/ctdb/brlock.tdb.0
lrwx------. 1 root root 64 Jun 22 09:43 35 -> /var/lib/ctdb/connections.tdb.0
lrwx------. 1 root root 64 Jun 22 09:43 36 -> /var/lib/ctdb/sessionid.tdb.0
lrwx------. 1 root root 64 Jun 22 09:43 37 -> /var/lib/ctdb/g_lock.tdb.0
lrwx------. 1 root root 64 Jun 22 09:43 38 -> /var/lib/ctdb/serverid.tdb.0
l-wx------. 1 root root 64 Jun 22 09:43 4 -> /var/log/log.ctdb
lrwx------. 1 root root 64 Jun 22 09:43 5 -> socket:[1580347021]
lrwx------. 1 root root 64 Jun 22 09:43 6 -> anon_inode:[eventpoll]
lr-x------. 1 root root 64 Jun 22 09:43 7 -> pipe:[1580347169]
l-wx------. 1 root root 64 Jun 22 09:43 8 -> pipe:[1580347169]
lr-x------. 1 root root 64 Jun 22 09:43 9 -> pipe:[1580347174]
Comment 5 Amitay Isaacs 2016-06-22 02:44:38 UTC
Please do not post such large output in the comments. Attach the output, so it's easier to read.

Here CTDB is consuming around 29MB and most of it is related to the databases.

Where do you see CTDB is consuming 700MB?

The column you are referring to is showing resident memory (RSS).  That will increase and decrease in the lifetime of CTDB depending on the activity.  Similarly virtual size (VSZ).

Are you seeing the memory increase in RSS if you don't run Samba?
Comment 6 James 2016-06-22 06:07:35 UTC
In our production system, running more than 100 days, the memory increased to 700M, the impact of the system operation, the production system has been restarted ctdb; now analyze the problem, found that every day in the leak;

root     14179  0.1  0.6 656284 39272 ?        Ssl  Jan12 367:09 /usr/local/hstor/sms_install/ucsfs/sbin/ucsfs --volfile-server=172.16.53.56 --volfile-id=/cluster_lock_volume /root/lock
root     14219  0.5 11.5 745064 713884 ?       SLs  Jan12 1148:44 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE

Began to suspect that the impact of samba, after closing the samba, ctdb is still continuing to increase the RES memory;

In our application, each node has a periodic call to ctdb status, ctdb ip query ctdb node information
Comment 7 James 2016-06-29 06:57:54 UTC
Hi all
  update ctdb lib source,talloc to version 2.1.7,tdb to version 1.2.13,tevent to 0.9.28, recompile ctdb ,There are still memory leaks.
  Who can give advice, how to analyze and solve the problem;
  Using Valgrind detection, too many alarms; using fortify tools to check, also did not find valuable information
Comment 8 Amitay Isaacs 2016-06-29 08:18:41 UTC
(In reply to James from comment #7)

We first need to establish that there is a memory leak.  Increase in the process memory does not necessarily translate into memory leak.

Can you provide following details before and after (memory increase)?

1. /proc/<pid>/smaps
2. ctdb dumpmemory

Please attach the output as files.

Also, note that CTDB 2.5.x is out of support.  There will not be any new releases.  Please upgrade to newer version of CTDB.
Comment 9 James 2016-06-29 13:19:22 UTC
Hi all
ctdb info:
cat /proc/16798/smaps >> smaps
ctdb dumpmemory >> dumpmemory.txt

Fri Jun 24 16:06:48 CST 2016
root     16798  0.3  0.1  27944 15592 ?        SLs  10:55   1:09 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
root     17437  0.0  0.0  22088  5020 ?        S    10:55   0:06 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
Fri Jun 24 16:06:49 CST 2016
root     16798  0.3  0.1  27944 15592 ?        SLs  10:55   1:09 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
root     17437  0.0  0.0  22088  5020 ?        S    10:55   0:06 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
Fri Jun 24 16:06:50 CST 2016
root     16798  0.3  0.1  27944 15592 ?        SLs  10:55   1:09 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
root     17437  0.0  0.0  22088  5020 ?        S    10:55   0:06 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/


Wed Jun 29 21:04:42 CST 2016
root     16798  0.3  0.2  55720 43352 ?        SLs  Jun24  30:51 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
root     17437  0.0  0.0  22088  5020 ?        S    Jun24   2:36 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
Wed Jun 29 21:04:43 CST 2016
root     16798  0.3  0.2  55720 43352 ?        SLs  Jun24  30:51 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
root     17437  0.0  0.0  22088  5020 ?        S    Jun24   2:36 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE

ctdb running 5 days, ctdb memory increase from 15592KB to 43352 KB
Comment 10 James 2016-06-29 13:21:10 UTC
Created attachment 12234 [details]
smaps
Comment 11 James 2016-06-29 13:22:15 UTC
Created attachment 12235 [details]
dumpmemory Wed Jun 29 21:18:51
Comment 12 James 2016-06-29 13:23:58 UTC
Created attachment 12236 [details]
ps ctdb  per second
Comment 13 James 2016-06-29 13:31:16 UTC
Hi,
 ctdb-2.5.6.tar.gz Is the final version,Is there an updated version?
 I install ctdb-2.5.6,Is also the same leakage;
 Memory should be allocated, the use of complete, will be released, should not be a single direction has been growing;
Comment 14 Amitay Isaacs 2016-06-29 13:46:29 UTC
(In reply to James from comment #9)

Can you please provide the complete information?

I need two sets of outputs to compare the memory utilization.

Do you have the smaps output from Jun 24th?  If not, you can wait a day or two and provide smaps information for the same process.
Comment 15 Amitay Isaacs 2016-06-29 13:48:15 UTC
(In reply to James from comment #13)

There will no more CTDB 2.5.x releases.  You can switch to the latest stable release 4.4.x.

CTDB is now part of Samba and you can download the latest Samba release 4.4.x from samba.org.
Comment 16 James 2016-06-30 01:22:01 UTC
Created attachment 12243 [details]
ctdb dumpmemory Thu Jun 30 09:18:34 CST 2016
Comment 17 James 2016-06-30 01:27:08 UTC
Created attachment 12244 [details]
cat /proc/16798/smaps Thu Jun 30 09:23:45 CST 2016
Comment 18 James 2016-06-30 01:42:59 UTC
HI,
  Above is the information that ctdb runs for 11 hours. 
  Wed Jun 29 21:04:42 CST 2016
root     16798  0.3  0.2  55720 43352 ?        SLs  Jun24  30:51 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
[root@vx1600-cjf50 ~]# date
Thu Jun 30 09:38:47 CST 2016
[root@vx1600-cjf50 ~]# 
[root@vx1600-cjf50 ~]# 
[root@vx1600-cjf50 ~]# ps aux | grep ctdb 
root      2967  0.0  0.0   7956   844 pts/0    S+   09:38   0:00 grep ctdb
root     16798  0.3  0.3  58292 46052 ?        SLs  Jun24  34:10 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
root     17437  0.0  0.0  22088  5020 ?        S    Jun24   2:52 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
[root@vx1600-cjf50 ~]# 
  ctdb memory from 43352KB to 46052 KB
Comment 19 James 2016-07-04 07:03:49 UTC
Hi
 I download samba-4.4.4,compile and install ctdb ;
 ctdb status
Number of nodes:5
pnn:0 192.168.77.91    OK
pnn:1 192.168.77.50    OK (THIS NODE)
pnn:2 192.168.77.35    OK
pnn:3 192.168.77.26    OK
pnn:4 192.168.77.60    OK
Generation:222815519
Size:5
hash:0 lmaster:0
hash:1 lmaster:1
hash:2 lmaster:2
hash:3 lmaster:3
hash:4 lmaster:4
Recovery mode:NORMAL (0)
Recovery master:4
ctdb version
CTDB version: 4.4.4
 the 5 node ctdb cluster running 5day ; every node ctdb memory increase;
 The phenomenon is as follows :
26 node :
Thu Jun 30 19:51:44  26 node memory is 32244 KB
Sun Jul  3 04:25:42  26 node memory is 34292 KB
between Thu Jun 30 19:51:44 and  Sun Jul  3 04:25:42 ctdb memory no change;
start at this time Sun Jul  3 21:12:40 CST 2016
ctdb memory increase 4KB  per minute !

26 node info :
Thu Jun 30 19:51:44 CST 2016
root     28510  0.1  0.2  48052 32244 ?        SLs  18:13   0:09 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
root     28779  0.0  0.0  22088  5032 ?        S    18:13   0:01 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
Thu Jun 30 19:51:45 CST 2016

Sun Jul  3 04:25:41 CST 2016
root     28510  0.1  0.2  48052 32244 ?        SLs  Jun30   5:22 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
root     28779  0.0  0.0  22088  5032 ?        S    Jun30   0:39 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
Sun Jul  3 04:25:42 CST 2016
root     28510  0.1  0.2  48052 34292 ?        SLs  Jun30   5:22 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
root     28779  0.0  0.0  22088  5032 ?        S    Jun30   0:39 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
Sun Jul  3 04:25:43 CST 2016

root     28510  0.1  0.2  48052 34292 ?        SLs  Jun30   6:58 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
root     28779  0.0  0.0  22088  5032 ?        S    Jun30   0:50 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
Sun Jul  3 21:12:40 CST 2016
root     28510  0.1  0.2  48052 34296 ?        SLs  Jun30   6:58 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
root     28779  0.0  0.0  22088  5032 ?     
root     28510  0.1  0.2  48052 34292 ?        SLs  Jun30   6:58 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
root     28779  0.0  0.0  22088  5032 ?        S    Jun30   0:50 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
Sun Jul  3 21:12:40 CST 2016
root     28510  0.1  0.2  48052 34296 ?        SLs  Jun30   6:58 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
root     28779  0.0  0.0  22088  5032 ?        S    Jun30   0:50 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE

91 node :
ctdb memory Has been increasing ! from 16992KB to 38592KB
Thu Jun 30 22:27:22 CST 2016
root      8220  0.1  0.1  30296 16992 ?        SLs  18:13   0:24 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
root      8493  0.0  0.0  22088  5032 ?        S    18:13   0:03 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
Thu Jun 30 22:27:23 CST 2016
Mon Jul  4 14:23:15 CST 2016
root      8220  0.1  0.2  51828 38592 ?        SLs  Jun30   8:16 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
root      8493  0.0  0.0  22088  5032 ?        S    Jun30   1:25 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
Comment 20 Amitay Isaacs 2016-07-04 07:24:50 UTC
(In reply to James from comment #19)

CTDB process memory will increase depending on the Samba activity and the database sizes.

Do you notice CTDB memory increasing when there is no Samba activity?

> 91 node :
> ctdb memory Has been increasing ! from 16992KB to 38592KB

This is possible if the sizes of databases have grown.  Please provide smaps output (before and after) whenever reporting memory increase in the ctdbd process.  Also include output of "ctdb statistics" command.
Comment 21 James 2016-07-05 02:30:04 UTC
Created attachment 12260 [details]
ctdb info from 07-04-1444 to 07-05-0926
Comment 22 James 2016-07-05 02:41:17 UTC
Hi,
  I'm sure it's closed down Samba,no smb ,no nmb,no winbind running ;
[root@vx1600-cjf91 ~]# ps aux | grep smb
root      8380  0.0  0.0   7956   860 pts/2    S+   10:28   0:00 grep smb
[root@vx1600-cjf91 ~]# ps aux | grep nmb
root      8647  0.0  0.0   7956   860 pts/2    S+   10:28   0:00 grep nmb
[root@vx1600-cjf91 ~]# ps aux | grep winbind
root      9592  0.0  0.0   7956   868 pts/2    S+   10:28   0:00 grep winbind
[root@vx1600-cjf91 ~]# 
  I set ctdb setdebug 3 ,only see eventscript monitor and tmp tdb Vacuuming in operation;Whether the Tmp tdb Vacuuming operation may have a memory leak.

now ,ctdb memory increase to 43560 KB.
  Collected information on the above。
Comment 23 James 2016-07-11 06:58:34 UTC
Hi all 

   In the memory leak, I opened the debug switch,mode 4, found that there are the following log, data memory structure abnormalities;
  2016/07/11 16:40:40.519305 [22853]: Ending timer event 0x27f6910 "queue_next_trigger"
2016/07/11 16:40:40.521533 [22853]: server/ctdb_daemon.c:927 Created SOCKET FD:13 to connected child
2016/07/11 16:40:40.521575 [22853]: server/ctdb_daemon.c:889 client request 2147483448 of type 7 length 64 from node 0 to 4026531841
2016/07/11 16:40:40.521591 [22853]: Running timer event 0x27d6f00 "queue_next_trigger"
2016/07/11 16:40:40.521597 [22853]: server/ctdb_server.c:292 ctdb request 13157570 of type 7 length 64 from node 1 to 1
   node num 4026531841 is too big ; 
   Problem may be the reason, how to continue to analyze the abnormal location;
Comment 24 James 2016-07-13 08:49:25 UTC
Hi Amitay,
  I use valgind tools to check memory, commands are as follows
  valgrind --leak-check=full --log-file=/var/log/log1 --show-reachable=yes /usr/sbin/ctdbd --nosetsched --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
  Found a memory leak location, can help confirm it.
  ==31999== 89,216 bytes in 2,720 blocks are definitely lost in loss record 112 of 112
==31999==    at 0x4A05FDE: malloc (vg_replace_malloc.c:236)
==31999==    by 0x4EBFDCF: scandir (in /lib64/libc-2.12.so)
==31999==    by 0x4375A0: ctdb_get_script_list (eventscript.c:168)
==31999==    by 0x439385: ctdb_event_script_callback_v (eventscript.c:768)
==31999==    by 0x4396AF: ctdb_event_script_callback (eventscript.c:824)
==31999==    by 0x42299E: ctdb_check_health (ctdb_monitor.c:383)
==31999==    by 0x48507D: tevent_common_loop_timer_delay (tevent_timed.c:343)
==31999==    by 0x48BF4A: epoll_event_loop (tevent_epoll.c:667)
==31999==    by 0x48C7CA: epoll_event_loop_once (tevent_epoll.c:931)
==31999==    by 0x487D9A: std_event_loop_once (tevent_standard.c:112)
==31999==    by 0x483AC0: _tevent_loop_once (tevent.c:530)
==31999==    by 0x483D38: tevent_common_loop_wait (tevent.c:634)
==31999== 
==31999== LEAK SUMMARY:
==31999==    definitely lost: 101,024 bytes in 3,080 blocks
==31999==    indirectly lost: 0 bytes in 0 blocks
==31999==      possibly lost: 197,463 bytes in 136 blocks
==31999==    still reachable: 46,681 bytes in 24 blocks
==31999==         suppressed: 0 bytes in 0 blocks
Does this scandir function use error,in linux platform
Comment 25 Jeremy Allison 2016-07-13 16:28:24 UTC
This looks like a memory leak that was fixed in 2015 with the commit:

----------------------------------------------------------------------------
commit c1e8bfb186c5cbeafbce9f2767db82edb579d5e1
Author: Volker Lendecke <vl@samba.org>
Date:   Fri Mar 13 14:12:41 2015 +0000

    ctdb: Fix memleak in ctdb_get_script_list
    
    scandir allocates every name individually, see example code in susv4 or man
    scandir
    
    Signed-off-by: Volker Lendecke <vl@samba.org>
    Reviewed-by: Michael Adam <obnox@samba.org>

diff --git a/ctdb/server/eventscript.c b/ctdb/server/eventscript.c
index 4cbb846..a47396a 100644
--- a/ctdb/server/eventscript.c
+++ b/ctdb/server/eventscript.c
@@ -177,8 +177,7 @@ static struct ctdb_scripts_wire *ctdb_get_script_list(struct ctdb_context *ctdb,
                                   + sizeof(scripts->scripts[0]) * count);
        if (scripts == NULL) {
                DEBUG(DEBUG_ERR, (__location__ " Failed to allocate scripts\n"));
-               free(namelist);
-               return NULL;
+               goto done;
        }
        scripts->num_scripts = count;
 
@@ -191,6 +190,10 @@ static struct ctdb_scripts_wire *ctdb_get_script_list(struct ctdb_context *ctdb,
                }
        }
 
+done:
+       for (i=0; i<count; i++) {
+               free(namelist[i]);
+       }
        free(namelist);
        return scripts;
 }
----------------------------------------------------------------------------

Can you confirm your ctdb code has this fix ?
Comment 26 Amitay Isaacs 2016-07-13 16:41:06 UTC
Good catch, James.

Volker's patch is not present in CTDB 2.5.x.  That explains the memory leak.

However, this patch is present in CTDB 4.4.4.  So there should not be any memory leak in ctdb_get_script_list().