Hi all Ctdb was found to have a persistent memory leak, with the increase in memory usage per 4K size; In order to simplify the problem analysis, close the samba service, or continue to leak; Use ctdb dumpmemory rddumpmemory analysis, there is no memory leak found the structure Related information is as follows: [root@vx1600-cjf35 ~]# ctdb status Number of nodes:5 pnn:0 192.168.77.91 OK pnn:1 192.168.77.50 OK pnn:2 192.168.77.35 OK (THIS NODE) pnn:3 192.168.77.26 OK pnn:4 192.168.77.60 OK Generation:1795168275 Size:5 hash:0 lmaster:0 hash:1 lmaster:1 hash:2 lmaster:2 hash:3 lmaster:3 hash:4 lmaster:4 Recovery mode:NORMAL (0) Recovery master:0 [root@vx1600-cjf35 ~]# ctdb ip Public IPs on node 2 173.16.10.1 0 173.16.10.2 3 173.16.10.3 1 173.16.10.4 3 173.16.10.5 2 173.16.10.6 4 173.16.10.7 0 173.16.10.8 1 173.16.10.9 0 173.16.10.10 2 Memory leak,continue 4k, root 19817 0.3 0.1 33468 21184 ? SLs Jun17 6:38 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE root 20178 0.0 0.0 22084 5016 ? S Jun17 0:31 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE Sat Jun 18 23:40:42 CST 2016 root 19817 0.3 0.1 33468 21184 ? SLs Jun17 6:38 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE root 20178 0.0 0.0 22084 5016 ? S Jun17 0:31 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE Sat Jun 18 23:40:43 CST 2016 from 21184 increase to 21188 ,4K increase root 19817 0.3 0.1 33468 21188 ? SLs Jun17 6:38 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE root 20178 0.0 0.0 22084 5016 ? S Jun17 0:31 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE Sat Jun 18 23:40:44 CST 2016 root 19817 0.3 0.1 33468 21188 ? SLs Jun17 6:38 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE root 20178 0.0 0.0 22084 5016 ? S Jun17 0:31 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE Sat Jun 18 23:40:45 CST 2016 root 19817 0.3 0.1 33468 21188 ? SLs Jun17 6:38 /usr/sbin/ctdbd --rec persistent memory leak,Leaked 4M memory every day Sun Jun 19 02:18:23 CST 2016 root 19817 0.3 0.1 34008 21760 ? SLs Jun17 7:12 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE root 20178 0.0 0.0 22084 5016 ? S Jun17 0:34 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE Sun Jun 19 02:18:24 CST 2016 root 19817 0.3 0.1 34008 21764 ? SLs Jun17 7:12 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE root 20178 0.0 0.0 22084 5016 ? S Jun17 0:34 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE Sun Jun 19 02:18:26 CST 2016 Best Regards James
Created attachment 12193 [details] dumpmemory ,rddumpmemory Running a few months, ctdb memory increased to 700M
(In reply to James from comment #1) Can you provide the output of following commands: cat /proc/19817/maps cat /proc/19817/smaps ls -l /proc/19817/fd assuming 19817 is the ctdb daemon pid or use "ctdb getpid". Amitay.
The problem can be reproduced: 191.168.17.91 ctdb info: [root@vx1600-cjf91 ~]# ctdb getpid Pid:25775 [root@vx1600-cjf91 ~]# ps aux | grep ctdb root 25775 0.1 0.1 29768 17396 ? SLs Jun21 1:19 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE root 26050 0.0 0.0 22080 5028 ? S Jun21 0:16 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE root 26997 0.0 0.0 7960 892 pts/0 S+ 09:48 0:00 grep ctdb [root@vx1600-cjf91 ~]# 00400000-004c5000 r-xp 00000000 08:02 3369 /usr/sbin/ctdbd 006c5000-006c7000 rw-p 000c5000 08:02 3369 /usr/sbin/ctdbd 00ef0000-00f11000 rw-p 00000000 00:00 0 00f11000-00f61000 rw-p 00000000 00:00 0 00f61000-01260000 rw-p 00000000 00:00 0 3547400000-3547420000 r-xp 00000000 08:02 11428 /lib64/ld-2.12.so 354761f000-3547620000 r--p 0001f000 08:02 11428 /lib64/ld-2.12.so 3547620000-3547621000 rw-p 00020000 08:02 11428 /lib64/ld-2.12.so 3547621000-3547622000 rw-p 00000000 00:00 0 7eff2f51b000-7eff2f65b000 rw-s 00000000 08:02 14406 /var/lib/ctdb/printer_list.tdb.0 7eff2f65b000-7eff2f781000 rw-s 00000000 08:02 14404 /var/lib/ctdb/brlock.tdb.0 7eff2f781000-7eff2f8a7000 rw-s 00000000 08:02 14403 /var/lib/ctdb/connections.tdb.0 7eff2f8a7000-7eff2f9cd000 rw-s 00000000 08:02 14401 /var/lib/ctdb/g_lock.tdb.0 7eff2f9cd000-7eff2faf3000 rw-s 00000000 08:02 14400 /var/lib/ctdb/serverid.tdb.0 7eff2faf3000-7eff2fc19000 rw-s 00000000 08:02 14402 /var/lib/ctdb/sessionid.tdb.0 7eff2fc19000-7eff2fd3f000 rw-s 00000000 08:02 14405 /var/lib/ctdb/locking.tdb.0 7eff2fd3f000-7eff2fe7f000 rw-s 00000000 08:02 14393 /var/lib/ctdb/persistent/ctdb.tdb.0 7eff2fe7f000-7eff2ffbf000 rw-s 00000000 08:02 14394 /var/lib/ctdb/persistent/group_mapping.tdb.0 7eff2ffbf000-7eff300ff000 rw-s 00000000 08:02 14399 /var/lib/ctdb/persistent/passdb.tdb.0 7eff300ff000-7eff3023f000 rw-s 00000000 08:02 14397 /var/lib/ctdb/persistent/account_policy.tdb.0 7eff3023f000-7eff3037f000 rw-s 00000000 08:02 14395 /var/lib/ctdb/persistent/registry.tdb.0 7eff3037f000-7eff304bf000 rw-s 00000000 08:02 14398 /var/lib/ctdb/persistent/share_info.tdb.0 7eff304bf000-7eff305ff000 rw-s 00000000 08:02 14396 /var/lib/ctdb/persistent/secrets.tdb.0 7eff305ff000-7eff3060b000 r-xp 00000000 08:02 124 /lib64/libnss_files-2.12.so 7eff3060b000-7eff3080b000 ---p 0000c000 08:02 124 /lib64/libnss_files-2.12.so 7eff3080b000-7eff3080c000 r--p 0000c000 08:02 124 /lib64/libnss_files-2.12.so 7eff3080c000-7eff3080d000 rw-p 0000d000 08:02 124 /lib64/libnss_files-2.12.so 7eff3080d000-7eff30810000 rw-p 00000000 00:00 0 7eff30810000-7eff309a7000 r-xp 00000000 08:02 86 /lib64/libc-2.12.so 7eff309a7000-7eff30ba7000 ---p 00197000 08:02 86 /lib64/libc-2.12.so 7eff30ba7000-7eff30bab000 r--p 00197000 08:02 86 /lib64/libc-2.12.so 7eff30bab000-7eff30bac000 rw-p 0019b000 08:02 86 /lib64/libc-2.12.so 7eff30bac000-7eff30bb1000 rw-p 00000000 00:00 0 7eff30bb1000-7eff30bb9000 r-xp 00000000 08:02 29 /lib64/libpopt.so.0.0.0 7eff30bb9000-7eff30db9000 ---p 00008000 08:02 29 /lib64/libpopt.so.0.0.0 7eff30db9000-7eff30dba000 rw-p 00008000 08:02 29 /lib64/libpopt.so.0.0.0 7eff30dbc000-7eff30dbd000 rw-s 00000000 08:02 13887 /var/lib/ctdb/state/persistent_health.tdb.0 7eff30dbd000-7eff30dbe000 rw-p 00000000 00:00 0 7fff32703000-7fff32718000 rw-p 00000000 00:00 0 [stack] 7fff327ff000-7fff32800000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] 00400000-004c5000 r-xp 00000000 08:02 3369 /usr/sbin/ctdbd Size: 788 kB Rss: 788 kB Pss: 636 kB Shared_Clean: 304 kB Shared_Dirty: 0 kB Private_Clean: 484 kB Private_Dirty: 0 kB Referenced: 788 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 006c5000-006c7000 rw-p 000c5000 08:02 3369 /usr/sbin/ctdbd Size: 8 kB Rss: 8 kB Pss: 8 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 8 kB Referenced: 8 kB Anonymous: 8 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 00ef0000-00f11000 rw-p 00000000 00:00 0 Size: 132 kB Rss: 132 kB Pss: 122 kB Shared_Clean: 0 kB Shared_Dirty: 20 kB Private_Clean: 0 kB Private_Dirty: 112 kB Referenced: 132 kB Anonymous: 132 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 00f11000-00f61000 rw-p 00000000 00:00 0 Size: 320 kB Rss: 320 kB Pss: 312 kB Shared_Clean: 0 kB Shared_Dirty: 16 kB Private_Clean: 0 kB Private_Dirty: 304 kB Referenced: 320 kB Anonymous: 320 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 00f61000-01260000 rw-p 00000000 00:00 0 Size: 3068 kB Rss: 3012 kB Pss: 3012 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 3012 kB Referenced: 3012 kB Anonymous: 3012 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 3547400000-3547420000 r-xp 00000000 08:02 11428 /lib64/ld-2.12.so Size: 128 kB Rss: 128 kB Pss: 14 kB Shared_Clean: 116 kB Shared_Dirty: 0 kB Private_Clean: 12 kB Private_Dirty: 0 kB Referenced: 128 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 354761f000-3547620000 r--p 0001f000 08:02 11428 /lib64/ld-2.12.so Size: 4 kB Rss: 4 kB Pss: 2 kB Shared_Clean: 0 kB Shared_Dirty: 4 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 4 kB Anonymous: 4 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 3547620000-3547621000 rw-p 00020000 08:02 11428 /lib64/ld-2.12.so Size: 4 kB Rss: 4 kB Pss: 4 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 4 kB Referenced: 4 kB Anonymous: 4 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 3547621000-3547622000 rw-p 00000000 00:00 0 Size: 4 kB Rss: 4 kB Pss: 2 kB Shared_Clean: 0 kB Shared_Dirty: 4 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 4 kB Anonymous: 4 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff2f51b000-7eff2f65b000 rw-s 00000000 08:02 14406 /var/lib/ctdb/printer_list.tdb.0 Size: 1280 kB Rss: 796 kB Pss: 796 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 796 kB Private_Dirty: 0 kB Referenced: 796 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff2f65b000-7eff2f781000 rw-s 00000000 08:02 14404 /var/lib/ctdb/brlock.tdb.0 Size: 1176 kB Rss: 784 kB Pss: 784 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 784 kB Private_Dirty: 0 kB Referenced: 784 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff2f781000-7eff2f8a7000 rw-s 00000000 08:02 14403 /var/lib/ctdb/connections.tdb.0 Size: 1176 kB Rss: 784 kB Pss: 784 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 784 kB Private_Dirty: 0 kB Referenced: 784 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff2f8a7000-7eff2f9cd000 rw-s 00000000 08:02 14401 /var/lib/ctdb/g_lock.tdb.0 Size: 1176 kB Rss: 784 kB Pss: 784 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 784 kB Private_Dirty: 0 kB Referenced: 784 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff2f9cd000-7eff2faf3000 rw-s 00000000 08:02 14400 /var/lib/ctdb/serverid.tdb.0 Size: 1176 kB Rss: 784 kB Pss: 784 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 784 kB Private_Dirty: 0 kB Referenced: 784 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff2faf3000-7eff2fc19000 rw-s 00000000 08:02 14402 /var/lib/ctdb/sessionid.tdb.0 Size: 1176 kB Rss: 784 kB Pss: 784 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 784 kB Private_Dirty: 0 kB Referenced: 784 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff2fc19000-7eff2fd3f000 rw-s 00000000 08:02 14405 /var/lib/ctdb/locking.tdb.0 Size: 1176 kB Rss: 784 kB Pss: 784 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 784 kB Private_Dirty: 0 kB Referenced: 784 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff2fd3f000-7eff2fe7f000 rw-s 00000000 08:02 14393 /var/lib/ctdb/persistent/ctdb.tdb.0 Size: 1280 kB Rss: 796 kB Pss: 796 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 796 kB Private_Dirty: 0 kB Referenced: 796 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff2fe7f000-7eff2ffbf000 rw-s 00000000 08:02 14394 /var/lib/ctdb/persistent/group_mapping.tdb.0 Size: 1280 kB Rss: 796 kB Pss: 796 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 796 kB Private_Dirty: 0 kB Referenced: 796 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff2ffbf000-7eff300ff000 rw-s 00000000 08:02 14399 /var/lib/ctdb/persistent/passdb.tdb.0 Size: 1280 kB Rss: 796 kB Pss: 796 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 796 kB Private_Dirty: 0 kB Referenced: 796 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff300ff000-7eff3023f000 rw-s 00000000 08:02 14397 /var/lib/ctdb/persistent/account_policy.tdb.0 Size: 1280 kB Rss: 796 kB Pss: 796 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 796 kB Private_Dirty: 0 kB Referenced: 796 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff3023f000-7eff3037f000 rw-s 00000000 08:02 14395 /var/lib/ctdb/persistent/registry.tdb.0 Size: 1280 kB Rss: 836 kB Pss: 836 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 836 kB Private_Dirty: 0 kB Referenced: 836 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff3037f000-7eff304bf000 rw-s 00000000 08:02 14398 /var/lib/ctdb/persistent/share_info.tdb.0 Size: 1280 kB Rss: 796 kB Pss: 796 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 796 kB Private_Dirty: 0 kB Referenced: 796 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff304bf000-7eff305ff000 rw-s 00000000 08:02 14396 /var/lib/ctdb/persistent/secrets.tdb.0 Size: 1280 kB Rss: 796 kB Pss: 796 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 796 kB Private_Dirty: 0 kB Referenced: 796 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff305ff000-7eff3060b000 r-xp 00000000 08:02 124 /lib64/libnss_files-2.12.so Size: 48 kB Rss: 48 kB Pss: 11 kB Shared_Clean: 40 kB Shared_Dirty: 0 kB Private_Clean: 8 kB Private_Dirty: 0 kB Referenced: 48 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff3060b000-7eff3080b000 ---p 0000c000 08:02 124 /lib64/libnss_files-2.12.so Size: 2048 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff3080b000-7eff3080c000 r--p 0000c000 08:02 124 /lib64/libnss_files-2.12.so Size: 4 kB Rss: 4 kB Pss: 2 kB Shared_Clean: 0 kB Shared_Dirty: 4 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 4 kB Anonymous: 4 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff3080c000-7eff3080d000 rw-p 0000d000 08:02 124 /lib64/libnss_files-2.12.so Size: 4 kB Rss: 4 kB Pss: 2 kB Shared_Clean: 0 kB Shared_Dirty: 4 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 4 kB Anonymous: 4 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff3080d000-7eff30810000 rw-p 00000000 00:00 0 Size: 12 kB Rss: 12 kB Pss: 8 kB Shared_Clean: 0 kB Shared_Dirty: 8 kB Private_Clean: 0 kB Private_Dirty: 4 kB Referenced: 12 kB Anonymous: 12 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff30810000-7eff309a7000 r-xp 00000000 08:02 86 /lib64/libc-2.12.so Size: 1628 kB Rss: 1628 kB Pss: 598 kB Shared_Clean: 1096 kB Shared_Dirty: 0 kB Private_Clean: 532 kB Private_Dirty: 0 kB Referenced: 1628 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff309a7000-7eff30ba7000 ---p 00197000 08:02 86 /lib64/libc-2.12.so Size: 2048 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff30ba7000-7eff30bab000 r--p 00197000 08:02 86 /lib64/libc-2.12.so Size: 16 kB Rss: 16 kB Pss: 8 kB Shared_Clean: 0 kB Shared_Dirty: 16 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 16 kB Anonymous: 16 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff30bab000-7eff30bac000 rw-p 0019b000 08:02 86 /lib64/libc-2.12.so Size: 4 kB Rss: 4 kB Pss: 4 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 4 kB Referenced: 4 kB Anonymous: 4 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff30bac000-7eff30bb1000 rw-p 00000000 00:00 0 Size: 20 kB Rss: 20 kB Pss: 18 kB Shared_Clean: 0 kB Shared_Dirty: 4 kB Private_Clean: 0 kB Private_Dirty: 16 kB Referenced: 20 kB Anonymous: 20 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff30bb1000-7eff30bb9000 r-xp 00000000 08:02 29 /lib64/libpopt.so.0.0.0 Size: 32 kB Rss: 32 kB Pss: 29 kB Shared_Clean: 4 kB Shared_Dirty: 0 kB Private_Clean: 28 kB Private_Dirty: 0 kB Referenced: 32 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff30bb9000-7eff30db9000 ---p 00008000 08:02 29 /lib64/libpopt.so.0.0.0 Size: 2048 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff30db9000-7eff30dba000 rw-p 00008000 08:02 29 /lib64/libpopt.so.0.0.0 Size: 4 kB Rss: 4 kB Pss: 2 kB Shared_Clean: 0 kB Shared_Dirty: 4 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 4 kB Anonymous: 4 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff30dbc000-7eff30dbd000 rw-s 00000000 08:02 13887 /var/lib/ctdb/state/persistent_health.tdb.0 Size: 4 kB Rss: 4 kB Pss: 4 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 4 kB Private_Dirty: 0 kB Referenced: 4 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff30dbd000-7eff30dbe000 rw-p 00000000 00:00 0 Size: 4 kB Rss: 4 kB Pss: 2 kB Shared_Clean: 0 kB Shared_Dirty: 4 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 4 kB Anonymous: 4 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7fff32703000-7fff32718000 rw-p 00000000 00:00 0 [stack] Size: 88 kB Rss: 84 kB Pss: 50 kB Shared_Clean: 0 kB Shared_Dirty: 68 kB Private_Clean: 0 kB Private_Dirty: 16 kB Referenced: 84 kB Anonymous: 84 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7fff327ff000-7fff32800000 r-xp 00000000 00:00 0 [vdso] Size: 4 kB Rss: 4 kB Pss: 0 kB Shared_Clean: 4 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 4 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] Size: 4 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB total 0 lr-x------. 1 root root 64 Jun 22 09:43 0 -> /dev/null l-wx------. 1 root root 64 Jun 22 09:43 1 -> pipe:[1580347174] lrwx------. 1 root root 64 Jun 22 09:43 10 -> /var/lib/ctdb/state/persistent_health.tdb.0 lrwx------. 1 root root 64 Jun 22 09:43 11 -> socket:[1580385812] lrwx------. 1 root root 64 Jun 22 09:43 12 -> socket:[1597056131] lrwx------. 1 root root 64 Jun 22 09:43 13 -> /var/lib/ctdb/persistent/secrets.tdb.0 lrwx------. 1 root root 64 Jun 22 09:43 14 -> /var/lib/ctdb/persistent/share_info.tdb.0 lrwx------. 1 root root 64 Jun 22 09:43 15 -> /var/lib/ctdb/persistent/registry.tdb.0 lrwx------. 1 root root 64 Jun 22 09:43 16 -> /var/lib/ctdb/persistent/account_policy.tdb.0 lrwx------. 1 root root 64 Jun 22 09:43 17 -> /var/lib/ctdb/persistent/passdb.tdb.0 lrwx------. 1 root root 64 Jun 22 09:43 18 -> /var/lib/ctdb/persistent/group_mapping.tdb.0 lrwx------. 1 root root 64 Jun 22 09:43 19 -> /var/lib/ctdb/persistent/ctdb.tdb.0 l-wx------. 1 root root 64 Jun 22 09:43 2 -> pipe:[1580347174] lrwx------. 1 root root 64 Jun 22 09:43 20 -> socket:[1580400159] lrwx------. 1 root root 64 Jun 22 09:43 21 -> socket:[1597091707] lrwx------. 1 root root 64 Jun 22 09:43 23 -> socket:[1580450766] lrwx------. 1 root root 64 Jun 22 09:43 24 -> socket:[1580425862] lrwx------. 1 root root 64 Jun 22 09:43 25 -> socket:[1580425970] lrwx------. 1 root root 64 Jun 22 09:43 27 -> socket:[1580425874] lrwx------. 1 root root 64 Jun 22 09:43 28 -> socket:[1580425881] lrwx------. 1 root root 64 Jun 22 09:43 29 -> socket:[1580444029] lrwx------. 1 root root 64 Jun 22 09:43 3 -> anon_inode:[eventpoll] l-wx------. 1 root root 64 Jun 22 09:43 31 -> pipe:[1580450764] lrwx------. 1 root root 64 Jun 22 09:43 32 -> /var/lib/ctdb/printer_list.tdb.0 lrwx------. 1 root root 64 Jun 22 09:43 33 -> /var/lib/ctdb/locking.tdb.0 lrwx------. 1 root root 64 Jun 22 09:43 34 -> /var/lib/ctdb/brlock.tdb.0 lrwx------. 1 root root 64 Jun 22 09:43 35 -> /var/lib/ctdb/connections.tdb.0 lrwx------. 1 root root 64 Jun 22 09:43 36 -> /var/lib/ctdb/sessionid.tdb.0 lrwx------. 1 root root 64 Jun 22 09:43 37 -> /var/lib/ctdb/g_lock.tdb.0 lrwx------. 1 root root 64 Jun 22 09:43 38 -> /var/lib/ctdb/serverid.tdb.0 l-wx------. 1 root root 64 Jun 22 09:43 4 -> /var/log/log.ctdb lrwx------. 1 root root 64 Jun 22 09:43 5 -> socket:[1580347021] lrwx------. 1 root root 64 Jun 22 09:43 6 -> anon_inode:[eventpoll] lr-x------. 1 root root 64 Jun 22 09:43 7 -> pipe:[1580347169] l-wx------. 1 root root 64 Jun 22 09:43 8 -> pipe:[1580347169] lr-x------. 1 root root 64 Jun 22 09:43 9 -> pipe:[1580347174] Tue Jun 21 20:16:52 CST 2016 root 25775 0.2 0.0 26796 14464 ? SLs 20:06 0:01 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE root 26050 0.0 0.0 22080 5028 ? S 20:06 0:00 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE Tue Jun 21 20:16:53 CST 2016 root 25775 0.1 0.0 26796 14464 ? SLs 20:06 0:02 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE root 26050 0.0 0.0 22080 5028 ? S 20:06 0:00 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE Tue Jun 21 20:30:52 CST 2016 root 25775 0.1 0.0 26928 14468 ? SLs 20:06 0:02 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE root 26050 0.0 0.0 22080 5028 ? S 20:06 0:00 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE Tue Jun 21 20:30:53 CST 2016 continue memory leak, Wed Jun 22 09:39:42 CST 2016 root 25775 0.1 0.1 29768 17360 ? SLs Jun21 1:18 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE root 26050 0.0 0.0 22080 5028 ? S Jun21 0:15 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE Wed Jun 22 09:39:43 CST 2016 root 25775 0.1 0.1 29768 17364 ? SLs Jun21 1:18 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE root 26050 0.0 0.0 22080 5028 ? S Jun21 0:15 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE Wed Jun 22 09:39:44 CST 2016
An hour or so, again to collect information : [root@vx1600-cjf91 ~]# date Wed Jun 22 10:32:13 CST 2016 [root@vx1600-cjf91 ~]# [root@vx1600-cjf91 ~]# ps aux | grep ctdb root 18553 0.0 0.0 7956 872 pts/0 S+ 10:23 0:00 grep ctdb root 25775 0.1 0.1 29900 17524 ? SLs Jun21 1:22 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE root 26050 0.0 0.0 22080 5028 ? S Jun21 0:16 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE [root@vx1600-cjf91 ~]# cat /proc/25775/maps >> b.txt [root@vx1600-cjf91 ~]# cat /proc/25775/smaps >> b.txt [root@vx1600-cjf91 ~]# ls -l /proc/25775/fd >> b.txt [root@vx1600-cjf91 ~]# 00400000-004c5000 r-xp 00000000 08:02 3369 /usr/sbin/ctdbd 006c5000-006c7000 rw-p 000c5000 08:02 3369 /usr/sbin/ctdbd 00ef0000-00f11000 rw-p 00000000 00:00 0 00f11000-00f61000 rw-p 00000000 00:00 0 00f61000-01281000 rw-p 00000000 00:00 0 3547400000-3547420000 r-xp 00000000 08:02 11428 /lib64/ld-2.12.so 354761f000-3547620000 r--p 0001f000 08:02 11428 /lib64/ld-2.12.so 3547620000-3547621000 rw-p 00020000 08:02 11428 /lib64/ld-2.12.so 3547621000-3547622000 rw-p 00000000 00:00 0 7eff2f51b000-7eff2f65b000 rw-s 00000000 08:02 14406 /var/lib/ctdb/printer_list.tdb.0 7eff2f65b000-7eff2f781000 rw-s 00000000 08:02 14404 /var/lib/ctdb/brlock.tdb.0 7eff2f781000-7eff2f8a7000 rw-s 00000000 08:02 14403 /var/lib/ctdb/connections.tdb.0 7eff2f8a7000-7eff2f9cd000 rw-s 00000000 08:02 14401 /var/lib/ctdb/g_lock.tdb.0 7eff2f9cd000-7eff2faf3000 rw-s 00000000 08:02 14400 /var/lib/ctdb/serverid.tdb.0 7eff2faf3000-7eff2fc19000 rw-s 00000000 08:02 14402 /var/lib/ctdb/sessionid.tdb.0 7eff2fc19000-7eff2fd3f000 rw-s 00000000 08:02 14405 /var/lib/ctdb/locking.tdb.0 7eff2fd3f000-7eff2fe7f000 rw-s 00000000 08:02 14393 /var/lib/ctdb/persistent/ctdb.tdb.0 7eff2fe7f000-7eff2ffbf000 rw-s 00000000 08:02 14394 /var/lib/ctdb/persistent/group_mapping.tdb.0 7eff2ffbf000-7eff300ff000 rw-s 00000000 08:02 14399 /var/lib/ctdb/persistent/passdb.tdb.0 7eff300ff000-7eff3023f000 rw-s 00000000 08:02 14397 /var/lib/ctdb/persistent/account_policy.tdb.0 7eff3023f000-7eff3037f000 rw-s 00000000 08:02 14395 /var/lib/ctdb/persistent/registry.tdb.0 7eff3037f000-7eff304bf000 rw-s 00000000 08:02 14398 /var/lib/ctdb/persistent/share_info.tdb.0 7eff304bf000-7eff305ff000 rw-s 00000000 08:02 14396 /var/lib/ctdb/persistent/secrets.tdb.0 7eff305ff000-7eff3060b000 r-xp 00000000 08:02 124 /lib64/libnss_files-2.12.so 7eff3060b000-7eff3080b000 ---p 0000c000 08:02 124 /lib64/libnss_files-2.12.so 7eff3080b000-7eff3080c000 r--p 0000c000 08:02 124 /lib64/libnss_files-2.12.so 7eff3080c000-7eff3080d000 rw-p 0000d000 08:02 124 /lib64/libnss_files-2.12.so 7eff3080d000-7eff30810000 rw-p 00000000 00:00 0 7eff30810000-7eff309a7000 r-xp 00000000 08:02 86 /lib64/libc-2.12.so 7eff309a7000-7eff30ba7000 ---p 00197000 08:02 86 /lib64/libc-2.12.so 7eff30ba7000-7eff30bab000 r--p 00197000 08:02 86 /lib64/libc-2.12.so 7eff30bab000-7eff30bac000 rw-p 0019b000 08:02 86 /lib64/libc-2.12.so 7eff30bac000-7eff30bb1000 rw-p 00000000 00:00 0 7eff30bb1000-7eff30bb9000 r-xp 00000000 08:02 29 /lib64/libpopt.so.0.0.0 7eff30bb9000-7eff30db9000 ---p 00008000 08:02 29 /lib64/libpopt.so.0.0.0 7eff30db9000-7eff30dba000 rw-p 00008000 08:02 29 /lib64/libpopt.so.0.0.0 7eff30dbc000-7eff30dbd000 rw-s 00000000 08:02 13887 /var/lib/ctdb/state/persistent_health.tdb.0 7eff30dbd000-7eff30dbe000 rw-p 00000000 00:00 0 7fff32703000-7fff32718000 rw-p 00000000 00:00 0 [stack] 7fff327ff000-7fff32800000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] 00400000-004c5000 r-xp 00000000 08:02 3369 /usr/sbin/ctdbd Size: 788 kB Rss: 788 kB Pss: 636 kB Shared_Clean: 304 kB Shared_Dirty: 0 kB Private_Clean: 484 kB Private_Dirty: 0 kB Referenced: 788 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 006c5000-006c7000 rw-p 000c5000 08:02 3369 /usr/sbin/ctdbd Size: 8 kB Rss: 8 kB Pss: 8 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 8 kB Referenced: 8 kB Anonymous: 8 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 00ef0000-00f11000 rw-p 00000000 00:00 0 Size: 132 kB Rss: 132 kB Pss: 122 kB Shared_Clean: 0 kB Shared_Dirty: 20 kB Private_Clean: 0 kB Private_Dirty: 112 kB Referenced: 132 kB Anonymous: 132 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 00f11000-00f61000 rw-p 00000000 00:00 0 Size: 320 kB Rss: 320 kB Pss: 312 kB Shared_Clean: 0 kB Shared_Dirty: 16 kB Private_Clean: 0 kB Private_Dirty: 304 kB Referenced: 320 kB Anonymous: 320 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 00f61000-01281000 rw-p 00000000 00:00 0 Size: 3200 kB Rss: 3156 kB Pss: 3156 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 3156 kB Referenced: 3156 kB Anonymous: 3156 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 3547400000-3547420000 r-xp 00000000 08:02 11428 /lib64/ld-2.12.so Size: 128 kB Rss: 128 kB Pss: 14 kB Shared_Clean: 116 kB Shared_Dirty: 0 kB Private_Clean: 12 kB Private_Dirty: 0 kB Referenced: 128 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 354761f000-3547620000 r--p 0001f000 08:02 11428 /lib64/ld-2.12.so Size: 4 kB Rss: 4 kB Pss: 2 kB Shared_Clean: 0 kB Shared_Dirty: 4 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 4 kB Anonymous: 4 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 3547620000-3547621000 rw-p 00020000 08:02 11428 /lib64/ld-2.12.so Size: 4 kB Rss: 4 kB Pss: 4 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 4 kB Referenced: 4 kB Anonymous: 4 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 3547621000-3547622000 rw-p 00000000 00:00 0 Size: 4 kB Rss: 4 kB Pss: 2 kB Shared_Clean: 0 kB Shared_Dirty: 4 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 4 kB Anonymous: 4 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff2f51b000-7eff2f65b000 rw-s 00000000 08:02 14406 /var/lib/ctdb/printer_list.tdb.0 Size: 1280 kB Rss: 796 kB Pss: 796 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 796 kB Private_Dirty: 0 kB Referenced: 796 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff2f65b000-7eff2f781000 rw-s 00000000 08:02 14404 /var/lib/ctdb/brlock.tdb.0 Size: 1176 kB Rss: 784 kB Pss: 784 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 784 kB Private_Dirty: 0 kB Referenced: 784 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff2f781000-7eff2f8a7000 rw-s 00000000 08:02 14403 /var/lib/ctdb/connections.tdb.0 Size: 1176 kB Rss: 784 kB Pss: 784 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 784 kB Private_Dirty: 0 kB Referenced: 784 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff2f8a7000-7eff2f9cd000 rw-s 00000000 08:02 14401 /var/lib/ctdb/g_lock.tdb.0 Size: 1176 kB Rss: 784 kB Pss: 784 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 784 kB Private_Dirty: 0 kB Referenced: 784 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff2f9cd000-7eff2faf3000 rw-s 00000000 08:02 14400 /var/lib/ctdb/serverid.tdb.0 Size: 1176 kB Rss: 784 kB Pss: 784 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 784 kB Private_Dirty: 0 kB Referenced: 784 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff2faf3000-7eff2fc19000 rw-s 00000000 08:02 14402 /var/lib/ctdb/sessionid.tdb.0 Size: 1176 kB Rss: 784 kB Pss: 784 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 784 kB Private_Dirty: 0 kB Referenced: 784 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff2fc19000-7eff2fd3f000 rw-s 00000000 08:02 14405 /var/lib/ctdb/locking.tdb.0 Size: 1176 kB Rss: 784 kB Pss: 784 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 784 kB Private_Dirty: 0 kB Referenced: 784 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff2fd3f000-7eff2fe7f000 rw-s 00000000 08:02 14393 /var/lib/ctdb/persistent/ctdb.tdb.0 Size: 1280 kB Rss: 796 kB Pss: 796 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 796 kB Private_Dirty: 0 kB Referenced: 796 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff2fe7f000-7eff2ffbf000 rw-s 00000000 08:02 14394 /var/lib/ctdb/persistent/group_mapping.tdb.0 Size: 1280 kB Rss: 796 kB Pss: 796 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 796 kB Private_Dirty: 0 kB Referenced: 796 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff2ffbf000-7eff300ff000 rw-s 00000000 08:02 14399 /var/lib/ctdb/persistent/passdb.tdb.0 Size: 1280 kB Rss: 796 kB Pss: 796 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 796 kB Private_Dirty: 0 kB Referenced: 796 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff300ff000-7eff3023f000 rw-s 00000000 08:02 14397 /var/lib/ctdb/persistent/account_policy.tdb.0 Size: 1280 kB Rss: 796 kB Pss: 796 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 796 kB Private_Dirty: 0 kB Referenced: 796 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff3023f000-7eff3037f000 rw-s 00000000 08:02 14395 /var/lib/ctdb/persistent/registry.tdb.0 Size: 1280 kB Rss: 836 kB Pss: 836 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 836 kB Private_Dirty: 0 kB Referenced: 836 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff3037f000-7eff304bf000 rw-s 00000000 08:02 14398 /var/lib/ctdb/persistent/share_info.tdb.0 Size: 1280 kB Rss: 796 kB Pss: 796 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 796 kB Private_Dirty: 0 kB Referenced: 796 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff304bf000-7eff305ff000 rw-s 00000000 08:02 14396 /var/lib/ctdb/persistent/secrets.tdb.0 Size: 1280 kB Rss: 796 kB Pss: 796 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 796 kB Private_Dirty: 0 kB Referenced: 796 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff305ff000-7eff3060b000 r-xp 00000000 08:02 124 /lib64/libnss_files-2.12.so Size: 48 kB Rss: 48 kB Pss: 11 kB Shared_Clean: 40 kB Shared_Dirty: 0 kB Private_Clean: 8 kB Private_Dirty: 0 kB Referenced: 48 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff3060b000-7eff3080b000 ---p 0000c000 08:02 124 /lib64/libnss_files-2.12.so Size: 2048 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff3080b000-7eff3080c000 r--p 0000c000 08:02 124 /lib64/libnss_files-2.12.so Size: 4 kB Rss: 4 kB Pss: 2 kB Shared_Clean: 0 kB Shared_Dirty: 4 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 4 kB Anonymous: 4 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff3080c000-7eff3080d000 rw-p 0000d000 08:02 124 /lib64/libnss_files-2.12.so Size: 4 kB Rss: 4 kB Pss: 2 kB Shared_Clean: 0 kB Shared_Dirty: 4 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 4 kB Anonymous: 4 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff3080d000-7eff30810000 rw-p 00000000 00:00 0 Size: 12 kB Rss: 12 kB Pss: 8 kB Shared_Clean: 0 kB Shared_Dirty: 8 kB Private_Clean: 0 kB Private_Dirty: 4 kB Referenced: 12 kB Anonymous: 12 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff30810000-7eff309a7000 r-xp 00000000 08:02 86 /lib64/libc-2.12.so Size: 1628 kB Rss: 1628 kB Pss: 595 kB Shared_Clean: 1100 kB Shared_Dirty: 0 kB Private_Clean: 528 kB Private_Dirty: 0 kB Referenced: 1628 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff309a7000-7eff30ba7000 ---p 00197000 08:02 86 /lib64/libc-2.12.so Size: 2048 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff30ba7000-7eff30bab000 r--p 00197000 08:02 86 /lib64/libc-2.12.so Size: 16 kB Rss: 16 kB Pss: 8 kB Shared_Clean: 0 kB Shared_Dirty: 16 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 16 kB Anonymous: 16 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff30bab000-7eff30bac000 rw-p 0019b000 08:02 86 /lib64/libc-2.12.so Size: 4 kB Rss: 4 kB Pss: 4 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 4 kB Referenced: 4 kB Anonymous: 4 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff30bac000-7eff30bb1000 rw-p 00000000 00:00 0 Size: 20 kB Rss: 20 kB Pss: 18 kB Shared_Clean: 0 kB Shared_Dirty: 4 kB Private_Clean: 0 kB Private_Dirty: 16 kB Referenced: 20 kB Anonymous: 20 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff30bb1000-7eff30bb9000 r-xp 00000000 08:02 29 /lib64/libpopt.so.0.0.0 Size: 32 kB Rss: 32 kB Pss: 29 kB Shared_Clean: 4 kB Shared_Dirty: 0 kB Private_Clean: 28 kB Private_Dirty: 0 kB Referenced: 32 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff30bb9000-7eff30db9000 ---p 00008000 08:02 29 /lib64/libpopt.so.0.0.0 Size: 2048 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff30db9000-7eff30dba000 rw-p 00008000 08:02 29 /lib64/libpopt.so.0.0.0 Size: 4 kB Rss: 4 kB Pss: 2 kB Shared_Clean: 0 kB Shared_Dirty: 4 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 4 kB Anonymous: 4 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff30dbc000-7eff30dbd000 rw-s 00000000 08:02 13887 /var/lib/ctdb/state/persistent_health.tdb.0 Size: 4 kB Rss: 4 kB Pss: 4 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 4 kB Private_Dirty: 0 kB Referenced: 4 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7eff30dbd000-7eff30dbe000 rw-p 00000000 00:00 0 Size: 4 kB Rss: 4 kB Pss: 2 kB Shared_Clean: 0 kB Shared_Dirty: 4 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 4 kB Anonymous: 4 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7fff32703000-7fff32718000 rw-p 00000000 00:00 0 [stack] Size: 88 kB Rss: 84 kB Pss: 50 kB Shared_Clean: 0 kB Shared_Dirty: 68 kB Private_Clean: 0 kB Private_Dirty: 16 kB Referenced: 84 kB Anonymous: 84 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB 7fff327ff000-7fff32800000 r-xp 00000000 00:00 0 [vdso] Size: 4 kB Rss: 4 kB Pss: 0 kB Shared_Clean: 4 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 4 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] Size: 4 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB total 0 lr-x------. 1 root root 64 Jun 22 09:43 0 -> /dev/null l-wx------. 1 root root 64 Jun 22 09:43 1 -> pipe:[1580347174] lrwx------. 1 root root 64 Jun 22 09:43 10 -> /var/lib/ctdb/state/persistent_health.tdb.0 lrwx------. 1 root root 64 Jun 22 09:43 11 -> socket:[1580385812] lrwx------. 1 root root 64 Jun 22 09:43 12 -> socket:[1597056131] lrwx------. 1 root root 64 Jun 22 09:43 13 -> /var/lib/ctdb/persistent/secrets.tdb.0 lrwx------. 1 root root 64 Jun 22 09:43 14 -> /var/lib/ctdb/persistent/share_info.tdb.0 lrwx------. 1 root root 64 Jun 22 09:43 15 -> /var/lib/ctdb/persistent/registry.tdb.0 lrwx------. 1 root root 64 Jun 22 09:43 16 -> /var/lib/ctdb/persistent/account_policy.tdb.0 lrwx------. 1 root root 64 Jun 22 09:43 17 -> /var/lib/ctdb/persistent/passdb.tdb.0 lrwx------. 1 root root 64 Jun 22 09:43 18 -> /var/lib/ctdb/persistent/group_mapping.tdb.0 lrwx------. 1 root root 64 Jun 22 09:43 19 -> /var/lib/ctdb/persistent/ctdb.tdb.0 l-wx------. 1 root root 64 Jun 22 09:43 2 -> pipe:[1580347174] lrwx------. 1 root root 64 Jun 22 09:43 20 -> socket:[1580400159] lrwx------. 1 root root 64 Jun 22 09:43 21 -> socket:[1597091707] lrwx------. 1 root root 64 Jun 22 10:24 22 -> socket:[486250748] lrwx------. 1 root root 64 Jun 22 09:43 23 -> socket:[1580450766] lrwx------. 1 root root 64 Jun 22 09:43 24 -> socket:[1580425862] lrwx------. 1 root root 64 Jun 22 09:43 25 -> socket:[1580425970] lrwx------. 1 root root 64 Jun 22 09:43 27 -> socket:[1580425874] lrwx------. 1 root root 64 Jun 22 09:43 28 -> socket:[1580425881] lrwx------. 1 root root 64 Jun 22 09:43 29 -> socket:[1580444029] lrwx------. 1 root root 64 Jun 22 09:43 3 -> anon_inode:[eventpoll] l-wx------. 1 root root 64 Jun 22 09:43 31 -> pipe:[1580450764] lrwx------. 1 root root 64 Jun 22 09:43 32 -> /var/lib/ctdb/printer_list.tdb.0 lrwx------. 1 root root 64 Jun 22 09:43 33 -> /var/lib/ctdb/locking.tdb.0 lrwx------. 1 root root 64 Jun 22 09:43 34 -> /var/lib/ctdb/brlock.tdb.0 lrwx------. 1 root root 64 Jun 22 09:43 35 -> /var/lib/ctdb/connections.tdb.0 lrwx------. 1 root root 64 Jun 22 09:43 36 -> /var/lib/ctdb/sessionid.tdb.0 lrwx------. 1 root root 64 Jun 22 09:43 37 -> /var/lib/ctdb/g_lock.tdb.0 lrwx------. 1 root root 64 Jun 22 09:43 38 -> /var/lib/ctdb/serverid.tdb.0 l-wx------. 1 root root 64 Jun 22 09:43 4 -> /var/log/log.ctdb lrwx------. 1 root root 64 Jun 22 09:43 5 -> socket:[1580347021] lrwx------. 1 root root 64 Jun 22 09:43 6 -> anon_inode:[eventpoll] lr-x------. 1 root root 64 Jun 22 09:43 7 -> pipe:[1580347169] l-wx------. 1 root root 64 Jun 22 09:43 8 -> pipe:[1580347169] lr-x------. 1 root root 64 Jun 22 09:43 9 -> pipe:[1580347174]
Please do not post such large output in the comments. Attach the output, so it's easier to read. Here CTDB is consuming around 29MB and most of it is related to the databases. Where do you see CTDB is consuming 700MB? The column you are referring to is showing resident memory (RSS). That will increase and decrease in the lifetime of CTDB depending on the activity. Similarly virtual size (VSZ). Are you seeing the memory increase in RSS if you don't run Samba?
In our production system, running more than 100 days, the memory increased to 700M, the impact of the system operation, the production system has been restarted ctdb; now analyze the problem, found that every day in the leak; root 14179 0.1 0.6 656284 39272 ? Ssl Jan12 367:09 /usr/local/hstor/sms_install/ucsfs/sbin/ucsfs --volfile-server=172.16.53.56 --volfile-id=/cluster_lock_volume /root/lock root 14219 0.5 11.5 745064 713884 ? SLs Jan12 1148:44 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE Began to suspect that the impact of samba, after closing the samba, ctdb is still continuing to increase the RES memory； In our application, each node has a periodic call to ctdb status, ctdb ip query ctdb node information
Hi all update ctdb lib source,talloc to version 2.1.7,tdb to version 1.2.13,tevent to 0.9.28, recompile ctdb ,There are still memory leaks. Who can give advice, how to analyze and solve the problem； Using Valgrind detection, too many alarms; using fortify tools to check, also did not find valuable information
(In reply to James from comment #7) We first need to establish that there is a memory leak. Increase in the process memory does not necessarily translate into memory leak. Can you provide following details before and after (memory increase)? 1. /proc/<pid>/smaps 2. ctdb dumpmemory Please attach the output as files. Also, note that CTDB 2.5.x is out of support. There will not be any new releases. Please upgrade to newer version of CTDB.
Hi all ctdb info: cat /proc/16798/smaps >> smaps ctdb dumpmemory >> dumpmemory.txt Fri Jun 24 16:06:48 CST 2016 root 16798 0.3 0.1 27944 15592 ? SLs 10:55 1:09 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE root 17437 0.0 0.0 22088 5020 ? S 10:55 0:06 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE Fri Jun 24 16:06:49 CST 2016 root 16798 0.3 0.1 27944 15592 ? SLs 10:55 1:09 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE root 17437 0.0 0.0 22088 5020 ? S 10:55 0:06 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE Fri Jun 24 16:06:50 CST 2016 root 16798 0.3 0.1 27944 15592 ? SLs 10:55 1:09 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE root 17437 0.0 0.0 22088 5020 ? S 10:55 0:06 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ Wed Jun 29 21:04:42 CST 2016 root 16798 0.3 0.2 55720 43352 ? SLs Jun24 30:51 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE root 17437 0.0 0.0 22088 5020 ? S Jun24 2:36 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE Wed Jun 29 21:04:43 CST 2016 root 16798 0.3 0.2 55720 43352 ? SLs Jun24 30:51 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE root 17437 0.0 0.0 22088 5020 ? S Jun24 2:36 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE ctdb running 5 days, ctdb memory increase from 15592KB to 43352 KB
Created attachment 12234 [details] smaps
Created attachment 12235 [details] dumpmemory Wed Jun 29 21:18:51
Created attachment 12236 [details] ps ctdb per second
Hi, ctdb-2.5.6.tar.gz Is the final version,Is there an updated version？ I install ctdb-2.5.6,Is also the same leakage； Memory should be allocated, the use of complete, will be released, should not be a single direction has been growing;
(In reply to James from comment #9) Can you please provide the complete information? I need two sets of outputs to compare the memory utilization. Do you have the smaps output from Jun 24th? If not, you can wait a day or two and provide smaps information for the same process.
(In reply to James from comment #13) There will no more CTDB 2.5.x releases. You can switch to the latest stable release 4.4.x. CTDB is now part of Samba and you can download the latest Samba release 4.4.x from samba.org.
Created attachment 12243 [details] ctdb dumpmemory Thu Jun 30 09:18:34 CST 2016
Created attachment 12244 [details] cat /proc/16798/smaps Thu Jun 30 09:23:45 CST 2016
HI, Above is the information that ctdb runs for 11 hours. Wed Jun 29 21:04:42 CST 2016 root 16798 0.3 0.2 55720 43352 ? SLs Jun24 30:51 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE [root@vx1600-cjf50 ~]# date Thu Jun 30 09:38:47 CST 2016 [root@vx1600-cjf50 ~]# [root@vx1600-cjf50 ~]# [root@vx1600-cjf50 ~]# ps aux | grep ctdb root 2967 0.0 0.0 7956 844 pts/0 S+ 09:38 0:00 grep ctdb root 16798 0.3 0.3 58292 46052 ? SLs Jun24 34:10 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE root 17437 0.0 0.0 22088 5020 ? S Jun24 2:52 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE [root@vx1600-cjf50 ~]# ctdb memory from 43352KB to 46052 KB
Hi I download samba-4.4.4,compile and install ctdb ; ctdb status Number of nodes:5 pnn:0 192.168.77.91 OK pnn:1 192.168.77.50 OK (THIS NODE) pnn:2 192.168.77.35 OK pnn:3 192.168.77.26 OK pnn:4 192.168.77.60 OK Generation:222815519 Size:5 hash:0 lmaster:0 hash:1 lmaster:1 hash:2 lmaster:2 hash:3 lmaster:3 hash:4 lmaster:4 Recovery mode:NORMAL (0) Recovery master:4 ctdb version CTDB version: 4.4.4 the 5 node ctdb cluster running 5day ; every node ctdb memory increase; The phenomenon is as follows : 26 node : Thu Jun 30 19:51:44 26 node memory is 32244 KB Sun Jul 3 04:25:42 26 node memory is 34292 KB between Thu Jun 30 19:51:44 and Sun Jul 3 04:25:42 ctdb memory no change; start at this time Sun Jul 3 21:12:40 CST 2016 ctdb memory increase 4KB per minute ! 26 node info : Thu Jun 30 19:51:44 CST 2016 root 28510 0.1 0.2 48052 32244 ? SLs 18:13 0:09 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE root 28779 0.0 0.0 22088 5032 ? S 18:13 0:01 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE Thu Jun 30 19:51:45 CST 2016 Sun Jul 3 04:25:41 CST 2016 root 28510 0.1 0.2 48052 32244 ? SLs Jun30 5:22 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE root 28779 0.0 0.0 22088 5032 ? S Jun30 0:39 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE Sun Jul 3 04:25:42 CST 2016 root 28510 0.1 0.2 48052 34292 ? SLs Jun30 5:22 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE root 28779 0.0 0.0 22088 5032 ? S Jun30 0:39 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE Sun Jul 3 04:25:43 CST 2016 root 28510 0.1 0.2 48052 34292 ? SLs Jun30 6:58 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE root 28779 0.0 0.0 22088 5032 ? S Jun30 0:50 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE Sun Jul 3 21:12:40 CST 2016 root 28510 0.1 0.2 48052 34296 ? SLs Jun30 6:58 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE root 28779 0.0 0.0 22088 5032 ? root 28510 0.1 0.2 48052 34292 ? SLs Jun30 6:58 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE root 28779 0.0 0.0 22088 5032 ? S Jun30 0:50 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE Sun Jul 3 21:12:40 CST 2016 root 28510 0.1 0.2 48052 34296 ? SLs Jun30 6:58 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE root 28779 0.0 0.0 22088 5032 ? S Jun30 0:50 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE 91 node : ctdb memory Has been increasing ! from 16992KB to 38592KB Thu Jun 30 22:27:22 CST 2016 root 8220 0.1 0.1 30296 16992 ? SLs 18:13 0:24 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE root 8493 0.0 0.0 22088 5032 ? S 18:13 0:03 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE Thu Jun 30 22:27:23 CST 2016 Mon Jul 4 14:23:15 CST 2016 root 8220 0.1 0.2 51828 38592 ? SLs Jun30 8:16 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE root 8493 0.0 0.0 22088 5032 ? S Jun30 1:25 /usr/sbin/ctdbd --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE
(In reply to James from comment #19) CTDB process memory will increase depending on the Samba activity and the database sizes. Do you notice CTDB memory increasing when there is no Samba activity? > 91 node : > ctdb memory Has been increasing ! from 16992KB to 38592KB This is possible if the sizes of databases have grown. Please provide smaps output (before and after) whenever reporting memory increase in the ctdbd process. Also include output of "ctdb statistics" command.
Created attachment 12260 [details] ctdb info from 07-04-1444 to 07-05-0926
Hi, I'm sure it's closed down Samba,no smb ,no nmb,no winbind running ; [root@vx1600-cjf91 ~]# ps aux | grep smb root 8380 0.0 0.0 7956 860 pts/2 S+ 10:28 0:00 grep smb [root@vx1600-cjf91 ~]# ps aux | grep nmb root 8647 0.0 0.0 7956 860 pts/2 S+ 10:28 0:00 grep nmb [root@vx1600-cjf91 ~]# ps aux | grep winbind root 9592 0.0 0.0 7956 868 pts/2 S+ 10:28 0:00 grep winbind [root@vx1600-cjf91 ~]# I set ctdb setdebug 3 ,only see eventscript monitor and tmp tdb Vacuuming in operation；Whether the Tmp tdb Vacuuming operation may have a memory leak. now ,ctdb memory increase to 43560 KB. Collected information on the above。
Hi all In the memory leak, I opened the debug switch，mode 4, found that there are the following log, data memory structure abnormalities； 2016/07/11 16:40:40.519305 [22853]: Ending timer event 0x27f6910 "queue_next_trigger" 2016/07/11 16:40:40.521533 [22853]: server/ctdb_daemon.c:927 Created SOCKET FD:13 to connected child 2016/07/11 16:40:40.521575 [22853]: server/ctdb_daemon.c:889 client request 2147483448 of type 7 length 64 from node 0 to 4026531841 2016/07/11 16:40:40.521591 [22853]: Running timer event 0x27d6f00 "queue_next_trigger" 2016/07/11 16:40:40.521597 [22853]: server/ctdb_server.c:292 ctdb request 13157570 of type 7 length 64 from node 1 to 1 node num 4026531841 is too big ; Problem may be the reason, how to continue to analyze the abnormal location;
Hi Amitay, I use valgind tools to check memory, commands are as follows valgrind --leak-check=full --log-file=/var/log/log1 --show-reachable=yes /usr/sbin/ctdbd --nosetsched --reclock=/root/lock/lockfile --pidfile=/var/run/ctdb/ctdbd.pid --nlist=/etc/ctdb/nodes --public-addresses=/etc/ctdb/public_addresses -d NOTICE Found a memory leak location, can help confirm it. ==31999== 89,216 bytes in 2,720 blocks are definitely lost in loss record 112 of 112 ==31999== at 0x4A05FDE: malloc (vg_replace_malloc.c:236) ==31999== by 0x4EBFDCF: scandir (in /lib64/libc-2.12.so) ==31999== by 0x4375A0: ctdb_get_script_list (eventscript.c:168) ==31999== by 0x439385: ctdb_event_script_callback_v (eventscript.c:768) ==31999== by 0x4396AF: ctdb_event_script_callback (eventscript.c:824) ==31999== by 0x42299E: ctdb_check_health (ctdb_monitor.c:383) ==31999== by 0x48507D: tevent_common_loop_timer_delay (tevent_timed.c:343) ==31999== by 0x48BF4A: epoll_event_loop (tevent_epoll.c:667) ==31999== by 0x48C7CA: epoll_event_loop_once (tevent_epoll.c:931) ==31999== by 0x487D9A: std_event_loop_once (tevent_standard.c:112) ==31999== by 0x483AC0: _tevent_loop_once (tevent.c:530) ==31999== by 0x483D38: tevent_common_loop_wait (tevent.c:634) ==31999== ==31999== LEAK SUMMARY: ==31999== definitely lost: 101,024 bytes in 3,080 blocks ==31999== indirectly lost: 0 bytes in 0 blocks ==31999== possibly lost: 197,463 bytes in 136 blocks ==31999== still reachable: 46,681 bytes in 24 blocks ==31999== suppressed: 0 bytes in 0 blocks Does this scandir function use error，in linux platform
This looks like a memory leak that was fixed in 2015 with the commit: ---------------------------------------------------------------------------- commit c1e8bfb186c5cbeafbce9f2767db82edb579d5e1 Author: Volker Lendecke <vl@samba.org> Date: Fri Mar 13 14:12:41 2015 +0000 ctdb: Fix memleak in ctdb_get_script_list scandir allocates every name individually, see example code in susv4 or man scandir Signed-off-by: Volker Lendecke <vl@samba.org> Reviewed-by: Michael Adam <obnox@samba.org> diff --git a/ctdb/server/eventscript.c b/ctdb/server/eventscript.c index 4cbb846..a47396a 100644 --- a/ctdb/server/eventscript.c +++ b/ctdb/server/eventscript.c @@ -177,8 +177,7 @@ static struct ctdb_scripts_wire *ctdb_get_script_list(struct ctdb_context *ctdb, + sizeof(scripts->scripts[0]) * count); if (scripts == NULL) { DEBUG(DEBUG_ERR, (__location__ " Failed to allocate scripts\n")); - free(namelist); - return NULL; + goto done; } scripts->num_scripts = count; @@ -191,6 +190,10 @@ static struct ctdb_scripts_wire *ctdb_get_script_list(struct ctdb_context *ctdb, } } +done: + for (i=0; i<count; i++) { + free(namelist[i]); + } free(namelist); return scripts; } ---------------------------------------------------------------------------- Can you confirm your ctdb code has this fix ?
Good catch, James. Volker's patch is not present in CTDB 2.5.x. That explains the memory leak. However, this patch is present in CTDB 4.4.4. So there should not be any memory leak in ctdb_get_script_list().
Hi James, Were you ever able to show a memory leak with valgrind in a 4.x release? If not then we might close this, since it won't be fixed in 2.5.x and we believe it is fixed in 4.x. Thanks...
No update for a year about whether this is reproducible in 4.x. Assuming this is fixed in 4.x. Closing...