nsd schrijft /tmp vol

[huub]

/tmp raakt vol door groot aantal nsd-xfr files bij test-nodes (SOM2, CeTIM2, etc).

[huub]

nsd is ook gestopt en start niet meer op, op vosko2 (met volle /tmp):
Vosko2# service nsd restart
nsd not running? (check /var/run/nsd/nsd.pid).
Starting nsd.
[2018-12-18 15:48:37.007] nsd[94987]: notice: nsd starting (NSD 4.1.25)
[2018-12-18 15:48:37.009] nsd[94987]: error: ...stale pid file from process 94969

[huub]

/tmp ziet er zo uit:
Vosko2# ls
.ICE-unix nsd-xfr-66294
.X11-unix nsd-xfr-66329
.XIM-unix nsd-xfr-66374
.font-unix nsd-xfr-66398
.snap nsd-xfr-66429
authenticated.txt nsd-xfr-66906
autologin.txt nsd-xfr-67094
db.16.172.in-addr.arpa nsd-xfr-67247
db.17.172.in-addr.arpa nsd-xfr-67258
db.18.172.in-addr.arpa nsd-xfr-67288
db.19.172.in-addr.arpa nsd-xfr-67344
db.20.172.in-addr.arpa nsd-xfr-67470
db.21.172.in-addr.arpa nsd-xfr-67489
db.22.172.in-addr.arpa nsd-xfr-67566
db.23.172.in-addr.arpa nsd-xfr-67630
db.24.172.in-addr.arpa nsd-xfr-67982
db.25.172.in-addr.arpa nsd-xfr-6809
db.26.172.in-addr.arpa nsd-xfr-681
db.27.172.in-addr.arpa nsd-xfr-68154
db.28.172.in-addr.arpa nsd-xfr-68201
db.29.172.in-addr.arpa nsd-xfr-68257
db.30.172.in-addr.arpa nsd-xfr-68267
db.31.172.in-addr.arpa nsd-xfr-68291
db.wleiden.net nsd-xfr-68330
dns.tar.gz.enc nsd-xfr-68383
lvrouted.log nsd-xfr-68396
lvrouted.mytree nsd-xfr-68414
lvrouted.packet-172.16.3.153 nsd-xfr-68656
lvrouted.packet-172.16.4.124 nsd-xfr-68700
lvrouted.packet-172.16.4.140 nsd-xfr-68884
lvrouted.packet-172.16.4.36 nsd-xfr-68894
lvrouted.packet-172.16.4.84 nsd-xfr-68904
lvrouted.packet-172.16.4.9 nsd-xfr-68936
lvrouted.packet-172.17.175.71 nsd-xfr-69056
lvrouted.packet-172.17.175.73 nsd-xfr-69082
lvrouted.tree-172.16.4.124 nsd-xfr-69138
lvrouted.tree-172.16.4.140 nsd-xfr-69157
lvrouted.tree-172.16.4.36 nsd-xfr-69236
lvrouted.tree-172.16.4.84 nsd-xfr-69427
lvrouted.tree-172.16.4.9 nsd-xfr-69480
lvrouted.tree-172.17.175.71 nsd-xfr-69537
lvrouted.tree-172.17.175.73 nsd-xfr-69621
network.status nsd-xfr-69655
nsd-xfr-1250 nsd-xfr-69695
nsd-xfr-12529 nsd-xfr-69753
nsd-xfr-1260 nsd-xfr-6979
nsd-xfr-1270 nsd-xfr-69866
nsd-xfr-1305 nsd-xfr-69964
nsd-xfr-1373 nsd-xfr-70014
nsd-xfr-1439 nsd-xfr-70042
nsd-xfr-1465 nsd-xfr-70208
nsd-xfr-14759 nsd-xfr-70346
nsd-xfr-1516 nsd-xfr-70384
etc.

en bijv.
Vosko2# ls -alh nsd-xfr-658
total 44
drwx------ 2 nsd wheel 512B Dec 16 03:10 .
drwxrwxrwt 773 root wheel 23K Dec 18 15:38 ..
-rw------- 1 nsd wheel 288B Dec 16 03:10 nsd.658.task.0
-rw------- 1 nsd wheel 288B Dec 16 03:10 nsd.658.task.1
-rw------- 1 nsd wheel 1.2K Dec 16 03:10 nsd.658.zstat.0
-rw------- 1 nsd wheel 1.2K Dec 16 03:10 nsd.658.zstat.1

[huub]

Vosko2# nsd-checkconf -v /etc/local/nsd/nsd.conf
# Read file /etc/local/nsd/nsd.conf: 17 patterns, 17 fixed-zones, 0 keys.
# Config settings.
server:

debug-mode: no
ip-transparent: no
ip-freebind: no
reuseport: no
do-ip4: yes
do-ip6: yes
hide-version: no
database: ""
#identity:
#version:
#nsid:
#logfile:
server-count: 1
tcp-count: 100
tcp-query-count: 0
tcp-timeout: 120
tcp-mss: 0
outgoing-tcp-mss: 0
ipv4-edns-size: 4096
ipv6-edns-size: 4096
pidfile: "/var/run/nsd/nsd.pid"
port: "5353"
statistics: 0
chroot: ""
username: "nsd"
zonesdir: "/usr/local/etc/nsd"
xfrdfile: "/var/db/nsd/xfrd.state"
zonelistfile: "/var/db/nsd/zone.list"
xfrdir: "/tmp"
xfrd-reload-timeout: 1
log-time-ascii: yes
round-robin: no
minimal-responses: no
refuse-any: no
verbosity: 0
ip-address: "127.0.0.1"
rrl-size: 1000000
rrl-ratelimit: 200
rrl-slip: 2
rrl-ipv4-prefix-length: 24
rrl-ipv6-prefix-length: 64
rrl-whitelist-ratelimit: 2000
zonefiles-check: yes
zonefiles-write: 3600

[rick]

Als ik kijk in de nsd ​broncode dan is de /tmp/nsd-xfr-<pid> directory een normaal verschijnsel. Met zeer beperkte grootte (20kB). Tijdens het stoppen van nsd word deze tijdelijke directory netjes verwijderd. Het feit dat er vele instanties van deze directory bestaan, betekend dat het process crashed en/of niet netjes wordt herstart. De reden hiervan is mij niet nog bekend.

[rick]

Bingo, nsd gebruikt teveel geheugen:

/tmp: optimization changed from SPACE to TIME
pid 77199 (nsd), uid 216, was killed: out of swap space
pid 97290 (nsd), uid 216, was killed: out of swap space
pid 97302 (nsd), uid 216, was killed: out of swap space
pid 97278 (xz), uid 0, was killed: out of swap space
pid 97311 (nsd), uid 216, was killed: out of swap space
pid 5167 (nsd), uid 216, was killed: out of swap space
pid 5293 (nsd), uid 216, was killed: out of swap space
pid 5524 (nsd), uid 216, was killed: out of swap space
pid 6179 (nsd), uid 216, was killed: out of swap space
pid 6293 (nsd), uid 216, was killed: out of swap space
pid 6603 (nsd), uid 216, was killed: out of swap space
pid 7001 (nsd), uid 216, was killed: out of swap space
pid 7076 (nsd), uid 216, was killed: out of swap space
pid 7351 (nsd), uid 216, was killed: out of swap space
pid 7676 (nsd), uid 216, was killed: out of swap space
pid 7892 (nsd), uid 216, was killed: out of swap space
pid 8096 (nsd), uid 216, was killed: out of swap space
pid 8221 (nsd), uid 216, was killed: out of swap space

nu nog een oplossing vinden ...

[rick]

Some suggested options by the NSD mailinglist:

--disable-radix-tree
--enable-packed

how-ever they need to be included during compile time...

[rick]

In 14277/hybrid:

Add options for reduced nsd memory footprint

Related to issue nodefactory#245

[rick]

The options does not seems to reduce the memory footprint by some 15 percent how-ever the base footprint is still rather large, in the order of 100MB. I will try to evaluate alternatives ​YADIFA and ​gdnsd.

[huub]

ik heb een nieuwe image gebouwd. nsd gebruikt nu in totaal 117 MB, dat was 155 MB. Dus een kwart minder.

[huub]

Software op alle 11.2 nodes geinstalleerd (dat zijn er inmiddels 19, binnenstad Leiden); voorlopig dwz 14 uur) geen swap space problemen met nsd meer.

[huub]

Probleem lijkt opgelost.