Context Navigation

← Previous Ticket
Next Ticket →

#149 closed incident (fixed)

lvrouted crashed on CNodeIMI (/var full)

Reported by:	rick	Owned by:	rick
Keywords:		Cc:
Location:	Generiek

Description

CNodeIMI# tail -10 /tmp/lvrouted.log
1332813702[1]: Unhandled exception in main loop
1332813707[1]: Unhandled exception in main loop
1332813734[1]: Unhandled exception in main loop
1332813735[1]: Unhandled exception in main loop
1332813744[1]: Unhandled exception in main loop
1332813771[1]: Unhandled exception in main loop
1332813772[1]: Unhandled exception in main loop
1332813781[1]: Unhandled exception in main loop
1332813806[1]: Unhandled exception in main loop
1332813815[1]: Unhandled exception in main loop

 $ cd /tmp 
 $ ktrace -p 5000
 $ ktrace -c -p 5000
 $ kdump

CNodeIMI# kdump 
  5000 lvrouted RET   select 1
  5000 lvrouted CALL  recvfrom(0x3,0xbfbfa994,0x4000,0,0xbfbfa924,0xbfbfa920)
  5000 lvrouted GIO   fd 3 read 1596 bytes
       0x0000 7108 41fe 998f 37cf 7710 0f07 1dbf e2dc 42c6 e645 886b 71cf 0060 0000 0090 0611 0000 02f2 0000 116c 0004 8901 0000  |q.A...7.w.......B..E.kq..`.............l......|
       0x002e 0335 0000 0352 0000 0631 0060 0351 0050 05e9 0004 9001 0000 1181 0000 0491 0050 0494 0001 1701 0000 0361 0004 9141  |.5...R...1.`.Q.P...............P.........a...A|
       0x005c 0004 9181 0004 9101 0060 1184 0070 035a 0005 0101 0000 0511 0000 0519 0000 0524 0000 0531 0050 0521 0070 030c 0008  |.........`...p.Z...............$...1.P.!.p....|
       0x008a 1001 0000 038d 0000 0055 0000 122c 0000 0562 0040 0056 0000 01ba 0009 4041 0000 018e 0009 4001 0040 1229 0000 1226  |.........U...,...b.@.V......@A......@..@.)...&|
       0x00b8 0000 1221 0009 3801 0009 3401 0000 0309 0000 04f1 0000 0599 0002 2001 0060 0534 0030 10ac 0001 0d01 0001 1401 0001  |...!..8...4............... ..`.4.0............|
       0x00e6 0d41 000b 8141 000b 8181 0000 10a9 0000 059c 000b 8101 0000 0359 0000 0509 0000 0529 0000 0539 0005 0001 0060 05e4  |.A...A...................Y.......)...9.....`..|
       0x0114 0004 0801 0000 0474 0000 05dc 0004 0841 0090 0471 0071 af47 0001 ae01 0001 ae41 0000 1029 0000 1039 0030 102c 0001  |.......t.......A...q.q.G.......A...)...9.0.,..|
       0x0142 3041 0000 1031 0001 3001 0040 103c 0050 0102 0000 01ae 0005 1081 0005 1061 0005 1041 0005 1001 0000 051d 0000 0101  |0A...1..0..@.<.P.............a...A............|
       0x0170 0005 0801 0091 af49 0030 055c 0001 a901 0001 a941 0001 a942 0000 0544 0000 04fc 0040 044c 0000 033e 0000 b042 0003  |.......I.0.\.......A...B...D.....@.L...>...B..|
       0x019e 8301 0000 0459 0000 0541 0000 0559 0000 0449 0000 04f9 0001 b001 0040 119c 0001 a001 0000 04d4 0001 a041 0070 04d1  |.....Y...A...Y...I.........@.............A.p..|
       0x01cc 0000 04c9 0040 04c4 0001 0f01 0001 1301 0001 1101 0000 0355 0040 04b9 0001 0301 0000 0499 0000 1001 0000 049c 0000  |.....@.................U.@....................|
       0x01fa 04ba 0000 04c1 0000 04cc 0001 0501 0070 047c 0001 1001 0001 1081 0000 041e 0000 040c 0001 1041 0070 0409 0003 8942  |...............p.|...................A.p.....B|
       0x0228 0060 1011 0001 0201 0000 1189 0000 038a 0001 0241 0000 0389 0030 118c 0000 0414 0002 3441 0002 3401 0003 8941 0000  |.`.................A.....0........4A..4....A..|
       0x0256 1014 0000 05d1 0000 0381 0003 8801 0070 0419 0030 11cc 0001 8c01 0001 8c41 0000 11d1 0000 0022 0000 10f9 0000 11c9  |...............p...0.........A......."........|
       0x0284 0001 8b01 0001 8901 0000 0069 0020 0399 0008 0001 0008 0041 0000 1199 0000 0479 0000 039b 0001 af48 0001 af01 00c0  |...........i. .........A.......y.......H......|
       0x02b2 05d9 00c3 8043 0000 11e9 0000 1149 0000 042d 0000 11f1 0000 1069 0000 1161 0003 8701 0060 106c 0040 107c 0000 0272  |.....C.......I...-.......i...a.....`.l.@.|...r|
       0x02e0 0000 020a 0003 a801 0003 a841 0000 1071 0000 1074 0000 1045 0000 1079 0003 8501 0050 114c 0050 1144 000b 0001 0000  |...........A...q...t...E...y.....P.L.P.D......|
       0x030e 1201 0000 0304 0000 0301 0030 1204 0000 1211 0000 1209 000b 1001 0080 101c 0000 1091 0001 0e01 0000 1049 0009 5a44  |...........0.............................I..ZD|
       0x033c 0020 104c 0000 10e1 0001 0b01 0069 5a41 0030 108c 0009 5a01 0000 10f1 0000 10e9 0020 1024 0001 0c01 0000 03e9 0000  |. .L.........iZA.0....Z.......... .$..........|
       0x036a 1064 0000 1021 0000 1089 0001 0a01 0009 5a42 0039 5a43 0000 1059 0001 0021 0001 0901 0000 1019 0000 1141 000b 0801  |.d...!..........ZB.9ZC...Y...!...........A....|
       0x0398 0040 1164 0060 1159 0007 0c01 0000 fd01 0000 0601 0000 0609 0000 0604 0000 fd02 0007 1a41 0000 115c 0007 1a01 0060  |.@.d.`.Y...........................A...\.....`|
       0x03c6 11ec 0030 061c 0007 1901 0007 1941 0007 1942 0000 0369 0000 0619 0000 036c 0007 1041 0007 1001 0040 11f4 0060 0621  |...0.........A...B...i.......l...A.....@...`.!|
       0x03f4 0007 0d01 0000 11f9 0000 007d 0000 0371 0040 0374 0000 0489 0000 0334 0001 1201 0001 1501 0000 11fc 0000 0624 0007  |...........}...q.@.t.......4...............$..|
       0x0422 1841 0007 1801 0063 8042 0006 0801 0000 0401 0000 1081 0000 0011 0000 00a5 0040 0404 0000 033a 0000 0339 0000 b041  |.A.....c.B.....................@.....:...9...A|
       0x0450 0003 8401 0070 0454 0003 8201 0000 0469 0000 0315 0000 04a1 0000 0461 0000 0464 0040 04a4 0000 11e1 0000 05a1 0000  |.....p.T.......i...........a...d.@............|
       0x047e 11e4 0003 9801 0060 03bc 0001 0801 0001 0842 0000 019a 0000 03b4 0040 0199 0000 03d9 0000 03c1 0000 03dc 0003 8901  |.......`.........B.........@..................|
       0x04ac 0100 03b1 0000 11c4 0000 0574 0040 03a9 0005 a001 0000 04e1 0000 11d9 0030 11dc 0000 1194 0003 0881 0003 0801 0000  |...........t.@...............0................|
       0x04da 034c 0000 03ac 0001 1901 0000 03e1 0000 0571 0000 11c1 0001 1d01 0000 03f1 0001 1e01 0000 03cd 0000 0481 0000 0349  |.L...............q...........................I|
       0x0508 0001 8d01 0040 0346 0006 0001 0000 03fb 0000 0649 0006 0041 0003 8041 0001 4002 0000 03b9 0000 0451 0000 0345 0003  |.....@.F...........I...A...A..@........Q...E..|
       0x0536 8001 0001 2301 0000 05ec 0000 0121 0000 05e1 0004 8801 0020 0634 0000 0391 0001 0481 0070 1169 0000 1174 0050 0439  |....#........!......... .4.........p.i...t.P.9|
       0x0564 0002 3101 0000 05b4 0002 3148 0052 3147 0060 032c 0005 a801 0000 0239 0000 05c9 0000 0584 0005 a841 0060 0581 0000  |..1.......1H.R1G.`.,.......9...........A.`....|
       0x0592 058c 0000 0579 0000 057c 0000 0589 0000 0641 0005 9001 0000 05c1 0000 1054 0000 0329 0002 3001 0052 3149 0070 05b9  |.....y...|.......A...........T...)..0..R1I.p..|
       0x05c0 0002 8001 0000 03d4 0000 05ac 0000 0435 0000 03d1 0000 0436 0090 05a9 0011 0001 0001 0701 0000 11b9 0000 0594 0001  |...............5.......6......................|
       0x05ee 0002 0001 0702 0000 0591 0000 05b1 0000 11bc 0001 0101 0000 05bc 0000 04e9 0002 3201 0000 03a1 0000 02fe 0000 043c  |................................2............<|
       0x061c 0000 02fd 0000 1171 0004 8c01 0004 8061 0000 05f1 0000 0614 0000 0441 0004 8001                                     |.......q.......a...........A....|

  5000 lvrouted STRU  struct sockaddr { AF_INET, 172.16.5.241:12345 }
  5000 lvrouted RET   recvfrom 1596/0x63c
  5000 lvrouted CALL  open(0x2862d070,0x601<O_WRONLY|O_CREAT|O_TRUNC>,0x1b6<S_IRUSR|S_IWUSR|S_IRGRP|S_IWGRP|S_IROTH|S_IWOTH>)
  5000 lvrouted NAMI  "/tmp/lvrouted.packet-172.16.5.241"
  5000 lvrouted RET   open -1 errno 24 Too many open files
  5000 lvrouted CALL  clock_gettime(0xd,0xbfbfe974)
  5000 lvrouted RET   clock_gettime 0
  5000 lvrouted CALL  write(0x5,0x285f1034,0x1000)
  5000 lvrouted RET   write -1 errno 28 No space left on device
  5000 lvrouted CALL  select(0x5,0xbfbfe918,0xbfbfe898,0xbfbfe818,0xbfbfe80c)
  5000 lvrouted RET   select 1
  5000 lvrouted CALL  read(0x4,0x287acc00,0x400)
  5000 lvrouted GIO   fd 4 read 30 bytes
       0x0000 1e00 0512 0c00 776c 616e 3000 0000 0000 0000 0000 0000 6800 50cc f876 9c44                                          |......wlan0...........h.P..v.D|

  5000 lvrouted RET   read 30/0x1e
  5000 lvrouted CALL  gettimeofday(0xbfbfe9a8,0)
  5000 lvrouted RET   gettimeofday 0
  5000 lvrouted CALL  gettimeofday(0xbfbfe978,0)
  5000 lvrouted RET   gettimeofday 0
  5000 lvrouted CALL  gettimeofday(0xbfbfe98c,0)
  5000 lvrouted RET   gettimeofday 0
  5000 lvrouted CALL  gettimeofday(0xbfbfe934,0)
  5000 lvrouted RET   gettimeofday 0
  5000 lvrouted CALL  gettimeofday(0xbfbfe91c,0)
  5000 lvrouted RET   gettimeofday 0
  5000 lvrouted CALL  gettimeofday(0xbfbfe904,0)
  5000 lvrouted RET   gettimeofday 0
  5000 lvrouted CALL  gettimeofday(0xbfbfe8dc,0)
  5000 lvrouted RET   gettimeofday 0
  5000 lvrouted CALL  gettimeofday(0xbfbfe8d0,0)
  5000 lvrouted RET   gettimeofday 0
  5000 lvrouted CALL  gettimeofday(0xbfbfe8c4,0)
  5000 lvrouted RET   gettimeofday 0
  5000 lvrouted CALL  gettimeofday(0xbfbfe8b8,0)
  5000 lvrouted RET   gettimeofday 0
  5000 lvrouted CALL  gettimeofday(0xbfbfe8dc,0)
  5000 lvrouted RET   gettimeofday 0
  5000 lvrouted CALL  gettimeofday(0xbfbfe8d0,0)
  5000 lvrouted RET   gettimeofday 0
  5000 lvrouted CALL  gettimeofday(0xbfbfe8f4,0)
  5000 lvrouted RET   gettimeofday 0
  5000 lvrouted CALL  gettimeofday(0xbfbfe8e8,0)
  5000 lvrouted RET   gettimeofday 0
  5000 lvrouted CALL  gettimeofday(0xbfbfe8ec,0)
  5000 lvrouted RET   gettimeofday 0
  5000 lvrouted CALL  gettimeofday(0xbfbfe904,0)
  5000 lvrouted RET   gettimeofday 0
  5000 lvrouted CALL  gettimeofday(0xbfbfe91c,0)
  5000 lvrouted RET   gettimeofday 0
  5000 lvrouted CALL  gettimeofday(0xbfbfe904,0)
  5000 lvrouted RET   gettimeofday 0
  5000 lvrouted CALL  gettimeofday(0xbfbfe91c,0)
  5000 lvrouted RET   gettimeofday 0
  5000 lvrouted CALL  gettimeofday(0xbfbfe934,0)
  5000 lvrouted RET   gettimeofday 0
  5000 lvrouted CALL  gettimeofday(0xbfbfe91c,0)
  5000 lvrouted RET   gettimeofday 0
  5000 lvrouted CALL  gettimeofday(0xbfbfe904,0)
  5000 lvrouted RET   gettimeofday 0
  5000 lvrouted CALL  gettimeofday(0xbfbfe934,0)
  5000 lvrouted RET   gettimeofday 0
  5000 lvrouted CALL  gettimeofday(0xbfbfe90c,0)
  5000 lvrouted RET   gettimeofday 0
  5000 lvrouted CALL  gettimeofday(0xbfbfe900,0)
  5000 lvrouted RET   gettimeofday 0
  5000 lvrouted CALL  gettimeofday(0xbfbfe8f4,0)
  5000 lvrouted RET   gettimeofday 0
  5000 lvrouted CALL  gettimeofday(0xbfbfe8e8,0)
  5000 lvrouted RET   gettimeofday 0
  5000 lvrouted CALL  gettimeofday(0xbfbfe8dc,0)
  5000 lvrouted RET   gettimeofday 0
  5000 lvrouted CALL  gettimeofday(0xbfbfe8d0,0)
  5000 lvrouted RET   gettimeofday 0
  5000 lvrouted CALL  gettimeofday(0xbfbfe934,0)
  5000 lvrouted RET   gettimeofday 0
  5000 lvrouted CALL  select(0x5,0xbfbfe918,0xbfbfe898,0xbfbfe818,0xbfbfe80c)
  5000 lvrouted RET   select 1
  5000 lvrouted CALL  recvfrom(0x3,0xbfbfa994,0x4000,0,0xbfbfa924,0xbfbfa920)
  5000 lvrouted GIO   fd 3 read 1596 bytes
       0x0000 a6ef acc3 0d87 e12a 4c37 42b3 b24d 4bfe 5d57 5543 946b 71cf 00b0 0000 0050 11cc 0001 8c01 0001 8c41 0000 11d1 0000  |.......*L7B..MK.]WUC.kq......P.........A......|
       0x002e 11c4 0140 11c1 0030 0574 0001 a901 0001 a941 0001 a942 0090 0541 0001 b001 0000 04f9 0000 04c9 0000 0449 0000 0559  |...@...0.t.......A...B...A...............I...Y|
       0x005c 0000 044c 0000 04cc 0040 04fc 0000 0355 0001 1101 0001 1301 0001 0f01 0000 055c 0070 03d4 0002 8001 0000 05ac 0000  |...L.....@.....U...............\.p............|
       0x008a 05b9 0000 0435 0000 0436 0000 05a9 0060 05bc 0002 3148 0032 3147 0002 3001 0000 1054 0000 05c1 0002 3149 0000 04e9  |.....5...6.....`....1H.21G..0....T......1I....|
       0x00b8 0002 3201 0000 03a1 0000 03b4 0060 03a9 0005 a001 0000 04e1 0000 11d9 0000 0579 0060 057c 0000 058c 0060 0584 0005  |..2..........`.................y.`.|.....`....|
       0x00e6 a801 0000 0239 0000 05c9 0000 032c 0005 a841 0000 0329 0000 0581 0000 0589 0000 0641 0005 9001 0030 11dc 0000 1194  |.....9.......,...A...)...........A.....0......|
       0x0114 0003 0881 0003 0801 0040 034c 0003 8401 0000 b041 0000 0339 0000 033a 0000 03ac 0001 1901 0000 03e1 0000 03d1 0000  |.........@.L.......A...9...:..................|
       0x0142 0571 0001 1d01 0000 03f1 0000 0544 0001 1e01 0000 03cd 0000 03b1 0000 0481 0000 0349 0001 8d01 0070 0461 0003 8201  |.q...........D...................I.....p.a....|
       0x0170 0000 0469 0000 0315 0000 04a1 0000 0454 00c0 0451 00c3 8043 0000 11e9 0000 1149 0000 042d 0000 11f1 0000 1069 0000  |...i...........T...Q...C.......I...-.......i..|
       0x019e 1161 0003 8701 0000 106c 0050 114c 0050 1144 000b 0001 0000 1201 0000 0304 0000 0301 0030 1204 0000 1211 0000 1209  |.a.......l.P.L.P.D.................0..........|
       0x01cc 000b 1001 0080 101c 0000 1091 0001 0e01 0000 1049 0009 5a44 0020 104c 0000 10e1 0001 0b01 0069 5a41 0030 108c 0009  |...................I..ZD. .L.........iZA.0....|
       0x01fa 5a01 0000 10f1 0000 10e9 0020 1024 0001 0c01 0000 03e9 0000 1064 0000 1021 0000 1089 0001 0a01 0009 5a42 0039 5a43  |Z.......... .$...........d...!..........ZB.9ZC|
       0x0228 0000 1059 0001 0021 0001 0901 0000 1019 0000 1141 000b 0801 0040 1164 0060 1159 0007 0c01 0000 fd01 0000 0601 0000  |...Y...!...........A.....@.d.`.Y..............|
       0x0256 0609 0000 0604 0000 fd02 0007 1a41 0000 115c 0007 1a01 0060 11ec 0030 061c 0007 1901 0007 1941 0007 1942 0000 0369  |.............A...\.....`...0.........A...B...i|
       0x0284 0000 0619 0000 036c 0007 1041 0007 1001 0040 11f4 0060 0621 0007 0d01 0000 11f9 0000 007d 0000 0371 0040 0374 0000  |.......l...A.....@...`.!...........}...q.@.t..|
       0x02b2 0489 0000 0334 0001 1201 0001 1501 0000 11fc 0000 0624 0007 1841 0007 1801 0063 8042 0006 0801 0000 0401 0000 1081  |.....4...............$...A.....c.B............|
       0x02e0 0000 0011 0000 00a5 0000 0404 0000 05dc 0020 03bc 0001 0801 0001 0842 0040 0346 0006 0001 0000 03fb 0000 0649 0006  |................. .........B.@.F...........I..|
       0x030e 0041 0003 8041 0001 4002 0000 03b9 0000 05d9 0000 0345 0003 8001 0001 2301 0040 04a4 0060 11e1 0001 0301 0000 04b9  |.A...A..@............E......#..@...`..........|
       0x033c 0000 0499 0000 1001 0040 049c 0000 0459 0003 8301 0000 b042 0000 033e 0000 04ba 0000 05a1 0000 11e4 0003 9801 0070  |.........@.....Y.......B...>.................p|
       0x036a 041e 0001 1001 0001 1081 0000 047c 0000 040c 0001 1041 0070 0409 0063 8942 0003 8901 0000 03dc 0000 03c1 0000 0199  |.............|.......A.p...c.B................|
       0x0398 0000 019a 0000 03d9 00a0 1011 0001 0201 0000 1189 0000 11b9 0000 1071 0000 038a 0001 0241 0000 0389 0040 1074 0040  |...........................q.......A.....@.t.@|
       0x03c6 107c 0000 0272 0000 020a 0003 a801 0003 a841 0000 1045 0000 1079 0003 8501 0030 118c 0000 0414 0002 3441 0002 3401  |.|...r...........A...E...y.....0........4A..4.|
       0x03f4 0080 11bc 0011 0001 0001 0701 0030 05b4 0002 3101 0000 0439 0030 043c 0000 02fe 0000 02fd 0004 8c01 0090 0594 0008  |.............0....1....9.0.<..................|
       0x0422 1001 0000 038d 0000 0055 0000 122c 0000 0562 0000 030c 0040 0056 0000 01ba 0009 4041 0000 018e 0009 4001 0050 0309  |.........U...,...b.....@.V......@A......@..P..|
       0x0450 0050 0524 0005 0101 0000 0511 0000 0519 0000 0531 0060 0534 0030 10ac 0001 0d01 0001 1401 0001 0d41 000b 8141 000b  |.P.$...............1.`.4.0.............A...A..|
       0x047e 8181 0000 10a9 0000 059c 000b 8101 0000 04f1 0000 0521 0000 0599 0002 2001 0040 1229 0000 1226 0000 1221 0009 3801  |.....................!...... ..@.)...&...!..8.|
       0x04ac 0009 3401 0001 0002 0001 0702 0000 0591 0000 05b1 0001 0101 0003 8941 0000 1014 0000 05d1 0000 0381 0003 8801 0090  |..4........................A..................|
       0x04da 0479 0071 af47 0001 ae01 0001 ae41 0000 1029 0000 1039 0030 102c 0001 3041 0000 1031 0001 3001 0040 103c 0050 0102  |.y.q.G.......A...)...9.0.,..0A...1..0..@.<.P..|
       0x0508 0000 01ae 0005 1081 0005 1061 0005 1041 0005 1001 0000 051d 0000 0101 0005 0801 0001 af49 0040 119c 0001 a001 0000  |...........a...A...................I.@........|
       0x0536 04d4 0001 a041 0030 04d1 0000 04c4 0000 04c1 0001 0501 0040 0474 0004 0801 0000 05e4 0004 0841 0060 05e1 0070 05e9  |.....A.0...............@.t...........A.`...p..|
       0x0564 0004 9001 0000 1174 0000 1181 0000 0491 0050 0494 0001 1701 0000 0361 0004 9141 0004 9181 0004 9101 0000 1171 0060  |.......t.........P.........a...A...........q.`|
       0x0592 1184 0000 035a 0000 0359 0000 0509 0000 0529 0000 0539 0005 0001 0090 0352 0000 02f2 0000 116c 0004 8901 0000 0611  |.....Z...Y.......)...9.......R.......l........|
       0x05c0 0000 0335 0000 0631 0040 0614 0004 8061 0000 05f1 0000 0441 0004 8001 0020 0634 0000 0391 0001 0481 0000 1169 0000  |...5...1.@.....a.......A..... .4...........i..|
       0x05ee 0351 0000 05ec 0000 0121 0004 8801 0020 0399 0008 0001 0008 0041 0000 1199 0000 0471 0000 039b 0001 af48 0001 af01  |.Q.......!..... .........A.......q.......H....|
       0x061c 0000 0022 0000 0464 0000 10f9 0000 11c9 0000 0419 0001 8b01 0001 8901 0000 0069                                     |..."...d.......................i|

  5000 lvrouted STRU  struct sockaddr { AF_INET, 172.16.16.249:12345 }
  5000 lvrouted RET   recvfrom 1596/0x63c
  5000 lvrouted CALL  open(0x2862d070,0x601<O_WRONLY|O_CREAT|O_TRUNC>,0x1b6<S_IRUSR|S_IWUSR|S_IRGRP|S_IWGRP|S_IROTH|S_IWOTH>)
  5000 lvrouted NAMI  "/tmp/lvrouted.packet-172.16.16.249"
CNodeIMI#

/var was

Change History (1)

comment:1 by rick, 12 years ago

Resolution:	→ fixed
Status:	new → closed

/var/log/pen.log was not rotating properly. Adding it to the nodefactory.

Note: See TracTickets for help on using tickets.

Download in other formats: