Dziwny zwis serwera

[webmaniac 0]

Cześć, oto log z mojego hosta - z ostatnich sekund przed zgonem dzisiejszego popołudnia. Na co zwrócić uwagę, aby taki zgon więcej nie wystąpił?

 

Czy dobrze rozumuję, że serwer zaliczył śmierć przez nadmierne jedzenie ramu?

 

May 18 14:45:12 sd243 kernel: [844796.717428] php-cgi invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
May 18 14:45:12 sd243 kernel: [844796.717433] php-cgi cpuset=/ mems_allowed=0
May 18 14:45:12 sd243 kernel: [844796.717436] Pid: 31838, comm: php-cgi Not tainted 2.6.32-5-amd64 #1
May 18 14:45:12 sd243 kernel: [844796.717437] Call Trace:
May 18 14:45:12 sd243 kernel: [844796.717444]  [<ffffffff810b6444>] ? oom_kill_process+0x7f/0x23f
May 18 14:45:12 sd243 kernel: [844796.717447]  [<ffffffff810b6968>] ? __out_of_memory+0x12a/0x141
May 18 14:45:12 sd243 kernel: [844796.717450]  [<ffffffff810b6abf>] ? out_of_memory+0x140/0x172
May 18 14:45:12 sd243 kernel: [844796.717453]  [<ffffffff810ba824>] ? __alloc_pages_nodemask+0x4ec/0x5fc
May 18 14:45:12 sd243 kernel: [844796.717456]  [<ffffffff810bbd89>] ? __do_page_cache_readahead+0x9b/0x1b4
May 18 14:45:12 sd243 kernel: [844796.717459]  [<ffffffff810bbebe>] ? ra_submit+0x1c/0x20
May 18 14:45:12 sd243 kernel: [844796.717462]  [<ffffffff810b4b30>] ? filemap_fault+0x11c/0x2f6
May 18 14:45:12 sd243 kernel: [844796.717466]  [<ffffffff810cab06>] ? __do_fault+0x54/0x3c3
May 18 14:45:12 sd243 kernel: [844796.717468]  [<ffffffff810c3d22>] ? vma_prio_tree_insert+0x20/0x3d
May 18 14:45:12 sd243 kernel: [844796.717471]  [<ffffffff810cce5a>] ? handle_mm_fault+0x3b8/0x80f
May 18 14:45:12 sd243 kernel: [844796.717474]  [<ffffffff810d04b5>] ? get_unmapped_area+0xd7/0x139
May 18 14:45:12 sd243 kernel: [844796.717478]  [<ffffffff812ff1a6>] ? do_page_fault+0x2e0/0x2fc
May 18 14:45:12 sd243 kernel: [844796.717481]  [<ffffffff812fd045>] ? page_fault+0x25/0x30
May 18 14:45:12 sd243 kernel: [844796.717483] Mem-Info:
May 18 14:45:12 sd243 kernel: [844796.717485] Node 0 DMA per-cpu:
May 18 14:45:12 sd243 kernel: [844796.717487] CPU	0: hi:	0, btch:   1 usd:   0
May 18 14:45:12 sd243 kernel: [844796.717489] CPU	1: hi:	0, btch:   1 usd:   0
May 18 14:45:12 sd243 kernel: [844796.717491] CPU	2: hi:	0, btch:   1 usd:   0
May 18 14:45:12 sd243 kernel: [844796.717492] CPU	3: hi:	0, btch:   1 usd:   0
May 18 14:45:12 sd243 kernel: [844796.717494] Node 0 DMA32 per-cpu:
May 18 14:45:12 sd243 kernel: [844796.717496] CPU	0: hi:  186, btch:  31 usd:  24
May 18 14:45:12 sd243 kernel: [844796.717498] CPU	1: hi:  186, btch:  31 usd:  39
May 18 14:45:12 sd243 kernel: [844796.717499] CPU	2: hi:  186, btch:  31 usd:  43
May 18 14:45:12 sd243 kernel: [844796.717501] CPU	3: hi:  186, btch:  31 usd:   7
May 18 14:45:12 sd243 kernel: [844796.717502] Node 0 Normal per-cpu:
May 18 14:45:12 sd243 kernel: [844796.717504] CPU	0: hi:  186, btch:  31 usd:  33
May 18 14:45:12 sd243 kernel: [844796.717506] CPU	1: hi:  186, btch:  31 usd: 113
May 18 14:45:12 sd243 kernel: [844796.717508] CPU	2: hi:  186, btch:  31 usd:  58
May 18 14:45:12 sd243 kernel: [844796.717510] CPU	3: hi:  186, btch:  31 usd: 145
May 18 14:45:12 sd243 kernel: [844796.717514] active_anon:1695500 inactive_anon:207566 isolated_anon:2
May 18 14:45:12 sd243 kernel: [844796.717515]  active_file:309 inactive_file:422 isolated_file:128
May 18 14:45:12 sd243 kernel: [844796.717516]  unevictable:0 dirty:0 writeback:2155 unstable:0
May 18 14:45:12 sd243 kernel: [844796.717517]  free:11319 slab_reclaimable:7788 slab_unreclaimable:22023
May 18 14:45:12 sd243 kernel: [844796.717518]  mapped:361 shmem:476 pagetables:69151 bounce:0
May 18 14:45:12 sd243 kernel: [844796.717520] Node 0 DMA free:15844kB min:20kB low:24kB high:28kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15276kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
May 18 14:45:12 sd243 kernel: [844796.717529] lowmem_reserve[]: 0 3477 8022 8022
May 18 14:45:12 sd243 kernel: [844796.717532] Node 0 DMA32 free:22948kB min:4960kB low:6200kB high:7440kB active_anon:3057384kB inactive_anon:215468kB active_file:96kB inactive_file:212kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3561184kB mlocked:0kB dirty:0kB writeback:736kB mapped:0kB shmem:192kB slab_reclaimable:11072kB slab_unreclaimable:30424kB kernel_stack:4752kB pagetables:68976kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:19860 all_unreclaimable? yes
May 18 14:45:12 sd243 kernel: [844796.717541] lowmem_reserve[]: 0 0 4545 4545
May 18 14:45:12 sd243 kernel: [844796.717544] Node 0 Normal free:6484kB min:6484kB low:8104kB high:9724kB active_anon:3724616kB inactive_anon:614796kB active_file:1140kB inactive_file:1476kB unevictable:0kB isolated(anon):8kB isolated(file):512kB present:4654080kB mlocked:0kB dirty:0kB writeback:7884kB mapped:1472kB shmem:1712kB slab_reclaimable:20080kB slab_unreclaimable:57668kB kernel_stack:10720kB pagetables:207628kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:48260 all_unreclaimable? yes
May 18 14:45:12 sd243 kernel: [844796.717555] lowmem_reserve[]: 0 0 0 0
May 18 14:45:12 sd243 kernel: [844796.717557] Node 0 DMA: 3*4kB 3*8kB 4*16kB 2*32kB 3*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15844kB
May 18 14:45:12 sd243 kernel: [844796.717565] Node 0 DMA32: 196*4kB 925*8kB 197*16kB 57*32kB 17*64kB 27*128kB 18*256kB 2*512kB 0*1024kB 0*2048kB 0*4096kB = 23336kB
May 18 14:45:12 sd243 kernel: [844796.717573] Node 0 Normal: 1061*4kB 16*8kB 2*16kB 9*32kB 8*64kB 4*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 6484kB
May 18 14:45:12 sd243 kernel: [844796.717580] 29197 total pagecache pages
May 18 14:45:12 sd243 kernel: [844796.717582] 27663 pages in swap cache
May 18 14:45:12 sd243 kernel: [844796.717584] Swap cache stats: add 101838, delete 74175, find 1301847/1302652
May 18 14:45:12 sd243 kernel: [844796.717585] Free swap  = 0kB
May 18 14:45:12 sd243 kernel: [844796.717586] Total swap = 248756kB
May 18 14:45:12 sd243 kernel: [844796.742459] 2097136 pages RAM
May 18 14:45:12 sd243 kernel: [844796.742461] 56438 pages reserved
May 18 14:45:12 sd243 kernel: [844796.742462] 2261349 pages shared
May 18 14:45:12 sd243 kernel: [844796.742463] 2022241 pages non-shared
May 18 14:45:12 sd243 kernel: [844797.463054] httpd invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
May 18 14:45:12 sd243 kernel: [844797.463058] httpd cpuset=/ mems_allowed=0
May 18 14:45:12 sd243 kernel: [844797.463061] Pid: 31360, comm: httpd Not tainted 2.6.32-5-amd64 #1
May 18 14:45:12 sd243 kernel: [844797.463063] Call Trace:
May 18 14:45:12 sd243 kernel: [844797.463070]  [<ffffffff810b6444>] ? oom_kill_process+0x7f/0x23f
May 18 14:45:12 sd243 kernel: [844797.463072]  [<ffffffff810b6968>] ? __out_of_memory+0x12a/0x141
May 18 14:45:12 sd243 kernel: [844797.463075]  [<ffffffff810b6abf>] ? out_of_memory+0x140/0x172
May 18 14:45:12 sd243 kernel: [844797.463078]  [<ffffffff810ba824>] ? __alloc_pages_nodemask+0x4ec/0x5fc
May 18 14:45:12 sd243 kernel: [844797.463082]  [<ffffffff81065033>] ? autoremove_wake_function+0x9/0x2e
May 18 14:45:12 sd243 kernel: [844797.463085]  [<ffffffff810b4c31>] ? filemap_fault+0x21d/0x2f6
May 18 14:45:12 sd243 kernel: [844797.463089]  [<ffffffff810cab06>] ? __do_fault+0x54/0x3c3
May 18 14:45:12 sd243 kernel: [844797.463092]  [<ffffffff810eedaa>] ? do_sync_write+0xce/0x113
May 18 14:45:12 sd243 kernel: [844797.463095]  [<ffffffff810cce5a>] ? handle_mm_fault+0x3b8/0x80f
May 18 14:45:12 sd243 kernel: [844797.463099]  [<ffffffff812ff1a6>] ? do_page_fault+0x2e0/0x2fc
May 18 14:45:12 sd243 kernel: [844797.463102]  [<ffffffff812fd045>] ? page_fault+0x25/0x30
May 18 14:45:12 sd243 kernel: [844797.463104] Mem-Info:
May 18 14:45:12 sd243 kernel: [844797.463105] Node 0 DMA per-cpu:
May 18 14:45:12 sd243 kernel: [844797.463107] CPU	0: hi:	0, btch:   1 usd:   0
May 18 14:45:12 sd243 kernel: [844797.463109] CPU	1: hi:	0, btch:   1 usd:   0
May 18 14:45:12 sd243 kernel: [844797.463111] CPU	2: hi:	0, btch:   1 usd:   0
May 18 14:45:12 sd243 kernel: [844797.463113] CPU	3: hi:	0, btch:   1 usd:   0
May 18 14:45:12 sd243 kernel: [844797.463114] Node 0 DMA32 per-cpu:
May 18 14:45:12 sd243 kernel: [844797.463116] CPU	0: hi:  186, btch:  31 usd:   8
May 18 14:45:12 sd243 kernel: [844797.463118] CPU	1: hi:  186, btch:  31 usd:  13
May 18 14:45:12 sd243 kernel: [844797.463120] CPU	2: hi:  186, btch:  31 usd:   0
May 18 14:45:12 sd243 kernel: [844797.463121] CPU	3: hi:  186, btch:  31 usd:   5
May 18 14:45:12 sd243 kernel: [844797.463123] Node 0 Normal per-cpu:
May 18 14:45:12 sd243 kernel: [844797.463125] CPU	0: hi:  186, btch:  31 usd:   6
May 18 14:45:12 sd243 kernel: [844797.463127] CPU	1: hi:  186, btch:  31 usd:   9
May 18 14:45:12 sd243 kernel: [844797.463128] CPU	2: hi:  186, btch:  31 usd:  20
May 18 14:45:12 sd243 kernel: [844797.463130] CPU	3: hi:  186, btch:  31 usd:   7
May 18 14:45:12 sd243 kernel: [844797.463134] active_anon:1694195 inactive_anon:208974 isolated_anon:2
May 18 14:45:12 sd243 kernel: [844797.463135]  active_file:412 inactive_file:478 isolated_file:160
May 18 14:45:12 sd243 kernel: [844797.463136]  unevictable:0 dirty:0 writeback:2101 unstable:0
May 18 14:45:12 sd243 kernel: [844797.463137]  free:11292 slab_reclaimable:7759 slab_unreclaimable:21972
May 18 14:45:12 sd243 kernel: [844797.463138]  mapped:405 shmem:476 pagetables:69151 bounce:0
May 18 14:45:12 sd243 kernel: [844797.463140] Node 0 DMA free:15844kB min:20kB low:24kB high:28kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15276kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
May 18 14:45:12 sd243 kernel: [844797.463149] lowmem_reserve[]: 0 3477 8022 8022
May 18 14:45:12 sd243 kernel: [844797.463152] Node 0 DMA32 free:22972kB min:4960kB low:6200kB high:7440kB active_anon:3057524kB inactive_anon:215596kB active_file:176kB inactive_file:344kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3561184kB mlocked:0kB dirty:0kB writeback:604kB mapped:112kB shmem:192kB slab_reclaimable:10984kB slab_unreclaimable:30488kB kernel_stack:4752kB pagetables:68976kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:56377 all_unreclaimable? yes
May 18 14:45:12 sd243 kernel: [844797.463161] lowmem_reserve[]: 0 0 4545 4545
May 18 14:45:12 sd243 kernel: [844797.463164] Node 0 Normal free:6352kB min:6484kB low:8104kB high:9724kB active_anon:3719256kB inactive_anon:620300kB active_file:1472kB inactive_file:1568kB unevictable:0kB isolated(anon):8kB isolated(file):640kB present:4654080kB mlocked:0kB dirty:0kB writeback:7800kB mapped:1508kB shmem:1712kB slab_reclaimable:20052kB slab_unreclaimable:57400kB kernel_stack:10720kB pagetables:207628kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:62912 all_unreclaimable? yes
May 18 14:45:12 sd243 kernel: [844797.463174] lowmem_reserve[]: 0 0 0 0
May 18 14:45:12 sd243 kernel: [844797.463176] Node 0 DMA: 3*4kB 3*8kB 4*16kB 2*32kB 3*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15844kB
May 18 14:45:12 sd243 kernel: [844797.463184] Node 0 DMA32: 159*4kB 929*8kB 198*16kB 57*32kB 17*64kB 27*128kB 18*256kB 2*512kB 0*1024kB 0*2048kB 0*4096kB = 23236kB
May 18 14:45:12 sd243 kernel: [844797.463192] Node 0 Normal: 1055*4kB 7*8kB 2*16kB 9*32kB 8*64kB 4*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 6388kB
May 18 14:45:12 sd243 kernel: [844797.463199] 29169 total pagecache pages
May 18 14:45:12 sd243 kernel: [844797.463201] 27663 pages in swap cache
May 18 14:45:12 sd243 kernel: [844797.463202] Swap cache stats: add 101838, delete 74175, find 1301847/1302652
May 18 14:45:12 sd243 kernel: [844797.463204] Free swap  = 0kB
May 18 14:45:12 sd243 kernel: [844797.463205] Total swap = 248756kB
May 18 14:45:12 sd243 kernel: [844797.483415] 2097136 pages RAM
May 18 14:45:12 sd243 kernel: [844797.483417] 56438 pages reserved
May 18 14:45:12 sd243 kernel: [844797.483418] 2254364 pages shared
May 18 14:45:12 sd243 kernel: [844797.483419] 2022650 pages non-shared
May 18 14:45:28 sd243 kernel: [844811.606923] php-cgi invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
May 18 14:45:29 sd243 kernel: [844811.606928] php-cgi cpuset=/ mems_allowed=0

[patryk 451]

Zabrakło Ci pamięci, a spowodował to proces php-cgi. Wtedy zaczyna się swapowanie, a co za tym idzie kosmiczny load, a kiedy już i swapu brakuje (co widać jakoś w połowie tego loga) kernel zaczyna killować co się da, żeby uratować sytuację, gdybyś go tak zostawił na 10-15 minut to pewnie obyłoby się bez restartu :-).

[webmaniac 0]

Zmartwię Cię:

 

W sytuacji zorientowałem się dopiero o godzinie 20:00 (nie jest to serwer produkcyjny, więc rzadko tam zaglądam), więc minęło 6 godzin a serwer sam sobie nie poradził i się zabił (nawet technik podpięty bezpośrednio do maszyny nic nie wskórał).

 

Da się jakoś ustawić kolejkę rzeczy, które są killowane w momencie tworzenia się swapu? W pierwszej kolejności chciałbym pogasić PHP-CGI. W sumie - to moja wina, bo dałem aplikacji więcej rzeczy do parsowania niż zamierzałem.

[Arhu 4]

ja bym przede wszystkim zmienił php-cgi na php-fpm i ustawił limity w [ifdebian] /etc/security/limits.conf