togotoday
2015年1月19日 14時11分
お世話になります。坂本と申します。
Armadillo-840を使用させていただいております。
連続稼働試験を実施していますが、３〜４日連続で稼働すると、以下のようなメモリーallocエラーが発生し、
アプリ（Java）が落ちます（swapper: page allocation failure）。連続稼働の目標日数は30日ですので、
こんなに早くアプリが落ちるとは、ということで困っています。以下ログになります。
Jan 15 10:53:58 debian kernel: java invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Jan 15 10:53:58 debian kernel: java cpuset=/ mems_allowed=0
Jan 15 10:53:58 debian kernel: Backtrace:
Jan 15 10:53:58 debian kernel: [] (dump_backtrace+0x0/0x110) from [] (dump_stack+0x18/0x1c)
Jan 15 10:53:58 debian kernel: r6:000201da r5:00000000 r4:c0f52000 r3:00000002
Jan 15 10:53:58 debian kernel: [] (dump_stack+0x0/0x1c) from [] (dump_header.isra.13+0x8c/0x1a0)
Jan 15 10:53:58 debian kernel: [] (dump_header.isra.13+0x0/0x1a0) from [] (oom_kill_process+0x78/0x294)
Jan 15 10:53:58 debian kernel: [] (oom_kill_process+0x0/0x294) from [] (out_of_memory+0x160/0x208)
Jan 15 10:53:58 debian kernel: [] (out_of_memory+0x0/0x208) from [] (__alloc_pages_nodemask+0x504/0x5fc)
Jan 15 10:53:58 debian kernel: [] (__alloc_pages_nodemask+0x0/0x5fc) from [] (filemap_fault+0x2bc/0x3e4)
Jan 15 10:53:58 debian kernel: [] (filemap_fault+0x0/0x3e4) from [] (__do_fault+0xc4/0x4b4)
Jan 15 10:53:58 debian kernel: [] (__do_fault+0x0/0x4b4) from [] (handle_pte_fault+0x304/0x860)
Jan 15 10:53:58 debian kernel: [] (handle_pte_fault+0x0/0x860) from [] (handle_mm_fault+0xd4/0xf0)
Jan 15 10:53:58 debian kernel: [] (handle_mm_fault+0x0/0xf0) from [] (do_page_fault+0x104/0x340)
Jan 15 10:53:58 debian kernel: [] (do_page_fault+0x0/0x340) from [] (do_PrefetchAbort+0x3c/0xa0)
Jan 15 10:53:58 debian kernel: [] (do_PrefetchAbort+0x0/0xa0) from [] (ret_from_exception+0x0/0x10)
Jan 15 10:53:58 debian kernel: Exception stack(0xc0f53fb0 to 0xc0f53ff8)
Jan 15 10:53:58 debian kernel: 3fa0: 00415a38 ffffffff 00000000 00415a30
Jan 15 10:53:58 debian kernel: 3fc0: 00415a34 a5960af4 00415a2c 00000001 b6d21000 00415f38 00415a38 00000006
Jan 15 10:53:58 debian kernel: 3fe0: b061df60 a5960a28 b6bed7b3 b6ace4f0 40000030 ffffffff
Jan 15 10:53:58 debian kernel: r7:00000001 r6:ffffffff r5:40000030 r4:b6ace4f0
Jan 15 10:53:58 debian kernel: Mem-info:
Jan 15 10:53:58 debian kernel: Normal per-cpu:
Jan 15 10:53:58 debian kernel: CPU 0: hi: 18, btch: 3 usd: 0
Jan 15 10:53:58 debian kernel: active_anon:11067 inactive_anon:505 isolated_anon:0
Jan 15 10:53:58 debian kernel: active_file:95 inactive_file:138 isolated_file:0
Jan 15 10:53:58 debian kernel: unevictable:0 dirty:0 writeback:0 unstable:0
Jan 15 10:53:58 debian kernel: free:309 slab_reclaimable:340 slab_unreclaimable:1462
Jan 15 10:53:58 debian kernel: mapped:93 shmem:530 pagetables:116 bounce:0
Jan 15 10:53:58 debian kernel: Normal free:1236kB min:1236kB low:1544kB high:1852kB active_anon:44268kB inactive_anon:2020kB active_file:380kB inactive_file:552kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:97536kB mlocked:0kB dirty:0kB writeback:0kB mapped:372kB shmem:2120kB slab_reclaimable:1360kB slab_unreclaimable:5848kB kernel_stack:616kB pagetables:464kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:3542 all_unreclaimable? yes
Jan 15 10:53:58 debian kernel: lowmem_reserve[]: 0 0 0
Jan 15 10:53:58 debian kernel: Normal: 31*4kB 33*8kB 15*16kB 1*32kB 1*64kB 2*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1236kB
Jan 15 10:53:58 debian kernel: 763 total pagecache pages
Jan 15 10:53:58 debian kernel: 0 pages in swap cache
Jan 15 10:53:58 debian kernel: Swap cache stats: add 0, delete 0, find 0/0
Jan 15 10:53:58 debian kernel: Free swap = 0kB
Jan 15 10:53:58 debian kernel: Total swap = 0kB
Jan 15 10:53:58 debian kernel: 24576 pages of RAM
Jan 15 10:53:58 debian kernel: 449 free pages
Jan 15 10:53:58 debian kernel: 1916 reserved pages
Jan 15 10:53:58 debian kernel: 1802 slab pages
Jan 15 10:53:58 debian kernel: 385 pages shared
Jan 15 10:53:58 debian kernel: 0 pages swap cached
Jan 15 10:53:58 debian kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Jan 15 10:53:58 debian kernel: [ 1329] 0 1329 667 198 0 -17 -1000 udevd
Jan 15 10:53:58 debian kernel: [ 1539] 0 1539 666 197 0 -17 -1000 udevd
Jan 15 10:53:58 debian kernel: [ 1540] 0 1540 666 198 0 -17 -1000 udevd
Jan 15 10:53:58 debian kernel: [ 1798] 0 1798 320 23 0 0 0 logsave
Jan 15 10:53:58 debian kernel: [ 2415] 0 2415 320 23 0 0 0 logsave
Jan 15 10:53:58 debian kernel: [ 2811] 0 2811 1019 428 0 0 0 dhclient
Jan 15 10:53:58 debian kernel: [ 2942] 0 2942 466 49 0 0 0 rpcbind
Jan 15 10:53:58 debian kernel: [ 2975] 106 2975 544 101 0 0 0 rpc.statd
Jan 15 10:53:58 debian kernel: [ 2991] 0 2991 564 45 0 0 0 rpc.idmapd
Jan 15 10:53:58 debian kernel: [ 3220] 0 3220 6835 106 0 0 0 rsyslogd
Jan 15 10:53:58 debian kernel: [ 3266] 105 3266 73610 9353 0 0 0 java
Jan 15 10:53:58 debian kernel: [ 3339] 0 3339 423 32 0 0 0 atd
Jan 15 10:53:58 debian kernel: [ 3381] 0 3381 456 40 0 0 0 cron
Jan 15 10:53:58 debian kernel: [ 3411] 103 3411 639 58 0 0 0 dbus-daemon
Jan 15 10:53:58 debian kernel: [ 3440] 108 3440 1162 98 0 0 0 ntpd
Jan 15 10:53:58 debian kernel: [ 3470] 0 3470 382 25 0 0 0 inetd
Jan 15 10:53:58 debian kernel: [ 3531] 0 3531 1290 105 0 -17 -1000 sshd
Jan 15 10:53:58 debian kernel: [ 3589] 0 3589 689 85 0 0 0 login
Jan 15 10:53:58 debian kernel: [ 3593] 1001 3593 718 181 0 0 0 bash
Jan 15 11:03:28 debian kernel: swapper: page allocation failure: order:3, mode:0x20
Jan 15 11:03:28 debian kernel: Backtrace:
Jan 15 11:03:28 debian kernel: [] (dump_backtrace+0x0/0x110) from [] (dump_stack+0x18/0x1c)
Jan 15 11:03:28 debian kernel: r6:00000000 r5:00000020 r4:00000002 r3:c05fa238
Jan 15 11:03:28 debian kernel: [] (dump_stack+0x0/0x1c) from [] (warn_alloc_failed+0x100/0x120)
Jan 15 11:03:28 debian kernel: [] (warn_alloc_failed+0x0/0x120) from [] (__alloc_pages_nodemask+0x578/0x5fc)
Jan 15 11:03:28 debian kernel: r3:c05e1b54 r2:00000000
Jan 15 11:03:28 debian kernel: r7:00000003 r6:c05e0000 r5:c05e0000 r4:00000020
Jan 15 11:03:28 debian kernel: [] (__alloc_pages_nodemask+0x0/0x5fc) from [] (cache_alloc_refill+0x2d4/0x5a8)
Jan 15 11:03:28 debian kernel: [] (cache_alloc_refill+0x0/0x5a8) from [] (__kmalloc+0x98/0xec)
Jan 15 11:03:28 debian kernel: [] (__kmalloc+0x0/0xec) from [] (__alloc_skb+0x50/0xec)
Jan 15 11:03:28 debian kernel: r7:c18195c0 r6:c1b30600 r5:00004041 r4:00000020
Jan 15 11:03:28 debian kernel: [] (__alloc_skb+0x0/0xec) from [] (rx_submit+0x30/0x2a0)
Jan 15 11:03:28 debian kernel: r8:c10dabc0 r7:c10daf40 r6:00004000 r5:c1b5e240 r4:c1b11ba0
Jan 15 11:03:28 debian kernel: r3:ffffffff
Jan 15 11:03:28 debian kernel: [] (rx_submit+0x0/0x2a0) from [] (rx_complete+0x734/0x798)
Jan 15 11:03:28 debian kernel: r8:c10dabc0 r7:c10daf40 r6:80000193 r5:c1b5e240 r4:c1b11ba0
Jan 15 11:03:28 debian kernel: [] (rx_complete+0x0/0x798) from [] (usb_hcd_giveback_urb+0x78/0xc0)
Jan 15 11:03:28 debian kernel: [] (usb_hcd_giveback_urb+0x0/0xc0) from [] (ehci_urb_done+0xb0/0xbc)
Jan 15 11:03:28 debian kernel: r5:c193c000 r4:c10daf40
Jan 15 11:03:28 debian kernel: [] (ehci_urb_done+0x0/0xbc) from [] (qh_completions+0xac/0x3c0)
Jan 15 11:03:28 debian kernel: r6:ffffff8d r5:c2041458 r4:c1a8c240 r3:c2041420
Jan 15 11:03:28 debian kernel: [] (qh_completions+0x0/0x3c0) from [] (ehci_work+0xb8/0x918)
Jan 15 11:03:28 debian kernel: [] (ehci_work+0x0/0x918) from [] (ehci_irq+0x248/0x294)
Jan 15 11:03:28 debian kernel: [] (ehci_irq+0x0/0x294) from [] (usb_hcd_irq+0x38/0x48)
Jan 15 11:03:28 debian kernel: [] (usb_hcd_irq+0x0/0x48) from [] (handle_irq_event_percpu+0x38/0x198)
Jan 15 11:03:28 debian kernel: r4:c1976240 r3:c02d0414
Jan 15 11:03:28 debian kernel: [] (handle_irq_event_percpu+0x0/0x198) from [] (handle_irq_event+0x5c/0x7c)
Jan 15 11:03:28 debian kernel: [] (handle_irq_event+0x0/0x7c) from [] (handle_fasteoi_irq+0xac/0x110)
Jan 15 11:03:28 debian kernel: r5:00000000 r4:c180edc0
Jan 15 11:03:28 debian kernel: [] (handle_fasteoi_irq+0x0/0x110) from [] (generic_handle_irq+0x28/0x38)
Jan 15 11:03:28 debian kernel: r4:0000010a r3:c006df34
Jan 15 11:03:28 debian kernel: [] (generic_handle_irq+0x0/0x38) from [] (handle_IRQ+0x68/0x8c)
Jan 15 11:03:28 debian kernel: r4:0000010a r3:00000160
Jan 15 11:03:28 debian kernel: [] (handle_IRQ+0x0/0x8c) from [] (gic_handle_irq+0x3c/0x54)
Jan 15 11:03:28 debian kernel: r5:c05e8074 r4:c6802000
Jan 15 11:03:28 debian kernel: [] (gic_handle_irq+0x0/0x54) from [] (__irq_svc+0x40/0x70)
Jan 15 11:03:28 debian kernel: Exception stack(0xc05e1f28 to 0xc05e1f70)
Jan 15 11:03:28 debian kernel: 1f20: 00000001 00000000 c05e1f68 00000000 c05e0000 c0623088
Jan 15 11:03:28 debian kernel: 1f40: c05e804c c05d9b24 c072e500 412fc093 00000000 c05e1f7c c05e1f80 c05e1f70
Jan 15 11:03:28 debian kernel: 1f60: c000e60c c000e610 60000013 ffffffff
Jan 15 11:03:28 debian kernel: r6:ffffffff r5:60000013 r4:c000e610 r3:c000e60c
Jan 15 11:03:28 debian kernel: [] (default_idle+0x0/0x34) from [] (cpu_idle+0x74/0xc0)
Jan 15 11:03:28 debian kernel: [] (cpu_idle+0x0/0xc0) from [] (rest_init+0x70/0x88)
Jan 15 11:03:28 debian kernel: r6:ffffffff r5:c0622fc0 r4:00000002 r3:00000000
Jan 15 11:03:28 debian kernel: [] (rest_init+0x0/0x88) from [] (start_kernel+0x2b0/0x308)
Jan 15 11:03:28 debian kernel: r4:c05e8148 r3:c05fa484
Jan 15 11:03:28 debian kernel: [] (start_kernel+0x0/0x308) from [<40008040>] (0x40008040)
Jan 15 11:03:28 debian kernel: Mem-info:
Jan 15 11:03:28 debian kernel: Normal per-cpu:
Jan 15 11:03:28 debian kernel: CPU 0: hi: 18, btch: 3 usd: 14
Jan 15 11:03:28 debian kernel: active_anon:3650 inactive_anon:502 isolated_anon:0
Jan 15 11:03:28 debian kernel: active_file:2819 inactive_file:4647 isolated_file:0
Jan 15 11:03:28 debian kernel: unevictable:0 dirty:134 writeback:0 unstable:0
Jan 15 11:03:28 debian kernel: free:318 slab_reclaimable:503 slab_unreclaimable:1486
Jan 15 11:03:28 debian kernel: mapped:1499 shmem:538 pagetables:96 bounce:0
Jan 15 11:03:28 debian kernel: Normal free:1272kB min:1236kB low:1544kB high:1852kB active_anon:14600kB inactive_anon:2008kB active_file:11276kB inactive_file:18588kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:97536kB mlocked:0kB dirty:536kB writeback:0kB mapped:5996kB shmem:2152kB slab_reclaimable:2012kB slab_unreclaimable:5944kB kernel_stack:512kB pagetables:384kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Jan 15 11:03:28 debian kernel: lowmem_reserve[]: 0 0 0
Jan 15 11:03:28 debian kernel: Normal: 318*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1272kB
Jan 15 11:03:28 debian kernel: 8010 total pagecache pages
Jan 15 11:03:28 debian kernel: 0 pages in swap cache
Jan 15 11:03:28 debian kernel: Swap cache stats: add 0, delete 0, find 0/0
Jan 15 11:03:28 debian kernel: Free swap = 0kB
Jan 15 11:03:28 debian kernel: Total swap = 0kB
Jan 15 11:03:28 debian kernel: SLAB: Unable to allocate memory on node 0 (gfp=0x20)
Jan 15 11:03:28 debian kernel: cache: size-32768, object size: 32768, order: 3
Jan 15 11:03:28 debian kernel: node 0: slabs: 5/5, objs: 5/5, free: 0
Jan 15 11:03:28 debian kernel: swapper: page allocation failure: order:3, mode:0x20
Jan 15 11:03:28 debian kernel: Backtrace:
Jan 15 11:03:28 debian kernel: [] (dump_backtrace+0x0/0x110) from [] (dump_stack+0x18/0x1c)
Jan 15 11:03:28 debian kernel: r6:00000000 r5:00000020 r4:00000002 r3:c05fa484
Jan 15 11:03:28 debian kernel: [] (dump_stack+0x0/0x1c) from [] (warn_alloc_failed+0x100/0x120)
Jan 15 11:03:28 debian kernel: [] (warn_alloc_failed+0x0/0x120) from [] (__alloc_pages_nodemask+0x578/0x5fc)
Jan 15 11:03:28 debian kernel: r3:c05e1ce4 r2:00000000
Jan 15 11:03:28 debian kernel: r7:00000003 r6:c05e0000 r5:c05e0000 r4:00000020
Jan 15 11:03:28 debian kernel: [] (__alloc_pages_nodemask+0x0/0x5fc) from [] (cache_alloc_refill+0x2d4/0x5a8)
Jan 15 11:03:28 debian kernel: [] (cache_alloc_refill+0x0/0x5a8) from [] (__kmalloc+0x98/0xec)
Jan 15 11:03:28 debian kernel: [] (__kmalloc+0x0/0xec) from [] (__alloc_skb+0x50/0xec)
Jan 15 11:03:28 debian kernel: r7:c18195c0 r6:c1b30600 r5:00004041 r4:00000020
Jan 15 11:03:28 debian kernel: [] (__alloc_skb+0x0/0xec) from [] (rx_submit+0x30/0x2a0)
Jan 15 11:03:28 debian kernel: r8:0000000a r7:c10daf40 r6:00004000 r5:00000004 r4:c1b11ba0
Jan 15 11:03:28 debian kernel: r3:ffffffff
Jan 15 11:03:28 debian kernel: [] (rx_submit+0x0/0x2a0) from [] (hw_bh+0x1c8/0x2cc)
Jan 15 11:03:28 debian kernel: r8:0000000a r7:0000000a r6:00000003 r5:00000004 r4:c1b11ba0
Jan 15 11:03:28 debian kernel: [] (hw_bh+0x0/0x2cc) from [] (tasklet_action+0x70/0xa8)
Jan 15 11:03:28 debian kernel: r7:00000102 r6:c18ee480 r5:00000000 r4:c05fa4cc
Jan 15 11:03:28 debian kernel: [] (tasklet_action+0x0/0xa8) from [] (__do_softirq+0x90/0x128)
Jan 15 11:03:28 debian kernel: r6:c0634d18 r5:00000001 r4:c05e0000 r3:c00238c4
Jan 15 11:03:28 debian kernel: [] (__do_softirq+0x0/0x128) from [] (irq_exit+0x50/0xa4)
Jan 15 11:03:28 debian kernel: [] (irq_exit+0x0/0xa4) from [] (handle_IRQ+0x6c/0x8c)
Jan 15 11:03:28 debian kernel: [] (handle_IRQ+0x0/0x8c) from [] (gic_handle_irq+0x3c/0x54)
Jan 15 11:03:28 debian kernel: r5:c05e8074 r4:c6802000
Jan 15 11:03:28 debian kernel: [] (gic_handle_irq+0x0/0x54) from [] (__irq_svc+0x40/0x70)
Jan 15 11:03:28 debian kernel: Exception stack(0xc05e1f28 to 0xc05e1f70)
この件に関しまして、何かアドバイスを頂けますと、大変ありがたいです。何卒よろしくお願い申し上げます。
コメント
togotoday
at_takashi.sasayama
2015年1月19日 16時50分
笹山です。
> Armadillo-840を使用させていただいております。
> 連続稼働試験を実施していますが、３〜４日連続で稼働すると、以下のようなメモリーallocエラーが発生し、
> アプリ（Java）が落ちます（swapper: page allocation failure）。連続稼働の目標日数は30日ですので、
> こんなに早くアプリが落ちるとは、ということで困っています。以下ログになります。
> ...
> Jan 15 10:53:58 debian kernel: [ 3266] 105 3266 73610 9353 0 0 0 java
java の total_vm が 73610 ですので、
約 280MB 近くメモリを必要とした形跡がありますね。
他のプロセスに比べると１桁使用量が違っています。
javaアプリが大きくメモリを使用する設計になっていたりしますか?
メモリがまったく足りない状況になっていますので、
スワップ領域を作成してみてはいかがでしょうか?
以下は、256MBのスワップ領域を作成する手順です。
Armadillo-840 を Debian で起動し、スワップが有効でないことを free コマンドで確認します。
root@debian:~# free total used free shared buffers cached Mem: 382712 78424 304288 0 2360 20164 -/+ buffers/cache: 55900 326812 Swap: 0 0 0 # Swap: total が 0 であることを確認する。スワップ領域となる 256MBのファイルを作成後、スワップを有効化します。
再度、freeコマンドを実行して、スワップ領域のサイズを確認します。
root@debian:~# free total used free shared buffers cached Mem: 382712 346264 36448 0 2724 282332 -/+ buffers/cache: 61208 321504 Swap: 262140 # Swap: total が 約256MBに設定されていることを確認する。スワップを無効にする場合は、以下のコマンドを実行してください。