netconsoleを使ってみる
netconsoleは、dmesgの出力をUDPでリモートホストへ送る機能。
この機能のポイントは、カーネルがpanicした後でも送信してくれる（無理かもしれないけど努力はする）という事。
panic時のbacktraceとかも取れる。
送信側の設定
netconsoleモジュールの引数か、カーネル引数でローカル・リモートのアドレス・ポートを指定。
フォーマットはnetconsole=src-port@src-ip/ifname,dst-port@dst-ip/dst-macって感じ。
$ sudo modprobe netconsole netconsole=6665@192.168.100.12/eth0,6666@192.168.100.11/04:0c:ce:xx:xx:xx
受信側の設定（OSXの場合）
syslogdで受けても良い。
今回は取り敢えず受けて表示したかっただけなので、netcatで。
$ nc -l -u 192.168.100.11 6666
なお、受け側ノードがLinuxの場合はncの引数フォーマットが異なるかも。
実験（送信側）
hogeを4回dmesgへ書いて、最後にpanicさせる。
$ su - # echo hoge > /dev/kmsg # echo hoge > /dev/kmsg # echo hoge > /dev/kmsg # echo hoge > /dev/kmsg # echo c > /proc/sysrq-trigger
実験（受信側）
以下の様な出力が得られる。
$ nc -l -u 192.168.100.11 6666 [ 26.660685] hoge [ 27.250444] hoge [ 27.722372] hoge [ 28.146850] hoge [ 37.013289] SysRq : Trigger a crash [ 37.013534] BUG: unable to handle kernel NULL pointer dereference at (null) [ 37.013850] IP: [<ffffffff8122f454>] sysrq_handle_crash+0xd/0x16 [ 37.014078] PGD 3d7f9067 PUD 374f1067 PMD 0 [ 37.014389] Oops: 0002 [#1] SMP [ 37.014626] CPU 0 [ 37.014705] Modules linked in: netconsole configfs nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc loop btusb bluetooth rfkill crc16 coretemp crc32c_intel ghash_clmulni_intel aesni_intel joydev aes_x86_64 aes_generic parport_pc parport vmw_balloon usbhid hid psmouse cryptd pcspkr serio_raw evdev snd_ens1371 snd_ac97_codec snd_rawmidi snd_seq_device snd_pcm snd_page_alloc snd_timer snd soundcore processor ac power_supply ac97_bus container thermal_sys gameport vmwgfx ttm i2c_piix4 button drm i2c_core shpchp ext3 mbcache jbd sg sr_mod uhci_hcd cdrom sd_mod ata_generic crc_t10dif floppy ehci_hcd usbcore e1000 usb_common ata_piix libata mptspi scsi_transport_spi mptscsih mptbase scsi_mod [last unloaded: scsi_wait_scan] [ 37.020622] [ 37.020732] Pid: 2608, comm: bash Not tainted 3.2.0-4-amd64 #1 Debian 3.2.41-2 VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform [ 37.021209] RIP: 0010:[<ffffffff8122f454>] [<ffffffff8122f454>] sysrq_handle_crash+0xd/0x16 [ 37.021460] RSP: 0018:ffff88003d72be80 EFLAGS: 00010092 [ 37.021599] RAX: 0000000000000010 RBX: ffffffff8164bb90 RCX: 00000000000022ea [ 37.021752] RDX: 0000000000000000 RSI: 0000000000000046 RDI: 0000000000000063 [ 37.021921] RBP: 000000000000000a R08: 0000000000000000 R09: 000000003bf10202 [ 37.022094] R10: 0000000000000000 R11: ffff88003770a000 R12: 0000000000000063 [ 37.022251] R13: 0000000000000246 R14: 0000000000000000 R15: 0000000000000000 [ 37.022405] FS: 00007f187b50f700(0000) GS:ffff88003f600000(0000) knlGS:0000000000000000 [ 37.022577] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 37.022703] CR2: 0000000000000000 CR3: 0000000037265000 CR4: 00000000000406f0 [ 37.023249] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 37.023477] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 37.023621] Process bash (pid: 2608, threadinfo ffff88003d72a000, task ffff88003731f160) [ 37.023853] Stack: [ 37.023951] ffffffff8122f8a7 0000000000000002 0000000000000000 ffff88003d72bf58 [ 37.024430] 0000000000000002 0000000000000001 ffffffff8122f96d ffff88003d72bf58 [ 37.025891] ffff88003bbfb540 ffff88003b419ec0 ffffffff8113f3c0 0000000000000002 [ 37.026270] Call Trace: [ 37.026375] [<ffffffff8122f8a7>] ? __handle_sysrq+0x94/0x128 [ 37.026510] [<ffffffff8122f96d>] ? write_sysrq_trigger+0x32/0x39 [ 37.026659] [<ffffffff8113f3c0>] ? proc_reg_write+0x78/0x91 [ 37.026796] [<ffffffff810f9e6d>] ? vfs_write+0xa2/0xe9 [ 37.027153] [<ffffffff810fa04a>] ? sys_write+0x45/0x6b [ 37.027267] [<ffffffff81352952>] ? system_call_fastpath+0x16/0x1b [ 37.027404] Code: 00 01 8a 81 23 91 7f 81 19 d2 83 e0 8f f7 d2 83 e2 03 c1 e2 04 09 d0 88 81 23 91 7f 81 c3 c7 05 e3 86 53 00 01 00 00 00 0f ae f8 <c6> 04 25 00 00 00 00 01 c3 53 8d 5f d0 31 c0 48 c7 c7 39 00 50 [ 37.030866] RIP [<ffffffff8122f454>] sysrq_handle_crash+0xd/0x16 [ 37.031061] RSP <ffff88003d72be80> [ 37.031152] CR2: 0000000000000000 [ 37.031246] ---[ end trace 1de8fbb6f262746a ]--- [ 37.031341] Kernel panic - not syncing: Fatal exception [ 37.031446] Pid: 2608, comm: bash Tainted: G D 3.2.0-4-amd64 #1 Debian 3.2.41-2 [ 37.031582] Call Trace: [ 37.031673] [<ffffffff81347128>] ? panic+0x95/0x1a2 [ 37.031773] [<ffffffff8134e7c6>] ? oops_end+0xa9/0xb6 [ 37.031880] [<ffffffff81346a71>] ? no_context+0x1ff/0x20e [ 37.031983] [<ffffffff8135083b>] ? do_page_fault+0x210/0x337 [ 37.032093] [<ffffffff81070c4d>] ? arch_local_irq_save+0x11/0x17 [ 37.032200] [<ffffffff8134d9a4>] ? _raw_spin_lock_irqsave+0x9/0x25 [ 37.032327] [<ffffffff810634eb>] ? up+0xb/0x34 [ 37.032425] [<ffffffff81047321>] ? console_unlock+0x1f7/0x206 [ 37.032566] [<ffffffff8134d9a4>] ? _raw_spin_lock_irqsave+0x9/0x25 [ 37.032682] [<ffffffff810477b7>] ? vprintk+0x39e/0x3d9 [ 37.032809] [<ffffffff8134df35>] ? page_fault+0x25/0x30 [ 37.032918] [<ffffffff8122f454>] ? sysrq_handle_crash+0xd/0x16 [ 37.033058] [<ffffffff8122f8a7>] ? __handle_sysrq+0x94/0x128 [ 37.033164] [<ffffffff8122f96d>] ? write_sysrq_trigger+0x32/0x39 [ 37.033277] [<ffffffff8113f3c0>] ? proc_reg_write+0x78/0x91 [ 37.033382] [<ffffffff810f9e6d>] ? vfs_write+0xa2/0xe9 [ 37.033486] [<ffffffff810fa04a>] ? sys_write+0x45/0x6b [ 37.033588] [<ffffffff81352952>] ? system_call_fastpath+0x16/0x1b
