On Sat, 2012-05-26 at 06:34 -0700, Anton Vorontsov wrote:
Hi all,
Sorry for the late reply, Monday was US holiday, and I've been working days on finding (and fixing) a very nasty bug.
With this support kernel can save functions call chain log into a persistent ram buffer that can be decoded and dumped after reboot through pstore filesystem. It can be used to determine what function was last called before a hang or an unexpected reset (caused by, for example, a buggy driver that abuses HW).
Nice!
Here's a "nano howto", to get the idea:
# mount -t debugfs debugfs /sys/kernel/debug/ # cd /sys/kernel/debug/tracing # echo persistent > current_tracer # reboot -f [...] # mount -t pstore pstore /mnt/ # tail /mnt/ftrace-ramoops 0 ffffffff8101ea64 ffffffff8101bcda native_apic_mem_read <- disconnect_bsp_APIC+0x6a/0xc0 0 ffffffff8101ea44 ffffffff8101bcf6 native_apic_mem_write <- disconnect_bsp_APIC+0x86/0xc0 0 ffffffff81020084 ffffffff8101a4b5 hpet_disable <- native_machine_shutdown+0x75/0x90 0 ffffffff81005f94 ffffffff8101a4bb iommu_shutdown_noop <- native_machine_shutdown+0x7b/0x90 0 ffffffff8101a6a1 ffffffff8101a437 native_machine_emergency_restart <- native_machine_restart+0x37/0x40 0 ffffffff811f9876 ffffffff8101a73a acpi_reboot <- native_machine_emergency_restart+0xaa/0x1e0 0 ffffffff8101a514 ffffffff8101a772 mach_reboot_fixups <- native_machine_emergency_restart+0xe2/0x1e0 0 ffffffff811d9c54 ffffffff8101a7a0 __const_udelay <- native_machine_emergency_restart+0x110/0x1e0 0 ffffffff811d9c34 ffffffff811d9c80 __delay <- __const_udelay+0x30/0x40 0 ffffffff811d9d14 ffffffff811d9c3f delay_tsc <- __delay+0xf/0x20
Mostly the code comes from trace_persistent.c driver found in the Android git tree, written by Colin Cross ccross@android.com (according to sign-off history). I reworked the driver a little bit, and ported it to pstore subsystem.
The patches depend on a pile of other pstore-related work, so if anyone is willing to try the patches, they would rather grab the whole git tree: git://git.infradead.org/users/cbou/linux-pstore.git or gitweb: http://git.infradead.org/users/cbou/linux-pstore.git
Note that so far I've tried to not change the original idea, but if we consider inclusion, there are some open questions:
- Should we merge persistent tracer with normal function tracer, i.e. add some flag that makes function tracer to duplicate the events into pstore (via a callback to pstore)?
What about making it a trace option? That is, in .../debug/tracing/options, have 'persistent' and when that's selected, we change to using the persistent buffer. This could be for both function tracing and event tracing.
I'm currently working on getting 'multi' buffers working. I plan on presenting it at Linux Plumbers for more feedback. But it will be adding a lot of hooks that would probably make this rather trivial to implement.
-- Steve
- If we keep the two separate, should the code live in fs/pstore or kernel/trace?.. I can see valid points for both approaches.
Thanks,