In public cloud scenario, if kdump service works abnormally, users cannot get vmcore. Without vmcore, user has no idea why the kernel crashed. Meanwhile, there is no additional information to find the reason why the kdump service is abnormal.
One way is to obtain console messages through VNC. The drawback is that VNC is real-time, if user missed the timing to get the VNC output, the crash needs to be retriggered.
Another way is to enable the console frontend of pstore and record the console messages to the pstore backend. On the one hand, the console logs only contain kernel printk logs and does not cover user-mode print logs. Although we can redirect user-mode logs to the pmsg frontend provided by pstore, user-mode information related to booting and kdump service vary from systemd, kdump.sh, and so on which makes redirection troublesome. So we added a tty frontend and save all logs of tty driver to the pstore backend.
Another problem is that currently pstore only supports a single backend. For debugging kdump problems, we hope to save the console logs and tty logs to the ramoops backend of pstore, as it will not be lost after rebooting. If the user has enabled another backend, the ramoops backend will not be registered. To this end, we add the multi-backend function to support simultaneous registration of multiple backends.
Based on the above changes, we can enable pstore in the crashdump kernel and save the console logs and tty logs to the ramoops backend of pstore. After rebooting, we can view the relevant logs by mounting the pstore file system.
Furthermore, we also modified kexec-tools referring to crash-utils for reading memory, so that pstore ramoops information can be read without enabling pstore in first kernel. As we set the address and size of ramoops, as well as the sizes of console and tty, we can infer the physical address of console logs and tty logs in memory. Referring to the read method of crash-utils, the console logs and tty logs are read from the memory, user can get pstore debug information without affecting the first kernel at all.
kexec-tools modification can be seen at https://github.com/shuyuanmen/kexec-tools/blob/main/Add-pstore-segment.patch
Yuanhe Shu (5): pstore: add tty frontend pstore: add multi-backends support pstore: add subdirs for multi-backends pstore: remove the module parameter "backend" tools/pstore: update pstore selftests
drivers/tty/n_tty.c | 1 + fs/pstore/Kconfig | 23 ++ fs/pstore/Makefile | 2 + fs/pstore/blk.c | 10 + fs/pstore/ftrace.c | 22 +- fs/pstore/inode.c | 86 ++++++- fs/pstore/internal.h | 16 +- fs/pstore/platform.c | 238 ++++++++++++-------- fs/pstore/pmsg.c | 23 +- fs/pstore/ram.c | 40 +++- fs/pstore/tty.c | 56 +++++ fs/pstore/zone.c | 42 +++- include/linux/pstore.h | 33 +++ include/linux/pstore_blk.h | 3 + include/linux/pstore_ram.h | 1 + include/linux/pstore_zone.h | 2 + include/linux/tty.h | 14 ++ tools/testing/selftests/pstore/common_tests | 4 - 18 files changed, 500 insertions(+), 116 deletions(-) create mode 100644 fs/pstore/tty.c