This patch series introduces a new configfs attribute that enables sending messages directly through netconsole without going through the kernel's logging infrastructure.
This feature allows users to send custom messages, alerts, or status updates directly to netconsole receivers by writing to /sys/kernel/config/netconsole/<target>/send_msg, without poluting kernel buffers, and sending msgs to the serial, which could be slow.
At Meta this is currently used in two cases right now (through printk by now):
a) When a new workload enters or leave the machine. b) From time to time, as a "ping" to make sure the netconsole/machine is alive.
The implementation reuses the existing message transmission functions (send_msg_udp() and send_ext_msg_udp()) to handle both basic and extended message formats.
Regarding code organization, this version uses forward declarations for send_msg_udp() and send_ext_msg_udp() functions rather than relocating them within the file. While forward declarations do add a small amount of redundancy, they avoid the larger churn that would result from moving entire function definitions.
--- Breno Leitao (4): netconsole: extract message fragmentation into send_msg_udp() netconsole: Add configfs attribute for direct message sending selftests/netconsole: Switch to configfs send_msg interface Documentation: netconsole: Document send_msg configfs attribute
Documentation/networking/netconsole.rst | 40 +++++++++++++++ drivers/net/netconsole.c | 59 ++++++++++++++++++---- .../selftests/drivers/net/netcons_sysdata.sh | 2 +- 3 files changed, 91 insertions(+), 10 deletions(-) --- base-commit: ab084f0b8d6d2ee4b1c6a28f39a2a7430bdfa7f0 change-id: 20251127-netconsole_send_msg-89813956dc23
Best regards, -- Breno Leitao leitao@debian.org
Add documentation for the new send_msg configfs attribute that allows sending custom messages directly through netconsole targets.
The documentation covers: - How to use the send_msg attribute - Key features and requirements - Use cases for direct message sending - Example of periodic health check implementation
This feature enables userspace applications to inject custom messages into the netconsole stream without going through the kernel's printk infrastructure, which is useful for application monitoring, testing, and debugging purposes.
Signed-off-by: Breno Leitao leitao@debian.org --- Documentation/networking/netconsole.rst | 40 +++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+)
diff --git a/Documentation/networking/netconsole.rst b/Documentation/networking/netconsole.rst index 4ab5d7b05cf1..229d5fe9a3b3 100644 --- a/Documentation/networking/netconsole.rst +++ b/Documentation/networking/netconsole.rst @@ -139,6 +139,7 @@ The interface exposes these parameters of a netconsole target to userspace: local_mac Local interface's MAC address (read-only) remote_mac Remote agent's MAC address (read-write) transmit_errors Number of packet send errors (read-only) + send_msg Send custom messages directly (write-only) =============== ================================= ============
The "enabled" attribute is also used to control whether the parameters of @@ -158,6 +159,45 @@ You can also update the local interface dynamically. This is especially useful if you want to use interfaces that have newly come up (and may not have existed when netconsole was loaded / initialized).
+Direct Message Sending +---------------------- + +The `send_msg` attribute allows sending custom messages directly through a +netconsole target without going through the kernel's printk infrastructure. +This is a write-only attribute that can be used to send arbitrary text to +the configured remote logging agent. + +To send a message directly:: + + echo "Custom status message" > /sys/kernel/config/netconsole/target1/send_msg + +Key features: + +* Messages can be sent only when the target is enabled +* The network interface must be up and running +* For extended targets, messages are sent with the extended header format +* For non-extended targets, messages are fragmented if they exceed the + maximum chunk size +* Messages bypass the kernel log buffer entirely + +This is useful for: + +* Sending application-level alerts or status updates +* Injecting custom markers or delimiters into the log stream +* Sending diagnostic information from userspace scripts +* Testing netconsole connectivity without generating kernel messages + +Example use case - sending periodic health checks:: + + while true; do + echo "[$(date)] System health: OK" > /sys/kernel/config/netconsole/target1/send_msg + sleep 60 + done + +.. note:: + The `send_msg` attribute requires the target to be enabled. Unlike other + parameters, you do not need to disable the target to use this attribute. + Netconsole targets defined at boot time (or module load time) with the `netconsole=` param are assigned the name `cmdline<index>`. For example, the first target in the parameter is named `cmdline0`. You can control and modify
Extract the message fragmentation logic from write_msg() into a dedicated send_msg_udp() function. This improves code readability and prepares for future enhancements.
The new send_msg_udp() function handles splitting messages that exceed MAX_PRINT_CHUNK into smaller fragments and sending them sequentially. This function is placed before send_ext_msg_udp() to maintain a logical ordering of related functions.
No functional changes - this is purely a refactoring commit.
Signed-off-by: Breno Leitao leitao@debian.org Reviewed-by: Petr Mladek pmladek@suse.com --- drivers/net/netconsole.c | 24 +++++++++++++++--------- 1 file changed, 15 insertions(+), 9 deletions(-)
diff --git a/drivers/net/netconsole.c b/drivers/net/netconsole.c index 9cb4dfc242f5..dc3bd7c9b049 100644 --- a/drivers/net/netconsole.c +++ b/drivers/net/netconsole.c @@ -1725,12 +1725,24 @@ static void write_ext_msg(struct console *con, const char *msg, spin_unlock_irqrestore(&target_list_lock, flags); }
+static void send_msg_udp(struct netconsole_target *nt, const char *msg, + unsigned int len) +{ + const char *tmp = msg; + int frag, left = len; + + while (left > 0) { + frag = min(left, MAX_PRINT_CHUNK); + send_udp(nt, tmp, frag); + tmp += frag; + left -= frag; + } +} + static void write_msg(struct console *con, const char *msg, unsigned int len) { - int frag, left; unsigned long flags; struct netconsole_target *nt; - const char *tmp;
if (oops_only && !oops_in_progress) return; @@ -1747,13 +1759,7 @@ static void write_msg(struct console *con, const char *msg, unsigned int len) * at least one target if we die inside here, instead * of unnecessarily keeping all targets in lock-step. */ - tmp = msg; - for (left = len; left;) { - frag = min(left, MAX_PRINT_CHUNK); - send_udp(nt, tmp, frag); - tmp += frag; - left -= frag; - } + send_msg_udp(nt, msg, len); } } spin_unlock_irqrestore(&target_list_lock, flags);
Add a new write-only configfs attribute "send_msg" to netconsole targets that allows sending arbitrary messages directly through netconsole.
This feature enables users to send custom messages through netconsole without having to go through the kernel logging infrastructure. Messages can be sent by simply writing to: /sys/kernel/config/netconsole/<target>/send_msg
The implementation: - Checks if the target is enabled before sending - Verifies the network interface is running - Handles both basic and extended message formats - Fragments large messages when needed for basic targets - Reuses existing send_msg_udp() and send_ext_msg_udp() functions
Unfortunately this patch has two forward declaration, which is not ideal, but, moving send_msg_udp() functions earlier would cause too many changes, and this could be done in an idependent patch.
Signed-off-by: Breno Leitao leitao@debian.org --- drivers/net/netconsole.c | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+)
diff --git a/drivers/net/netconsole.c b/drivers/net/netconsole.c index dc3bd7c9b049..245ed2584bbb 100644 --- a/drivers/net/netconsole.c +++ b/drivers/net/netconsole.c @@ -1194,6 +1194,39 @@ static const struct config_item_type userdata_type = { .ct_owner = THIS_MODULE, };
+/* Forward declarations */ +static void send_ext_msg_udp(struct netconsole_target *nt, const char *msg, + int msg_len); +static void send_msg_udp(struct netconsole_target *nt, const char *msg, + unsigned int len); + +static ssize_t send_msg_store(struct config_item *item, const char *buf, + size_t count) +{ + struct netconsole_target *nt = to_target(item); + ssize_t ret = -EINVAL; + unsigned long flags; + + mutex_lock(&dynamic_netconsole_mutex); + if (!nt->enabled) + goto out; + + if (!netif_running(nt->np.dev)) + goto out; + + spin_lock_irqsave(&target_list_lock, flags); + if (nt->extended) + send_ext_msg_udp(nt, buf, count); + else + send_msg_udp(nt, buf, count); + spin_unlock_irqrestore(&target_list_lock, flags); + + ret = count; +out: + mutex_unlock(&dynamic_netconsole_mutex); + + return ret; +} CONFIGFS_ATTR(, enabled); CONFIGFS_ATTR(, extended); CONFIGFS_ATTR(, dev_name); @@ -1205,6 +1238,7 @@ CONFIGFS_ATTR_RO(, local_mac); CONFIGFS_ATTR(, remote_mac); CONFIGFS_ATTR(, release); CONFIGFS_ATTR_RO(, transmit_errors); +CONFIGFS_ATTR_WO(, send_msg);
static struct configfs_attribute *netconsole_target_attrs[] = { &attr_enabled, @@ -1218,6 +1252,7 @@ static struct configfs_attribute *netconsole_target_attrs[] = { &attr_local_mac, &attr_remote_mac, &attr_transmit_errors, + &attr_send_msg, NULL, };
Update the netcons_sysdata test to use the configfs send_msg attribute for message injection instead of /dev/kmsg. This validates the new direct message sending functionality that bypasses the kernel's printk infrastructure.
Only move this test to the new mechanism, given the traditional printk() flow continues to be default path, and the one that should be mostly tested.
Signed-off-by: Breno Leitao leitao@debian.org --- tools/testing/selftests/drivers/net/netcons_sysdata.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tools/testing/selftests/drivers/net/netcons_sysdata.sh b/tools/testing/selftests/drivers/net/netcons_sysdata.sh index baf69031089e..27df730dc8f3 100755 --- a/tools/testing/selftests/drivers/net/netcons_sysdata.sh +++ b/tools/testing/selftests/drivers/net/netcons_sysdata.sh @@ -195,7 +195,7 @@ function runtest { # Wait for socat to start and listen to the port. wait_local_port_listen "${NAMESPACE}" "${PORT}" udp # Send the message - taskset -c "${CPU}" echo "${MSG}: ${TARGET}" > /dev/kmsg + taskset -c "${CPU}" echo "${MSG}: ${TARGET}" > "$NETCONS_PATH"/send_msg # Wait until socat saves the file to disk busywait "${BUSYWAIT_TIMEOUT}" test -s "${OUTPUT_FILE}" }
On Fri, Nov 28, 2025 at 06:20:48AM -0800, Breno Leitao wrote:
Update the netcons_sysdata test to use the configfs send_msg attribute for message injection instead of /dev/kmsg. This validates the new direct message sending functionality that bypasses the kernel's printk infrastructure.
Only move this test to the new mechanism, given the traditional printk() flow continues to be default path, and the one that should be mostly tested.
Signed-off-by: Breno Leitao leitao@debian.org
Reviewed-by: Simon Horman horms@kernel.org
On Fri, Nov 28, 2025 at 06:20:46AM -0800, Breno Leitao wrote:
Extract the message fragmentation logic from write_msg() into a dedicated send_msg_udp() function. This improves code readability and prepares for future enhancements.
The new send_msg_udp() function handles splitting messages that exceed MAX_PRINT_CHUNK into smaller fragments and sending them sequentially. This function is placed before send_ext_msg_udp() to maintain a logical ordering of related functions.
No functional changes - this is purely a refactoring commit.
Signed-off-by: Breno Leitao leitao@debian.org Reviewed-by: Petr Mladek pmladek@suse.com
Reviewed-by: Simon Horman horms@kernel.org
On Fri, Nov 28, 2025 at 06:20:47AM -0800, Breno Leitao wrote:
Add a new write-only configfs attribute "send_msg" to netconsole targets that allows sending arbitrary messages directly through netconsole.
This feature enables users to send custom messages through netconsole without having to go through the kernel logging infrastructure. Messages can be sent by simply writing to: /sys/kernel/config/netconsole/<target>/send_msg
The implementation:
- Checks if the target is enabled before sending
- Verifies the network interface is running
- Handles both basic and extended message formats
- Fragments large messages when needed for basic targets
- Reuses existing send_msg_udp() and send_ext_msg_udp() functions
Unfortunately this patch has two forward declaration, which is not ideal, but, moving send_msg_udp() functions earlier would cause too many changes, and this could be done in an idependent patch.
Signed-off-by: Breno Leitao leitao@debian.org
Reviewed-by: Simon Horman horms@kernel.org
On Fri, Nov 28, 2025 at 06:20:49AM -0800, Breno Leitao wrote:
Add documentation for the new send_msg configfs attribute that allows sending custom messages directly through netconsole targets.
The documentation covers:
- How to use the send_msg attribute
- Key features and requirements
- Use cases for direct message sending
- Example of periodic health check implementation
This feature enables userspace applications to inject custom messages into the netconsole stream without going through the kernel's printk infrastructure, which is useful for application monitoring, testing, and debugging purposes.
Signed-off-by: Breno Leitao leitao@debian.org
Reviewed-by: Simon Horman horms@kernel.org
On Fri, Nov 28, 2025 at 06:20:45AM -0800, Breno Leitao wrote:
This patch series introduces a new configfs attribute that enables sending messages directly through netconsole without going through the kernel's logging infrastructure.
Minor nit:
Hi Breno,
The patchset looks good to me. But the cover-letter (this email) has no subject.
On Fri, 28 Nov 2025 06:20:45 -0800 Breno Leitao wrote:
This patch series introduces a new configfs attribute that enables sending messages directly through netconsole without going through the kernel's logging infrastructure.
This feature allows users to send custom messages, alerts, or status updates directly to netconsole receivers by writing to /sys/kernel/config/netconsole/<target>/send_msg, without poluting kernel buffers, and sending msgs to the serial, which could be slow.
At Meta this is currently used in two cases right now (through printk by now):
a) When a new workload enters or leave the machine. b) From time to time, as a "ping" to make sure the netconsole/machine is alive.
The implementation reuses the existing message transmission functions (send_msg_udp() and send_ext_msg_udp()) to handle both basic and extended message formats.
Regarding code organization, this version uses forward declarations for send_msg_udp() and send_ext_msg_udp() functions rather than relocating them within the file. While forward declarations do add a small amount of redundancy, they avoid the larger churn that would result from moving entire function definitions.
The two questions we need to address here are : - why is the message important in the off-host message stream but not important in local dmesg stream. You mention "serial, which could be slow" - we need more details here. - why do we need the kernel API, netcons is just a UDP message, which is easy enough to send from user space. A little bit more detail about the advantages would be good to have. The 2nd point is trivial, the first one is what really gives me pause. Why do we not care about the logs on host? If the serial is very slow presumably it impacts a lot of things, certainly boot speed, so... perhaps it should be configured to only log messages at a high level?
Hello Jakub,
On Mon, Dec 01, 2025 at 04:36:22PM -0800, Jakub Kicinski wrote:
On Fri, 28 Nov 2025 06:20:45 -0800 Breno Leitao wrote:
This patch series introduces a new configfs attribute that enables sending messages directly through netconsole without going through the kernel's logging infrastructure.
This feature allows users to send custom messages, alerts, or status updates directly to netconsole receivers by writing to /sys/kernel/config/netconsole/<target>/send_msg, without poluting kernel buffers, and sending msgs to the serial, which could be slow.
At Meta this is currently used in two cases right now (through printk by now):
a) When a new workload enters or leave the machine. b) From time to time, as a "ping" to make sure the netconsole/machine is alive.
The implementation reuses the existing message transmission functions (send_msg_udp() and send_ext_msg_udp()) to handle both basic and extended message formats.
Regarding code organization, this version uses forward declarations for send_msg_udp() and send_ext_msg_udp() functions rather than relocating them within the file. While forward declarations do add a small amount of redundancy, they avoid the larger churn that would result from moving entire function definitions.
The two questions we need to address here are :
- why is the message important in the off-host message stream but not important in local dmesg stream. You mention "serial, which could be slow" - we need more details here.
Thanks for the questions, and I would like to share my view of the world. The way I see and use netconsole at my company (Meta) is a "kernel message" on steroids, where it provides more information about the system than what is available in kernel log buffers (dmesg)
These netconsole messages already have extra data, which provides information to each message, such as:
* scheduler configuration (for sched_ext contenxt) * THP memory configuration * Job/workload running * CPU id * task->curr name * etc
So, netconsole already sends extra information today that is not visible on kernel console (dmesg), and this has proved to be super useful, so useful that 16 entries are not enough and Gustavo need to do a dynamic allocation instead of limiting it to 16.
On top of that, printk() has a similar mechanism where extra data is not printed to the console. printk buffers has a dictionary of structured data attached to the message that is not printed to the screen, but, sent through netconsole.
This feature (in this patchset) is just one step ahead, giving some more power to netconsole, where extra information could be sent beyond what is in dmesg.
- why do we need the kernel API, netcons is just a UDP message, which is easy enough to send from user space. A little bit more detail about the advantages would be good to have.
The primary advantage is leveraging the existing configured netconsole infrastructure. At Meta, for example, we have a "continuous ping" mechanism configured by our Configuration Management software that simply runs 'echo "ping" > /dev/kmsg'.
A userspace solution would require deploying a binary to millons of machines, parsing /sys/kernel/configfs/netconsole/cmdline0/configs and sends packets directly.
While certainly feasible, it's less convenient than using the existing infrastructure (though I may just be looking for the easier path here).
The 2nd point is trivial, the first one is what really gives me pause. Why do we not care about the logs on host? If the serial is very slow presumably it impacts a lot of things, certainly boot speed, so...
This is spot-on - slow serial definitely impacts things like boot speed.
See my constant complains here, about slow boot
https://lore.kernel.org/all/aGVn%2FSnOvwWewkOW@gmail.com/
And the something similar in reboot/kexec path:
https://lore.kernel.org/all/sqwajvt7utnt463tzxgwu2yctyn5m6bjwrslsnupfexeml6h...
perhaps it should be configured to only log messages at a high level?
Chris is actually working on per-console log levels to solve exactly this problem, so we could filter serial console messages while keeping everything in other consoles (aka netconsole):
https://lore.kernel.org/all/cover.1764272407.git.chris@chrisdown.name/
That work has been in progress for years though, and I'm not sure when/if it'll land upstream. But if it does, we'd be able to have different log levels per console and then use your suggested approach.
Thanks for the review, and feel free to yell at me if I am missing the point, --breno
On Tue, 2 Dec 2025 02:18:44 -0800 Breno Leitao wrote:
On Mon, Dec 01, 2025 at 04:36:22PM -0800, Jakub Kicinski wrote:
On Fri, 28 Nov 2025 06:20:45 -0800 Breno Leitao wrote:
This patch series introduces a new configfs attribute that enables sending messages directly through netconsole without going through the kernel's logging infrastructure.
This feature allows users to send custom messages, alerts, or status updates directly to netconsole receivers by writing to /sys/kernel/config/netconsole/<target>/send_msg, without poluting kernel buffers, and sending msgs to the serial, which could be slow.
At Meta this is currently used in two cases right now (through printk by now):
a) When a new workload enters or leave the machine. b) From time to time, as a "ping" to make sure the netconsole/machine is alive.
The implementation reuses the existing message transmission functions (send_msg_udp() and send_ext_msg_udp()) to handle both basic and extended message formats.
Regarding code organization, this version uses forward declarations for send_msg_udp() and send_ext_msg_udp() functions rather than relocating them within the file. While forward declarations do add a small amount of redundancy, they avoid the larger churn that would result from moving entire function definitions.
The two questions we need to address here are :
- why is the message important in the off-host message stream but not important in local dmesg stream. You mention "serial, which could be slow" - we need more details here.
Thanks for the questions, and I would like to share my view of the world. The way I see and use netconsole at my company (Meta) is a "kernel message" on steroids, where it provides more information about the system than what is available in kernel log buffers (dmesg)
These netconsole messages already have extra data, which provides information to each message, such as:
- scheduler configuration (for sched_ext contenxt)
- THP memory configuration
- Job/workload running
- CPU id
- task->curr name
- etc
So, netconsole already sends extra information today that is not visible on kernel console (dmesg), and this has proved to be super useful, so useful that 16 entries are not enough and Gustavo need to do a dynamic allocation instead of limiting it to 16.
On top of that, printk() has a similar mechanism where extra data is not printed to the console. printk buffers has a dictionary of structured data attached to the message that is not printed to the screen, but, sent through netconsole.
This feature (in this patchset) is just one step ahead, giving some more power to netconsole, where extra information could be sent beyond what is in dmesg.
Having extra metadata makes sense, since the interpretation happens in a different environment. But here we're talking about having extra messages, not extra metadata.
- why do we need the kernel API, netcons is just a UDP message, which is easy enough to send from user space. A little bit more detail about the advantages would be good to have.
The primary advantage is leveraging the existing configured netconsole infrastructure. At Meta, for example, we have a "continuous ping" mechanism configured by our Configuration Management software that simply runs 'echo "ping" > /dev/kmsg'.
A userspace solution would require deploying a binary to millons of machines, parsing /sys/kernel/configfs/netconsole/cmdline0/configs and sends packets directly.
While certainly feasible, it's less convenient than using the existing infrastructure (though I may just be looking for the easier path here).
If this was your objective, instead of having a uAPI for sending arbitrary message you should be adding some "keepalive" timer / empty message sender... With the patches are posted you still need something to run the echo.
The 2nd point is trivial, the first one is what really gives me pause. Why do we not care about the logs on host? If the serial is very slow presumably it impacts a lot of things, certainly boot speed, so...
This is spot-on - slow serial definitely impacts things like boot speed.
See my constant complains here, about slow boot
https://lore.kernel.org/all/aGVn%2FSnOvwWewkOW@gmail.com/
And the something similar in reboot/kexec path:
https://lore.kernel.org/all/sqwajvt7utnt463tzxgwu2yctyn5m6bjwrslsnupfexeml6h...
perhaps it should be configured to only log messages at a high level?
Chris is actually working on per-console log levels to solve exactly this problem, so we could filter serial console messages while keeping everything in other consoles (aka netconsole):
https://lore.kernel.org/all/cover.1764272407.git.chris@chrisdown.name/
Excellent! Unless I'm missing more context Chris does seem to be attacking the problem at a more suitable layer.
That work has been in progress for years though, and I'm not sure when/if it'll land upstream. But if it does, we'd be able to have different log levels per console and then use your suggested approach.
Thanks for the review, and feel free to yell at me if I am missing the point, --breno
On Tue 2025-12-02 10:24:42, Jakub Kicinski wrote:
On Tue, 2 Dec 2025 02:18:44 -0800 Breno Leitao wrote:
On Mon, Dec 01, 2025 at 04:36:22PM -0800, Jakub Kicinski wrote:
On Fri, 28 Nov 2025 06:20:45 -0800 Breno Leitao wrote:
This patch series introduces a new configfs attribute that enables sending messages directly through netconsole without going through the kernel's logging infrastructure.
This feature allows users to send custom messages, alerts, or status updates directly to netconsole receivers by writing to /sys/kernel/config/netconsole/<target>/send_msg, without poluting kernel buffers, and sending msgs to the serial, which could be slow.
At Meta this is currently used in two cases right now (through printk by now):
a) When a new workload enters or leave the machine. b) From time to time, as a "ping" to make sure the netconsole/machine is alive.
The implementation reuses the existing message transmission functions (send_msg_udp() and send_ext_msg_udp()) to handle both basic and extended message formats.
This feature (in this patchset) is just one step ahead, giving some more power to netconsole, where extra information could be sent beyond what is in dmesg.
Having extra metadata makes sense, since the interpretation happens in a different environment. But here we're talking about having extra messages, not extra metadata.
The 2nd point is trivial, the first one is what really gives me pause. Why do we not care about the logs on host? If the serial is very slow presumably it impacts a lot of things, certainly boot speed, so...
This is spot-on - slow serial definitely impacts things like boot speed.
See my constant complains here, about slow boot
https://lore.kernel.org/all/aGVn%2FSnOvwWewkOW@gmail.com/
And the something similar in reboot/kexec path:
https://lore.kernel.org/all/sqwajvt7utnt463tzxgwu2yctyn5m6bjwrslsnupfexeml6h...
perhaps it should be configured to only log messages at a high level?
Chris is actually working on per-console log levels to solve exactly this problem, so we could filter serial console messages while keeping everything in other consoles (aka netconsole):
https://lore.kernel.org/all/cover.1764272407.git.chris@chrisdown.name/
Excellent! Unless I'm missing more context Chris does seem to be attacking the problem at a more suitable layer.
This would help to bypass slow serial consoles. But the extra messages would still get stored into the kernel ring buffer and passed back to user space logs, for example journalctl.
I do not have strong opinion whether adding the /sys/kernel/config/netconsole/<target>/send_msg is a good idea or not.
Ah, I have sent it prematurely.
On Thu 2025-12-04 11:46:21, Petr Mladek wrote:
On Tue 2025-12-02 10:24:42, Jakub Kicinski wrote:
On Tue, 2 Dec 2025 02:18:44 -0800 Breno Leitao wrote:
On Mon, Dec 01, 2025 at 04:36:22PM -0800, Jakub Kicinski wrote:
On Fri, 28 Nov 2025 06:20:45 -0800 Breno Leitao wrote:
This patch series introduces a new configfs attribute that enables sending messages directly through netconsole without going through the kernel's logging infrastructure.
This feature allows users to send custom messages, alerts, or status updates directly to netconsole receivers by writing to /sys/kernel/config/netconsole/<target>/send_msg, without poluting kernel buffers, and sending msgs to the serial, which could be slow.
At Meta this is currently used in two cases right now (through printk by now):
a) When a new workload enters or leave the machine. b) From time to time, as a "ping" to make sure the netconsole/machine is alive.
The implementation reuses the existing message transmission functions (send_msg_udp() and send_ext_msg_udp()) to handle both basic and extended message formats.
This feature (in this patchset) is just one step ahead, giving some more power to netconsole, where extra information could be sent beyond what is in dmesg.
Having extra metadata makes sense, since the interpretation happens in a different environment. But here we're talking about having extra messages, not extra metadata.
The 2nd point is trivial, the first one is what really gives me pause. Why do we not care about the logs on host? If the serial is very slow presumably it impacts a lot of things, certainly boot speed, so...
This is spot-on - slow serial definitely impacts things like boot speed.
See my constant complains here, about slow boot
https://lore.kernel.org/all/aGVn%2FSnOvwWewkOW@gmail.com/
And the something similar in reboot/kexec path:
https://lore.kernel.org/all/sqwajvt7utnt463tzxgwu2yctyn5m6bjwrslsnupfexeml6h...
perhaps it should be configured to only log messages at a high level?
Chris is actually working on per-console log levels to solve exactly this problem, so we could filter serial console messages while keeping everything in other consoles (aka netconsole):
https://lore.kernel.org/all/cover.1764272407.git.chris@chrisdown.name/
Excellent! Unless I'm missing more context Chris does seem to be attacking the problem at a more suitable layer.
This would help to bypass slow serial consoles. But the extra messages would still get stored into the kernel ring buffer and passed back to user space logs, for example journalctl.
It might actually make sense for the "workload enters or leaves" messages. But I am not sure about the "ping" messages.
I do not have strong opinion whether adding the /sys/kernel/config/netconsole/<target>/send_msg is a good idea or not.
I just wanted to point out that it is not only about slow serial consoles.
Best Regards, Petr
On Thu, Dec 04, 2025 at 11:51:58AM +0100, Petr Mladek wrote:
perhaps it should be configured to only log messages at a high level?
Chris is actually working on per-console log levels to solve exactly this problem, so we could filter serial console messages while keeping everything in other consoles (aka netconsole):
https://lore.kernel.org/all/cover.1764272407.git.chris@chrisdown.name/
Excellent! Unless I'm missing more context Chris does seem to be attacking the problem at a more suitable layer.
This would help to bypass slow serial consoles. But the extra messages would still get stored into the kernel ring buffer and passed back to user space logs, for example journalctl.
It might actually make sense for the "workload enters or leaves" messages. But I am not sure about the "ping" messages.
Agree. Let me back up and explain my "ping" messages better, which I think might add more information about this topic.
Meta has millions of servers, and all of them must have netconsole running 100% of the time.
Of course that this is not reality, and problems happen for different reasons, the ones that interest me here are:
1) Top of the rack switch MAC address changes (mostly associated with network hardware (top of the rack switches and gateway) replacement) a) Keep in mind that netconsole target has the destination MAC as part of its configuration.
2) Netconsole got associated with the wrong network port, which comes in two different flavors. a) The machine got provisioned wrongly since day one (Most common case) b) The machine NIC changed and: i) The target doesn't bind correctly anymore (if netconsole target is bound by mac address) * This is easier to detect, given the target will never be enabled.
3) Netconsd (the daemon that listen to netconsole packets) is buggy or dead
4) Network failures across the route
Possible Solutions ==================
In order to detect those issues above, I think the best (or only) way is to send messages from the host, and check if they got received. If not, raise an alarm (in the common distributed way).
This could be done in very different ways, tho. Such as:
1) Have a binary in each machine: a) This binary reads the netconsole target that is configured, and mimics "ping" UDP netconsole packet.
Pro: * It doesn't need any kernel change Cons: * It needs to reimplement the netconsole logic in userspace * This needs also a widely distributed binary on all machines
2) Send a ping directly to the console a) Something as 'echo ping from $hostname" > /dev/kmsg')
Pro: * No kernel changes Cons: * These debug messages will be sent to journalctl and to the console, polluting both
3) Using per-loglevel patchset. a) Same as above, but, setting netconsole loglevel to DEBUG, while all other consoles to INFO.
Pro: * No changes on netconsole * Netconsole "buffers" continues to be synchronized with kernel buffers. Everything in the same page, but, netconsole data has one loglevel higher. * Sending a message to netconsole-only message is not special at all. It uses the same workflow we have today, through `/dev/kmsg' Cons: * Needs to change printk/console code (Chris' patch) that is on review for years now. Will it ever get accepted? * These "ping" message will be in kernel buffers and journalctl, and are useless in there (!?) * It is not possible to send a message to a single netconsole target.
4) send messages only to netconsole (this patchset) Pro: * It is easy to test netconsole connective (problem above), without kernel buffers/journal pollution * It doesn't depend on the per-loglevel patchset * Adds flexibility to netconsole targets. - only certain netconsole targets receive certain messages Cons: * Messages sent to netconsole is a superset of messages in the kernel buffer. In other words, "dmesg" and machine logs/journal will not be able to see messages that were sent directly to netconsole. - It might be seen as a back channel (!?) * Different netconsole targets may receive different messages. Too much flexibility might be bad (!?)
Anyway, options 1 and 2 are available today. In order to make the problem easier to solve, we are deciding between approach 3) and 4), which will require kernel changes, either in printk/console or netconsole.
Sorry for the long email, I just wanted to do the brain dump about my view of the world, so, we can decide it from a community perspective in the open.
Thanks for the discussion, --breno
On Fri 2025-12-05 02:21:08, Breno Leitao wrote:
On Thu, Dec 04, 2025 at 11:51:58AM +0100, Petr Mladek wrote:
perhaps it should be configured to only log messages at a high level?
Chris is actually working on per-console log levels to solve exactly this problem, so we could filter serial console messages while keeping everything in other consoles (aka netconsole):
https://lore.kernel.org/all/cover.1764272407.git.chris@chrisdown.name/
Excellent! Unless I'm missing more context Chris does seem to be attacking the problem at a more suitable layer.
This would help to bypass slow serial consoles. But the extra messages would still get stored into the kernel ring buffer and passed back to user space logs, for example journalctl.
It might actually make sense for the "workload enters or leaves" messages. But I am not sure about the "ping" messages.
Agree. Let me back up and explain my "ping" messages better, which I think might add more information about this topic.
Meta has millions of servers, and all of them must have netconsole running 100% of the time.
Of course that this is not reality, and problems happen for different reasons, the ones that interest me here are:
Top of the rack switch MAC address changes (mostly associated with network hardware (top of the rack switches and gateway) replacement) a) Keep in mind that netconsole target has the destination MAC as part of its configuration.
Netconsole got associated with the wrong network port, which comes in two different flavors. a) The machine got provisioned wrongly since day one (Most common case) b) The machine NIC changed and: i) The target doesn't bind correctly anymore (if netconsole target is bound by mac address) * This is easier to detect, given the target will never be enabled.
Netconsd (the daemon that listen to netconsole packets) is buggy or dead
Network failures across the route
Possible Solutions
In order to detect those issues above, I think the best (or only) way is to send messages from the host, and check if they got received. If not, raise an alarm (in the common distributed way).
This could be done in very different ways, tho. Such as:
Have a binary in each machine: a) This binary reads the netconsole target that is configured, and mimics "ping" UDP netconsole packet.
Pro: * It doesn't need any kernel change Cons: * It needs to reimplement the netconsole logic in userspace * This needs also a widely distributed binary on all machines
Send a ping directly to the console a) Something as 'echo ping from $hostname" > /dev/kmsg')
Pro:
- No kernel changes
Cons:
- These debug messages will be sent to journalctl and to the console, polluting both
Using per-loglevel patchset. a) Same as above, but, setting netconsole loglevel to DEBUG, while all other consoles to INFO.
Pro:
- No changes on netconsole
- Netconsole "buffers" continues to be synchronized with kernel buffers. Everything in the same page, but, netconsole data has one loglevel higher.
- Sending a message to netconsole-only message is not special at all. It uses the same workflow we have today, through `/dev/kmsg'
Cons:
- Needs to change printk/console code (Chris' patch) that is on review for years now. Will it ever get accepted?
- These "ping" message will be in kernel buffers and journalctl, and are useless in there (!?)
- It is not possible to send a message to a single netconsole target.
JFYI, I am going to review the last version of the per-console loglevel patchset later this week. IMHO, we are very close to get it merged.
BTW: How often do you ping the netconsole, please?
IMHO, adding a short message once-per-hour might be bearable, once-per-minute might be questionable for the kernel buffer but still fine for journalctl.
Also it depends on the size of the kernel buffer and whether you use a crash dump. I mean that it might be handy to have some useful messages in the kernel buffer when the crash dump is generated and used for debugging. Otherwise, the only important thing is whether they get stored externally either via console or journalctl.
- send messages only to netconsole (this patchset) Pro:
Cons:
- It is easy to test netconsole connective (problem above), without kernel buffers/journal pollution
- It doesn't depend on the per-loglevel patchset
- Adds flexibility to netconsole targets.
- only certain netconsole targets receive certain messages
- Messages sent to netconsole is a superset of messages in the kernel buffer. In other words, "dmesg" and machine logs/journal will not be able to see messages that were sent directly to netconsole.
- It might be seen as a back channel (!?)
- Different netconsole targets may receive different messages. Too much flexibility might be bad (!?)
I do not have strong opinion about this.
That said, the location /sys/kernel/config/netconsole/<target>/send_msg looks a bit weird to me. I would rather expect /dev/netconsole_msg or so. But I do not have strong opinion. It might be an overkill.
How important is it to trigger the ping from userspace, please? It might be sent by an existing watchdog.
Best Regards, Petr
On Fri, 5 Dec 2025 02:21:08 -0800 Breno Leitao wrote:
- Have a binary in each machine:
- Send a ping directly to the console
- Using per-loglevel patchset.
- send messages only to netconsole (this patchset)
I think I was alluding that another option (not saying that it's the best but IIUC your requirements it'd be the best fit)):
5) Add a keepalive configfs knob, if set to a non-zero value netconsole will send an empty (?) message at given interval
Pros: - truly does not require a user binary to run periodically, netcons would set a timer in the kernel Cons: - does not provide the arbitrary "console bypass" message functionality
Hello Petr,
On Mon, Dec 08, 2025 at 03:52:37PM +0100, Petr Mladek wrote:
On Fri 2025-12-05 02:21:08, Breno Leitao wrote:
JFYI, I am going to review the last version of the per-console loglevel patchset later this week. IMHO, we are very close to get it merged.
BTW: How often do you ping the netconsole, please? IMHO, adding a short message once-per-hour might be bearable, once-per-minute might be questionable for the kernel buffer but still fine for journalctl.
It is not very often today, about once a week. This is mostly due to the pollution of kernel buffers.
Ideally we can set it to multiple times a day, but less than hourly.
- send messages only to netconsole (this patchset)
I do not have strong opinion about this.
That said, the location /sys/kernel/config/netconsole/<target>/send_msg looks a bit weird to me. I would rather expect /dev/netconsole_msg or so. But I do not have strong opinion. It might be an overkill.
How important is it to trigger the ping from userspace, please? It might be sent by an existing watchdog.
Medium importance, I would say. I am inclined to distributed design, where each machine detects broken netconsole and report itself. Instead of something else (a service) finding broken-netconsole hosts in the "fleet".
Something as:
1. Machine boots 2. userspace sends the ping (or knows that the ping was already sent) 3. the configuration management (chef, ansible, etc) makes sure that the message got received.
So, step number 3 needs to be executed only after step 2. Initiating the ping (step 2) from userspace is the easiest way, to control when the machine can go to step 3.
Thanks --breno
Hello Jakub,
On Tue, Dec 09, 2025 at 04:37:45PM +0900, Jakub Kicinski wrote:
On Fri, 5 Dec 2025 02:21:08 -0800 Breno Leitao wrote:
- Have a binary in each machine:
- Send a ping directly to the console
- Using per-loglevel patchset.
- send messages only to netconsole (this patchset)
I think I was alluding that another option (not saying that it's the best but IIUC your requirements it'd be the best fit)):
- Add a keepalive configfs knob, if set to a non-zero value netconsole
will send an empty (?) message at given interval
Pros:
- truly does not require a user binary to run periodically, netcons would set a timer in the kernel
Cons:
- does not provide the arbitrary "console bypass" message functionality
This is a good idea if we change it slightly. What about a "ping" configfs item that send sit when I touch it?
Something as:
# echo 1 > /sys/kernel/configs/<target>/ping And it would ping the host with a predefined "ping" message, and nothing else.
That would work, for my current problem, honestly.
One drawback compared to a more flexible "send_msg" is that I don't have complete flexibility on the message format. Thus, if I want to pass extra information such as a Nonce, timestamp, host state, interface name, health state, it will not be possible, which is fine for now, given I am NOT planning to use it at this stage.
Thanks for the idea and discussion, --breno
On Tue, 9 Dec 2025 09:46:51 -0800 Breno Leitao wrote:
I think I was alluding that another option (not saying that it's the best but IIUC your requirements it'd be the best fit)):
- Add a keepalive configfs knob, if set to a non-zero value netconsole
will send an empty (?) message at given interval
Pros:
- truly does not require a user binary to run periodically, netcons would set a timer in the kernel
Cons:
- does not provide the arbitrary "console bypass" message functionality
This is a good idea if we change it slightly. What about a "ping" configfs item that send sit when I touch it?
Something as:
# echo 1 > /sys/kernel/configs/<target>/ping
And it would ping the host with a predefined "ping" message, and nothing else.
That would work, for my current problem, honestly.
One drawback compared to a more flexible "send_msg" is that I don't have complete flexibility on the message format. Thus, if I want to pass extra information such as a Nonce, timestamp, host state, interface name, health state, it will not be possible, which is fine for now, given I am NOT planning to use it at this stage.
If you still want to tickle it from user space periodically, I guess send_msg is more flexible. I think the main advantage of keepalive would be to remove the need for periodic userspace work.
linux-kselftest-mirror@lists.linaro.org
-
Breno Leitao -
Jakub Kicinski -
Petr Mladek -
Simon Horman