Message-ID: <aTbmNdilvUq78bjc@pathway>
Date: Mon, 8 Dec 2025 15:52:37 +0100
From: Petr Mladek <pmladek@...e.com>
To: Breno Leitao <leitao@...ian.org>
Cc: kuba@...nel.org, Andrew Lunn <andrew+netdev@...n.ch>,
	"David S. Miller" <davem@...emloft.net>,
	Eric Dumazet <edumazet@...gle.com>, Paolo Abeni <pabeni@...hat.com>,
	Shuah Khan <shuah@...nel.org>, Simon Horman <horms@...nel.org>,
	Jonathan Corbet <corbet@....net>, netdev@...r.kernel.org,
	linux-kernel@...r.kernel.org, linux-kselftest@...r.kernel.org,
	linux-doc@...r.kernel.org, gustavold@...il.com, asantostc@...il.com,
	calvin@...nvd.org, kernel-team@...a.com, davej@...emonkey.org.uk
Subject: Re: [PATCH net-next 0/4] (no cover subject)

On Fri 2025-12-05 02:21:08, Breno Leitao wrote:
> On Thu, Dec 04, 2025 at 11:51:58AM +0100, Petr Mladek wrote:
> > > > > > perhaps it should be configured to only log messages at a high level?  
> > > > > 
> > > > > Chris is actually working on per-console log levels to solve exactly
> > > > > this problem, so we could filter serial console messages while keeping
> > > > > everything in other consoles (aka netconsole):
> > > > > 
> > > > > 	https://lore.kernel.org/all/cover.1764272407.git.chris@chrisdown.name/
> > > > 
> > > > Excellent! Unless I'm missing more context Chris does seem to be
> > > > attacking the problem at a more suitable layer.
> > > 
> > > This would help to bypass slow serial consoles. But the extra messages
> > > would still get stored into the kernel ring buffer and passed back
> > > to user space logs, for example journalctl.
> > 
> > It might actually make sense for the "workload enters or leaves" messages.
> > But I am not sure about the "ping" messages.
> 
> Agree. Let me back up and explain my "ping" messages better, which
> I think might add more information about this topic.
> 
> Meta has millions of servers, and all of them must have netconsole
> running 100% of the time.
> 
> Of course that this is not reality, and problems happen for different
> reasons, the ones that interest me here are:
> 
> 1) Top of the rack switch MAC address changes (mostly associated with
>    network hardware (top of the rack switches and gateway) replacement)
>     a) Keep in mind that netconsole target has the destination MAC as
>        part of its configuration.
> 
> 2) Netconsole got associated with the wrong network port, which comes in
>    two different flavors.
>    a) The machine got provisioned wrongly since day one (Most common
>       case)
>    b) The machine NIC changed and: 
>       i) The target doesn't bind correctly anymore (if netconsole
>          target is bound by mac address)
>       	   * This is easier to detect, given the target will never be
> 	     enabled.
> 
> 3) Netconsd (the daemon that listen to netconsole packets) is buggy or
>    dead
> 
> 4) Network failures across the route
> 
> 
> Possible Solutions
> ==================
> 
> In order to detect those issues above, I think the best (or only) way is
> to send messages from the host, and check if they got received. If not,
> raise an alarm (in the common distributed way).
> 
> This could be done in very different ways, tho. Such as:
> 
> 1) Have a binary in each machine:
> 	a) This binary reads the netconsole target that is configured,
> 	   and mimics "ping" UDP netconsole packet.
> 
> 	Pro: 
> 	     * It doesn't need any kernel change
> 	Cons:
> 	     * It needs to reimplement the netconsole logic in userspace
> 	     * This needs also a widely distributed binary on all
> 	       machines
> 
> 2) Send a ping directly to the console
> 	a) Something as 'echo ping from $hostname" > /dev/kmsg')
> 
> 	Pro:
> 		* No kernel changes
> 	Cons:
> 		* These debug messages will be sent to journalctl and to
> 		  the console, polluting both
>
> 3) Using per-loglevel patchset.
> 	a) Same as above, but, setting netconsole loglevel to DEBUG, while
> 	   all other consoles to INFO.
> 
> 	Pro:
> 		* No changes on netconsole
> 		* Netconsole "buffers" continues to be synchronized with
> 		  kernel buffers. Everything in the same page, but,
> 		  netconsole data has one loglevel higher.
> 		* Sending a message to netconsole-only message is not
> 		  special at all. It uses the same workflow we have
> 		  today, through `/dev/kmsg'
> 	Cons:
> 		* Needs to change printk/console code (Chris' patch)
> 		  that is on review for years now. Will it ever get
> 		  accepted?
> 		* These "ping" message will be in kernel buffers and
> 		  journalctl, and are useless in there (!?)
> 		* It is not possible to send a message to a single
> 		  netconsole target.

JFYI, I am going to review the last version of the per-console
loglevel patchset later this week. IMHO, we are very close to
get it merged.

BTW: How often do you ping the netconsole, please?

     IMHO, adding a short message once-per-hour might be bearable,
     once-per-minute might be questionable for the kernel buffer
     but still fine for journalctl.

     Also it depends on the size of the kernel buffer and whether
     you use a crash dump. I mean that it might be handy to have
     some useful messages in the kernel buffer when the crash dump
     is generated and used for debugging. Otherwise, the only
     important thing is whether they get stored externally either
     via console or journalctl.


> 4) send messages only to netconsole (this patchset)
> 	Pro:
> 		* It is easy to test netconsole connective (problem above),
> 		  without kernel buffers/journal pollution
> 		* It doesn't depend on the per-loglevel patchset
> 		* Adds flexibility to netconsole targets.
> 			- only certain netconsole targets receive
> 			  certain messages
> 	Cons:
> 		* Messages sent to netconsole is a superset of messages in the
> 		  kernel buffer. In other words, "dmesg" and machine
> 		  logs/journal will not be able to see messages that
> 		  were sent directly to netconsole.
> 			- It might be seen as a back channel (!?)
> 		* Different netconsole targets may receive different
> 		  messages. Too much flexibility might be bad (!?)

I do not have strong opinion about this.

That said, the location /sys/kernel/config/netconsole/<target>/send_msg
looks a bit weird to me. I would rather expect /dev/netconsole_msg
or so. But I do not have strong opinion. It might be an overkill.

How important is it to trigger the ping from userspace, please?
It might be sent by an existing watchdog.

Best Regards,
Petr

Powered by blists - more mailing lists