Message-ID: <7jdruzcpkeyhuudwi6uzg2vsc5mhgpq7qz4ym7vqqmgs7j3524@cvtnzneddg2d>
Date: Fri, 5 Dec 2025 02:21:08 -0800
From: Breno Leitao <leitao@...ian.org>
To: Petr Mladek <pmladek@...e.com>, kuba@...nel.org
Cc: Jakub Kicinski <kuba@...nel.org>, Andrew Lunn <andrew+netdev@...n.ch>, 
	"David S. Miller" <davem@...emloft.net>, Eric Dumazet <edumazet@...gle.com>, 
	Paolo Abeni <pabeni@...hat.com>, Shuah Khan <shuah@...nel.org>, Simon Horman <horms@...nel.org>, 
	Jonathan Corbet <corbet@....net>, netdev@...r.kernel.org, linux-kernel@...r.kernel.org, 
	linux-kselftest@...r.kernel.org, linux-doc@...r.kernel.org, gustavold@...il.com, 
	asantostc@...il.com, calvin@...nvd.org, kernel-team@...a.com, davej@...emonkey.org.uk
Subject: Re: [PATCH net-next 0/4] (no cover subject)

On Thu, Dec 04, 2025 at 11:51:58AM +0100, Petr Mladek wrote:
> > > > > perhaps it should be configured to only log messages at a high level?  
> > > > 
> > > > Chris is actually working on per-console log levels to solve exactly
> > > > this problem, so we could filter serial console messages while keeping
> > > > everything in other consoles (aka netconsole):
> > > > 
> > > > 	https://lore.kernel.org/all/cover.1764272407.git.chris@chrisdown.name/
> > > 
> > > Excellent! Unless I'm missing more context Chris does seem to be
> > > attacking the problem at a more suitable layer.
> > 
> > This would help to bypass slow serial consoles. But the extra messages
> > would still get stored into the kernel ring buffer and passed back
> > to user space logs, for example journalctl.
> 
> It might actually make sense for the "workload enters or leaves" messages.
> But I am not sure about the "ping" messages.

Agree. Let me back up and explain my "ping" messages better, which
I think might add more information about this topic.

Meta has millions of servers, and all of them must have netconsole
running 100% of the time.

Of course that this is not reality, and problems happen for different
reasons, the ones that interest me here are:

1) Top of the rack switch MAC address changes (mostly associated with
   network hardware (top of the rack switches and gateway) replacement)
    a) Keep in mind that netconsole target has the destination MAC as
       part of its configuration.

2) Netconsole got associated with the wrong network port, which comes in
   two different flavors.
   a) The machine got provisioned wrongly since day one (Most common
      case)
   b) The machine NIC changed and: 
      i) The target doesn't bind correctly anymore (if netconsole
         target is bound by mac address)
      	   * This is easier to detect, given the target will never be
	     enabled.

3) Netconsd (the daemon that listen to netconsole packets) is buggy or
   dead

4) Network failures across the route


Possible Solutions
==================

In order to detect those issues above, I think the best (or only) way is
to send messages from the host, and check if they got received. If not,
raise an alarm (in the common distributed way).

This could be done in very different ways, tho. Such as:

1) Have a binary in each machine:
	a) This binary reads the netconsole target that is configured,
	   and mimics "ping" UDP netconsole packet.

	Pro: 
	     * It doesn't need any kernel change
	Cons:
	     * It needs to reimplement the netconsole logic in userspace
	     * This needs also a widely distributed binary on all
	       machines

2) Send a ping directly to the console
	a) Something as 'echo ping from $hostname" > /dev/kmsg')

	Pro:
		* No kernel changes
	Cons:
		* These debug messages will be sent to journalctl and to
		  the console, polluting both

3) Using per-loglevel patchset.
	a) Same as above, but, setting netconsole loglevel to DEBUG, while
	   all other consoles to INFO.

	Pro:
		* No changes on netconsole
		* Netconsole "buffers" continues to be synchronized with
		  kernel buffers. Everything in the same page, but,
		  netconsole data has one loglevel higher.
		* Sending a message to netconsole-only message is not
		  special at all. It uses the same workflow we have
		  today, through `/dev/kmsg'
	Cons:
		* Needs to change printk/console code (Chris' patch)
		  that is on review for years now. Will it ever get
		  accepted?
		* These "ping" message will be in kernel buffers and
		  journalctl, and are useless in there (!?)
		* It is not possible to send a message to a single
		  netconsole target.

4) send messages only to netconsole (this patchset)
	Pro:
		* It is easy to test netconsole connective (problem above),
		  without kernel buffers/journal pollution
		* It doesn't depend on the per-loglevel patchset
		* Adds flexibility to netconsole targets.
			- only certain netconsole targets receive
			  certain messages
	Cons:
		* Messages sent to netconsole is a superset of messages in the
		  kernel buffer. In other words, "dmesg" and machine
		  logs/journal will not be able to see messages that
		  were sent directly to netconsole.
			- It might be seen as a back channel (!?)
		* Different netconsole targets may receive different
		  messages. Too much flexibility might be bad (!?)

Anyway, options 1 and 2 are available today. In order to make the
problem easier to solve, we are deciding between approach 3) and 4),
which will require kernel changes, either in printk/console or
netconsole.

Sorry for the long email, I just wanted to do the brain dump about my
view of the world, so, we can decide it from a community perspective in
the open.

Thanks for the discussion,
--breno

Powered by blists - more mailing lists