Message-Id: <1420730220-20224-1-git-send-email-hadarh@mellanox.com>
Date:	Thu,  8 Jan 2015 17:16:57 +0200
From:	Hadar Hen Zion <hadarh@...lanox.com>
To:	"David S. Miller" <davem@...emloft.net>, gregkh@...uxfoundation.org
Cc:	netdev@...r.kernel.org, Amir Vadai <amirv@...lanox.com>,
	Hadar Har-Zion <hadarh@...lanox.com>,
	Yevgeny Petrilin <yevgenyp@...lanox.com>,
	Or Gerlitz <ogerlitz@...lanox.com>, shannon.nelson@...el.com,
	Doug Ledford <dledford@...hat.com>, greearb@...delatech.com
Subject: [RFC net-next 0/3] devconf: New infrastructure for setting pre-load parameters for a module

Hi,
    
When configuring a device at an early boot stage, most kernel drivers
use module parameters (the parameters' settings can be determined in
modprobe.d config files).
These parameters are difficult to manage, and one of the reasons is that
module parameters are set per driver and not per device (NICs using the
same driver cannot be set with different configurations).
Furthermore, using other existing configuration tools like ethtool,
ifconfig, ip link commands or sysfs entries is not applicable, since
they all rely on having a netdev already set up.

In the past, 'request_firmware' solution for configuration parameters
was suggested by Shannon Nelson from Intel[1]. The idea was rejected by
Greg KH, who claimed it was abusive of the request_firmware mechanism.
Greg suggested using configfs for device configuration instead (as done
by the USB gadget driver).

As a solution, we introduce a new kernel infrastructure using configfs
to allow the configuration of the device. The goal is to set low-level
device functionality that needs to be sorted out before a module is
loaded.

Configfs Solution:
----------------------
The implemented configfs solution is composed of one generic module
('devconf') that can load configuration modules per driver.
The devconf should be loaded at a very early boot stage, before other
kernel modules are loaded (or compiled as in-tree). After configfs is
mounted, it creates a new directory under configfs, called 'devices'.

In the example presented below, systemd mechanism is being used, but the
configuration can also work with older init systems.

1. A systemd service, scheduled by the OS to run before kernel modules
are loaded, creates a directory under 'devices' with the name of the driver.
The devconf module will try to load a configuration module for this
driver (if exists).

2. Drivers using this mechanism will be made of two parts:
a configuration module and the driver itself.

3. Later on, when the OS loads the driver, it uses the configuration
sub-tree.

To avoid dependencies between the modules, in case the configuration
module does not exist or is not loaded, the driver will use default
values.

Example:
--------
The below mlx4_core configuration file is only an example. Since the
infrastructure is generic, each vendor can choose how to identify its
devices (for example: bdf, mac address, uuid etc.).

If /etc/devconf.d/mlx4_core.conf is as follows (see below), systemd
service will use it to run the following commands (see below).

mlx4_core.conf file:
--------------------
[pdevs]
	[0000:00:08.0]
		dmfs = 0
		[ports]
			[1]
			type = 2
			[2]
			type =2
	[0000:00:04.0]
		dmfs = 0
		[ports]
			[1]
			type = 5
			[2]
			type =1

Systemd commands:
-----------------
$ mkdir /sys/kernel/config/devices/mlx4_core

devconf module will try to load dev_c_mlx4.ko

$ mkdir /sys/kernel/config/devices/mlx4_core/pdevs
$ mkdir /sys/kernel/config/devices/mlx4_core/pdevs/0000:00:08.0
$ mkdir /sys/kernel/config/devices/mlx4_core/pdevs/0000:00:08.0/ports
$ mkdir /sys/kernel/config/devices/mlx4_core/pdevs/0000:00:08.0/ports/1
$ mkdir /sys/kernel/config/devices/mlx4_core/pdevs/0000:00:08.0/ports/2

$ echo 5 > /sys/kernel/config/devices/mlx4_core/pdevs/0000:00:08.0/ports/1/type
$ echo 1 > /sys/kernel/config/devices/mlx4_core/pdevs/0000:00:08.0/ports/2/type
$ echo 0 > /sys/kernel/config/devices/mlx4_core/pdevs/0000:00:08.0/dmfs
[...]

dev_c_mlx4 will reject the above configuration, printing a clear message
to dmesg:
"mlx4_core: Unknown port type '5' for device 0000:00:08.0. Use 0 for
auto, 1 for IB and 2 for ETH"

The driver configuration module validates and prepares configfs sub-tree
for the driver itself.
A configuration sub-tree for a specific module will be created under
/sys/kernel/config/devices/<driver name>, and propagated from a
configuration file on the OS (located under: /etc/devconf.d/).

Configfs Sub-tree Example (for mlx4_core):
------------------------------------------
$ cd /sys/kernel/config/
$ tree devices/
devices/
└── mlx4_core
    └── pdevs
	    ├── 0000:00:04.0
	    │   ├── dmfs
	    │   └── ports
	    │       ├── 1
	    │       │   └── type
	    │       └── 2
	    │		└── type
	    └── 0000:00:08.0
	 	├── dmfs
	 	└── ports
	 	    ├── 1
	 	    │ 	└── type
	 	    └── 2
	 		└── type

The systemd service that populates configfs is part of the sysinit
target. The service is executed after mounting configfs and before udev
load kernel modules.

To use devconf infrastructure, the following should be included:
1. Devconf systemd service
2. Configuration file in the right format under: /etc/devconf.d/

This suggested solution is generic and designed to suite any type of
device. The goal is to make this solution generic enough so all kinds of
drivers are able to use it.
As a start, this RFC is being sent to the netdev mailing list for
feedback, but will eventually be submitted to the LKML mailing list as a
generic kernel infrastructure.

Hadar.

[1] - https://lkml.org/lkml/2013/1/10/606

Hadar Hen Zion (3):
  net/dev_c_mlx4: Support mlx4_core pre-load configuration
  devconf: Add configuration module for setting pre-load parameters
  net/mlx4_core: Set port_type value according to configuration module

 Documentation/filesystems/configfs/devconf.txt |  135 ++++++++++
 drivers/Kconfig                                |    2 +
 drivers/Makefile                               |    1 +
 drivers/devconf/Kconfig                        |    6 +
 drivers/devconf/Makefile                       |    3 +
 drivers/devconf/driver.c                       |  160 ++++++++++++
 drivers/devconf/main.c                         |  103 ++++++++
 drivers/net/ethernet/mellanox/mlx4/Kconfig     |    6 +
 drivers/net/ethernet/mellanox/mlx4/Makefile    |    4 +
 drivers/net/ethernet/mellanox/mlx4/main.c      |   55 ++++-
 drivers/net/ethernet/mellanox/mlx4/mlx4.h      |    1 +
 drivers/net/ethernet/mellanox/mlx4/mlx4_conf.c |  323 ++++++++++++++++++++++++
 include/linux/devconf.h                        |   69 +++++
 13 files changed, 867 insertions(+), 1 deletions(-)
 create mode 100644 Documentation/filesystems/configfs/devconf.txt
 create mode 100644 drivers/devconf/Kconfig
 create mode 100644 drivers/devconf/Makefile
 create mode 100644 drivers/devconf/driver.c
 create mode 100644 drivers/devconf/main.c
 create mode 100644 drivers/net/ethernet/mellanox/mlx4/mlx4_conf.c
 create mode 100644 include/linux/devconf.h

-- 
1.7.8.2

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Powered by blists - more mailing lists