Date:   Wed, 10 Aug 2022 19:23:00 -0700
From:   Jakub Kicinski <kuba@...nel.org>
To:     netdev@...r.kernel.org, davem@...emloft.net, edumazet@...gle.com,
        pabeni@...hat.com
Cc:     sdf@...gle.com, jacob.e.keller@...el.com, vadfed@...com,
        johannes@...solutions.net, jiri@...nulli.us, dsahern@...nel.org,
        stephen@...workplumber.org, fw@...len.de,
        linux-doc@...r.kernel.org, Jakub Kicinski <kuba@...nel.org>
Subject: [RFC net-next 0/4] ynl: YAML netlink protocol descriptions

Netlink seems simple and reasonable to those who understand it.
It appears cumbersome and arcane to those who don't.

This RFC introduces machine readable netlink protocol descriptions
in YAML, in an attempt to make creation of truly generic netlink
libraries a possibility. Truly generic netlink library here means
a library which does not require changes to support a new family
or a new operation.

Each YAML spec lists attributes and operations the family supports.
The specs are fully standalone, meaning that there is no dependency
on existing uAPI headers in C. Numeric values of all attribute types,
operations, enums, and defines and listed in the spec (or unambiguous).
This property removes the need to manually translate the headers for
languages which are not compatible with C.

The expectation is that the spec can be used to either dynamically
translate between whatever types the high level language likes (see
the Python example below) or codegen a complete libarary / bindings
for a netlink family at compilation time (like popular RPC libraries
do).

Currently only genetlink is supported, but the "old netlink" should
be supportable as well (I don't need it myself).

On the kernel side the YAML spec can be used to generate:
 - the C uAPI header
 - documentation of the protocol as a ReST file
 - policy tables for input attribute validation
 - operation tables

We can also codegen parsers and dump helpers, but right now the level
of "creativity & cleverness" when it comes to netlink parsing is so
high it's quite hard to generalize it for most families without major
refactoring.

Being able to generate the header, documentation and policy tables
should balance out the extra effort of writing the YAML spec.

Here is a Python example I promised earlier:

  ynl = YnlFamily("path/to/ethtool.yaml")
  channels = ynl.channels_get({'header': {'dev_name': 'eni1np1'}})

If the call was successful "channels" will hold a standard Python dict,
e.g.:

  {'header': {'dev_index': 6, 'dev_name': 'eni1np1'},
   'combined_max': 1,
   'combined_count': 1}

for a netdevsim device with a single combined queue.

YnlFamily is an implementation of a YAML <> netlink translator (patch 3).
It takes a path to the YAML spec - hopefully one day we will make
the YAMLs themselves uAPI and distribute them like we distribute
C headers. Or get them distributed to a standard search path another
way. Until then, the YNL library needs a full path to the YAML spec and
application has to worry about the distribution of those.

The YnlFamily reads all the info it needs from the spec, resolves
the genetlink family id, and creates methods based on the spec.
channels_get is such a dynamically-generated method (i.e. grep for
channels_get in the python code shows nothing). The method can be called
passing a standard Python dict as an argument. YNL will look up each key
in the YAML spec and render the appropriate binary (netlink TLV)
representation of the value. It then talks thru a netlink socket
to the kernel, and deserilizes the response, converting the netlink
TLVs into Python types and constructing a dictionary.

Again, the YNL code is completely generic and has no knowledge specific
to ethtool. It's fairly simple an incomplete (in terms of types
for example), I wrote it this afternoon. I'm also pretty bad at Python,
but it's the only language I can type which allows the method
magic, so please don't judge :) I have a rather more complete codegen
for C, with support for notifications, kernel -> user policy/type
verification, resolving extack attr offsets into a path
of attribute names etc, etc. But that stuff needs polishing and
is less suitable for an RFC.

The ability for a high level language like Python to talk to the kernel
so easily, without ctypes, manually packing structs, copy'n'pasting
values for defines etc. excites me more than C codegen, anyway.


Patch 1 adds a bit of documentation under Documentation/, it talks
more about the schemas themselves.

Patch 2 contains the YAML schema for the YAML specs.

Patch 3 adds the YNL Python library.

Patch 4 adds a sample schema for ethtool channels and a demo script.


Jakub Kicinski (4):
  ynl: add intro docs for the concept
  ynl: add the schema for the schemas
  ynl: add a sample python library
  ynl: add a sample user for ethtool

 Documentation/index.rst                     |   1 +
 Documentation/netlink/bindings/ethtool.yaml | 115 +++++++
 Documentation/netlink/index.rst             |  13 +
 Documentation/netlink/netlink-bindings.rst  | 104 ++++++
 Documentation/netlink/schema.yaml           | 242 ++++++++++++++
 tools/net/ynl/samples/ethtool.py            |  30 ++
 tools/net/ynl/samples/ynl.py                | 342 ++++++++++++++++++++
 7 files changed, 847 insertions(+)
 create mode 100644 Documentation/netlink/bindings/ethtool.yaml
 create mode 100644 Documentation/netlink/index.rst
 create mode 100644 Documentation/netlink/netlink-bindings.rst
 create mode 100644 Documentation/netlink/schema.yaml
 create mode 100755 tools/net/ynl/samples/ethtool.py
 create mode 100644 tools/net/ynl/samples/ynl.py

-- 
2.37.1

Powered by blists - more mailing lists