The notification handling in ynl is currently very simple, using sleep()
to wait a period of time and then handling all the buffered messages in
a single batch.
This patch adds async notification handling so that messages can be
processed as they are received. This makes it possible to use ynl as a
library that supplies notifications in a timely manner.
- Add poll_ntf() to be a generator that yields 1 notification at a
time and blocks until a notification is available.
- Add a --duration parameter to the CLI, with --sleep as an alias.
./tools/net/ynl/cli.py \
--spec <SPEC> --subscribe <TOPIC> [ --duration <SECS> ]
The cli will report any notifications for duration seconds and then
exit. If duration is not specified, then it will poll forever, until
interrupted.
Here is an example python snippet that shows how to use ynl as a library
for receiving notifications:
ynl = YnlFamily(f"{dir}/rt_route.yaml")
ynl.ntf_subscribe('rtnlgrp-ipv4-route')
for event in ynl.poll_ntf():
handle(event)
Signed-off-by: Donald Hunter <donald.hunter@gmail.com>
Link: https://patch.msgid.link/20241113090843.72917-3-donald.hunter@gmail.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
This reverts commit 1bf70e6c3a5346966c25e0a1ff492945b25d3f80.
This modification to check_ntf() is being reverted so that its behaviour
remains equivalent to ynl_ntf_check() in the C YNL. Instead a new
poll_ntf() will be added in a separate patch.
Signed-off-by: Donald Hunter <donald.hunter@gmail.com>
Link: https://patch.msgid.link/20241113090843.72917-2-donald.hunter@gmail.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
The notification handling in ynl is currently very simple, using sleep()
to wait a period of time and then handling all the buffered messages in
a single batch.
This patch changes the notification handling so that messages are
processed as they are received. This makes it possible to use ynl as a
library that supplies notifications in a timely manner.
- Change check_ntf() to be a generator that yields 1 notification at a
time and blocks until a notification is available.
- Use the --sleep parameter to set an alarm and exit when it fires.
This means that the CLI has the same interface, but notifications get
printed as they are received:
./tools/net/ynl/cli.py --spec <SPEC> --subscribe <TOPIC> [ --sleep <SECS> ]
Here is an example python snippet that shows how to use ynl as a library
for receiving notifications:
ynl = YnlFamily(f"{dir}/rt_route.yaml")
ynl.ntf_subscribe('rtnlgrp-ipv4-route')
for event in ynl.check_ntf():
handle(event)
Signed-off-by: Donald Hunter <donald.hunter@gmail.com>
Tested-by: Kory Maincent <kory.maincent@bootlin.com>
Link: https://patch.msgid.link/20241018093228.25477-1-donald.hunter@gmail.com
Signed-off-by: Paolo Abeni <pabeni@redhat.com>
I often forget the exact naming of ops and have to look at
the spec to find it. Add support for listing the operations:
$ ./cli.py --spec .../netdev.yaml --list-ops
dev-get [ do, dump ]
page-pool-get [ do, dump ]
page-pool-stats-get [ do, dump ]
queue-get [ do, dump ]
napi-get [ do, dump ]
qstats-get [ dump ]
For completeness also support listing all ops (including
notifications:
# ./cli.py --spec .../netdev.yaml --list-msgs
dev-get [ dump, do ]
dev-add-ntf [ notify ]
dev-del-ntf [ notify ]
dev-change-ntf [ notify ]
page-pool-get [ dump, do ]
page-pool-add-ntf [ notify ]
page-pool-del-ntf [ notify ]
page-pool-change-ntf [ notify ]
page-pool-stats-get [ dump, do ]
queue-get [ dump, do ]
napi-get [ dump, do ]
qstats-get [ dump ]
Use double space after the name for slightly easier to read
output.
Reviewed-by: Jiri Pirko <jiri@nvidia.com>
Reviewed-by: Donald Hunter <donald.hunter@gmail.com>
Link: https://lore.kernel.org/r/20240502164043.2130184-1-kuba@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Add a "--multi <do-op> <json>" command line to ynl that makes it
possible to add several operations to a single netlink request payload.
The --multi command line option is repeated for each operation.
This is used by the nftables family for transaction batches. For
example:
./tools/net/ynl/cli.py \
--spec Documentation/netlink/specs/nftables.yaml \
--multi batch-begin '{"res-id": 10}' \
--multi newtable '{"name": "test", "nfgen-family": 1}' \
--multi newchain '{"name": "chain", "table": "test", "nfgen-family": 1}' \
--multi batch-end '{"res-id": 10}'
[None, None, None, None]
It can also be used for bundling get requests:
./tools/net/ynl/cli.py \
--spec Documentation/netlink/specs/nftables.yaml \
--multi gettable '{"name": "test", "nfgen-family": 1}' \
--multi getchain '{"name": "chain", "table": "test", "nfgen-family": 1}' \
--output-json
[{"name": "test", "use": 1, "handle": 1, "flags": [],
"nfgen-family": 1, "version": 0, "res-id": 2},
{"table": "test", "name": "chain", "handle": 1, "use": 0,
"nfgen-family": 1, "version": 0, "res-id": 2}]
Signed-off-by: Donald Hunter <donald.hunter@gmail.com>
Link: https://lore.kernel.org/r/20240418104737.77914-4-donald.hunter@gmail.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
ynl does not handle NlError exceptions so they get reported like program
failures. Handle the NlError exceptions and report the netlink errors
more cleanly.
Example now:
Netlink error: No such file or directory
nl_len = 44 (28) nl_flags = 0x300 nl_type = 2
error: -2 extack: {'bad-attr': '.op'}
Example before:
Traceback (most recent call last):
File "/home/donaldh/net-next/./tools/net/ynl/cli.py", line 81, in <module>
main()
File "/home/donaldh/net-next/./tools/net/ynl/cli.py", line 69, in main
reply = ynl.dump(args.dump, attrs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/donaldh/net-next/tools/net/ynl/lib/ynl.py", line 906, in dump
return self._op(method, vals, [], dump=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/donaldh/net-next/tools/net/ynl/lib/ynl.py", line 872, in _op
raise NlError(nl_msg)
lib.ynl.NlError: Netlink error: No such file or directory
nl_len = 44 (28) nl_flags = 0x300 nl_type = 2
error: -2 extack: {'bad-attr': '.op'}
Signed-off-by: Donald Hunter <donald.hunter@gmail.com>
Link: https://lore.kernel.org/r/20240306231046.97158-3-donald.hunter@gmail.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Most "production" netlink clients use large buffers to
make dump efficient, which means that handling of dump
continuation in the kernel is not very well tested.
Add an option for debugging / testing handling of dumps.
It enables printing of extra netlink-level debug and
lowers the recv() buffer size in one go. When used
without any argument (--dbg-small-recv) it picks
a very small default (4000), explicit size can be set,
too (--dbg-small-recv 5000).
Example:
$ ./cli.py [...] --dbg-small-recv
Recv: read 3712 bytes, 29 messages
nl_len = 128 (112) nl_flags = 0x0 nl_type = 19
[...]
nl_len = 128 (112) nl_flags = 0x0 nl_type = 19
Recv: read 3968 bytes, 31 messages
nl_len = 128 (112) nl_flags = 0x0 nl_type = 19
[...]
nl_len = 128 (112) nl_flags = 0x0 nl_type = 19
Recv: read 532 bytes, 5 messages
nl_len = 128 (112) nl_flags = 0x0 nl_type = 19
[...]
nl_len = 128 (112) nl_flags = 0x0 nl_type = 19
nl_len = 20 (4) nl_flags = 0x2 nl_type = 3
(the [...] are edits to shorten the commit message).
Note that the first message of the dump is sized conservatively
by the kernel.
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Reviewed-by: Donald Hunter <donald.hunter@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
The ynl cli currently emits python pretty printed structures which is
hard to consume. Add a new --output-json argument to emit JSON.
Signed-off-by: Donald Hunter <donald.hunter@gmail.com>
Reviewed-by: Breno Leitao <leitao@debian.org>
Reviewed-by: Jiri Pirko <jiri@nvidia.com>
Link: https://lore.kernel.org/r/20240129223458.52046-2-donald.hunter@gmail.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
In case the kernel sends message back containing attribute not defined
in family spec, following exception is raised to the user:
$ sudo ./tools/net/ynl/cli.py --spec Documentation/netlink/specs/devlink.yaml --do trap-get --json '{"bus-name": "netdevsim", "dev-name": "netdevsim1", "trap-name": "source_mac_is_multicast"}'
Traceback (most recent call last):
File "/home/jiri/work/linux/tools/net/ynl/lib/ynl.py", line 521, in _decode
attr_spec = attr_space.attrs_by_val[attr.type]
~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
KeyError: 132
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/jiri/work/linux/./tools/net/ynl/cli.py", line 61, in <module>
main()
File "/home/jiri/work/linux/./tools/net/ynl/cli.py", line 49, in main
reply = ynl.do(args.do, attrs, args.flags)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jiri/work/linux/tools/net/ynl/lib/ynl.py", line 731, in do
return self._op(method, vals, flags)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jiri/work/linux/tools/net/ynl/lib/ynl.py", line 719, in _op
rsp_msg = self._decode(decoded.raw_attrs, op.attr_set.name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jiri/work/linux/tools/net/ynl/lib/ynl.py", line 525, in _decode
raise Exception(f"Space '{space}' has no attribute with value '{attr.type}'")
Exception: Space 'devlink' has no attribute with value '132'
Introduce a command line option "process-unknown" and pass it down to
YnlFamily class constructor to allow user to process unknown
attributes and types and print them as binaries.
$ sudo ./tools/net/ynl/cli.py --spec Documentation/netlink/specs/devlink.yaml --do trap-get --json '{"bus-name": "netdevsim", "dev-name": "netdevsim1", "trap-name": "source_mac_is_multicast"}' --process-unknown
{'UnknownAttr(129)': {'UnknownAttr(0)': b'\x00\x00\x00\x00\x00\x00\x00\x00',
'UnknownAttr(1)': b'\x00\x00\x00\x00\x00\x00\x00\x00',
'UnknownAttr(2)': b'\x0e\x00\x00\x00\x00\x00\x00\x00'},
'UnknownAttr(132)': b'\x00',
'UnknownAttr(133)': b'',
'UnknownAttr(134)': {'UnknownAttr(0)': b''},
'bus-name': 'netdevsim',
'dev-name': 'netdevsim1',
'trap-action': 'drop',
'trap-group-name': 'l2_drops',
'trap-name': 'source_mac_is_multicast'}
Signed-off-by: Jiri Pirko <jiri@nvidia.com>
Link: https://lore.kernel.org/r/20231027092525.956172-1-jiri@resnulli.us
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Add support for using NLM_F_REPLACE, _EXCL, _CREATE and _APPEND flags
in requests.
Signed-off-by: Donald Hunter <donald.hunter@gmail.com>
Reviewed-by: Jacob Keller <jacob.e.keller@intel.com>
Link: https://lore.kernel.org/r/20230825122756.7603-10-donald.hunter@gmail.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
I was intending to make all the Netlink Spec code BSD-3-Clause
to ease the adoption but it appears that:
- I fumbled the uAPI and used "GPL WITH uAPI note" there
- it gives people pause as they expect GPL in the kernel
As suggested by Chuck re-license under dual. This gives us benefit
of full BSD freedom while fulfilling the broad "kernel is under GPL"
expectations.
Link: https://lore.kernel.org/all/20230304120108.05dd44c5@kernel.org/
Link: https://lore.kernel.org/r/20230306200457.3903854-1-kuba@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
The scripts require Python 3 and some distros are dropping
Python 2 support.
Reported-by: Stanislav Fomichev <sdf@google.com>
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
The CLI script tries to validate jsonschema by default.
It's seems better to validate too many times than too few.
However, when copying the scripts to random servers having
to install jsonschema is tedious. Load jsonschema via
importlib, and let the user opt out.
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
When I wrote the first version of the Python code I was quite
excited that we can generate class methods directly from the
spec. Unfortunately we need to use valid identifiers for method
names (specifically no dashes are allowed). Don't reuse those
names on the CLI, it's much more natural to use the operation
names exactly as listed in the spec.
Instead of:
./cli --do rings_get
use:
./cli --do rings-get
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Move the CLI code out of samples/ and the library part
of it into tools/net/ynl/lib/. This way we can start
sharing some code with the code gen.
Initially I thought that code gen is too C-specific to
share anything but basic stuff like calculating values
for enums can easily be shared.
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
