** Background **
Currently, OVS supports several packet sampling mechanisms (sFlow,
per-bridge IPFIX, per-flow IPFIX). These end up being translated into a
userspace action that needs to be handled by ovs-vswitchd's handler
threads only to be forwarded to some third party application that
will somehow process the sample and provide observability on the
datapath.
A particularly interesting use-case is controller-driven
per-flow IPFIX sampling where the OpenFlow controller can add metadata
to samples (via two 32bit integers) and this metadata is then available
to the sample-collecting system for correlation.
** Problem **
The fact that sampled traffic share netlink sockets and handler thread
time with upcalls, apart from being a performance bottleneck in the
sample extraction itself, can severely compromise the datapath,
yielding this solution unfit for highly loaded production systems.
Users are left with little options other than guessing what sampling
rate will be OK for their traffic pattern and system load and dealing
with the lost accuracy.
Looking at available infrastructure, an obvious candidated would be
to use psample. However, it's current state does not help with the
use-case at stake because sampled packets do not contain user-defined
metadata.
** Proposal **
This series is an attempt to fix this situation by extending the
existing psample infrastructure to carry a variable length
user-defined cookie.
The main existing user of psample is tc's act_sample. It is also
extended to forward the action's cookie to psample.
Finally, a new OVS action (OVS_SAMPLE_ATTR_EMIT_SAMPLE) is created.
It accepts a group and an optional cookie and uses psample to
multicast the packet and the metadata.
--
v2 -> v3:
- Addressed comments from Simon, Aaron and Ilya.
- Dropped probability propagation in nested sample actions.
- Dropped patch v2's 7/9 in favor of a userspace implementation and
consume skb if emit_sample is the last action, same as we do with
userspace.
- Split ovs-dpctl.py features in independent patches.
v1 -> v2:
- Create a new action ("emit_sample") rather than reuse existing
"sample" one.
- Add probability semantics to psample's sampling rate.
- Store sampling probability in skb's cb area and use it in emit_sample.
- Test combining "emit_sample" with "trunc"
- Drop group_id filtering and tracepoint in psample.
rfc_v2 -> v1:
- Accomodate Ilya's comments.
- Split OVS's attribute in two attributes and simplify internal
handling of psample arguments.
- Extend psample and tc with a user-defined cookie.
- Add a tracepoint to psample to facilitate troubleshooting.
rfc_v1 -> rfc_v2:
- Use psample instead of a new OVS-only multicast group.
- Extend psample and tc with a user-defined cookie.
Adrian Moreno (10):
net: psample: add user cookie
net: sched: act_sample: add action cookie to sample
net: psample: skip packet copy if no listeners
net: psample: allow using rate as probability
net: openvswitch: add emit_sample action
net: openvswitch: store sampling probability in cb.
selftests: openvswitch: add emit_sample action
selftests: openvswitch: add userspace parsing
selftests: openvswitch: parse trunc action
selftests: openvswitch: add emit_sample test
Documentation/netlink/specs/ovs_flow.yaml | 17 ++
include/net/psample.h | 5 +-
include/uapi/linux/openvswitch.h | 30 +-
include/uapi/linux/psample.h | 11 +-
include/uapi/linux/tc_act/tc_sample.h | 1 +
net/openvswitch/Kconfig | 1 +
net/openvswitch/actions.c | 63 +++-
net/openvswitch/datapath.h | 3 +
net/openvswitch/flow_netlink.c | 33 ++-
net/openvswitch/vport.c | 1 +
net/psample/psample.c | 16 +-
net/sched/act_sample.c | 12 +
.../selftests/net/openvswitch/openvswitch.sh | 110 ++++++-
.../selftests/net/openvswitch/ovs-dpctl.py | 272 +++++++++++++++++-
14 files changed, 559 insertions(+), 16 deletions(-)
--
2.45.1
Hello,
KernelCI is hosting a bi-weekly call on Thursday to discuss improvements
to existing upstream tests, the development of new tests to increase
kernel testing coverage, and the enablement of these tests in KernelCI.
In recent months, we at Collabora have focused on various kernel areas,
assessing the tests already available upstream and contributing patches
to make them easily runnable in CIs.
Below is a list of the tests we've been working on and their latest
status updates, as discussed in the last meeting held on 2024-06-13:
*ACPI probe kselftest*
- Proposing new kselftest to detect unprobed devices on ACPI platforms:
https://lore.kernel.org/all/20240308144933.337107-1-laura.nao@collabora.com
- Regression on acpi_fan driver detected and fixed upstream after
preliminary testing in KernelCI. Sent follow-up on RFCv2.
*USB/PCI devices kselftest*
- Upstream test to detect unprobed devices on discoverable buses:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?…
- Preparing to enable the test in KernelCI on MediaTek Chromebooks in the
Collabora LAVA lab - see: https://github.com/kernelci/kernelci-pipeline/issues/579
*Watchdog kselftest*
- Adding KTAP output and limiting the ping loop to run the test
individually in CIs: https://lore.kernel.org/all/20240506111359.224579-1-laura.nao@collabora.com
- Received feedback on series, discussion ongoing
*Watchdog reset test*
- Implementing out-of-tree test in KernelCI to detect device reset after
watchdog timeout expiration:
https://github.com/kernelci/kernelci-pipeline/issues/608
- Enabled test in KernelCI for all ARM64 and AMD Chromebooks, currently
monitoring the results
*Error log test*
- Proposing new kselftest to report device log errors:
https://lore.kernel.org/all/20240423-dev-err-log-selftest-v1-0-690c1741d68b…
- Standardizing error log format suggested for easier parsing, including
for kernel panic/oops/warning messages.
*Suspend/resume in cpufreq kselftest*
- Enabling suspend/resume test within the cpufreq kselftest in KernelCI
- Adding parameter support to kselftest script in KernelCI:
https://github.com/Linaro/test-definitions/pull/511
- Planning to add support for rtcwake in the test, to enable automated
resume
- Considering measuring latency for suspend and resume processes
*TAP conformance in kselftests*
- Ongoing improvements to KVM selftests, discussion ongoing upstream
https://lore.kernel.org/all/20240422170842.2073979-1-usama.anjum@collabora.…https://lore.kernel.org/all/20240612104500.425012-1-usama.anjum@collabora.c…
- Enabling more kselftests in KernelCI (dmabuf-heaps, exec, and iommu):
https://github.com/kernelci/kernelci-pipeline/pull/640/
*Boot time test*
- Investigating possibility of adding new test upstream to measure the
kernel boot time and detect regressions
- Need to investigate available interfaces for reporting boot/probe times
- Multiple measurement points may be necessary, need to establish a clear
definition of "boot" first
- Influenced by defconfig; testing with various kernel configurations
recommended
*Possible improvements to kselftest documentation*
- Document tests excluded from default kselftest run and parameter
support
*Test configuration reporting in KernelCI*
- Discussing reporting more details on the test configuration used:
https://github.com/kernelci/kernelci-pipeline/issues/636
Please reply to this thread if you'd like to join the call or discuss
any of the topics further. We look forward to collaborating with the
community to improve upstream tests and expand coverage to more areas
of interest within the kernel.
Best regards,
Laura Nao
Netlink flags, although they don't have payload at the netlink level,
are represented as having "True" as value in pyroute2.
Without it, trying to add a flow with a flag-type action (e.g: pop_vlan)
fails with the following traceback:
Traceback (most recent call last):
File "[...]/ovs-dpctl.py", line 2498, in <module>
sys.exit(main(sys.argv))
^^^^^^^^^^^^^^
File "[...]/ovs-dpctl.py", line 2487, in main
ovsflow.add_flow(rep["dpifindex"], flow)
File "[...]/ovs-dpctl.py", line 2136, in add_flow
reply = self.nlm_request(
^^^^^^^^^^^^^^^^^
File "[...]/pyroute2/netlink/nlsocket.py", line 822, in nlm_request
return tuple(self._genlm_request(*argv, **kwarg))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "[...]/pyroute2/netlink/generic/__init__.py", line 126, in
nlm_request
return tuple(super().nlm_request(*argv, **kwarg))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "[...]/pyroute2/netlink/nlsocket.py", line 1124, in nlm_request
self.put(msg, msg_type, msg_flags, msg_seq=msg_seq)
File "[...]/pyroute2/netlink/nlsocket.py", line 389, in put
self.sendto_gate(msg, addr)
File "[...]/pyroute2/netlink/nlsocket.py", line 1056, in sendto_gate
msg.encode()
File "[...]/pyroute2/netlink/__init__.py", line 1245, in encode
offset = self.encode_nlas(offset)
^^^^^^^^^^^^^^^^^^^^^^^^
File "[...]/pyroute2/netlink/__init__.py", line 1560, in encode_nlas
nla_instance.setvalue(cell[1])
File "[...]/pyroute2/netlink/__init__.py", line 1265, in setvalue
nlv.setvalue(nla_tuple[1])
~~~~~~~~~^^^
IndexError: list index out of range
Signed-off-by: Adrian Moreno <amorenoz(a)redhat.com>
---
tools/testing/selftests/net/openvswitch/ovs-dpctl.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tools/testing/selftests/net/openvswitch/ovs-dpctl.py b/tools/testing/selftests/net/openvswitch/ovs-dpctl.py
index 1dd057afd3fb..9f8dec2f6539 100644
--- a/tools/testing/selftests/net/openvswitch/ovs-dpctl.py
+++ b/tools/testing/selftests/net/openvswitch/ovs-dpctl.py
@@ -531,7 +531,7 @@ class ovsactions(nla):
for flat_act in parse_flat_map:
if parse_starts_block(actstr, flat_act[0], False):
actstr = actstr[len(flat_act[0]):]
- self["attrs"].append([flat_act[1]])
+ self["attrs"].append([flat_act[1], True])
actstr = actstr[strspn(actstr, ", ") :]
parsed = True
--
2.45.1
openvswitch.sh makes use of substitutions of the form ${ns:0:1}, to
obtain the first character of $ns. Empirically, this is works with bash
but not dash. When run with dash these evaluate to an empty string and
printing an error to stdout.
# dash -c 'ns=client; echo "${ns:0:1}"' 2>error
# cat error
dash: 1: Bad substitution
# bash -c 'ns=client; echo "${ns:0:1}"' 2>error
c
# cat error
This leads to tests that neither pass nor fail.
F.e.
TEST: arp_ping [START]
adding sandbox 'test_arp_ping'
Adding DP/Bridge IF: sbx:test_arp_ping dp:arpping {, , }
create namespaces
./openvswitch.sh: 282: eval: Bad substitution
TEST: ct_connect_v4 [START]
adding sandbox 'test_ct_connect_v4'
Adding DP/Bridge IF: sbx:test_ct_connect_v4 dp:ct4 {, , }
./openvswitch.sh: 322: eval: Bad substitution
create namespaces
Resolve this by making openvswitch.sh a bash script.
Fixes: 918423fda910 ("selftests: openvswitch: add an initial flow programming case")
Signed-off-by: Simon Horman <horms(a)kernel.org>
---
tools/testing/selftests/net/openvswitch/openvswitch.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tools/testing/selftests/net/openvswitch/openvswitch.sh b/tools/testing/selftests/net/openvswitch/openvswitch.sh
index 5cae53543849..15bca0708717 100755
--- a/tools/testing/selftests/net/openvswitch/openvswitch.sh
+++ b/tools/testing/selftests/net/openvswitch/openvswitch.sh
@@ -1,4 +1,4 @@
-#!/bin/sh
+#!/bin/bash
# SPDX-License-Identifier: GPL-2.0
#
# OVS kernel module self tests
Although "TAP" word is being used already in documentation, but it hasn't
been defined in informative way for developers that how to write TAP
conformant tests and what are the benefits. Write a short brief about it.
Signed-off-by: Muhammad Usama Anjum <usama.anjum(a)collabora.com>
---
Changes since v1:
- Update documentation by modifying and removing sentences
---
Documentation/dev-tools/kselftest.rst | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/Documentation/dev-tools/kselftest.rst b/Documentation/dev-tools/kselftest.rst
index dcf634e411bd9..f3766e326d1e3 100644
--- a/Documentation/dev-tools/kselftest.rst
+++ b/Documentation/dev-tools/kselftest.rst
@@ -228,6 +228,13 @@ In general, the rules for selftests are
* Don't cause the top-level "make run_tests" to fail if your feature is
unconfigured.
+ * The output of tests must conform to the TAP standard to ensure high
+ testing quality and to capture failures/errors with specific details.
+ The kselftest.h and kselftest_harness.h headers provide wrappers for
+ outputting test results. These wrappers should be used for pass,
+ fail, exit, and skip messages. CI systems can easily parse TAP output
+ messages to detect test results.
+
Contributing new tests (details)
================================
--
2.39.2
Changes since v2:
a) After some disussion with David Hildenbrand, simplified the "fix
missing __NR_mseal" patch down to just two lines of diff, after all.
b) Improved the "kvm, mdwe fixes to avoid requiring "make headers""
patch by taking a snapshot of the prctl.h, instead of manually creating
defines. Thanks also to David Hildenbrand for that.
c) Fixed up the final patch to use asm-generic/unistd.h.
d) Added acks.
e) Rebased onto 6.10-rc4+ (today's top of tree).
Changes since v1:
Jeff Xu, I apologize for this churn: I was forced to drop your
Reviewed-by and Tested-by tags from 2 of the 3 mseal patches, because
the __NR_mseal fix is completely different now.
a) Reworked the mseal fix to use the kernel's in-tree unistd*.h files,
instead of hacking in a __NR_mseal definition directly. (Thanks to David
Hildenbrand for pointing out that this needed to be done.)
b) Fixed the subject line of the kvm and mdwe patch.
c) Reordered the patches so as to group the mseal changes together.
d) ADDED an additional patch, 6/6, to remove various __NR_xx items and
checks from the mm selftests.
Cover letter, updated for v2:
Eventually, once the build succeeds on a sufficiently old distro, the
idea is to delete $(KHDR_INCLUDES) from the selftests/mm build, and then
after that, from selftests/lib.mk and all of the other selftest builds.
For now, this series merely achieves a clean build of selftests/mm on a
not-so-old distro: Ubuntu 23.04. In other words, after this series is
applied, it is possible to delete $(KHDR_INCLUDES) from
selftests/mm/Makefile and the build will still succeed.
1. Add tools/uapi/asm/unistd_[32|x32|64].h files, which include
definitions of __NR_mseal, and include them (indirectly) from the files
that use __NR_mseal. The new files are copied from ./usr/include/asm,
which is how we have agreed to do this sort of thing, see [1].
2. Add fs.h, similarly created: it was copied directly from a snapshot
of ./usr/include/linux/fs.h after running "make headers".
3. Add a few selected prctl.h values that the ksm and mdwe tests require.
4. Factor out some common code from mseal_test.c and seal_elf.c, into a
new mseal_helpers.h file.
5. Remove local __NR_* definitions and checks.
[1] commit e076eaca5906 ("selftests: break the dependency upon local
header files")
John Hubbard (6):
selftests/mm: mseal, self_elf: fix missing __NR_mseal
selftests/mm: mseal, self_elf: factor out test macros and other
duplicated items
selftests/mm: mseal, self_elf: rename TEST_END_CHECK to
REPORT_TEST_PASS
selftests/mm: fix vm_util.c build failures: add snapshot of fs.h
selftests/mm: kvm, mdwe fixes to avoid requiring "make headers"
selftests/mm: remove local __NR_* definitions
tools/include/uapi/linux/fs.h | 392 ++++++++++++++++++
tools/include/uapi/linux/prctl.h | 331 +++++++++++++++
tools/testing/selftests/mm/hugepage-mremap.c | 2 +-
.../selftests/mm/ksm_functional_tests.c | 8 +-
tools/testing/selftests/mm/memfd_secret.c | 14 +-
tools/testing/selftests/mm/mkdirty.c | 8 +-
tools/testing/selftests/mm/mlock2.h | 1 +
tools/testing/selftests/mm/mseal_helpers.h | 41 ++
tools/testing/selftests/mm/mseal_test.c | 143 +++----
tools/testing/selftests/mm/pagemap_ioctl.c | 2 +-
tools/testing/selftests/mm/protection_keys.c | 2 +-
tools/testing/selftests/mm/seal_elf.c | 37 +-
tools/testing/selftests/mm/uffd-common.c | 4 -
tools/testing/selftests/mm/uffd-stress.c | 16 +-
tools/testing/selftests/mm/uffd-unit-tests.c | 14 +-
15 files changed, 824 insertions(+), 191 deletions(-)
create mode 100644 tools/include/uapi/linux/fs.h
create mode 100644 tools/include/uapi/linux/prctl.h
create mode 100644 tools/testing/selftests/mm/mseal_helpers.h
base-commit: 14d7c92f8df9c0964ae6f8b813c1b3ac38120825
--
2.45.2