There are a number of tests that check features of the Linux networking stack. By running them on suitable interfaces, one can exercise the mlxsw offloading code. However none of these tests attempts to push mlxsw to the limits supported by the ASIC.
As an additional wrinkle, the "limits supported by the ASIC" themselves may not be a set of fixed numbers, but rather depend on a profile that determines how the ASIC resources are allocated for different purposes.
This patchset introduces several tests that verify capability of mlxsw to offload amounts of routes, flower rules, and mirroring sessions that match predicted ASIC capacity, at different configuration profiles. Additionally they verify that amounts exceeding the predicted capacity can *not* be offloaded.
These are not generic tests, but ones that are tailored for mlxsw specifically. For that reason they are not added to net/forwarding selftests subdirectory, but rather to a newly-added drivers/net/mlxsw.
Patches #1, #2 and #3 tweak the generic forwarding/lib.sh to support the new additions.
In patches #4 and #5, new libraries for interfacing with devlink are introduced, first a generic one, then a Spectrum-specific one.
In patch #6, a devlink resource test is introduced.
Patches #7 and #8, #9 and #10, and #11 and #12 introduce three scale tests: router, flower and mirror-to-gretap. The first of each pair of patches introduces a generic portion of the test (mlxsw-specific), the second introduces a Spectrum-specific wrapper.
Patch #13 then introduces a scale test driver that runs (possibly a subset of) the tests introduced by patches from previous paragraph.
Arkadi Sharshevsky (1): selftests: mlxsw: Add router test
Petr Machata (8): selftests: forwarding: lib: Add check_err_fail() selftests: forwarding: lib: Parameterize NUM_NETIFS in two functions selftests: forwarding: Add devlink_lib.sh selftests: mlxsw: Add devlink_lib_spectrum.sh selftests: mlxsw: Add tc flower scale test selftests: mlxsw: Add target for tc flower test on spectrum selftests: mlxsw: Add scale test for mirror-to-gretap selftests: mlxsw: Add target for mirror-to-gretap test on spectrum
Yuval Mintz (4): selftests: forwarding: Allow lib.sh sourcing from other directories selftests: mlxsw: Add devlink KVD resource test selftests: mlxsw: Add target for router test on spectrum selftests: mlxsw: Add scale test for resources
MAINTAINERS | 1 + .../drivers/net/mlxsw/mirror_gre_scale.sh | 197 +++++++++++++++++++++ .../selftests/drivers/net/mlxsw/router_scale.sh | 167 +++++++++++++++++ .../net/mlxsw/spectrum/devlink_lib_spectrum.sh | 119 +++++++++++++ .../net/mlxsw/spectrum/devlink_resources.sh | 117 ++++++++++++ .../drivers/net/mlxsw/spectrum/mirror_gre_scale.sh | 13 ++ .../drivers/net/mlxsw/spectrum/resource_scale.sh | 55 ++++++ .../drivers/net/mlxsw/spectrum/router_scale.sh | 18 ++ .../drivers/net/mlxsw/spectrum/tc_flower_scale.sh | 19 ++ .../selftests/drivers/net/mlxsw/tc_flower_scale.sh | 134 ++++++++++++++ .../selftests/net/forwarding/devlink_lib.sh | 108 +++++++++++ tools/testing/selftests/net/forwarding/lib.sh | 30 +++- 12 files changed, 974 insertions(+), 4 deletions(-) create mode 100644 tools/testing/selftests/drivers/net/mlxsw/mirror_gre_scale.sh create mode 100644 tools/testing/selftests/drivers/net/mlxsw/router_scale.sh create mode 100644 tools/testing/selftests/drivers/net/mlxsw/spectrum/devlink_lib_spectrum.sh create mode 100755 tools/testing/selftests/drivers/net/mlxsw/spectrum/devlink_resources.sh create mode 100644 tools/testing/selftests/drivers/net/mlxsw/spectrum/mirror_gre_scale.sh create mode 100755 tools/testing/selftests/drivers/net/mlxsw/spectrum/resource_scale.sh create mode 100644 tools/testing/selftests/drivers/net/mlxsw/spectrum/router_scale.sh create mode 100644 tools/testing/selftests/drivers/net/mlxsw/spectrum/tc_flower_scale.sh create mode 100644 tools/testing/selftests/drivers/net/mlxsw/tc_flower_scale.sh create mode 100644 tools/testing/selftests/net/forwarding/devlink_lib.sh
From: Yuval Mintz yuvalm@mellanox.com
The devlink related scripts are mlxsw-specific. As a result, they'll reside in a different directory - but would still need the common logic implemented in lib.sh. So as a preliminary step, allow lib.sh to be sourced from other directories as well.
Signed-off-by: Yuval Mintz yuvalm@mellanox.com Signed-off-by: Petr Machata petrm@mellanox.com --- tools/testing/selftests/net/forwarding/lib.sh | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
diff --git a/tools/testing/selftests/net/forwarding/lib.sh b/tools/testing/selftests/net/forwarding/lib.sh index d1f14f83979e..59272824ef37 100644 --- a/tools/testing/selftests/net/forwarding/lib.sh +++ b/tools/testing/selftests/net/forwarding/lib.sh @@ -14,8 +14,13 @@ PAUSE_ON_CLEANUP=${PAUSE_ON_CLEANUP:=no} NETIF_TYPE=${NETIF_TYPE:=veth} NETIF_CREATE=${NETIF_CREATE:=yes}
-if [[ -f forwarding.config ]]; then - source forwarding.config +relative_path="${BASH_SOURCE%/*}" +if [[ "$relative_path" == "${BASH_SOURCE}" ]]; then + relative_path="." +fi + +if [[ -f $relative_path/forwarding.config ]]; then + source "$relative_path/forwarding.config" fi
##############################################################################
In the scale testing scenarios, one usually has a condition that is expected to either fail, or pass, depending on which side of the scale is being tested.
To capture this logic, add a function check_err_fail(), which dispatches either to check_err() or check_fail(), depending on the value of the first argument, should_fail.
Signed-off-by: Petr Machata petrm@mellanox.com Reviewed-by: Yuval Mintz yuvalm@mellanox.com --- tools/testing/selftests/net/forwarding/lib.sh | 13 +++++++++++++ 1 file changed, 13 insertions(+)
diff --git a/tools/testing/selftests/net/forwarding/lib.sh b/tools/testing/selftests/net/forwarding/lib.sh index 59272824ef37..5f4b7ed2c65a 100644 --- a/tools/testing/selftests/net/forwarding/lib.sh +++ b/tools/testing/selftests/net/forwarding/lib.sh @@ -156,6 +156,19 @@ check_fail() fi }
+check_err_fail() +{ + local should_fail=$1; shift + local err=$1; shift + local what=$1; shift + + if ((should_fail)); then + check_fail $err "$what succeeded, but should have failed" + else + check_err $err "$what failed" + fi +} + log_test() { local test_name=$1
setup_wait() and tc_offload_check() both assume that all NUM_NETIFS interfaces are relevant for a given test. However, the scale test script acts as an umbrella for a number of sub-tests, some of which may not require all the interfaces.
Thus it's suboptimal for tc_offload_check() to query all the interfaces. In case of setup_wait() it's incorrect, because the sub-test in question of course doesn't configure any interfaces beyond what it needs, and setup_wait() then ends up waiting indefinitely for the extraneous interfaces to come up.
For that reason, give setup_wait() and tc_offload_check() an optional parameter with a number of interfaces to probe. Fall back to global NUM_NETIFS if the parameter is not given.
Signed-off-by: Petr Machata petrm@mellanox.com Reviewed-by: Yuval Mintz yuvalm@mellanox.com --- tools/testing/selftests/net/forwarding/lib.sh | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/tools/testing/selftests/net/forwarding/lib.sh b/tools/testing/selftests/net/forwarding/lib.sh index 5f4b7ed2c65a..e073918bbe15 100644 --- a/tools/testing/selftests/net/forwarding/lib.sh +++ b/tools/testing/selftests/net/forwarding/lib.sh @@ -220,7 +220,9 @@ setup_wait_dev()
setup_wait() { - for i in $(eval echo {1..$NUM_NETIFS}); do + local num_netifs=${1:-$NUM_NETIFS} + + for ((i = 1; i <= num_netifs; ++i)); do setup_wait_dev ${NETIFS[p$i]} done
@@ -481,7 +483,9 @@ forwarding_restore()
tc_offload_check() { - for i in $(eval echo {1..$NUM_NETIFS}); do + local num_netifs=${1:-$NUM_NETIFS} + + for ((i = 1; i <= num_netifs; ++i)); do ethtool -k ${NETIFS[p$i]} \ | grep "hw-tc-offload: on" &> /dev/null if [[ $? -ne 0 ]]; then
This helper library contains wrappers to devlink functionality agnostic to the underlying device.
Signed-off-by: Yuval Mintz yuvalm@mellanox.com [petrm@mellanox.com: Split this out from another patch.] Signed-off-by: Petr Machata petrm@mellanox.com --- .../selftests/net/forwarding/devlink_lib.sh | 108 +++++++++++++++++++++ 1 file changed, 108 insertions(+) create mode 100644 tools/testing/selftests/net/forwarding/devlink_lib.sh
diff --git a/tools/testing/selftests/net/forwarding/devlink_lib.sh b/tools/testing/selftests/net/forwarding/devlink_lib.sh new file mode 100644 index 000000000000..5ab1e5f43022 --- /dev/null +++ b/tools/testing/selftests/net/forwarding/devlink_lib.sh @@ -0,0 +1,108 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +############################################################################## +# Source library + +relative_path="${BASH_SOURCE%/*}" +if [[ "$relative_path" == "${BASH_SOURCE}" ]]; then + relative_path="." +fi + +source "$relative_path/lib.sh" + +############################################################################## +# Defines + +DEVLINK_DEV=$(devlink port show | grep "${NETIFS[p1]}" | \ + grep -v "${NETIFS[p1]}[0-9]" | cut -d" " -f1 | \ + rev | cut -d"/" -f2- | rev) +if [ -z "$DEVLINK_DEV" ]; then + echo "SKIP: ${NETIFS[p1]} has no devlink device registered for it" + exit 1 +fi +if [[ "$(echo $DEVLINK_DEV | grep -c pci)" -eq 0 ]]; then + echo "SKIP: devlink device's bus is not PCI" + exit 1 +fi + +DEVLINK_VIDDID=$(lspci -s $(echo $DEVLINK_DEV | cut -d"/" -f2) \ + -n | cut -d" " -f3) + +############################################################################## +# Sanity checks + +devlink -j resource show "$DEVLINK_DEV" &> /dev/null +if [ $? -ne 0 ]; then + echo "SKIP: iproute2 too old, missing devlink resource support" + exit 1 +fi + +############################################################################## +# Devlink helpers + +devlink_resource_names_to_path() +{ + local resource + local path="" + + for resource in "${@}"; do + if [ "$path" == "" ]; then + path="$resource" + else + path="${path}/$resource" + fi + done + + echo "$path" +} + +devlink_resource_get() +{ + local name=$1 + local resource_name=.[]["$DEVLINK_DEV"] + + resource_name="$resource_name | .[] | select (.name == "$name")" + + shift + for resource in "${@}"; do + resource_name="${resource_name} | .["resources"][] | \ + select (.name == "$resource")" + done + + devlink -j resource show "$DEVLINK_DEV" | jq "$resource_name" +} + +devlink_resource_size_get() +{ + local size=$(devlink_resource_get "$@" | jq '.["size_new"]') + + if [ "$size" == "null" ]; then + devlink_resource_get "$@" | jq '.["size"]' + else + echo "$size" + fi +} + +devlink_resource_size_set() +{ + local new_size=$1 + local path + + shift + path=$(devlink_resource_names_to_path "$@") + devlink resource set "$DEVLINK_DEV" path "$path" size "$new_size" + check_err $? "Failed setting path $path to size $size" +} + +devlink_reload() +{ + local still_pending + + devlink dev reload "$DEVLINK_DEV" &> /dev/null + check_err $? "Failed reload" + + still_pending=$(devlink resource show "$DEVLINK_DEV" | \ + grep -c "size_new") + check_err $still_pending "Failed reload - There are still unset sizes" +}
This library builds on top of devlink_lib.sh and contains functionality specific to Spectrum ASICs, e.g., re-partitioning the various KVD sub-parts.
Signed-off-by: Yuval Mintz yuvalm@mellanox.com [petrm@mellanox.com: Split this out from another patch. Fix line length in devlink_sp_read_kvd_defaults().] Signed-off-by: Petr Machata petrm@mellanox.com --- MAINTAINERS | 1 + .../net/mlxsw/spectrum/devlink_lib_spectrum.sh | 119 +++++++++++++++++++++ 2 files changed, 120 insertions(+) create mode 100644 tools/testing/selftests/drivers/net/mlxsw/spectrum/devlink_lib_spectrum.sh
diff --git a/MAINTAINERS b/MAINTAINERS index f3e766e504d8..f3dd6c4f61e1 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -9154,6 +9154,7 @@ S: Supported W: http://www.mellanox.com Q: http://patchwork.ozlabs.org/project/netdev/list/ F: drivers/net/ethernet/mellanox/mlxsw/ +F: tools/testing/selftests/drivers/net/mlxsw/
MELLANOX FIRMWARE FLASH LIBRARY (mlxfw) M: mlxsw@mellanox.com diff --git a/tools/testing/selftests/drivers/net/mlxsw/spectrum/devlink_lib_spectrum.sh b/tools/testing/selftests/drivers/net/mlxsw/spectrum/devlink_lib_spectrum.sh new file mode 100644 index 000000000000..73035e25085d --- /dev/null +++ b/tools/testing/selftests/drivers/net/mlxsw/spectrum/devlink_lib_spectrum.sh @@ -0,0 +1,119 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +source "../../../../net/forwarding/devlink_lib.sh" + +if [ "$DEVLINK_VIDDID" != "15b3:cb84" ]; then + echo "SKIP: test is tailored for Mellanox Spectrum" + exit 1 +fi + +# Needed for returning to default +declare -A KVD_DEFAULTS + +KVD_CHILDREN="linear hash_single hash_double" +KVDL_CHILDREN="singles chunks large_chunks" + +devlink_sp_resource_minimize() +{ + local size + local i + + for i in $KVD_CHILDREN; do + size=$(devlink_resource_get kvd "$i" | jq '.["size_min"]') + devlink_resource_size_set "$size" kvd "$i" + done + + for i in $KVDL_CHILDREN; do + size=$(devlink_resource_get kvd linear "$i" | \ + jq '.["size_min"]') + devlink_resource_size_set "$size" kvd linear "$i" + done +} + +devlink_sp_size_kvd_to_default() +{ + local need_reload=0 + local i + + for i in $KVD_CHILDREN; do + local size=$(echo "${KVD_DEFAULTS[kvd_$i]}" | jq '.["size"]') + current_size=$(devlink_resource_size_get kvd "$i") + + if [ "$size" -ne "$current_size" ]; then + devlink_resource_size_set "$size" kvd "$i" + need_reload=1 + fi + done + + for i in $KVDL_CHILDREN; do + local size=$(echo "${KVD_DEFAULTS[kvd_linear_$i]}" | \ + jq '.["size"]') + current_size=$(devlink_resource_size_get kvd linear "$i") + + if [ "$size" -ne "$current_size" ]; then + devlink_resource_size_set "$size" kvd linear "$i" + need_reload=1 + fi + done + + if [ "$need_reload" -ne "0" ]; then + devlink_reload + fi +} + +devlink_sp_read_kvd_defaults() +{ + local key + local i + + KVD_DEFAULTS[kvd]=$(devlink_resource_get "kvd") + for i in $KVD_CHILDREN; do + key=kvd_$i + KVD_DEFAULTS[$key]=$(devlink_resource_get kvd "$i") + done + + for i in $KVDL_CHILDREN; do + key=kvd_linear_$i + KVD_DEFAULTS[$key]=$(devlink_resource_get kvd linear "$i") + done +} + +KVD_PROFILES="default scale ipv4_max" + +devlink_sp_resource_kvd_profile_set() +{ + local profile=$1 + + case "$profile" in + scale) + devlink_resource_size_set 64000 kvd linear + devlink_resource_size_set 15616 kvd linear singles + devlink_resource_size_set 32000 kvd linear chunks + devlink_resource_size_set 16384 kvd linear large_chunks + devlink_resource_size_set 128000 kvd hash_single + devlink_resource_size_set 48000 kvd hash_double + devlink_reload + ;; + ipv4_max) + devlink_resource_size_set 64000 kvd linear + devlink_resource_size_set 15616 kvd linear singles + devlink_resource_size_set 32000 kvd linear chunks + devlink_resource_size_set 16384 kvd linear large_chunks + devlink_resource_size_set 144000 kvd hash_single + devlink_resource_size_set 32768 kvd hash_double + devlink_reload + ;; + default) + devlink_resource_size_set 98304 kvd linear + devlink_resource_size_set 16384 kvd linear singles + devlink_resource_size_set 49152 kvd linear chunks + devlink_resource_size_set 32768 kvd linear large_chunks + devlink_resource_size_set 87040 kvd hash_single + devlink_resource_size_set 60416 kvd hash_double + devlink_reload + ;; + *) + check_err 1 "Unknown profile $profile" + esac +}
From: Yuval Mintz yuvalm@mellanox.com
Add a selftest that can be used to perform basic sanity of the devlink resource API as well as test the behavior of KVD manipulation in the driver.
This is the first case of a HW-only test - in order to test the devlink resource a driver capable of exposing resources has to be provided first.
Signed-off-by: Yuval Mintz yuvalm@mellanox.com [petrm@mellanox.com: Extracted two patches out of this patch. Tweaked commit message.] Signed-off-by: Petr Machata petrm@mellanox.com --- .../net/mlxsw/spectrum/devlink_resources.sh | 117 +++++++++++++++++++++ 1 file changed, 117 insertions(+) create mode 100755 tools/testing/selftests/drivers/net/mlxsw/spectrum/devlink_resources.sh
diff --git a/tools/testing/selftests/drivers/net/mlxsw/spectrum/devlink_resources.sh b/tools/testing/selftests/drivers/net/mlxsw/spectrum/devlink_resources.sh new file mode 100755 index 000000000000..b1fe960e398a --- /dev/null +++ b/tools/testing/selftests/drivers/net/mlxsw/spectrum/devlink_resources.sh @@ -0,0 +1,117 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +NUM_NETIFS=1 +source devlink_lib_spectrum.sh + +setup_prepare() +{ + devlink_sp_read_kvd_defaults +} + +cleanup() +{ + pre_cleanup + devlink_sp_size_kvd_to_default +} + +trap cleanup EXIT + +setup_prepare + +profiles_test() +{ + local i + + log_info "Running profile tests" + + for i in $KVD_PROFILES; do + RET=0 + devlink_sp_resource_kvd_profile_set $i + log_test "'$i' profile" + done + + # Default is explicitly tested at end to ensure it's actually applied + RET=0 + devlink_sp_resource_kvd_profile_set "default" + log_test "'default' profile" +} + +resources_min_test() +{ + local size + local i + local j + + log_info "Running KVD-minimum tests" + + for i in $KVD_CHILDREN; do + RET=0 + size=$(devlink_resource_get kvd "$i" | jq '.["size_min"]') + devlink_resource_size_set "$size" kvd "$i" + + # In case of linear, need to minimize sub-resources as well + if [[ "$i" == "linear" ]]; then + for j in $KVDL_CHILDREN; do + devlink_resource_size_set 0 kvd linear "$j" + done + fi + + devlink_reload + devlink_sp_size_kvd_to_default + log_test "'$i' minimize [$size]" + done +} + +resources_max_test() +{ + local min_size + local size + local i + local j + + log_info "Running KVD-maximum tests" + for i in $KVD_CHILDREN; do + RET=0 + devlink_sp_resource_minimize + + # Calculate the maximum possible size for the given partition + size=$(devlink_resource_size_get kvd) + for j in $KVD_CHILDREN; do + if [ "$i" != "$j" ]; then + min_size=$(devlink_resource_get kvd "$j" | \ + jq '.["size_min"]') + size=$((size - min_size)) + fi + done + + # Test almost maximum size + devlink_resource_size_set "$((size - 128))" kvd "$i" + devlink_reload + log_test "'$i' almost maximize [$((size - 128))]" + + # Test above maximum size + devlink resource set "$DEVLINK_DEV" \ + path "kvd/$i" size $((size + 128)) &> /dev/null + check_fail $? "Set kvd/$i to size $((size + 128)) should fail" + log_test "'$i' Overflow rejection [$((size + 128))]" + + # Test maximum size + if [ "$i" == "hash_single" ] || [ "$i" == "hash_double" ]; then + echo "SKIP: Observed problem with exact max $i" + continue + fi + + devlink_resource_size_set "$size" kvd "$i" + devlink_reload + log_test "'$i' maximize [$size]" + + devlink_sp_size_kvd_to_default + done +} + +profiles_test +resources_min_test +resources_max_test + +exit "$RET"
From: Arkadi Sharshevsky arkadis@mellanox.com
This test aims for both stand alone and internal usage by the resource infra. The test receives the number routes to offload and checks: - The routes were offloaded correctly - Traffic for each route.
Signed-off-by: Arkadi Sharshevsky arkadis@mellanox.com Signed-off-by: Yuval Mintz yuvalm@mellanox.com Reviewed-by: Petr Machata petrm@mellanox.com Signed-off-by: Petr Machata petrm@mellanox.com --- .../selftests/drivers/net/mlxsw/router_scale.sh | 167 +++++++++++++++++++++ 1 file changed, 167 insertions(+) create mode 100644 tools/testing/selftests/drivers/net/mlxsw/router_scale.sh
diff --git a/tools/testing/selftests/drivers/net/mlxsw/router_scale.sh b/tools/testing/selftests/drivers/net/mlxsw/router_scale.sh new file mode 100644 index 000000000000..d231649b4f01 --- /dev/null +++ b/tools/testing/selftests/drivers/net/mlxsw/router_scale.sh @@ -0,0 +1,167 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +ROUTER_NUM_NETIFS=4 + +router_h1_create() +{ + simple_if_init $h1 192.0.1.1/24 + ip route add 193.0.0.0/8 via 192.0.1.2 dev $h1 +} + +router_h1_destroy() +{ + ip route del 193.0.0.0/8 via 192.0.1.2 dev $h1 + simple_if_fini $h1 192.0.1.1/24 +} + +router_h2_create() +{ + simple_if_init $h2 192.0.2.1/24 + tc qdisc add dev $h2 handle ffff: ingress +} + +router_h2_destroy() +{ + tc qdisc del dev $h2 handle ffff: ingress + simple_if_fini $h2 192.0.2.1/24 +} + +router_create() +{ + ip link set dev $rp1 up + ip link set dev $rp2 up + + ip address add 192.0.1.2/24 dev $rp1 + ip address add 192.0.2.2/24 dev $rp2 +} + +router_destroy() +{ + ip address del 192.0.2.2/24 dev $rp2 + ip address del 192.0.1.2/24 dev $rp1 + + ip link set dev $rp2 down + ip link set dev $rp1 down +} + +router_setup_prepare() +{ + h1=${NETIFS[p1]} + rp1=${NETIFS[p2]} + + rp2=${NETIFS[p3]} + h2=${NETIFS[p4]} + + h1mac=$(mac_get $h1) + rp1mac=$(mac_get $rp1) + + vrf_prepare + + router_h1_create + router_h2_create + + router_create +} + +router_offload_validate() +{ + local route_count=$1 + local offloaded_count + + offloaded_count=$(ip route | grep -o 'offload' | wc -l) + [[ $offloaded_count -ge $route_count ]] +} + +router_routes_create() +{ + local route_count=$1 + local count=0 + + ROUTE_FILE="$(mktemp)" + + for i in {0..255} + do + for j in {0..255} + do + for k in {0..255} + do + if [[ $count -eq $route_count ]]; then + break 3 + fi + + echo route add 193.${i}.${j}.${k}/32 via \ + 192.0.2.1 dev $rp2 >> $ROUTE_FILE + ((count++)) + done + done + done + + ip -b $ROUTE_FILE &> /dev/null +} + +router_routes_destroy() +{ + if [[ -v ROUTE_FILE ]]; then + rm -f $ROUTE_FILE + fi +} + +router_test() +{ + local route_count=$1 + local should_fail=$2 + local count=0 + + RET=0 + + router_routes_create $route_count + + router_offload_validate $route_count + check_err_fail $should_fail $? "Offload of $route_count routes" + if [[ $RET -ne 0 ]] || [[ $should_fail -eq 1 ]]; then + return + fi + + tc filter add dev $h2 ingress protocol ip pref 1 flower \ + skip_sw dst_ip 193.0.0.0/8 action drop + + for i in {0..255} + do + for j in {0..255} + do + for k in {0..255} + do + if [[ $count -eq $route_count ]]; then + break 3 + fi + + $MZ $h1 -c 1 -p 64 -a $h1mac -b $rp1mac \ + -A 192.0.1.1 -B 193.${i}.${j}.${k} \ + -t ip -q + ((count++)) + done + done + done + + tc_check_packets "dev $h2 ingress" 1 $route_count + check_err $? "Offload mismatch" + + tc filter del dev $h2 ingress protocol ip pref 1 flower \ + skip_sw dst_ip 193.0.0.0/8 action drop + + router_routes_destroy +} + +router_cleanup() +{ + pre_cleanup + + router_routes_destroy + router_destroy + + router_h2_destroy + router_h1_destroy + + vrf_cleanup +}
From: Yuval Mintz yuvalm@mellanox.com
IPv4 routes in Spectrum are based on the kvd single-hash, but as it's a hash we need to assume we cannot reach 100% of its capacity.
Add a wrapper that provides us with good/bad target numbers for the Spectrum ASIC.
Signed-off-by: Yuval Mintz yuvalm@mellanox.com Reviewed-by: Petr Machata petrm@mellanox.com [petrm@mellanox.com: Drop shebang.] Signed-off-by: Petr Machata petrm@mellanox.com --- .../drivers/net/mlxsw/spectrum/router_scale.sh | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100644 tools/testing/selftests/drivers/net/mlxsw/spectrum/router_scale.sh
diff --git a/tools/testing/selftests/drivers/net/mlxsw/spectrum/router_scale.sh b/tools/testing/selftests/drivers/net/mlxsw/spectrum/router_scale.sh new file mode 100644 index 000000000000..21c4697d5bab --- /dev/null +++ b/tools/testing/selftests/drivers/net/mlxsw/spectrum/router_scale.sh @@ -0,0 +1,18 @@ +# SPDX-License-Identifier: GPL-2.0 +source ../router_scale.sh + +router_get_target() +{ + local should_fail=$1 + local target + + target=$(devlink_resource_size_get kvd hash_single) + + if [[ $should_fail -eq 0 ]]; then + target=$((target * 85 / 100)) + else + target=$((target + 1)) + fi + + echo $target +}
Add test of capacity to offload flower.
This is a generic portion of the test that is meant to be called from a driver that supplies a particular number of rules to be tested with.
Signed-off-by: Petr Machata petrm@mellanox.com Reviewed-by: Yuval Mintz yuvalm@mellanox.com --- .../selftests/drivers/net/mlxsw/tc_flower_scale.sh | 134 +++++++++++++++++++++ 1 file changed, 134 insertions(+) create mode 100644 tools/testing/selftests/drivers/net/mlxsw/tc_flower_scale.sh
diff --git a/tools/testing/selftests/drivers/net/mlxsw/tc_flower_scale.sh b/tools/testing/selftests/drivers/net/mlxsw/tc_flower_scale.sh new file mode 100644 index 000000000000..a6d733d2a4b4 --- /dev/null +++ b/tools/testing/selftests/drivers/net/mlxsw/tc_flower_scale.sh @@ -0,0 +1,134 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +# Test for resource limit of offloaded flower rules. The test adds a given +# number of flower matches for different IPv6 addresses, then generates traffic, +# and ensures each was hit exactly once. This file contains functions to set up +# a testing topology and run the test, and is meant to be sourced from a test +# script that calls the testing routine with a given number of rules. + +TC_FLOWER_NUM_NETIFS=2 + +tc_flower_h1_create() +{ + simple_if_init $h1 + tc qdisc add dev $h1 clsact +} + +tc_flower_h1_destroy() +{ + tc qdisc del dev $h1 clsact + simple_if_fini $h1 +} + +tc_flower_h2_create() +{ + simple_if_init $h2 + tc qdisc add dev $h2 clsact +} + +tc_flower_h2_destroy() +{ + tc qdisc del dev $h2 clsact + simple_if_fini $h2 +} + +tc_flower_setup_prepare() +{ + h1=${NETIFS[p1]} + h2=${NETIFS[p2]} + + vrf_prepare + + tc_flower_h1_create + tc_flower_h2_create +} + +tc_flower_cleanup() +{ + pre_cleanup + + tc_flower_h2_destroy + tc_flower_h1_destroy + + vrf_cleanup + + if [[ -v TC_FLOWER_BATCH_FILE ]]; then + rm -f $TC_FLOWER_BATCH_FILE + fi +} + +tc_flower_addr() +{ + local num=$1; shift + + printf "2001:db8:1::%x" $num +} + +tc_flower_rules_create() +{ + local count=$1; shift + local should_fail=$1; shift + + TC_FLOWER_BATCH_FILE="$(mktemp)" + + for ((i = 0; i < count; ++i)); do + cat >> $TC_FLOWER_BATCH_FILE <<-EOF + filter add dev $h2 ingress \ + prot ipv6 \ + pref 1000 \ + flower $tcflags dst_ip $(tc_flower_addr $i) \ + action drop + EOF + done + + tc -b $TC_FLOWER_BATCH_FILE + check_err_fail $should_fail $? "Rule insertion" +} + +__tc_flower_test() +{ + local count=$1; shift + local should_fail=$1; shift + local last=$((count - 1)) + + tc_flower_rules_create $count $should_fail + + for ((i = 0; i < count; ++i)); do + $MZ $h1 -q -c 1 -t ip -p 20 -b bc -6 \ + -A 2001:db8:2::1 \ + -B $(tc_flower_addr $i) + done + + MISMATCHES=$( + tc -j -s filter show dev $h2 ingress | + jq -r '[ .[] | select(.kind == "flower") | .options | + values as $rule | .actions[].stats.packets | + select(. != 1) | "(.) on ($rule.keys.dst_ip)" ] | + join(", ")' + ) + + test -z "$MISMATCHES" + check_err $? "Expected to capture 1 packet for each IP, but got $MISMATCHES" +} + +tc_flower_test() +{ + local count=$1; shift + local should_fail=$1; shift + + # We use lower 16 bits of IPv6 address for match. Also there are only 16 + # bits of rule priority space. + if ((count > 65536)); then + check_err 1 "Invalid count of $count. At most 65536 rules supported" + return + fi + + if ! tc_offload_check $TC_FLOWER_NUM_NETIFS; then + check_err 1 "Could not test offloaded functionality" + return + fi + + tcflags="skip_sw" + __tc_flower_test $count $should_fail +}
Add a wrapper around mlxsw/tc_flower_scale.sh that parameterizes the generic tc flower scale test template with Spectrum-specific target values.
Signed-off-by: Petr Machata petrm@mellanox.com Reviewed-by: Yuval Mintz yuvalm@mellanox.com --- .../drivers/net/mlxsw/spectrum/tc_flower_scale.sh | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 tools/testing/selftests/drivers/net/mlxsw/spectrum/tc_flower_scale.sh
diff --git a/tools/testing/selftests/drivers/net/mlxsw/spectrum/tc_flower_scale.sh b/tools/testing/selftests/drivers/net/mlxsw/spectrum/tc_flower_scale.sh new file mode 100644 index 000000000000..f9bfd8937765 --- /dev/null +++ b/tools/testing/selftests/drivers/net/mlxsw/spectrum/tc_flower_scale.sh @@ -0,0 +1,19 @@ +# SPDX-License-Identifier: GPL-2.0 +source ../tc_flower_scale.sh + +tc_flower_get_target() +{ + local should_fail=$1; shift + + # 6144 (6x1024) is the theoretical maximum. + # One bank of 512 rules is taken by the 18-byte MC router rule. + # One rule is the ACL catch-all. + # 6144 - 512 - 1 = 5631 + local target=5631 + + if ((! should_fail)); then + echo $target + else + echo $((target + 1)) + fi +}
Test that it's possible to offload a given number of mirrors.
Signed-off-by: Petr Machata petrm@mellanox.com Reviewed-by: Jiri Pirko jiri@mellanox.com --- .../drivers/net/mlxsw/mirror_gre_scale.sh | 197 +++++++++++++++++++++ 1 file changed, 197 insertions(+) create mode 100644 tools/testing/selftests/drivers/net/mlxsw/mirror_gre_scale.sh
diff --git a/tools/testing/selftests/drivers/net/mlxsw/mirror_gre_scale.sh b/tools/testing/selftests/drivers/net/mlxsw/mirror_gre_scale.sh new file mode 100644 index 000000000000..6f3a70df63bc --- /dev/null +++ b/tools/testing/selftests/drivers/net/mlxsw/mirror_gre_scale.sh @@ -0,0 +1,197 @@ +# SPDX-License-Identifier: GPL-2.0 + +# Test offloading a number of mirrors-to-gretap. The test creates a number of +# tunnels. Then it adds one flower mirror for each of the tunnels, matching a +# given host IP. Then it generates traffic at each of the host IPs and checks +# that the traffic has been mirrored at the appropriate tunnel. +# +# +--------------------------+ +--------------------------+ +# | H1 | | H2 | +# | + $h1 | | $h2 + | +# | | 2001:db8:1:X::1/64 | | 2001:db8:1:X::2/64 | | +# +-----|--------------------+ +--------------------|-----+ +# | | +# +-----|-------------------------------------------------------------|-----+ +# | SW o--> mirrors | | +# | +---|-------------------------------------------------------------|---+ | +# | | + $swp1 BR $swp2 + | | +# | +---------------------------------------------------------------------+ | +# | | +# | + $swp3 + gt6-<X> (ip6gretap) | +# | | 2001:db8:2:X::1/64 : loc=2001:db8:2:X::1 | +# | | : rem=2001:db8:2:X::2 | +# | | : ttl=100 | +# | | : tos=inherit | +# | | : | +# +-----|--------------------------------:----------------------------------+ +# | : +# +-----|--------------------------------:----------------------------------+ +# | H3 + $h3 + h3-gt6-<X> (ip6gretap) | +# | 2001:db8:2:X::2/64 loc=2001:db8:2:X::2 | +# | rem=2001:db8:2:X::1 | +# | ttl=100 | +# | tos=inherit | +# | | +# +-------------------------------------------------------------------------+ + +source ../../../../net/forwarding/mirror_lib.sh + +MIRROR_NUM_NETIFS=6 + +mirror_gre_ipv6_addr() +{ + local net=$1; shift + local num=$1; shift + + printf "2001:db8:%x:%x" $net $num +} + +mirror_gre_tunnels_create() +{ + local count=$1; shift + local should_fail=$1; shift + + MIRROR_GRE_BATCH_FILE="$(mktemp)" + for ((i=0; i < count; ++i)); do + local match_dip=$(mirror_gre_ipv6_addr 1 $i)::2 + local htun=h3-gt6-$i + local tun=gt6-$i + + ((mirror_gre_tunnels++)) + + ip address add dev $h1 $(mirror_gre_ipv6_addr 1 $i)::1/64 + ip address add dev $h2 $(mirror_gre_ipv6_addr 1 $i)::2/64 + + ip address add dev $swp3 $(mirror_gre_ipv6_addr 2 $i)::1/64 + ip address add dev $h3 $(mirror_gre_ipv6_addr 2 $i)::2/64 + + tunnel_create $tun ip6gretap \ + $(mirror_gre_ipv6_addr 2 $i)::1 \ + $(mirror_gre_ipv6_addr 2 $i)::2 \ + ttl 100 tos inherit allow-localremote + + tunnel_create $htun ip6gretap \ + $(mirror_gre_ipv6_addr 2 $i)::2 \ + $(mirror_gre_ipv6_addr 2 $i)::1 + ip link set $htun vrf v$h3 + matchall_sink_create $htun + + cat >> $MIRROR_GRE_BATCH_FILE <<-EOF + filter add dev $swp1 ingress pref 1000 \ + protocol ipv6 \ + flower $tcflags dst_ip $match_dip \ + action mirred egress mirror dev $tun + EOF + done + + tc -b $MIRROR_GRE_BATCH_FILE + check_err_fail $should_fail $? "Mirror rule insertion" +} + +mirror_gre_tunnels_destroy() +{ + local count=$1; shift + + for ((i=0; i < count; ++i)); do + local htun=h3-gt6-$i + local tun=gt6-$i + + ip address del dev $h3 $(mirror_gre_ipv6_addr 2 $i)::2/64 + ip address del dev $swp3 $(mirror_gre_ipv6_addr 2 $i)::1/64 + + ip address del dev $h2 $(mirror_gre_ipv6_addr 1 $i)::2/64 + ip address del dev $h1 $(mirror_gre_ipv6_addr 1 $i)::1/64 + + tunnel_destroy $htun + tunnel_destroy $tun + done +} + +__mirror_gre_test() +{ + local count=$1; shift + local should_fail=$1; shift + + mirror_gre_tunnels_create $count $should_fail + if ((should_fail)); then + return + fi + + sleep 5 + + for ((i = 0; i < count; ++i)); do + local dip=$(mirror_gre_ipv6_addr 1 $i)::2 + local htun=h3-gt6-$i + local message + + icmp6_capture_install $htun + mirror_test v$h1 "" $dip $htun 100 10 + icmp6_capture_uninstall $htun + done +} + +mirror_gre_test() +{ + local count=$1; shift + local should_fail=$1; shift + + if ! tc_offload_check $TC_FLOWER_NUM_NETIFS; then + check_err 1 "Could not test offloaded functionality" + return + fi + + tcflags="skip_sw" + __mirror_gre_test $count $should_fail +} + +mirror_gre_setup_prepare() +{ + h1=${NETIFS[p1]} + swp1=${NETIFS[p2]} + + swp2=${NETIFS[p3]} + h2=${NETIFS[p4]} + + swp3=${NETIFS[p5]} + h3=${NETIFS[p6]} + + mirror_gre_tunnels=0 + + vrf_prepare + + simple_if_init $h1 + simple_if_init $h2 + simple_if_init $h3 + + ip link add name br1 type bridge vlan_filtering 1 + ip link set dev br1 up + + ip link set dev $swp1 master br1 + ip link set dev $swp1 up + tc qdisc add dev $swp1 clsact + + ip link set dev $swp2 master br1 + ip link set dev $swp2 up + + ip link set dev $swp3 up +} + +mirror_gre_cleanup() +{ + mirror_gre_tunnels_destroy $mirror_gre_tunnels + + ip link set dev $swp3 down + + ip link set dev $swp2 down + + tc qdisc del dev $swp1 clsact + ip link set dev $swp1 down + + ip link del dev br1 + + simple_if_fini $h3 + simple_if_fini $h2 + simple_if_fini $h1 + + vrf_cleanup +}
Add a wrapper around mlxsw/mirror_gre_scale.sh that parameterized number of offloadable mirrors on Spectrum machines.
Signed-off-by: Petr Machata petrm@mellanox.com Reviewed-by: Jiri Pirko jiri@mellanox.com --- .../drivers/net/mlxsw/spectrum/mirror_gre_scale.sh | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100644 tools/testing/selftests/drivers/net/mlxsw/spectrum/mirror_gre_scale.sh
diff --git a/tools/testing/selftests/drivers/net/mlxsw/spectrum/mirror_gre_scale.sh b/tools/testing/selftests/drivers/net/mlxsw/spectrum/mirror_gre_scale.sh new file mode 100644 index 000000000000..8d2186c7c62b --- /dev/null +++ b/tools/testing/selftests/drivers/net/mlxsw/spectrum/mirror_gre_scale.sh @@ -0,0 +1,13 @@ +# SPDX-License-Identifier: GPL-2.0 +source ../mirror_gre_scale.sh + +mirror_gre_get_target() +{ + local should_fail=$1; shift + + if ((! should_fail)); then + echo 3 + else + echo 4 + fi +}
From: Yuval Mintz yuvalm@mellanox.com
Add a scale test capable of validating that offloaded network functionality is indeed functional at scale when configured to the different KVD profiles available.
Start by testing offloaded routes are functional at scale by passing traffic on each one of them in turn.
Signed-off-by: Yuval Mintz yuvalm@mellanox.com Signed-off-by: Petr Machata petrm@mellanox.com --- .../drivers/net/mlxsw/spectrum/resource_scale.sh | 55 ++++++++++++++++++++++ 1 file changed, 55 insertions(+) create mode 100755 tools/testing/selftests/drivers/net/mlxsw/spectrum/resource_scale.sh
diff --git a/tools/testing/selftests/drivers/net/mlxsw/spectrum/resource_scale.sh b/tools/testing/selftests/drivers/net/mlxsw/spectrum/resource_scale.sh new file mode 100755 index 000000000000..a0a80e1a69e8 --- /dev/null +++ b/tools/testing/selftests/drivers/net/mlxsw/spectrum/resource_scale.sh @@ -0,0 +1,55 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +NUM_NETIFS=6 +source ../../../../net/forwarding/lib.sh +source ../../../../net/forwarding/tc_common.sh +source devlink_lib_spectrum.sh + +current_test="" + +cleanup() +{ + pre_cleanup + if [ ! -z $current_test ]; then + ${current_test}_cleanup + fi + devlink_sp_size_kvd_to_default +} + +devlink_sp_read_kvd_defaults +trap cleanup EXIT + +ALL_TESTS="router tc_flower mirror_gre" +for current_test in ${TESTS:-$ALL_TESTS}; do + source ${current_test}_scale.sh + + num_netifs_var=${current_test^^}_NUM_NETIFS + num_netifs=${!num_netifs_var:-$NUM_NETIFS} + + for profile in $KVD_PROFILES; do + RET=0 + devlink_sp_resource_kvd_profile_set $profile + if [[ $RET -gt 0 ]]; then + log_test "'$current_test' [$profile] setting" + continue + fi + + for should_fail in 0 1; do + RET=0 + target=$(${current_test}_get_target "$should_fail") + ${current_test}_setup_prepare + setup_wait $num_netifs + ${current_test}_test "$target" "$should_fail" + ${current_test}_cleanup + if [[ "$should_fail" -eq 0 ]]; then + log_test "'$current_test' [$profile] $target" + else + log_test "'$current_test' [$profile] overflow $target" + fi + done + done +done +current_test="" + +exit "$RET"
From: Petr Machata petrm@mellanox.com Date: Sat, 30 Jun 2018 02:44:23 +0200
There are a number of tests that check features of the Linux networking stack. By running them on suitable interfaces, one can exercise the mlxsw offloading code. However none of these tests attempts to push mlxsw to the limits supported by the ASIC.
As an additional wrinkle, the "limits supported by the ASIC" themselves may not be a set of fixed numbers, but rather depend on a profile that determines how the ASIC resources are allocated for different purposes.
This patchset introduces several tests that verify capability of mlxsw to offload amounts of routes, flower rules, and mirroring sessions that match predicted ASIC capacity, at different configuration profiles. Additionally they verify that amounts exceeding the predicted capacity can *not* be offloaded.
...
Good stuff, series applied, thanks! -- To unsubscribe from this list: send the line "unsubscribe linux-kselftest" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
linux-kselftest-mirror@lists.linaro.org