We (Linaro) started exploring options to run packetdrill on Linux kernel for improving network subsystem test coverage. The following results obtained on arm64 and x86_64 architectures after installing required packages on the debian rootfs. The run time of testing is less than a minute. Full run test log attached to this email.
Please suggest ways to improve the number of PASS. Our plan is to add the packetdrill test suite to LKFT [1] test plan.
arm64: Ran 234 tests: 199 passing, 35 failing, 0 timed out (36.00 sec): .
x86_64: Ran 234 tests: 192 passing, 42 failing, 0 timed out (23.14 sec): .
steps: - apt-get update - apt-get -y install build-essential python git python3 automake - apt-get -y install iproute2 net-tools ethtool bison flex - git clone https://github.com/google/packetdrill - cd packetdrill/gtests/net/packetdrill/ - ./configure - make - cd ../ - ./packetdrill/run_all.py -v -l -L
Errors list: ------------- - error handling packet: live packet payload: expected 1000 bytes vs actual 2000 bytes - runtime error in socket call: Expected non-negative result but got -1 with errno 93 (Protocol not supported) - timing error: expected system call return at 0.300556 sec but happened at 0.310599 sec; tolerance 0.010000 sec - sysctl: cannot stat /proc/sys/net/ipv4/tcp_syncookies: No such file or directory - comment CONFIG_SYN_COOKIES not enabled on arm64 but set on x86_64. - sysctl: cannot stat /proc/sys/net/ipv4/tcp_syncookies: No such file or directory - cwnd-moderation-ecn-enter-cwr-no-moderation-700.pkt:24: error in Python code Traceback (most recent call last): File "/tmp/code_nVK70l", line 195, in <module> assert tcpi_snd_cwnd == 4, tcpi_snd_cwnd AssertionError: 5 cwnd-moderation-ecn-enter-cwr-no-moderation-700.pkt: error executing code: '/usr/bin/python' returned non-zero status 1
[1] https://lkft.linaro.org/tests/
Ping. May i request to review this email please.
On Mon, 22 Feb 2021 at 12:48, Naresh Kamboju naresh.kamboju@linaro.org wrote:
We (Linaro) started exploring options to run packetdrill on Linux kernel for improving network subsystem test coverage. The following results obtained on arm64 and x86_64 architectures after installing required packages on the debian rootfs. The run time of testing is less than a minute. Full run test log attached to this email.
Please suggest ways to improve the number of PASS. Our plan is to add the packetdrill test suite to LKFT [1] test plan.
arm64: Ran 234 tests: 199 passing, 35 failing, 0 timed out (36.00 sec): .
x86_64: Ran 234 tests: 192 passing, 42 failing, 0 timed out (23.14 sec): .
steps: - apt-get update - apt-get -y install build-essential python git python3 automake - apt-get -y install iproute2 net-tools ethtool bison flex - git clone https://github.com/google/packetdrill - cd packetdrill/gtests/net/packetdrill/ - ./configure - make - cd ../ - ./packetdrill/run_all.py -v -l -L
Errors list:
- error handling packet: live packet payload: expected 1000 bytes vs
actual 2000 bytes
- runtime error in socket call: Expected non-negative result but got
-1 with errno 93 (Protocol not supported)
- timing error: expected system call return at 0.300556 sec but
happened at 0.310599 sec; tolerance 0.010000 sec
- sysctl: cannot stat /proc/sys/net/ipv4/tcp_syncookies: No such file
or directory - comment CONFIG_SYN_COOKIES not enabled on arm64 but set on x86_64.
- sysctl: cannot stat /proc/sys/net/ipv4/tcp_syncookies: No such file
or directory
- cwnd-moderation-ecn-enter-cwr-no-moderation-700.pkt:24: error in Python code Traceback (most recent call last): File "/tmp/code_nVK70l", line 195, in <module> assert tcpi_snd_cwnd == 4, tcpi_snd_cwnd AssertionError: 5
cwnd-moderation-ecn-enter-cwr-no-moderation-700.pkt: error executing code: '/usr/bin/python' returned non-zero status 1
[1] https://lkft.linaro.org/tests/
-- Linaro LKFT https://lkft.linaro.org
Hi,
Thanks for the report!
What exact kernel version are you testing?
We can offer advice, but I'm afraid our team does not have cycles to track down and fix those test failures in your environment.
In some cases, you will probably want to update your environment. For example, I would recommend enabling CONFIG_SYN_COOKIES, since that should give a much more robust and typical Linux TCP stack.
In other cases, your kernels may have different behavior, for various reasons (local patches or atypical .config), that will require you to modify the tests to expect your behavior.
In other cases the upstream net-next behavior may have diverged from what these tests in the github packetdrill repo expect. If someone in the Linux networking community wants to submit pull requests to update the tests to expect the latest net-next behavior, we are happy to review and merge those. Our team has cycles to maintain our internal branches of these tests to match the behavior in our environment, but unfortunately does not have time to strictly keep all upstream tests up-to-date with respect to matching net-next behavior.
best, neal
On Mon, Feb 22, 2021 at 2:19 AM Naresh Kamboju naresh.kamboju@linaro.org wrote:
We (Linaro) started exploring options to run packetdrill on Linux kernel for improving network subsystem test coverage. The following results obtained on arm64 and x86_64 architectures after installing required packages on the debian rootfs. The run time of testing is less than a minute. Full run test log attached to this email.
Please suggest ways to improve the number of PASS. Our plan is to add the packetdrill test suite to LKFT [1] test plan.
arm64: Ran 234 tests: 199 passing, 35 failing, 0 timed out (36.00 sec): .
x86_64: Ran 234 tests: 192 passing, 42 failing, 0 timed out (23.14 sec): .
steps: - apt-get update - apt-get -y install build-essential python git python3 automake - apt-get -y install iproute2 net-tools ethtool bison flex - git clone https://github.com/google/packetdrill - cd packetdrill/gtests/net/packetdrill/ - ./configure - make - cd ../ - ./packetdrill/run_all.py -v -l -L
Errors list:
- error handling packet: live packet payload: expected 1000 bytes vs
actual 2000 bytes
- runtime error in socket call: Expected non-negative result but got
-1 with errno 93 (Protocol not supported)
- timing error: expected system call return at 0.300556 sec but
happened at 0.310599 sec; tolerance 0.010000 sec
- sysctl: cannot stat /proc/sys/net/ipv4/tcp_syncookies: No such file
or directory - comment CONFIG_SYN_COOKIES not enabled on arm64 but set on x86_64.
- sysctl: cannot stat /proc/sys/net/ipv4/tcp_syncookies: No such file
or directory
- cwnd-moderation-ecn-enter-cwr-no-moderation-700.pkt:24: error in Python
code Traceback (most recent call last): File "/tmp/code_nVK70l", line 195, in <module> assert tcpi_snd_cwnd == 4, tcpi_snd_cwnd AssertionError: 5 cwnd-moderation-ecn-enter-cwr-no-moderation-700.pkt: error executing code: '/usr/bin/python' returned non-zero status 1
[1] https://lkft.linaro.org/tests/
-- Linaro LKFT https://lkft.linaro.org
On Mon, 1 Mar 2021 at 21:19, Neal Cardwell ncardwell@google.com wrote:
Hi,
Thanks for the report!
You are welcome !
What exact kernel version are you testing?
We are running on Linux mainline master.
We can offer advice, but I'm afraid our team does not have cycles to track down and fix those test failures in your environment.
No. We are not expecting that. We would like to understand your testing environment (Testing board setup / Interfaces / userspace tools ) and Kconfigs that you are building with. I have attached the Kconfig file that we are building at our end.
In some cases, you will probably want to update your environment. For example, I would recommend enabling CONFIG_SYN_COOKIES, since that should give a much more robust and typical Linux TCP stack.
The config CONFIG_SYN_COOKIES is enabled now in our test CI.
In other cases, your kernels may have different behavior, for various reasons (local patches or atypical .config), that will require you to modify the tests to expect your behavior.
We (Linaro) are testing on the tip of Linux mainline and Linux next master branches and stable branch kernels. Packetdrill is not yet added to LKFT [1] test plan.
In other cases the upstream net-next behavior may have diverged from what these tests in the github packetdrill repo expect. If someone in the Linux networking community wants to submit pull requests to update the tests to expect the latest net-next behavior, we are happy to review and merge those. Our team has cycles to maintain our internal branches of these tests to match the behavior in our environment, but unfortunately does not have time to strictly keep all upstream tests up-to-date with respect to matching net-next behavior.
I completely understand that.
While running tests we have seen python errors, @Anders fixed this problem. The patch attached here.
Complaining that print is missing parentheses.
File ". packetdrill run_all.py", line 114 print 'stdout: ' ^ SyntaxError: Missing parentheses in call to 'print'. Did you mean print'stdout: '?
I have parsed the results in a LAVA[2] friendly way.
Results link, [1] https://lkft.linaro.org/tests/ [2] https://lavalab.nxp.com/results/285999/0_packetdrill?search=&length=1000...
- Naresh
-
Naresh Kamboju -
Neal Cardwell