On Thu, Feb 4, 2021 at 2:54 PM Daniel Latypov dlatypov@google.com wrote:
When using `kunit.py run` to run tests, users must populate a `kunitconfig` file to select the options the tests are hidden behind and all their dependencies.
The patch [1] to allow specifying a path to kunitconfig promises to make this nicer as we can have checked in files corresponding to different sets of tests.
But it's still annoying
- when trying to run a subet of tests
- when you want to run tests that don't have such a pre-existing
kunitconfig and selecting all the necessary options is tricky.
This patch series aims to alleviate both:
- `kunit.py run 'my-suite-*'`
I.e. use my current kunitconfig, but just run suites that match this glob 2) `kunit.py run --alltests 'my-suite-*'` I.e. use allyesconfig so I don't have to worry about writing a kunitconfig at all.
See the first commit message for more details and discussion about future work.
This patch series also includes a bugfix for a latent bug that can't be triggered right now but has worse consequences as a result of the changes needed to plumb in this suite name glob.
[1] https://lore.kernel.org/linux-kselftest/20210201205514.3943096-1-dlatypov@go...
I tried applying this series on top of several of your other patches over the past cycle and running the kunit_tool unit tests (kunit_tool_test.py) and I got several failures:
====================================================================== FAIL: test_exec_builddir (__main__.KUnitMainTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "tools/testing/kunit/kunit_tool_test.py", line 417, in test_exec_builddir self.linux_source_mock.run_kernel.assert_called_once_with(build_dir=build_dir, timeout=300) File "/usr/lib/python3.8/unittest/mock.py", line 925, in assert_called_once_with return self.assert_called_with(*args, **kwargs) File "/usr/lib/python3.8/unittest/mock.py", line 913, in assert_called_with raise AssertionError(_error_message()) from cause AssertionError: expected call not found. Expected: run_kernel(build_dir='.kunit', timeout=300) Actual: run_kernel(timeout=300, filter_glob='', build_dir='.kunit')
====================================================================== FAIL: test_exec_passes_args_pass (__main__.KUnitMainTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "tools/testing/kunit/kunit_tool_test.py", line 337, in test_exec_passes_args_pass self.linux_source_mock.run_kernel.assert_called_once_with(build_dir='.kunit', timeout=300) File "/usr/lib/python3.8/unittest/mock.py", line 925, in assert_called_once_with return self.assert_called_with(*args, **kwargs) File "/usr/lib/python3.8/unittest/mock.py", line 913, in assert_called_with raise AssertionError(_error_message()) from cause AssertionError: expected call not found. Expected: run_kernel(build_dir='.kunit', timeout=300) Actual: run_kernel(timeout=300, filter_glob='', build_dir='.kunit')
====================================================================== FAIL: test_exec_timeout (__main__.KUnitMainTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "tools/testing/kunit/kunit_tool_test.py", line 385, in test_exec_timeout self.linux_source_mock.run_kernel.assert_called_once_with(build_dir='.kunit', timeout=timeout) File "/usr/lib/python3.8/unittest/mock.py", line 925, in assert_called_once_with return self.assert_called_with(*args, **kwargs) File "/usr/lib/python3.8/unittest/mock.py", line 913, in assert_called_with raise AssertionError(_error_message()) from cause AssertionError: expected call not found. Expected: run_kernel(build_dir='.kunit', timeout=3453) Actual: run_kernel(timeout=3453, filter_glob='', build_dir='.kunit')
====================================================================== FAIL: test_run_builddir (__main__.KUnitMainTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "tools/testing/kunit/kunit_tool_test.py", line 400, in test_run_builddir self.linux_source_mock.run_kernel.assert_called_once_with( File "/usr/lib/python3.8/unittest/mock.py", line 925, in assert_called_once_with return self.assert_called_with(*args, **kwargs) File "/usr/lib/python3.8/unittest/mock.py", line 913, in assert_called_with raise AssertionError(_error_message()) from cause AssertionError: expected call not found. Expected: run_kernel(build_dir='.kunit', timeout=300) Actual: run_kernel(timeout=300, filter_glob='', build_dir='.kunit')
====================================================================== FAIL: test_run_passes_args_pass (__main__.KUnitMainTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "tools/testing/kunit/kunit_tool_test.py", line 344, in test_run_passes_args_pass self.linux_source_mock.run_kernel.assert_called_once_with( File "/usr/lib/python3.8/unittest/mock.py", line 925, in assert_called_once_with return self.assert_called_with(*args, **kwargs) File "/usr/lib/python3.8/unittest/mock.py", line 913, in assert_called_with raise AssertionError(_error_message()) from cause AssertionError: expected call not found. Expected: run_kernel(build_dir='.kunit', timeout=300) Actual: run_kernel(timeout=300, filter_glob='', build_dir='.kunit')
====================================================================== FAIL: test_run_timeout (__main__.KUnitMainTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "tools/testing/kunit/kunit_tool_test.py", line 392, in test_run_timeout self.linux_source_mock.run_kernel.assert_called_once_with( File "/usr/lib/python3.8/unittest/mock.py", line 925, in assert_called_once_with return self.assert_called_with(*args, **kwargs) File "/usr/lib/python3.8/unittest/mock.py", line 913, in assert_called_with raise AssertionError(_error_message()) from cause AssertionError: expected call not found. Expected: run_kernel(build_dir='.kunit', timeout=3453) Actual: run_kernel(timeout=3453, filter_glob='', build_dir='.kunit')
---------------------------------------------------------------------- Ran 37 tests in 0.082s
FAILED (failures=6)