On Thu, Oct 7, 2021 at 1:00 AM Daniel Latypov dlatypov@google.com wrote:
From: Rae Moar rmoar@google.com
Update to kunit_parser to improve compatibility with KTAP specification including arbitrarily nested tests. Patch accomplishes three major changes:
- Use a general Test object to represent all tests rather than TestCase
and TestSuite objects. This allows for easier implementation of arbitrary levels of nested tests and promotes the idea that both test suites and test cases are tests.
- Print errors incrementally rather than all at once after the
parsing finishes to maximize information given to the user in the case of the parser given invalid input and to increase the helpfulness of the timestamps given during printing. Note that kunit.py parse does not print incrementally yet. However, this fix brings us closer to this feature.
- Increase compatibility for different formats of input. Arbitrary levels
of nested tests supported. Also, test cases and test suites are now supported to be present on the same level of testing.
This patch now implements the draft KTAP specification here: https://lore.kernel.org/linux-kselftest/CA+GJov6tdjvY9x12JsJT14qn6c7NViJxqaJ... We'll update the parser as the spec evolves.
This patch adjusts the kunit_tool_test.py file to check for the correct outputs from the new parser and adds a new test to check the parsing for a KTAP result log with correct format for multiple nested subtests (test_is_test_passed-all_passed_nested.log).
This patch also alters the kunit_json.py file to allow for arbitrarily nested tests.
Signed-off-by: Rae Moar rmoar@google.com Reviewed-by: Brendan Higgins brendanhiggins@google.com Signed-off-by: Daniel Latypov dlatypov@google.com Reviewed-by: David Gow davidgow@google.com
NOTE: this patch is now applied on top of https://lore.kernel.org/linux-kselftest/20210930222048.1692635-5-dlatypov@go... to resolve a conflict.
Change log from v5: https://lore.kernel.org/linux-kselftest/20211006001447.20919-1-dlatypov@goog...
- Tweak commit message to reflect the KTAP spec is a draft
- Add missing Signed-off-by
- Tweak docstrings
Change log from v3,4: https://lore.kernel.org/linux-kselftest/20210901190623.315736-1-rmoar@google...
- Move test_kselftest_nested from LinuxSourceTreeTest => KUnitParserTest.
- Resolve conflict with hermetic testing patches.
- max_status is no longer defined, so we need to use the TestCounts type now. And to keep --raw_output working, we need to set this to SUCCESS to avoid the default assumption that the kernel crashed.
Ignore v4, was accidentally based on v2.
Change log from v2: https://lore.kernel.org/linux-kselftest/20210826195505.3066755-1-rmoar@googl...
- Fixes bug of type disagreement in kunit_json.py for build_dir
- Removes raw_output()
- Changes docstrings in kunit_parser.py (class docstring, LineStream docstrings, add_error(), total(), get_status(), all parsing methods)
- Fixes bug of not printing diagnostic log in the case of end of lines
- Sets default status of all tests to TEST_CRASHED
- Adds and prints empty tests with crashed status in case of missing tests
- Prints 'subtest' in instance of 1 subtest instead of 'subtests'
- Includes checking for 'BUG:' message in search of crash messages in log (note that parse_crash_in_log method could be removed but would require deleting tests in kunit_tool_test.py that include the crash message that is no longer used. If removed, parser would still print log in cases of test crashed or failure, which would now include missing subtests)
- Fixes bug of including directives (other than SKIP) in test name when matching name in result line for subtests
Change log from v1: https://lore.kernel.org/linux-kselftest/20210820200032.2178134-1-rmoar@googl...
- Rebase onto kselftest/kunit branch
- Add tests to kunit_tool_test.py to check parser is correctly stripping hyphen, producing correct json objects with nested tests, correctly passing kselftest TAP output, and correctly deals with missing test plan.
- Fix bug to correctly match test name in instance of a missing test plan.
- Fix bug in kunit_tool_test.py pointed out by Daniel where it was not correctly checking for a proper match to the '0 tests run!' error message. Reverts changes back to original.
- A few minor changes to commit message using Daniel's comments.
- Change docstrings using Daniel's comments to reduce:
- Shortens some docstrings to be one-line or just description if it is self explanatory.
- Remove explicit respecification of types of parameters and returns because this is already specified in the function annoations. However, some descriptions of the parameters and returns remain and some contain the type for context. Additionally, the types of public attributes of classes remain.
- Remove any documentation of 'Return: None'
- Remove docstrings of helper methods within other methods
Looks good to me, thanks!
This is still: Reviewed-by: David Gow davidgow@google.com
Cheers, -- David