# Background
KUnit currently lacks any first-class support for mocking.
For an overview and discussion on the pros and cons, see
https://martinfowler.com/articles/mocksArentStubs.html
This patch set introduces the basic machinery needed for mocking:
setting and validating expectations, setting default actions, etc.
Using that basic infrastructure, we add macros for "class mocking", as
it's probably the easiest type of mocking to start with.
## Class mocking
By "class mocking", we're referring mocking out function pointers stored
in structs like:
struct sender {
int (*send)(struct sender *sender, int data);
};
or in ops structs
struct sender {
struct send_ops *ops; // contains `send`
};
After the necessary DEFINE_* macros, we can then write code like
struct MOCK(sender) mock_sender = CONSTRUCT_MOCK(sender, test);
/* Fake an error for a specific input. */
handle = KUNIT_EXPECT_CALL(send(<omitted>, kunit_int_eq(42)));
handle->action = kunit_int_return(test, -EINVAL);
/* Pass the mocked object to some code under test. */
KUNIT_EXPECT_EQ(test, -EINVAL, send_message(...));
I.e. the goal is to make it easier to test
1) with less dependencies (we don't need to setup a real `sender`)
2) unusual/error conditions more easily.
In the future, we hope to build upon this to support mocking in more
contexts, e.g. standalone funcs, etc.
# TODOs
## Naming
This introduces a number of new macros for dealing with mocks,
e.g:
DEFINE_STRUCT_CLASS_MOCK(METHOD(foo), CLASS(example),
RETURNS(int),
PARAMS(struct example *, int));
...
KUNIT_EXPECT_CALL(foo(mock_get_ctrl(mock_example), ...);
For consistency, we could prefix everything with KUNIT, e.g.
`KUNIT_DEFINE_STRUCT_CLASS_MOCK` and `kunit_mock_get_ctrl`, but it feels
like the names might be long enough that they would hinder readability.
## Usage
For now the only use of class mocking is in kunit-example-test.c
As part of changing this from an RFC to a real patch set, we're hoping
to include at least one example.
Pointers to bits of code where this would be useful that aren't too
hairy would be appreciated.
E.g. could easily add a test for tools/perf/ui/progress.h, e.g. that
ui_progress__init() calls ui_progress_ops.init(), but that likely isn't
useful to anyone.
---
v2:
* Pass `struct kunit *` to mock init's to allow allocating ops structs.
* Update kunit-example-test.cc to do so as a more realistic example.
v1: https://lore.kernel.org/linux-kselftest/20200918183114.2571146-1-dlatypov@g…
---
Brendan Higgins (9):
kunit: test: add kunit_stream a std::stream like logger
kunit: test: add concept of post conditions
checkpatch: add support for struct MOCK(foo) syntax
kunit: mock: add parameter list manipulation macros
kunit: mock: add internal mock infrastructure
kunit: mock: add basic matchers and actions
kunit: mock: add class mocking support
kunit: mock: add struct param matcher
kunit: mock: implement nice, strict and naggy mock distinctions
Daniel Latypov (2):
Revert "kunit: move string-stream.h to lib/kunit"
kunit: expose kunit_set_failure() for use by mocking
Marcelo Schmitt (1):
kunit: mock: add macro machinery to pick correct format args
include/kunit/assert.h | 3 +-
include/kunit/kunit-stream.h | 94 +++
include/kunit/mock.h | 902 +++++++++++++++++++++++++
include/kunit/params.h | 305 +++++++++
{lib => include}/kunit/string-stream.h | 2 +
include/kunit/test.h | 9 +
lib/kunit/Makefile | 9 +-
lib/kunit/assert.c | 2 -
lib/kunit/common-mocks.c | 409 +++++++++++
lib/kunit/kunit-example-test.c | 98 +++
lib/kunit/kunit-stream.c | 110 +++
lib/kunit/mock-macro-test.c | 241 +++++++
lib/kunit/mock-test.c | 531 +++++++++++++++
lib/kunit/mock.c | 370 ++++++++++
lib/kunit/string-stream-test.c | 3 +-
lib/kunit/string-stream.c | 5 +-
lib/kunit/test.c | 15 +-
scripts/checkpatch.pl | 4 +
18 files changed, 3099 insertions(+), 13 deletions(-)
create mode 100644 include/kunit/kunit-stream.h
create mode 100644 include/kunit/mock.h
create mode 100644 include/kunit/params.h
rename {lib => include}/kunit/string-stream.h (95%)
create mode 100644 lib/kunit/common-mocks.c
create mode 100644 lib/kunit/kunit-stream.c
create mode 100644 lib/kunit/mock-macro-test.c
create mode 100644 lib/kunit/mock-test.c
create mode 100644 lib/kunit/mock.c
base-commit: 10b82d5176488acee2820e5a2cf0f2ec5c3488b6
--
2.28.0.1011.ga647a8990f-goog
ASSERT_GE() is defined as:
/**
* ASSERT_GE(expected, seen)
*
* @expected: expected value
* @seen: measured value
*
* ASSERT_GE(expected, measured): expected >= measured
*/
#define ASSERT_GE(expected, seen) \
__EXPECT(expected, #expected, seen, #seen, >=, 1)
but that means that logically, if you want to write "assert that the
measured PID X is >= the expected value 0", you actually have to use
ASSERT_LE(0, X). That's really awkward. Normally you'd be talking
about how the seen value compares to the expected one, not the other
way around.
At the moment I see tests that are instead written like ASSERT_GE(X,
0), but then that means that the expected and seen values are the
wrong way around.
It might be good if someone could refactor the definitions of
ASSERT_GE and such to swap around which number is the expected and
which is the seen one.
Nowadays, there are increasing requirements to benchmark the performance
of dma_map and dma_unmap particually while the device is attached to an
IOMMU.
This patchset provides the benchmark infrastruture for streaming DMA
mapping. The architecture of the code is pretty much similar with GUP
benchmark:
* mm/gup_benchmark.c provides kernel interface;
* tools/testing/selftests/vm/gup_benchmark.c provides user program to
call the interface provided by mm/gup_benchmark.c.
In our case, kernel/dma/map_benchmark.c is like mm/gup_benchmark.c;
tools/testing/selftests/dma/dma_map_benchmark.c is like tools/testing/
selftests/vm/gup_benchmark.c
A major difference with GUP benchmark is DMA_MAP benchmark needs to run
on a device. Considering one board with below devices and IOMMUs
device A ------- IOMMU 1
device B ------- IOMMU 2
device C ------- non-IOMMU
Different devices might attach to different IOMMU or non-IOMMU. To make
benchmark run, we can either
* create a virtual device and hack the kernel code to attach the virtual
device to IOMMU1, IOMMU2 or non-IOMMU.
* use the existing driver_override mechinism, unbind device A,B, OR c from
their original driver and bind A to dma_map_benchmark platform driver or
pci driver for benchmarking.
In this patchset, I prefer to use the driver_override and avoid the ugly
hack in kernel. We can dynamically switch device behind different IOMMUs
to get the performance of IOMMU or non-IOMMU.
-v2:
* add PCI support; v1 supported platform devices only
* replace ssleep by msleep_interruptible() to permit users to exit
benchmark before it is completed
* many changes according to Robin's suggestions, thanks! Robin
- add standard deviation output to reflect the worst case
- check users' parameters strictly like the number of threads
- make cache dirty before dma_map
- fix unpaired dma_map_page and dma_unmap_single;
- remove redundant "long long" before ktime_to_ns();
- use devm_add_action();
- wakeup all threads together after they are ready
Barry Song (2):
dma-mapping: add benchmark support for streaming DMA APIs
selftests/dma: add test application for DMA_MAP_BENCHMARK
MAINTAINERS | 6 +
kernel/dma/Kconfig | 8 +
kernel/dma/Makefile | 1 +
kernel/dma/map_benchmark.c | 295 ++++++++++++++++++
tools/testing/selftests/dma/Makefile | 6 +
tools/testing/selftests/dma/config | 1 +
.../testing/selftests/dma/dma_map_benchmark.c | 87 ++++++
7 files changed, 404 insertions(+)
create mode 100644 kernel/dma/map_benchmark.c
create mode 100644 tools/testing/selftests/dma/Makefile
create mode 100644 tools/testing/selftests/dma/config
create mode 100644 tools/testing/selftests/dma/dma_map_benchmark.c
--
2.25.1
Nowadays, there are increasing requirements to benchmark the performance
of dma_map and dma_unmap particually while the device is attached to an
IOMMU.
This patchset provides the benchmark infrastruture for streaming DMA
mapping. The architecture of the code is pretty much similar with GUP
benchmark:
* mm/gup_benchmark.c provides kernel interface;
* tools/testing/selftests/vm/gup_benchmark.c provides user program to
call the interface provided by mm/gup_benchmark.c.
In our case, kernel/dma/map_benchmark.c is like mm/gup_benchmark.c;
tools/testing/selftests/dma/dma_map_benchmark.c is like tools/testing/
selftests/vm/gup_benchmark.c
A major difference with GUP benchmark is DMA_MAP benchmark needs to run
on a device. Considering one board with below devices and IOMMUs
device A ------- IOMMU 1
device B ------- IOMMU 2
device C ------- non-IOMMU
Different devices might attach to different IOMMU or non-IOMMU. To make
benchmark run, we can either
* create a virtual device and hack the kernel code to attach the virtual
device to IOMMU1, IOMMU2 or non-IOMMU.
* use the existing driver_override mechinism, unbind device A,B, or c from
their original driver and bind them to "dma_map_benchmark" platform_driver
or pci_driver for benchmarking.
In this patchset, I prefer to use the driver_override and avoid the various
hack in kernel. We can dynamically switch devices behind different IOMMUs
to get the performance of dma map on IOMMU or non-IOMMU.
Barry Song (2):
dma-mapping: add benchmark support for streaming DMA APIs
selftests/dma: add test application for DMA_MAP_BENCHMARK
MAINTAINERS | 6 +
kernel/dma/Kconfig | 8 +
kernel/dma/Makefile | 1 +
kernel/dma/map_benchmark.c | 202 ++++++++++++++++++
tools/testing/selftests/dma/Makefile | 6 +
tools/testing/selftests/dma/config | 1 +
.../testing/selftests/dma/dma_map_benchmark.c | 72 +++++++
7 files changed, 296 insertions(+)
create mode 100644 kernel/dma/map_benchmark.c
create mode 100644 tools/testing/selftests/dma/Makefile
create mode 100644 tools/testing/selftests/dma/config
create mode 100644 tools/testing/selftests/dma/dma_map_benchmark.c
--
2.25.1
For simplcity, strip all trailing whitespace from parsed output.
I imagine no one is printing out meaningful trailing whitespace via
KUNIT_FAIL() or similar, and that if they are, they really shouldn't.
`isolate_kunit_output()` yielded liens with trailing \n, which results
in artifacty output like this:
$ ./tools/testing/kunit/kunit.py run
[16:16:46] [FAILED] example_simple_test
[16:16:46] # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29
[16:16:46] Expected 1 + 1 == 3, but
[16:16:46] 1 + 1 == 2
[16:16:46] 3 == 3
[16:16:46] not ok 1 - example_simple_test
[16:16:46]
After this change:
[16:16:46] # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29
[16:16:46] Expected 1 + 1 == 3, but
[16:16:46] 1 + 1 == 2
[16:16:46] 3 == 3
[16:16:46] not ok 1 - example_simple_test
[16:16:46]
We should *not* be expecting lines to end with \n in kunit_tool_test.py
for this reason.
Do the same for `raw_output()` as well which suffers from the same
issue.
This is a followup to [1], but rebased onto kunit-fixes to pick up the
other raw_output() fix and fixes for kunit_tool_test.py.
[1] https://lore.kernel.org/linux-kselftest/20201020233219.4146059-1-dlatypov@g…
Signed-off-by: Daniel Latypov <dlatypov(a)google.com>
---
tools/testing/kunit/kunit_parser.py | 3 ++-
tools/testing/kunit/kunit_tool_test.py | 4 ++--
2 files changed, 4 insertions(+), 3 deletions(-)
diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py
index 84a1af2581f5..edd6fbd1cf18 100644
--- a/tools/testing/kunit/kunit_parser.py
+++ b/tools/testing/kunit/kunit_parser.py
@@ -54,6 +54,7 @@ kunit_end_re = re.compile('(List of all partitions:|'
def isolate_kunit_output(kernel_output):
started = False
for line in kernel_output:
+ line = line.rstrip() # line always has a trailing \n
if kunit_start_re.search(line):
prefix_len = len(line.split('TAP version')[0])
started = True
@@ -65,7 +66,7 @@ def isolate_kunit_output(kernel_output):
def raw_output(kernel_output):
for line in kernel_output:
- print(line)
+ print(line.rstrip())
DIVIDER = '=' * 60
diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py
index 0b60855fb819..497ab51bc170 100755
--- a/tools/testing/kunit/kunit_tool_test.py
+++ b/tools/testing/kunit/kunit_tool_test.py
@@ -102,7 +102,7 @@ class KUnitParserTest(unittest.TestCase):
'test_data/test_output_isolated_correctly.log')
file = open(log_path)
result = kunit_parser.isolate_kunit_output(file.readlines())
- self.assertContains('TAP version 14\n', result)
+ self.assertContains('TAP version 14', result)
self.assertContains(' # Subtest: example', result)
self.assertContains(' 1..2', result)
self.assertContains(' ok 1 - example_simple_test', result)
@@ -115,7 +115,7 @@ class KUnitParserTest(unittest.TestCase):
'test_data/test_pound_sign.log')
with open(log_path) as file:
result = kunit_parser.isolate_kunit_output(file.readlines())
- self.assertContains('TAP version 14\n', result)
+ self.assertContains('TAP version 14', result)
self.assertContains(' # Subtest: kunit-resource-test', result)
self.assertContains(' 1..5', result)
self.assertContains(' ok 1 - kunit_resource_test_init_resources', result)
base-commit: cab67acc8a18d7c6f1850313e3da1a030abe8fc4
--
2.29.1.341.ge80a0c044ae-goog
For simplcity, strip all trailing whitespace from parsed output.
I imagine no one is printing out meaningful trailing whitespace via
KUNIT_FAIL() or similar, and that if they are, they really shouldn't.
At some point, the lines from `isolate_kunit_output()` started having
trailing \n, which results in artifacty output like this:
$ ./tools/testing/kunit/kunit.py run
[16:16:46] [FAILED] example_simple_test
[16:16:46] # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29
[16:16:46] Expected 1 + 1 == 3, but
[16:16:46] 1 + 1 == 2
[16:16:46] 3 == 3
[16:16:46] not ok 1 - example_simple_test
[16:16:46]
After this change:
[16:16:46] # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29
[16:16:46] Expected 1 + 1 == 3, but
[16:16:46] 1 + 1 == 2
[16:16:46] 3 == 3
[16:16:46] not ok 1 - example_simple_test
[16:16:46]
Signed-off-by: Daniel Latypov <dlatypov(a)google.com>
---
tools/testing/kunit/kunit_parser.py | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py
index 8019e3dd4c32..e68b1c66a73f 100644
--- a/tools/testing/kunit/kunit_parser.py
+++ b/tools/testing/kunit/kunit_parser.py
@@ -342,7 +342,8 @@ def parse_run_tests(kernel_output) -> TestResult:
total_tests = 0
failed_tests = 0
crashed_tests = 0
- test_result = parse_test_result(list(isolate_kunit_output(kernel_output)))
+ test_result = parse_test_result(list(
+ l.rstrip() for l in isolate_kunit_output(kernel_output)))
if test_result.status == TestStatus.NO_TESTS:
print(red('[ERROR] ') + yellow('no tests run!'))
elif test_result.status == TestStatus.FAILURE_TO_PARSE_TESTS:
base-commit: c4d6fe7311762f2e03b3c27ad38df7c40c80cc93
--
2.29.0.rc1.297.gfa9743e501-goog
When JSON support was added in [1], the KunitParseRequest tuple was
updated to contain a 'build_dir' field, but kunit.py parse doesn't
accept --build_dir as an option. The code nevertheless tried to access
it, resulting in this error:
AttributeError: 'Namespace' object has no attribute 'build_dir'
Given that the parser only uses the build_dir variable to set the
'build_environment' json field, we set it to None (which gives the JSON
'null') for now. Ultimately, we probably do want to be able to set this,
but since it's new functionality which (for the parse subcommand) never
worked, this is the quickest way of getting it back up and running.
[1]: https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git/c…
Fixes: 21a6d1780d5bbfca0ce9b8104ca6233502fcbf86 ("kunit: tool: allow generating test results in JSON")
Signed-off-by: David Gow <davidgow(a)google.com>
---
This is a quick fix because kunit.py parse is completely broken: it
appears it was introduced in the rebase of the JSON parser after the
separation of concerns patch.
tools/testing/kunit/kunit.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py
index ebf5f5763dee..a6d5f219f714 100755
--- a/tools/testing/kunit/kunit.py
+++ b/tools/testing/kunit/kunit.py
@@ -337,7 +337,7 @@ def main(argv, linux=None):
kunit_output = f.read().splitlines()
request = KunitParseRequest(cli_args.raw_output,
kunit_output,
- cli_args.build_dir,
+ None,
cli_args.json)
result = parse_tests(request)
if result.status != KunitStatus.SUCCESS:
base-commit: c4d6fe7311762f2e03b3c27ad38df7c40c80cc93
--
2.29.0.rc1.297.gfa9743e501-goog