Sorry for the delayed reply. I was on vacation.
On Fri, Oct 11, 2019 at 5:16 AM Theodore Ts'o theodore.tso@gmail.com wrote:
On Friday, October 11, 2019 at 7:19:49 AM UTC-4, Brendan Higgins wrote:
Should we maybe drop `--build_dir` in favor of `O`?
How about if "make kunit" results in "./tools/testing/kunit/kunit.py run --build_dir=/.kunit --allconfig"
... where --allconfig automatically creates kunitconfig but in includes all of the CONFIG options which depends on CONFIG_KUNIT, so that all unit tests are run? That way, we make it super easy for people to run the unit tests. Since most users are used using make targets, this I bet will significantly increase the number of developers using kunit, because it will be super-duper convenient for them.
Also, it would be nice if kunit.py first looks for kunitconfig in build_dir, and then in the top-level of the kernel sources, and we put .kunit in .gitignore. That will make "git status" look nice and clean.
What do folks think?
Having something like --allconfig is the ultimate goal. I had been talking to Luis and Shuah about this for some time.
I think the best way to make this work would be for kunit_tool to be able to detect all the tests with CONFIG_KUNIT as you suggest (or something like it). Luis actually already suggested it; however, we identified that this would likely not be as easy as it sounds as it is possible to have mutually exclusive CONFIGs. Luis pointed out that some researchers are currently working on a sat solver for Kconfig that we could use to potentially address this problem. Nevertheless, a complete solution in this regard is actually somewhat difficult.
Shuah's solution was just to use CONFIG fragments in the meantime similar to what kselftest already does. I was leaning in that direction since kselftest already does that and we know that it works.
Shuah, Luis, does this still match what you have been thinking?
On Wed, Oct 16, 2019 at 02:02:52PM -0700, Brendan Higgins wrote:
Shuah's solution was just to use CONFIG fragments in the meantime similar to what kselftest already does. I was leaning in that direction since kselftest already does that and we know that it works.
Shuah, Luis, does this still match what you have been thinking?
I personally never use the selftest full config thing myself, however I do use subcomponent selftests configs as hints to edit my .config to add what I need and then run 'make menuconfig', in hopes that that leaves a .config with all that is needed.
So indeed, I believe ethis works well for now, and it works for me.
I've hinted elsewhere that there is a difference between what kernel features you have enabled Vs what components are needed / should we built to test the current target kernel .config. And even then, what we test in userspace is in my view different than what should be configured in the kernel. To scale this I think a respective .config for userspace and respective symbols for testing may be in order, this way the userspace tests can only be visible say if you enabled certain features in your kernel. How this gets exposed, etc, is a separate question, however I think this can be addressed later, and I believe Knut will likely be dealing with it during the KTF merge to kunit work as currently it addresses this via generic netlink, and we want something simple to start off with.
Luis
On Fri, Oct 18, 2019 at 5:43 AM Luis Chamberlain mcgrof@kernel.org wrote:
On Wed, Oct 16, 2019 at 02:02:52PM -0700, Brendan Higgins wrote:
Shuah's solution was just to use CONFIG fragments in the meantime similar to what kselftest already does. I was leaning in that direction since kselftest already does that and we know that it works.
Shuah, Luis, does this still match what you have been thinking?
I personally never use the selftest full config thing myself, however I do use subcomponent selftests configs as hints to edit my .config to add what I need and then run 'make menuconfig', in hopes that that leaves a .config with all that is needed.
So indeed, I believe ethis works well for now, and it works for me.
Okay, good to know. So that is probably the right thing to do until we can come up with a good solution to the dynamically generating an allconfig problem.
I've hinted elsewhere that there is a difference between what kernel features you have enabled Vs what components are needed / should we built to test the current target kernel .config. And even then, what we test in userspace is in my view different than what should be configured in the kernel. To scale this I think a respective .config for userspace
Sure.
and respective symbols for testing may be in order, this way the userspace tests can only be visible say if you enabled certain features in your kernel. How this gets exposed, etc, is a separate question,
I think that is a reasonable idea, but I don't think that really applies here. I don't think it really makes sense to have the `make kunit` that Ted is describing here do anything involving userspace. That should just run the KUnit tests in the kernel. In anycase, I expressed my ideas on the matter in more detail on the hybrid testing thread[1].
however I think this can be addressed later, and I believe Knut will likely be dealing with it during the KTF merge to kunit work as currently it addresses this via generic netlink, and we want something simple to start off with.
Cheers!
[1] https://lore.kernel.org/linux-kselftest/CAFd5g459xmO+=QPhnnXVO8+dB_t1PViXxK-...
linux-kselftest-mirror@lists.linaro.org
-
Brendan Higgins -
Luis Chamberlain