On Mon, Sep 14, 2020 at 01:59:15PM -0700, Linus Torvalds wrote:
> On Mon, Sep 14, 2020 at 1:45 PM Thomas Gleixner <tglx(a)linutronix.de> wrote:
> >
> > Recently merged code does:
> >
> > gfp = preemptible() ? GFP_KERNEL : GFP_ATOMIC;
> >
> > Looks obviously correct, except for the fact that preemptible() is
> > unconditionally false for CONFIF_PREEMPT_COUNT=n, i.e. all allocations in
> > that code use GFP_ATOMIC on such kernels.
>
> I don't think this is a good reason to entirely get rid of the no-preempt thing.
>
> The above is just garbage. It's bogus. You can't do it.
>
> Blaming the no-preempt code for this bug is extremely unfair, imho.
>
> And the no-preempt code does help make for much better code generation
> for simple spinlocks.
>
> Where is that horribly buggy recent code? It's not in that exact
> format, certainly, since 'grep' doesn't find it.
It would be convenient for that "gfp =" code to work, as this would
allow better cache locality while invoking RCU callbacks, and would
further provide better robustness to callback floods. The full story
is quite long, but here are alternatives have not yet been proven to be
abject failures:
1. Use workqueues to do the allocations in a clean context.
While waiting for the allocations, the callbacks are queued
in the old cache-busting manner. This functions correctly,
but in the meantime (which on busy systems can be some time)
the cache locality and robustness are lost.
2. Provide the ability to allocate memory in raw atomic context.
This is extremely effective, especially when used in combination
with #1 above, but as you might suspect, the MM guys don't like
it much.
In contrast, with Thomas's patch series, call_rcu() and kvfree_rcu()
could just look at preemptible() to see whether or not it was safe to
allocate memory, even in !PREEMPT kernels -- and in the common case,
it almost always would be safe. It is quite possible that this approach
would work in isolation, or failing that, that adding #1 above would do
the trick.
I understand that this is all very hand-wavy, and I do apologize for that.
If you really want the full sad story with performance numbers and the
works, let me know!
Thanx, Paul
Hi,
This fixes a couple of minor aggravating factors that I ran across while
trying to do some changes in selftests/vm. These are simple things, but
like most things with GNU Make, it's rarely obvious what's wrong until
you understand *the entire Makefile and all of its includes*.
So while there is, of course, joy in learning those details, I thought I'd
fix these little things, so as to allow others to skip out on the Joy if
they so choose. :)
First of all, if you have an item (let's choose userfaultfd for an
example) that fails to build, you might do this:
$ make -j32
# ...you observe a failed item in the threaded output
# OK, let's get a closer look
$ make
# ...but now the build quietly "succeeds".
That's what Patch 0001 fixes.
Second, if you instead attempt this approach for your closer look (a casual
mistake, as it's not supported):
$ make userfaultfd
# ...userfaultfd fails to link, due to incomplete LDLIBS
That's what Patch 0002 fixes.
John Hubbard (2):
selftests/vm: fix false build success on the second and later attempts
selftests/vm: fix incorrect gcc invocation in some cases
tools/testing/selftests/vm/Makefile | 17 +++++++++++++++++
1 file changed, 17 insertions(+)
--
2.28.0
On Tue, 15 Sep 2020 at 01:43, Linus Torvalds
<torvalds(a)linux-foundation.org> wrote:
>
> On Mon, Sep 14, 2020 at 3:24 PM Linus Torvalds
> <torvalds(a)linux-foundation.org> wrote:
> >
> > Ard and Herbert added to participants: see
> > chacha20poly1305_crypt_sg_inplace(), which does
> >
> > flags = SG_MITER_TO_SG;
> > if (!preemptible())
> > flags |= SG_MITER_ATOMIC;
> >
> > introduced in commit d95312a3ccc0 ("crypto: lib/chacha20poly1305 -
> > reimplement crypt_from_sg() routine").
>
> As far as I can tell, the only reason for this all is to try to use
> "kmap()" rather than "kmap_atomic()".
>
> And kmap() actually has the much more complex "might_sleep()" tests,
> and apparently the "preemptible()" check wasn't even the proper full
> debug check, it was just a complete hack to catch the one that
> triggered.
>
This was not driven by a failing check.
The documentation of kmap_atomic() states the following:
* The use of kmap_atomic/kunmap_atomic is discouraged - kmap/kunmap
* gives a more generic (and caching) interface. But kmap_atomic can
* be used in IRQ contexts, so in some (very limited) cases we need
* it.
so if this is no longer accurate, perhaps we should fix it?
But another reason I tried to avoid kmap_atomic() is that it disables
preemption unconditionally, even on 64-bit architectures where HIGHMEM
is irrelevant. So using kmap_atomic() here means that the bulk of
WireGuard packet encryption runs with preemption disabled, essentially
for legacy reasons.
> From a quick look, that code should probably just get rid of
> SG_MITER_ATOMIC entirely, and alwayse use kmap_atomic().
>
> kmap_atomic() is actually the faster and proper interface to use
> anyway (never mind that any of this matters on any sane hardware). The
> old kmap() and kunmap() interfaces should generally be avoided like
> the plague - yes, they allow sleeping in the middle and that is
> sometimes required, but if you don't need that, you should never ever
> use them.
>
> We used to have a very nasty kmap_atomic() that required people to be
> very careful and know exactly which atomic entry to use, and that was
> admitedly quite nasty.
>
> So it _looks_ like this code started using kmap() - probably back when
> kmap_atomic() was so cumbersome to use - and was then converted
> (conditionally) to kmap_atomic() rather than just changed whole-sale.
> Is there actually something that wants to use those sg_miter functions
> and sleep?
>
> Because if there is, that choice should come from the outside, not
> from inside lib/scatterlist.c trying to make some bad guess based on
> the wrong thing entirely.
>
> Linus
This patchset contains everything needed to integrate KASAN and KUnit.
KUnit will be able to:
(1) Fail tests when an unexpected KASAN error occurs
(2) Pass tests when an expected KASAN error occurs
Convert KASAN tests to KUnit with the exception of copy_user_test
because KUnit is unable to test those.
Add documentation on how to run the KASAN tests with KUnit and what to
expect when running these tests.
The dependencies for this patchset are all present in 5.9-rc1+.
Changes from v13:
- Fix some compile warnings in test_kasan_module[9]
Changes from v12:
- Rebased on top of mainline (ab29a807)
- Updated to match latest KUnit guidelines (no longer rename the test)
- Fix some small issues with the documentation to match the correct
test name and mention the module name.
Changes from v11:
- Rebased on top of latest -next (20200810)
- Fixed a redundant memchr() call in kasan_memchr()
- Added Andrey's "Tested-by" to everything.
Changes from v10:
- Fixed some whitespace issues in patch 2.
- Split out the renaming of the KUnit test suite into a separate patch.
Changes from v9:
- Rebased on top of linux-next (20200731) + kselftest/kunit and [7]
- Note that the kasan_rcu_uaf test has not been ported to KUnit, and
remains in test_kasan_module. This is because:
(a) KUnit's expect failure will not check if the RCU stacktraces
show.
(b) KUnit is unable to link the failure to the test, as it occurs in
an RCU callback.
Changes from v8:
- Rebased on top of kselftest/kunit
- (Which, with this patchset, should rebase cleanly on 5.8-rc7)
- Renamed the KUnit test suite, config name to patch the proposed
naming guidelines for KUnit tests[6]
Changes from v7:
- Rebased on top of kselftest/kunit
- Rebased on top of v4 of the kunit resources API[1]
- Rebased on top of v4 of the FORTIFY_SOURCE fix[2,3,4]
- Updated the Kconfig entry to support KUNIT_ALL_TESTS
Changes from v6:
- Rebased on top of kselftest/kunit
- Rebased on top of Daniel Axtens' fix for FORTIFY_SOURCE
incompatibilites [2]
- Removed a redundant report_enabled() check.
- Fixed some places with out of date Kconfig names in the
documentation.
Changes from v5:
- Split out the panic_on_warn changes to a separate patch.
- Fix documentation to fewer to the new Kconfig names.
- Fix some changes which were in the wrong patch.
- Rebase on top of kselftest/kunit (currently identical to 5.7-rc1)
Changes from v4:
- KASAN no longer will panic on errors if both panic_on_warn and
kasan_multishot are enabled.
- As a result, the KASAN tests will no-longer disable panic_on_warn.
- This also means panic_on_warn no-longer needs to be exported.
- The use of temporary "kasan_data" variables has been cleaned up
somewhat.
- A potential refcount/resource leak should multiple KASAN errors
appear during an assertion was fixed.
- Some wording changes to the KASAN test Kconfig entries.
Changes from v3:
- KUNIT_SET_KASAN_DATA and KUNIT_DO_EXPECT_KASAN_FAIL have been
combined and included in KUNIT_DO_EXPECT_KASAN_FAIL() instead.
- Reordered logic in kasan_update_kunit_status() in report.c to be
easier to read.
- Added comment to not use the name "kasan_data" for any kunit tests
outside of KUNIT_EXPECT_KASAN_FAIL().
Changes since v2:
- Due to Alan's changes in [1], KUnit can be built as a module.
- The name of the tests that could not be run with KUnit has been
changed to be more generic: test_kasan_module.
- Documentation on how to run the new KASAN tests and what to expect
when running them has been added.
- Some variables and functions are now static.
- Now save/restore panic_on_warn in a similar way to kasan_multi_shot
and renamed the init/exit functions to be more generic to accommodate.
- Due to [4] in kasan_strings, kasan_memchr, and
kasan_memcmp will fail if CONFIG_AMD_MEM_ENCRYPT is enabled so return
early and print message explaining this circumstance.
- Changed preprocessor checks to C checks where applicable.
Changes since v1:
- Make use of Alan Maguire's suggestion to use his patch that allows
static resources for integration instead of adding a new attribute to
the kunit struct
- All KUNIT_EXPECT_KASAN_FAIL statements are local to each test
- The definition of KUNIT_EXPECT_KASAN_FAIL is local to the
test_kasan.c file since it seems this is the only place this will
be used.
- Integration relies on KUnit being builtin
- copy_user_test has been separated into its own file since KUnit
is unable to test these. This can be run as a module just as before,
using CONFIG_TEST_KASAN_USER
- The addition to the current task has been separated into its own
patch as this is a significant enough change to be on its own.
[1] https://lore.kernel.org/linux-kselftest/CAFd5g46Uu_5TG89uOm0Dj5CMq+11cwjBns…
[2] https://lore.kernel.org/linux-mm/20200424145521.8203-1-dja@axtens.net/
[3] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?…
[4] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?…
[5] https://bugzilla.kernel.org/show_bug.cgi?id=206337
[6] https://lore.kernel.org/linux-kselftest/20200620054944.167330-1-davidgow@go…
[7] https://lkml.org/lkml/2020/7/31/571
[8] https://lore.kernel.org/linux-kselftest/8d43e88e-1356-cd63-9152-209b81b1674…
[9] https://www.spinics.net/lists/kernel/msg3660451.html
David Gow (1):
mm: kasan: Do not panic if both panic_on_warn and kasan_multishot set
Patricia Alfonso (4):
Add KUnit Struct to Current Task
KUnit: KASAN Integration
KASAN: Port KASAN Tests to KUnit
KASAN: Testing Documentation
Documentation/dev-tools/kasan.rst | 70 +++
include/kunit/test.h | 5 +
include/linux/kasan.h | 6 +
include/linux/sched.h | 4 +
lib/Kconfig.kasan | 22 +-
lib/Makefile | 4 +-
lib/kunit/test.c | 13 +-
lib/test_kasan.c | 728 ++++++++++++------------------
lib/test_kasan_module.c | 111 +++++
mm/kasan/report.c | 34 +-
10 files changed, 554 insertions(+), 443 deletions(-)
create mode 100644 lib/test_kasan_module.c
--
2.28.0.618.gf4bc123cb7-goog
v1 -> v2:
- Move check_pseudo_btf_id from check_ld_imm() to
replace_map_fd_with_map_ptr() and rename the latter.
- Add bpf_this_cpu_ptr().
- Use bpf_core_types_are_compat() in libbpf.c for checking type
compatibility.
- Rewrite typed ksym extern type in BTF with int to save space.
- Minor revision of bpf_per_cpu_ptr()'s comments.
- Avoid using long in tests that use skeleton.
- Refactored test_ksyms.c by moving kallsyms_find() to trace_helpers.c
- Fold the patches that sync include/linux/uapi and
tools/include/linux/uapi.
rfc -> v1:
- Encode VAR's btf_id for PSEUDO_BTF_ID.
- More checks in verifier. Checking the btf_id passed as
PSEUDO_BTF_ID is valid VAR, its name and type.
- Checks in libbpf on type compatibility of ksyms.
- Add bpf_per_cpu_ptr() to access kernel percpu vars. Introduced
new ARG and RET types for this helper.
This patch series extends the previously added __ksym externs with
btf support.
Right now the __ksym externs are treated as pure 64-bit scalar value.
Libbpf replaces ld_imm64 insn of __ksym by its kernel address at load
time. This patch series extend those externs with their btf info. Note
that btf support for __ksym must come with the kernel btf that has
VARs encoded to work properly. The corresponding chagnes in pahole
is available at [1] (with a fix at [2] for gcc 4.9+).
The first 3 patches in this series add support for general kernel
global variables, which include verifier checking (01/06), libpf
support (02/06) and selftests for getting typed ksym extern's kernel
address (03/06).
The next 3 patches extends that capability further by introducing
helpers bpf_per_cpu_ptr() and bpf_this_cpu_ptr(), which allows accessing
kernel percpu variables correctly (04/06 and 05/06).
The tests of this feature were performed against pahole that is extended
with [1] and [2]. For kernel BTF that does not have VARs encoded, the
selftests will be skipped.
[1] https://git.kernel.org/pub/scm/devel/pahole/pahole.git/commit/?id=f3d9054ba…
[2] https://www.spinics.net/lists/dwarves/msg00451.html
Hao Luo (6):
bpf: Introduce pseudo_btf_id
bpf/libbpf: BTF support for typed ksyms
bpf/selftests: ksyms_btf to test typed ksyms
bpf: Introduce bpf_per_cpu_ptr()
bpf: Introduce bpf_this_cpu_ptr()
bpf/selftests: Test for bpf_per_cpu_ptr() and bpf_this_cpu_ptr()
include/linux/bpf.h | 4 +
include/linux/bpf_verifier.h | 4 +
include/linux/btf.h | 26 +++
include/uapi/linux/bpf.h | 69 ++++++-
kernel/bpf/btf.c | 25 ---
kernel/bpf/verifier.c | 176 +++++++++++++++++-
kernel/trace/bpf_trace.c | 32 ++++
tools/include/uapi/linux/bpf.h | 69 ++++++-
tools/lib/bpf/libbpf.c | 116 ++++++++++--
.../testing/selftests/bpf/prog_tests/ksyms.c | 31 +--
.../selftests/bpf/prog_tests/ksyms_btf.c | 73 ++++++++
.../selftests/bpf/progs/test_ksyms_btf.c | 49 +++++
tools/testing/selftests/bpf/trace_helpers.c | 26 +++
tools/testing/selftests/bpf/trace_helpers.h | 4 +
14 files changed, 615 insertions(+), 89 deletions(-)
create mode 100644 tools/testing/selftests/bpf/prog_tests/ksyms_btf.c
create mode 100644 tools/testing/selftests/bpf/progs/test_ksyms_btf.c
--
2.28.0.526.ge36021eeef-goog