This test fails routinely in our prod testing environment, and I can reproduce it locally as well.
The test allocates dcache inside a cgroup, then drops the memory limit and checks that usage drops correspondingly. The reason it fails is because dentries are freed with an RCU delay - a debugging sleep shows that usage drops as expected shortly after.
Insert a 1s sleep after dropping the limit. This should be good enough, assuming that machines running those tests are otherwise not very busy.
Signed-off-by: Johannes Weiner hannes@cmpxchg.org --- tools/testing/selftests/cgroup/test_kmem.c | 4 ++++ 1 file changed, 4 insertions(+)
diff --git a/tools/testing/selftests/cgroup/test_kmem.c b/tools/testing/selftests/cgroup/test_kmem.c index 258ddc565deb..1b2cec9d18a4 100644 --- a/tools/testing/selftests/cgroup/test_kmem.c +++ b/tools/testing/selftests/cgroup/test_kmem.c @@ -70,6 +70,10 @@ static int test_kmem_basic(const char *root) goto cleanup;
cg_write(cg, "memory.high", "1M"); + + /* wait for RCU freeing */ + sleep(1); + slab1 = cg_read_key_long(cg, "memory.stat", "slab "); if (slab1 <= 0) goto cleanup;
On Tue, Aug 01, 2023 at 09:56:32AM -0400, Johannes Weiner wrote:
This test fails routinely in our prod testing environment, and I can reproduce it locally as well.
The test allocates dcache inside a cgroup, then drops the memory limit and checks that usage drops correspondingly. The reason it fails is because dentries are freed with an RCU delay - a debugging sleep shows that usage drops as expected shortly after.
Insert a 1s sleep after dropping the limit. This should be good enough, assuming that machines running those tests are otherwise not very busy.
Signed-off-by: Johannes Weiner hannes@cmpxchg.org
I am putting together something more formal, but this will certainly improve things, as Johannes says, assuming the system goes mostly idle during that one-second wait. So:
Acked-by: Paul E. McKenney paulmck@kernel.org
Yes, there are corner cases, such as the system having millions of RCU callbacks queued and being unable to invoke them all during that one-second interval. But that is a corner case, and that is exactly why I will be putting together something more formal. ;-)
Thanx, Paul
tools/testing/selftests/cgroup/test_kmem.c | 4 ++++ 1 file changed, 4 insertions(+)
diff --git a/tools/testing/selftests/cgroup/test_kmem.c b/tools/testing/selftests/cgroup/test_kmem.c index 258ddc565deb..1b2cec9d18a4 100644 --- a/tools/testing/selftests/cgroup/test_kmem.c +++ b/tools/testing/selftests/cgroup/test_kmem.c @@ -70,6 +70,10 @@ static int test_kmem_basic(const char *root) goto cleanup; cg_write(cg, "memory.high", "1M");
- /* wait for RCU freeing */
- sleep(1);
- slab1 = cg_read_key_long(cg, "memory.stat", "slab "); if (slab1 <= 0) goto cleanup;
-- 2.41.0
On Tue, Aug 01, 2023 at 09:39:28AM -0700, Paul E. McKenney wrote:
On Tue, Aug 01, 2023 at 09:56:32AM -0400, Johannes Weiner wrote:
This test fails routinely in our prod testing environment, and I can reproduce it locally as well.
The test allocates dcache inside a cgroup, then drops the memory limit and checks that usage drops correspondingly. The reason it fails is because dentries are freed with an RCU delay - a debugging sleep shows that usage drops as expected shortly after.
Insert a 1s sleep after dropping the limit. This should be good enough, assuming that machines running those tests are otherwise not very busy.
Signed-off-by: Johannes Weiner hannes@cmpxchg.org
I am putting together something more formal, but this will certainly improve things, as Johannes says, assuming the system goes mostly idle during that one-second wait. So:
Acked-by: Paul E. McKenney paulmck@kernel.org
Yes, there are corner cases, such as the system having millions of RCU callbacks queued and being unable to invoke them all during that one-second interval. But that is a corner case, and that is exactly why I will be putting together something more formal. ;-)
Thanx, Paul
tools/testing/selftests/cgroup/test_kmem.c | 4 ++++ 1 file changed, 4 insertions(+)
diff --git a/tools/testing/selftests/cgroup/test_kmem.c b/tools/testing/selftests/cgroup/test_kmem.c index 258ddc565deb..1b2cec9d18a4 100644 --- a/tools/testing/selftests/cgroup/test_kmem.c +++ b/tools/testing/selftests/cgroup/test_kmem.c @@ -70,6 +70,10 @@ static int test_kmem_basic(const char *root) goto cleanup; cg_write(cg, "memory.high", "1M");
- /* wait for RCU freeing */
- sleep(1);
- slab1 = cg_read_key_long(cg, "memory.stat", "slab "); if (slab1 <= 0) goto cleanup;
-- 2.41.0
The same issue exists in the test case test_kmem_memcg_deletion. I wouldn't mind posting the patch, but it seems you want to propose something more formal. Let me know your opinion.
Thanks, Lucas
On Thu, Aug 03, 2023 at 12:13:26PM -0400, Lucas Karpinski wrote:
On Tue, Aug 01, 2023 at 09:39:28AM -0700, Paul E. McKenney wrote:
On Tue, Aug 01, 2023 at 09:56:32AM -0400, Johannes Weiner wrote:
This test fails routinely in our prod testing environment, and I can reproduce it locally as well.
The test allocates dcache inside a cgroup, then drops the memory limit and checks that usage drops correspondingly. The reason it fails is because dentries are freed with an RCU delay - a debugging sleep shows that usage drops as expected shortly after.
Insert a 1s sleep after dropping the limit. This should be good enough, assuming that machines running those tests are otherwise not very busy.
Signed-off-by: Johannes Weiner hannes@cmpxchg.org
I am putting together something more formal, but this will certainly improve things, as Johannes says, assuming the system goes mostly idle during that one-second wait. So:
Acked-by: Paul E. McKenney paulmck@kernel.org
Yes, there are corner cases, such as the system having millions of RCU callbacks queued and being unable to invoke them all during that one-second interval. But that is a corner case, and that is exactly why I will be putting together something more formal. ;-)
Thanx, Paul
tools/testing/selftests/cgroup/test_kmem.c | 4 ++++ 1 file changed, 4 insertions(+)
diff --git a/tools/testing/selftests/cgroup/test_kmem.c b/tools/testing/selftests/cgroup/test_kmem.c index 258ddc565deb..1b2cec9d18a4 100644 --- a/tools/testing/selftests/cgroup/test_kmem.c +++ b/tools/testing/selftests/cgroup/test_kmem.c @@ -70,6 +70,10 @@ static int test_kmem_basic(const char *root) goto cleanup; cg_write(cg, "memory.high", "1M");
- /* wait for RCU freeing */
- sleep(1);
- slab1 = cg_read_key_long(cg, "memory.stat", "slab "); if (slab1 <= 0) goto cleanup;
-- 2.41.0
The same issue exists in the test case test_kmem_memcg_deletion. I wouldn't mind posting the patch, but it seems you want to propose something more formal. Let me know your opinion.
I am proposing a /sys/module/rcutree/parameters/do_rcu_barrier file. Writing a "1" into this file results in an rcu_barrier() in the kernel, but set up so that there is no more than a single rcu_barrier() call per second.
So you could do the following:
run-a-test echo 1 > /sys/module/rcutree/parameters/do_rcu_barrier # As root # All RCU callbacks from run-a-test have now been invoked run-another-test
Please note that this handles only RCU, as in call_rcu(), and not SRCU, Tasks RCU, and so on.
Thanx, Paul
linux-kselftest-mirror@lists.linaro.org
-
Johannes Weiner -
Lucas Karpinski -
Paul E. McKenney