On Sun, Jan 1, 2023 at 2:16 PM Joel Fernandes (Google) joel@joelfernandes.org wrote:
During shutdown of rcutorture, the shutdown thread in rcu_torture_cleanup() calls torture_cleanup_begin() which sets fullstop to FULLSTOP_RMMOD. This is enough to cause the rcutorture threads for readers and fakewriters to breakout of their main while loop and start shutting down.
Once out of their main loop, they then call torture_kthread_stopping() which in turn waits for kthread_stop() to be called, however rcu_torture_cleanup() has not even called kthread_stop() on those threads yet, it does that a bit later. However, before it gets a chance to do so, torture_kthread_stopping() calls schedule_timeout_interruptible(1) in a tight loop. Tracing confirmed this makes the timer softirq constantly execute timer callbacks, while never returning back to the softirq exit path and is essentially "locked up" because of that. If the softirq preempts the shutdown thread, kthread_stop() may never be called.
This commit improves the situation dramatically, by increasing timeout passed to schedule_timeout_interruptible() 1/20th of a second. This causes the timer softirq to not lock up a CPU and everything works fine. Testing has shown 100 runs of TREE07 passing reliably, which was not the case before because of RCU stalls.
On my Dell PowerEdge R720 with two Intel(R) Xeon(R) CPU E5-2660 128G memory: 1) before this patch: 3 of 80 rounds failed with "rcu: INFO: rcu_sched detected stalls on CPUs/tasks" [1] 2) after this patch all 80 rounds passed
Tested-by: Zhouyi Zhou zhouzhouyi@gmail.com
Thanks Zhouyi
[1] http://154.220.3.115/logs/20230101/console.log
Cc: Paul McKenney paulmck@kernel.org Cc: Frederic Weisbecker fweisbec@gmail.com Cc: Zhouyi Zhou zhouzhouyi@gmail.com Cc: stable@vger.kernel.org # 6.0.x Signed-off-by: Joel Fernandes (Google) joel@joelfernandes.org
kernel/torture.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/kernel/torture.c b/kernel/torture.c index 29afc62f2bfe..d024f3b7181f 100644 --- a/kernel/torture.c +++ b/kernel/torture.c @@ -915,7 +915,7 @@ void torture_kthread_stopping(char *title) VERBOSE_TOROUT_STRING(buf); while (!kthread_should_stop()) { torture_shutdown_absorb(title);
schedule_timeout_uninterruptible(1);
schedule_timeout_uninterruptible(HZ/20); }} EXPORT_SYMBOL_GPL(torture_kthread_stopping); -- 2.39.0.314.g84b9a713c41-goog