On Sun, 2022-09-11 at 12:43 +0000, David Laight wrote:
From: NeilBrown
Sent: 08 September 2022 01:58
On Thu, 08 Sep 2022, Eugeniu Rosca wrote:
Hello all,
On Mo, Apr 18, 2022 at 02:10:03 +0200, Greg Kroah-Hartman wrote:
From: NeilBrown neilb@suse.de
commit 3848e96edf4788f772d83990022fa7023a233d83 upstream.
xprt_destory() claims XPRT_LOCKED and then calls del_timer_sync(). Both xprt_unlock_connect() and xprt_release() call ->release_xprt() which drops XPRT_LOCKED and *then* xprt_schedule_autodisconnect() which calls mod_timer().
This may result in mod_timer() being called *after* del_timer_sync(). When this happens, the timer may fire long after the xprt has been freed, and run_timer_softirq() will probably crash.
The pairing of ->release_xprt() and xprt_schedule_autodisconnect() is always called under ->transport_lock. So if we take -
transport_lock to
call del_timer_sync(), we can be sure that mod_timer() will run first (if it runs at all).
Cc: stable@vger.kernel.org Signed-off-by: NeilBrown neilb@suse.de Signed-off-by: Trond Myklebust trond.myklebust@hammerspace.com Signed-off-by: Greg Kroah-Hartman gregkh@linuxfoundation.org
net/sunrpc/xprt.c | 7 +++++++ 1 file changed, 7 insertions(+)
--- a/net/sunrpc/xprt.c +++ b/net/sunrpc/xprt.c @@ -1520,7 +1520,14 @@ static void xprt_destroy(struct rpc_xprt */ wait_on_bit_lock(&xprt->state, XPRT_LOCKED, TASK_UNINTERRUPTIBLE);
+ /* + * xprt_schedule_autodisconnect() can run after XPRT_LOCKED + * is cleared. We use ->transport_lock to ensure the mod_timer() + * can only run *before* del_time_sync(), never after. + */ + spin_lock(&xprt->transport_lock); del_timer_sync(&xprt->timer); + spin_unlock(&xprt->transport_lock);
I think it is sufficient to change the to spin_{,un}lock_bh() in older kernels. The spinlock call need to match other uses of the same lock.
Every time I see this patch it looks wrong. You need something to stop the code that is calling mod_timer() running after the spin_unlock(). Now it might be that there is some other state that is already set - in which case you only need to wait for the spin_lock to be released - since it can't be obtained again (to start the timer).
So I'd expect to see: spin_lock(); if (nothing_set_earlier) xprt->destroying = 1; spin_unlock() del_timer_sync();
Looking at the code (for a change) is looks even worse.
del_timer_sync() isn't anywhere near enough. All the timer callback function does is schedule some work. So you also need to wait for the work to complete.
Changing it all to use delayed_work might reduce the problems.
Oh, any using proper mutex/locks instead of wait_on_bit_lock().
I suggest you read the code one more time, then.
Holding the bitlock XPRT_LOCKED until the transport is completely destroyed is what ensures that nothing will ever schedule xprt->timer again for that transport. This patch was needed in order to fix a minor race when xprt_unlock_connect() needs to first release XPRT_LOCKED before scheduling the xprt->timer.
...and, no. We're not going to break our aio model by replacing XPRT_LOCKED with a mutex. We optimise for speed of processing of the RPC message queue, and mutexes would break that by forcing the workqueue thread to sleep instead of just re-queuing the message until the lock is available as we do now. The price of having to use wait_on_bit_lock() in the final shutdown path is well worth it.