Apparently despite it being marked inline, the compiler may not inline __down_read_common() which makes it difficult to identify the cause of lock contention, as the blocked function in traceevents will always be listed as __down_read_common().
So this patch adds __always_inline annotation to the common function (as well as the inlined helper callers) to force it to be inlined so the blocking function will be listed (via Wchan) in traceevents.
Cc: Minchan Kim minchan@kernel.org Cc: Tim Murray timmurray@google.com Cc: Peter Zijlstra peterz@infradead.org Cc: Ingo Molnar mingo@redhat.com Cc: Will Deacon will@kernel.org Cc: Waiman Long longman@redhat.com Cc: Boqun Feng boqun.feng@gmail.com Cc: kernel-team@android.com Cc: stable@vger.kernel.org Fixes: c995e638ccbb ("locking/rwsem: Fold __down_{read,write}*()") Reviewed-by: Waiman Long longman@redhat.com Reported-by: Tim Murray timmurray@google.com Signed-off-by: John Stultz jstultz@google.com --- v2: Reworked to use __always_inline instead of __sched as suggested by Waiman Long
v3: Add __always_inline annotations to currently inlined users of __down_read_common() to avoid the compiler later doing the same thing there. (Suggested by Peter). --- kernel/locking/rwsem.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/kernel/locking/rwsem.c b/kernel/locking/rwsem.c index acb5a50309a1..9eabd585ce7a 100644 --- a/kernel/locking/rwsem.c +++ b/kernel/locking/rwsem.c @@ -1240,7 +1240,7 @@ static struct rw_semaphore *rwsem_downgrade_wake(struct rw_semaphore *sem) /* * lock for reading */ -static inline int __down_read_common(struct rw_semaphore *sem, int state) +static __always_inline int __down_read_common(struct rw_semaphore *sem, int state) { int ret = 0; long count; @@ -1258,17 +1258,17 @@ static inline int __down_read_common(struct rw_semaphore *sem, int state) return ret; }
-static inline void __down_read(struct rw_semaphore *sem) +static __always_inline void __down_read(struct rw_semaphore *sem) { __down_read_common(sem, TASK_UNINTERRUPTIBLE); }
-static inline int __down_read_interruptible(struct rw_semaphore *sem) +static __always_inline int __down_read_interruptible(struct rw_semaphore *sem) { return __down_read_common(sem, TASK_INTERRUPTIBLE); }
-static inline int __down_read_killable(struct rw_semaphore *sem) +static __always_inline int __down_read_killable(struct rw_semaphore *sem) { return __down_read_common(sem, TASK_KILLABLE); }