Hi,
I still see high (upto 30%) ksoftirqd cpu use with 4.19.101+ after these 2 back ports went in for 4.19.101 (had all 4 backports applied earlier to our tree):
commit f6783319737f28e4436a69611853a5a098cbe974 sched/fair: Fix insertion in rq->leaf_cfs_rq_list commit 5d299eabea5a251fbf66e8277704b874bbba92dc sched/fair: Add tmp_alone_branch assertion
perf shows for any given ksoftirqd, with 20k-30k processes on the system with high scheduler load: 58.88% ksoftirqd/0 [kernel.kallsyms] [k] update_blocked_averages
Can we backport these 2 also, confirmed that it fixes this behavior of ksoftirqd.
commit 039ae8bcf7a5f4476f4487e6bf816885fb3fb617 upstream commit 31bc6aeaab1d1de8959b67edbed5c7a4b3cdbe7c upstream
The second one doesn’t apply cleanly, there’s a small change for the last diff, where pelt needs renaming to task (cfg_rq_clock_pelt was cfs_rq_clock_task in 4.19.y) in that unchanged part of the diff:
@@ -7700,10 +7720,6 @@ static void update_blocked_averages(int cpu) for_each_leaf_cfs_rq(rq, cfs_rq) { struct sched_entity *se;
- /* throttled entities do not contribute to load */ - if (throttled_hierarchy(cfs_rq)) - continue; - if (update_cfs_rq_load_avg(cfs_rq_clock_pelt(cfs_rq), cfs_rq)) update_tg_load_avg(cfs_rq, 0);
I can post that patch with that rename if required.
Thanks Vishnu
On Fri, Feb 28, 2020 at 12:13:54PM -0800, Vishnu Rangayyan wrote:
Hi,
I still see high (upto 30%) ksoftirqd cpu use with 4.19.101+ after these 2 back ports went in for 4.19.101 (had all 4 backports applied earlier to our tree):
commit f6783319737f28e4436a69611853a5a098cbe974 sched/fair: Fix insertion in rq->leaf_cfs_rq_list commit 5d299eabea5a251fbf66e8277704b874bbba92dc sched/fair: Add tmp_alone_branch assertion
perf shows for any given ksoftirqd, with 20k-30k processes on the system with high scheduler load: 58.88% ksoftirqd/0 [kernel.kallsyms] [k] update_blocked_averages
Can we backport these 2 also, confirmed that it fixes this behavior of ksoftirqd.
commit 039ae8bcf7a5f4476f4487e6bf816885fb3fb617 upstream commit 31bc6aeaab1d1de8959b67edbed5c7a4b3cdbe7c upstream
Do we also need bef69dd87828 ("sched/cpufreq: Move the cfs_rq_util_change() call to cpufreq_update_util()") then?
Hi Sasha,
Not sure of this, looks relevant but I'm no expert on this code. This particular change bef69dd87828 doesn't apply cleanly, need to backport it. I'll do that now and retest on the failing setup and report back.
Best Vishnu
On 2/29/20 7:11 PM, Sasha Levin wrote:
On Fri, Feb 28, 2020 at 12:13:54PM -0800, Vishnu Rangayyan wrote:
Hi,
I still see high (upto 30%) ksoftirqd cpu use with 4.19.101+ after these 2 back ports went in for 4.19.101 (had all 4 backports applied earlier to our tree):
commit f6783319737f28e4436a69611853a5a098cbe974 sched/fair: Fix insertion in rq->leaf_cfs_rq_list commit 5d299eabea5a251fbf66e8277704b874bbba92dc sched/fair: Add tmp_alone_branch assertion
perf shows for any given ksoftirqd, with 20k-30k processes on the system with high scheduler load: 58.88% ksoftirqd/0 [kernel.kallsyms] [k] update_blocked_averages
Can we backport these 2 also, confirmed that it fixes this behavior of ksoftirqd.
commit 039ae8bcf7a5f4476f4487e6bf816885fb3fb617 upstream commit 31bc6aeaab1d1de8959b67edbed5c7a4b3cdbe7c upstream
Do we also need bef69dd87828 ("sched/cpufreq: Move the cfs_rq_util_change() call to cpufreq_update_util()") then?
On Tue, 3 Mar 2020 at 00:13, Vishnu Rangayyan vishnu.rangayyan@apple.com wrote:
Hi Sasha,
Not sure of this, looks relevant but I'm no expert on this code. This particular change bef69dd87828 doesn't apply cleanly, need to backport it. I'll do that now and retest on the failing setup and report back.
Best Vishnu
On 2/29/20 7:11 PM, Sasha Levin wrote:
On Fri, Feb 28, 2020 at 12:13:54PM -0800, Vishnu Rangayyan wrote:
Hi,
I still see high (upto 30%) ksoftirqd cpu use with 4.19.101+ after these 2 back ports went in for 4.19.101 (had all 4 backports applied earlier to our tree):
commit f6783319737f28e4436a69611853a5a098cbe974 sched/fair: Fix insertion in rq->leaf_cfs_rq_list commit 5d299eabea5a251fbf66e8277704b874bbba92dc sched/fair: Add tmp_alone_branch assertion
perf shows for any given ksoftirqd, with 20k-30k processes on the system with high scheduler load: 58.88% ksoftirqd/0 [kernel.kallsyms] [k] update_blocked_averages
Can we backport these 2 also, confirmed that it fixes this behavior of ksoftirqd.
commit 039ae8bcf7a5f4476f4487e6bf816885fb3fb617 upstream commit 31bc6aeaab1d1de8959b67edbed5c7a4b3cdbe7c upstream
Do we also need bef69dd87828 ("sched/cpufreq: Move the cfs_rq_util_change() call to cpufreq_update_util()") then?
This patch is not related to the 2 patches above. It removes some spurious call to cfs_rq_util_change() and some wrong ordering but this is a fixed overhead which is not impacted by the number of cgroups unlike the 2 patches above. This patch will not help with the high load of update_blocked_averages
On Tue, Mar 03, 2020 at 09:03:15AM +0100, Vincent Guittot wrote:
On Tue, 3 Mar 2020 at 00:13, Vishnu Rangayyan vishnu.rangayyan@apple.com wrote:
Hi Sasha,
Not sure of this, looks relevant but I'm no expert on this code. This particular change bef69dd87828 doesn't apply cleanly, need to backport it. I'll do that now and retest on the failing setup and report back.
Best Vishnu
On 2/29/20 7:11 PM, Sasha Levin wrote:
On Fri, Feb 28, 2020 at 12:13:54PM -0800, Vishnu Rangayyan wrote:
Hi,
I still see high (upto 30%) ksoftirqd cpu use with 4.19.101+ after these 2 back ports went in for 4.19.101 (had all 4 backports applied earlier to our tree):
commit f6783319737f28e4436a69611853a5a098cbe974 sched/fair: Fix insertion in rq->leaf_cfs_rq_list commit 5d299eabea5a251fbf66e8277704b874bbba92dc sched/fair: Add tmp_alone_branch assertion
perf shows for any given ksoftirqd, with 20k-30k processes on the system with high scheduler load: 58.88% ksoftirqd/0 [kernel.kallsyms] [k] update_blocked_averages
Can we backport these 2 also, confirmed that it fixes this behavior of ksoftirqd.
commit 039ae8bcf7a5f4476f4487e6bf816885fb3fb617 upstream commit 31bc6aeaab1d1de8959b67edbed5c7a4b3cdbe7c upstream
Do we also need bef69dd87828 ("sched/cpufreq: Move the cfs_rq_util_change() call to cpufreq_update_util()") then?
This patch is not related to the 2 patches above. It removes some spurious call to cfs_rq_util_change() and some wrong ordering but this is a fixed overhead which is not impacted by the number of cgroups unlike the 2 patches above. This patch will not help with the high load of update_blocked_averages
Thanks for letting us know, the other two patches are now queued up.
greg k-h
linux-stable-mirror@lists.linaro.org
-
gregkh＠linuxfoundation.org -
Sasha Levin -
Vincent Guittot -
Vishnu Rangayyan