Hi,
I tried an experiment this weekend - basically I have RT threads bound to big CPUs running a fixed period load, with hack bench running with all CPUs allowed. The system is a Pixel2 ARM big.LITTLE 8-core (4x4).
Basically, I changed capacity_orig_of to capacity_of in capacity_spare_wake and wake_cap and I see a good performance improvement. That makes sense because wake_cap would send the task wake up to the slow-path if RT capacity was eating into the CFS capacity on prev/current CPU, and capacity_spare_wake would find a better group with spare-capacity deducted by the RT pressure capacity.
One of the concerns for such a change to wake_cap, that I had, was that it might affect upstream cases that may still want to do a select_idle_sibling even if the capacity on the previous/waker's CPU was not enough after deducting RT pressure. In that case, the wake_cap change to use capacity_of would cause it to enter the slow-path for those cases I think.
Could you let me know your thoughts about such a change? I heard that capacity_of was attempted before and there might be some cases to consider. Anything from your previous experiences with this change that you could share? Atleast for capacity_spare_wake, the improvements seems to be worthwhile and dramatic in some cases. I also have some more changes I am thinking off to find_idlest_group but I wanted to start a discussion on the spare capacity idea first.
This is related to Rohit's work on RT Capacity awareness, I was talking to him and we were discussing ideas on the implementation.
thanks,
- Joel
On 19/10/17 21:43, Joel Fernandes wrote:
Hi,
I tried an experiment this weekend - basically I have RT threads bound to big CPUs running a fixed period load, with hack bench running with all CPUs allowed. The system is a Pixel2 ARM big.LITTLE 8-core (4x4).
Basically, I changed capacity_orig_of to capacity_of in capacity_spare_wake and wake_cap and I see a good performance improvement. That makes sense because wake_cap would send the task wake up to the slow-path if RT capacity was eating into the CFS capacity on prev/current CPU, and capacity_spare_wake would find a better group with spare-capacity deducted by the RT pressure capacity.
Just so I can follow. The change in capacity_spare_wake() is now part of [PATCH 1/3] sched/fair: Introduce scaled capacity awareness in find_idlest_cpu code path ?
https://patchwork.kernel.org/patch/10002411
One of the concerns for such a change to wake_cap, that I had, was that it might affect upstream cases that may still want to do a select_idle_sibling even if the capacity on the previous/waker's CPU was not enough after deducting RT pressure. In that case, the wake_cap change to use capacity_of would cause it to enter the slow-path for those cases I think.
So then you guys could switch to capacity_of(cpu) only in capacity_spare_wake(). You mentioned in the next paragraph that this change is especially worthwhile.
At the end, wake_cap() and capacity_spare_wake() have different roles and IMHO the benefit of taking rt-pressure into consideration is easier to see with the latter one.
Could you let me know your thoughts about such a change? I heard that capacity_of was attempted before and there might be some cases to consider. Anything from your previous experiences with this change that you could share? Atleast for capacity_spare_wake, the improvements seems to be worthwhile and dramatic in some cases. I also have some more changes I am thinking off to find_idlest_group but I wanted to start a discussion on the spare capacity idea first.
I'm not aware of any use cases which might suffer from using capacity_of(cpu) in both functions. It's just that it is much harder to understand the entire range of behaviours once we switch from original to dynamic capacity. The synthetic testcase you've chosen clearly shows advantage here.
This is related to Rohit's work on RT Capacity awareness, I was talking to him and we were discussing ideas on the implementation.
Unfortunately, his v5 doesn't consider the latest changes in the slow path yet but I see how your test relates to his ideas.
- Dietmar
[...]
Hi Dietmar,
On Wed, Oct 25, 2017 at 3:04 AM, Dietmar Eggemann dietmar.eggemann@arm.com wrote:
On 19/10/17 21:43, Joel Fernandes wrote:
Hi,
I tried an experiment this weekend - basically I have RT threads bound to big CPUs running a fixed period load, with hack bench running with all CPUs allowed. The system is a Pixel2 ARM big.LITTLE 8-core (4x4).
Basically, I changed capacity_orig_of to capacity_of in capacity_spare_wake and wake_cap and I see a good performance improvement. That makes sense because wake_cap would send the task wake up to the slow-path if RT capacity was eating into the CFS capacity on prev/current CPU, and capacity_spare_wake would find a better group with spare-capacity deducted by the RT pressure capacity.
Just so I can follow. The change in capacity_spare_wake() is now part of [PATCH 1/3] sched/fair: Introduce scaled capacity awareness in find_idlest_cpu code path ?
Yes, but the above patch which touch the slow path is being abandoned and will be superseded by mine. In mine I only change capacity_spare_wake and one of other thing in find_idlest_group so its a lot simpler. I will post this out by this week - I have held back until now since its being tested more - but I will have it out on eas-dev soon.
Me and Rohit have been working on validating just the effect of capacity_of in capacity_spare_wake and it is looking good in testing.
One of the concerns for such a change to wake_cap, that I had, was that it might affect upstream cases that may still want to do a select_idle_sibling even if the capacity on the previous/waker's CPU was not enough after deducting RT pressure. In that case, the wake_cap change to use capacity_of would cause it to enter the slow-path for those cases I think.
So then you guys could switch to capacity_of(cpu) only in capacity_spare_wake(). You mentioned in the next paragraph that this change is especially worthwhile.
Yes, that's right.
At the end, wake_cap() and capacity_spare_wake() have different roles and IMHO the benefit of taking rt-pressure into consideration is easier to see with the latter one.
Yes, that's my feeling too.
Could you let me know your thoughts about such a change? I heard that capacity_of was attempted before and there might be some cases to consider. Anything from your previous experiences with this change that you could share? Atleast for capacity_spare_wake, the improvements seems to be worthwhile and dramatic in some cases. I also have some more changes I am thinking off to find_idlest_group but I wanted to start a discussion on the spare capacity idea first.
I'm not aware of any use cases which might suffer from using capacity_of(cpu) in both functions. It's just that it is much harder to understand the entire range of behaviours once we switch from original to dynamic capacity. The synthetic testcase you've chosen clearly shows advantage here.
True. Currently both my synthetic test and Rohit's tests with ping + hackbench on large number of CPU servers are showing good results with capacity_spare_wake changes. I will summarize the results and post them to eas-dev soon once all the testing data is back.
This is related to Rohit's work on RT Capacity awareness, I was talking to him and we were discussing ideas on the implementation.
Unfortunately, his v5 doesn't consider the latest changes in the slow path yet but I see how your test relates to his ideas.
Yes. His next version which is planned to be posted after mine will only be affecting the fast-path and should result in further improvement. We decided to take it one step at a time so its easier to get these in upstream.
Regards, - Joel
-
Dietmar Eggemann -
Joel Fernandes