eas-dev

eas-dev@lists.linaro.org

310 discussions

by Daniel Lezcano

The current approach to select an idle state is based on the idle period statistics computation. Useless to say this approach satisfied everyone as a solution to find the best trade-off between the performances and the energy saving via the menu governor. However, the kernel is evolving to act pro-actively regarding the energy constraints with the scheduler and the different power management subsystems are not collaborating with the scheduler as the conductor of the decisions, they all act independently. In order to integrate the cpuidle framework into the scheduler, we have to radically change the approach by clearly identifying what is causing a wake up and how it behaves. The cpuidle governors are based on idle period statistics, hence without knowledge of what woke up the cpu. In these sources of wakes up, the IPI are of course accounted which results in doing statistics on the scheduler behavior too. It is no sense to let the scheduler to take a decision based on a next prediction of its own decisions. This serie inverts the logic. First there is a small statistic library do to basic and fast statistics computation, put in the library directory and make it available to everyone. It is mathematically proven there is no overflow in the code (check the log and comments). The second patch provides a callback to be registered in the irq subsystem and to be called when an interrupt is handled with a timestamp. Interrupts related to timers are discarded. The third patch uses the callback provided by the patch above to compute an average for each interrupt on each cpu. When the interrupt intervals are in standard deviation +/- mean value, then the source of wake up is considered stable and enters in the 'predictable' category. Then the next prediction wakeup for a specific cpu is the minimum remaining time of each interrupt's next prediction / or the timer. These are the results with a workload emulator (mp3, video, browser, ...) on a Dual Xeon 6 cores. Each test has been run 10 times. -------------------------- successful predictions (%) -------------------------- scripts/rt-app-browser.sh.menu.dat: N min max sum mean stddev 10 56.51 68.61 631.27 63.127 3.6882 scripts/rt-app-browser.sh.irq.dat: N min max sum mean stddev 10 72.88 79.94 774.43 77.443 2.10055 -------------------------- Successful predictions (%) -------------------------- scripts/rt-app-mp3.sh.menu.dat: N min max sum mean stddev 10 65.4 69.53 675.51 67.551 1.42503 scripts/rt-app-mp3.sh.irq.dat: N min max sum mean stddev 10 82.03 92.13 854.69 85.469 2.63553 -------------------------- Successful predictions (%) -------------------------- scripts/rt-app-video.sh.menu.dat: N min max sum mean stddev 10 57.69 77.72 625.58 62.558 5.54488 scripts/rt-app-video.sh.irq.dat: N min max sum mean stddev 10 73.19 75.2 742.33 74.233 0.752316 -------------------------- Successful predictions (%) -------------------------- scripts/video.sh.menu.dat: N min max sum mean stddev 10 40.7 59.08 463.02 46.302 5.25094 scripts/video.sh.irq.dat: N min max sum mean stddev 10 29.64 84.59 425.58 42.558 16.007 The next prediction algorithm is very simple at the moment but it opens the door for the following improvements: - Detect patterns (eg. 1, 1, 3, 1, 1, 3, ...) - Each devices behave differently, thus the prediction algorithm can be per interrupt. Eg. disk ios have a burst of fast interrupt followed by a couple of slow interrupts. If a simplistic algorithm gives better results than the menu governor, there is a high probability an optimized one will do much better. * Regarding how this integrates into the scheduler At the moment the integration is the first step, hence there is just a very small integration when the scheduler tries to find a cpu it will prevent to use an idle cpu where the idle period did not reach the energy break even. Invoking the API to enter idle is simplified on purpose to let the scheduler to take a decision between it asks when is expected the next wakeup on the cpu and when it enters idle. - sched_idle_next_wakeup() => returns a s64 telling the remaining time before a wakeup occurs - sched_idle(duration, latency) => goes idle with the specified duration and the latency constraint Daniel Lezcano (9): lib: Add a simple statistics library irq: Add a framework to measure interrupt timings sched: idle: IRQ based next prediction for idle period sched-idle: Plug sched idle with the idle task cpuidle: Add statistics and debug information with debugfs cpuidle: Store the idle start time stamp sched: fair: Fix wrong idle timestamp usage sched/fair: Prevent to break the target residency sched-idle: Add a debugfs entry to switch from cpuidle to sched-idle Nicolas Pitre (1): idle-sched: Add a trace event when an interrupt occurs drivers/cpuidle/Kconfig | 12 ++ drivers/cpuidle/Makefile | 2 + drivers/cpuidle/cpuidle.c | 16 +- drivers/cpuidle/debugfs.c | 232 +++++++++++++++++++++++ drivers/cpuidle/debugfs.h | 19 ++ include/linux/cpuidle.h | 16 ++ include/linux/interrupt.h | 45 +++++ include/linux/irqdesc.h | 3 + include/linux/stats.h | 29 +++ include/trace/events/irq.h | 44 +++++ kernel/irq/Kconfig | 3 + kernel/irq/handle.c | 12 ++ kernel/irq/manage.c | 65 ++++++- kernel/sched/Makefile | 1 + kernel/sched/fair.c | 44 +++-- kernel/sched/idle-sched.c | 449 +++++++++++++++++++++++++++++++++++++++++++++ kernel/sched/idle.c | 11 +- kernel/sched/sched.h | 20 ++ lib/Makefile | 3 +- lib/stats.c | 235 ++++++++++++++++++++++++ 20 files changed, 1239 insertions(+), 22 deletions(-) create mode 100644 drivers/cpuidle/debugfs.c create mode 100644 drivers/cpuidle/debugfs.h create mode 100644 include/linux/stats.h create mode 100644 kernel/sched/idle-sched.c create mode 100644 lib/stats.c -- 1.9.1

10 years, 10 months

PELT load-tracking problem while running Geekbench on MT8173

by Koan-Sin Tan

Hi, Geekbench is a widely used benchmark on Android and iOs devices. It's also available on Linux and Mac OS X. It somehow provides a cross-platform benchmark which we can use as an index of how powerful the tested CPUs are. I was trying to run Geekbench on MTK's internal EAS-enabled Android-based device and found Geekbench scores are lower than expected because of relative slow response time of PELT. See the attached pdf for a bit more description of a GeekBench trace analysed using TRAPpy. -- // freedom

10 years, 10 months

Regarding EAS usefulness for SMP system

by Nitish Ambastha

Dear All I am going through the EAS project work and trying to port them on my ARM based SMP system (3.10 Linux version) Could you please help me clarify, will EAS be helpful in terms of power/performance for SMP systems as well? Thanks & Regards Nitish Ambastha

10 years, 10 months

decaying of idle CPU utilization

by Steve Muckle

To summarize the current problem with idle CPU capacity votes: - When the last task on a CPU (say CPU X) sleeps and the CPU goes idle, we currently drop its capacity vote to zero. We do not immediately update the cluster frequency based on this information however. - It depends on when other CPUs in the frequency domain have an event which forces re-evaluation of the capacity votes and corresponding frequency. It could occur right away, lowering the frequency only to require raising it again immediately if CPU X is idle a very short time. Or it could be a very long time before such an event occurs which will leave the cluster at an unnecessarily high OPP and waste energy. I have a draft of a change which modifies the nohz idle balance path a bit to ensure that update_blocked_averages() is called for tickless idle CPUs at least every X ms. This alone won't solve the above problems though. You need to force re-evaluation of the capacity votes somewhere to update the cluster frequency. I was originally going to call into cpufreq_sched as idle CPU loads are decayed to update the frequency there but folks didn't seem to like this during Thursday's call. We could get rid of the clearing of the capacity vote when entering idle and use a passive update when decaying idle CPU utilizations (setting the capacity vote but not triggering a re-evaluation of cluster frequency). That would solve the problem of risking the cluster frequency dropping to fmin during a very short idle and having to be immediately ramped up again. It will not solve the issue of the cluster potentially getting stuck idle at fmax/high frequency for long periods of time and wasting energy though. There's been some discussion on this issue in the context of integration of cpuidle with cpufreq and the scheduler (see attached). Rather than force regular load decay updates via the load balancer and figure out when to force frequency re-evaluation I'm inclined to just remove the clearing of the capacity vote in dequeue_task_fair when going idle and tackle this problem within cpuidle as part of an energy aware/platform aware decision (see #2 in the attachment). A possible policy in cpuidle might look like: - If it's a short idle, don't bother removing capacity vote. - If it's a long idle and the system doesn't burn extra power in idle at elevated frequency, passively remove the capacity vote. Frequency gets adjusted if another CPU has a freq-evaluating event, like today. - If it's a long idle and the system burns extra power in idle, actively remove the capacity vote, immediately adjusting frequency if needed. A slack timer mechanism may still be desirable in cpuidle to guard against the prediction being wrong (you think it's a short idle and leave a high capacity vote in, but it ends up being a long idle). Thanks if you've read this far! Also, I hope to migrate these discussions to lkml+linux-pm. Perhaps after the next sched-freq RFC posting which will surely spawn discussions there anyway and get everyone up to speed on our current status and issues, making it a good cutover point.

10 years, 11 months

enqueue_task_fair and task_tick_fair race when setting opp

by Vikram Mulukutla

Hi Juri, Steve, It looks like if cpuX sets its own OPP level in task_tick_fair (to capacity_orig), another cpuY can override this to any value (at least via enqueue_task_fair) before cpuX's request can take effect (i.e. before the throttling timestamp is updated via the kcpufreq thread). The request from cpuX at the next tick may be throttled or the task may go to sleep and its load is decayed enough that the next request after wakeup no longer crosses the threshold and hence we lose the opportunity to go to FMAX. It seems like we need to have a mechanism where a current higher request from cpu for its own capacity should override any other cpu's lower request? Thanks, Vikram

10 years, 11 months

cpufreq_sched: reset capacity in pick_next_task_idle?

by Vikram Mulukutla

Hello, I'm using the EASv5 3.18 tree with cpufreq_sched. With the sched governor enabled I've noticed that after a migration or after a switch from a non-fair task to the idle task, the source CPU goes idle and its (possibly max) capacity request stays in place, preventing other requests from going through until that source CPU decides to wake up and take up some work. I know that there are some ongoing discussions about how to actually enforce a frequency reduction when a CPU enters idle to save power, but this seems to be a more immediate problem since the other CPU(s)' requests are also basically ignored. How about a reset_capacity call in pick_next_task_idle? Throttling is a concern I suppose, but I think the check in dequeue_task_fair is doing the same thing already, so the following would just repeat for non_fair_class->idle_task. diff --git a/kernel/sched/idle_task.c b/kernel/sched/idle_task.c index c65dac8..555c21d 100644 --- a/kernel/sched/idle_task.c +++ b/kernel/sched/idle_task.c @@ -28,6 +28,8 @@ pick_next_task_idle(struct rq *rq, struct task_struct *prev) { put_prev_task(rq, prev); + cpufreq_sched_reset_cap(cpu_of(rq)); + schedstat_inc(rq, sched_goidle); return rq->idle; } Thanks, Vikram

10 years, 11 months

delaying update of setting OPP

by Steve Muckle

In cpufreq_sched_set_cap we currently have this: /* * We only change frequency if this cpu's capacity request represents a * new max. If another cpu has requested a capacity greater than the * previous max then we rely on that cpu to hit this code path and make * the change. IOW, the cpu with the new max capacity is responsible * for setting the new capacity/frequency. * * If this cpu is not the new maximum then bail */ if (capacity_max > capacity) goto out; But this can lead to situations like (2 CPU cluster, CPUs start with cap request of 0): 1. CPU0 gets heavily loaded, requests cap = 1024 (fmax) 2. CPU1 gets lightly loaded, requests cap = 10 3. CPU0's load goes away, requests cap = 0 4. CPU1's load of 10 persists for a long time In step #3 we could've set the cluster capacity to 10/1024 but did not because the CPU we were working with at the time (CPU0) was not the CPU driving the new cluster maximum capacity request. As a result we run unnecessarily at fmax for a long time. Any reason to not set the OPP associated with the new max capacity request immediately, regardless of what CPU is driving it? thanks, Steve

10 years, 11 months

[eas-dev] CPU invariant usage tracking question

by Vikram Mulukutla

Hi Dietmar, Juri, I'm evaluating EAS RFCv5 on Qualcomm SoCs. I have a question about the CPU invariant utilization tracking in __update_entity_load_avg. I have Juri's arch-support patches for arm64 such that arch_scale_cpu_capacity(cpu) - returns the maximum capacity of the CPU from the energy model Consider a dual-cluster system that has equally capable CPUs in both clusters with each cluster on its own clock source, but with different max OPP levels for each cluster. If both the clusters are at the same OPP level, the lower-max-opp cluster would accumulate utilization_avg at a slower rate. This doesn't seem right; at the same frequency, a task should exhibit equal performance on either cluster since the CPUs are otherwise equally capable. On such a system scale_cpu should be 1024 for both clusters at least in the context of utilization tracking. What do you think? Also, please confirm my understanding with respect to traditional Big.Little - where utilization will accumulate slower on a little CPU due to the CPU invariance factor. I can understand the frequency scaling factor - a task consumes more absolute CPU cycles at a higher frequency; now it would seem that given the same frequency scaling invariance factor (say little CPU's is 500MHz/1000MHz and big CPU is 1000MHz/2000MHz), we still want to have the little CPU accumulate utilization slower because the amount of work done (in IPC terms perhaps) is less on the little CPU? Thanks, Vikram -- Sent by an employee of the Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, hosted by The Linux Foundation.

10 years, 11 months

cpufreq/cpuidle integration

by Steve Muckle

Hi Daniel, I've reviewed the draft doc you sent on cpuidle/cpufreq integration. Things have progressed a bit since the initial round of comments in the doc in April, also I thought it'd be good to open the discussion on eas-dev, so I figured I'd try and summarize the main points of the doc here and comment on them. Apologies if I mis-state anything, please correct me if necessary. Some high level thoughts: - I have yet to see a platform where race to idle is a win for power. Because of the exponential shape of the power/perf curves and other issues such as random wakeups interrupting deep sleep, it's always been better in my experience to run at as low an OPP as possible within the performance requirements of the workload. As a general policy at least. - The validation tests mentioned in the doc seemed to be focused uniformly on performance but I think power measurements must be given equal consideration and mention in validating any of these changes. My comments on each of the individual proposals: 1. Managing frequency during idle when blocked load is high. The proposal in this section was to keep the frequency unchanged when entering idle and there is significant blocked load. Blocked load has since been included in the utilization metric which determines frequency. But the policy in sched-freq on what to do when the last task on a runqueue blocks (i.e. the CPU goes idle) is still evolving. Currently the CPU's CFS capacity vote in the frequency domain is passively dropped when that CPU goes idle. It zeros out its capacity vote but does not trigger a recalculation of the frequency domain's new overall required capacity or set a new frequency. This means that the frequency will remain as-is until a different event occurs which forces re-evaluation of the CPU capacity votes in the frequency domain. I won't go into enumerating those events here, suffice it to say I don't think the current policy will work. It's possible for the CPU to stay at an elevated frequency for far too long which would have an unacceptable power impact. I'm also concerned about the way blocked load is included in the utilization metric and potentially keying off that for frequency during periods of idle. It's certainly a more power-hungry policy than what is in place today, plus there's really no way to tune it since it's part of the per-entity load tracking scheme. The interactive governor had a tunable (the slack timer) which controlled how long you could sit idle at a frequency greater than fmin. Schedtune aims to provide a per-task tunable which can boost/scale up the load value for that task calculated by PELT. So that would provide some mechanism to tweak this although it affects the task's contribution at all times rather than only when it is blocked. It also currently only is built to inflate/scale up a task's demand rather than decrease it. 2. Make the idle task in charge of changing the frequency to minimum Proposal is to have idle main loop set frequency to minimum, do idle, and then restore frequency coming out of idle. My thinking is that when we enter idle, somewhere there has to be a decision made as to whether it is worth it to change frequency. The input for that decision could include - the current frequency - the energy data of the target (power consumption at each frequency at the C-state we will be entering) - the expected duration of the idle period - the latency of changing between the two frequencies in question, as well as the latency of C-state entry and exit - performance and latency requirements for the currently blocked tasks The idle task seems to me like a reasonable place for the decision logic, which could then call some not-yet-existent API into sched-freq to ask for the frequency change. Sched class capacity votes would be retained and reinstated on idle exit. The temporary idle frequency would respect any min or max freq constraints that had been previously registered in cpufreq. 3. The expected sleep duration is less than the frequency transition latency Agreed this needs to be considered, a full implementation of the logic I mentioned in #2 would cover this one. 4. Align frequency with task priority I'd agree with MikeT's comment in the doc that changing frequency according to the niceness of the running tasks would be a pretty big change in semantics that we should stay away from. Schedtune may offer a way to negatively bias the performance of a task, although currently it only can be used to inflate a task's performance demands. 5. Consider CPU frequency when evaluating wake-up latency Agreed this should be taken into account. Again would be part of the logic I mentioned in #2 I think, deciding whether we can even change frequencies during idle at all (possibly ruled out due to QoS constraints), and then if it is possible, whether it is worth it from an energy standpoint. 6. Multiple freq governors as input to cpufreq core I'd agree this seems like it'd have value but barring a major shift in priorities, it's going to be a while before this would get focus given the effort required just to get the basic sched-freq feature merged. 7. Increase freq of little cluster when freq of big cluster increases, to make migrations back to little faster. This wouldn't be for everyone IMO due to the power impact. I'm not sure where exactly this policy would go, especially given the current crusade in the community against plugin governors and tunables. Perhaps in the platform-level cpufreq driver. thanks, Steve

10 years, 11 months

alternative name for sched-dvfs

by Steve Muckle

It was pointed out in today's technical syncup that sched-dvfs should rely on its own static key, and that static key should not indicate a dependency on EAS. The static key was named sched_energy_freq. I've renamed/shortened it to sched_freq. I also think we should try and use this name for the feature in general since I believe less people are familiar with the term/acronym DVFS. A git grep -i dvfs suggests it's almost exclusively an ARM term, at least as far as the kernel source goes, with a little usage in sound codecs and GPU. If anyone has a better name or concerns please share... thanks, Steve

10 years, 11 months

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

eas-dev