eas-dev

eas-dev@lists.linaro.org

310 discussions

by Leo Yan

Hi all, Below are raw power data on Hikey board; with this power data i'd like to create the power model for Hikey. - Measure method: On Hikey board, we cannot measure buck1 which is dedicated for AP subsystem; so turned to measure VDD_4V2 to remove R247 and remount shunt resistor 470mOhm. At a result, the power data includes many other LDO's power data. +--------------+ +-------------+ 4.2v | | Buck1 | | ---- Shunt Resistor --->| PMIC: Hi6553 |------>| SoC: Hi6220 | ^ ^ | | | ACPU | | | +--------------+ +-------------+ |-> Energy Probe <-| - Measured raw data: sys_suspend: AP system suspend state cluster_off: two clusters are powered off cluster_on: 1 cluster is powered on, all cpus are powered off cpu_wfi: 1 cluster is powered on, last cpu enters 'wfi' but other cpus are powered off voltage: voltage for every OPP # OPP sys_suspend cluster_off cluster_on cpu_wfi cpu_on voltage 208 328 347 366 374 435 1.04 432 328 344 374 388 499 1.04 729 331 351 400 409 606 1.09 960 329 353 430 443 750 1.18 1200 331 365 486 506 988 1.33 Hikey Power Model Power [mW] 500 ++--------+-------------+---------+---------++ cluster_off **A*** + + + + D cluster_on ##B### 450 ++.........................................%%+ cpu_wfi $$C$$$ 400 ++.......................................%%.++ cpu p-state %%D%%% | : : : %% | 350 ++...................................%%.....++ | : : :%% | 300 ++...............................%D.........++ 250 ++...........................%%%%...........++ | : : %%% : | 200 ++....................%%D%..................++ | : %%%%% : : | 150 ++...........%%%%...........................++ | %%D%% : : #####B 100 ++.%%%%%.....................#####B#####....++ 50 D%%..............#######B####...............++ A*********A*************A*********A**********A 0 C$$$$$$$$$C$$$$$$$$$$$$$C$$$$$$$$$C$$$$$----++ 208 432 729 960 1200 Frequency [MHz] Hikey Power Efficiency Power Efficiency [mW/MHz] Voltage [v] 0.45 ++-------+----------+--------+-------++ 3 Cluster Power **A*** + + + + + CPU Static Power ##B### 0.4 ++...................................$C CPU Dynamic power $$C$$$ | : : : $$$++ 2.5 Voltage %%D%%% 0.35 ++.............................$$$...++ | : : $$C$ | 0.3 C$$$$$$$$..............$$$$..........++ 2 | C$$$$$$$$$$C$$ : | 0.25 ++...................................++ 1.5 0.2 ++................................%%%%D | : : %%%%D%%%% | 0.15 D%%%%%%%%D%%%%%%%%%%D%%%%............++ 1 | : : : | 0.1 A********.........................****A | A**********A********A**** ++ 0.5 0.05 ++...................................++ B########B##########+ ####B########B 0 ++-------+----------B####----+-------++ 0 208 432 729 960 1200 Frequency [MHz] - Power Model On Hikey: According to before we have discussed for power model, i think below is the prefered power data for Hikey which calculated from raw power date: static struct idle_state idle_states_cluster_a53[] = { { .power = 0 }, { .power = 0 }, }; /* * Use (cluster_on - cluster_off) for every OPP */ static struct capacity_state cap_states_cluster_a53[] = { /* Power per cluster */ { .cap = 178, .power = 19, }, { .cap = 369, .power = 30, }, { .cap = 622, .power = 49, }, { .cap = 819, .power = 77, }, { .cap = 1024, .power = 121, }, }; /* * Use (cpu_wfi - cluster_on) for every OPP, then calculate the * average value for wfi's power data; But we can see actually * the idle state of "WFI" will be impacted by voltage. */ static struct idle_state idle_states_core_a53[] = { { .power = 12 }, { .power = 0 }, }; /* * Use (cpu_on - cluster_off) for every OPP */ static struct capacity_state cap_states_core_a53[] = { /* Power per cpu */ { .cap = 178, .power = 69, }, /* 208MHz */ { .cap = 369, .power = 125, }, /* 432MHz */ { .cap = 622, .power = 206, }, /* 729MHz */ { .cap = 819, .power = 320, }, /* 960MHz */ { .cap = 1024, .power = 502, }, /* 1.2GHz */ }; If have any questions or issues for upper energy model data, please let me know; Appreciate review and comments in advance. - Some other questions Q1: Jian & Dan, voltage for 1.2GHz is quite high, could you help check the voltage table for OPPs, if there have any unexpected value? Q2: Morten, if i want to do more profiling on EAS, do you suggest i should refer which branch now? I think now EASv5 patches are relative old, so want to check if we have better candidate or not. i downloaded the git repo: git://www.linux-arm.org/linux-power.git, branch energy_model_rfc_v5.1; but there have no sched-dvfs related patches. Thanks, Leo Yan

10 years, 11 months

jank measurement in workload automation suite

by Steve Muckle

I see there is a jank measurement capability in workload automation: http://pythonhosted.org/wlauto/_modules/wlauto/instrumentation/fps.html Has anyone used this? Limitations, prerequisites etc? I'd like to use this to measure janks during scrolls. I'll give it a shot but I've never used WA before so I thought I'd check first if anyone had comments/experiences to share. thanks, Steve

10 years, 11 months

Re: [Eas-dev] cpufreq_sched policy for combining requests from multiple sched classes

by Juri Lelli

[+eas-dev] On 19/10/15 09:06, Vincent Guittot wrote: > On 8 October 2015 at 12:28, Morten Rasmussen <morten.rasmussen(a)arm.com> wrote: >> On Thu, Oct 08, 2015 at 10:54:15AM +0200, Vincent Guittot wrote: >>> On 8 October 2015 at 02:59, Steve Muckle <steve.muckle(a)linaro.org> wrote: >>>> At Linaro Connect a couple weeks back there was some sentiment that >>>> taking the max of multiple capacity requests would be the only viable >>>> policy when cpufreq_sched is extended to support multiple sched classes. >>>> But I'm concerned that this is not workable - if CFS is requesting 1GHz >>>> worth of bandwidth on a 2GHz CPU, and DEADLINE is also requesting 1GHz >>>> of bandwidth, we would run the CPU at 1GHz and starve the CFS tasks >>>> indefinitely. >>>> >>>> I'd think there has to be a summing of bandwidth requests from scheduler >>>> class clients. MikeT raised the concern that in such schemes you often >>>> end up with a bunch of extra overhead because everyone adds their own >>>> fudge factor (Mike please correct me if I'm misstating our concern >>>> here). We should be able to control this in the scheduler classes though >>>> and ensure headroom is only added after the requests are combined. >>>> >>>> Thoughts? >>> >>> I have always been in favor of using summing instead of maximum >>> because of the example you mentioned above. IIRC, we also said that >>> the scheduler classes should not request more than needed capacity >>> with regards of schedtune knob position it means that if schedtune is >>> set to max power save, no margin should be taken any scheduler class >>> other than to filter uncertainties in cpu util computation). >>> Regarding RT, it's a bit less straight foward as we much ensure an >>> unknown responsiveness constraint (unlike deadline) so we could easily >>> request the max capacity to be sure to ensure this unknown constraint. >> >> Agreed. I'm in favor with summing the requests, but with a minor twist. >> As Steve points out, and I have discussed it with Juri as well, with >> three sched classes and using the max capacity request we would always >> request too little capacity if more than one class has tasks. Worst case >> we would only request a third of the required capacity. Deadline would >> take it all and leave nothing for RT and CFS. >> >> Summing the requests instead should be fine, but deadline might cause >> us to reserve too much capacity if we have short deadline tasks with a >> tight deadline. For example, a 2ms task (@max capacity) with a 4ms >> deadline and a 10ms period. In this case deadline would have to request >> 50% capacity (at least) to meet its deadline but it only uses 20% >> capacity (scale-invariant). Since deadline has higher priority than RT >> and CFS we can safely assume that they can use the remaining 30% without >> harming the deadline task. We can take this into account if we let >> deadline provide a utilization request (20%) and a minimum capacity >> request (50%). We would sum the utilization request with the utilization >> requests of RT and CFS. If sum < deadline_min_capacity, we would choose >> deadline_min_capacity instead of the sum to determine the capacity. What >> do you think? It might not be worth the trouble as there are plenty of >> other scenarios where we would request too much capacity for deadline >> tasks that can't be fixed. > > I have some concern with a deadline min capacity field. If we take the > example above, the request seems to be a bit too much static, deadline We should be able to get away from having a special "min capacity" field for SCHED_DEADLINE, yes. Requests coming from SCHED_DEADLINE should always concern minimum requirements (too meet deadlines). However, I'll have to play a bit with all this before being 100% sure. > scheduler should request 50% only when the task is running. The 50% > make only sense if the task start ro run at t he beg of the period and > inorder to run the complete 4ms time slot of the deadline. But, the > request might even have to be increased to 100% if for some reasons > like another deadline task or a irq or a disable preemption, the task > starts to run in the last 2ms of the deadline time slot. Once the task > has finished its running period, the request should go back to 0. Capacity requests of SCHED_DEADLINE are supposed to be more stable, I think it's built in how the thing works. There is a particular instant of time, relative to each period, called "0-lag" point after which, if the task is not running, we are sure (by construction) that we can safely release a capacity request relative to a certain task. It is about theory behind SCHED_DEADLINE implementation, but we should be able to use this information to ask for the right capacity. As said above, I'll need more time to think this through and experiment with it. > I agree that reality is a bit more complex because we don't have > "immediate" change of the freq/capacity so we must take into account > the time needed to change the capacity of the CPU but we should try to > make the requirement as close as possible to the reality. > > Using a min value just mean that we are not able to evaluate the > current capacity requirement of the deadline class and that we will > just steal the capacity requested by other class which is not a good > solution IMHO > > Juri, what will be the granularity of the computation of the bandwidth > of the patches you are going to send ? > I'm not sure I get what you mean by granularity here. The patches will add a 0..100% bandwidth number, that we'll have to normalize to 0..1024, for SCHED_DEADLINE tasks currently active. Were you expecting something else? Thanks, - Juri >> >> As Vincent points out, RT is a bit tricky. AFAIK, it doesn't have any >> utilization tracking at all. I think we have to fix that somehow. >> Regarding responsiveness, RT doesn't provide any guarantees by design >> (in the same way deadline does) so we shouldn't be violating any >> policies by slowing RT tasks down. The users might not be happy though >> so we could favor performance for RT tasks to avoid breaking legacy >> software and ask users that care about energy to migrate to deadline >> where we actually know the performance constraints of the tasks. >

10 years, 11 months

sched-dvfs response

by Steve Muckle

(changing subject, +eas-dev) On 10/19/2015 12:34 AM, Vincent Guittot wrote: >> FWIW as I tested with the browser-short rt-app workload I noticed the >> > rt-app calibration causes the CPU frequency to oscillate between fmin >> > and fmax. > > It's a normal behavior. During calibration step, we try to force the > governor to use the max OPP to evaluate the minimum ns per loop. We > have 2 sequence: The 1st one just loop on the calibration loop until > the ns per loop value reach a stable value. The second one alternate > run and sleep phase to prevent the trig of thermal mitigation which > can be triggered during 1 sequence Understood that rt-app would want to hit fmax for calibration, but the oscillation directly between fmin and fmax, and the timing of the transitions, seem concerning. I've copied a trace of this to http://smuckle.net/calibrate.txt . As an example at ~41.808: - rt-app starts executing and executes continuously at fmin for about 85ms. That's a long time IMO to underserve a continuously running workload before changing frequency. - The frequency then goes directly to fmax and rt-app runs for 3ms more. We should be going to an intermediate frequency first. This has come up on lkml and I think everyone wants that change but I'm including it here for completeness. - The system then sits idle at fmax for almost 200ms since nothing is decaying the usage on the idle CPU. This also came up on lkml though it's probably worth mentioning that it's so easy to reproduce with rt-app. Just curious, is rt-app doing a fixed amount of work in these bursts of execution? Or is it watching cpufreq nodes so that it knows when the CPU frequency has hit fmax, so it can then do calibration work?

11 years

sched-dvfs response

by Steve Muckle

11 years

Re: [Eas-dev] cpufreq_sched policy for combining requests from multiple sched classes

by Steve Muckle

(adding eas-dev) On 10/09/2015 01:41 AM, Patrick Bellasi wrote: >>> The users might not be happy though >>> > > so we could favor performance for RT tasks to avoid breaking legacy >>> > > software and ask users that care about energy to migrate to deadline >>> > > where we actually know the performance constraints of the tasks. >> > >> > Given that at the moment RT tasks are treated no differently than CFS >> > tasks w.r.t. cpu frequency I'd expect that we could get away without any >> > sort of perf bias for RT bandwidth, which I think would be cost >> > prohibitive for power. > > Are you specifically considering instantaneous power? > Because from a power standpoint I cannot see any difference for > example w.r.t having a batch CFS task. > > From an energy standpoint instead, do not you think that a > "race-to-idle" policy could be better at least for RT-BATCH tasks? Sorry I should have said energy rather than power... >From my experience race to idle has never panned out as an energy-efficient strategy, presumably due to the nonlinear increase in power cost as performance increases. Because of this I think a policy of increasing the OPP when RT tasks are runnable will cause a net increase in energy consumption, which need not be incurred since RT tasks do not receive this preferential OPP treatment today.

11 years

Re: [Eas-dev] cpufreq_sched policy for combining requests from multiple sched classes

by Patrick Bellasi

[+eas-dev] On Mon, Oct 12, 2015 at 02:16:42AM -0700, Michael Turquette wrote: > Quoting Patrick Bellasi (2015-10-12 01:51:29) > > On Fri, Oct 09, 2015 at 11:58:22AM +0100, Michael Turquette wrote: > > > Steve, > > > > > > On Thu, Oct 8, 2015 at 1:59 AM, Steve Muckle <steve.muckle(a)linaro.org> wrote: > > > > At Linaro Connect a couple weeks back there was some sentiment that > > > > taking the max of multiple capacity requests would be the only viable > > > > policy when cpufreq_sched is extended to support multiple sched classes. > > > > But I'm concerned that this is not workable - if CFS is requesting 1GHz > > > > worth of bandwidth on a 2GHz CPU, and DEADLINE is also requesting 1GHz > > > > of bandwidth, we would run the CPU at 1GHz and starve the CFS tasks > > > > indefinitely. > > > > > > For the scheduler I think that summing makes sense. For peripheral > > > devices it places a much higher burden on the system integrator to > > > figure out how much compute is needed to achieve a specific use case. > > > > Are we still thinking about exposing an interface to device drivers? > > > > With sched-DVFS we are able to select the CPU's OPP based on real > > (expected) task demand. Thus, I expect constraints from device drivers > > being useful only in these use cases: > > > > a) the tasks activated by the driver are not "big enough" to > > generate an OPP switch, but still we want to race-to-idle their > > execution > > e.g. a light-weight control thread for an external accelerator, > > which could benefit from running on an higher OPP to reduce > > overall processing latencies > > b) we do not know which tasks require a boost in performances > > e.g. something like the "boost pulse" exposed by the Interactive > > governor, where the input subsystem is considered to trigger > > latency sensitive operations and thus the system deserves to be > > globally boosted > > > > There are other classes of use-cases? > > > > If these are the only use-cases we are thinking about, I'm wondering > > if we could not try to cover all them via SchedTune and the boost > > value it exposes. > > > > The use-case a) is already covered by the first implementation of > > SchedTune. If we know which task should be boosted, technically we > > could expose the SchedTune interface to drivers to allow them to boosts > > specific tasks. > > It sounds like could work for me. > > My concerns are bandwidth-related use cases. I've observed high speed > MMC controllers, WiFi chips, GPUs and other devices whose CPU tasks were > "small", but their performance was adversely affected when the CPU runs > at a slower rate. This is true even on relatively quiet system. I see your point, I'm wondering if most of these use-case should not be better implemented using DEADLINE instead of FIFO/RR. For these cases the problem will be solved once DL is properly integrated with sched-DVFS. For the remaining use-case where we still want (or we are "limited") to use FIFO/RT, I'm more on the idea that a race-to-idle strategy could just work. > Tasks running on the CPU can be viewed as latencies from the perspective > of a peripheral/IO device. TI and many other vendors have implemented > out-of-tree solutions to hold a CPU at a minimum OPP/frequency when > these drivers are running (usually with terrible hacks, but in some > cases nicely wrapped up in runtime_pm_{get,put} callbacks). I know very well these scenarios, in the past I experimented a lot with x86 machines running OpenCL workloads offloaded on GPGPUs. Quite frequently the bottleneck was the control thread running on the CPU, at the point that by co-scheduling two apps on the same GPU you can get better performance (for both apps) than just running one app at the time. This calls out for a kind of coordination, between workloads running on the CPU and the accelerator, on the frequency selection. But if you consider the specific GPGPU use-case, many time the source of knowledge about the required bandwidth is not kernel-space but instead in user-space. The OpenCL run-time, as well as many other run-time, could provide valuable input to the scheduler about these dependencies. > Does this fit with your model of how schedtune is supposed to work? I I think it's worth to have a try... provided that, in case of promising results, we are eventually satisfied to replace the CPUFreq specific API exposed to drivers with a more generic interface exposed to both kernel- and user-space. If instead we end up with two different APIs to achieve the same goal this will be just confusing. > have not looked at that stuff at all... are start_the_work() and > stop_the_work(() critical-section functions exposed to drivers? Mmm... I cannot get that question. Which functions/critical-sections are you referring to? > Regards, > Mike Cheers Patrick > > > > Regarding the second use-case b), this is a feature we try to address > > using the global boost value. Right now the only consumer of that > > global boost value is the FAIR scheduling class. However, it should be > > quite easy to extend it with the integration of SchedDVFS into other > > scheduling classes. > > > > > However, I might be the only one that is concerned with use case right > > > now. > > > > > > > > > > > I'd think there has to be a summing of bandwidth requests from scheduler > > > > class clients. MikeT raised the concern that in such schemes you often > > > > end up with a bunch of extra overhead because everyone adds their own > > > > fudge factor (Mike please correct me if I'm misstating our concern > > > > here). > > > > > > To be clear, I raised that point because I've actually seen that in > > > the past when TI implemented out-of-tree cpu frequency constraint > > > systems. I'm not being hypothetical :-) > > > > That's the point, SchedTune aims at becoming a sort of (hopefully) > > official solution to setup "frequency constraints". > > > > > > We should be able to control this in the scheduler classes though > > > > and ensure headroom is only added after the requests are combined. > > > > > > Sounds promising. Everyone else in this thread supports aggregation > > > (or summing) over maximum value, and I definitely won't argue the > > > point on the list unless it presents a real problem in testing. > > > > > > Regards, > > > Mike > > > > > > > > > > > Thoughts? > > > > > > > > > > > > -- > > > Michael Turquette > > > CEO > > > BayLibre - At the Heart of Embedded Linux > > > http://baylibre.com/ > > > > Cheers Patrick > > > > -- > > #include <best/regards.h> > > > > Patrick Bellasi > > > -- #include <best/regards.h> Patrick Bellasi

11 years

Re: [Eas-dev] Per CPU deadline scheduler bandwidth

by Steve Muckle

On 10/12/2015 06:55 AM, Vincent Guittot wrote: >> For the RT story, are you thinking to use rq->rt_avg in some way? > > Yes that was my goal but the deadline is also accounted in rq->rt_avg > which is then used in scale_rt_capacity. I was planning to remove the > deadline class from rq->rt_avg and to use the per cpu deadline > bandwidth in scale_rt_capacity instead If the deadline contribution to rq->rt_avg is left in place, can RT and deadline both be accounted for via that one mechanism for the purposes of sched-dvfs? I'm not sure how the accuracy/behavior of the deadline accounting to rq->rt_avg differs from the forthcoming patchset Juri mentioned, but if it's not as good, perhaps the mechanisms could be combined so that rt_avg (and by extension the per-CPU capacity adjustments in CFS) benefit as well.

11 years

Thoughts and Questions For EAS Energy Model

by Leo Yan

Hi all, Below are some thoughts and questions after reviewed EAS's energy model; my purpose is want to get clear the energy model from user's perspective, so below question will _ONLY_ focus on the model and not dig into the implementation. This email is related long, but i think if use formulas, we can easily get the same page; So i lists the energy model's formulas, then based on them i try to match with TC2's power data and bring up some questions. Look forward to your suggestions and comments. * Basic Energy and Power Calculation Formulas From the doc Documentation/scheduler/sched-energy.txt, we can get to know the energy can be calculated with: Energy [j] = Power [w] * Time [s] (F.1) So let's assume there have one piece of code, which has fixed instruction numbers will be executed on CPU, the execution duration is depend on CPU's pipeline and CPU's frequency. So can convert F.1 to F.2: Code [instructions] Energy [j] = Power [w] * ------------------------------ (Inst Per Cycle) * Frequency Code [instructions] = Power [w] * ------------------------------ (F.2) MIPS(f) `-> 'f' is factor of frequency Because MIPS(f) can be normalize as the CPU's capacity corresponding to OPP, so we can simply convert from F.2 to F.3: Code [instructions] Energy [j] = Power [w] * ------------------------------ (F.3) CPU_Capacity(f) If breakdown Power[w], we can split it into two parts: static leakage, and dynamic leakage: Power [w] = Ps [w] + Pd [w] (F.4) Static power leakage can be calculated with below formula: Ps [w] = i * V [v] (F.5) `-> 'i' is coefficient for according to silicon's process V [v] is voltage according to OPP Dynamic power leakage can be calculated with below formula: Pd [w] = b * V [v] * V [v] * frequency (F.6) `-> 'b' is coefficient for according to silicon's process V [v] is voltage according to OPP Here have two special cases, if the island's clock is gated, then Pd [w] = 0, So: Power [w] = Ps [w] (F.7) If the island is powered off, then Ps [w] = 0, Pd [w] = 0; So: Power [w] = 0 (F.8) So energy can be calculated as (come from F.3 and F.4): Code [instructions] Energy [j] = (Ps [w] + Pd [w]) * ---------------------- (F.9) CPU_Capacity(f) * Formulas for duty cycle We separate the logic (cluster or CPU) into two states: P-state and C-state, for P-state and C-state they have different power data, this is because after the logic enter C-state, it will be clock gating or powered off. So if we expand the time axis for relative long time, we need calculate CPU's utilization percentage (for CPU is full running, util = 100%). Let's simplize the ratio between "Code [instructions]" and "CPU_Capactity(f)" as the utilization, So the energy calculation can be depicted as: Code [instructions] Util(f) = -------------------------- (F.10) CPU_Capacity(f) Energy [j] = Power_Pstate [w] * Util(f) + Power_Cstate [w] * (1 - Util(f)) (F.11) (F.12) Energy [j] = Sum(i=0..MAX_OPP)(Power_Pstate [w](i) * Util_OPP(i)) + Sum(i=0..MAX_IDL)(Power_Cstate [w](i) * Util_IDL(i)) Sum(i=0..MAX_OPP)Util_OPP(i) + Sum(i..MAX_IDLE)Util_IDL(i) = 1 * Formulas for clusters (F.13) Energy [j] = Energy_cluster [j] + Sum(i=0..MAX_CPU_PER_CLUSTER)Energy_cpu(i) [j] (F.14) Energy_cluster [j] = Sum(i=0..MAX_OPP)(Power_Pstate [w](i) * Util_OPP(i)) + Sum(i=WFI, ClusterOff)(Power_Cstate [w](i) * Util_IDL(i)) (F.15) Energy_cpu [j] = Sum(i=0..MAX_OPP)(Power_Pstate [w](i) * Util_OPP(i)) + Sum(i=WFI, CPUOff)(Power_Cstate [w](i) * Util_IDL(i)) * Thoughts and Questions - Let's summary EAS's energy model as below: CPU::capacity_state::power : CPU's power [w] for specific OPP Power(OPP) = Ps [w] + Pd [w] CPU::idle_state::power : CPU's power [w] for specific idle state Power(IDLE_WFI) = Ps [w] Power(IDLE_CPUOff) = 0 CPU's IDLE_WFI means: CPU is clock gating, so has static leakage but don't include dynamic leakage. CLUSTER::capacity_state::power : Cluster's power [w] for specific OPP Power(OPP) = Ps [w] + Pd [w] CLUSTER::idle_state::power : CPU's power [w] for specific idle state Power(IDLE_WFI) = Ps [w] + Pd [w] Power(IDLE_CLSOff) = 0 Cluster's IDLE_WFI is quite special, means all CPUs in cluster have been powered off, but cluster's logic (L2$ and SCU, etc) is powered on and clock is enabled, so it includes cluster level's static power and dynamic power. Are these formulas matching the original design? - TC2's data for cluster's sleep: static struct idle_state idle_states_cluster_a7[] = { { .power = 25 }, /* WFI */ { .power = 10 }, /* cluster-sleep-l */ }; static struct idle_state idle_states_cluster_a15[] = { { .power = 70 }, /* WFI */ { .power = 25 }, /* cluster-sleep-b */ }; For cluster level's sleep, the clock is gating and domain is powered off, so the dynamic leakage and static leakge should be zero, right? - TC2's data for CPU's idle state: static struct idle_state idle_states_core_a7[] = { { .power = 0 }, /* WFI */ }; static struct idle_state idle_states_core_a15[] = { { .power = 0 }, /* WFI */ }; CPU has two idle state, one is 'WFI' and another is 'C2'; For 'WFI' state, the power will not be zero, this is because 'WFI' state means internal clock gating, so according to F.7, there should have static leakage. BTW, for TC2, there have no corresponding idle state for 'C2', this is weird. Could you confirm it has been delibrately removed? - TC2's data for P-state: static struct capacity_state cap_states_cluster_a7[] = { /* Cluster only power */ { .cap = 150, .power = 2967, }, /* 350 MHz */ [...] }; static struct capacity_state cap_states_core_a7[] = { /* Power per cpu */ { .cap = 150, .power = 187, }, /* 350 MHz */ [...] }; From previous experience, the CPU level's power leakage is very higher than cluster level's leakage. For example, for CA7, if only power on cluster (all CPUs in cluster are powered off), the power delta is ~10mA@156MHz; if power on one CPUs, the power delta is about 30mA@156MHz. I also checked the data for CA53, it has similar result. So this is confilict with TC2's power data, you can see the cluster level's power leakage is quite high (almost 15 times than CPU level). This means almostly we cannot get much benefit from CPU level's low power state, due cluster level will contribute most of power consumption. This is not make sense. - From formula F.4, we can combine power with static leakage and dynamic leakage; IPA also used static/dynamic leakage to depict energy model. But EAS uses another way, which provide the power data according to every OPP and idle state. So that means on one platform, we need provide two kinds of power data. IMHO, i think the static and dynamic leakage is more simple; because usually we will use (mW/MHz) to describe the power efficiency for specific CPU, though (mW/MHz) cannot very accurately for power consumption if the voltage has been changed (See formula F.6, usually the voltage will be increased at higher frequency). But if we use mW/MHz, maybe we can calculate with very simple way for we can just only use it to mulitplate with frequency to get dynamic power. So we only need provide below parameters: P-state: static leakage, power efficiency (mW/MHz), capacity (DMIPS/MHz); C-state: static leakage, power efficiency (mW/MHz); What's the thoughts for unify the energy model? Thanks, Leo Yan

11 years

EASv5 Profiling On Hikey (6th Sept)

by Leo Yan

Hi all, This is the third round profiling for EASv5 patches on Hikey board (6th, Sept); Welcome any comment and suggestion. * Overview - 3rd round vs 2nd round: Add two patches based on EASv5, the first patch will select small group/cpu if the groups have same capacity, so it will place tasks into the first cluster for the LITTLE.LITTLE case; the second patch will fix the case: if the sched domain is already the highest level, need directly use its group to calculate shared capacity and energy difference. Also have enclosed these two patches for review. - 2nd round vs 1st round: According to review comments from eas-dev mailing list, refined the energy model for Hikey, also developed several python/shell scripts for automatic analysis. So profiling data will be devided into three parts: - C-state profiling data - P-state profiling data - Scheduler performance profiling data * Hardware Environment - Platform: 96boards Hikey - SoC: Hi6220, 2 clusters, 4xCA53 CPUs in each cluster - CPU clock: 2 clusters with 8 CPUs have same clock source and support 208MHz/432MHz/729MHz/960MHz/1200MHz - Support CPU and cluster level low power mode * Software Environment - Kernel (4.2 + EAS RFCv5) + extra two patches [1] - ARM-TF [2] - Enable CPUIdle with PSCI - Enable CPUFreq with cpufreq-dt driver - Profiling scritps: calc_idle_diff.py [3]: calculate C-state's difference for different configurations calc_pstate_time.py [4]: calculate P-state's difference for different configurations calc_sched_preformance.py [5]: calculate scheduler performance * Conventions Below are some conventions which used in below tables: CLS0: Cluster 0 CLS1: Cluster 1 WFI: CPU WFI state C2: CPU power down state M2: Cluster power down state DC: Duty cycle Configuration | Mailine EASv5 Enable CPUFreq CPUFreq | ENERGY_AWARE ondemand sched ------------- | ------- ----- ------------ -------- ------- Mainline (ndm) | Yes No No Yes No noEAS (ndm) | Yes Yes No Yes No EAS (ndm) | Yes Yes Yes Yes No EAS (sched) | Yes Yes Yes No Yes * Profiling: C-state & P-state The detailed profiling result have been uploaded to git-hub [6]; - Case MP3: C-state DC Mainline (ndm) noEAS (ndm) EAS (ndm) EAS (sched) clusterA: WFI 104.68ms -35.3ms +36.6ms -65.0ms clusterA: C2 2.11s -2.0s -1.8s -973.0ms clusterA: M2 26.45s -6.7s -4.5s -15.7s clusterB: WFI 3.07ms -3.1ms -3.1ms -3.1ms clusterB: C2 98.88ms -56.4ms -98.9ms -88.8ms clusterB: M2 19.52s +9.0s +10.4s +10.3s P-State Statistics ------------------------------------------------------------------------------------------------- | Cluster Level (ms) | ------ ------------------------------------------------------------------------------------------ | Item | 208MHz | 432MHz | 729MHz | 960MHz | 1.2GHz | Cycles | ------------------------------------------------------------------------------------------------- | MAINLINE | 870.39 | 4872.34 | 93.79 | 27.43 | 5880.00 | 9436597.71 | | EAS DIS | +2.87%| -6.05%| -14.43%| -1.53%| 0.00%| -1.40%| | EAS NDM | -91.80%| -78.45%| +146.91%| +1384.29%| -0.16%| -14.46%| | EAS SCHED | +1771.55%| -79.00%| -100.00%| -100.00%| -84.15%| -47.56%| ------------------------------------------------------------------------------------------------- | CPU Level (ms) | ------ ------------------------------------------------------------------------------------------ | Item | 208MHz | 432MHz | 729MHz | 960MHz | 1.2GHz | Cycles | ------------------------------------------------------------------------------------------------- | MAINLINE | 919.34 | 4882.11 | 95.01 | 27.20 | 5888.56 | 9461934.73 | | EAS DIS | +6.26%| -3.38%| -11.19%| +0.18%| +0.94%| -0.01%| | EAS NDM | -88.31%| -78.24%| +155.38%| +1374.49%| -0.23%| -14.47%| | EAS SCHED | +1684.96%| -78.96%| -100.00%| -100.00%| -84.14%| -47.40%| ------------------------------------------------------------------------------------------------- - Case rt-app 6%: C-state DC Mainline (ndm) noEAS (ndm) EAS (ndm) EAS (sched) clusterA: WFI 8.99s -425.4ms -4.9s -462.3ms clusterA: C2 2.18s +810.8ms -2.2s -2.2s clusterA: M2 9.20ms -5.6ms -6.0ms -4.4ms clusterB: WFI 8.69s +135.2ms -8.6s -8.7s clusterB: C2 1.46s -229.9ms -1.5s -1.4s clusterB: M2 191.43ms -304us +19.8s +21.2s P-State Statistics ------------------------------------------------------------------------------------------------- | Cluster Level (ms) | ------ ------------------------------------------------------------------------------------------ | Item | 208MHz | 432MHz | 729MHz | 960MHz | 1.2GHz | Cycles | ------------------------------------------------------------------------------------------------- | MAINLINE | 16.73 | 18640.00 | 42.41 | 9.79 | 176.25 | 8307775.13 | | EAS DIS | -2.99%| -1.29%| -76.22%| +1.71%| +0.16%| -1.53%| | EAS NDM | -26.18%| -99.72%| +35.65%| +161267.72%| -62.97%| +84.31%| | EAS SCHED | +2939.63%| -100.00%| +26837.28%| +150.46%| +486.62%| +16.74%| ------------------------------------------------------------------------------------------------- | CPU Level (ms) | ------ ------------------------------------------------------------------------------------------ | Item | 208MHz | 432MHz | 729MHz | 960MHz | 1.2GHz | Cycles | ------------------------------------------------------------------------------------------------- | MAINLINE | 56.28 | 42010.00 | 76.46 | 9.86 | 181.47 | 18442998.06 | | EAS DIS | -4.92%| -1.33%| -85.96%| -2.70%| +0.50%| -1.57%| | EAS NDM | -59.28%| -99.66%| +14.01%| +160382.48%| -61.11%| -16.44%| | EAS SCHED | +879.42%| -100.00%| +28572.37%| +355.98%| +489.88%| -5.52%| ------------------------------------------------------------------------------------------------- - Case rt-app 13%: C-state DC Mainline (ndm) noEAS (ndm) EAS (ndm) EAS (sched) clusterA: WFI 7.08s -25.3ms -3.3s -2.3s clusterA: C2 4.11s -15.6ms -4.1s -4.1s clusterA: M2 6.80ms -3.4ms -695us +11.0ms clusterB: WFI 6.47s -329.8ms -6.5s -6.5s clusterB: C2 4.13s +497.4ms -4.1s -4.1s clusterB: M2 71.32ms +121.8ms +20.0s +21.3s P-State Statistics ------------------------------------------------------------------------------------------------- | Cluster Level (ms) | ------ ------------------------------------------------------------------------------------------ | Item | 208MHz | 432MHz | 729MHz | 960MHz | 1.2GHz | Cycles | ------------------------------------------------------------------------------------------------- | MAINLINE | 16.74 | 9730.00 | 8510.00 | 0.00 | 58.70 | 10481071.92 | | EAS DIS | -2.03%| -10.89%| +10.69%| +1864.00%| +190.19%| +3.41%| | EAS NDM | +0.78%| -99.77%| -98.47%| +1602036.00%| +222.13%| +49.94%| | EAS SCHED | +3304.36%| -81.57%| -79.87%| +1004144.00%| +4555.06%| +43.69%| ------------------------------------------------------------------------------------------------- | CPU Level (ms) | ------ ------------------------------------------------------------------------------------------ | Item | 208MHz | 432MHz | 729MHz | 960MHz | 1.2GHz | Cycles | ------------------------------------------------------------------------------------------------- | MAINLINE | 52.98 | 32160.00 | 24740.00 | 0.00 | 64.87 | 32017443.84 | | EAS DIS | +3.59%| -6.69%| +10.02%| +3186.00%| +164.31%| +3.24%| | EAS NDM | +3.68%| -99.78%| -98.58%| +3109608.80%| +275.04%| -4.92%| | EAS SCHED | +1057.87%| -78.78%| -75.17%| +1954311.10%| +5900.74%| -3.22%| ------------------------------------------------------------------------------------------------- - Case rt-app 19%: C-state DC Mainline (ndm) noEAS (ndm) EAS (ndm) EAS (sched) clusterA: WFI 8.59s +730.0ms -5.2s -4.6s clusterA: C2 795.08ms -479.8ms -792.0ms -791.9ms clusterA: M2 6.67ms +1.4ms -5.0ms -510us clusterB: WFI 6.78s +1.7s -6.7s -6.8s clusterB: C2 2.77s -2.1s -2.8s -2.8s clusterB: M2 73.12ms +2.9ms +20.0s +21.3s P-State Statistics ------------------------------------------------------------------------------------------------- | Cluster Level (ms) | ------ ------------------------------------------------------------------------------------------ | Item | 208MHz | 432MHz | 729MHz | 960MHz | 1.2GHz | Cycles | ------------------------------------------------------------------------------------------------- | MAINLINE | 16.91 | 6330.00 | 14440.00 | 321.43 | 58.54 | 13643658.08 | | EAS DIS | -1.54%| -42.34%| +21.88%| -100.00%| -0.07%| +6.14%| | EAS NDM | -3.08%| -100.00%| -99.19%| +5041.93%| +206.22%| +18.52%| | EAS SCHED | +3458.49%| -98.23%| -80.94%| +3764.90%| +2791.53%| +18.27%| ------------------------------------------------------------------------------------------------- | CPU Level (ms) | ------ ------------------------------------------------------------------------------------------ | Item | 208MHz | 432MHz | 729MHz | 960MHz | 1.2GHz | Cycles | ------------------------------------------------------------------------------------------------- | MAINLINE | 55.47 | 23900.00 | 45480.00 | 944.74 | 64.38 | 44475464.16 | | EAS DIS | +0.25%| -40.50%| +9.78%| -100.00%| +2.95%| -4.14%| | EAS NDM | -3.62%| -100.00%| -99.21%| +4529.65%| +185.78%| -4.48%| | EAS SCHED | +1466.81%| -98.46%| -78.07%| +3391.27%| +4333.30%| -4.00%| ------------------------------------------------------------------------------------------------- - Case rt-app 25%: C-state DC Mainline (ndm) noEAS (ndm) EAS (ndm) EAS (sched) clusterA: WFI 9.41s +80.7ms -5.2s -3.1s clusterA: C2 6.62ms +2.5ms -2.6ms -386us clusterA: M2 9.02ms -5.5ms -1.8ms +12.0ms clusterB: WFI 9.35s +72.1ms -9.3s -9.3s clusterB: C2 11.49ms +208us +2.0ms +40.2ms clusterB: M2 121.57ms +67.8ms +19.8s +20.4s P-State Statistics ------------------------------------------------------------------------------------------------- | Cluster Level (ms) | ------ ------------------------------------------------------------------------------------------ | Item | 208MHz | 432MHz | 729MHz | 960MHz | 1.2GHz | Cycles | ------------------------------------------------------------------------------------------------- | MAINLINE | 16.86 | 0.00 | 21240.00 | 9.90 | 109.76 | 15628682.88 | | EAS DIS | -5.69%| +1592.00%| -0.85%| +95.05%| +64.24%| -0.20%| | EAS NDM | +3.80%| 0.00%| -62.71%| +27272.12%| +4797.52%| -5.11%| | EAS SCHED | +3999.88%| +62.70%| -97.79%| +132936.06%| -57.80%| -15.63%| ------------------------------------------------------------------------------------------------- | CPU Level (ms) | ------ ------------------------------------------------------------------------------------------ | Item | 208MHz | 432MHz | 729MHz | 960MHz | 1.2GHz | Cycles | ------------------------------------------------------------------------------------------------- | MAINLINE | 55.41 | 0.00 | 76010.00 | 9.64 | 122.04 | 55578519.60 | | EAS DIS | -3.66%| +6163.00%| -2.84%| +318.59%| +66.44%| -2.56%| | EAS NDM | +0.43%| 0.00%| -59.18%| +91362.04%| +16748.26%| +0.34%| | EAS SCHED | +1258.89%| +159.70%| -99.03%| +136755.93%| -34.05%| -75.79%| ------------------------------------------------------------------------------------------------- - Case rt-app 31%: C-state DC Mainline (ndm) noEAS (ndm) EAS (ndm) EAS (sched) clusterA: WFI 8.64s -193.4ms -9.0ms -6.2s clusterA: C2 2.23ms -1.0ms +3.8ms +2.2ms clusterA: M2 8.42ms +1.2ms -1.4ms -5.0ms clusterB: WFI 8.64s -210.4ms -11.2ms -7.7s clusterB: C2 n.a. +1.6ms +4.9ms +79.7ms clusterB: M2 190.32ms -439us -119.2ms +18.9s P-State Statistics ------------------------------------------------------------------------------------------------- | Cluster Level (ms) | ------ ------------------------------------------------------------------------------------------ | Item | 208MHz | 432MHz | 729MHz | 960MHz | 1.2GHz | Cycles | ------------------------------------------------------------------------------------------------- | MAINLINE | 16.49 | 0.00 | 22710.00 | 18.56 | 181.44 | 16794565.52 | | EAS DIS | +0.12%| 0.00%| +1.81%| -46.77%| +0.23%| +1.73%| | EAS NDM | +2.55%| 0.00%| +0.18%| -100.00%| -61.71%| -0.73%| | EAS SCHED | +320.07%| 0.00%| -95.70%| +4179.96%| +9229.78%| +29.82%| ------------------------------------------------------------------------------------------------- | CPU Level (ms) | ------ ------------------------------------------------------------------------------------------ | Item | 208MHz | 432MHz | 729MHz | 960MHz | 1.2GHz | Cycles | ------------------------------------------------------------------------------------------------- | MAINLINE | 55.15 | 0.00 | 89550.00 | 9.60 | 205.88 | 65549695.12 | | EAS DIS | -3.25%| 0.00%| +1.82%| +0.33%| +0.66%| +1.81%| | EAS NDM | -0.25%| 0.00%| +0.45%| -100.00%| -51.82%| +0.24%| | EAS SCHED | +102.30%| 0.00%| -96.24%| +30805.83%| +23883.23%| -1.48%| ------------------------------------------------------------------------------------------------- - Case rt-app 38%: C-state DC Mainline (ndm) noEAS (ndm) EAS (ndm) EAS (sched) clusterA: WFI 7.38s -68.4ms -3.2s -4.7s clusterA: C2 3.81ms +3.0ms +7.7ms +4.4ms clusterA: M2 8.30ms -6.5ms -5.4ms -2.6ms clusterB: WFI 7.38s -6.7ms -2.6s -3.6s clusterB: C2 655us -655us +191.1ms +447.0ms clusterB: M2 71.73ms +52.2ms +8.7s +10.5s P-State Statistics ------------------------------------------------------------------------------------------------- | Cluster Level (ms) | ------ ------------------------------------------------------------------------------------------ | Item | 208MHz | 432MHz | 729MHz | 960MHz | 1.2GHz | Cycles | ------------------------------------------------------------------------------------------------- | MAINLINE | 16.43 | 0.00 | 13740.00 | 11510.00 | 70.25 | 21153777.44 | | EAS DIS | +1.77%| 0.00%| +0.44%| +0.09%| +70.23%| +0.53%| | EAS NDM | -0.79%| 0.00%| -90.94%| -72.37%| +25593.95%| +21.13%| | EAS SCHED | +2751.25%| 0.00%| -74.16%| -99.45%| +29707.83%| +31.77%| ------------------------------------------------------------------------------------------------- | CPU Level (ms) | ------ ------------------------------------------------------------------------------------------ | Item | 208MHz | 432MHz | 729MHz | 960MHz | 1.2GHz | Cycles | ------------------------------------------------------------------------------------------------- | MAINLINE | 55.33 | 0.00 | 53790.00 | 44650.00 | 103.91 | 82213110.64 | | EAS DIS | +0.56%| 0.00%| 0.00%| -1.81%| +50.12%| -0.87%| | EAS NDM | -1.66%| 0.00%| -91.35%| -75.88%| +53561.82%| -1.89%| | EAS SCHED | +821.76%| 0.00%| -77.45%| -99.53%| +55525.06%| -4.51%| ------------------------------------------------------------------------------------------------- - Case rt-app 44%: C-state DC Mainline (ndm) noEAS (ndm) EAS (ndm) EAS (sched) clusterA: WFI 5.62s +123.1ms -4.0s -1.7s clusterA: C2 4.22ms +933us -606us -1.6ms clusterA: M2 3.77ms +86us -697us +2.6ms clusterB: WFI 5.61s +87.6ms -46.5ms -859.0ms clusterB: C2 1.36ms -1.4ms +48.7ms +80.8ms clusterB: M2 71.19ms +10.7ms +4.1s +2.4s P-State Statistics ------------------------------------------------------------------------------------------------- | Cluster Level (ms) | ------ ------------------------------------------------------------------------------------------ | Item | 208MHz | 432MHz | 729MHz | 960MHz | 1.2GHz | Cycles | ------------------------------------------------------------------------------------------------- | MAINLINE | 15.51 | 19.62 | 15660.00 | 12720.00 | 451.61 | 24180973.92 | | EAS DIS | +5.74%| -100.00%| +0.19%| -5.82%| +115.02%| -0.30%| | EAS NDM | +13.73%| -100.00%| -74.27%| +44.65%| +1343.72%| +17.57%| | EAS SCHED | +210.57%| -100.00%| +30.46%| -90.64%| +1664.80%| +5.91%| ------------------------------------------------------------------------------------------------- | CPU Level (ms) | ------ ------------------------------------------------------------------------------------------ | Item | 208MHz | 432MHz | 729MHz | 960MHz | 1.2GHz | Cycles | ------------------------------------------------------------------------------------------------- | MAINLINE | 51.45 | 76.43 | 61390.00 | 50060.00 | 1482.15 | 94633209.36 | | EAS DIS | +6.03%| -100.00%| -1.24%| -6.07%| +129.76%| -1.26%| | EAS NDM | +12.05%| -100.00%| -74.51%| +15.60%| +1314.79%| -2.64%| | EAS SCHED | +72.01%| -100.00%| +27.51%| -91.65%| +1547.61%| -4.47%| ------------------------------------------------------------------------------------------------- - Case rt-app 50%: C-state DC Mainline (ndm) noEAS (ndm) EAS (ndm) EAS (sched) clusterA: WFI 6.34s +227.7ms +67.6ms +304.1ms clusterA: C2 3.35ms -1.5ms -2.0ms -1.7ms clusterA: M2 7.73ms -1.6ms -1.9ms -3.1ms clusterB: WFI 6.34s +231.2ms +69.2ms +308.0ms clusterB: C2 n.a. n.a. +2.0ms +2.2ms clusterB: M2 188.71ms -64.3ms -120.2ms +1.3s P-State Statistics ------------------------------------------------------------------------------------------------- | Cluster Level (ms) | ------ ------------------------------------------------------------------------------------------ | Item | 208MHz | 432MHz | 729MHz | 960MHz | 1.2GHz | Cycles | ------------------------------------------------------------------------------------------------- | MAINLINE | 16.81 | 0.00 | 9.95 | 27340.00 | 169.39 | 26460416.57 | | EAS DIS | +0.54%| 0.00%| +208.91%| -1.79%| -26.08%| -1.92%| | EAS NDM | -3.51%| 0.00%| +98.23%| -0.55%| -63.03%| -1.00%| | EAS SCHED | +2926.71%| 0.00%| -100.00%| -2.93%| +587.69%| +1.97%| ------------------------------------------------------------------------------------------------- | CPU Level (ms) | ------ ------------------------------------------------------------------------------------------ | Item | 208MHz | 432MHz | 729MHz | 960MHz | 1.2GHz | Cycles | ------------------------------------------------------------------------------------------------- | MAINLINE | 55.44 | 0.00 | 10.62 | 108060.00 | 170.45 | 103961414.23 | | EAS DIS | +0.41%| 0.00%| +743.42%| -1.43%| +2.25%| -1.37%| | EAS NDM | -1.75%| 0.00%| +90.32%| -0.92%| -50.47%| -1.01%| | EAS SCHED | +894.37%| 0.00%| -100.00%| -3.08%| +865.46%| -1.28%| ------------------------------------------------------------------------------------------------- * Profiling: performance sysbench --test=cpu --num-threads=1 --max-time=10 run rt-app performance is calculate with below formula: task performance = slack/(c_period - c_run) * 1024 energy mainline (ndm) noeas (ndm) eas (ndm) eas (sched) prf prf prf prf sysbench 100 100 100 92 rt-app 6% 662 665 393 615 rt-app 13% 648 645 465 394 rt-app 19% 610 648 479 57 rt-app 25% 649 664 306 518 rt-app 31% 600 585 596 366 rt-app 38% 576 584 259 -166 rt-app 44% 466 487 30 -349 rt-app 50% 583 602 598 612 * Summary - After applied the two extra patches, the profiling result is consistent and stable for EAS (ndm) and EAS (sched). The tasks will be placed into first cluster for LITTLE.LITTLE; so EAS (ndm) and EAS (sched) are much better for cluster level's idle duty cycle compared with noEAS (ndm). - If the tasks are placed only on one cluster, the CPU's cycles will not change too much, but cluster level's cycles will increase much higher with EAS (ndm) and EAS (sched); So after packed tasks into one cluster, the cluster level will run longer time. - With "sched" governor, it is more aggressive than "ondemand" governor, the CPUs will easily run at high OPPs (1.2GHz) and low OPP (208MHz); With "ondemand" governor, CPUs have many chances run at middle OPPs (729MHz or 960MHz). - Need investigate rt-app 31% case, it is abnormal for cpu/cluster's idle duty cycle. - Need investigate rt-app 25% case, it is abnormal for "ondemand" governor, which stay at OPP 1.2GHz much longer than other configurations. [1] https://github.com/Leo-Yan/linux/tree/profile_easv5_hikey_round3 [2] https://github.com/96boards/arm-trusted-firmware/tree/hikey [3] https://github.com/Leo-Yan/utility/blob/master/profile_eas/calc_idle_diff.py [4] https://github.com/Leo-Yan/utility/blob/master/profile_eas/calc_pstate_time… [5] https://github.com/Leo-Yan/utility/blob/master/profile_eas/calc_sched_prefo… [6] https://github.com/Leo-Yan/utility/tree/master/profile_eas/hikey_easv5_roun… Thanks, Leo Yan

11 years, 1 month

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

eas-dev