Hi, My first mail here, please tell me if this is not the right list for my queries. I have recently got myself a thundersoft s835 development kit and set it up for power measurement. I was looking at creating the energy model for this from scratch and measuring the impact. I stumbled across something which led me to write this mail. The eas integration document mentions no support currently exists for SMT. However, my kernel configuration for some reason has SCHED_SMT enabled. The s835 is not hyper-threaded and so I am thinking it's a mistake. My questions are: 1) Is disabling SCHED_SMT enough or are there any other configurations that need to be changed as well ? 2) I'd like to understand the technical difficulties with enabling SMT support since I have a mips device which has hyperthreading. Is it that the cpu-cluster energy model would not be enough to handle proper task placement when it comes to virtual cores ? 3) Is there any on-going work on adding SMT support or is it considered as not being worth the effort ? Thanks for reading. -Jason
Hi,
On 22/03/18 16:33, Jason Cren wrote:
Hi, My first mail here, please tell me if this is not the right list for my queries.
You have knocked at the right door :)
I have recently got myself a thundersoft s835 development kit and set it up for power measurement. I was looking at creating the energy model for this from scratch and measuring the impact. I stumbled across something which led me to write this mail. The eas integration document mentions no support currently exists for SMT. However, my kernel configuration for some reason has SCHED_SMT enabled. The s835 is not hyper-threaded and so I am thinking it's a mistake. My questions are
- Is disabling SCHED_SMT enough or are there any other configurations that need to be changed as well ?
Regarding SCHED_SMT, if you have:
zcat /proc/config.z | grep SCHED_SMT
CONFIG_SCHED_SMT=y
Then that's not necessarily a wrong configuration. All this does is allow the SMT code to be compiled in the kernel ([1]) - if it is used depends on what is there at runtime ([2]). However, it is broken with EAS so you should disable it if you want to use EAS.
- I'd like to understand the technical difficulties with enabling SMT support since I have a mips device which has hyperthreading. Is it that the cpu-cluster energy model would not be enough to handle proper task placement when it comes to virtual cores ?
So I'm really not well versed in SMT but I think the biggest problem is that IIRC SMT adds another scheduler domain ((DIE, MC) + SMT).
Plus there are assumptions in EAS that make sense for big.LITTLE but not really for SMT: I recall that the capacity of two virtual cores is smaller than the capacity of the physical core they reside in, which can potentially mess up the heuristics that use capacity.
- Is there any on-going work on adding SMT support or is it considered as not being worth the effort ?
I know there's work being done on bugfixes, but that's to make the kernel not crash with SCHED_SMT (see my answer to 1), not to make EAS work on SMT HW. Beyond that I'm not the right person to ask, sorry :(
Thanks for reading. -Jason
Cheers, Valentin
[1]: https://android.googlesource.com/kernel/common/+/android-4.9/kernel/sched/fa... [2]:
That's where the topology code comes into play. Most of it is still somewhat obscure to me (the only cure is staring at it long enough really), but you can see that there are flags that say "my hardware has SMT": [3].
You can see the flags that are defined on your system by issuing:
cat /proc/sys/kernel/sched_domain/cpu*/domain*/flags
For instance, I get for my HiKey960:
cat /proc/sys/kernel/sched_domain/cpu0/domain*/flags
4671 4223
Flag values are defined in [4], and that lets me see that SD_ASYM_CPUCAPACITY is set for domain1 but not for domain0. We have some tests [5] that check those flags if you want to do a comparison with your setup.
[3]: https://android.googlesource.com/kernel/common/+/android-4.9/kernel/sched/co... [4]: https://android.googlesource.com/kernel/common/+/android-4.9/include/linux/s... [5]: https://github.com/mdigiorgio/lisa/blob/b9157d7d5b20a13c6e420ba2bb63e26f983a...
Hi Valentin,
Sent: Friday, March 23, 2018 at 5:44 AM From: "Valentin Schneider" valentin.schneider@arm.com To: "Jason Cren" jason.cren@mail.com, eas-dev@lists.linaro.org Subject: Re: [Eas-dev] eas question
Hi,
On 22/03/18 16:33, Jason Cren wrote:
Hi, My first mail here, please tell me if this is not the right list for my queries.
You have knocked at the right door :)
Thanks for the reply.
I have recently got myself a thundersoft s835 development kit and set it up for power measurement. I was looking at creating the energy model for this from scratch and measuring the impact. I stumbled across something which led me to write this mail. The eas integration document mentions no support currently exists for SMT. However, my kernel configuration for some reason has SCHED_SMT enabled. The s835 is not hyper-threaded and so I am thinking it's a mistake. My questions are
- Is disabling SCHED_SMT enough or are there any other configurations that need to be changed as well ?
Regarding SCHED_SMT, if you have:
zcat /proc/config.z | grep SCHED_SMT
CONFIG_SCHED_SMT=y
This is what I have.
Then that's not necessarily a wrong configuration. All this does is allow the SMT code to be compiled in the kernel ([1]) - if it is used depends on what is there at runtime ([2]). However, it is broken with EAS so you should disable it if you want to use EAS.
Ok, I wasn't aware of the run-time check. I was under the faulty impression that having SCHED_SMT enabled would be a problem even without eas. In conclusion, for my eas experiments I should disable SCHED_SMT.
- I'd like to understand the technical difficulties with enabling SMT support since I have a mips device which has hyperthreading. Is it that the cpu-cluster energy model would not be enough to handle proper task placement when it comes to virtual cores ?
So I'm really not well versed in SMT but I think the biggest problem is that IIRC SMT adds another scheduler domain ((DIE, MC) + SMT).
Plus there are assumptions in EAS that make sense for big.LITTLE but not really for SMT: I recall that the capacity of two virtual cores is smaller than the capacity of the physical core they reside in, which can potentially mess up the heuristics that use capacity.
So, the issue is with the calculation of correct capacities for the logical cores and addition of SMT domain. I am not familiar with the code other than what is in the eas intergration guide and a doc on how to create the energy model for a new board. I will start staring at some of the code :) and then things might be clearer.
- Is there any on-going work on adding SMT support or is it considered as not being worth the effort ?
I know there's work being done on bugfixes, but that's to make the kernel not crash with SCHED_SMT (see my answer to 1), not to make EAS work on SMT HW. Beyond that I'm not the right person to ask, sorry :(
Appreciate your responses Valentin. Perhaps someone else on the list might chime in regarding this.
Thanks for reading. -Jason
Cheers, Valentin
[2]:
That's where the topology code comes into play. Most of it is still somewhat obscure to me (the only cure is staring at it long enough really), but you can see that there are flags that say "my hardware has SMT": [3].
You can see the flags that are defined on your system by issuing:
cat /proc/sys/kernel/sched_domain/cpu*/domain*/flags
For instance, I get for my HiKey960:
cat /proc/sys/kernel/sched_domain/cpu0/domain*/flags
4671 4223
Flag values are defined in [4], and that lets me see that SD_ASYM_CPUCAPACITY is set for domain1 but not for domain0. We have some tests [5] that check those flags if you want to do a comparison with your setup.
Thanks for the links, will go through them and get back.
-Jason
Hi Jason,
On 03/23/2018 05:12 AM, Jason Cren wrote:
Hi Valentin,
Sent: Friday, March 23, 2018 at 5:44 AM From: "Valentin Schneider" valentin.schneider@arm.com To: "Jason Cren" jason.cren@mail.com, eas-dev@lists.linaro.org Subject: Re: [Eas-dev] eas question
[...]
I have recently got myself a thundersoft s835 development kit and set it up for power measurement. I was looking at creating the energy model for this from scratch and measuring the impact. I stumbled across something which led me to write this mail. The eas integration document mentions no support currently exists for SMT. However, my kernel configuration for some reason has SCHED_SMT enabled. The s835 is not hyper-threaded and so I am thinking it's a mistake. My questions are
- Is disabling SCHED_SMT enough or are there any other configurations that need to be changed as well ?
Regarding SCHED_SMT, if you have:
zcat /proc/config.z | grep SCHED_SMT
CONFIG_SCHED_SMT=y
This is what I have >> Then that's not necessarily a wrong configuration. All this does is
allow the
SMT code to be compiled in the kernel ([1]) - if it is used depends on what is there at runtime ([2]). However, it is broken with EAS so you should disable it if you want to use EAS.
Ok, I wasn't aware of the run-time check. I was under the faulty impression that having SCHED_SMT enabled would be a problem even without eas. In conclusion, for my eas experiments I should disable SCHED_SMT.
Like Valentin said already, disable it for now. EAS requires that if there is energy model (em) data on sched domain (sd) level x (e.g. MC) then there must be em data also on sd level x-1 (child of x) (e.g. SMT).
init_sched_groups_energy()
+ if (sd->child && !sd->child->groups->sge) { + pr_err("BUG: EAS setup borken for CPU%d\n", cpu); +#ifdef CONFIG_SCHED_DEBUG + pr_err(" energy data on %s but not on %s domain\n", + sd->name, sd->child->name); +#endif + return; + }
Normally, even if you have CONFIG_SMT set on a non-smt platform, it should work since the scheduler init code figures out that it can remove the smt level (have a look into the sd degenerate functions in kernel/sched/topology.c). But EAS init code currently gets executed when this transient smt level still exists in the system and this is the bugfix Valentin referred to.
- I'd like to understand the technical difficulties with enabling SMT support since I have a mips device which has hyperthreading. Is it that the cpu-cluster energy model would not be enough to handle proper task placement when it comes to virtual cores ?
So I'm really not well versed in SMT but I think the biggest problem is that IIRC SMT adds another scheduler domain ((DIE, MC) + SMT).
Plus there are assumptions in EAS that make sense for big.LITTLE but not really for SMT: I recall that the capacity of two virtual cores is smaller than the capacity of the physical core they reside in, which can potentially mess up the heuristics that use capacity.
So, the issue is with the calculation of correct capacities for the logical cores and addition of SMT domain. I am not familiar with the code other than what is in the eas intergration guide and a doc on how to create the energy model for a new board. I will start staring at some of the code :) and then things might be clearer.
- Is there any on-going work on adding SMT support or is it considered as not being worth the effort ?
I know there's work being done on bugfixes, but that's to make the kernel not crash with SCHED_SMT (see my answer to 1), not to make EAS work on SMT HW. Beyond that I'm not the right person to ask, sorry :(
Appreciate your responses Valentin. Perhaps someone else on the list might chime in regarding this.
We're currently fixing the issue to let EAS come up nicely on a non-smp platform with CONFIG_SMP set. I don't remember seeing the kernel crash but the em bring-up code gets confused.
SMT in the fair scheduler (kernel/sched/fair) behaves much different than the other sd level (MC, DIE). Look out for SD_PREFER_SIBLING and prefer_sibling in update_sd_lb_stats() and you will see that the current EAS heuristics and algorithms are not applicable to SMT.
We're currently not working on enabling EAS on SMT but if you have an idea how to do it, please share it with us.
Thanks,
-- Dietmar
[...]
-
Dietmar Eggemann -
Jason Cren -
Valentin Schneider