Hi,
One of the top questions that comes up on CoreSight is how it interacts with CPU power saving, and I'd like to get a handle on where we are with this. This will help us understand if any more work needs to be done.
I'd suggest three levels of support:
- transparent: use of CoreSight has no effect on CPU power saving - if an idle CPU would have been powered down it's still powered down. Any increased power draw from CoreSight comes from debug/trace blocks being powered up as necessary, not from keeping entire CPUs powered up.
- automatic wakelock: use of CoreSight has the effect of disabling powering off of idle CPUs, so there may be a significant increase in power consumption, but it's done automatically and works out of the box. CoreSight itself is fully functional irrespective of how the system is configured.
- invasive: power saving must be disabled manually - i.e. you have to get a manual (and possibly device-specific) recipe from somewhere. If you don't then things will break (loss of trace at best, crash at worst).
I would hope that perf (all modes) is transparent, and direct use of sysfs is at worst a wakelock... but where are we now? Are there still boards that need manual recipes with the current kernel - either with perf or with sysfs?
Thanks,
Al
On Fri, 18 Jan 2019 at 02:44, Al Grant Al.Grant@arm.com wrote:
Hi,
One of the top questions that comes up on CoreSight is how it interacts with CPU power saving, and I'd like to get a handle on where we are with this. This will help us understand if any more work needs to be done.
I'd suggest three levels of support:
- transparent: use of CoreSight has no effect on CPU power saving - if an idle
CPU would have been powered down it's still powered down. Any increased power draw from CoreSight comes from debug/trace blocks being powered up as necessary, not from keeping entire CPUs powered up.
Yes, that is the ideal scenario and also how Juno is currently working. It uses the TRCPDRC:PU bit and requires FW and PMIC support. To the best of my knowledge it is the only platform that works this way. It is hard to implement because it requires involvement from different teams (kernel, FW, PMIC) and the HW needs to have been designed properly.
Seeing the improbability to get all this right on all platform, three years ago I set out to fix the solution in the kernel by using the genPD subsystem. That was swiftly refused by the Juno maintainers, leaving only the above as an option.
- automatic wakelock: use of CoreSight has the effect of disabling powering
off of idle CPUs, so there may be a significant increase in power consumption, but it's done automatically and works out of the box. CoreSight itself is fully functional irrespective of how the system is configured.
This is functionality that I will not accept upstream and will have to be carried (and maintained) out of tree by ARM. To me it is better to spend time and efforts on making things right using either the FW/PMIC mechanic or genPD.
- invasive: power saving must be disabled manually - i.e. you have to get a
manual (and possibly device-specific) recipe from somewhere. If you don't then things will break (loss of trace at best, crash at worst).
That is how all platforms (except Juno) currently work. CPUidle needs to be manually disabled at compilation time or at runtime via sysfs.
I would hope that perf (all modes) is transparent, and direct use of sysfs is at worst a wakelock... but where are we now? Are there still boards that need manual recipes with the current kernel - either with perf or with sysfs?
I think the first thing that needs to be worked on is the integration of CPUs with genPD so that CPU power domains can be controlled by genPD rather than CPUidle. That way power management can be done by the kernel rather than in FW. I'm being told the functionality is now supported in genPD but I wouldn't bet a penny that it is adequate for CS or that it does exactly what we want.
Second is properly handling CPUhotplug operation when perf or sysfs sessions are ongoing.
Mathieu
Thanks,
Al _______________________________________________ CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight
On Fri, Jan 18, 2019 at 04:11:33PM -0700, Mathieu Poirier wrote:
On Fri, 18 Jan 2019 at 02:44, Al Grant Al.Grant@arm.com wrote:
Hi,
One of the top questions that comes up on CoreSight is how it interacts with CPU power saving, and I'd like to get a handle on where we are with this. This will help us understand if any more work needs to be done.
I'd suggest three levels of support:
- transparent: use of CoreSight has no effect on CPU power saving - if an idle
CPU would have been powered down it's still powered down. Any increased power draw from CoreSight comes from debug/trace blocks being powered up as necessary, not from keeping entire CPUs powered up.
Yes, that is the ideal scenario and also how Juno is currently working. It uses the TRCPDRC:PU bit and requires FW and PMIC support. To the best of my knowledge it is the only platform that works this way. It is hard to implement because it requires involvement from different teams (kernel, FW, PMIC) and the HW needs to have been designed properly.
Sorry jump into this topic and if introduce noise :)
Just want to remind, for dynamic power management, at least we need to consider below factors:
- Clock domain; - Power domain; - Voltage domain; - Context saving and restoring;
With reading roughly for the spec when I wrote driver for CPU debug module, TRCPDRC:PU is used as an indicate signal for power controller (Usually the power controller means the logic in SoC, it either can be one MCU or it's integrated logic in SoC) to enable power domian for CoreSight logic. But clock domain and voltage domain controlling is separated from power domain controlling (actually this kind design is quite common for Arm SoCs), hence from the software pespective we need to handle all of them and it's not always valid for clock domain/power domain/voltage domain can response to TRCPDRC:PU bit, especially for the voltage domain which is controlled by external PMIC.
Furthermore, if the registers value will be lost during CPUIdle (e.g. for ETM), we need to provide mechanism to save and restore the content for CPU power cycle. This part is missed.
Seeing the improbability to get all this right on all platform, three years ago I set out to fix the solution in the kernel by using the genPD subsystem. That was swiftly refused by the Juno maintainers, leaving only the above as an option.
- automatic wakelock: use of CoreSight has the effect of disabling powering
off of idle CPUs, so there may be a significant increase in power consumption, but it's done automatically and works out of the box. CoreSight itself is fully functional irrespective of how the system is configured.
This is functionality that I will not accept upstream and will have to be carried (and maintained) out of tree by ARM. To me it is better to spend time and efforts on making things right using either the FW/PMIC mechanic or genPD.
- invasive: power saving must be disabled manually - i.e. you have to get a
manual (and possibly device-specific) recipe from somewhere. If you don't then things will break (loss of trace at best, crash at worst).
That is how all platforms (except Juno) currently work. CPUidle needs to be manually disabled at compilation time or at runtime via sysfs.
I would hope that perf (all modes) is transparent, and direct use of sysfs is at worst a wakelock... but where are we now? Are there still boards that need manual recipes with the current kernel - either with perf or with sysfs?
I think the first thing that needs to be worked on is the integration of CPUs with genPD so that CPU power domains can be controlled by genPD rather than CPUidle. That way power management can be done by the kernel rather than in FW. I'm being told the functionality is now supported in genPD but I wouldn't bet a penny that it is adequate for CS or that it does exactly what we want.
Just curious for the genPD usage with CoreSight.
For a sane solution, we can use genPD to manage the dependency within different CoreSight components, and also manage the dependency between CoreSight and CPU. But let's see if we can support dynamic power management without genPD (and maybe it's simple for the first step :))
Seems to me we can get rid of the dependency between CoreSight and CPU simply based on the task switching, e.g. when the task is switched out and CPU runs with idle task, we firstly disable logic for CoreSight and then the CPU can be safely powered off by CPUIdle. Should this be sufficient for perf (per-thread mode and CPU wide mode)?
Using genPD so we can get benefit for wider tracing scope (e.g. using SysFS method for firmwares tracing). If so, genPD is used as wakelock for this case.
Second is properly handling CPUhotplug operation when perf or sysfs sessions are ongoing.
Since the CPU hotplug will migrate all tasks to other online CPUs and at the end the idle task is running, so if we simply diable CoreSight when switch to idle task then CPU hotplug can be easily handled. Thoughts?
Thanks, Leo Yan
On Fri, 18 Jan 2019 at 20:03, Leo Yan leo.yan@linaro.org wrote:
On Fri, Jan 18, 2019 at 04:11:33PM -0700, Mathieu Poirier wrote:
On Fri, 18 Jan 2019 at 02:44, Al Grant Al.Grant@arm.com wrote:
Hi,
One of the top questions that comes up on CoreSight is how it interacts with CPU power saving, and I'd like to get a handle on where we are with this. This will help us understand if any more work needs to be done.
I'd suggest three levels of support:
- transparent: use of CoreSight has no effect on CPU power saving - if an idle
CPU would have been powered down it's still powered down. Any increased power draw from CoreSight comes from debug/trace blocks being powered up as necessary, not from keeping entire CPUs powered up.
Yes, that is the ideal scenario and also how Juno is currently working. It uses the TRCPDRC:PU bit and requires FW and PMIC support. To the best of my knowledge it is the only platform that works this way. It is hard to implement because it requires involvement from different teams (kernel, FW, PMIC) and the HW needs to have been designed properly.
Sorry jump into this topic and if introduce noise :)
Just want to remind, for dynamic power management, at least we need to consider below factors:
- Clock domain;
- Power domain;
- Voltage domain;
- Context saving and restoring;
With reading roughly for the spec when I wrote driver for CPU debug module, TRCPDRC:PU is used as an indicate signal for power controller (Usually the power controller means the logic in SoC, it either can be one MCU or it's integrated logic in SoC) to enable power domian for CoreSight logic. But clock domain and voltage domain controlling is separated from power domain controlling (actually this kind design is quite common for Arm SoCs), hence from the software pespective we need to handle all of them and it's not always valid for clock domain/power domain/voltage domain can response to TRCPDRC:PU bit, especially for the voltage domain which is controlled by external PMIC.
Furthermore, if the registers value will be lost during CPUIdle (e.g. for ETM), we need to provide mechanism to save and restore the content for CPU power cycle. This part is missed.
That is only the case when operated from sysfs. When talking about perf, the only thing that is not supported is if a trace session is started *and then* a CPU is hotplugged in. Tracing on that CPU will not work.
Seeing the improbability to get all this right on all platform, three years ago I set out to fix the solution in the kernel by using the genPD subsystem. That was swiftly refused by the Juno maintainers, leaving only the above as an option.
- automatic wakelock: use of CoreSight has the effect of disabling powering
off of idle CPUs, so there may be a significant increase in power consumption, but it's done automatically and works out of the box. CoreSight itself is fully functional irrespective of how the system is configured.
This is functionality that I will not accept upstream and will have to be carried (and maintained) out of tree by ARM. To me it is better to spend time and efforts on making things right using either the FW/PMIC mechanic or genPD.
- invasive: power saving must be disabled manually - i.e. you have to get a
manual (and possibly device-specific) recipe from somewhere. If you don't then things will break (loss of trace at best, crash at worst).
That is how all platforms (except Juno) currently work. CPUidle needs to be manually disabled at compilation time or at runtime via sysfs.
I would hope that perf (all modes) is transparent, and direct use of sysfs is at worst a wakelock... but where are we now? Are there still boards that need manual recipes with the current kernel - either with perf or with sysfs?
I think the first thing that needs to be worked on is the integration of CPUs with genPD so that CPU power domains can be controlled by genPD rather than CPUidle. That way power management can be done by the kernel rather than in FW. I'm being told the functionality is now supported in genPD but I wouldn't bet a penny that it is adequate for CS or that it does exactly what we want.
Just curious for the genPD usage with CoreSight.
For a sane solution, we can use genPD to manage the dependency within different CoreSight components, and also manage the dependency between CoreSight and CPU. But let's see if we can support dynamic power management without genPD (and maybe it's simple for the first step :))
Seems to me we can get rid of the dependency between CoreSight and CPU simply based on the task switching, e.g. when the task is switched out and CPU runs with idle task, we firstly disable logic for CoreSight and then the CPU can be safely powered off by CPUIdle. Should this be sufficient for perf (per-thread mode and CPU wide mode)?
The only reason to add genPD into the mix is to provide an alternative to the TRCPDRC:PU bit manipulation so that boards that can't use it (for whatever reason) have a fallback option.
Using genPD so we can get benefit for wider tracing scope (e.g. using SysFS method for firmwares tracing). If so, genPD is used as wakelock for this case.
Second is properly handling CPUhotplug operation when perf or sysfs sessions are ongoing.
Hotplug handling with perf: As mentionned above, this is only needed when a session (CPU wide or per-thread) has been started and a CPU is hotplugged in.
Hotplug handling with sysfs: That needs to be addressed.
Since the CPU hotplug will migrate all tasks to other online CPUs and at the end the idle task is running, so if we simply diable CoreSight when switch to idle task then CPU hotplug can be easily handled.
Again, it is very important to specify this is only relevant for sysfs. There is already a hotplug state machine that takes care of signalling drivers that a CPU is going down (or up).
Thoughts?
Thanks, Leo Yan
Hi,Been looking around at this both for CTI drivers and since it was raised...
On Mon, 21 Jan 2019 at 17:01, Mathieu Poirier mathieu.poirier@linaro.org wrote:
On Fri, 18 Jan 2019 at 20:03, Leo Yan leo.yan@linaro.org wrote:
On Fri, Jan 18, 2019 at 04:11:33PM -0700, Mathieu Poirier wrote:
On Fri, 18 Jan 2019 at 02:44, Al Grant Al.Grant@arm.com wrote:
Hi,
One of the top questions that comes up on CoreSight is how it interacts with CPU power saving, and I'd like to get a handle on where we are with this. This will help us understand if any more work needs to be done.
I'd suggest three levels of support:
- transparent: use of CoreSight has no effect on CPU power saving - if an idle
CPU would have been powered down it's still powered down. Any increased power draw from CoreSight comes from debug/trace blocks being powered up as necessary, not from keeping entire CPUs powered up.
Yes, that is the ideal scenario and also how Juno is currently working. It uses the TRCPDRC:PU bit and requires FW and PMIC support. To the best of my knowledge it is the only platform that works this way. It is hard to implement because it requires involvement from different teams (kernel, FW, PMIC) and the HW needs to have been designed properly.
Sorry jump into this topic and if introduce noise :)
Just want to remind, for dynamic power management, at least we need to consider below factors:
- Clock domain;
- Power domain;
- Voltage domain;
- Context saving and restoring;
With reading roughly for the spec when I wrote driver for CPU debug module, TRCPDRC:PU is used as an indicate signal for power controller (Usually the power controller means the logic in SoC, it either can be one MCU or it's integrated logic in SoC) to enable power domian for CoreSight logic. But clock domain and voltage domain controlling is separated from power domain controlling (actually this kind design is quite common for Arm SoCs), hence from the software pespective we need to handle all of them and it's not always valid for clock domain/power domain/voltage domain can response to TRCPDRC:PU bit, especially for the voltage domain which is controlled by external PMIC.
Furthermore, if the registers value will be lost during CPUIdle (e.g. for ETM), we need to provide mechanism to save and restore the content for CPU power cycle. This part is missed.
That is only the case when operated from sysfs. When talking about perf, the only thing that is not supported is if a trace session is started *and then* a CPU is hotplugged in. Tracing on that CPU will not work.
The impression I got from looking at the code, was that the hotplug would re-enble ETM trace for both perf and sysfs when a core came up - assuming that the relevant config data was present. i.e. the 'start_cpu()' callback checks that the ETM mode is _not_ disabled and will therefore enable it.
Both CTI and ETM drivers keep a lot of register copies as part of the driver data, only to transfer it on hardware enable. Thus quite a bit of restore is present for both CTI and ETM. The CTI does not differentiate between perf and sysfs - it is enabled whenever any associated coresight component in enbled - hence core CTIs should enable alongside core ETMs.
There is an issue with save status when a core is hotplugged out - no attempt is made to save values of dynamic elements like counters. However these are currently unused in perf directly, but can be set by sysfs - e.g. when using the AFDO patch for ETM strobing. This can and should be addressed when we make changes for the complex configuration of CoreSight.
Seeing the improbability to get all this right on all platform, three years ago I set out to fix the solution in the kernel by using the genPD subsystem. That was swiftly refused by the Juno maintainers, leaving only the above as an option.
- automatic wakelock: use of CoreSight has the effect of disabling powering
off of idle CPUs, so there may be a significant increase in power consumption, but it's done automatically and works out of the box. CoreSight itself is fully functional irrespective of how the system is configured.
This is functionality that I will not accept upstream and will have to be carried (and maintained) out of tree by ARM. To me it is better to spend time and efforts on making things right using either the FW/PMIC mechanic or genPD.
- invasive: power saving must be disabled manually - i.e. you have to get a
manual (and possibly device-specific) recipe from somewhere. If you don't then things will break (loss of trace at best, crash at worst).
That is how all platforms (except Juno) currently work. CPUidle needs to be manually disabled at compilation time or at runtime via sysfs.
I would hope that perf (all modes) is transparent, and direct use of sysfs is at worst a wakelock... but where are we now? Are there still boards that need manual recipes with the current kernel - either with perf or with sysfs?
I think the first thing that needs to be worked on is the integration of CPUs with genPD so that CPU power domains can be controlled by genPD rather than CPUidle. That way power management can be done by the kernel rather than in FW. I'm being told the functionality is now supported in genPD but I wouldn't bet a penny that it is adequate for CS or that it does exactly what we want.
Just curious for the genPD usage with CoreSight.
For a sane solution, we can use genPD to manage the dependency within different CoreSight components, and also manage the dependency between CoreSight and CPU. But let's see if we can support dynamic power management without genPD (and maybe it's simple for the first step :))
Seems to me we can get rid of the dependency between CoreSight and CPU simply based on the task switching, e.g. when the task is switched out and CPU runs with idle task, we firstly disable logic for CoreSight and then the CPU can be safely powered off by CPUIdle. Should this be sufficient for perf (per-thread mode and CPU wide mode)?
The only reason to add genPD into the mix is to provide an alternative to the TRCPDRC:PU bit manipulation so that boards that can't use it (for whatever reason) have a fallback option.
Using genPD so we can get benefit for wider tracing scope (e.g. using SysFS method for firmwares tracing). If so, genPD is used as wakelock for this case.
Second is properly handling CPUhotplug operation when perf or sysfs sessions are ongoing.
Hotplug handling with perf: As mentionned above, this is only needed when a session (CPU wide or per-thread) has been started and a CPU is hotplugged in.
Hotplug handling with sysfs: That needs to be addressed.
Since the CPU hotplug will migrate all tasks to other online CPUs and at the end the idle task is running, so if we simply diable CoreSight when switch to idle task then CPU hotplug can be easily handled.
Again, it is very important to specify this is only relevant for sysfs. There is already a hotplug state machine that takes care of signalling drivers that a CPU is going down (or up).
Thoughts?
Thanks, Leo Yan
CoreSight mailing list CoreSight@lists.linaro.org https://lists.linaro.org/mailman/listinfo/coresight
Having a look around for info on genPD I did find that it appears to be a scheme for collating multiple runtime_pm information to establish when all devices in a domain are idle to enable the powering off of the domain. Now if EMT and CPU are registered in the domain - does this not mean that ETM is going to act as a wakelock and prevent CPUIdle?
(there is a patch http://lkml.iu.edu/hypermail/linux/kernel/1806.2/04025.html from last June that appears to add CPUs to genPD but I cannot find any evidence of how it has progressed.)
I also noted that at present the PMU register driver attached to the cores uses notifiers to save / restore when the CPU enters / exits idle. It seems that it may be prudent to do the same for CPU bound coresight devices such as ETM and CTI., even if they are later attached to genPD. In this way we ensure that Core bound CoreSight closes down when the CPU would like to idle.
Regards
Mike
-- Mike Leach Principal Engineer, ARM Ltd. Manchester Design Centre. UK
-
Al Grant -
Leo Yan -
Mathieu Poirier -
Mike Leach