A quick poll of the ARM platforms that implement CPU Hotplug support shows that every platform treats CPU 0 as a special case that cannot be hotplugged. In fact every platform has identical code for platform_cpu_die which returns -EPERM in the case of CPU 0.
The user-facing sysfs interfaces should reflect this by not populating an 'online' entry for CPU 0 at all. This better reflects reality by making it clear to users that CPU 0 cannot be hotplugged.
This patch prevents CPU 0 from being marked as hotpluggable on all ARM platforms during CPU registration. This in turn prevents the creation of an 'online' sysfs interface for that CPU.
Signed-off-by: Mike Turquette mturquette@ti.com --- arch/arm/kernel/setup.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-)
diff --git a/arch/arm/kernel/setup.c b/arch/arm/kernel/setup.c index ed11fb0..a5fc969 100644 --- a/arch/arm/kernel/setup.c +++ b/arch/arm/kernel/setup.c @@ -940,7 +940,8 @@ static int __init topology_init(void)
for_each_possible_cpu(cpu) { struct cpuinfo_arm *cpuinfo = &per_cpu(cpu_data, cpu); - cpuinfo->cpu.hotpluggable = 1; + if (cpu) + cpuinfo->cpu.hotpluggable = 1; register_cpu(&cpuinfo->cpu, cpu); }
On 07/20/2011 06:32 PM, Mike Turquette wrote:
A quick poll of the ARM platforms that implement CPU Hotplug support shows that every platform treats CPU 0 as a special case that cannot be hotplugged. In fact every platform has identical code for platform_cpu_die which returns -EPERM in the case of CPU 0.
The user-facing sysfs interfaces should reflect this by not populating an 'online' entry for CPU 0 at all. This better reflects reality by making it clear to users that CPU 0 cannot be hotplugged.
This patch prevents CPU 0 from being marked as hotpluggable on all ARM platforms during CPU registration. This in turn prevents the creation of an 'online' sysfs interface for that CPU.
Unless there is a kernel limitation why CPU0 can't be hot unplugged, then this should remain a platform decision. This may be another case of everybody just copying other platforms' code, not a platform limitation.
Rob
Signed-off-by: Mike Turquette mturquette@ti.com
arch/arm/kernel/setup.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-)
diff --git a/arch/arm/kernel/setup.c b/arch/arm/kernel/setup.c index ed11fb0..a5fc969 100644 --- a/arch/arm/kernel/setup.c +++ b/arch/arm/kernel/setup.c @@ -940,7 +940,8 @@ static int __init topology_init(void) for_each_possible_cpu(cpu) { struct cpuinfo_arm *cpuinfo = &per_cpu(cpu_data, cpu);
cpuinfo->cpu.hotpluggable = 1;
if (cpu)cpuinfo->cpu.hotpluggable = 1;
On 7/21/2011 8:32 AM, Rob Herring wrote:
On 07/20/2011 06:32 PM, Mike Turquette wrote:
A quick poll of the ARM platforms that implement CPU Hotplug support shows that every platform treats CPU 0 as a special case that cannot be hotplugged. In fact every platform has identical code for platform_cpu_die which returns -EPERM in the case of CPU 0.
The user-facing sysfs interfaces should reflect this by not populating an 'online' entry for CPU 0 at all. This better reflects reality by making it clear to users that CPU 0 cannot be hotplugged.
This patch prevents CPU 0 from being marked as hotpluggable on all ARM platforms during CPU registration. This in turn prevents the creation of an 'online' sysfs interface for that CPU.
Unless there is a kernel limitation why CPU0 can't be hot unplugged, then this should remain a platform decision. This may be another case of everybody just copying other platforms' code, not a platform limitation.
Just talking on behalf of OMAP, we can't offline CPU0 and limitation would remain in future OMAPs too.
Regards Santosh
On Thu, Jul 21, 2011 at 11:03:04AM +0530, Santosh Shilimkar wrote:
Just talking on behalf of OMAP, we can't offline CPU0 and limitation would remain in future OMAPs too.
Apart from the broken IRQ migration, and the way CPU0 immediately reawakes if it is offlined on OMAP4 (even when IRQs are migrated off CPU0) because omap_read_auxcoreboot0() returns 0, is there any other reason?
With fixed IRQ migration and forcing CPU0 into an infinite WFI loop, I can offline CPU0 and still have a running system.
On 7/21/2011 7:00 PM, Russell King - ARM Linux wrote:
On Thu, Jul 21, 2011 at 11:03:04AM +0530, Santosh Shilimkar wrote:
Just talking on behalf of OMAP, we can't offline CPU0 and limitation would remain in future OMAPs too.
Apart from the broken IRQ migration, and the way CPU0 immediately reawakes if it is offlined on OMAP4 (even when IRQs are migrated off CPU0) because omap_read_auxcoreboot0() returns 0, is there any other reason?
With fixed IRQ migration and forcing CPU0 into an infinite WFI loop, I can offline CPU0 and still have a running system.
The secure software runs only on CPU0 and that's the biggest limitation. Other issues like hand-shake as you pointed out, power sequencing etc can be worked around.
Regards Santosh
From: linux-arm-kernel-bounces@lists.infradead.org [mailto:linux-arm- kernel-bounces@lists.infradead.org] On Behalf Of Shilimkar, Santosh
With fixed IRQ migration and forcing CPU0 into an infinite WFI loop, I can offline CPU0 and still have a running system.
The secure software runs only on CPU0 and that's the biggest limitation. Other issues like hand-shake as you pointed out, power sequencing etc can be worked around.
As you know well some of the secure APIs work on CPU1 and others do not.
I notice in other thread Russell made assumption about CPU1 being able to use all because it could run some. This is not the case.
I'll re-add because of the many power states omap4 adds atop of standard 3 arm states there exists a lot of states and not all state transitions are valid. This will increase complexity of practical usage.
On 7/22/2011 6:15 PM, Woodruff, Richard wrote:
From: linux-arm-kernel-bounces@lists.infradead.org [mailto:linux-arm- kernel-bounces@lists.infradead.org] On Behalf Of Shilimkar, Santosh
With fixed IRQ migration and forcing CPU0 into an infinite WFI loop, I can offline CPU0 and still have a running system.
The secure software runs only on CPU0 and that's the biggest limitation. Other issues like hand-shake as you pointed out, power sequencing etc can be worked around.
As you know well some of the secure APIs work on CPU1 and others do not.
I notice in other thread Russell made assumption about CPU1 being able to use all because it could run some. This is not the case.
I clarified that on the other thread.
Regards Santosh
On Fri, Jul 22, 2011 at 5:53 AM, Santosh Shilimkar santosh.shilimkar@ti.com wrote:
On 7/22/2011 6:15 PM, Woodruff, Richard wrote:
From: linux-arm-kernel-bounces@lists.infradead.org [mailto:linux-arm- kernel-bounces@lists.infradead.org] On Behalf Of Shilimkar, Santosh
With fixed IRQ migration and forcing CPU0 into an infinite WFI loop, I can offline CPU0 and still have a running system.
The secure software runs only on CPU0 and that's the biggest limitation. Other issues like hand-shake as you pointed out, power sequencing etc can be worked around.
As you know well some of the secure APIs work on CPU1 and others do not.
I notice in other thread Russell made assumption about CPU1 being able to use all because it could run some. This is not the case.
I clarified that on the other thread.
I've asked a few other ARM folks (out of band) to weigh in on this thread to determine if their platform has similar limitations as OMAP. Unfortunately no one else has responded.
Still the limitation for OMAP remains. Earlier in this thread I provided an alternative to blacklisting CPU0 for all ARM platforms by instead using a config option, but it received no comments. What is the best way to move forward?
Thanks, Mike
Regards Santosh
On Wed, Jul 20, 2011 at 04:32:25PM -0700, Mike Turquette wrote:
A quick poll of the ARM platforms that implement CPU Hotplug support shows that every platform treats CPU 0 as a special case that cannot be hotplugged. In fact every platform has identical code for platform_cpu_die which returns -EPERM in the case of CPU 0.
Are you sure that's just not because everyone copied what Realview has been doing (highly likely)?
I suspect that there's no reason that CPU0 can't be taken down, especially on those platforms which don't take the CPU fully offline but just put it into a WFI loop.
Those which restart the CPUs through the boot loader probably detect CPU0 as the boot CPU, so they probably can't take CPU0 down.
From: linux-arm-kernel-bounces@lists.infradead.org [mailto:linux-arm- kernel-bounces@lists.infradead.org] On Behalf Of Russell King - ARM Linux
On Wed, Jul 20, 2011 at 04:32:25PM -0700, Mike Turquette wrote:
A quick poll of the ARM platforms that implement CPU Hotplug support shows that every platform treats CPU 0 as a special case that cannot be hotplugged. In fact every platform has identical code for platform_cpu_die which returns -EPERM in the case of CPU 0.
Are you sure that's just not because everyone copied what Realview has been doing (highly likely)?
I suspect that there's no reason that CPU0 can't be taken down, especially on those platforms which don't take the CPU fully offline but just put it into a WFI loop.
Those which restart the CPUs through the boot loader probably detect CPU0 as the boot CPU, so they probably can't take CPU0 down.
This was a hot topic a couple years back for hw folks...
On 443x when you go to ARM-MPCore power states other than normal (dormant, powered-off) there are hardware and ti-trustzone-monitor constraints which dictate the CPU power state transition matrix (cpu1 can't make all calls cpu0 can and some hardware combinations were not validated).
For MPU-domain (mpcore-cluster + l2cache ++) and its sub-domains (cpu0, cpu1, ...) there exists ability to set multiple states which wfi is a gate into (on, inactive, retention, partial-off, off). A big matrix of possible states results. Things were coded and system tested for only a subset of these. The ARM code did line up with these constraints.
Maybe an offline to simple wfi loop is ok, but other lower states could not be entered from that state.
Some folks were talking about relaxing cpu0/cpu1 constraints moving forward. Doing this could result in many more states-combinations to be validated at hardware level. Do you think Linux code would be simplified/enhanced such that it is worth the extra hardware validation effort?
On Thu, Jul 21, 2011 at 12:45 AM, Russell King - ARM Linux linux@arm.linux.org.uk wrote:
On Wed, Jul 20, 2011 at 04:32:25PM -0700, Mike Turquette wrote:
A quick poll of the ARM platforms that implement CPU Hotplug support shows that every platform treats CPU 0 as a special case that cannot be hotplugged. In fact every platform has identical code for platform_cpu_die which returns -EPERM in the case of CPU 0.
Are you sure that's just not because everyone copied what Realview has been doing (highly likely)?
Copy/paste is always likely. Would be nice for other platform folks to weigh in on this.
I suspect that there's no reason that CPU0 can't be taken down, especially on those platforms which don't take the CPU fully offline but just put it into a WFI loop.
Those which restart the CPUs through the boot loader probably detect CPU0 as the boot CPU, so they probably can't take CPU0 down.
OMAP does seem to have this limitation and Santosh/Richard can provide much better details than me in the parallel thread. Again, would be nice to hear if other platforms have similar limitations from their stakeholders.
The idea here is to not mark a CPU hotpluggable if it cannot be hotplugged. If the limitations turn out to be legitimate for some platforms but not others, what's the best way to handle it? Either move the topology initialization to platform code or allow platforms to set some config option. Patches for the latter are below, but I think the current discussion on whether or not other platforms can actually hotplug CPU0 should run its course before considering below patches too seriously.
From b734cedf23a9f8366faa1a961d87cb4bc221291e Mon Sep 17 00:00:00 2001
From: Mike Turquette mturquette@ti.com Date: Mon, 18 Jul 2011 17:33:22 -0700 Subject: [PATCH 1/2] ARM: conditionally allow master CPU hotplugging
Currently all CPUs are marked as hotpluggable in topology_init(). This is not true for all ARM SMP platforms including OMAP4. In the OMAP4 case, CPU0 is considered as privileged due to TrustZone software and other considerations.
This patch introduces a new config option, HOTPLUG_CPU_MASTER, which prevents the first CPU in possible_cpu_mask from being marked hotpluggable if set.
Signed-off-by: Mike Turquette mturquette@ti.com --- arch/arm/Kconfig | 8 ++++++++ arch/arm/kernel/setup.c | 5 +++++ 2 files changed, 13 insertions(+), 0 deletions(-)
diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig index 9adc278..a6e34f5 100644 --- a/arch/arm/Kconfig +++ b/arch/arm/Kconfig @@ -1393,6 +1393,14 @@ config HOTPLUG_CPU Say Y here to experiment with turning CPUs off and on. CPUs can be controlled through /sys/devices/system/cpu.
+config HOTPLUG_CPU_MASTER + bool "Prevent first CPU (master) from hotplugging" + depends on HOTPLUG_CPU + help + Say Y here if your platform treats the first CPU as a special + "master" CPU and cannot allow it to hotplug. This prevents the + "online" entry in sysfs for that CPU. + config LOCAL_TIMERS bool "Use local timer interrupts" depends on SMP diff --git a/arch/arm/kernel/setup.c b/arch/arm/kernel/setup.c index ed11fb0..3785420 100644 --- a/arch/arm/kernel/setup.c +++ b/arch/arm/kernel/setup.c @@ -940,7 +940,12 @@ static int __init topology_init(void)
for_each_possible_cpu(cpu) { struct cpuinfo_arm *cpuinfo = &per_cpu(cpu_data, cpu); +#ifdef CONFIG_HOTPLUG_CPU_MASTER + if (cpu) + cpuinfo->cpu.hotpluggable = 1; +#else cpuinfo->cpu.hotpluggable = 1; +#endif register_cpu(&cpuinfo->cpu, cpu); }
-
Mike Turquette -
Rob Herring -
Russell King - ARM Linux -
Santosh Shilimkar -
Turquette, Mike
-
Woodruff, Richard