On Thu, 5 Sep 2013, Catalin Marinas wrote:
I'll let the firmware guys discuss what is and is not upgradable at some point in the future (when the generic firmware will be available). This is not related just to PSCI but to other things like secure-only errata bits.
In the meantime I'll try to focus on the Linux interaction with the secure firmware (and PSCI). Also cc'ing Mark Rutland since he's working on CPU hotplug for arm64.
On Wed, Sep 04, 2013 at 02:48:48PM +0100, Nicolas Pitre wrote:
On Wed, 4 Sep 2013, Catalin Marinas wrote:
On 3 Sep 2013, at 19:53, Nicolas Pitre nicolas.pitre@linaro.org wrote:
On Tue, 3 Sep 2013, Catalin Marinas wrote:
On Tue, Sep 03, 2013 at 05:16:17AM +0100, Nicolas Pitre wrote
> For example, MCPM provides callbacks into the platform > code when a CPU goes down to disable coherency, flush caches etc. and > this code must call back into the MCPM to complete the CPU tear-down. If > you want such thing, you need a different PSCI specification.
Hmmm... The above statement makes no sense to me. Sorry I must have missed something.
OK, let me clarify. I took the dcscb.c example for CPU going down (please correct me if I got it wrong):
mcpm_cpu_power_down() dcscb_power_down() flush_cache_all() - both secure and non-secure set_auxcr() - secure-only cci_disable_port_by_cpu() - secure-only? (__mcpm_outbound_leave_critical()) __mcpm_cpu_down() wfi()
So the dcscb back-end calls back into MCPM to update the state machine. If you are to run Linux in non-secure mode, set_auxcr() and CCI would need secure access. Cache flushing also needs to affect the secure cachelines and disable the caches on the secure side. So from the flush_cache_all() point, you need to get into EL3. But MCPM requires (as I see in the dcscb back-end) a return from such EL3 code to call __mcpm_cpu_down() and do WFI on the non-secure side. This is incompatible with the (current) PSCI specification.
The MCPM backend doesn't _need_ to call __mcpm_cpu_down() and friends. Those are helpers for when there is no firmware and proper synchronization needs to be done between different cores.
The reason people currently ask for MCPM is exactly this synchronisation which they don't want to do in the firmware.
And this is a hell of a good reason. I'm scared to death by the prospect of seeing this kind of algorithmic complexity shoved into closed firmware.
OK, so if I understand you correctly, you don't want to see PSCI firmware in use at all or at least not in it's current form (which *you* also reviewed) because you think it is too complex to be bug-free or impractical to upgrade.
Well... PSCI as an *interface* definition is sound. I reviewed this interface definition which is great for virtualized systems as well as simple hardware.
What I didn't fully grasp at the time is the implied _complexity_ when doing power management on a cluster based system. Partly because I had not started the work on MCPM at the time and I seriously under-estimated the effort needed to 1) get it done, and 2) get it right.
Now PSCI is very abstract which is both a good and a bad thing. It is good because it makes for a very nice and clean interface that hides all the platform particularities. And I do understand why you as the maintainer of a new architecture could be fond of it as it keeps the kernel clean and free of machine specific quirks.
However it implies that a lot of the complexity currently handled by MCPM in the kernel has to move behind the PSCI interface i.e. in the secure firmware. And *that* is bad.
For some secure firmware to actually be secure, it has to be simple and obvious. Security always gets compromised to some degree with added complexity.
I respect your opinion but do you have a more concrete proposal? The options so far:
(current status) Don't use PSCI firmware, let Linux handle all the CPU power management (possibly under the MCPM framework). If not all power-related actions can be done at the non-secure level, just implement non-standard SMC calls as needed. If these are changed (because in time vendor may have other security needs), add them to the driver and hope they have a way to detect or just not upstream the #ifdef'ed code.
New standard firmware interface, simpler and less error-prone. Handle most power management in Linux (with an MCPM-like state machine) and have guaranteed race-free calls to the firmware. In the process, also convince the secure OS guys that Linux is part of their trusted environment (I personally trust Linux more than the trusted OS ;) but this doesn't hold any water for certification purposes). Basically if you can disable coherency from the non-secure OS (e.g. CCI or just powering down a CPU without the secure OS having the chance to flush its caches), the only workaround for the secure OS would be to run UP (which is probably the case now) or flush caches at every return to the non-secure world.
Very similar to 2, with PSCI firmware interface but without the requirements to do cluster state coordination in firmware (with some semantic changes to the affinity arguments). Linux handles the state coordination (MCPM state machine) but PSCI firmware does the necessary flushing, coherency disabling based on the specified affinity level (it doesn't question that because it does not track the "last man"). Slightly better security model than 2 as it can flush the secure OS caches but I'm not entirely sure PSCI can avoid a state machine and whether this has other security implications.
MCPM state machine on top of full PSCI. Here I don't see the point of tracking cluster/last-man state in Linux if PSCI does it already. If PSCI got it wrong (broken coherency, deadlocks), MCPM cannot really solve it. Also, are there additional races by having two separate state machines for the same thing (I can't think of any now)?
Full PSCI with a light wrapper (smp_operations) allowing SoC to hook additional code (only if needed for parts which aren't handled by firmware). This is similar to the mcpm_platform_ops registration but *without* the MCPM state machine and without the separate cpu/cluster arguments (just linearise this space for flexibility). Other key point is that the re-entry address is driven via PSCI rather than mcpm_entry_vectors. Platform ops registration is optional, just available for flexibility (which could also mean that it is not the platform ops driving the final PSCI call, different from the MCPM framework). This approach does not enforce a secure model, it's up to the SoC vendor to allow/prevent non-secure access to power controller, CCI etc. But it still mandates common kernel entry/exit path via PSCI.
Full PSCI with generic CPU hotplug and cpuidle drivers. I won't list the pros/cons, that's what this thread is mainly about.
Any other options?
My goal is for 6 but 5 could be a more practical/flexible approach.
Both those options (5 and 6) imply the state machine is in the firmware. And that's where complexity lies. So that wouldn't be my choice at all.
Option 4 is rather useless. In fact the MCPM backend for PSCI I showed you doesn't exercise the MCPM state machine. And the firmware still implements a state machine.
Differences between 2,3,4 are a bit fuzzy to me. I understand the issue with having a secure OS that needs to protect itself from the nasty Linux world. However, if I understand the model right, the secure OS is there to provide special services to the non-secure OS and not the reverse. Therefore the secure OS should simply pack and hide its things when told to do so, right?
Of course option 1 is the most flexible in terms of implementation efficiency, but it has drawbacks as well.
As I said in a previous post, I'm not against MCPM as such but against the back-ends which will eventually get non-standard secure calls.
Maybe the secure call standardization should focus on providing less abstract and more low-level interfaces then. Right now, the PSCI definition implies that everything is performed behind the interface.
Sorry but you are late for this call. PSCI has been discussed for nearly two years and you have been involved.
And two years later I've gained enough experience to state that this might not be such a good idea for all cases after all. Sometimes you may not realize all the ramifications of a concept or design until you do actual work on it.
And this is not my call to make either. System vendors will choose their own poison for themselves. Between risk inducing complexity in secure firmware and non-standard low-level machine specific L3 calls I don't think there is much to rejoice about.
But my point is that there is legitimacy in both models and right now I don't see a single "true" solution. Hence my claim that you should not insist on PSCI or nothing.
Nicolas