On Tue, Sep 03, 2013 at 05:16:17AM +0100, Nicolas Pitre wrote:
On Mon, 2 Sep 2013, Catalin Marinas wrote:
(sorry if you get a duplicate email, the SMTP server I was using keeps reporting delays, so I'm resending with different settings)
This is the first time I see your reply.
I should have said "sorry for the future duplicate email that you may receive" (we try to work around IT here but sometimes they break our back doors ;)).
On Sat, Aug 31, 2013 at 03:19:46AM +0100, Nicolas Pitre wrote:
On Fri, 30 Aug 2013, Catalin Marinas wrote:
My position for the arm64 kernel support is to use the PSCI and implement the cluster power synchronisation in the firmware. IOW, no MCPM in the arm64 kernel :(. To help with this, ARM is going to provide a generic firmware implementation that SoC vendors can expand for their needs.
I am open for discussing a common API that could be shared between MCPM-based code and the PSCI one. But I'm definitely not opting for a light-weight PSCI back-end to a heavy-weight MCPM implementation.
Also note that IKS won't be supported on arm64.
I find the above statements a bit rigid.
While I think the reasoning behind PSCI is sound, I suspect some people will elect not to add it to their system. Either because the hardware doesn't support all the necessary priviledge levels, or simply because they prefer the easiest solution in terms of maintenance and upgradability which means making the kernel in charge. And that may imply MCPM. I know that ARM would like to see PSCI be adopted everywhere but I doubt it'll be easy.
I agree PSCI is not always possible because of technical reasons (missing certain exception levels), so let me refine the above statement: if EL3 mode is available on a CPU implementation, PSCI must be supported (which I think is the case for Marvell).
I may agree to that.
It indeed sounds rigid but what's the point of trying to standardise something if you don't enforce it? People usually try to find the easiest path for them which may not necessarily be the cleanest longer term.
I'll say this only once, and then I'll shut up on this very issue.
I actually enjoy this discussion ;)
<<< Beginning of my editorial on firmware >>>
...
So no, I don't have this faith in firmware like you do. And I've discussed this with you in the past as well, and my mind on this issue has not changed. In particular, when you give a presentation on ARMv8 and your answer to a question from the audience is: "firmware will take care of this" I may only think you'll regret this answer one day.
Well, I don't trust firmware either and I've seen it causing hard to debug issues in the past. But actually a better statement would be that I don't trust any software (not even Linux) unless I can see the source. The big advantage Linux has is that it requires (well, unless you are NVidia ;)) opening the code and more people look at it.
My hope for the generic firmware is that it will turn into a proper open-source project (with a friendly license). This would make it different from old BIOS implementations.
Of course, bugs can happen and the firmware is harder to update but really not impossible.
As a past example, how many SoC maintainers were willing to convert their code to DT if there wasn't a clear statement from arm-soc guys mandating this?
This is a wrong comparison. DT is more or less a more structured way to describe hardware. Just like ACPI. The kernel is not *forced* to follow DT information if that turns out to be wrong. And again experience has shown that this does happen.
The comparison is not meant for DT vs PSCI but an example that even though DT has benefits and less risks, people didn't rush into adopting it unless it was mandated for new platforms. For arm64 we try not to get SoC code under arch/arm64/ (mach-virt like approach) but I still get people asking in private about copying code into arch/arm64/mach-* directories for the same easy path reasons.
But let's get back to firmware. Delegating power management to firmware is a completely different type of contract. In that case you're moving a critical functionality with potentially significant algorithmic complexity (as evidenced by the MCPM development effort) out of the kernel's control. The more functionality and complexity you delegate to the firmware, the greater the risk you'll end up with a crippled system the kernel will have to cope with eventually. Because this will happen at some point there is no doubt about that.
I agree with your arguments that Linux is more flexible and easily upgradable. However, the point I want to emphasize is that unless Linux is playing all the roles of firmware/secure/non-secure code, you must have firmware and calls to it from the non-secure OS. On ARMv8, EL3 is the first mode CPU is running out of reset and it needs to get back to this mode for power-related (e.g. coherency) settings. Whether we (Linux people) like it or not, that's the reality.
How much you leave on the secure or the non-secure side goes through a great deal of discussions/reviews both with the general purpose OS and secure OS people, hence the creation of PSCI (really, it's not just about standardising the SoC support on arm64 Linux).
MCPM is well suited if you don't have a trusted execution environment but once you do, the secure OS could have the same needs as the non-secure one in terms of CPU availability. Would it trust the non-secure OS to handle the cluster shutdown, coherency? I doubt it.
Think for a minute when the kernel used to call the APM BIOS for power management in the old days, and then the kernel was enhanced to be preemptive. APM was not re-entrant and therefore conceptually incompatible with this kernel context at the time. Obviously things where even more difficult when SMP showed up. And just ask the X86 maintainers what they think about the firmware induced NMIs *today*.
So, is e.g. PSCI compatible with Linux-RT now?
I think it is as much compatible as MCPM. What you need for RT is bounded time for going down and coming back from a lower power state. On ARM and other architectures this involves firmware (at least after reset), so not related to PSCI. Those time bounds could be provided by the SoC vendor for their firmware (whether you use MCPM or PSCI). If you don't like them, you can simply use WFI, PSCI does not mandate a call for such idle state (not sure whether APM required this in the past), only deeper sleep states.
Is there another kernel execution model innovation coming up for which the firmware will be an obstacle on ARM?
Firmwares evolve in time but as I said, even if we would like to, we can't eliminate them.
I'm just extremely worried about the simple presence of firmware and the need to call into it for complex operations such as intelligent power management when the kernel would be a much better location to perform such operations. The kernel is also the only place where those things may be improved over time, even for free from third parties, , whereas the firmware is going to be locked down with no possible evolution beyond its shipping state.
For example, after working on MCPM for a while, I do have ideas for additional performance and power usage optimization. But if that functionality is getting burned into firmware controlled by secure world then there is just no hope for _me_ to optimize things any further.
I don't dispute the above but I don't have a better solution that would accommodate secure/non-secure worlds. With proper education, SoC vendors can learn to allow upgradable (parts of) firmware. But if you have a better proposal and can get all the parts (including secure OS people) to agree, I'm open to it.
Not that I can do anything about it anyway. But I had to vent my discomfort about anything firmware-like. Yet I was the one who suggested to Will Deacon and Marc Zyngier they should use PSCI to control KVM instances instead of their ad hoc interface. However, if someone has a legitimate case for _not_ using firmware calls and use machine specific extensions in the kernel then I'll support them.
Legitimate calls yes (like no EL3). But if people would like MCPM just for temporary kernel bring-up, I don't agree with. They start with an simple MCPM back-end and later realise that they need Linux to run in non-secure mode (well, like getting EL2 or just because they get a secure OS), so they modify the MCPM back-end for (non-standard) secure calls. Some time later they decide they need a secure OS (e.g. mobile payments) and yet again they modify their back-end, possibly ignoring MCPM because the secure OS asks them to handle such state machine at EL3. In Linux we end up carrying all those intermediate implementations.
For example, MCPM provides callbacks into the platform code when a CPU goes down to disable coherency, flush caches etc. and this code must call back into the MCPM to complete the CPU tear-down. If you want such thing, you need a different PSCI specification.
Hmmm... The above statement makes no sense to me. Sorry I must have missed something.
OK, let me clarify. I took the dcscb.c example for CPU going down (please correct me if I got it wrong):
mcpm_cpu_power_down() dcscb_power_down() flush_cache_all() - both secure and non-secure set_auxcr() - secure-only cci_disable_port_by_cpu() - secure-only? (__mcpm_outbound_leave_critical()) __mcpm_cpu_down() wfi()
So the dcscb back-end calls back into MCPM to update the state machine. If you are to run Linux in non-secure mode, set_auxcr() and CCI would need secure access. Cache flushing also needs to affect the secure cachelines and disable the caches on the secure side. So from the flush_cache_all() point, you need to get into EL3. But MCPM requires (as I see in the dcscb back-end) a return from such EL3 code to call __mcpm_cpu_down() and do WFI on the non-secure side. This is incompatible with the (current) PSCI specification.