Ok Heinrich, thanks for the clarification. I also read through this thread again and understand the intention you would like to have some sentences in the UEFI spec.
I realize that we are trying to put sentences in the RISC-V calling convention in UEFI spec to support the specific UEFI firmware implementation of using gp/tp. Uboot UEFI firmware uses tp in HART to record the HART ID and uses gp in the boot-HART to maintain the global data structure of uboot. The reason I can think of the reason to use tp/gp is because S-Mode UEFI firmware (or S-mode payload?) is not able to get mhartid, so uboot stores HART ID in tp. Because there is nowhere to maintain the global data structure for uboot and the mscratch is occupied by opensbi, thus uboot uses gp to maintain the pointer to global context. Because both tp and gp are non-privileged registers so uboot (or payload?) can get the information from those registers easily, right? (But is this a security hole that any software can hack tp/gp? However, this is the different topic, just ignore this) Did we spec out above behavior in uboot RISC-V spec or the implementation design guide at somewhere? I am not quite familiar with uboot area though.
Another problem comes to my mind is even though we save gp/tp upon the entry of ExitBootService (or other UEFI APIs) and replaced with the uboot ones, what if the interrupt/exception happens and delegated to the payload event handler during ExitBootService (or other UEFI APIs)? The ownership is messed up at this point, right?
In edk2 RISC-V port, we use "firmware_context" in sbi_platforms which provided opensbi through mscratch to maintain the firmware specific context. With this, firmware can maintain its own data structure without bothering any other industrial specs. Because S-Mode firmware can't access to mhartid and the sbi-scratch which is stored in mscratch, so we have Edk2OpensbiLib that registers the firmware specific sbi extension functions using "Firmware Code Base Specific SBI Extension Space, Extension Ids 0x0A000000 through 0x0AFFFFFF" defined in sbi spec. That is a separate spec for RISC-V edk2 firmware sbi extension function (This reminds me I forget to document it). Then edk2 can get the necessary information using edk2 extension sbi call.
You would agree with that we don’t have to mention the gp/tp usage in the calling convention just because of the uboot implementation. And we don’t have mention any connection/relationship between gp/tp usage and the specific UEFI firmware implementation. The sentence must be also generic to other UEFI firmware solutions such as edk2 which doesn't care about destroying gp/tp (so far).
In the RISC-V assembly programmer's handbook in RISC-V spec, gp/tp are not given either the caller or callee to save the value, so we can spec out something in UEFI spec based on this calling convention.
The sentence which I consider reasonable is , "In view of gp and tp registers are not assigned the specific owner to save and restore their values in the calling convention (refer to "RISC-V assembly programmer's handbook" section in RISC-V Unprivileged ISA specification), UEFI firmware must not having assumption of owning the write access to these register at any circumstances such as in EFI Boot service, EFI Runtime service, EFI Management Mode service and in any UEFI firmware interfaces which may invoked by the external firmware payload, or altering the values of gp/tp registers results in the interference in the asynchronous callback to the external firmware payload . Whether and how to preserve gp and tp during UEFI firmware environment is UEFI firmware implementation-specific. RISC-V platform firmware integrator should refer to the corresponding design specification of the particular UEFI firmware solution."
I believe this initial sentence still needs some revises, please just modify above paragraph if you don't like it. We can keep discussing this based on above paragraph.
Thanks
-----Original Message----- From: Heinrich Schuchardt [mailto:xypron.glpk@gmx.de] Sent: Thursday, September 24, 2020 2:20 AM To: Chang, Abner (HPS SW/FW Technologist) abner.chang@hpe.com; boot-architecture@lists.linaro.org; Atish Patra atishp@atishpatra.org Cc: Rick Chen rick@andestech.com; Atish Patra atish.patra@wdc.com; Grant Likely grant.likely@arm.com; Ard Biesheuvel ardb@kernel.org Subject: Re: EBBR: RISC-V handoff to OS
On 9/23/20 5:01 PM, Chang, Abner (HPS SW/FW Technologist) wrote:
-----Original Message----- From: Chang, Abner (HPS SW/FW Technologist) Sent: Wednesday, September 23, 2020 10:55 PM To: 'Heinrich Schuchardt' xypron.glpk@gmx.de; boot- architecture@lists.linaro.org; Atish Patra atishp@atishpatra.org Cc: Rick Chen rick@andestech.com; Atish Patra atish.patra@wdc.com; Grant Likely grant.likely@arm.com; Ard Biesheuvel ardb@kernel.org Subject: RE: EBBR: RISC-V handoff to OS
-----Original Message----- From: Heinrich Schuchardt [mailto:xypron.glpk@gmx.de] Sent: Wednesday, September 23, 2020 3:42 PM To: Chang, Abner (HPS SW/FW Technologist) abner.chang@hpe.com; boot-architecture@lists.linaro.org; Atish Patra atishp@atishpatra.org Cc: Rick Chen rick@andestech.com; Atish Patra atish.patra@wdc.com; Grant Likely grant.likely@arm.com; Ard Biesheuvel ardb@kernel.org Subject: Re: EBBR: RISC-V handoff to OS
On 23.09.20 08:51, Chang, Abner (HPS SW/FW Technologist) wrote:
-----Original Message----- From: Heinrich Schuchardt [mailto:xypron.glpk@gmx.de] Sent: Wednesday, September 23, 2020 2:18 PM To: Chang, Abner (HPS SW/FW Technologist)
boot-architecture@lists.linaro.org; Atish Patra atishp@atishpatra.org Cc: Rick Chen rick@andestech.com; Atish Patra atish.patra@wdc.com; Grant Likely grant.likely@arm.com; Ard Biesheuvel ardb@kernel.org Subject: Re: EBBR: RISC-V handoff to OS
On 9/23/20 7:24 AM, Chang, Abner (HPS SW/FW Technologist) wrote: > > >> -----Original Message----- >> From: Heinrich Schuchardt [mailto:xypron.glpk@gmx.de] >> Sent: Tuesday, September 22, 2020 5:30 PM >> To: boot-architecture@lists.linaro.org; Chang, Abner (HPS SW/FW >> Technologist) abner.chang@hpe.com; Atish Patra >> atishp@atishpatra.org >> Cc: boot-architecture@lists.linaro.org; Rick Chen >> rick@andestech.com; Atish Patra atish.patra@wdc.com; Grant >> Likely grant.likely@arm.com; Ard Biesheuvel ardb@kernel.org >> Subject: RE: EBBR: RISC-V handoff to OS >> >> Am 22. September 2020 08:30:57 MESZ schrieb "Chang, Abner (HPS SW/FW >> Technologist)" abner.chang@hpe.com: >>> >>> >>>> -----Original Message----- >>>> From: Atish Patra [mailto:atishp@atishpatra.org] >>>> Sent: Tuesday, September 22, 2020 2:08 PM >>>> To: Chang, Abner (HPS SW/FW Technologist)
>>>> Cc: Heinrich Schuchardt xypron.glpk@gmx.de; Atish Patra >>>> atish.patra@wdc.com; boot-architecture@lists.linaro.org; >>>> Grant >>> Likely >>>> grant.likely@arm.com; Ard Biesheuvel ardb@kernel.org;
Rick
Chen >>>> rick@andestech.com >>>> Subject: Re: EBBR: RISC-V handoff to OS >>>> >>>> On Mon, Sep 21, 2020 at 6:11 PM Chang, Abner (HPS SW/FW >>>> Technologist) abner.chang@hpe.com wrote: >>>>> >>>>> >>>>> >>>>>> -----Original Message----- >>>>>> From: Atish Patra [mailto:atishp@atishpatra.org] >>>>>> Sent: Tuesday, September 22, 2020 2:28 AM >>>>>> To: Heinrich Schuchardt xypron.glpk@gmx.de >>>>>> Cc: Atish Patra atish.patra@wdc.com; >>>>>> boot-architecture@lists.linaro.org; >>>>>> Grant Likely grant.likely@arm.com; Ard Biesheuvel >>>>>> ardb@kernel.org; Rick Chen rick@andestech.com; Chang, Abner >>>> (HPS >>>>>> SW/FW Technologist) abner.chang@hpe.com >>>>>> Subject: Re: EBBR: RISC-V handoff to OS >>>>>> >>>>>> On Mon, Sep 21, 2020 at 9:23 AM Heinrich Schuchardt >>>>>> xypron.glpk@gmx.de wrote: >>>>>>> >>>>>>> Hello Atish, >>>>>>> >>>>>>> the UEFI spec has this sentence: >>>>>>> >>>>>>> "When UEFI firmware handoff control to OS, the RISC-V is >>> operated >>>>>>> in machine-mode privilege." (M-mode is the equivalent to >>>>>>> EL3 in >>> ARM). >>>>> Yes, this is a pretty old section in UEFI spec even before >>>>> OpenSBI >>> as I can >>>> remember and haven't been updated to sync with latest status >>>> RISC-V >>> works. >>>>> >>>>>>> >>>>>>> This does not make any sense to me when using a secure >>> execution >>>>>>> environement (SEE) like OpenSBI. >>>>>>> >>>>>>> The hand-off should occur in S-Mode if the CPU supports it >>>>>>> and only in M-Mode when the CPU only supports M-mode. >>>>>>> >>>>>> +Abner >>>>>> >>>>>> Yes. Abner has already submitted an ECR for this & few other >>> RISC-V >>>>>> related changes to UEFI spec. I am not sure about the >>>>>> current >>> status >>>> though. >>>>>> >>>>>> @Abner: Do you know the latest status ? >>>>> Yes, the ECR was submitted to USWG for review, however the meeting >>>>> canceled often recently and the process goes slow. I will >>>>> keep following up >>>>> >>>>>> Maybe you also attach the latest ECR here for a broader
review.
>>>>> I may not allowed to public ECR here unless all people in the >>> mailing >>>>> list are the members of UEFI org… but the sentence we revised >>>>> in >>> ECR >>>>> is as below, >>>>> >>>>> "When UEFI firmware handoff control to Supervisor mode OS, >>>>> RISC-V >>> boot >>>> hart must be operated in Supervisor mode, and the memory addressing >>>> must be operated in Bare mode which is no memory address >>>> translation >>> or >>>> protection through the virtual page table entry." >>>>> >>>> >>>> Thanks. >>>> >>>>> We didn’t mention hand-off to M-Mode if CPU only support M-
Mode
>>>> because we only verified edk2 RISC-V port in S-Mode with >>>> OpenSBI, but didn’t try it on M-Mode even though the code is
ready there.
>>>>> >>>> >>>> That's correct. We can't run UEFI applications run time >>>> services in >>> M-mode as >>>> it requires virtual memory in current setup. >>>> I think it is better to keep that way there is a specific >>>> demand and >>> value in >>>> running UEFI applications in M-mode. >>>> >>>>>> >>>>>>> We should prescribe this in the EBBR and somehow get the >>>>>>> UEFI >>> spec >>>>>>> fixed afterwards. >>>>>>> >>>>>> >>>>>> I will add it to the RISC-V EBBR PR (haven't sent it yet). >>>>>> >>>>>>> An other issue is the calling convention. Chapter "2.3.7.3 >>>>>>> Detailed Calling Convention" does not describe which >>>>>>> registers >>> are >>>>>>> saved by the UEFI payload's entry point and restored by the >>>>>>> payload before calling the UEFI API or returning to the >>>>>>> UEFI payload. This concerns especially registers gp (x3) >>>>>>> and tp >>> (x4). >>>>>>> >>>>>>> Into the EBBR or UEFI spec we should put a link to the >>>>>>> "RISC-V >>> ELF >>>>>>> psABI specification" >>>>>>> >>> https://github.com/riscv/riscv-elf-psabi-doc/blob/master/riscv- >>> e >>> lf >>>>>>> .md which is referenced by "The RISC-V Instruction Set
Manual".
>>>>>>> >>>>>>> From the "RISC-V ELF psABI specification" one might >>>>>>> conclude >>> that >>>>>>> the UEFI payload should not be allowed to change gp and tp >>> before >>>>>>> calling >>>>>>> ExitBootServices() or SetVirtualAddressMap() whichever >>>>>>> occurs >>> last. >>>>>>> >>>>>> >>>>>> Agreed. Thanks for pointing this out. I think this should go >>>>>> into the UEFI spec instead of EBBR spec. Any suggestions ? >>>>> To have this external link is good, I will add this link in ECR. >>>>>> >>>> >>>> Thanks again :). As per Heinrich's suggestion, it would also >>>> be good >>> to add >>>> some text about the state of gp & tp. >>> >>> Ok. I add some text, copied from psABI spec. :) >>> >>> "........ Registers tp and gp should not be modified in the >>> function, >> >> I would not know what "in the function" refers to here. >> >> We have a value of tp and gp passed from the firmware to the >> entry-point of the firmware. We need a definition describing >> what assumptions the firmware may or may not make about the >> value of tp and gp up to ExitBootServices and in >> SetVirtualAddressMap. I would not assume gp and tp to have any >> firmware related value
afterwards.
> Ok, I got your point. But do we mention "value of tp and gp > passed from the firmware to the entry-point of the firmware" somewhere in the particular spec? What is the spec of values you pass in tp/gp to the entry point of firmware? > >> >> It would be perfectly ok if the spec said that the firmware >> should make no assumptions about the value of these registers > I don’t think firmware has assumption of value in these registers > because the calling convention doesn't use these two regs. Maybe > the assembly code use these two regs for some purpose but > firmware has to preserve it. The revised version in below, > >> but has to keep the values >> intact during an API call. >> >> The important thing is that we have a clear definition of the >> interface between the firmware and the payload. > > How about this, > "Registers tp and gp are unallocatable and not preserved across > the function calls, UEFI firmware implementation of RISC-V platform should keep the values in these two registers intact during entire UEFI firmware boot process, UEFI boot service and runtime service. See “Link to UEFI Specification-Related Document” on https://uefi.org/uefi under the heading “RISC-V EFL psABI Specification” for the more descriptions of RISC-V calling convention."
Current situation in U-Boot
With U-Boot we currently have the following boot flows:
U-Boot -> payload OpenSBI -> payload U-Boot SPL -> OpenSBI -> U-Boot -> payload
Upon entry U-Boot overwrites tp with the HART id on all HARTs and the boot HART's gp with a pointer to U-Boot's global data structure.
At boot time when the payload makes an API call U-Boot replaces the payload's gp by its own value upon entry and restores the payload's gp before returning to the payload.
At runtime UBoot neither touches gp nor tp.
Interrupt handling
Before ExitBootServices() handling of interrupts and exceptions is the task of the firmware. As both may rely on the values of tp and gp the payload should not overwrite them.
When the operating system takes over it also has gets the responsibility for handling interrupts and exceptions and will set its own
values of tp and gp.
Some operating systems call SetVirtualAddressMap() after ExitBootServices(), others don't. On Linux this is controlled by kernel command line parameter efi=novamap.
Conclusion
I would suggest the following rules:
Until ExitBootServices() returns to the payload the firmware is the owner of registers gp and tp. The payload MUST NOT alter these
values.
This implies that when handling EVT_SIGNAL_EXIT_BOOT_SERVICES
events
the payload MUST NOT alter gp and tp.
After ExitBootServices() the payload is the owner of registers gp and tp. The firmware MUST NOT alter these values.
The firmware solution you mentioned above is the payload solution with
either uboot or opensbi. However, edk2 firmware solution is not the
same.
RISC-V edk2 firmware solution uses OpenSBI as the library. Thus we don't have to mention specific firmware solution in UEFI spec.
I think the final sentence already covered all use cases you mentioned
above, that includes the EFI POST time and the UEFI service exposed to user such as EFI boot service and EFI runtime service. We don’t have to mention specific rule in the spec. I revised it again and add EFI Management Mode service.
My references to U-Boot where not meant for inclusion in any spec but as background information only.
"Registers tp and gp are unallocatable and not preserved across the
function calls, UEFI firmware implementation of RISC-V platform should keep the values in these two registers intact during entire UEFI firmware boot process, EFI boot services, EFI runtime services and EFI management mode services. See “Link to UEFI Specification-Related Document” on https://uefi.org/uefi under the heading “RISC-V EFL psABI Specification” for the more descriptions of
RISC-V calling convention."
Please, have a look at RFC 2119 "Key words for use in RFCs to Indicate Requirement Levels" [INVALID URI REMOVED
3A__tools.ietf.org_html_rfc2119&d=DwIFaQ&c=C5b8zRQO1miGmBeVZ2LFW
g&r=_SN6FZBN4Vgi4Ulkskz6qU3NYRO03nHp9P7Z5q59A3E&m=M_APpMPr8z
v5FK0ys5Noq10akQQqL0eNc9vlZqmOkpw&s=DBCkr05bpuKrQWdyNn5bJSow
sLV7UDvDJIJc3nT4jd0&e= ]. Here "should" has the meaning of "recommended" and may be ignored "in particular circumstances".
Did you mean SHOULD or MUST?
MUST is better, to make UEFI firmware solution simple. UEFI edk2 implementation could be a stand along firmware or to be built as the payload (we don’t have this implementation yet).
"are unallocatable" is vague. It does not describe who is not allowed to change the values. Is it the firmware or is it the payload?
From UEFI perspective, UEFI firmware payload is not allowed to change the value. I will rephrase this sentence.
"keep the values intact" is vague. It could mean that the firmware SHOULD reset gp and tp to the values they had had before StartImage() whenever an API call is made.
We need a definition that cannot be misunderstood. In my definition it is at least clear who is the owner of tp and gp.
Please, consider that before ExitBootServices() many different payloads may be loaded by the firmware: applications, boot service drivers, runtime service drivers. This is why to me it does not make sense to allow any other software but the firmware to define the values of
tp and gp before ExitBootServices().
I am not against that firmware to define the values of tp and gp. I think the "firmware" you mentioned here refer to uboot but not edk2
firmware.
I refer to the firmware providing the UEFI API. U-Boot and EDK II are alternative implementations.
I am also fine to mention UEFI firmware should not (Must not) use gp and tp in spec. I would like to restrict using these two registers in UEFI firmware no matter the RISC-V UEFI firmware implementation is a stand along system firmware or a payload for other firmware such as uboot/opensbi, this would be easier for us to maintain RISC-V edk2 port. Hope I don’t misunderstand
Both U-Boot and EDK II may be started by an SEE, e.g. OpenSBI.
the intention which you brought up here, which is the UEFI firmware payload must not altering the value of gp and tp, right?
However, I don’t want to see that we list the specific functions such as ExitBootServices, startImage, SetVirtualAddress in the UEFI spec, we can just give the strict criteria to UEFI firmware to not using gp/tp. Says UEFI firmware can't alter tp/gp values,
- After firmware enters UEFI firmware payload and before returning
to firmware>> - In BS
- In RTS
- In SMM service
Everytime U-Boot code is entered during boot services the payloads gp is saved. When returning to the payload the payloads value of the gp is restored. So the payload cannot observe any change in gp. At runtime U- Boot does not touch gp.
I hope that is in line with your suggestion.
Best regards
Heinrich
These should cover all possibilities of gp/tp value changed in UEFI
firmware.
We don’t even have to define the ownership of these registers in UEFI
spec.
BTW, above is based on the edk2 UEFI implementation. I am not sure if
other UEFI firmware implementation can avoid to use these registers.
Best regards
Heinrich
>> >>> because signal handlers may rely upon their values. See “Link >>> to UEFI Specification-Related Document” on >>> https://uefi.org/uefi under the heading “RISC-V EFL psABI >>> Specification” for the more descriptions of RISC-V calling
convention."
>>> >>> >>> I will upload this update to USWG once Heinrich agrees with
above.
>>> >>>> >>>>>>> Due to this missing clarification U-Boot is currently >>>>>>> saving gp before calling the entry point of the payload and >>>>>>> restores it >>> on >>>>>>> reentry or on entry of an API call. Nothing is done for tp. >>>>>>>