Hi, Reinette, thanks for your great contribution for EDMM Linux kernel patch. I am trying to follow the newest patch now, and I have some questions on it.
It seems that `sgx_enclave_restrict_permissions()` is able to do permission restrictions for multiple enclave’s pages. After driver invokes ENCLS[EMODPR] to restrict the page’s permission, it should then invoke ENCLS[ETRACK] and send IPIs to ensure stale TLB entries have been flushed. Only in this way, ENCLU[EACCEPT] inside enclave can only succeed.
Current implementation invokes `sgx_enclave_etrack(encl)` after every `__emodpr(…)` in the for loop. My question is:
Can we move the `sgx_enclave_etrack(encl)` out of the for loop? After doing so, `sgx_enclave_etrack(encl)` is invoked **one** time for multiple enclave pages’ permission restriction, instead of N times (N = `modp -> length / PAGE_SIZE`). We may gain some performance optimization from it.
Please correct my if my understanding is incorrect. Looking forward to your reply and Thanks for your time!
BR, Bojun
Hi Bojun
On 9/1/2022 8:55 AM, zhubojun wrote:
It seems that `sgx_enclave_restrict_permissions()` is able to do permission restrictions for multiple enclave’s pages. After driver invokes ENCLS[EMODPR] to restrict the page’s permission, it should then invoke ENCLS[ETRACK] and send IPIs to ensure stale TLB entries have been flushed. Only in this way, ENCLU[EACCEPT] inside enclave can only succeed.
Correct.
Current implementation invokes `sgx_enclave_etrack(encl)` after every `__emodpr(…)` in the for loop. My question is:
Can we move the `sgx_enclave_etrack(encl)` out of the for loop? After doing so, `sgx_enclave_etrack(encl)` is invoked **one** time for multiple enclave pages’ permission restriction, instead of N times (N = `modp -> length / PAGE_SIZE`). We may gain some performance optimization from it.
How important is the performance of page permission restriction? How about the performance of page type modification?
Please correct my if my understanding is incorrect. Looking forward to your reply and Thanks for your time!
From the hardware perspective, a single ETRACK can be run after EMODPR is run on a range of pages.
Some things to keep in mind when considering making this change:
Note that the enclave's mutex is obtained and released every time an enclave page is modified. This is needed to (a) avoid softlockups when modifying a large range of pages and (b) give the reclaimer opportunity make space to load pages that will be modified.
Moving the ETRACK flow out of the for loop implies that the mutex would be released between the time the page is modified and ETRACK flow is run (with enclave mutex held). It is thus possible for other changes to be made or attempted on a page between the time it is modified and the ETRACK flow. The possible interactions between different page modifications (both initiated from user space and the OS via the reclaimer) need to be studied if it is considered to split this flow in two parts.
With the ETRACK flow done while the enclave page being modified is loaded there is a guarantee that the SECS page is loaded also. When the ETRACK flow is isolated there needs to be changes to ensure that the SECS page is loaded.
It needs to be considered how errors will be communicated to user space and how possible inconsistent state could affect user space. In support of partial success the ioctl() returns a count indicating how many pages were successfully modified. With the configuration and ETRACK done per page and their failures handled, the meaning of this count is clear. This needs to be considered because it is not possible for the kernel to undo an EMODPR. So if all (or some of) the EMODPRs succeed but the final ETRACK fails for some reason then the successful EMODPR cannot be undone yet all will be considered failed? How should this be reported to user space? Variations may be ok since EMODPR can be repeated from what I can tell but I envision scenarios where some pages in a range may have their permissions restricted (and thus have EPCM.PR set) but which pages have this state is not clear to user space. I don't know what would be acceptable here. Looking at the EACCEPT flow in the SDM it does not seem as though EPCM.PR is one of the EPC page settings that are verified.
Reinette
Hi, Reinette. Sorry for late reply! Appreciate for you detailed explanation!
On Sep 2, 2022, at 23:22, Reinette Chatre reinette.chatre@intel.com wrote:
How important is the performance of page permission restriction? How about the performance of page type modification?
Enclave applications may need to change its page permissions at runtime. If they change page permissions frequently, it may introduce substantial overhead, compared to applications running on native Linux. (`mprotect()` is more lightweight compared to restricting and extending page permissions inside enclave)
But I have not profiled the detailed of the page permission restriction’s performance. So I’m not sure how much benefit we will gain for performance if we move the ETRACK flow outside the `for loop`.
From the hardware perspective, a single ETRACK can be run after EMODPR is run on a range of pages.
Some things to keep in mind when considering making this change:
Note that the enclave's mutex is obtained and released every time an enclave page is modified. This is needed to (a) avoid softlockups when modifying a large range of pages and (b) give the reclaimer opportunity make space to load pages that will be modified.
Moving the ETRACK flow out of the for loop implies that the mutex would be released between the time the page is modified and ETRACK flow is run (with enclave mutex held). It is thus possible for other changes to be made or attempted on a page between the time it is modified and the ETRACK flow. The possible interactions between different page modifications (both initiated from user space and the OS via the reclaimer) need to be studied if it is considered to split this flow in two parts.
With the ETRACK flow done while the enclave page being modified is loaded there is a guarantee that the SECS page is loaded also. When the ETRACK flow is isolated there needs to be changes to ensure that the SECS page is loaded.
Thanks for pointing out such case which I have not considered yet!
It needs to be considered how errors will be communicated to user space and how possible inconsistent state could affect user space. In support of partial success the ioctl() returns a count indicating how many pages were successfully modified. With the configuration and ETRACK done per page and their failures handled, the meaning of this count is clear. This needs to be considered because it is not possible for the kernel to undo an EMODPR. So if all (or some of) the EMODPRs succeed but the final ETRACK fails for some reason then the successful EMODPR cannot be undone yet all will be considered failed? How should this be reported to user space? Variations may be ok since EMODPR can be repeated from what I can tell but I envision scenarios where some pages in a range may have their permissions restricted (and thus have EPCM.PR set) but which pages have this state is not clear to user space. I don't know what would be acceptable here. Looking at the EACCEPT flow in the SDM it does not seem as though EPCM.PR is one of the EPC page settings that are verified.
Reinette
I agree with you. It is hard to handle when there the final ETRACK fails but EMODPR succeeds.
Thanks for showing the case I have not considered but needs thinking deeply!
BR, Bojun
Hi, Reinette. Sorry for late reply! Appreciate for you detailed explanation!
On Sep 2, 2022, at 23:22, Reinette Chatre reinette.chatre@intel.com wrote:
How important is the performance of page permission restriction? How about the performance of page type modification?
Enclave applications may need to change its page permissions at runtime. If they change page permissions frequently, it may introduce substantial overhead, compared to applications running on native Linux. (`mprotect()` is more lightweight compared to restricting and extending page permissions inside enclave)
But I have not profiled the detailed of the page permission restriction’s performance. So I’m not sure how much benefit we will gain for performance if we move the ETRACK flow outside the `for loop`.
From the hardware perspective, a single ETRACK can be run after EMODPR is run on a range of pages.
Some things to keep in mind when considering making this change:
Note that the enclave's mutex is obtained and released every time an enclave page is modified. This is needed to (a) avoid softlockups when modifying a large range of pages and (b) give the reclaimer opportunity make space to load pages that will be modified.
Moving the ETRACK flow out of the for loop implies that the mutex would be released between the time the page is modified and ETRACK flow is run (with enclave mutex held). It is thus possible for other changes to be made or attempted on a page between the time it is modified and the ETRACK flow. The possible interactions between different page modifications (both initiated from user space and the OS via the reclaimer) need to be studied if it is considered to split this flow in two parts.
With the ETRACK flow done while the enclave page being modified is loaded there is a guarantee that the SECS page is loaded also. When the ETRACK flow is isolated there needs to be changes to ensure that the SECS page is loaded.
Thanks for pointing out such case which I have not considered yet!
It needs to be considered how errors will be communicated to user space and how possible inconsistent state could affect user space. In support of partial success the ioctl() returns a count indicating how many pages were successfully modified. With the configuration and ETRACK done per page and their failures handled, the meaning of this count is clear. This needs to be considered because it is not possible for the kernel to undo an EMODPR. So if all (or some of) the EMODPRs succeed but the final ETRACK fails for some reason then the successful EMODPR cannot be undone yet all will be considered failed? How should this be reported to user space? Variations may be ok since EMODPR can be repeated from what I can tell but I envision scenarios where some pages in a range may have their permissions restricted (and thus have EPCM.PR set) but which pages have this state is not clear to user space. I don't know what would be acceptable here. Looking at the EACCEPT flow in the SDM it does not seem as though EPCM.PR is one of the EPC page settings that are verified.
Reinette
I agree with you. It is hard to handle when there the final ETRACK fails but EMODPR succeeds.
Thanks for showing the case I have not considered but needs thinking deeply!
BR, Bojun
linux-kselftest-mirror@lists.linaro.org
-
Reinette Chatre -
zhubojun