Hello,
There was discussion that started on github (https://github.com/OP-TEE/optee_os/issues/1019) and then continued in this mailing list. We discussed how OP-TEE can interact with virtualization and came to conclusion that one of tasks is to rework shared memory mechanism. So I want to discuss my vision on this topic.
Currently shared buffer is carved out from RAM by some higher powers and presented to OP-TEE. OP-TEE informs Normal World about its location. And then OP-TEE uses Normal World's allocator to allocate buffers for various needs. This approach has two drawbacks: shared region size is limited and it is wastes memory space when it is not used. On most platforms it have size of 1 megabyte which is not enough to run GP tests (according to comments in board files).
We have agreed that we don't need this shared carveout at all. This region is unsecure by definition, so Normal World can allocate in anywhere it wishes and then just pass PAs to OP-TEE.
I already did most of the work on kernel side. You can check it there: https://github.com/lorc/linux/tree/optee-new-shm This code still needs to be tidied up, possibly some caching should be added. I also hacked OP-TEE core to make it work. So I want to discuss proper changes to OP-TEE memory management subsystem.
Main requirements are: - MM should be dynamic: we can add and remove mappings in runtime. - There can be arbitrary number of maps. - Every map should be associated with guest virtual machine, so we can clean up easily when one of Normal Worlds dies. - Preserve existing APIs if possible. I don't want to rework all MM API calls and you probably don't want to review this :)
Here is how I see this: there will be two types of mappings: static and dynamic. Most of the areas will be left static, but the area for shared memory and probably the area for device IO will be made dynamic. Actually I borrowed this idea from Linux's vmalloc.c. Lets take SHM as an example:
There will be region in virtual address space for SHM. But it will not be backed by any physical memory at the start. When OP-TEE needs shared buffer it issues RPC call to Normal World, Normal World provides physical page(s). OP-TEE then allocates part of SHM virtual space and maps there pages provided by Normal World. When Normal World allocates SHM on its own, it calls Secure World with `OPTEE_SMC_ADD_SHM_BUFFER` where it provides list of pages. OP-TEE then maps pages in the same way as in previous case. There will be complementary functions and SMC calls to unmap shared buffer and free allocated VA space.
We will need some some allocator to track areas in SHM VA space. I plan to use rb-tree allocator like it is used in vmalloc.c. Probably we will need second rb-tree with PA's to speed up `phys_to_virt` lookups. For every allocated area in VA space we will hold virtual machine id, so we can free this region when virtual machine dies.
APIs in core_memprot.h will be preserved. There just will be another handling for SHM region cases.
Also corresponding changes to MMU drivers should be done. We need to be able to map/unmap individual pages.
The same mechanism can be used for memory mapped peripherals, so device drivers can use plain old `ioremap` to gain access to device registers.
I hope this is not overengineered :)
I will appreciate any comments on this, especially if I missed something.