Assume you have a system with quite some ZONE_MOVABLE memory (esp. in virtualized environments), eating up a significant amount of !ZONE_MOVABLE memory dynamically at runtime can lead to non-obvious issues. It looks like you have plenty of free memory, but the kernel might still OOM when trying to do kernel allocations e.g., for pagetables. With CMA we at least know what we're dealing with - it behaves like ZONE_MOVABLE except for the owner that can place unmovable pages there. We can use it to compute statically the amount of ZONE_MOVABLE memory we can have in the system without doing harm to the system.
Why would you say that secretmem allocates from !ZONE_MOVABLE? If we put boot time reservations aside, the memory allocation for secretmem follows the same rules as the memory allocations for any file descriptor. That means we allocate memory with GFP_HIGHUSER_MOVABLE.
Oh, okay - I missed that! I had the impression that pages are unmovable and allocating from ZONE_MOVABLE would be a violation of that?
After the allocation the memory indeed becomes unmovable but it's not like we are eating memory from other zones here.
... and here you have your problem. That's a no-no. We only allow it in very special cases where it can't be avoided - e.g., vfio having to pin guest memory when passing through memory to VMs.
Hotplug memory, online it to ZONE_MOVABLE. Allocate secretmem. Try to unplug the memory again -> endless loop in offline_pages().
Or have a CMA area that gets used with GFP_HIGHUSER_MOVABLE. Allocate secretmem. The owner of the area tries to allocate memory - always fails. Purpose of CMA destroyed.
Ideally, we would want to support page migration/compaction and allow for allocation from ZONE_MOVABLE as well. Would involve temporarily mapping, copying, unmapping. Sounds feasible, but not sure which roadblocks we would find on the way.
We can support migration/compaction with temporary mapping. The first roadblock I've hit there was that migration allocates 4K destination page and if we use it in secret map we are back to scrambling the direct map into 4K pieces. It still sounds feasible but not as trivial :)
That sounds like the proper way for me to do it then.
But again, there is nothing in the current form of secretmem that prevents allocation from ZONE_MOVABLE.
Oh, there is something: That the pages are not movable.