On 03/15/23 21:03, David Hildenbrand wrote:
On 10.03.23 19:28, Stefan Roesch wrote:
Stefan, can you do me a favor and investigate which pages we end up deduplicating -- especially if it's mostly only the zeropage and if it's still that significant when disabling THP?
I'm currently investigating with some engineers on playing with enabling KSM on some selected processes (enabling it blindly on all VMAs of that process via madvise() ).
One thing we noticed is that such (~50 times) 20MiB processes end up saving ~2MiB of memory per process. That made me suspicious, because it's the THP size.
What I think happens is that we have a 2 MiB area (stack?) and only touch a single page. We get a whole 2 MiB THP populated. Most of that THP is zeroes.
KSM somehow ends up splitting that THP and deduplicates all resulting zeropages. Thus, we "save" 2 MiB. Actually, it's more like we no longer "waste" 2 MiB. I think the processes with KSM have less (none) THP than the processes with THP enabled, but I only took a look at a sample of the process' smaps so far.
I recall that there was a proposal to split underutilized THP and free up the zeropages (IIRC Rik was involved).
I also recall that Mike reported memory waste due to THP.
Interesting!
2MB stacks were also involved in our case. That stack would first get a write fault allocating a THP. The write fault would be followed by a mprotect(PROT_NONE) of the 4K page at the bottom of the stack to create a guard page. The mprotect would result in the THP being split resulting in 510 zero filled pages. I suppose KSM could dedup those zero pages.