On 03.04.23 18:34, Stefan Roesch wrote:
In contrast to e.g.:
THP resulted in many zeropages we end up deduplicating again. The THP placement was unfortunate.
Unoptimized memory allocators that leave many identical pages mapped after freeing up memory (e.g., zeroed pages, pages all filled with poison values) instead of e.g., using MADV_DONTNEED to free up that memory.
I repeated an experiment with and without KSM. In terms of THP there is no huge difference between the two. On a 64GB main memory machine I see between 100 - 400MB in AnonHugePages.
/sys/kernel/mm/ksm/pages_shared is over 10000 when we run this on an Instagram workload. The workload consists of 36 processes plus a few sidecar processes.
Thanks! To which value is /sys/kernel/mm/ksm/max_page_sharing set in that environment?
It's set to the standard value of 256.
In the meantime I have run experiments with different settings for pages_to_scan. With the default value of 100, we only get a relatively small benefit of KSM. If I increase the value to for instance to 2000 or 3000 the savings are substantial. (The workload is memory bound, not CPU bound).
Interesting.
Here are some stats for setting pages_to_scan to 3000:
full_scans: 560 general_profit: 20620539008 max_page_sharing: 256 merge_across_nodes: 1 pages_shared: 125446 pages_sharing: 5259506 pages_to_scan: 3000 pages_unshared: 1897537 pages_volatile: 12389223 run: 1 sleep_millisecs: 20 stable_node_chains: 176 stable_node_chains_prune_millisecs: 2000 stable_node_dups: 2604 use_zero_pages: 0 zero_pages_sharing: 0
What would be interesting is pages_shared after max_page_sharing was set to a very high number such that pages_shared does not include duplicates. Then pages_shared actually expresses how many different pages we deduplicate. No need to run without THP in that case.
Thats on my list for the next set of experiments.
Splendid.
Similarly, enabling "use_zero_pages" could highlight if your workload ends up deduplciating a lot of zeropages. But maxing out max_page_sharing would be sufficient to understand what's happening.
I already run experiments with use_zero_pages, but they didn't make a difference. I'll repeat the experiment with a higher pages_to_scan value.
Okay, so it's most certainly not the zeropage. Thanks for that information and running the experiments!