On Tue, Nov 03, 2026 at 12:46:24PM -0800, Yabin Cui wrote:

...

In field telemetry and local ftrace on Pixel devices (prior to 18049c8cff9c, so __GFP_NORETRY was already in place), we observed heavy rb_alloc_aux() -> __alloc_pages_direct_compact() stalls alongside kcompactd0 pinned on big cores scanning order-9, and this accounted for

...

20% of all perceptible app kills (via lmkd) across dogfood devices

until we stopped high-order AUX allocations.

Thanks for the detailed explanation. My understanding is that repeated reclaim/compaction during AUX allocation raises memory PSI and can lead to lmkd killing applications. The problem to address is therefore the cost of obtaining higher-order pages under memory fragmentation.

Initially we worked around this in userspace (simpleperf) by setting perf_event_attr::aux_watermark = 4096 so get_order(watermark) == 0. That was fine for CoreSight ETR (which uses its own separate scatter-gather buffer and only copies into AUX on flush/stop), but when enabling TRBE (and SPE) it caused hardware maintenance interrupts every 4 KiB instead of every 2 MiB. Also, in overwrite/snapshot mode rb_alloc_aux() ignores aux_watermark and uses ilog2(nr_pages). That is why I sent 18049c8cff9c.

Agreed. Reducing watermark to 4096 also increases maintenance interrupt frequency for SPE/TRBE. We should refine buffer allocation rather than using watermark as a workaround.

### Possible solutions

For PMUs without PERF_PMU_CAP_AUX_NO_SG (such as SPE, TRBE, and modern Intel PT), higher-order AUX pages are an opportunistic optimization rather than a functional requirement -- very similar to THP allocations on fault (GFP_TRANSHUGE_LIGHT) or opportunistic high-order slabs in SLUB (allocate_slab()).

Instead of reverting to GFP_KERNEL reclaim/compaction for all high-order attempts, could we do the following?

1. Make opportunistic high-order AUX allocations no-reclaim: When !(event->pmu->capabilities & PERF_PMU_CAP_AUX_NO_SG) and order > 0, clear __GFP_RECLAIM (and add __GFP_NOMEMALLOC, matching SLUB's allocate_slab() and GFP_TRANSHUGE_LIGHT):

   gfp_t gfp = PERF_AUX_GFP;

   if (order && !has_no_sg) gfp = (PERF_AUX_GFP | __GFP_NOMEMALLOC) & ~__GFP_RECLAIM;

   When order falls back to 0 (or when PERF_PMU_CAP_AUX_NO_SG is set), keep PERF_AUX_GFP so single-page allocations still reclaim normally under memory pressure.

   That way, when free order-9 / order-4 blocks exist, SPE and TRBE get them immediately for PMD / contiguous-PTE vmap() mappings (and pages stay grouped together); when memory is fragmented on a long-running device, high-order probes fail fast on the freelist without waking kcompactd/kswapd or triggering direct compaction and reclaim.

This looks reasonable for me: try to obtain larger blocks cheaply, then then retain the PERF_AUX_GFP policy for order-0 allocations. I also agree with preserving the allocation policy for PERF_PMU_CAP_AUX_NO_SG case.

Could you compare the memory PSI stall deltas before and after applying this option? I think measurements from your production devices would be convinced evidence for benefits.

2. Clamp max_order in rb_alloc_aux() after fallback: If rb_alloc_aux_page() requested `order` and had to fall back to a smaller order, update `max_order = page_private(page)` in rb_alloc_aux() so subsequent iterations for the rest of the buffer do not keep re-probing higher orders that just failed.

This also can avoid reclaim repeatedly, but the side effect is subsequent iterations would no longer try high order, which have no chance to benefit from large mapping.

I would prefer to evaluate option 1 first.

(Optionally, if server/benchmark workloads ever want synchronous compaction to form huge AUX mappings even on fragmented hosts, that could be gated behind an opt-in attr/sysfs knob similar to THP defrag modes, with light/no-reclaim as the default.)

I will follow up with Will separately about the userspace interface.

Would this approach work for your SPE/TRBE use cases? Happy to help test on our side.

If option 1 works, we can confidently revert 18049c8cff9c, as it would restore higher-order allocations for TRBE/SPE without introducing the regression you mentioned.

Thanks, Leo