Hi all,
This quite long email (sorry!) has two purposes: to announce userland lowmemory killer daemon to a broader audience, and to resume discussion on lowmemory notifications.
The userland lowmemory killer daemon (ulmkd), behaves the same way as kernel's lowmemorykiller (LMK) driver, except that the policy now lives in the userland, and the daemon expects some 'low memory notification' services from the kernel (currently, the main backend is cgroups).
Plus, with userland approach now we can send not only SIGKILL, but also user-specific events, upon which programs would not just quit, but would e.g. release/garbage collect memory or we could even try to preemptively suspend and put selected "currently not important" processes into swap (something, I believe, Windows 8 does nowadays, sorry for the analogy. :-) This also seem to slightly relate to fallocate volatile work: the trend is to make resource management a bit smarter, export userland's knowledge about resources to the kernel, give the kernel some hints.
ulmkd is a drop-in replacement for lowmemorykiller driver; one can disable CONFIG_ANDROID_LOW_MEMORY_KILLER in the kernel config, start ulmkd, and everything should behave the same way.
Also, I do hope the code would be useful not only for Android, so if anybody wants to extend it, you're more than welcome.
The code is tiny, and is available in this git repo:
git://git.infradead.org/users/cbou/ulmkd.git
(The repo is three months old since the stuff seem to just work, at least with cgroups.)
The daemon consists of two parts,
- Low memory notifications handling; - Task list management.
For notifications, there are two backends: cgroups and vmevent. Vmevent support is quite outdated, but it is still there just to show the idea. I plan to substitute it with deferred timer polling + shrinker notifications (see below).
For task list management, two methods implemented: /proc based (the daemon reads PIDs and oom_adj values from the /proc directory), and shared memory based, where it is expected that Android Activity Manager (or Maemo, or Tizen manager, or whatever) would keep the task list in the memory, and share it with the killer daemon. The demo_shm.c file provides a small example, it "proxies" task list from /proc to a shared memory. (The Android Activity Manager already manages its own task list, we just need to teach it to share it with the daemon.)
Note that we have to implement LMK as a separate small daemon, the reason behind this is best described in Android example: in JVM we can't guarantee 'no new new memory allocations', we're out of control of what JVM does with memory. Plus, we don't want the killer to be swapped out, and so in ulmkd we call mlockall(), thus locking just the small daemon, not the whole JVM.
Some words about latency: the reaction time is not a big issue in "Low Memory Killer" duties, this is because LMK triggers when we have plenty of free memory and time (tens and hundreds of megabytes), and OOMK (in-kernel OOM killer) will help us if we're too slow. So ulmkd by no means is going to "replace" OOMK.
Note that no matter if we choose to kill processes from kernel or userspace, current in-kernel LMK driver would still need a lot of rework to get it right.
The main problem is vm_stat counters. The vm_stat counters are per-node, per-cpu, and gathering the statistics from all the nodes might be quite expensive: e.g. on SMP to synchronize global counters, we'd need to issue an IPI, which, if we presume that we need a low-latency LMK, would disturb the system quite a lot, and the whole point of "light weight" LMK driver defeats itself.
In-kernel LMK started when most users where UP/"embedded", so it was all straightforward. But now SMP is quite common setup even on embedded devices, and so we will need to "adjust" LMK to a new reality, sooner or later. And we'd better do it in the best possible way, right from the start.
(Note that adding another LRUs, like "easily reclaimable list", doesn't solve the vm_stat issue. Identifying which pages are easily reclaimable is one thing, but statistics is another.)
So, in-kernel LMK shares the same issues with vmevent lowmemory notification approach, because both use vm_stat, which we can't update frequently, and so the statistics are not up to date anyway.
In ulmkd I want to try another approach (in addition to cgroups):
- Considering that we don't have to be super-low-latency, we can just poll /proc/vmstat from userland very infrequently *and* using deferred timers approach to save power, as we did in vmevents -- we won't wake up the system needlessly. As far as I can see, there is no such thing as deferred timers for userland yet, so this is going to be a key part.
- Export shrinker notifications to userland (via vmevents API?). This would zap all the discussions about what to consider "low memory", as shrinker is just a small hint that kernel is short on the memory, and we'll OOM pretty soon (assuming no swap).
Does it sound viable? Note that nothing is set in stone here, before going all-in into it, I'd really want to hear opinions and more ideas.
Thanks!
Anton.