Can't that fire when changing the max count from user space at just the wrong time?
You are right: technically it's possible if it was raised between the time of checking and when the new VMA is added.
I assume we'll have to tolerated that and might just want to drop this patch from the series.
It is compiled out in !CONFIG_VM_DEBUG builds, would we still want to drop it?
Due to the racy nature I think any kinds of assertions would be wrong. Without any such races possible I would agree that the checks would be nice to have.