[PATCH 5.15.y] arm64: Restrict pagetable teardown to avoid false warning