[PATCH 6.1.y] arm64: Restrict pagetable teardown to avoid false warning