[PATCH 5.4.y] arm64: Restrict pagetable teardown to avoid false warning