This is a variant of the cache aliasing problem we see with virtually indexed caches: You may end up with multiple cache lines for the same physical address, with different contents. The results are unpredictable, so most CPU architectures explicitly forbid this.
I think the extra wrinkle here is the presence of the unity mapping as cached, even if you never access it, causes a problem. I totally understand why you wouldn't want to access mappings with different attributes, but just having them hang around seems like it shouldn't in general be a problem. How does powerpc handle it when you need an uncached page for dma?
You don't need uncached pages for DMA, Linux only supports systems that are coherent on powerpc, which has pretty much solved the problem by forcing hardware designers to do it the easy way, rather than requiring the software add extra overhead to work around it.
Thats not entirely true, I've nearly certain I've spent a bit of time with Ben tracking down non-coherent PCI issues on powerpc
Dave.