[Linaro-mm-sig] Memory region attribute bits and multiple mappings
Rebecca Schultz Zavin
rebecca at android.com
Tue Apr 19 22:05:50 UTC 2011
On Tue, Apr 19, 2011 at 2:49 PM, Nicolas Pitre <nicolas.pitre at linaro.org>wrote:
> On Tue, 19 Apr 2011, Rebecca Schultz Zavin wrote:
> > Hey all,
> > While we are working out requirements, I was hoping to get some more
> > information about another related issue that keeps coming up on mailing
> > lists and in discussions.
> > ARM has stated that if you have the same physical memory mapped with two
> > different sets of attribute bits you get undefined behavior. I think
> > going to be a requirement that some of the memory allocated via the
> > memory manager is mapped uncached. However, because all of memory is
> > cached into the unity map at boot, we already have two mappings with
> > different attributes. I want to understand the mechanism of the problem,
> > because none of the solutions I can come up with are particularly nice.
> > also like to know exactly which architectures are affected, since the fix
> > may be costly in performance, memory or both. Can someone at ARM explain
> > me why this causes a problem. I have a theory, but it's mostly a guess.
> My own guess is that the cacheable attribute is tied to cache entries
> which are physically tagged. Access to one mapping could establish some
> attributes that the second mapping would inherit on cache hit.
My guess was that if a line was present in the L2, even accesses via an
uncached mapping would get serviced from there. Presumably the prefetcher
would have had to access the cached mapping, causing it to get populated
into the l2. Then when the uncached mapping is accessed later, we see the
old, possibly stale one. The only mechanism by which this makes sense to me
is if the cache attributes are only checked when deciding to put a line into
the l2 and not when retrieving one.
> > I especially want to understand if it's still a problem if we never
> > access the memory via the mapping in the unity map. I know
> > speculative prefetching is part of the issue, so I assume older
> > architectures without that feature don't exhibit this behaviour
> Yes, speculative prefetching is what will cause spurious accesses
> through the
> kernel direct mapping (that's how it is called in Linux) even if you
> don't access it explicitly. Older architectures don't have speculative
> prefetching, and even older ones have VIVT caches which has no problem
> with multiple different mappings.
> > If we really need all mappings of physical memory to have the same cache
> > attribute bits, I see three workarounds:
> > 1- set aside memory at boot that never gets mapped by the kernel. The
> > unified memory manager can then ensure there's only one mapping at a
> > Obvious drawbacks here are that you have to statically partition your
> > into memory you want accessible to the unified memory manager and memory
> > don't. This may not be that big a deal, since most current solutions,
> > cmem, et al basically do this. I can say that on Android devices running
> > a high resolution display (720p and above) we're easily talking about
> > needing 256M of memory or more to dedicate to this.
> This is obviously suboptimal.
> > 2- use highmem pages only for the unified memory manager. Highmem pages
> > only get mapped on demand.
> > This has some performance costs when the kernel allocates other metadata
> > highmem. Most embedded systems still don't have enough memory to need
> > highmem, though I'm guessing that'll follow the current trend and shift
> > the next couple of years.
> The kernel tries not to allocate its own data in highmem. Instead,
> highmem pages are used for user space processes or the buffer cache
> which can be populated directly by DMA and be largely untouched by the
> kernel. The highmem pages are also fairly easily reclaimable making
> them an easy target when large physically contiguous allocations are
> It is true that most systems might not have enough memory to require
> highmem, but they can make use of it nevertheless, simply by changing
> the direct mapped memory threshold.
> While highmem is not free in terms of overhead, it is still quite
> lightweight compared to other memory partitioning schemes, and above all
> it is already supported across the whole kernel and relied upon by many
> people already.
This is actually my favorite solution -- it's easy and not to gross. To be
honest we've been tweaking things so we get some highmem on some android
platforms for a while, because the binder is a huge kernel address space
hog. My plan for implementing a memory manager is to start with this and
solve the harder problem only if I have to.
> > 3- fix up the unity mapping so the attribute bits match those desired by
> > unified memory manger. This could be done by removing pages from the
> > map. It's complicated by the fact that the unity map makes use of large
> > pages, sections and supersections to reduce tlb pressure. I don't think
> > this is impossible if we restrict the set of contexts from which it can
> > happen, but I'm imagining that we will also need to maintain some kind of
> > pool of memory we've moved from cached to uncached since the process is
> > likely to be expensive. Quite likely we will have to iterate
> > over processes and update all their top level page tables.
> The kernel direct mapping share the same mapping entries across all
> processes. So if (part of) the kernel direct mapping uses second level
> page table entries, then the first level entries will share the same
> second level page table across all processes. Hence changing memory
> attributes for those pages covered by that second level table won't
> require any itteration over all processes. Obviously the drawback here
> is more TLB pressure, however if the memory put aside is not used by the
> kernel directly then the associated TLBs won't be involved.
I think the whole kernel direct mapping might be in 1st level page tables,
assuming it's an integer multiple of sections or supersections and aligned
properly. I suppose we could hack things to make sure there were at least 1
supersection's worth of second level page table entries required. I came to
the same conclusion about TLB pressure, that it didn't matter much if we
don't actually ever use those TLBs anyway. We actually had some patches
from folks at nvidia to solve part of this problem by just making the whole
kernel direct map page based. We never measured the performance impact,
just decided we were uncomfortable with the change.
> > These all have drawbacks, so I'd like to really understand the problem
> > before pursuing them. Can the linaro folks find someone who can explain
> > problem in more detail?
> I'm happy to discuss about the details when needed.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Linaro-mm-sig