Hello Sumit,

Sorry, I might have misled you. My mistake saying A9 is the exporter.

Yes I am using dri gembuffers here. And it is as per Rob Clark's suggestion that I used. He suggested to use dri gembuffers probably because that was the easiest way to get started and this was over an year back when my usecase was much more simpler. Using the dri gembuffers a lot of things came for free like (contiguous?) memory allocation from the kernel, 1:1 mapping mapping of virtual to physical in the remote processors, mmap for user space, the dmabuf implementation etc.

Yes, I am using dri's implementation as an attachment detachment mechanism. I understand that the DRI didnt have a need to implement this, but what I was saying was that I need to implement this. I dont know whether it is easier to implement another dma-buf exporter or hack dri's implementation for the same.

One of the reasons that I asked this question is because you say in Documentation/dma-buf-sharing.txt that you can consider extending dma-buf with a more explicit cache tracking scheme for userspace mappings. I am asking for this explicit cache tracking scheme for userspace mappings. How would you have gone about implementing it ?

"If the above shootdown dance turns out to be too expensive in certain scenarios, we can extend dma-buf with a more explicit cache tracking scheme for userspace mappings. But the current assumption is that using mmap is always a slower path, so some inefficiencies should be acceptable."

I think what happens currently is on mmap page fault, the driver marks the pages that are accessed and during map attachment it calls unmap_mapping_range to flush the ptes and cache(?). Sources are in the files given below.

http://lxr.free-electrons.com/source/drivers/gpu/drm/omapdrm/omap_gem_dmabuf.c

http://lxr.free-electrons.com/source/drivers/gpu/drm/omapdrm/omap_gem.c

Which linux kernel function/API should I use to cache flush the addresses modified by A9 and invalidate the addresses modified by the remote processors(M3 and DSP) ?

Thanks for your reply.

--Kiran

On Fri, Mar 28, 2014 at 6:06 AM, Sumit Semwal <sumit.semwal@linaro.org> wrote:

Hi Kiran,

On 27 March 2014 17:45, kiran <kiranchandramohan@gmail.com> wrote:
> Hello Sumit,
>
> Thanks for the reply.
>
> I am using the pandaboard. I am trying to partition data-parallel programs
> and run them in parallel on the ARM A9s, ARM M3s and DSP. All the processors
> have to work on the same arrays. Hence I allocate memory for these arrays
> using gembuffers and pass the physical address of these arrays to the remote

^^^ Do you mean you use dri GEM buffers for non-dri usage? Again, I
think that's probably not recommended, but I'll let the DRI experts
comment on it (Daniel, Rob: care to comment?)

> processors using rpmsg. Since there might be cross-processor dependencies, I
> have to do some sort of cache coherency which I am currently doing through
> the attachment-detachment mechanism. But this seems to be expensive.
>
> Documentation/dma-buf-sharing.txt, says that
> "Because existing importing subsystems might presume coherent mappings for
> userspace, the exporter needs to set up a coherent mapping. If that's not
> possible, it needs to fake coherency by manually shooting down ptes when
> leaving the cpu domain and flushing caches at fault time. ..... If the above
> shootdown dance turns out to be too expensive in certain scenarios, we can
> extend dma-buf with a more explicit cache tracking scheme for userspace
> mappings. But the current assumption is that using mmap is always a slower
> path, so some inefficiencies should be acceptable".
>
> I believe the attachment-detachment mechanism shoots down ptes and flushes
> caches to fake coherency. But i guess the ptes does not have to be shot down

Are you relying on the DRI Prime's implementation as an exporter
device to assume that its attachment-detachment mechanism is actually
doing it? AFAIK, DRI hasn't had a need to implement this, but I'd
again let Rob / Daniel comment on it.

> since they don't change during the program execution. Only the cache needs
> to be flushed.
>
> You say that, "To me, using attachment detachment mechanism to 'simulate'
> cache coherency sounds fairly wrong - Ideally, cache coherency management is
> to be done by the exporter in your system".
> Could you give some details on how the exporter(A9 here) can simulate cache
> coherency. The linux API or functions to use. If you can point me to some
> similar implementation in the linux kernel that would be very helpful.

I might be misunderstanding it, but from your statement above, it
seems you feel that A9 is the 'exporter' referred to in the
dma-buf-sharing.txt. That's not so; in fact, an exporter is any linux
driver / device subsystem (currently like DRI for gem buffers, or
V4L2...) that knows / wants to handle the allocation, coherency and
mapping aspects of the buffers. gembuffers are a part of DRI
framework, which is intended for graphics rendering and display
usecases.

For the needs you stated above, which seem to be fairly non-graphics
in nature, I'd think implementing a dma-buf exporter is a good idea -
you could then handle when or if you want to shoot down ptes or not,
and lot of usecase dependent things. You might also want to check with
the TI guys if they have some implementation already in place?

Hope this helps,
Best regards,
~Sumit.

> Thanks in advance.
>
> --Kiran
>
>
>
>
> On Thu, Mar 27, 2014 at 5:23 AM, Sumit Semwal <sumit.semwal@linaro.org>
> wrote:
>>
>> Hi Kiran,
>>
>> On 24 March 2014 02:35, kiran <kiranchandramohan@gmail.com> wrote:
>> > Hi,
>> >
>> > I was looking at some code(given below) which seems to perform very
>> > badly
>> > when attachments and detachments to used to simulate cache coherency.
>> Care to give more details on your use case, which processor(s) are you
>> targeting etc? To me, using attachment detachment mechanism to
>> 'simulate' cache coherency sounds fairly wrong - Ideally, cache
>> coherency management is to be done by the exporter in your system. I
>> presume you have had a chance to read through the
>> Documentation/dma-buf-sharing.txt file, which gives some good details
>> on what to expect from dma-bufs etc?
>>
>> Best regards,
>> ~Sumit.
>>
>> > In the code below, when remote_attach is false(ie no remote processors),
>> > using just the two A9 cores the following code runs in 8.8 seconds. But
>> > when
>> > remote_attach is true then even though there are other cores also
>> > executing
>> > and sharing the workload the following code takes 52.7 seconds. This
>> > shows
>> > that detach and attach is very heavy for this kind of code. (The system
>> > call
>> > detach performs dma_buf_unmap_attachment and dma_buf_detach, system call
>> > attach performs dma_buf_attach and dma_buf_map_attachment).
>> >
>> > for (k = 0; k < N; k++) {
>> > if(remote_attach) {
>> > detach(path) ;
>> > attach(path) ;
>> > }
>> >
>> > for(i = start_indx; i < end_indx; i++) {
>> > for (j = 0; j < N; j++) {
>> > if(path[i][j] < (path[i][k] + path[k][j])) {
>> > path[i][j] = path[i][k] + path[k][j] ;
>> > }
>> > }
>> > }
>> > }
>> >
>> > I would like to manage the cache explicitly and flush cache lines rather
>> > than pages to reduce overhead. I also want to access these buffers from
>> > the
>> > userspace. I can change some kernel code for this. Where should I start
>> > ?
>> >
>> > Thanks in advance.
>> >
>> > --Kiran
>> >
>> >
>> >
>> > _______________________________________________
>> > Linaro-mm-sig mailing list
>> > Linaro-mm-sig@lists.linaro.org
>> > http://lists.linaro.org/mailman/listinfo/linaro-mm-sig
>> >
>>
>>
>>
>> --
>> Thanks and regards,
>>
>> Sumit Semwal
>> Graphics Engineer - Graphics working group
>> Linaro.org │ Open source software for ARM SoCs
>
>
>
>
> --
> regards,
> Kiran C

--
Thanks and regards,

Sumit Semwal
Graphics Engineer - Graphics working group
Linaro.org │ Open source software for ARM SoCs

--
regards,
Kiran C