On Thu, Apr 10, 2025 at 04:53:17PM +0200, Bastien Curutchet wrote:
> Hi all,
>
> Many UIO users performing DMA from their UIO device need to access the
> DMA addresses of the allocated buffers. There are out-of-tree drivers
> that allow to do it but nothing in the mainline.
In which case it does not matter at all for mainline.
FYI the proper and safe way to do DMA from userspace is to use
vfio/iommufd.
Am 11.04.25 um 14:44 schrieb Philipp Stanner:
> On Fri, 2025-04-11 at 13:05 +0200, Christian König wrote:
>> Am 11.04.25 um 11:29 schrieb Philipp Stanner:
>>
>>> [SNIP]
>>>
>>> It could be, however, that at the same moment
>>> nouveau_fence_signal() is
>>> removing that entry, holding the appropriate lock.
>>>
>>> So we have a race. Again.
>>>
>>
>> Ah, yes of course. If signaled is called with or without the lock is
>> actually undetermined.
>>
>>
>>>
>>> You see, fixing things in Nouveau is difficult :)
>>> It gets more difficult if you want to clean it up "properly", so it
>>> conforms to rules such as those from dma_fence.
>>>
>>> I have now provided two fixes that both work, but you are not
>>> satisfied
>>> with from the dma_fence-maintainer's perspective. I understand
>>> that,
>>> but please also understand that it's actually not my primary task
>>> to
>>> work on Nouveau. I just have to fix this bug to move on with my
>>> scheduler work.
>>>
>>
>> Well I'm happy with whatever solution as long as it works, but as
>> far as I can see the approach with the callback simply doesn't.
>>
>> You just can't drop the fence reference for the list from the
>> callback.
>>
>>
>>>
>>> So if you have another idea, feel free to share it. But I'd like to
>>> know how we can go on here.
>>>
>>
>> Well the fence code actually works, doesn't it? The problem is
>> rather that setting the error throws a warning because it doesn't
>> expect signaled fences on the pending list.
>>
>> Maybe we should fix that instead.
> The fence code works as the author intended, but I would be happy if it
> were more explicitly documented.
>
> Regarding the WARN_ON: It occurs in dma_fence_set_error() because there
> is an attempt to set an error code on a signaled fence. I don't think
> that should be "fixed", it works as intended: You must not set an error
> code of a fence that was already signaled.
>
> The reason seems to be that once a fence is signaled, a third party
> might evaluate the error code.
Yeah, more or less correct. The idea is you can't declare an operation as having an error after the operation has already completed.
Because everyone will just wait for the completion and nobody checks the status again after that.
>
> But I think this wasn't wat you meant with "fix".
The idea was to avoid calling dma_fence_set_error() on already signaled fences. Something like this:
@@ -90,7 +90,7 @@ nouveau_fence_context_kill(struct nouveau_fence_chan *fctx, int error)
while (!list_empty(&fctx->pending)) {
fence = list_entry(fctx->pending.next, typeof(*fence), head);
- if (error)
+ if (error & !dma_fence_is_signaled_locked(&fence->base))
dma_fence_set_error(&fence->base, error);
if (nouveau_fence_signal(fence))
That would also improve the handling quite a bit since we now don't set errors on fences which are already completed even if we haven't realized that they are already completed yet.
> In any case, there must not be signaled fences in nouveau's pending-
> list. They must be removed immediately once they signal, and this must
> not race.
Why actually? As far as I can see the pending list is not for the unsignaled fences, but rather the pending interrupt processing.
So having signaled fences on the pending list is perfectly possible.
Regards,
Christian.
>
>>
>>
>>>
>>> I'm running out of ideas. What I'm wondering if we couldn't just
>>> remove
>>> performance hacky fastpath functions such as
>>> nouveau_fence_is_signaled() completely. It seems redundant to me.
>>>
>>
>> That would work for me as well.
> I'll test this approach. Seems a bit like the nuclear approach, but if
> it works we'd at least clean up a lot of this mess.
>
>
> P.
>
>
>>
>>
>>>
>>>
>>> Or we might add locking to it, but IDK what was achieved with RCU
>>> here.
>>> In any case it's definitely bad that Nouveau has so many redundant
>>> and
>>> half-redundant mechanisms.
>>>
>>
>> Yeah, agree messing with the locks even more won't help us here.
>>
>> Regards,
>> Christian.
>>
>>
>>>
>>>
>>>
>>> P.
>>>
>>>
>>>>
>>>>
>>>> P.
>>>>
>>>>
>>>>>
>>>>> Regards,
>>>>> Christian.
>>>>>
>>>>>
>>>>>>
>>>>>> P.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Regards,
>>>>>>> Christian.
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Replace the call to dma_fence_is_signaled() with
>>>>>>>> nouveau_fence_base_is_signaled().
>>>>>>>>
>>>>>>>> Cc: <stable(a)vger.kernel.org> # 4.10+, precise commit not
>>>>>>>> to
>>>>>>>> be
>>>>>>>> determined
>>>>>>>> Signed-off-by: Philipp Stanner <phasta(a)kernel.org>
>>>>>>>> ---
>>>>>>>> drivers/gpu/drm/nouveau/nouveau_fence.c | 2 +-
>>>>>>>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>>>>>>>
>>>>>>>> diff --git a/drivers/gpu/drm/nouveau/nouveau_fence.c
>>>>>>>> b/drivers/gpu/drm/nouveau/nouveau_fence.c
>>>>>>>> index 7cc84472cece..33535987d8ed 100644
>>>>>>>> --- a/drivers/gpu/drm/nouveau/nouveau_fence.c
>>>>>>>> +++ b/drivers/gpu/drm/nouveau/nouveau_fence.c
>>>>>>>> @@ -274,7 +274,7 @@ nouveau_fence_done(struct
>>>>>>>> nouveau_fence
>>>>>>>> *fence)
>>>>>>>> nvif_event_block(&fctx->event);
>>>>>>>> spin_unlock_irqrestore(&fctx->lock,
>>>>>>>> flags);
>>>>>>>> }
>>>>>>>> - return dma_fence_is_signaled(&fence->base);
>>>>>>>> + return test_bit(DMA_FENCE_FLAG_SIGNALED_BIT,
>>>>>>>> &fence-
>>>>>>>>
>>>>>>>>>
>>>>>>>>> base.flags);
>>>>>>>>>
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> static long
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>>
Hi Bastien,
Am 11.04.25 um 10:14 schrieb Bastien Curutchet:
> Hi Christian,
>
> On 4/11/25 9:30 AM, Christian König wrote:
>> Hi Thomas,
>>
>> Am 10.04.25 um 21:43 schrieb Thomas Petazzoni:
>>> Hello Christian,
>>>
>>> Thanks for your feedback!
>>>
>>> On Thu, 10 Apr 2025 18:29:12 +0200
>>> Christian König<christian.koenig(a)amd.com> wrote:
>>>
>>>>> Many UIO users performing DMA from their UIO device need to access the
>>>>> DMA addresses of the allocated buffers. There are out-of-tree drivers
>>>>> that allow to do it but nothing in the mainline.
>>>> Well that basically disqualifies this patch set in the first paragraph.
>>>>
>>>> To justify some kernel change we always need an in kernel user of the
>>>> interface, since this is purely for out-of-tree drivers this is a
>>>> no-go to begin with.
>>> I'm not sure to understand your comment here. This patch series is
>>> about extending the UIO interface... which is a user-space interface.
>>> So obviously it has no "in-kernel user" because it's meant to be used
>>> from user-space. Could you clarify what you meant here?
>>
>> Bastien wrote about "out-of-tree drivers" which is something the upstream kernel explicitly does not support.
>>
>
> Sorry maybe it wasn't clear, but what I meant is that the goal of this series is to replace 'out-of-tree drivers' with something upstream.
Ah! Yeah that wasn't really clear from the description.
But anyway please note that when you want to create new UAPI you need to provide an open source user of it. E.g. link to a repository or something similar in the covert letter should do it.
>> Well why do you then want to use DMA-buf in the first place? As far as I know what you describe can perfectly be done with the normal UIO memory management interfaces.
>>
>> DMA-buf is only interesting when you actually want to share something in between devices or between applications.
>>
>
> I wanted to use DMA-buf because it allows to dynamically allocate/release coherent buffers from userspace. UIO doesn't provide such interface.
> I'm aware that exposing DMA addresses to userspace isn't a good practice. That's why this series create a new heap specific to UIO that will be the only one implementing the new ioctl.
I don't know the UIO interfaces that well, but that is pretty clearly an abuse of DMA-buf and won't fly at all.
If you want coherent memory for your device you should use dma_alloc_coherent() for that.
Regards,
Christian.
>
>
> Best regards,
> Bastien
>
>
>
Am 11.04.25 um 11:29 schrieb Philipp Stanner:
> [SNIP]
> It could be, however, that at the same moment nouveau_fence_signal() is
> removing that entry, holding the appropriate lock.
>
> So we have a race. Again.
Ah, yes of course. If signaled is called with or without the lock is actually undetermined.
> You see, fixing things in Nouveau is difficult :)
> It gets more difficult if you want to clean it up "properly", so it
> conforms to rules such as those from dma_fence.
>
> I have now provided two fixes that both work, but you are not satisfied
> with from the dma_fence-maintainer's perspective. I understand that,
> but please also understand that it's actually not my primary task to
> work on Nouveau. I just have to fix this bug to move on with my
> scheduler work.
Well I'm happy with whatever solution as long as it works, but as far as I can see the approach with the callback simply doesn't.
You just can't drop the fence reference for the list from the callback.
> So if you have another idea, feel free to share it. But I'd like to
> know how we can go on here.
Well the fence code actually works, doesn't it? The problem is rather that setting the error throws a warning because it doesn't expect signaled fences on the pending list.
Maybe we should fix that instead.
> I'm running out of ideas. What I'm wondering if we couldn't just remove
> performance hacky fastpath functions such as
> nouveau_fence_is_signaled() completely. It seems redundant to me.
That would work for me as well.
>
> Or we might add locking to it, but IDK what was achieved with RCU here.
> In any case it's definitely bad that Nouveau has so many redundant and
> half-redundant mechanisms.
Yeah, agree messing with the locks even more won't help us here.
Regards,
Christian.
>
>
> P.
>
>>
>> P.
>>
>>> Regards,
>>> Christian.
>>>
>>>> P.
>>>>
>>>>
>>>>
>>>>> Regards,
>>>>> Christian.
>>>>>
>>>>>> Replace the call to dma_fence_is_signaled() with
>>>>>> nouveau_fence_base_is_signaled().
>>>>>>
>>>>>> Cc: <stable(a)vger.kernel.org> # 4.10+, precise commit not to
>>>>>> be
>>>>>> determined
>>>>>> Signed-off-by: Philipp Stanner <phasta(a)kernel.org>
>>>>>> ---
>>>>>> drivers/gpu/drm/nouveau/nouveau_fence.c | 2 +-
>>>>>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>>>>>
>>>>>> diff --git a/drivers/gpu/drm/nouveau/nouveau_fence.c
>>>>>> b/drivers/gpu/drm/nouveau/nouveau_fence.c
>>>>>> index 7cc84472cece..33535987d8ed 100644
>>>>>> --- a/drivers/gpu/drm/nouveau/nouveau_fence.c
>>>>>> +++ b/drivers/gpu/drm/nouveau/nouveau_fence.c
>>>>>> @@ -274,7 +274,7 @@ nouveau_fence_done(struct nouveau_fence
>>>>>> *fence)
>>>>>> nvif_event_block(&fctx->event);
>>>>>> spin_unlock_irqrestore(&fctx->lock, flags);
>>>>>> }
>>>>>> - return dma_fence_is_signaled(&fence->base);
>>>>>> + return test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &fence-
>>>>>>> base.flags);
>>>>>> }
>>>>>>
>>>>>> static long
Hi Thomas,
Am 10.04.25 um 21:43 schrieb Thomas Petazzoni:
> Hello Christian,
>
> Thanks for your feedback!
>
> On Thu, 10 Apr 2025 18:29:12 +0200
> Christian König <christian.koenig(a)amd.com> wrote:
>
>>> Many UIO users performing DMA from their UIO device need to access the
>>> DMA addresses of the allocated buffers. There are out-of-tree drivers
>>> that allow to do it but nothing in the mainline.
>> Well that basically disqualifies this patch set in the first paragraph.
>>
>> To justify some kernel change we always need an in kernel user of the
>> interface, since this is purely for out-of-tree drivers this is a
>> no-go to begin with.
> I'm not sure to understand your comment here. This patch series is
> about extending the UIO interface... which is a user-space interface.
> So obviously it has no "in-kernel user" because it's meant to be used
> from user-space. Could you clarify what you meant here?
Bastien wrote about "out-of-tree drivers" which is something the upstream kernel explicitly does not support.
When you make that UIO API and have an open source userspace driver then that is probably a good justification to do this.
What the kernel community tries to prevent here is that people start using the UAPI to write closed source drivers in userspace.
>> What you could potentially do is to create an UIO driver which
>> imports DMA-bufs, pins them and then provide the DMA addresses to
>> userspace.
>>
>> But please be aware that DMA-fences are fundamentally incompatible
>> with UIO. So you won't be able to do any form of synchronization
>> which probably makes the implementation pretty limited.
> Could you clarify why DMA-fences would be needed here, and for what
> synchronization?
In general DMA-buf is an interface which enables you do share device specific buffers between different drivers as well as between userspace processes.
For this to work with for example cameras, GPUs or RDMA NICs you need to some kind of synchronization primitive, e.g. device A can only starts it's DMA transaction when device B has completed his.
The problem is that this synchronization approach is fundamentally incompatible with UIO. See here for more details: https://www.kernel.org/doc/html/latest/driver-api/dma-buf.html#indefinite-d…
> The DMA buffers allocated here are DMA coherent buffers. So the
> user-space application that uses UIO will allocate such buffers once at
> application start, retrieve their DMA address, and then program DMA
> transfers as it sees fit using the memory-mapped registers accessible
> through UIO. I'm not sure which synchronization you are referring to.
> We are not "chaining" DMA transfers, with for example a camera
> interface feeding its DMA buffers to an ISP or something like that. The
> typical use case here is some IP block in an FPGA that does DMA
> transfers to transfer data to/from some sort of specialized I/O
> interface. We get an interrupt when the transfer is done, and we can
> re-use the buffer for the next transfer.
Well why do you then want to use DMA-buf in the first place? As far as I know what you describe can perfectly be done with the normal UIO memory management interfaces.
DMA-buf is only interesting when you actually want to share something in between devices or between applications.
Regards,
Christian.
> If you could clarify here as well, it would definitely help in
> understanding the shortcomings/limitations.
>
> Thanks a lot!
>
> Thomas Petazzoni
Am 10.04.25 um 16:53 schrieb Bastien Curutchet:
> Hi all,
>
> Many UIO users performing DMA from their UIO device need to access the
> DMA addresses of the allocated buffers. There are out-of-tree drivers
> that allow to do it but nothing in the mainline.
Well that basically disqualifies this patch set in the first paragraph.
To justify some kernel change we always need an in kernel user of the interface, since this is purely for out-of-tree drivers this is a no-go to begin with.
> I know DMA shouldn't be handled by userspace but, IMHO, since UIO
> drivers exist, it would be better if they offered a way of doing this.
Leaking DMA addresses to userspace is usually seen as quite some security hole, but on the other hand with UIO you don't have much other choice.
> This patch series use the dma-heap framework which already allows
> userspace to allocate DMA buffers. I tried to avoid 'polluting' the
> existing heaps to prevent inappropriate uses of this new feature by
> introducing a new UIO heap, which is the only one implementing this
> behavior.
Yeah, that won't fly at all.
What you could potentially do is to create an UIO driver which imports DMA-bufs, pins them and then provide the DMA addresses to userspace.
But please be aware that DMA-fences are fundamentally incompatible with UIO. So you won't be able to do any form of synchronization which probably makes the implementation pretty limited.
Regards,
Christian.
>
> PATCH 1 allows the creation of heaps that don't implement map/unmap_buf
> operations as UIO heap doesn't use them.
> PATCH 2 adds the DMA_BUF_IOCTL_GET_DMA_ADDR which transmits the DMA
> addresses to userspace.
> PATCH 3 implements the UIO heap.
>
> It has been tested with the uio_pci_generic driver on a PowerPC.
>
> Signed-off-by: Bastien Curutchet <bastien.curutchet(a)bootlin.com>
> ---
> Bastien Curutchet (3):
> dma-buf: Allow heap that doesn't provide map_buf/unmap_buf
> dma-buf: Add DMA_BUF_IOCTL_GET_DMA_ADDR
> uio: Add UIO_DMABUF_HEAP
>
> drivers/dma-buf/dma-buf.c | 29 +++++++++--
> drivers/uio/Kconfig | 9 ++++
> drivers/uio/Makefile | 1 +
> drivers/uio/uio.c | 4 ++
> drivers/uio/uio_heap.c | 120 +++++++++++++++++++++++++++++++++++++++++++
> include/linux/dma-buf.h | 1 +
> include/linux/uio_driver.h | 2 +
> include/uapi/linux/dma-buf.h | 1 +
> 8 files changed, 164 insertions(+), 3 deletions(-)
> ---
> base-commit: 5f13fa25acaa4f586aaed12efcf7436e004eeaf2
> change-id: 20250408-uio-dma-9b011e9e7f0b
>
> Best regards,