On Fri, Sep 23, 2022, at 2:56 PM, Linus Walleij wrote:

On Fri, Sep 23, 2022 at 1:58 PM Arnd Bergmann arnd@arndb.de wrote:

...

On Fri, Sep 23, 2022, at 1:38 PM, Linus Walleij wrote:

...

...

A recent change affecting the behaviour of phys_to_dma() to actually require the device tree ranges to work unmasked a bug in the Integrator DMA ranges.

The PL110 uses the CMA allocator to obtain coherent allocations from a dedicated 1MB video memory, leading to the following call chain:

drm_gem_cma_create() dma_alloc_attrs() dma_alloc_from_dev_coherent() __dma_alloc_from_coherent() dma_get_device_base() phys_to_dma() translate_phys_to_dma()

phys_to_dma() by way of translate_phys_to_dma() will nowadays not provide 1:1 mappings unless the ranges are properly defined in the device tree and reflected into the dev->dma_range_map.

I don't understand this yet, what did the kernel previously do to that allowed the correct DMA mapping when a wrong address was set in the DT?

Ah this goes way back. I think I need a different fixes tag. I hadn't tested this platform in a while.

What happens currently in translate_phys_to_dma() is that it returns DMA_MAPPING_ERROR (0xffffffff) because of the buggy device tree and that causes the problem.

Before the dma direct rework we selected ARCH_HAS_PHYS_TO_DMA and arch/arm/include/asm/dma-direct.h did this:

static inline dma_addr_t pfn_to_dma(struct device *dev, unsigned long pfn) { if (dev) pfn -= dev->dma_pfn_offset; return (dma_addr_t)__pfn_to_bus(pfn); } (...) static inline dma_addr_t __phys_to_dma(struct device *dev, phys_addr_t paddr) { unsigned int offset = paddr & ~PAGE_MASK; return pfn_to_dma(dev, __phys_to_pfn(paddr)) + offset; }

As dma_pfn_offset was 0 this resulted in a 1:1 map.

Ah, I see. So the offset was 0 because the dma-ranges did not actually point to RAM then, right?

Odd thing: this flag has no device tree bindings I could find. Maybe it's in the really old DT docs?

No idea. I do remember that in the old times, whether DMA was coherent or not was just a compile-time option, so ppc was usually assumed to be coherent while Arm was not, but then we needed something to identify machines that were different.

Here the special memory appears in the CPU address map.

    bus@c0000000 {
            compatible = "arm,integrator-ap-lm";
            #address-cells = <1>;
            #size-cells = <1>;
            ranges = <0xc0000000 0xc0000000 0x40000000>;
            dma-ranges;

(...) lm0: bus@c0000000 { compatible = "simple-bus"; ranges = <0x00000000 0xc0000000 0x10000000>; dma-ranges = <0x00000000 0xc0000000 0x10000000>; reg = <0xc0000000 0x10000000>; #address-cells = <1>; #size-cells = <1>;

                    display@1000000 {
                            compatible = "arm,pl110", "arm,primecell";
                            reg = <0x01000000 0x1000>;

(...) memory-region = <&impd1_ram>; dma-ranges; (...)

Here we find the reg properly at physical address 0xc1000000 thanks to the ranges: 0xc0000000 + 0x00000000 + 0x01000000 = 0xc1000000.

But the special memory region is in the root of the device tree, outside of the translation.

Now dma-ranges will assume that when we translate the memory region it is in the address space of the display controller, but it isn't, because the phandle goes to something in the root of the device tree.

0xc2000000 + 0x00000000 + 0x00000000 = 0xc2000000

Whenever I do ranges my head start spinning :/

If you have a better way to accomodate the DMA ranges I am happy to comply!

I'm still not sure I understand it. Is that reserved memory area in RAM or on the logic bus? The 'ranges' property makes it appear that all of 0xc0000000-0xcfffffff is the logic bus, but then the vram itself is also in that address range, which would indicate that it is behind that bus. However, using the 'reserved-memory' property indicates that this is actually just normal RAM that is carved out from what the OS manages.

Arnd

On Fri, Sep 23, 2022, at 3:54 PM, Linus Walleij wrote:

On Fri, Sep 23, 2022 at 3:21 PM Arnd Bergmann arnd@arndb.de wrote:

...

On Fri, Sep 23, 2022, at 2:56 PM, Linus Walleij wrote:

...

but then the vram itself is also in that address range, which would indicate that it is behind that bus.

Yep it is.

Ok.

...

However, using the 'reserved-memory' property indicates that this is actually just normal RAM that is carved out from what the OS manages.

The binding document Documentation/devicetree/bindings/reserved-memory/reserved-memory.yaml unfortunately does not mention anywhere that the reserved memory must be a subset of the system main RAM "carveout" or anything like that, it says that the memory should be "designed for special usage by various device drivers", and "excluded from normal usage" which it is, hence the misunderstanding.

We have used this for Vexpress like forever, but I did the change from some custom vexpress memory property to shared-dma-pool so I guess that's my fault. Incidentally this also works very well.

Is there some other mechanism we should be using for dedicated framebuffer memory on some odd address?

I think the correct way to do this would have been to have a 'reg' property in the device itself that points to the VRAM, instead of describing it as memory.

This in turn would mean that the driver would use some variant of ioremap_wc() to map the framebuffer into kernel space, rather than using dma_alloc_coherent().

I guess the author of the driver copied the code from another fbdev driver that was meant for an Arm SoC with a shared memory framebuffer rather than a dedicated VRAM...

It's probably too late to change now, and the new dma-ranges property is at least consistent with the usage.

Can you update the changelog text based on the above and send the patch again with the dma-coherent property removed?

Arnd