Hi Rob,
- flipping between multiple back buffers, perhaps not in order (to handle video formats with B-frames)
Oh, yes please. The closed source drivers seems to do this also all the time, and I never really understood why DRI is limiting the buffers to the OGL attachment points.
...
Current solutions use the GPU to do a scaled/colorconvert into a DRI2 buffer from the client context. The goal of this protocol change is to push the decision to use overlay or GPU blit to the xorg driver.
You left one corner case out, HDMI allows for the framebuffer to be in YUV 4:4:4 format. So it is possible to send YUV data to the display (usually a TV) without color conversion at all, but I think with the current state of X we are years away from that.
...
In many cases, an overlay will avoid several passes through memory (blit/scale/colorconvert to DRI back-buffer on client side, blit to front and fake-front, and then whatever compositing is done by the window manager). On the other hand, overlays can often be handled directly by the scanout engine in the display hardware, with the GPU switched off.
Actually AMD has thrown out the hardware support for overlay with the R7xx (or was it evergreen?) series, because they got support for turning shader pipes off separately and figured out that it use less power to turn off all shaders except one and then use this one for color conversion and scaling, compared to having a distinct hardware block doing the job. But there are tendencies to get a distinct color conversion block back again.
...
Note: video is not exactly the same as 3d, there are a number of other things to consider (scaling, colorconvert, multi-planar formats). But on the other hand the principle is similar (direct rendering from hw video codecs). And a lot infrastructure of connection, authentication, is same. So there are two options, either extend DRI2 or add a new protocol which duplicates some parts. I'd like to consider extending DRI2 first, but if people think the requirements for video are too much different from 3d, then I could split this into a new protocol.
If you ask me extending things seems the better way to do this.
..
@@ -184,6 +185,11 @@ DRI2ATTACHMENT { DRI2BufferFrontLeft These values describe various attachment points for DRI2 buffers.
- In the case of video driver (DRI2DriverXV) the attachment,
- other than DRI2BufferFrontLeft, just indicates buffer
- number and has no other special significance. There is no
- automatic maintenance of DRI2BufferFakeFrontLeft.
I think that will created compatibility problems with existing implementations, because the DDX side doesn't know if it's talking to a video or 3D client side.
...
- The "XV_OSD" attribute specifies the XID of a pixmap containing
- ARGB data to be non-destructively overlayed over the video. This
- could be used to implement subtiles, on-screen-menus, etc.
Why an XID? I'm not 100% sure about it, but using a DRI buffer name directly here seems to be the better alternative.
Are you targeting/limiting this to a particular API (or the customary limitations of overlay HW)? I ask because VDPAU allows clients to pass in an arbitrary colour conversion matrix rather than color standard/hue/sat/bri/con, so it wouldn't be possible to use this in that context.
Ideally it would something that could be used either from device-dependent VDPAU or VAAPI driver back-end, or something that could be used in a generic way, for example GStreamer sink element that could be used with software codecs.
AFAIK DRI mostly isn't a device driver dependent protocol, and the client side doesn't necessary know to which hardware it is talking, just look at how gallium 3D is working, talking with X over the DRI protocol is part of the driver independent state tracker, and NOT part of the driver itself.
So having this driver independent is just a must have, and not optional.
Well, this is the goal anyways. There is one slight other complication for use in a generic way.. it would need to be a bit better defined what the buffer 'name' is, so that the client side would know how to interpret it, mmap it if needed. But I think there is a solution brewing: http://lists.linaro.org/pipermail/linaro-mm-sig/2011-August/000509.html
That's indeed true, but currently it is the only parameter that must be interpreted in a driver dependent fashion. And I really think it should stay that way.
As far as color conversion matrix... well, the attribute system can have arbitrary device-dependent attributes. In the VDPAU case, I suppose the implementation on the client side knows which xorg driver it is talking to, and could introduce it's own attributes. Perhaps a bit awkward for communicating a matrix, but you could in theory have 4*3 different attributes (ie. XV_M00, XV_M01, ... XV_M23) for each entry in the matrix.
Yes, but you should define that clearly in the protocol, we also need something to make the client side know what is supported: individual values or matrix or both?
Also in general, their compositing API is a lot more flexible and allows for a background + multiple layers, rather than just a single layer. I suppose you could pre-flatten the layers into a single one, but the background would be problematic.
Yeah, pre-flatten into a single layer, I think. I mean, we *could* push that to xorg driver side too, but I was trying to not make things overly complicated in the protocol.
I'm not sure I caught the issue about background.. or are you thinking about video w/ AYUV? Is there any hw out there that supports overlays w/ YUV that has an alpha channel? If this is enough of a weird edge case, maybe it is ok to fall back in these cases to the old way of doing the blending on the client side and just looking like a 3d app to the xorg side. (I suspect in this sort of case you'd end up falling back to the GPU on the xorg side otherwise.) But I'm always interested to hear any other suggestions.
VDPAU indeed defines two YUVA formats you can use, but currently I haven't seen anybody using the background picture functionality so far, because there are just not so much YUVA videos around.
Christian.
2011/9/2 Christian König deathsimple@vodafone.de:
Hi Rob,
+ flipping between multiple back buffers, perhaps not in order (to handle video formats with B-frames)
Oh, yes please. The closed source drivers seems to do this also all the time, and I never really understood why DRI is limiting the buffers to the OGL attachment points.
...
Current solutions use the GPU to do a scaled/colorconvert into a DRI2 buffer from the client context. The goal of this protocol change is to push the decision to use overlay or GPU blit to the xorg driver.
You left one corner case out, HDMI allows for the framebuffer to be in YUV 4:4:4 format. So it is possible to send YUV data to the display (usually a TV) without color conversion at all, but I think with the current state of X we are years away from that.
...
In many cases, an overlay will avoid several passes through memory (blit/scale/colorconvert to DRI back-buffer on client side, blit to front and fake-front, and then whatever compositing is done by the window manager). On the other hand, overlays can often be handled directly by the scanout engine in the display hardware, with the GPU switched off.
Actually AMD has thrown out the hardware support for overlay with the R7xx (or was it evergreen?) series, because they got support for turning shader pipes off separately and figured out that it use less power to turn off all shaders except one and then use this one for color conversion and scaling, compared to having a distinct hardware block doing the job. But there are tendencies to get a distinct color conversion block back again.
The overlays are still there, but the old video overlay (with scaling and format conversion) was removed for r5xx. The current overlays are just overlay planes and mostly there for things like GL overlays. You could use them for video, but you'd need to do the scaling and format conversion in the 3D engine.
Alex
On Fre, 2011-09-02 at 12:04 +0200, Christian König wrote:
- flipping between multiple back buffers, perhaps not in order (to handle video formats with B-frames)
Oh, yes please. The closed source drivers seems to do this also all the time, and I never really understood why DRI is limiting the buffers to the OGL attachment points.
Probably because DRI2 was designed mainly with GL direct rendering in mind.
Note: video is not exactly the same as 3d, there are a number of other things to consider (scaling, colorconvert, multi-planar formats). But on the other hand the principle is similar (direct rendering from hw video codecs). And a lot infrastructure of connection, authentication, is same. So there are two options, either extend DRI2 or add a new protocol which duplicates some parts. I'd like to consider extending DRI2 first, but if people think the requirements for video are too much different from 3d, then I could split this into a new protocol.
If you ask me extending things seems the better way to do this.
Probably, though I'm not sure it's a good idea to extend existing DRI2 protocol requests. Even if it's theoretically possible, it'll probably be much easier and less problematic to add new requests instead.
Are you targeting/limiting this to a particular API (or the customary limitations of overlay HW)? I ask because VDPAU allows clients to pass in an arbitrary colour conversion matrix rather than color standard/hue/sat/bri/con, so it wouldn't be possible to use this in that context.
Ideally it would something that could be used either from device-dependent VDPAU or VAAPI driver back-end, or something that could be used in a generic way, for example GStreamer sink element that could be used with software codecs.
AFAIK DRI mostly isn't a device driver dependent protocol, and the client side doesn't necessary know to which hardware it is talking, just look at how gallium 3D is working, talking with X over the DRI protocol is part of the driver independent state tracker, and NOT part of the driver itself.
So having this driver independent is just a must have, and not optional.
Well, what's implemented in the Gallium DRI state tracker is just a de facto standard which works with the current X drivers. The intention was certainly for this to be opaque at the DRI2 protocol level. Once there is divergence between how drivers handle this, the Gallium DRI state tracker will probably need to grow some kind of driver specific hooks for this.
2011/9/2 Christian König deathsimple@vodafone.de:
Hi Rob,
+ flipping between multiple back buffers, perhaps not in order (to handle video formats with B-frames)
Oh, yes please. The closed source drivers seems to do this also all the time, and I never really understood why DRI is limiting the buffers to the OGL attachment points.
...
Current solutions use the GPU to do a scaled/colorconvert into a DRI2 buffer from the client context. The goal of this protocol change is to push the decision to use overlay or GPU blit to the xorg driver.
You left one corner case out, HDMI allows for the framebuffer to be in YUV 4:4:4 format. So it is possible to send YUV data to the display (usually a TV) without color conversion at all, but I think with the current state of X we are years away from that.
heh, yeah..
possibly w/ overlays, at least when fullscreen and gfx layer is not visible, we could send YUV direct to the TV.. but I'm not really sure the practicality of switching back and forth between RGB and YUV
...
In many cases, an overlay will avoid several passes through memory (blit/scale/colorconvert to DRI back-buffer on client side, blit to front and fake-front, and then whatever compositing is done by the window manager). On the other hand, overlays can often be handled directly by the scanout engine in the display hardware, with the GPU switched off.
Actually AMD has thrown out the hardware support for overlay with the R7xx (or was it evergreen?) series, because they got support for turning shader pipes off separately and figured out that it use less power to turn off all shaders except one and then use this one for color conversion and scaling, compared to having a distinct hardware block doing the job. But there are tendencies to get a distinct color conversion block back again.
The nice thing would be, by pushing this decision of *how* to render video to the DDX, the xorg driver can make intelligent choices about how best to implement it for a particular generation of gfx card or SoC. Nothing forces the use of overlays, but the goal is just to allow overlays to be used when/if it makes sense.
...
Note: video is not exactly the same as 3d, there are a number of other things to consider (scaling, colorconvert, multi-planar formats). But on the other hand the principle is similar (direct rendering from hw video codecs). And a lot infrastructure of connection, authentication, is same. So there are two options, either extend DRI2 or add a new protocol which duplicates some parts. I'd like to consider extending DRI2 first, but if people think the requirements for video are too much different from 3d, then I could split this into a new protocol.
If you ask me extending things seems the better way to do this.
Ok, I'm glad that folks don't seem too scared off by adding some video stuff in dri2proto :-)
..
@@ -184,6 +185,11 @@ DRI2ATTACHMENT { DRI2BufferFrontLeft These values describe various attachment points for DRI2 buffers.
- In the case of video driver (DRI2DriverXV) the attachment,
- other than DRI2BufferFrontLeft, just indicates buffer
- number and has no other special significance. There is no
- automatic maintenance of DRI2BufferFakeFrontLeft.
I think that will created compatibility problems with existing implementations, because the DDX side doesn't know if it's talking to a video or 3D client side.
On the xorg/DDX side, we'd have to expose the 'driverType' to the DRI2 implementation somehow. I'm not sure yet the best way to handle this, but I guess it is a solvable problem.
Once it seems like I'm basically the right track I'll start putting together some patches for xserver tree to go along with the proto changes and sort out these details. I wanted to at least make sure I was on the right track by extending dri2 as opposed to adding a new extension first.
...
- The "XV_OSD" attribute specifies the XID of a pixmap containing
- ARGB data to be non-destructively overlayed over the video. This
- could be used to implement subtiles, on-screen-menus, etc.
Why an XID? I'm not 100% sure about it, but using a DRI buffer name directly here seems to be the better alternative.
I'm debating back and forth between two approaches here..
1) attribute w/ pixmap XID (which itself could be DRI2CreateDrawable()'d if needed, which would give you DRI buffers)
2) have some special attachment points
In the second case, it seemed a bit awkward to me, because we'd probably need a way to SwapBuffers w/ both a src and dst specified (so you aren't always swapping to the front).. (well, maybe CopyRegion is sufficient). And also, now some of the non-front buffers are RGB and not scaled vs YUV and scaled. But with the first approach it is consistent, all non-front buffers are same dimensions/format.
Also, for a generic (non-3d) client, w/ the XID attribute approach, it is just a normal drawable that they could use normal X11 draw operations on. Which seemed like a benefit.
The drawback in the first case is that it feels like a sort of strange usage of an attribute. But beyond that I couldn't think of anything else I didn't like about the approach.
Of course, as always, I'm open to any other suggestions
Are you targeting/limiting this to a particular API (or the customary limitations of overlay HW)? I ask because VDPAU allows clients to pass in an arbitrary colour conversion matrix rather than color standard/hue/sat/bri/con, so it wouldn't be possible to use this in that context.
Ideally it would something that could be used either from device-dependent VDPAU or VAAPI driver back-end, or something that could be used in a generic way, for example GStreamer sink element that could be used with software codecs.
AFAIK DRI mostly isn't a device driver dependent protocol, and the client side doesn't necessary know to which hardware it is talking, just look at how gallium 3D is working, talking with X over the DRI protocol is part of the driver independent state tracker, and NOT part of the driver itself.
So having this driver independent is just a must have, and not optional.
Hmm, ok.. then I guess we need a way to expose which approach the driver supports via attributes? And perhaps a bit more flexibility in what the value of an attribute is in order to handle a matrix in a less awkward way.. I'll think a bit about this.
BR, -R
Well, this is the goal anyways. There is one slight other complication for use in a generic way.. it would need to be a bit better defined what the buffer 'name' is, so that the client side would know how to interpret it, mmap it if needed. But I think there is a solution brewing: http://lists.linaro.org/pipermail/linaro-mm-sig/2011-August/000509.html
That's indeed true, but currently it is the only parameter that must be interpreted in a driver dependent fashion. And I really think it should stay that way.
As far as color conversion matrix... well, the attribute system can have arbitrary device-dependent attributes. In the VDPAU case, I suppose the implementation on the client side knows which xorg driver it is talking to, and could introduce it's own attributes. Perhaps a bit awkward for communicating a matrix, but you could in theory have 4*3 different attributes (ie. XV_M00, XV_M01, ... XV_M23) for each entry in the matrix.
Yes, but you should define that clearly in the protocol, we also need something to make the client side know what is supported: individual values or matrix or both?
Also in general, their compositing API is a lot more flexible and allows for a background + multiple layers, rather than just a single layer. I suppose you could pre-flatten the layers into a single one, but the background would be problematic.
Yeah, pre-flatten into a single layer, I think. I mean, we *could* push that to xorg driver side too, but I was trying to not make things overly complicated in the protocol.
I'm not sure I caught the issue about background.. or are you thinking about video w/ AYUV? Is there any hw out there that supports overlays w/ YUV that has an alpha channel? If this is enough of a weird edge case, maybe it is ok to fall back in these cases to the old way of doing the blending on the client side and just looking like a 3d app to the xorg side. (I suspect in this sort of case you'd end up falling back to the GPU on the xorg side otherwise.) But I'm always interested to hear any other suggestions.
VDPAU indeed defines two YUVA formats you can use, but currently I haven't seen anybody using the background picture functionality so far, because there are just not so much YUVA videos around.
Christian.
xorg-devel@lists.x.org: X.Org development Archives: http://lists.x.org/archives/xorg-devel Info: http://lists.x.org/mailman/listinfo/xorg-devel
2011/9/2 Rob Clark rob.clark@linaro.org:
2011/9/2 Christian König deathsimple@vodafone.de:
Hi Rob,
+ flipping between multiple back buffers, perhaps not in order (to handle video formats with B-frames)
Oh, yes please. The closed source drivers seems to do this also all the time, and I never really understood why DRI is limiting the buffers to the OGL attachment points.
I guess you will also want to talk with mplayer folks about this. I'm quite sure they will be one the first to exploit any new video rendering interface.
...
Current solutions use the GPU to do a scaled/colorconvert into a DRI2 buffer from the client context. The goal of this protocol change is to push the decision to use overlay or GPU blit to the xorg driver.
You left one corner case out, HDMI allows for the framebuffer to be in YUV 4:4:4 format. So it is possible to send YUV data to the display (usually a TV) without color conversion at all, but I think with the current state of X we are years away from that.
heh, yeah..
possibly w/ overlays, at least when fullscreen and gfx layer is not visible, we could send YUV direct to the TV.. but I'm not really sure the practicality of switching back and forth between RGB and YUV
I'm not sure that's so much out of reach. IIRC there is some foundation for non-RGB visuals in Mesa and on X server side there is push to make the screens use separate buffers rather than one huge 'desktop' buffer with the ability to exchange these buffers on the fly as needed.
Thanks
Michal
-
Alex Deucher -
Christian König -
Michal Suchanek -
Michel Dänzer -
Rob Clark