Interesting topic, I have been part of Khronos work for quite some years but not anymore. I would not assume that Khronos as a standards body have any view on how the implementation is done since they want to be agnostic to OS etc. But I don't get your comment on why does OMX handle hardware, OMX main purpose is to handle multimedia hardware and offer an interface to that HW that looks identical indenpendent of the vendor delivering that hardware, much like the v4l2 or USB subsystems tries to do. And yes optimally it should be implemented in drivers/omx in Linux and a user space library on top of that. But a drivers/omx solution would set it to directly compete with v4l2 in Linux and that is not a good option, hence the needed enablers should be included in v4l2 etc instead. And it would need to be able to handle DSP/ISP firmware/load-modules etc. It might be that some alignment also needs to be made between 4vl2 and other OS's implementation, to ease developing drivers for many OSs (sorry I don't know these details, but you ST-E guys should know).
About your second premise, this have never been express inside Khronos by companies or the body (not at least so I have heard it) but might rather be how some companies manage their OMX implementation. I would say that for the most part the standards are designed to work alone and are developed in very seperate groups. But it would be foolish to design APIs that don't fit between each other inside the same standards body, hence cross API work is done and the nice SW stack pictures are drawn ;-).
By the way IL is about to finalize version 1.2 of OpenMAX IL which is more than a years work of aligning all vendors and fixing unclear and buggy parts. Hence, this makes this as a good candidate to find common ways of exposing HW that is implemented differently but have the same functionality. There is no point in exposing similare HW in many differnt ways, that just make it harder for users.
/Harald
-----Original Message----- From: Hans Verkuil [mailto:hverkuil@xs4all.nl] Sent: den 9 februari 2011 20:07 To: Linus Walleij Cc: linaro-dev@lists.linaro.org; Lee Jones; ST-Ericsson LT Mailing List; Harald Gustafsson Subject: Re: [st-ericsson] v4l2 vs omx for camera
On Wednesday, February 09, 2011 18:11:22 Linus Walleij wrote:
On Tue, Feb 8, 2011 at 9:30 AM, Lee Jones
lee.jones@linaro.org wrote:
Robert, Linus, what say you?
[I'm looping in Harald from Ericsson who worked with
Khronos so he can
correct me for all inevitable mistakes in trying to understand how Khronos works.]
I mainly come from the kernel for kernels' sake side of
things and my
remarks are basically the same as Arnd's: what is OMX doing
handling
hardware? If the OMX stuff *did* have some business with
the hardware
it should be a drivers/omx or lib/omx subsystem and everyone could make use of it from there and that's it. SCSI and VGA and whatever hardware protocol is in the kernel after all.
That's the simple answer. However I also know from
experience why the
Khronos Group folks don't see things this way.
Their problem is this: a plethora of operating systems, including Linux, Symbian, Windows Mobile, ITRON, VxWorks, QNX, OSE and a boatload of internal NIH systems. Some of them with nonexistant of half-brewn frameworks to handle the stuff that is done in kernel subsystems like V4L2, /media, or /sound (ALSA SoC). Their
objective is
to unify media pipelines across these desperately heterogeneous environments.
First premise of Khronos and OpenMAX: by describing very
strict APIs
for multimedia streams they set a stage where different
companies can
develop one implementation each and compete with each other. So, define API and compete by implementation.
Second premise: for each implementation, develop once, deploy everywhere. So each company create a large body of code, which they port to every OS they need to support. So that one of these OS:es happens to be Linux is just a technical detail. For OpenMAX
it's just
one world where the important stuff (i.e. the portable code and the Open* APIs) can live and thrive.
These basic premises permeates the work in Khronos so
deeply that it
becomes a paradigm or ontology if you like, it defines the world in which Open* development takes place. If you buy into one part of Khronos APIs you usually assume
an idea of
the world where all is done the Khronos way with no
compromises. E.g:
for OpenMAX you know that OpenMAX IL, AL and DL are
different things
but surely you expect that you will implement all of it down to the very last bit. (Yes, I know that this does not happen in
practice, but
such is the Khronos ontology.)
Some may think that OpenMAX is just an obtrusive layer of
APIs adding
nothing but complexity. If you live in an all-Linux world where say GStreamer and ALSA, V4L2 APIs go hand in hand this is true.
But this
is not the world where Khronos lives. Anyway, that's a separate subject. Let us say if Linux and GStreamer were dominating
the world
there would be no need for OpenMAX. Likewise if say Window Mobile dominated the world and had some framework for multimedia for that matter. Another implicit premise of the OpenMAX work is
obviously that
there will always be many OS:es involved in this game.
OpenMAX does not really conflict with the kernel way of doing business at all. It's natural place is within stuff like
GStreamer or
Android's counterpart, where they may or may not have some
perfect fit
- this is a matter of debate, admittedly the Linaro
multimedia working
group spend some time on it. The way I understand what they do is that they implement
interfaces in
say GStreamer like gst-openmax & Bellagio so that the big portable OpenMAX shebang that every vendor create can easily integrate into Linux.
The problem comes when you have a few DSP/ISP processors
and need some
kernel services, and then there's this V4L2 API you're
supposed to use
to get things done in the middle of your world, and in some
parts it's
not even fully defined, things in OpenMAX have no counterpart that maps 1-1 to V4L2 for example. From an OpenMAX point of view that's just an obstacle making your code bundle less portable to
VxWorks and
heterogenous, requiring new .c files and #ifdefs and whatever.
So naturally, if you can get away with implementing, say, your own userspace driver for video or audio sinks, and this can be
reused on
VxWorks and ITRON, you're happy because that means less hazzle. (In theory.)
The fact that the Linux community prefer a V4L2 API for e.g. a camera is totally irrelevant from an OpenMAX point of
view, that's
just a technical detail of Linux. Naturally, the Khronos mindset is that apps that need photographic images need to set up the
OpenMAX IL
chain and preferably conform to the OpenMAX AL API. However that is then handled behind the API is none of OpenMAX' business.
The goal of the portable OpenMAX is to make things like V4L2 become irrelevant to application developers. Compare how say Qt in theory makes the fact of whether you develop for Linux, Symbian or Windows irrelevant. There is not one word the specs that mandates
that you use
the native operating system APIs or frameworks behind the scenes at all, not even in the lowermost DL layer. (There is some portability glue in OpenKODE though, you can compare it to the portability code found in stuff like glib or nspr.) Anything providing the services will do. Needless to say, I don't think this is the view of say the V4L2 or ALSA developers. Or even the TI people working on drivers/staging/tidspbridge.
I would say that the problem is exactly this: both are
defining how to
do things with mediastreams, but the bridge inbetween is undefined, which makes for all kind of strange ad hoc solutions,
usually the most
comfortable one, from the implementers point of view, meaning they signed up for doing the OpenMAX API so they recognize that
they have
to develop and deploy this, but some of them think, that
even though
they are members of Linaro, they didn't sign up for the
ALSA or V4L2
APIs. Is that really so?
I leave the question open, because there was an inflamed debate on another very related subject the other week, which was that
of OpenGL
ES vs 3D hardware API's like DRI/DRM. Again a Khronos API
and similar
ideas of portable, proprietary, competing code bodies under a standardized API, and no reuse of the community interfaces.
I don't know if Khronos and Linaro could talk about this,
say come up
with something like "if you're member of both you're
actually supposed
to use the Linux APIs when implementing Khronos stuff" that kind of high-level politics is not my department. Our problem is that right now the inverse of that statement is assumed. Couldn't we
atleast say
that the native Linux APIs are "preferable" or something like that?
I agree with your mail.
Under linux APIs like ALSA and V4L give access to the multimedia hardware. So if you want to have your hardware drivers merged in the linux kernel, then those APIs are the way to go. OpenMax drivers will never be merged. Instead, the OpenMax framework should use the V4L/ALSA drivers to access the hardware.
Exceptions are DSPs/processors. While it is definitely possible to use V4L2 there as well, in practice I don't see this happening anytime soon. It would be a very interesting experiment though.
Of course, if you don't care about getting drivers in the kernel, then I won't stop you from using OpenMax. Personally I think that attempt to write a generic framework like OpenMax is highly problematic due to the wildly different video hardware implementations and the many different types of features.
BTW, if there are features missing in V4L2 that are needed to write an OpenMAX framework on top of it, then please let us know.
Regards,
Hans
-- Hans Verkuil - video4linux developer - sponsored by Cisco