Standard version: 1.5
Minor Change
Stereo motion pictures are now common, and various visual effects applications have been updated to include a stereo work flow. Typically this is done by embedding multiple views within an image stream.
The Foundry has implemented an unpublished extension to the OFX API in its Nuke compositing application to deal with multiple views within a clip. We are proposing cleaned up versions of these extensions to the OFX 2.0 Committee for adoption in the Image Effect API. It should work along side the Deep Images proposal.
An extra concept needs to be added to the API, which is a view, this a set of planes, for example the left view and the right view.
The basic idea behind this extension is orthogonal to the plane being fetched, as well as time. So when an effect fetches image data from the host it would specify the frame time, the view and the image plane to fetch, eg: the source clip's the colour plane of frame 10 of the left view.
The extensions will need supporting machinery, so that plug-ins advertise what views they process and need to render an output and so on.
The multi-view API would be an optional extension for host applications that support stereo streams.
An image stream in our proposed scheme is a fat hierarchical bundle of image data. Each clip has multiple frames, at each frame we have a set of views each of which consists of a set of image planes. The set of planes on each view are identical.
Not all effects need to be view aware, for example a simple blur plugin does not need to know which view is it being asked to process, it simply fetches the planes it is blurring, some parameter values and renders its outputs. View aware hosts have a straight forward problem to support such plugin.
View aware plug-ins are those that explicitly use images from more than one view to produce an output image. To enable such plug-ins to work, we need to extend the image effect API in a fairly straight forward manner,
All clips are considered to have all enumerated views, so that a plugin can fetch any view from any clip. A host may only have a single view available on that clip, in which case it should return the same image for either view. Unless a plugin sets itself to be view invariant it is expected that multiple calls to render will occur for each view.
A new, optional suite, OfxImageEffectViewSuite, will be needed to support this.
Views are passed around as indices, from 0 to nViews-1. The number and name of views can be recovered via functions on the view suite.
The one dimensional integer property kOfxImageEffectPropView is often used to identify a single view on various property sets.
A plugin and host advertise their view awareness via the descriptor property, kOfxImageEffectPropViewAware. This is a one dimensional integer property, set to zero if not view aware, or one if it is.
If this property does not exist on a descriptor (indicated by any of the property manipulation functions returning kOfxStatErrUnknown) then the corresponding host should not be considered view aware.
[pierre jasmin]: This is a bit complicated, perhaps we can ignore planes for now as a first pass?
Don't we only need something like:
+ maybe we need to say we support this, I am OK if I have to add a menu (process both, left only, right only - assuming I can get both views when rendering one) Also if getViewCount == 2 - maybe we can use predefined int values: ALL=0, LEFT=1, RIGHT=2?
[Gary Oberbrunner]: Like Pierre, I think a simple interface is better if it serves the need. Handling stereo is all we really need to do here, right? Being able to get disparity from the host would be nice, but (a) some hosts, even stereo-capable ones, might not have that, and (b) most plug-ins just need the actual left and right images.
One additional comment: what if we are in direct OpenGL mode? This suite has to interoperate with that one.
—
[Dennis Adams]: There are two needs that we identified:
We addressed these in Vegas Pro 10 using the following (shipping) OFX extensions:
1. Two new properties of the OfxImageEffectActionRender action describing the view being rendered, and how many views total:
2. Callback suite to get a specific view (using the standard callback gets you the current view being rendered):
Both of these seem to fit inside what Bruno has proposed (based on how Nuke works), but using numbers instead of names for the views.
To Gary's comment, these are compatible with the OpenGL extension and can be used at the same time.
Regarding taking it past stereoscopic: we may need a convention for describing views. For example, views needed for:
All of these can be sufficiently enumerated just with “view N of M” (or “view Nx,Ny of Mx,My” for #4) but for 2 and 3 you might want to know the angle described by each view.
[pierre jasmin]: I don't think we want decimation, tiling etc in there. Just maybe Stereo and Other.
Conceptually seems we need to separate “Planes” - multi-channel from multi-view
OfxStatus (* clipGetImagePlane)(OfxImageClipHandle clip,
OfxStatus (*clipGetImageView)(OfxImageClipHandle clip,
OfxStatus (* clipGetImageViewAndPlane)(OfxImageClipHandle clip,