×

Terminal Output

  • Welcome
  • Standards Discussion
  • Contact
  • Ofx small
  • Welcome
    • Why a Standard?
    • For Implementers
    • Association Business
  • API Documentation
    • API Reference
    • Programming Reference
    • Programming Guide
  • Standards Discussion
  • Contact
Back to standard change list


Multi View Effects Proposed

Standard version: 1.5

Minor Change

Stereo motion pictures are now common, and various visual effects applications have been updated to include a stereo work flow. Typically this is done by embedding multiple views within an image stream.

The Foundry has implemented an unpublished extension to the OFX API in its Nuke compositing application to deal with multiple views within a clip. We are proposing cleaned up versions of these extensions to the OFX 2.0 Committee for adoption in the Image Effect API. It should work along side the Deep Images proposal.

An extra concept needs to be added to the API, which is a view, this a set of planes, for example the left view and the right view.

The basic idea behind this extension is orthogonal to the plane being fetched, as well as time. So when an effect fetches image data from the host it would specify the frame time, the view and the image plane to fetch, eg: the source clip's the colour plane of frame 10 of the left view.

The extensions will need supporting machinery, so that plug-ins advertise what views they process and need to render an output and so on.

The multi-view API would be an optional extension for host applications that support stereo streams.

An image stream in our proposed scheme is a fat hierarchical bundle of image data. Each clip has multiple frames, at each frame we have a set of views each of which consists of a set of image planes. The set of planes on each view are identical.

Not all effects need to be view aware, for example a simple blur plugin does not need to know which view is it being asked to process, it simply fetches the planes it is blurring, some parameter values and renders its outputs. View aware hosts have a straight forward problem to support such plugin.

View aware plug-ins are those that explicitly use images from more than one view to produce an output image. To enable such plug-ins to work, we need to extend the image effect API in a fairly straight forward manner,

  • plugins and host need to advertise their view awareness,
  • plugins need to be able to ask how many views are present,
  • when fetching an image, a view needs to be specified,
  • plugins need to state what input views are needed to produce an output view,
  • at render, a plugin is told to produce a set of planes, at a specific time for a specific view.

All clips are considered to have all enumerated views, so that a plugin can fetch any view from any clip. A host may only have a single view available on that clip, in which case it should return the same image for either view. Unless a plugin sets itself to be view invariant it is expected that multiple calls to render will occur for each view.

A new, optional suite, OfxImageEffectViewSuite, will be needed to support this.

Identifying Views

Views are passed around as indices, from 0 to nViews-1. The number and name of views can be recovered via functions on the view suite.

The one dimensional integer property kOfxImageEffectPropView is often used to identify a single view on various property sets.

View Awareness

A plugin and host advertise their view awareness via the descriptor property, kOfxImageEffectPropViewAware. This is a one dimensional integer property, set to zero if not view aware, or one if it is.

If this property does not exist on a descriptor (indicated by any of the property manipulation functions returning kOfxStatErrUnknown) then the corresponding host should not be considered view aware.

Back to standard change list


Discussion

Comments

[pierre jasmin]: This is a bit complicated, perhaps we can ignore planes for now as a first pass?

Don't we only need something like:

typedefstruct
               OfxMultiViewImageEffectSuiteV1
               {
               OfxStatus
               (
               *
               clipGetMultiViewImage
               )
               (
               OfxImageClipHandle clip
               ,
               OfxTime
               time
               ,
               int
               iview
               ,
               OfxRectD
               *
               region
               ,
               OfxPropertySetHandle
               *
               imageHandle
               )
               ;
               OfxStatus
               (
               *
               getViewCount
               )
               (
               OfxImageEffectHandle instance
               ,
               int
               *
               nViews
               )
               ;
               // 2 when stereo
               OfxStatus
               (
               *
               getViewName
               )
               (
               OfxImageEffectHandle effect
               ,
               int
               view
               ,
               char
               **
               viewName
               )
               ;
               }
               OfxMultiViewImageSuiteV1
               ;
            

+ maybe we need to say we support this, I am OK if I have to add a menu (process both, left only, right only - assuming I can get both views when rendering one) Also if getViewCount == 2 - maybe we can use predefined int values: ALL=0, LEFT=1, RIGHT=2?

[Gary Oberbrunner]: Like Pierre, I think a simple interface is better if it serves the need. Handling stereo is all we really need to do here, right? Being able to get disparity from the host would be nice, but (a) some hosts, even stereo-capable ones, might not have that, and (b) most plug-ins just need the actual left and right images.

One additional comment: what if we are in direct OpenGL mode? This suite has to interoperate with that one.

—

[Dennis Adams]: There are two needs that we identified:

  1. Some stereo-aware plug-ins would like to know which view it is currently processing.
  2. Some stereo-aware plug-ins would like access to frames from views other than the one it is processing.

We addressed these in Vegas Pro 10 using the following (shipping) OFX extensions:

1. Two new properties of the OfxImageEffectActionRender action describing the view being rendered, and how many views total:

/** @brief Indicates the number of the view to render

       - Type - int x 1
       - Property Set - Render In Args (read only)
       - Valid Values - 0 (standard), 0 (left stereoscopic processing), 1 (right stereoscopic processing)

    */#define kOfxImageEffectPropRenderView OfxImageEffectPropRenderView/** @brief Indicates the number of total views to render

       - Type - int x 1
       - Property Set - Render In Args (read only)
       - Valid Values - 1 (standard), 2 (stereoscopic processing)

    */#define kOfxImageEffectPropViewsToRender OfxImageEffectPropViewsToRender

2. Callback suite to get a specific view (using the standard callback gets you the current view being rendered):

/** @brief Name of the Vegas stereoscopic image effect suite (used to get a clip image from the
               either view (otherwise getClipImage by default returns the current image)
    */#define kOfxVegasStereoscopicImageEffectSuite OfxVegasStereoscopicImageEffectSuitetypedefstruct
               OfxVegasStereoscopicImageEffectSuiteV1
               {
               OfxStatus
               (
               *
               clipGetStereoscopicImage
               )
               (
               OfxImageClipHandle clip
               ,
               OfxTime
               time
               ,
               int
               iview
               ,
               OfxRectD
               *
               region
               ,
               OfxPropertySetHandle
               *
               imageHandle
               )
               ;
               }
               OfxVegasStereoscopicImageSuiteV1
               ;
            

Both of these seem to fit inside what Bruno has proposed (based on how Nuke works), but using numbers instead of names for the views.

To Gary's comment, these are compatible with the OpenGL extension and can be used at the same time.

Regarding taking it past stereoscopic: we may need a convention for describing views. For example, views needed for:

  1. Stereoscopic (left and right).
  2. Autostereoscopic displays (I've seen these with 4 to 15 views, probably more someday).
  3. 360 displays like Sony RayModeler ( http://www.youtube.com/watch?v=6BFKC-NKRFw ).
  4. Multiple views in both X and Y view directions?

All of these can be sufficiently enumerated just with “view N of M” (or “view Nx,Ny of Mx,My” for #4) but for 2 and 3 you might want to know the angle described by each view.

[pierre jasmin]: I don't think we want decimation, tiling etc in there. Just maybe Stereo and Other.

Conceptually seems we need to separate “Planes” - multi-channel from multi-view

OfxStatus (* clipGetImagePlane)(OfxImageClipHandle clip,

                                OfxTime               time,
                                const char*           plane,
                                OfxRectD*             region,
                                OfxPropertySetHandle* imageHandle);

OfxStatus (*clipGetImageView)(OfxImageClipHandle clip,

                    OfxTime       time,
                    int           iview,
                    OfxRectD     *region,
                    OfxPropertySetHandle   *imageHandle);

OfxStatus (* clipGetImageViewAndPlane)(OfxImageClipHandle clip,

                                OfxTime               time,
                                int                   view,
                                const char*           plane,
                                OfxRectD*             region,
                                OfxPropertySetHandle* imageHandle);
unattributed | 9:49 am, 8 Mar 2014
Back to standard change list
  • OFX @ Github
  • Association Information

Copyright ©2023 The Open Effects Association