(sept 23, 2016) - this is to be simultaneously addressed with caching and multi-processing.
Currently only one kind of plug-in instance is recognised, this is used for everything, interface, caching state, rendering etc… Many host applications have severe problems with this model, as they typically have separate render objects that are either lightweight special purpose render objects, or cloned on demand copies of the effect.
To deal with the current API, they either have multiple instances of a plug-in around, continually syncing data from the persistant/UI version to the render version, or they do this on demand. This often leads to sub-optimal performance and a great deal of complexity.
Plug-ins also find this difficult, as they often cache data in an instance, expecting to be able to re-use that data on the next render (eg: motion vectors from a retimer). In a clone on demand host, as the instance is continually being re-created, this cached data is always being destroyed once the temporary render instance is released.
A good way to manage this would be to have the API support a special purpose lightweight render instance object. The full effect instance is asked to populate one of these with any data it needs for the render action (and related actions such as RoD/RoI?), this object is then called to do the actual pixel pushing. This decouples object persistence and interface from processing, and allows for better multi threaded behaviour.
This is a major and complex change to the API and needs to be thoroughly discussed between host devs and plug-in devs.
As plug-ins are currently caching data between renders within the persistent instance (as in our retimer example), we still need to allow a plug-in to cache data in some manner. A better way all round would be for an explicit caching API, which a render object or persistent object could both commit and retrieve data to a host managed cache in a thread safe manner.
In Natron 3 we clone all effects in the render tree for each time and view (as in the multiview extension) passed to the render action.
All actions that are typically called on a render threads are instead called on each of these clones.
Each clone gets a set of cloned parameters where their value is cloned from the main instance parameters for the requested time and view. These means that we can guarantee that a value will not change between 2 actions called for a single same render.
In case of an animated parameter, if the plug-in calls getValue instead of getValueAtTime, we are able to recover the current time of the current action of the plug-in because each ImageEffect clone was created to render a specific time.
Cloning effects allows to partially resolve the "loss of context" of the host when calling suite functions. In our case, we found out that it almost completely removes the need to use thread local storage on the host side, except for the multi-thread suite and strings storage.
In order for this to work nicely, it is important that the render clone does not perform expensive operations in the create instance action: it should only fetch its parameters.
A possible way to do it would be to have an action similar to the create instance action, but that would be specific to create an ImageEffect render instance, where the plug-in is expected not to do more than fetching parameters. This action should also get a pointer to the main-thread instance so that the render instance can nicely cache data in a convenient way.
Specification:
handle handle to the plug-in instance, cast to an OfxImageEffectHandle
inArgs : Contains the following property:
/** @brief Pointer property x1 pointing to the main-thread OfxImageEffectHandle
This action is similar to the kOfxActionCreateInstance action except that the ImageEffect is only to be used for render actions. This should be the first action called on a render thread so that the ImageEffect instance can have a consistent state throughout all actions of the render threads.
It is expected that this action only fetches its parameters.
The kOfxImageEffectMainInstanceHandle can be used to cache or retrieved cached data on the main-thread instance.
(e.g. a new *OfxPropExtraInstanceData per frame rendered) - my example: If I have a 3D table of 8192 floating point per column I collected by pressing a button and fetching an image or a file and this won't change, I would like to maintain that on InstanceData and ExtraPerFrameInstanceData could just have a pointer to the pointer in my InstanceData, as opposed to copy it all, no big deal for me to wait for render queue ended signal if I need to write to this again - and yes this can be stored in custom param with synch private data in project file for project load and save.
Pierre Jasmin | 9:29 pm, 24 Sep 2016
After a discussion on our side we propose the following behaviour:
- getParameterValuesAction: Has in inArgs property set the same arguments than passed to a render action (time, view, render scale etc...)
In outArgs allocates a structure created on plug-in side with all parameter values fetched into it that are going to be needed in any call of render actions, i.e:
The outArgs contain a kOfxImageEffectParamValuesHandle property of type pointer x1 which points to the struct returned by the plug-in.
If the pointer property is set the host must call the destroyParameterValuesAction to let a chance to the plug-in to clean-up the struct it allocated.
All the inArgs of the render actions aforementionned above get the new property kOfxImageEffectParamValuesHandle which can be null or not.
If non null the plug-in must make use of the values in the struct pointed to bykOfxImageEffectParamValuesHandle. Otherwise the plug-in can make use of regular calls to getValue/getValueAtTime on its parameters.
- destroyParameterValuesAction: In inArgs the kOfxImageEffectParamValuesHandle pointer property pointed to a struct previously allocated by getParameterValuesAction. Any call to getParameterValuesAction must have a matching call to destroyParameterValuesAction.
Typically the host would call getParameterValuesAction before calling any other render action listed above and when done would call the corresponding destroyParameterValuesAction.
This lets a chance to the plug-in to have a consistant state of values throughought the render of a frame for all actions used.
Without this extension the host has to come up with its own sort of cloning or locking of parameters which is a lot less efficient.
If all the getValue/getValueAtTime functions can have in inArgs an optional context pointer passed by the host, the host can then be aware of all the values fetched by a plug-in during the getParameterValuesAction. This can help the host in the purpose of caching.
The only unresolved point by our proposition is to address the parametric parameters extension: the current description of the curve is in no way enough for the plug-in to know what kind of spline and interpolation is used on the host side. It makes impossible for a plug-in used parametric parameters to use the getParameterValuesAction.
We would have to come up with a ParametricSuiteV2 describing curves with intepolations methods used by the host in order to support them in this new action.
Alexandre | 8:17 am, 15 Sep 2016
Phil Barret : Baselight generally renders in a different process, and very often on different hosts, so cloning render objects won't help me. But an
API
to reduce the amount of data streaming (e.g. declaring that some/all params will only be read at the render time, so I don't need to stream the animation) would be welcome.
Bruno Nicoletti : In Nuke, we definitely need light weight clones of the main instance, which are created/set single threaded, but get to render image data multi threaded. The setting is basically a whole load of getParameterAtTime calls pumping values into local variables. Your host could track what the lightweight clone requests on the main host and only send that data over to each of the nodes in the cluster for each of their clones. A bit of messing around, but it should work. Otherwise a serialisation of the render objects would be needed to satisfy us both.
Pierre Jasmin : If the need is to create a render instance that does not carry around the whole animation system in case + the equivalent of PF_PreRenderExtra in AE
API
(DOD,ROI) Looking at our base class, the non-trivial part is parameters. One idea would be to add an additional state to a parameter at creation in param suite V2 :
0. : I will only read this param at current time
N: always read a time window of N frames from current time (and at integer times?)
-1: randomly request param values at any time (then you would need to strore the whole parameter over time in your instance)
Not as efficient but this would get you there most of the time and then paramGetValue(AtTime) would not need to changed in render calls, and another action would not be needed just to store parameters per frame.
In Natron 3 we clone all effects in the render tree for each time and view (as in the multiview extension) passed to the render action.
All actions that are typically called on a render threads are instead called on each of these clones.
Each clone gets a set of cloned parameters where their value is cloned from the main instance parameters for the requested time and view. These means that we can guarantee that a value will not change between 2 actions called for a single same render.
In case of an animated parameter, if the plug-in calls getValue instead of getValueAtTime, we are able to recover the current time of the current action of the plug-in because each ImageEffect clone was created to render a specific time.
Cloning effects allows to partially resolve the "loss of context" of the host when calling suite functions. In our case, we found out that it almost completely removes the need to use thread local storage on the host side, except for the multi-thread suite and strings storage.
In order for this to work nicely, it is important that the render clone does not perform expensive operations in the create instance action: it should only fetch its parameters.
A possible way to do it would be to have an action similar to the create instance action, but that would be specific to create an ImageEffect render instance, where the plug-in is expected not to do more than fetching parameters. This action should also get a pointer to the main-thread instance so that the render instance can nicely cache data in a convenient way.
Specification:
handlehandle to the plug-in instance, cast to anOfxImageEffectHandleinArgs: Contains the following property:/** @brief Pointer property x1 pointing to the main-thread OfxImageEffectHandle
*/
#define kOfxImageEffectMainInstanceHandle "OfxImageEffectMainInstanceHandle"
outArgsis redundant and is set to NULL.This action is similar to the kOfxActionCreateInstance action except that the ImageEffect is only to be used for render actions. This should be the first action called on a render thread so that the ImageEffect instance can have a consistent state throughout all actions of the render threads.
It is expected that this action only fetches its parameters.
The kOfxImageEffectMainInstanceHandle can be used to cache or retrieved cached data on the main-thread instance.
#define kOfxActionCreateRenderInstance "OfxActionCreateRenderInstance"