Hi All,
So I have rotoscoped a video of a person on white background. Please refer to the link below
In this video the frames are all spontaneous. I am not sure how to put this but In need the vector elements to be interpolated.
The transition should be smooth. Below is the example of what I am trying to achieve.
So what I want is the smooth moving effect of the individual cells/ elements. like the transition of one frame to another should be smooth.
I know that the image above was roto'ed manually and the smoothing effect was given through an inhouse application known as rotoshop. Is it possible to get the same effect in SA5. I do not have the latest version. I am not going exactly for the example(Scanner Darkly) effect but yeah would be nice to know if its possible.
Regards,
Frost
Replies
We should first give a round of applause to the very talented team of animators and the 2 (plus?) years of their lives they spent making 'A Scanner Darkly'. I really love the film and the look they achieved.
My understanding of the process is that they manually build keyframe images and then interpolate between them. Which is probably what you are referring to when you mention interpolation.
A super quick fully automatic approach to this kind of thing could be something like the video example that appears at the very bottom of this post for some weird ning reason.
I made this using a very simple 2 action step Paint Action Sequence (PASeq) in Studio Artist.
The first Simplify Ip Op action step was setup as follows.
So why did i construct it this way?
I cannot stress enough the importance of preprocessing your video source, or your fixed source image if working in 2D.
And if you want something 'smooth', you want to get rid of the noise and extraneous detail before running a more elaborate effect on it.
There are various approaches to doing keyframe interpolation in Studio Artist. I'll try to put together some examples of different approaches and post them here over the next few days as time permits.
Anything you manually draw as paint action steps in a Paint Action Sequence (PASeq) can be keyframe animated.
You can also keyframe animate bezier paths or sets of bezier paths.
It's interesting to dive into how the animation example you showcased is put together.
So it would appear that the background was constructed first, and then the movement associated with a few image elements is laid down on top of that.
You can certainly do that kind of thing in Studio Artist. You also don't necessarily have to use multiple layers to do it, since you could just think of it as the ordering of elements within a PASeq applied to a single canvas layer.
I wonder if we couldn't use a different type of comparison between frames to create a mask of the image (compare the current frame with the previous frame), allowing only the changing areas to be processed (applying an alpha mask to the current frame). If you then take the current frame after processing, apply the same mask over the previous frame, it seems to me there should be background consistency.
How would I apply this multi-frame processing in SA?
Sure, you could come up with various approaches to doing this inside of a Paint Action Sequence (PASeq). I've done something similar in the past where i use something like the Vectorizer, then compute the error between the vectorizer output and the source it is working off of, then take that error image and vectorize that, then add the 2 together. I think i've also tried some things very similar to what you are proposing with movie processing. You can use Temporal Ip Op effects to generate the mask, or you can do it manually using another layer as the buffer for the previous frame. You then route the difference mask to the selection buffer, then you mask some process (vectorizer or ip op effect or whatever) with the mask when you run it. everything i just described can be done in a PASeq in Studio Artist.
I'll try an experiment or 2 later today to show you what i mean and post what i did here when i get some time.
I expected you could do it. I just haven't figured out how to deal with masks, buffers etc in SA yet. I'm used to node-based video processing, so it's taking me a while to learn the UI.
Any example you can give would be largely appreciated.
Studio Artist is more like a Turing machine, if you are looking for some kind of conceptual model for how it works, especially when considering things like Paint Action Sequences (PASeq).
You can view the current canvas layer and other hidden layers like registers. Actions in a PASeq take input from somewhere (current layer, source, hidden layer, selection buffer, style, etc), process it in some way, then the results are placed somewhere (current layer, source, hidden layer, selection buffer, style, etc).
MSG (modular synthesized graphics) works the same way. Processors in a processor chain list take input from somewhere (image streams, color palettes, color gradients, FSA buffers), run some kind of effect or processing action, then the results of that action are placed back somewhere (on the Stream Bus internal to the MSG processing system).
A node based system still has to run actions associated with the various nodes in a sequence somewhere, probably hidden from you. Studio Artist is very focused on providing an environment that allows for easy construction of recursive processing effects. A Turing machine like processing model makes that very easy to do.
We didn't actually set out to implement a Turing machine at the conception of the program, but if you analyse what is going on you can see that is really the conceptual structure in a lot of ways. The original design ideas were very much about trying to take ideas from music synthesis and how synthesizers work internally and reconceputalize them for visual artists. One of our beta testers pointed out the Turing machine analogy in the context of building bezier cellular automata effects, and what he said really clicked in my brain, it really is the underlying model if you are trying to grok what a Paint Action Sequence is really all about.
I'll post more on how to build the frame difference masking later.
So here is a single frame image where i used the Color Quantize Ip Op effect for creating a flat color region effect in video.
You can see that you kind of trade off one issue for a different one (if you goal is super smooth temporal processing). You can see the video on my art blog.
As a special effect in its own right this is kind of interesting. You get this smear effect.
Maybe if i did a better job on the temporal difference mask it would be different. I'll play around some more, but this is kind of what i remember from previous experiments with this kind of thing.
So how did i generate the effect (and the associated temporal masking). You can see in the PASeq below.
The first T_Difference1 Temporal Ip Op effect generates a frame difference mask. The specific effect settings i used are shown below.
I'm building the matte from 3 frames (as opposed to just 1). Note that the Composite parameter is set to Replace Region Selection, so this is the mechanism i'm using to take the output of the temporal effect and send it to the region selection buffer rather than the current canvas layer (Replace would do that).
I then used a Rank Area Filter Ip Op effect to widen the selection mask a little bit (as shown below).
Note that i'm using the Max algorithm (i want to spread out the white selected areas a little bit). Also note the Ip Source and Composite parameters are configured so that the contents of the region selection buffer are routed through the Max Rank Filter effect and then back to the region selection buffer.
You might wonder why i used the 3 frame difference mask with some additional widening, so let's take a look at using just a single frame difference and muting the Rank Area Filter below 32 frames in.
You get a different much more noticeable kind of fringing if you use the super tight temporal mask.
Now one could make an argument that the temporal masks being generated by the stock T_Difference1 temporal ip op are not ideal. So we could build a different temporal mask from scratch. And if you try and put something a little bit more elaborate together you get the following 74 frames in.
So the fringing in this one is much better (just a very low frequency background color shift variation if you look closely). The video for this run is also available on my art blog in the same post as the link above.
The approach i took to create the more elaborate temporal difference mask is shown below.
The first 7 non-muted action steps build the temporal difference mask. The Color Quantize Ip Op action step is generating the flat cartoon coloring. The Fixed Image routes the Style to the canvas. And the Canvas to Style routes the canvas to the Style.
So we are using the Style as a swap buffer in the PASeq. I could have used a second hidden layer, but it's way easier to do it this way (even easier in the next version we're working on since you can route any effect output directly to the style rather than putting it in the canvas and then routing it there).
The reason why i did the style-canvas-style swap trick is because i am generating the temporal difference mask in the canvas in the first 5 action steps, then i put that canvas in the selection buffer, and then i run the original Max Rank Filter effect we used above directly on the selection buffer. The first 2 Rank Area Filter action steps are median filters to try and clean up noise in the mask.
I tried a different experiment with the more elaborate temporal difference mask action steps i discussed in the previous post here. I used the Mondo Simplify Ip Op effect instead of the Color Quantize Ip Op effect for the cartoon styling.
Note that i have the Random parameter set to Use Seed rather than Unique. For any video processing like this you typically want to be sure to do that (helps reduce flicker from variations in the effect).
You can also note that the characteristics of the region patterning is different with this Mondo Simplify Ip Op effect as compared to the region patterning you get with the Color Quantize Ip Op effect. A little bit more like the patterning in the hand animated movie, but still different. The hand animated one has a very particular way they do the color gradation shading.
I also posted this effect without and with the temporal selection masking on my art blog. The temporal difference masking as implemented reduces the flicker in the background, but introduces noise artifacts in the face that i would probably consider worse for this kind of video processing effect.
Could you generate a better temporal difference mask for this particular source movie? Probably. I'll try some other variations as time permits.