I keep noticing that almost no one is approaching video processing in Studio Artist the way it's setup to really do it. And i think part of the problem is that folks are used to working in other video programs like FCP, AFTEFX, PRIMR etc, where their whole design is built around this notion of
'process an input frame into an output buffer - write out the buffer and wipe it clean for the next cycle in the loop'
Which is fine for what those programs are designed to do.
But at its core since day one (back when dinosaurs roamed the earth and 400 MB hard disks were the size of toaster ovens), Studio Artist has been about digital painting. And i don't mean silly drag a single nib image around as a brush kind of 1988 digital painting (although we are more than happy to let you do that all day in Studio Artist if you so desire).
Real world media has more finesse to it. More complexity. More chaos inherent in the system. You inherently get your hands dirty when you even mess with it. Things leak and spill all over the place.
And we wanted to build a digital system that behaves with the same level (or more) of rich organic aroma. If you can visualize that 'rich aroma' in a visual manor. Part of the fun when working with real art stuff is that it has a mind of its own. And if you follow that, you can end up in all kinds of unexpected really amazing places.
Or you can ignore it and just concentrate on the outcome. It looks vibrant (if that's what you're going for, again we try not to judge and limit what you can do, because of some mistaken pre-conceived notion of what is a proper visual aesthetic).
And we certainly aren't going to limit you to some even more limiting pre-conceived notion of what 'natural media' is or isn't. A mistake a lot of other digital art programs make. Are actually very religious about in fact.
And the feedback we get from visual artists about our approach confirms that. Oftentimes they are way more interested in a 'hyper digital natural media' kind of thing. Maybe it's reminiscent of some specific kind of natural media marking system, but its wildly unique and digitally imaginative, fresh and distinctive at the same time. They like the fact that it is its own thing.
Responsiveness is of course always important. It's at the heart of Studio Artist's internal parameter modulation structure. Real world art stuff is very responsive. To the touch. To the temperature of the room. To the characteristics of what you are trying to apply it too. To the bumps on that stick you are dragging around with paint on it. And we try to do the same (or more). It's built into the system from the bottom up. Parameter modulation everywhere, and lots of it. More and more all the time.
So we've always cared about painting. From day one.
But very quickly we also started looking into using Studio Artist to process movie files. It was built around Quicktime from day 1 (Quicktime being a cool and revolutionary thing back in the day, now extinct like the Dodo or Carrier Pigeon, killed by an evil entity).
But in the very beginning our painting focus was very much 'let's paint on a canvas'. Canvas being this 2D digital surface.
And Studio Artist also incorporated from the very, very beginning this whole notion of an artist painting a model source onto the canvas. And we wanted Studio Artist to look at and analyze the model source just like a human artist would do. So we built visual modeling into Studio Artist from day one as well. Human visual modeling based on cognitive neuroscience research that looks into how the visual cortex is structured, how it processes its input signals. And also theoretical computational visual models of what various folks thought was going on in those visual areas inside of the visual cortex.
We also incorporated AI Heuristics into the system as well. Lots of them, more and more all the time. If you want to think of that part as a rule based AI system based on expert knowledge from artists, sure why not. Different kinds of AI for different tasks (or mixed up together in the same tasks).
I think we need to make it clear that our business is making visual art. And enhancing what visual artists can do in the digital world. Building an extremely flexible creative tool for them to use in the digital art world. We're not trying to write a NIPS paper, where we present the perfect algorithm for some heavily restricted task run on a standardized data test set. We're trying to help people make art.
So anything goes really. We play loose and fast with some aspects of the modeling when we have to (computational limits are always there, or maybe a really bad model actually looks more interesting). And we love mistakes. Because oftentimes they look really really amazing. So why would you remove them from the program if that is the case. You just tie them to a specific parameter setting. Because anything that looks visually different is interesting, worth keeping around, something to use in your tool chest of visual effects. We aren't going to prevent you from using that, just because it was the result of a programming error (crashes are a different story, we're talking about algorithm mistakes that lead to interesting visual results).
Now the thing about painting is that there is this inherent dynamic tension between applying pigment to a surface, and the dissipative action of the fluid that holds and spreads that pigment. The fun stuff is in the interactions going on in the system. Pigment and fluid mixing. That amalgamation of pigment and fluid interacting with a surface. A textured brush or other applicator object interacting with all three. Time interacting with that whole system of discrete elements as the fluid dries out, or the surface gets wet, or doesn't.
All of that stuff interacting together is what generates all of the fun. Generates the visual fun of the end result. Generates the fun of driving that stuff applicator over the surface (over time), to create something interesting (one hopes).
So there is a very definite dynamic tension going on in the system between a dissipative effect, and a marking 'lay down stuff' application effect. They fight each other for control. So you want to be aware of that, and take advantage of that in your work.
And work is a real loose term. Because Studio Artist has always bridged these 2 product categories that seem to be kept completely separate by wizened marketing folks at other entities that release software into the world. The universe of 2D graphics programs. And the universe of video programs. And never shall the twain meet, except at the file : import / export level.
And when you start to look at video processing, movie file processing, digital rotoscoping, paint animation, whatever you want to call it, you want to make sure you apply the same set of principals you apply in the 2D 'spread it on the canvas' world to the temporal time-based 'frames in a row' video or movie canvas. You want to make sure you keep that rich and powerful dynamic tension going on. The dynamic tension between the dissipative forces, and the 'lay down the marking stuff' forces.
You really want to think of your video processing as 'painting in time'. And when you do that, not only will your creativity be expanded 10 fold or more, you will also start creating some really interesting visual results because of it. Visual results that happen over time. Because you are painting over time. You are building a temporal canvas.
And how did we say painting works? You lay down 'marking stuff' onto a surface. And some kind of 'fluid' is interacting with that stuff, and also interacting with the surface you are applying it all too. But the surface is a series of consecutive frames laid out over time. The 'fluid' (whatever that term even means, do not get literally or rigidly attached to it, remember anything goes when we are talking about art), fluid interacts over time as well. Interacts with those consecutive frames, and the consecutive laying down of stuff, and with the remnants of itself, all interacting over time.
Time is an essential component to the whole process. That's true for 2D painting, or time based painting. Time in 2D painting can be thought of as movement. Movement occurs over time. Movement builds up the final painted image. The dynamic tension between dissipative forces and marking forces on a surface occurs over time based on movement. That's what painting is really all about.
Painting on a time canvas has both kinds of movement over time. There is movement on the surface, or over time, or both going on simultaneously. But the canvas itself is time based (discrete elements of time we call frames).
Now once you wrap your head around this notion of video processing being painting over time, then you are free to envision it in a very different way than what is laid out for you in the conceptual design of those other video programs.
One key point is that you should really be thinking about painting on top of the last output frame. As opposed to wiping the slate clean and starting fresh with every new frame. Because if you do that (wipe the slate clean) you have immediately lost one of the main places where you can create dynamic tension between dissipative forces and marking forces, a dynamic tension that occurs over time, over multiple frame images. You really want to take advantage of this, capitalize on it, use it in your artistic process.
You also need to think heavily about what 'dissipative forces' means to you. Because how you approach that part can really help to define your visual style, make it unique, make it fresh, make it distinctive. We're going to educate you on some approaches. But you should really learn them, understand the mechanics of how they work, and then head off into the wilderness, boldly exploring where no one has gone before. Or maybe re-exploring what has been very heavily trodden (if it looks cool).
Is dissipative fluid like wet paint? Like a water emulation in the paint synthesizer? Sure, could be. I use that all the time in my work (maybe a little too much at times, like symmetry it's an easy crutch to fall on). But you certainly aren't restricted to water emulations to create a dissipative force.
You could use noise randomization (most likely in association with some kind of mixing, or blending). You could use warps, pushes, nudges. How they interact with what is already on the canvas, how that leads to movement over time, they all look different. Spend an afternoon playing around with different approaches to warping or nudging, to create a dissipative force. Learn how to use it as a tool in your tool chest.
Now for 2D painting, how the dissipative force moves around on the 2D surface is the deal. But when you are talking about painting over time, then you need to factor in some kind of recursive element, so that the force 'lingers' over several successive frames. The dissipative force is temporal in nature, in addition to probably having some 2D dissipative element as well.
So you need to think about different scenarios to pull this off, different generative strategies that play out over time in an animation, over time in processed video frames, that work to help generate the final painting over time, your digital painting on a time canvas.
You also need to lay down that 'marking stuff', whatever it may be (again, don't get hung up on it, there are lots of different approaches, all are valid). And by painting on top of your last output frame, you are inherently introducing temporal continuity into your time based painting. And you really want to shoot for that. You want that dynamic tension between interacting forces to play out over time.
So that was a mouth full. Something to think about before you head off to process some video frames in Studio Artist. Have fun.
Replies
I must confess that I'm hoping to embark, shortly, on my second time around the Version3.0 videos. I don't find the version 5.0 manual especially helpful, but the videos are great. At last I'm beginning to feel like I'm getting somewhere and beginning to explore the power and satisfaction from SA.
BUT - I still think that it would be helpful to have more videos to help make the leap from 3.0 to 5.0 (and maybe this is about to be addressed in 5.5.* :)....?). For example, I was bemoaning the process for selections but I now understand a little more though still have a long way to go. I give that example as its also been where I've begun to realise the thinking approach to SA that you refer to in your post.
So, turning to videos, you know whats coming now ;), I'm hoping to need a lot more helpful videos to even begin with videos. Or, to be more accurate, I'd like to know how to take some abstracted images, heavily utilising blanking, and get them heading towards an end result over time e.g. take a starting point of compositing a few sources with SA and have them morph over time into either an end result that's 'known' (another SA canvas for e.g.) or, indeed, to see where it takes me. But to be honest, the manual - whilst it has its uses (I had mine printed) is of little practical use to me, and posts that get too technical in language but don't meaningfully demonstrate the steps required become frustrating. Please don't misunderstand me, its all helpful stuff - streets, nay whole highways, ahead of competitors - a completely different league, but a picture speaks a thousand words etc.? :))
A host of new videos on SA you tube site - great, thank you! :))
What exactly do i mean by 'dissipative' force in conjunction with an 'apply-paint' force.
Basic Idea
Here's the world's simplest paint animation example.
https://vimeo.com/538397135
It was generated using the world's simplest paint action sequence (PASeq). I selected 2 factory paint preset that ship in Studio Artist V5.5 to generate it.
The first action step consists of the 'Subtle Drip 2' paint synthesizer preset.
It is a 'dissipative' action step. It simulates a wet water diffusion process applied to the canvas.
The 2nd action step consists of the 'Acrylic - Curvy' paint synthesizer preset.
It is an 'apply stuff' action step. 'Stuff' being colored paint that tries to simulate a very simple acrylic natural media look.
The 'dynamic tension' that occurs between these 2 'dissipative' vs 'apply stuff' forces acts to create the inherent complexity in the paint animation (the painting over time).
Note that i never erase the canvas. I'm always running the next animation cycle of the PASeq on top of the last output frame still sitting in the canvas. This 'art strategy' component works to help create temporal continuity in the resulting paint animation.
Note the ordering of the 2 action steps.
I start by using my dissipative action.
I then use my 'apply stuff' action.
What would happen if i reversed that order?
Exercise for the reader. Try it out and see.
Additional Details associated with this simple example
I also used a fixed FPS (frames per second) timing of 8 FPS for my animation output. I set that up in the Movie tab of the main Preferences dialog.
Note that i have Process Frame Timing set to MovieCodec FPS.
This means that the Movie Output FPS set in this preference panel is used for the movie output timing.
If i had set the Process Frame Timing to Source Movie, then the actual source movie timing would be passed through to the rendered output movie file (meaning that the Movie Output FPS preference setting would be ignored).
I started off with a blank white canvas.
if i wanted to build my PASeq to specifically include that i coudl do it like below.
Note that i setup an initial 'erase the canvas to white' action step at the very beginning. And i recorded a Mute keyframe for it at frame time 2. This means that the canvas will only be erased to white at the very beginning of the animation render process (frame time 1). That action step is muted for all other frames in the animation (the black keyframe is the mute keyframe).
To record a Mute keyframe, option click the keyframe while holding down the m key.
Some earlier pieces I’ve done that rarely erase a frame along the way:
https://victrolux.com/public-art-gallery/
~Victor
lovely website, victor.
thank you Donna. It was a long overdue refresh of my site. And the perfect project for the first 6 weeks of Covid shelter in place last April. I’ve been sheltering in place my whole life. So I just dug in and worked.