Still frame from a paint animation that mostly uses the paint synthesizer. It's built on the energy based model rendering principal, so you apply energy forces to push the image where you want it to go (in a statistical sense).
You need to be a member of Studio Artist to add comments!
Comments
I like this a lot but wish your explanations would be versed in a way that is understandable. Way too technical, not just on this... Thanks.
Yann LeCun has put together a whole unified field theory of deep learning neural nets called 'Energy Based Models'. Yann is one of the 3 individuals who won a Turing award (computer science Nobel prize equivalent) for their work developing the theory and practice of neural nets and deep learning.
Energy Based Models are a way of grouping and classifying many different kinds of at first thought disparate architectures and algorithms for building neural net or ai learning systems, and understanding how they all work with some consistent unifying principals. So it's a pretty technical way to look at how neural net or ai systems work, and to compare different kinds of historical neural net architectures. You can think of it as a conceptual framework for understanding and comparing these systems. The NYU deep learning course covers this stuff in great detail if anyone is interested (all the lectures are available online for free), and Yann has published various papers on the subject you can easily find with a google search and read if you want to. If anyone here is interested in the gory details, the papers include a lot of technical math analysis and historical perpective on a long line of different learning algorithms developed and used over the last 40 years.
What i realized is that one could apply the same principals to thinking about painting or rendering images.
I've talked before about how you can break down digital painting into 2 main components, the 'laying down of stuff on a substrate', and the dispersion of that stuff on the substrate itself. And this simple conceptual model does a great job of unifying different kinds of natural media marking technologies, different kinds of paints, different approaches to drawing, different printing technologies, etc.
Now digital painting is somewhat different than true natural media painting, since you can build a digital paint that adds energy to the system (or manipulates the internal energy of the system). Something that would be hard or impossible when working with real natural media, but is fairly trivial to do when you are digital painting if you want to approach it that way. Just pushing extra energy into the system leads to resonate phenomena in the resulting painting (like oscillation or dynamic textural patterning).
The straightforward 2 component digital painting model involves applying some kind of colored paint to the canvas, and then smearing or blending it in some way with what is already on the canvas.
So this is a literal simulation of what happens in the real world when you paint or mark a piece of paper or cloth or board with pigments. You place that colored stuff on the substrate, and then you smear it around on the substrate, mixing with what might already be on the substrate. Depending on what physical materials you are working with, what kind of mixing and smearing does or does not happen is the function of the physical properties of those materials, and how the artist physically interacts with them.
The Energy Based Model approach to digital painting is different. Rather than applying something new on top of what is already on the canvas (dropping color down onto the surface), you instead apply some kind of energetic force to what is already on the canvas. You induce what is already on the surface to change in some way. You subject it to energy forces that end up modifying what is already there into something different. A force designed to change the statistics of that part of the existing canvas in some way. It may seem like a subtle distinction, but it is a very different way to think about how digital painting could work. And it also has a very different visual appearance than the traditional approach.
What do we mean by statistics. Probably the simplest things to think about are the mean and variance of some small area of the canvas. The mean is just the average color of that local area. The variance is the distribution of change based around that mean value (are all of the pixels close to the mean, or far away, how far away, how different are the different pixels). We could get more elaborate and start to talk about different oriented spatial frequencies for more elaborate statistics, but simple mean and variance will get you quite far in many applications, and are fairly easy to think about and understand.
And there are all kinds of ways to configure the paint synthesizer to take the existing canvas under the brush and modify it's local statistics, the mean and variance. Or to move around some piece of the canvas and place it somewhere else close (doing so in a way that pushes the existing system towards an outcome you would like to see develop).
So that is a quick overview of what i am talking about when i talk about Energy based Model approach to image rendering or digital painting.
I can put together a tutorial that runs though how one could approach editing the paint synthesizer to do this kind of thing.
Everyone is very enamored by ai derived image synthesis these days. It all seems very mysterious and complex. But if you dive into how those systems actually work, it can all be broken down into recursive systems that try to modify their internal parameters to reduce some error measure. And that is pretty simple to warp your head around (probably after some initial conceptual leap). Studio Artist is built to enable all kinds of recursive processing algorithms and experiments. So you can start to apply the same elementary principals that these neural net systems use inside of the paint synthesizer, or a paint action sequence (PASeq).