The role of gesture and movement in digital art

I've been thinking a lot recently about how current approaches to multi-modal ai generative image synthesis works when contextualized against more traditional approaches for drawing and painting.  And also how it compares to the whole notion of spatial thinking in human perception and thought. So i thought i'd throw out a few observations here for discussion. I'm interested in other artist's thoughts on this.

It's easy to get swept up in the current generative image synthesis from text mania and lose sight of a bigger picture.

In general, current systems are extremely obsessed on the text description component of the system.  To the point where the whole notion of how one creates digital artwork involves writing a short textural description prompt mini novel down squeezed into a sentence or 2.  And indeed the structural mechanics of the underlying transformer deep learning architectures used in these systems have very hard limits on the number of characters or words that can be used to text prompt the system.

Don't get me wrong, these generative ai systems are fascinating and amazing.  What they actually 'know' or don't know is up for debate.  My current viewpoint is to think of the whole system as an image compression system that was trained on a huge collection of digital images scrapped from the internet.  Everyone here is familiar with compression systems like jpeg.  A latent diffusion generative image synthesis algorithm is just a more sophisticated version of this kind of compression system. One that incorporates a specific kind of neural net algorithm called a transformer that allows the positional statistics of text to be correlated to localized visual statistics in images.  And this addition opens up a whole world of new potential interactions with the resynthesis side of the image compression algorithm.

And we could dive into a whole analysis of what that kind of system learns or doesn't learn (and we can certainly do that if people are interested).  But i want to take a step back and focus on how people are interacting with the system.

The original roots of Studio Artist are very much planted in the whole notion of being a digital paint program.  Where 'painting' was all about the notion of physical movement in drawing.  Very focused in the beginning on harnessing the features of interactive graphics tablet systems.  Working with pen pressure, tilt, tilt orientation, axis twist in the context of pen movement to create a very visually expressive system for digital painting.

Studio Artist allows for totally manual painting, or intelligently assisted painting, or fully automatic painting.  And all 3 of these are very focused on this whole notion of gesture and movement.  Even if the system is running fully automatically, what is going on under the hood as the painting process proceeds is all about working with gesture and movement to drive the dynamic image synthesis.

 

I was watching a lecture by Barbara Tversky this morning titled 'Spatial Thinking is the Foundation of Thought'.  And i think it really drives home the whole notion of how using language only as a mechanism misses something fundamental about how people really think.  Many people now think that the whole underlying algorithmic mechanism for how the brains cortex actually works, what is actually going on when people think, is rooted in spatial perception and movement planning.  Ancient neural mechanisms for this process being reworked over time in evolution to be used in other sensory modality, and ultimately in abstract thought itself. 

Language being quantized conceptual tokens that are used for communication between people, but not the fundamental mechanism for how thought actually works.

There are a lot of interesting examples of the whole concept of 'spatial thinking' in the lecture. And they all drive how this notion that spatial thinking is a key component of human thought.  Some of them involve experiments where gesture leads to improved performance on various tasks.  There is a whole section near the end of the lecture about gesture and ambiguity associated with it in the way artists build and interpret sketches.  Of using gesture as a way to work out an artistic problem interactively through spatial manipulation.

 

So i keep coming back to this notion that there is something fundamentally wrong about how people are proceeding to build these new 'ai' generative image synthesis digital art systems.

In my own experiments it has become pretty obvious that the creative potential of what can be visually achieved by the system is really radically held back when you only work with text prompting.  One can use various approaches to visual modulation to really extend the potential range of imagery that the generative system can create.

But then there is this whole other notion of how to incorporate gesture and physical interactively into the system.  I'm still trying to work out in my head what that all means.  And i'm really curious to hear other peoples opinions on this.  Especially people coming from an extensive physical painting and drawing background.

 

Studio Artist at this point is a whole gigantor grab bag of different approaches to making digital art all thrown into this single package.  And in that process we lost some of the original focus on interactive motion based digital painting as i became more personally interested in this whole Brian Eno inspired notion of art strategies.  Where you build up a situation that you then let play out iteratively over time to create an image or animation.  Gallery Show is probably the ultimate example of how that kind of thinking played out.  But the recursive nature of so much of what you can do in Studio Artist is heavily tied into that notion as well.  Defining rules in the Studio Artist system (usually in a paint action sequence) that then play out over time to create artistic imagery.

And you can certainly view recent generative multi-modal image synthesis systems though that lens.  You build text prompts, see how they play out, interactively refine them.  And you can incorporate the whole notion of visual modulation we are working through here to extend what these systems are capable of.  But i keep wondering if everyone is missing a bigger picture.  One rooted in spatial thinking, in gesture and movement.  And we need to be thinking about how to incorporate gesture and movement more directly into generative image synthesis systems.

 

I'm happy to elaborate on any of this if it doesn't make sense.  And i'm very curious about other peoples thought on the issues i'm bringing up.

You need to be a member of Studio Artist to add comments!

Join Studio Artist

Replies

  • Fascinating stuff.  I will think about it

This reply was deleted.

On the closure of the User Forum : An open letter to John Dalton

Hi John I have no idea what you are going through. However the impact you have made not to continue with the user forum and your desire keep it as a permanent feature as promised in the past is a big shock. It also betrays the faith and love we have shown in you and Studio Artist.  Bluntly spoken, it sucks. That you have chosen to follow this direction in silence, allowing no compassion or understanding on our part is a betrayal of the deepest level. What is going on? This is not the John…

Read more…
4 Replies · Reply by Paul Perlow 2 hours ago

Interdimensional Coincidence Control

Hi everyone, I am glad the site is still here! Here is a new short video I made. All made in Studio Artist, several separate videos with alpha channels, then combined in layers with the music in Blender. A lot of MSG running through brushes, with several of the brush Path Starts being controlled by the MSG Scan Generator in the Generator part of the Path Start in the Paint Synthesizer. Also some MSG running through a brush, then making a video of that with an alpha channel, then making that a…

Read more…
3 Replies · Reply by Thor Johnson Mar 22

Whats going on with this site?

Has anyone else gotten a warning about this site disappearing? An email form just popped up, asking me to contract the owner and leave a message to let them know that they may loose their "network"Did Synthetik forget to pay it's bills, or is something else going on?I think 8 months is more than enough vacation time. Is anyone at Synthetik doing any development work at all? 

Read more…
7 Replies · Reply by Alf 10 hours ago