Info on Style functionality?

[Jeff Burger]

Hi,

I see references to Style-related functionaltiy in various places in 5.5.5 (like File > LoadStyle, Canvas > Canvas to Style, MSG Prefs > Palette Option > Gen from Style and probably other places I'm not remembering right now). Can't find anything in the docs or tutorials about Style and it's use. Would appreciate a pointer to more info or some direct intel.

Thanks!

[Synthetik]

You can use the Style buffer in a lot of different ways.

But at it's basic it provides a way to use 2 different images that modulate different artistic visual processes to create some stylized art image.

So the artistic visual image you are constructing is composed of artistic elements that are modulated by both the Source and the Canvas contents.

 

Here is an example of what i mean by this.

So the Source is the same image for all 3 examples.  And the visual effect being run is the same.  But the  Style is a different image for all 3 examples.

 

Now you don't want to get hung up thinking that 'style transfer' (maybe 'style modulation' works better as terminology) is just one particular thing.  There are an infinite number of different ways to approach it.  And they all look different.

 

Studio Artist from day one has been built around this notion of incorporating models of the human visual system under the hood.  Models that try to emulate the way different parts of the human visual cortex process and internally represent visual images.  And operation modes like the paint synthesizer were built to incorporate 'visual attribute modulation' as a part of how they worked internally. So 'visual attribute modulation' is a key component of what Studio Artist is all about.

The Style features we started to add to Studio Artist in the V5.5 release extend this whole notion of 'visual attribute modulation' so that you can work with 2 different sets of visual attributes from 2 different images at the same time in any given Studio Artist effect.

We snuck the LoadStyle feature into V5.5 at the last minute.  Really almost as a teaser for things we planned to come later.  It was hooked into Gallery Show and some limited Paint Synthesizer and IpOp and Vectorizer effects.

With the V5.5.5 release we had more Style modulation implemented throughout the program.

And we've continued that implementation of new style modulation features in the pre-V6 code we are working on.  So what we are doing with style modulation continues to increase as we continue to develop the program and work with it more.

 

Once the Style buffer was available, it almost immediately began to be subverted for other tasks.

And you can see that in some of the V5.5.5 PASeq presets.  Where it is extensively used as a 'swap' buffer within an overall PASeq art strategy.  You can think of a 'swap' buffer as an alternative approach to building effects that would require additional layers to pull off without it.  Building these effects with multiple layers is much more messy and complicated.  But building them using the Style buffer as a 'swap' buffer makes putting them together pretty easy.  And more robust, since they are not tied to specific layers and can therefore be used in a more generic way that PASeq presets that were built tied to specific layer positions.

 

I'll put together some additional posts over the next few days in this thread to run through the Style features in V5.5.5 and how one might approach using them.

There have been previous posts on Style in the users forum over the last year, so i'll see if i can dig up the links to those as well.

 

[Jeff Burger]

Thanks so much.... keep 'em coming! :)

[Synthetik]

You can load a single image or a folder of images as the loaded style.

In a previous post to this forum topic i mentioned how the Style buffer could be used as a swap buffer when building a PASeq. So if you do that, then the contents of the Style buffer are not going to match the specific image or image folder you loaded.  The ReLoad menu command lets you reload the specified image or image folder.

When i say image folder, keep in mind that only one image in the folder is going to be loaded at any given time.

Once an image is loaded as the current Style, it can also be subjected to some data augmentation options.

You can see above that i have 3 different data augmentation options checked, the RandFlip, SymmetryWarp, andBlockRandomize options.  And you can see the effect of those data augmentation options when you look at the current style in the canvas. I'm putting the current style in the canvas by erasing the canvas to the style.

There should really be some 'style info' dialog or display area in the interface that provides some info about what the style is current set to, but as i mentioned previously the style features are still a work in progress, so erasing to the styl eis the easiest way to see what it is set to currently.

If i turn off the data augmentation options and use the ReLoad menu, and then erase to the style again, we can see that the data augmentation is now gone.

Because i have an image folder loaded as the style, i will get a different image in that folder every time i reload.

The Seq Index vs Rnd Index style setting options let you select sequential indexing or random indexing through the images in a style image folder.

 

The current style data augmentation options are really just a teaser of things one could possible do with style data augmentation.  Those options will be extended in the next release.

One could of course manually build a PASeq that implements all kinds of different style image data augmentation options.

 

If you are confused about why one would want to load a folder of images as the style, think about how Gallery Show works.  Gallery Show runs a series of gallery show cycles. If you have a folder of images setup as your loaded style, then a new style image from the loaded style image folder will be setup at the beginning of every gallery show cycle.

It might help to think of a folder of images as a collection of statistical properties associated with those images in the folder.  If you put some care into selecting the specific images in the folder, then you can design the statistical properties associated with the folder to help create visual properties in the artwork you are creating. Remember, Studio Artist generates visual attributes for visual attribute modulation of different properties of visual effects or automatic painting. So color or textural or structural components of yor visual effect might be being modulated by what is currently in the style buffer. 

You can think of a folder of images as being a database of images in some sense.  Part of what we are trying to do with letting you work with folder of images for the source or the style in gallery show in Studio Artist is to give you the ability to think about building a painting or visual effect up from the aggregate statistical properties (statistical properties that determine the perceived visual appearance) of a database of images rather than just a single static still image.

[Synthetik]

Let's run through some simple examples of style modulation.

I'm going to start off by working with the paint synthesizer.  I ran the Initialize QuickEdit command to configure a default soft raster paint effect.  I also setup Size Modulation in the Brush Modulation control panel to use Color Range for a tighter paint effect (tighter based on source color range modulation).

So the default Path Angle settings are working off of the Source visual attributes.

I can change that to use the Style visual attributes instead.

You can see that the paint paths are different when i make that change.  The paint paths are being derived off of the style orientation rather than source orientation.

because i have the Size Modulation set to Color Range, and the color range is being derived from the source color visual attributes, it still does a pretty good job or painting a tight representation of the source image.  If i turn that off, you can now get a better sense of how the paint paths are being generated based on the current contents of the style buffer (oil rig photo).

if i now go back to the Path Angle control panel and set the visual attribute for the path angle generation back to Source, and also change the secondary popup in the Source Area from Paint to StylePaint, then the source coloring will be generated from the current style image rather than the source image.

So the paint paths are being generated from the clown image in the source, but the paint coloring is being generated from the current style image.

 

This is a really simple example, but it shows off how you can start thinking about working with 2 different images (or 2 different collections of images if you are working with folders of images), and then modulating different aspects of some visual effect or paint preset with different visual attributes from the 2 different images.

[Synthetik]

We introduced 3 Injection Channel modulators in the paint synthesizer in Studio Artist V5.5.5.

These live in the Miscellaneous control panel of the paint synthesizer Editor.

Each injection channel has a number of different modulator options you can choose from.

You can see that there are a number of different Style modulator options based on different style visual attributes. If i choose one like Style Orient, then the style orientation is routed through the Injection Channel #3 in this case.

If i then go to somewhere in the paint synthesizer that has a Inject Channel3 option for a modulator setting, the Style Orientation is what is going to be the modulator based on how i have it setup above.

So we can see above that the Size of the brush is being modulated by the Style Orientation, since that is what is being sent through Injection Channel 3.

If you are familiar with audio bus routing in an audio mixer, what i am describing probably make a a lot of sense.  If you are not, then it might take you a littl ebit of time to wrap your head around the concept.

 

Previous versions of Studio Artist had 3 different modulators for 2 handed input using wacom tablets that supported 2 handed input.  But wacom stopped supporting 2 handed input, so those 3 fixed modulators associated with it ended up not really doing anything.  In V5.5 we repurposed those 3 fixed modulator options available in a lot of different places in the paint synthesizer by using them to work with the 3 injection channels i discussed above.  So style modulation is avai;able throughout the paint synthesizer via using injection channel modulation.

At some point in the future we'll probably move to only using injection channels in the paint synthesizer as modulator options. It would clean up the interface quite a bit.  That probably will not happen until V7.

[Synthetik]

You can setup the Background Texture in the paint synthesizer to work off of the current style image if you want to. I've got an example of that shown below.

So the current style image is being used as a source for a crude form of texture resynthesis in this example.

[Synthetik]

The Gallery Show automatic selection mask generation supports different options for the image source that is being fed into the automatic selection generation algorithms.

The Style is an option in there.

You can also randomly flip back and forth between the source and the style, or source -style -canvas.

[Synthetik]

Gallery Show also has a source data augmentation option that randomly generates a transition effect between the source and the style, and then loads that transition image as the source for the current gallery show cycle.

So if you use this, and you have loaded an image folder as the style, then every gallery show cycle, a random source image is selected from the gallery show source folder, a random image is selected from the style folder, and then a random transition between those 2 images is generated, and that is loaded as the actual source that gallery show will use for the current gallery show cycle processing.

 

[Synthetik]

Different Image Operation (Ip Op) effects have Style modulation options available in them.

This was more limited in V5.5.5 than we would have liked, and the number of Ip Op style modulation options has expanded quite a bit in our development code for the next release.

 

However, you can configure any Ip Op effect to use the Style as the IP Source for the image processing effect.

Here i'm processing the style with an adaptive filter effect that is modulating the adaptive filter based on the inverse source luminance.

I can change the adaptive filter effect below so it is processing the current Source and is modulating the adaptive filter with the Style Luminance.

 

 

[Synthetik]

You can use the Canvas :

This menu command is recordable as an action step in a paint action sequence (PASeq).

So this is the mechanism you can use in V5.5.5 to use the style buffer as a swap buffer in a PASeq.