I know there are various users interested in scoring their Studio Artist visual effects to audio files. It's definitely something i'm personally interested in as well.
So i thought i'd open up a forum discussion related to this topic.
From my perspective i'm interested in better ways to put together hit point lists for key-framed PASeq timeline effects. So that i can build specific visual effects that match specific locations in the audio file of musical significance.
My musical projects tend to be rigidly tied to specific BPM bar/beat structures. So i want an easy way to make visual edits that can conform to that inherent structure.
At the same time, i'm well aware that other musicians hate that kind of rigid timing, and want the ability to have free reign over where they position their hit points within an audio file. And they might want transitions that happen over arbitrary time periods, as opposed to fixed sets of bars/beats.
Of course hearing the audio within the program would be nice to have. Especially if you want to tie specific live editing changes to musical hit points. Getting the synch right can be challenging the way things currently are.
I think you also want the ability to specify an initial offset to synch some point in the audio file with the beginning of our PASeq timeline synch to output frames in an animation.
I'm also going to draw a distinction between live editing of hit points vs live streaming output of whatever it is you are doing within Studio Artist. In the former you are recording editing changes in a PASeq timeline that you can then playback in an animation run for whatever final output you are interested in.
In the later you need to live stream output to matchup and synch with the audio. And ideally be able to record the audio in the video output if you are streaming out a movie file.
You could probably guess that i'm very interested in this notion of using procedural processes to build up transitions for you. Studio Artist V5.5.5 has some early steps towards providing a toolkit for doing this kind of thing.
So that's my quick take on some thoughts about beefing up the visual scoring to audio file capabilities. feel free to lay out your own thoughts on what you would like to see provided for this kind of thing. It's good feedback for us as we lay out our future plans in this area.
Replies
Given I have been working with/to/for/from music for more than 20 years (there’s over 300 videos of that work here: https://vimeo.com/jeandetheux, it is basically what my being forced by sudden and very severe allergies to abandon natural media for the digital “stuff” has made possible), there are a few things that matter to me in listening to music with my eyes and coming up with the dialog(s) that can happen between the music and the images.
Music has become my “motif” (as Cézanne used to say).
The worst thing I can think of is constant “Mickey-Mousing”, whereby they both do the same thing at the same time and leave no room for the viewer (which is also the doer) to enter.
Sure, there can/should be moments when they meet on the same millisecond, and other times at which they almost ignore each other, but the art of this language is to navigate between “too close” and “too far.”
Philip Glass has some very interesting things to say about that in this video: Interview with director Godfrey Reggio and composer Philip Glass on...
As for "the early steps towards providing a toolkit for exploring this kind of things” available in SA V5.5.5, I’d love to have a look at that.
that makes me feel better about how much of a struggle it was to please the musicians i was making videos for. my brain just doesn't work like that. koyannisqatsi effected me so much when it first came out that i couldn't wait to watch it again and it didn't have quite the same effect but i am interested in watching the video you shared.
Anything would be great, what I have done in the past is make a quicktime movie with the audio (and maybe rudimentary video) that I wanted and then take that into Studio Artist and work from there.
We're building up a set of new editing commands in V5.5.5 to help out with keyframing action events in the PASeq timeline.
Here are a few different examples:
So you can take a History Seq and keyframe append it to occur at a specific frame time in the PASeq timeline. Remember, you can import a PASeq into the History Sequence (even one with keyframes in it).
You can invert extend the PASeq timeline. So your keyframed action steps in a PASeq can play out over time as a palindrome animation.
You can also loop extend the PASeq timeline.
You can take a series of action steps in the History Sequence, and append all of them as one keyframed action step where the different action steps in the History Sequence are now individual keyframes recorded at different frame times in a single appended action step in the PASeq.
A work in progress is an edit command that lets you take a manual paint stroke in a History Sequence and turn it into individual keyframes in an appended PASeq paint action where the original paint path is broken out into keyframed position movements occurring over time.
You can think through how all of the different timeline edits i mention above relate to building visual events that could be synched to specific timing structures in a piece of music.
wow neat I did not know you could do some of that stuff
I'm trying to get everything in place so that we can build a generative animation system on top of it that could pump out AI generated animations that are several minutes in length with a ridiculous number of internal keyframes that timeout to bar/beat timing. You would point that at folders of PASeqs that would be dropped into the timeline with appropriate timing for the automated keyframe edits. Think gallery show, but it builds complete animations with a high level of internal complexity.
You could of course do it all by hand with manual keyframing, but that is so tedious that no one is ever actually going to do it.
That would be amazing
What I would love to be able to do is create graphical scores and execute them within SA. A system for that could be based on paths that can be referenced/injected throughout the various tools. For example
1) Draw and memorize curve A
2) Draw and memorize curve B,C,...
3) Set keyframe of first point of curve A at t0, set keyframe last point of curve A at t=100.
4) Drive the location and other parameters of a pen from the values of curve A interpolated by the timeline while curves B,C modify different settings/effects or other subtimelines
inspirational examples of graphical scores:
https://en.wikipedia.org/wiki/Metastaseis_(Xenakis)
https://www.youtube.com/watch?v=71hNl_skTZQ
now imagine that but driving visual parameters
That kind of visual score is kind of like painting in components of the audio spectra. The mapping makes sense to me.
It's a little bit different if you are using it to modulate editable 1-D parameters.
So are you asking for the ability to paint in a path that then gets mapped to an envelope for parameter modulation?
So vertical position would be parameter extent, and horizontal position would be how it changes over time?
So if someone draws a circle, how would the system interpret that?
If the path position was modulating 2 different parameters as the path progresses then the circle makes sense.
Or if the pressure in the path is what is doing the modulating as the path progresses, then it makes sense.
So mapping the change over time along the path seems to be the mapping that makes the most sense, which is different than the visual music scores.
in my fantasy it is something like this, where instead of user slider you have p1x, p1y. they could be normalized or screen coordinates and TG already has a mechanism for remapping values onto the time domain. moreover, you can have a global TG like in the misc settings that also serves for feeding say P1X to pressure over the global TG time domain. You also have things like phase (maybe with option to turn it on/off) or maybe ad Tmin Tmax from the MSG param window, but it looks like almost all the structure for implementing something like this may be already there
the I could have a layer that is just the score where I'd draw all the paths and have them drive the whole composition.
Xenakis built a spectral painter way back
https://www.youtube.com/watch?v=lNPWub-MNxg
but one interesting takeaway from the UPIC system is that it could also use the drawings to drive synth modulation, not just synthesis.
-
1
-
2
of 2 Next