Temporal image audio pass through

[mark g]

i was playing with various temporal settings and wondered if there was a way the audio could pass through and follow the mutations of the visuals as opposed to just playing straight?..i dont know how this would be done ..and it would probably be very abstract and unlistenable to.so just an academic question really

[johndalton]

You mean the temporal image processing section of Studio Artist 4?

I suppose you could work with audio delay effects where your delay times were synched to the temporal frame offsets you are using for the temporal ip op processing. Does this make sense?

[mark g]

yes i get that..i was just curious..it would probably be white noise anyway!

[johndalton]

Typically the kinds of processing that can be done to images that looks really visually interesting would just sound horrible and distorted when applied to audio. Especially anything involving nonlinear processing.

I guess you could build granular audio processing effects for audio analogies to things like the slit scan or scan tracker. Experimental musician Tom Dimuzio does some really interesting things were he takes a song or a series of songs and builds micro granules taken at sampling intervals and then reconstructs something new with them, so you could for example hear an entire 5 minute song play in 15 seconds.

[mark g]

it was the slit scan i was thinking of
i will have to look him up ..i have played with granular synthesis, but the other way round - making short samples vv long..
i know that visually interesting stuff produces pretty nasty sounds..i have tried with various other programs..i suppose one assumes nice image = nice sound...

[johndalton]

In the precursor to the MSG Evolver program you could export the generated image as an AIFF file.

Human auditory perception is very sensitive to harmonic distortion. Both in terms of harmonic structure defining the timbre of a sound and deviation from the harmonics series leading to perception of dissonance. So any kind of nonlinear processing that introduces edges is going to generate a ton of non-harmomic distortion in an audio signal. The visual system is all about perceiving edges, so this kind of distortion might be very visually appealing.

The slit scan effects are different, in that they are dealing with manipulating time slices. So there probably are direct audio analogies to a lot of them.

[mark g]

yes, i suppose in a way audio to visual is easier for us to deal with.ie nice sound wild image..
i played with the slit scan and put a little movie up on here..it was that that made me think how would that sound .. i assumed a rhythmic sound, as i saw the image rhythmically..

but things are never so simple

[Dennis Miller]

I wrote a roundup for electronic musician magazine last year that looked at a bunch of apps that converted image to sound and some that went the other way. The approaches are quite different:
http://emusician.com/software/say-with-pictures/

Also. see Brian Evans' work for image to sound conversion using a slit scan method.
-Dennis

[johndalton]

Cool article! I need to get back to reading EM on a regular basis.

[liveart]

Another interesting possibility would be to have the paint strokes that SA generates act as a gestural input for an audio processor. Rather than trying to process the image, the direction duration width and intensity of stroke would become inputs or modulators for sound synthesis.

In a sense this method would be processing the process vs processing the image. From another perspective it could make the pen (stylus) a musical instrument where the tilt, orientation, pressure and velocity mediate the sound output.

[Dennis Miller]

See Leafcutter John's Forrester for a nice gestural input interface - controls a bank of samples (free; Mac/PC). http://leafcutterjohn.com/?page_id=14 exactly what you describe.
Excellent tool, especially using a Wacom. I used it for all the audio in my Lines of Force (www.dennismiller.neu.edu)
Best.
d

[mark g]

yes i found this last year..v nice...i havnt tried with a wacom tho, didnt know that could be done!

[Jean Detheux]

Dennis,
Would it be possible to replace "the forest" with an image of one's own, or to start with a blank canvas and have the strokes produce the sounds accordingly?
I could overlay a Studio Artist Canvas window on top of Forester "forest" using a video mixer, and I will try just that, but that requires two computers and a video mixer (which I have).
Can something like that be done on a single computer?