Distorting video with live sound

Question

I am starting an art project that requires to create a program (running on my computer) that can be able to distort the image of the video projected according to the sound/noise that a mic captures live.

The distortion is supposed to respond to the duration and intensity of the sound, and I'm not looking for glitch art, I'm looking for a smoother distortion (but my biggest concern is not the way the distortion looks but the appropriate respond to the sound captured by the mic)

Is there anything like this? or do I have to create the program from scratch?

Answer 1

There are several programs available that allow you to do that. One very popular tool in the live visual industry is MAX (especially Max for Live). Its a visual programming language to create visuals for live performances that also integrates into a few tools like Ableton Live.

A program that I personally use at work is called vvvv, its a visual programming tool aswell that allows for very complex programs and is more "multipurpose" than MAX and offers a huge collection of add-ons to work with all kinds of peripherals and input devices. For getting started I highly recommend having a look at this cheat sheet, vvvv has a VERY minimalistic GUI and heavily relies on shortcuts and hidden menus for more advanced settings and features.

Unfortunately only available for Windows but unlike Max its absolutely free to use unless you want to use it commercially, then you have to buy a commercial license (no differnces to the free version, you still just download the same version but have permission to earn money with it).

A also a free solution but only available on Mac would be Quartz Composer which is coming with Xcode. Its also a visual programming language that works in a similar fashion like vvvv and MAX.

E: Another solution would be MadMapper, its more meant for projection mapping but also does video with filters that can react to sound. In can also integrate into Live afaik.

Answer 2

I recommend VVVV also. You could make a patch that controls many video effects with levels from microphone input. Take a look at Video Effects and Compositing Tutorials

Basic tutorials and examples on how to use TextureFX nodes to work with the same tricks and techniques you could previously achieve only in Photoshop or After Effects. And what's best: everything can be animated in real-time.

Answer 3

There's a variety of software available for live video manipulation, from the minimalistic, developer-oriented vvvv and pd, to the more aesthetically pleasing Max/MSP and Troikatronix Isadora.

I've used each of these in some capacity, and personally I prefer Isadora, as it is more intuitive to use and get a basic patch up and running in a short time than the rest. MAX was originally developed as a modular audio manipulation engine, and MSP was later developed to add video functionality.

vvvv and pd are both very powerful tools, but more suited to procedural content generation or content analysis than direct video manipulation, and they can be considered lacking in terms of an interface. As a result, they have a comparably steep learning curve. However, they are more extensible and versatile than Isadora and Max, and are worth digging into if you can devote the time to it.

I often find that it's possible to play to the strengths of each of these paradigms - for example, there have been many occasions where I've used vvvv and Isadora in tandem; vvvv performing real-time live audio analysis, and sending sync data (in terms of pulses and BPM) via OSC to Isadora, which then performs the video manipulation using these data. This may be worth looking into, if you have the chance.