There's a variety of software available for live video manipulation, from the minimalistic, developer-oriented vvvv and pd, to the more aesthetically pleasing Max/MSP and Troikatronix Isadora.
I've used each of these in some capacity, and personally I prefer Isadora, as it is more intuitive to use and get a basic patch up and running in a short time than the rest. MAX was originally developed as a modular audio manipulation engine, and MSP was later developed to add video functionality.
vvvv and pd are both very powerful tools, but more suited to procedural content generation or content analysis than direct video manipulation, and they can be considered lacking in terms of an interface. As a result, they have a comparably steep learning curve. However, they are more extensible and versatile than Isadora and Max, and are worth digging into if you can devote the time to it.
I often find that it's possible to play to the strengths of each of these paradigms - for example, there have been many occasions where I've used vvvv and Isadora in tandem; vvvv performing real-time live audio analysis, and sending sync data (in terms of pulses and BPM) via OSC to Isadora, which then performs the video manipulation using these data. This may be worth looking into, if you have the chance.