If software is all you're interested in, here are a few options:

 - [Quartz Composer](http://en.wikipedia.org/wiki/Quartz_Composer)
 - [Pixel Conduit](http://pixelconduit.com/)
 - [Resolume](http://resolume.com/)
 - [VDMX](http://vidvox.net/)
 - [Arkaos](https://www.arkaos.net/)
 - [Modul8](http://www.modul8.ch/)

Personally I've only tried **Quartz**. It's hugely powerful and free, but the learning curve can be quite steep. It's more of a node programming language for video graphics, that's what makes it so powerful.
Some downsides are that it's not actively developed anymore and that it only runs on OSX.  
I'm excited to try **Pixel Conduit** soon. It seems to be an attempt at a modern version of quartz, and the free version seems pretty powerful.

If you want to record the video as well you might have to look at a separate program called [Syphon](http://syphon.v002.info/).

I used a [Blackmagic Intensity Extreme box](https://www.blackmagicdesign.com/products/intensity/) to get the live video from our Sony XDCAMs into my Macbook Pro. I had to use [Black Syphon](http://vdmx.vidvox.net/blog/black-syphon) to get the feed from the Blackmagic box and send it into Quartz for live processing.

##UPDATE
I've been using **Resolume** to do exactly that. It works really well and is way more user friendly than Quartz. Though you can also use Quartz in it. Resolume also can record the video to an output file out of the box.  
**Pixel Conduit** didn't seem to have gained much traction unfortunately. So I never used it in a production environment, I only played with it.