If software is all you're interested in, here are a few options:

 - [Quartz Composer](http://en.wikipedia.org/wiki/Quartz_Composer)
 - [Pixel Conduit](http://pixelconduit.com/)
 - Resolume
 - VDMX
 - Arkaos
 - Modul8

Personally I've only tried **Quartz**. It's hugely powerful and free, but the learning curve can be quite steep. It's more of a node programming language for video graphics, that's what makes it so powerful.
Some downsides are that it's not actively developed anymore and that it only runs on OSX.  
I'm excited to try **Pixel Conduit** soon. It seems to be an attempt at a modern version of quartz, and the free version seems pretty powerful.

If you want to record the video as well you might have to look at a separate program called Syphon.

I would post more links but I don't have enough rep.