If software is all you're interested in, here are a few options:
- Quartz Composer
- Pixel Conduit
- Resolume
- VDMX
- Arkaos
- Modul8
Personally I've only tried Quartz. It's hugely powerful and free, but the learning curve can be quite steep. It's more of a node programming language for video graphics, that's what makes it so powerful.
Some downsides are that it's not actively developed anymore and that it only runs on OSX.
I'm excited to try Pixel Conduit soon. It seems to be an attempt at a modern version of quartz, and the free version seems pretty powerful.
If you want to record the video as well you might have to look at a separate program called Syphon.
I would post more links but I don't have enough rep.