Video editor that lets me code my own effects from scratch?

Question

I'm the type of guy who likes to produce beauty from code - I mostly experiment with shaders.

Up to this day, I've only been doing so for the sake of games, but an idea popped up in my mind that I could decorate an actual film footage using handwritten postprocessing.

Is there any software that, apart from facilitating basic video editing tasks, lets me code (and later apply) my own effect completely from scratch, that is, lets me manipulate each input frame pixel by pixel? Either in some shader language or maybe CUDA or even C++?

Answer 1

I second the vote for Processing, but if C++ is more your thing than Java you could also look at openFrameworks. It allows for more processor-intensive computation and as such is good at real-time synthesis and manipulation of video.

Another tool worth checking out is NodeBox. It allows for coding, but wraps it all in a node-based GUI, the aim being to simplify the boring stuff.

Final Cut Pro ≤ 7 used to have a built in language called fxcript (based on c, but weird and buggy) that gave you pixel level access to the video stream and integrated into the editor seamlessly. I wrote a few plugins for that back in the day, but I jumped ship after the initial FCPX fiasco so I can't comment on the new language for visual effects FXPlug. I believe it's quite powerful, and, like processing, leverages openGL to do the heavy lifting.

Answer 2

You should check out Processing dot org.

Answer 3

To add upon John's answer, one way to do this using FFmpeg would be to add your filters (in C), but another, more modular, way is to code a Frei0r plugin, which can then be used with various host applications, whether FFmpeg, MLT..etc

Frei0r is a minimalistic plugin API for video effects. The main emphasis is on simplicity for an API that will round up the most common video effects into simple filters, sources and mixers that can be controlled by parameters. It’s our hope that this way these simple effects can be shared between many applications, avoiding their reimplementation by different projects.

Answer 4

As an alternative to Processing, if you prefer to use C#, you can write plugins for VVVV using their C# APIs. Also if you prefer to just write pixel and vertex shaders which you can apply to your audio/video, they support that as well.

Answer 5

You might want to look into using FFmpeg - it seems like the kind of tool you're looking for. Adobe's After Effects allows some custom scripting, but not nearly to the extent you're looking for.

Answer 6

You can code custom plugins for Shotcut video editor using MLT, frei0r, or WebVfx plugin. MLT and frei0r plugins require C or C++ coding; whereas WebVfx lets you use HTML5, CSS3, JavaScript, SVG, and WebGL. : https://shotcut.org/notes/make-plugins/

I like unity3d for audio/video things

Obviously, have a go on Milkdrop if you want fluid dynamics code, else see if you can splice a video streaming code into the open source MD2 codebase, everyone would be well chuffed.

Research "VJ" programs, like dj programs for video.