What would be most similar in concept to VST effects but for visuals? In other words, is there a standardized interface for running VisFX on a video in the same way that one can run VST realtime effects on audio?
In addition to the AfterEffects plug-in API, there's also FxPlug, OpenFX, and AVX plug-ins.
FxPlugs often work in real-time because they tend to utilize the GPU. They work in FCPX and Motion.
OpenFX plug-ins can work in a variety of hosts (sometimes requiring an adaptor), but as far as I know, they aren't real-time usually. They work in Sony Vegas and I think Nuke. There was an adaptor for After Effects at one time, but I don't know if it's still maintained.
AVX plug-ins are used by Avid systems. Again, I don't know if they meet your realtime requirements.
It really depends on how much you care about realtime operation. VSTs tend to be used both for production and performance, but with video, because there's so much more processing involved, things tend to be split between production and performance. That being said, for production, After Effects plugins are probably the closest analogy. For performance, probably Freeframe.
Freeframe is a format that's specifically tailored to be cross-application, but on the production side of things, I don't know of anything that transcends applications the way that VSTs do. Each video production app tends to have its own format. The main reason I say AE plugins are the closest analogy to VSTs is because AE itself is almost as ubiquitous in the same application price range/platform.
So there's not really an exact analogy (if you consider all the various factors), but there are similar things out there.