I wish to stream a video with dynamic metadata. I know I could just steam the video and metadata separately, but ideally I'd like them to be bound permanently.
Is there an existing protocol / encoding / format that does this? I could store the data in each frame using steganography, but this is not an ideal solution and is rather hacky.
If the frame-rate is not too high and you can associate a frame with a precise timestamp then the easiest would be to add the metadata as textual subtitles.
The other option is to mux the metadata as standard headers in .mp4 or similar container format. .mp4 can be broken down into multiple fragments, each one with its own header, and theoretically this can be done per frame. However, doing that for more than a few frames is extremely inefficient in terms of file size and performance. A variation of this method would be to extract every frame as an image (e.g. JEPG), and set its metadata.
Yet another option is to use stenography without any hacking, you just encode the metadata into a separate stream which is timestamp-synced with the main video stream. Most modern container formats, including .mp4, can accommodate multiple video streams in the same container (e.g., you can label the encoded metadata stream as the same content but in another language).
I'm not sure how this fits into a streaming strategy, but the DPX file format includes per-frame metadata (potentially a great deal of it). This Wikipedia article has the basic info.
I'm also not clear on what you mean by 'bound permanently'. In the AVI file container the "I" stands for "Interleaved", where video and audio packets (roughly) alternate in the file. This is true for almost all container formats. The streams can be separated, but they natively co-exist in one meta-stream.
