I wish to stream a video with dynamic metadata. I know I could just steam the video and metadata separately, but ideally I'd like them to be bound permanently.

Is there an existing protocol / encoding / format that does this? I could store the data in each frame using steganography, but this is not an ideal solution and is rather hacky.


If the frame-rate is not too high and you can associate a frame with a precise timestamp then the easiest would be to add the metadata as textual subtitles.

The other option is to mux the metadata as standard headers in .mp4 or similar container format. .mp4 can be broken down into multiple fragments, each one with its own header, and theoretically this can be done per frame. However, doing that for more than a few frames is extremely inefficient in terms of file size and performance. A variation of this method would be to extract every frame as an image (e.g. JEPG), and set its metadata.

Yet another option is to use stenography without any hacking, you just encode the metadata into a separate stream which is timestamp-synced with the main video stream. Most modern container formats, including .mp4, can accommodate multiple video streams in the same container (e.g., you can label the encoded metadata stream as the same content but in another language).

  • Subtitle streams is what I was thinking of, too. Make sure you flag it as non-default, or add an empty subtitle stream as the first sub stream, so players that aren't aware of your custom sub format don't display it as text. – Peter Cordes Jun 5 '15 at 20:07
  • ffmpeg distinguishes between subtitle and other data streams. e.g. there's stuff like RTP metadata in some video files, and it's a separate metadata stream in the mp4 container. So depending on what you need, you might be able to just use a custom data stream muxed in mp4 or mkv. – Peter Cordes Jun 5 '15 at 20:09

I'm not sure how this fits into a streaming strategy, but the DPX file format includes per-frame metadata (potentially a great deal of it). This Wikipedia article has the basic info.

I'm also not clear on what you mean by 'bound permanently'. In the AVI file container the "I" stands for "Interleaved", where video and audio packets (roughly) alternate in the file. This is true for almost all container formats. The streams can be separated, but they natively co-exist in one meta-stream.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.