The goal is to play a rtsp video stream in the chrome browser using Websocket and MSE. The structure of the system looks almost identical to https://stackoverflow.com/questions/73562944/ffmpeg-live-streaming-for-media-source-extensionsmse. This setup works in most situations but the moment B-Frames come into play everything breaks. FFmpeg starts to output Non-monotonous DTS in output stream and shortly after the video played via MSE freezes for a few seconds. As a layman my understanding of the situation is that:
There are decode timestamp (DTS) and presentation timestamp (PTS) and an RTSP stream does not carry DTS information but allows somehow to relay PTS information.
Without B-Frames DTS=PTS but when B-Frames occur, the frames have to be decoded ahead of time for later frames to be displayed correctly, resulting in a PTS > DTS for these frames.
A mp4 muxer requires monotonously increasing DTS and DTS <= PTS. As the RTSP does not have DTS ffmpeg deduces DTS from PTS by default which results in jumps when a B-Frame occurs, hinted by the output
Non-monotonous DTS in output stream. FFmpeg repairs the DTS by increment from the last valid DTS.Now DTS is running behind video playback time the video presentation breaks down. Even the mode
segmentsdoes not help here as ffmpeg rewrote DTS to make it monotonous for the mp4 container and the underlying h264 does not include any timestamps so reordering can no longer happen.
As I have no control over the existence of B-Frames in the RTSP stream, I must assist FFmpeg in the deduction of DTS or at least fix the decoding order. Important: The live stream can only have a latency around one second so that expensive operations like reencoding the stream without B-Frames is not a valid option! I would also prefer to not build a specific application with ffmpeg's lib and instead use FFmpeg only via command line.
Idea 1: Buffer around one second of the rtsp packets, reoder them by PTS and then generate DTS based on PTS. The reordering should effectivly prevent the issues of non monotonous dts increase and presentation. While I think this is the most promising solution, it does not seem to be possible with ffmpeg cli?
Idea 2: Offset PTS by one second and set DTS to decode as soon as possible that frames are always ready for presentation. This assumes that B-Frames can be decoded after their respective partners are already decoded. I am not sure if this is even possible because the decoder may uses B-Frames as direct dependecies for the frames afterwards?
Idea 3: Somehow export ISO BMFF with non monotonous DTS for each packet and let the MediaSource sort the segments by DTS. As far as I understand the segments mode reorders by PTS and not DTS, but maybe someone has more knowledge here?
I already did a ton of research but was not able to implement the mentioned ideas with ffmpeg cli. Does someone know how to do it or has someone successfully used a similar system? I greatly appreciate other suggestions or technical feedback to improve my understanding. Solutions using ffmpeg as a programming library are also welcome but I am more interested in cli.
Here the current ffmpeg command that works well without B-Frames:
ffmpeg -nostats -hide_banner -loglevel fatal -i <rtsp_src> -c copy -an -f mp4 -frag_duration 50000 -movflags +frag_keyframe+empty_moov+default_base_moof pipe:1
My understanding is based on these sources:
https://stackoverflow.com/questions/60181618/how-to-write-a-video-stream-containing-b-frame-and-no-dts-to-a-mp4-container
https://stackoverflow.com/questions/24354179/does-rtp-timestamp-represents-the-pts-of-the-audio-video-present-in-media
https://stackoverflow.com/questions/2439096/h264-rtp-timestamp
https://www.ietf.org/rfc/rfc3550.txt
https://www.w3.org/TR/mse-byte-stream-format-isobmff/
