Combining two live RTMP streams into another RTMP stream, synchronization issues (with FFMPEG)

Question

I'm trying to combine (side by side) two live video streams coming over RTMP, using the following ffmpeg command:

ffmpeg -i "rtmp://first" -i "rtmp://first" -filter_complex "[0v][1v]xstack=inputs=2:layout=0_0|1920_0[stacked]" -map "[stacked]" -preset ultrafast -vcodec libx264 -tune zerolatency -an -f flv output.flv

In this example I actually use the same input stream two times, because the issue is more visible this way. And the issue is in the output two streams are out of sync by about 2-3 seconds. That is - I expect (since I have two identical inputs) to have exactly the same left and right sides in the output. Instead - left side is behind right side by 2-3 seconds.

What I believe is happening is ffmpeg connects to inputs in order (I see this in output log) and connection to each one takes 2-3 seconds (maybe it waits for I-frame, those streams have I-frame interval of 3 seconds). Then probably, it buffers frames received from first (already connected) input, while connecting to the second one. When second one is connected and frames from both inputs are ready to be put through the filter - first input buffer already contains 2-3 seconds of video - and result is out of sync.

Again, that's just my assumptions. So, how can achieve my goal? What I basically want is for ffmpeg to discard all "old" frames received before BOTH inputs are connected OR somehow put "empty" (black?) frames for second input, while waiting for that second input to become available. I tried play with various flags, with PTS (setpts filter), but to no avail.

Answer 1

I got it working by adding -fflags nobuffer -flags low_delay -strict experimental before all the inputs. Got helped reading this thread

Answer 2

-vsync parameter

Video sync method.

• 0: Each frame is passed with its timestamp from the demuxer to the muxer
• 1: Frames will be duplicated and dropped to achieve exactly the requested constant framerate.
• 2: Frames are passed through with their timestamp or dropped to prevent 2 frames from having the same timestamp
• -1: Chooses between 1 and 2 depending on muxer capabilities. This is the default method.

With -map you can select from which stream the timestamps should be taken. You can leave either video or audio unchanged and sync the remaining stream(s) to the unchanged one.

-async samples_per_second

Audio sync method. "Stretches/squeezes" the audio stream to match the timestamps, the parameter is the maximum samples per second by which the audio is changed. -async 1 is a special case where only the start of the audio stream is corrected without any later correction.

-copyts

Copy timestamps from input to output.