I'm setting up live streaming using three overlaid inputs in ffmpeg. O:v is a webcam directly connected to the streaming box (a headless linux box) while 2:v and 3:v are ultra-low latency UDP streams from two remote R-Pis.

The filter_complex argument I'm using (that works, with all three video inputs perfectly in sync) is [0:v] fifo [v1];[2:v] fifo [v2]; [3:v] fifo [v3];[v1]setpts=PTS-STARTPTS[sync1];[v2]setpts=PTS-STARTPTS[sync2]; [v3] setpts=PTS-STARTPTS+5/TB [sync3] ;[sync1][sync2] overlay=x=W-w:y=H-h [out1];[out1][sync3] overlay=x=0:y=H-h [vfinal]

What I don't understand is why I am having to add +5/TB to [sync3] via setpts (in bold above)? If I omit that, [3:v] is synced in to the stream noticeably ahead of the other two inputs. My best guess is that this may be to compensate for the processing time of the first overlay [sync1][sync2] which is introducing extra latency into that part of the stream.

Importantly - the value of the extra pts compensation has to be dialed in by hand and will be specific to the underlying hardware used. So the question is, is there a way of getting ffmpeg to calculate this automatically so I don't have to worry about keeping it dialed in, in production?

Thanks for reading! I'm at the limit of the ffmpeg docs here...


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.