Why video call has no latency, but no matter what options of ffmpeg I tried to transmit video, the lowest I can get is ~0.2s latency?

Question

Why video call has no (even if there is, it is lower than human perception can discern) latency, but no matter what options of ffmpeg I tried to transmit video to a remote server, the lowest I can get is ~0.2s latency (estimated, not sure how to profile)?

What is the technology behind video call that allows it to be much faster than ffmpeg?

I tried sending to the server and make the server send back the stream using the following commands:

CLIENT:

termimal 1-

ffmpeg -pixel_format mjpeg -s hd720 -r 30 -i /dev/video1 -c libx264 -preset ultrafast -tune zerolatency -crf 18 -b 5M -f matroska -vf hflip tcp://X:5000

termimal 2-

mplayer -benchmark ffmpeg://tcp://Y:5000?listen

SERVER:

ffmpeg -i tcp://X:5000 -c copy -f matroska tcp://Y:5000

However, the video I got back has a slight delay of about 0.5s (estimated, not sure how to profile). Is sending to and fro different that causes it to have longer delay?

Answer 1

Try using UDP. It should be faster than TCP but with possibility of losing paackets.

Answer 2

Your ffmpeg command is very strange at best:

1. Using hflip filter: you need fairly recent ffmpeg, otherwise it will add additional processing overhead.
2. Conflicting encoding options, you should use only preset for h264.

Some other remarks: have you benchmarked your encoding command without network transfer, how fast it is, 1x or 10x realtime?

Answer 3

Sources of any delays:

1. TCP protocol. Long handshake, possible error with time loss in future
2. Long encoder initialization (video call already prepared it)
3. Long decoder initialization (video call already prepared it), buffering for analyze data
4. Unoptimized protocol / format for low-time communication, like MKV
5. Prebuffering in encoder for B-frames and other optimizations

Video conferencing software also have many tricks to detect delays and resync streams. It why in video conferencing has frame drops and it not suitable for transmitting high quality video.