FFmpeg setpts filter produces wrong frame number

Question

Description

I found a strange behavior in setpts filter: sometimes when I use it to speed up a video it inserts more frames than expected.

My ffmpeg version is the following:

ffmpeg version 6.0-static https://johnvansickle.com/ffmpeg/  Copyright (c) 2000-2023 the FFmpeg developers
built with gcc 8 (Debian 8.3.0-6)
configuration: --enable-gpl --enable-version3 --enable-static --disable-debug --disable-ffplay --disable-indev=sndio --disable-outdev=sndio --cc=gcc --enable-fontconfig --enable-frei0r --enable-gnutls --enable-gmp --enable-libgme --enable-gray --enable-libaom --enable-libfribidi --enable-libass --enable-libvmaf --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librubberband --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libvorbis --enable-libopus --enable-libtheora --enable-libvidstab --enable-libvo-amrwbenc --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libdav1d --enable-libxvid --enable-libzvbi --enable-libzimg
libavutil      58.  2.100 / 58.  2.100
libavcodec     60.  3.100 / 60.  3.100
libavformat    60.  3.100 / 60.  3.100
libavdevice    60.  1.100 / 60.  1.100
libavfilter     9.  3.100 /  9.  3.100
libswscale      7.  1.100 /  7.  1.100
libswresample   4. 10.100 /  4. 10.100
libpostproc    57.  1.100 / 57.  1.100

Preparation

Use this video as input, taken from FFmpeg Wiki page about changing frame rate:

ffmpeg -f lavfi -i testsrc=duration=10:size=854x480:rate=60 -vf "drawtext=text=%{n}:fontsize=72:r=60:x=(w-tw)/2: y=h-(2*lh):fontcolor=white:box=1:boxcolor=0x00000099" test.mp4

The test.mp4 video has 60 fps, 600 total frames and a duration of 10 s.

Use this command to create a 2x version of the input.

ffmpeg -i test.mp4 -vf "setpts=0.5*PTS" test_2x.mp4

The output should be twice as fast as the input, dropping half the frames to maintain the same framerate in half the duration: it should be still at 60 fps, but have a duration of 5 s and 300 total frames.

Problem

The issue is that the output video has 302 frames, not the exact 300, and consequently it has a duration of 05.033333 s. As the input video has the frame number printed on it, I can see that the first frames are not correctly dropped.

This is what I get:

 input frame:  | 0 | 1 | 2 | 3 | 4 | 5 | 6 |  7 |  8 |  9 | ... |
output frame:  | 0 | 1 | 2 | 3 | 4 | 6 | 8 | 10 | 12 | 14 | ... |

Notice that the first 5 frames are replicated without any drop, while 2 of them (1 and 3) should have been dropped.

This is what I expected:

 input frame:  | 0 | 1 | 2 | 3 | 4 |  5 |  6 |  7 |  8 |  9 | ... |
output frame:  | 0 | 2 | 4 | 6 | 8 | 10 | 12 | 14 | 16 | 18 | ... |

The issue is present also if I use other encoders or containers to create my output file: I tried ffvhuff in an mkv file and raw video in an y4m file, with the same results.

The problem desappears if I add a fps=60 filter after the setps one.

Answer 1

setpts filter only retimes frames; it cannot add or drop frames.

The MP4 format is configured to require constant frame rate, and so ffmpeg's frame management (vsync) logic gets to work. This looks at the output frame rate, and the timestamps of the supplied frames to carry out frame duplication or drop decisions. But it first rebases the timestamps in terms of the output frame rate, and in this case, there seems to be some quirk in the output of that logic.

Instead of debugging it, I suggest you manually drop half the frames before retiming.

-vf "select=mod(n+1\,2),setpts=0.5*PTS"