FFMPEG VMAF consistently low rating despite quality encode

Question

I've seen similar issues but can't seem to come to a resolution myself. I have a 4K rip that is in hevc, 10bit depth, which my older phone can't play since it lacks GPU acceleration. Therefore, I'm reencoding it on ffmpeg to a YUV420p scheme. I've done this by GPU acceleration and without, at multiple qualities and bitrates. I figured I'd measure each output against the original video and simply use the one with the higher VMAF score as the one I use to reencode some other 4K videos with the same 10bit depth.

However, the VMAF score for every output, whether CPU or GPU encoded, 1M or 10M bitrate, is somewhere around 32.2%. This is despite the fact that the higher bitrate ones generally look pretty decent. After looking around, I saw that some people recommend using a filter for setpts=PTS-STARTPTS to sync up the frames, since maybe the encoding process desynced some frames? This still doesn't help.

Here's where it gets weird. I looked in the log when running the VMAF filter to pick one frame in particular and test it. I found one frame where the VMAF score reported a score of 0.0000:

    <frame frameNum="487" integer_motion2="7.639184" integer_motion="9.133736" integer_adm2="0.516201" integer_adm_scale0="0.902589" integer_adm_scale1="0.619710" integer_adm_scale2="0.445753" integer_adm_scale3="0.428904" integer_vif_scale0="0.443259" integer_vif_scale1="0.513409" integer_vif_scale2="0.532767" integer_vif_scale3="0.533751" vmaf="0.000000" />

Then in ffmpeg I ran a command to extract this frame in both source and encoded with something like:

.\ffmpeg.exe -i "(sourceVid)" -vf "select=eq(n\,487)" -vframes 1 source.png"

to just get a sample of the frame that VMAF has ranked at 0. Looking at the two pngs, they seem identical to the eye. When running the two pngs through VMAF in ffmpeg, they report a score of 98, which is what I actually expect and is obviously not 0.

This is driving me mad. I can't tell why VMAF would keep reporting a much lower rate than expected. It does seem like an issue with frame syncing, but I can't see a way to resolve it. Any help would be appreciated

Answer 1

I've noticed that sometimes either the encoder or FFmpeg itself likes to drop frames they deem as duplicates, possibly with the intent that a decoder will just display the same frame multiple times on playback and thus saving some bitrate by not having to encode the same frame twice (or more). This doesn't play nice with calculating VMAF or other metrics afterwards due to desynchronization of frames between the reference and target files, leading to invalid scores because different frames are being compared.

To get around this, I've started adding -fps_mode passthrough to my encoding commands (-vsync 0 for FFmpeg versions prior to 5.1). This way all frames are passed on as-is.

To see if this is the issue affecting you, simply count the number of frames in your files. This can be done with ffprobe:

ffprobe -v error -count_frames -select_streams v:0 -show_entries stream=nb_read_frames -of default=nokey=1:noprint_wrappers=1 -threads auto input.mp4