I have a source video (
source.mkv, which is a 25fps 1080p x264 encoded video), and I have reencoded it using SVT-AV1, like this:
ffmpeg -i source.mkv -nostdin -f rawvideo -pix_fmt yuv420p - | SvtAv1EncApp -i stdin -w 1920 -h 1080 --fps 25 -b output.ivf ffmpeg -i output.ivf -c:v copy -c:a copy output.mkv
output.mkv can be played successfully in any video player. When I compare it with the source video file, it looks like the video quality is barely the same. So I proceed to confirm this using vmaf. The procedure I'm following is:
# cut ffmpeg -i source.mkv -t 00:01:00.000 -c:v ffv1 -an ref.mkv ffmpeg -i output.mkv -t 00:01:00.000 -c:v ffv1 -an dis.mkv # convert to yuv ffmpeg -i ref.mkv -pix_fmt yuv420p -vsync 0 ref.yuv ffmpeg -i dis.mkv -pix_fmt yuv420p -vsync 0 dis.yuv # run vmaf vmafossexec yuv420p 1920 1080 ref.yuv dis.yuv vmaf_v0.6.1.pkl --log vmaf_output.xml
I'm cutting the videos to just measure a portion of them. As you can see, I'm reencoding the video files with
ffv1. This is just to cut as precise as possible (I've also tried copying the stream, without reencoding, but the result is the same). I've also tried
ffmpeg2vmaf instead of manually converting files to YUV before calling vmaf, but the result is the same. It is important to say that I have followed this same methodology to compare x264 with x265 and I had not any problems with the vmaf score. However, when I look at the vmaf output, I get a nonsense value (I am expecting more than 90):
[drnoob@linux svt-av1]$ grep -o 'aggregateVMAF="[^"]*"' vmaf_output.xml aggregateVMAF="24.0494"
ffmpeg, vmaf and SVT-AV1 have been compiled manually by me using gcc 8.3.0. Moreover, specific versions are:
- ffmpeg: N-97127-g5a0575e
- vmaf: commit 31565cf4
- SVT-AV1: commit 882e97f
My research so far
I compare both videos side by side (the original x264 one and the output using av1) and it seems like the quality is perfect. So, the first thing I thought was this issue is caused because of both videos being not synced. It is interesting to note that, although the aggregate VMAF score is so low, the first frame gets a score of 90. If I run ffmpeg on both, I can see there are differences:
[drnoob@linux svt-av1]$ ffmpeg -i ref.mkv Duration: 00:01:00.00, start: 0.040000, bitrate: 149417 kb/s Stream #0:0: Video: ffv1 (FFV1 / 0x31564646), yuv420p, 1920x1080, SAR 1:1 DAR 16:9, 25 fps, 25 tbr, 1k tbn, 1k tbc (default) Metadata: ENCODER : Lavc58.77.101 ffv1 DURATION : 00:01:00.000000000 [drnoob@linux svt-av1]$ ffmpeg -i dis.mkv Duration: 00:01:00.00, start: 0.000000, bitrate: 108269 kb/s Stream #0:0: Video: ffv1 (FFV1 / 0x31564646), yuv420p, 1920x1080, 25 fps, 25 tbr, 1k tbn, 1k tbc (default) Metadata: ENCODER : Lavc58.77.101 ffv1 DURATION : 00:01:00.000000000
As you can see, the start time is not the same. However, the difference should not be the cause of such a big vmaf loss, so there must be another reason. I am also suspecting that my procedure to convert the SVT-AV1 output to a file might not be the best (from ivf to mkv), and may be causing some problems. However, decoding the file with a video player shows no issues.
Any idea? I don't think this is a vmaf bug, there must be a problem with my approach.