Bad quality encodes to Prores4444 with alpha from ffmpeg

Question

EDIT: This bug has been fixed. If you're experiencing the problem below, try updating ffmpeg

I need to supply video with alpha as a prores4444 encoded file. I've been trying to use ffmpeg, but for once it seems to be letting me down (I know, gobsmacked!).

I'm using the prores_ks codec and following the intructions on the man page this is my command, testing on a QTRLE encoded file with 8-bit RGBA:

ffmpeg -i '.\alpha test.mov'  -c:v prores_ks -profile:v 4 -quant_mat 'hq' -pix_fmt yuva444p10le 'prorestest2.mov'

The output is:

ffmpeg version N-82664-g801b5c1 Copyright (c) 2000-2016 the FFmpeg developers
  built with gcc 5.4.0 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-dxva2 --enable-libmfx --enable-nvenc --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libfreetype --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-lzma --enable-decklink --enable-zlib
  libavutil      55. 41.101 / 55. 41.101
  libavcodec     57. 66.108 / 57. 66.108
  libavformat    57. 58.101 / 57. 58.101
  libavdevice    57.  2.100 / 57.  2.100
  libavfilter     6. 67.100 /  6. 67.100
  libswscale      4.  3.101 /  4.  3.101
  libswresample   2.  4.100 /  2.  4.100
  libpostproc    54.  2.100 / 54.  2.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '.\alpha test.mov':
  Metadata:
    major_brand     : qt
    minor_version   : 537199360
    compatible_brands: qt
    creation_time   : 2016-12-12T03:17:14.000000Z
  Duration: 00:00:10.00, start: 0.000000, bitrate: 406275 kb/s
    Stream #0:0(eng): Video: qtrle (rle  / 0x20656C72), bgra(progressive), 1920x1080, 405436 kb/s, SAR 1:1 DAR 16:9, 25 fps, 25 tbr, 25 tbn, 25 tbc (default)
    Metadata:
      creation_time   : 2016-12-12T03:17:14.000000Z
      handler_name    : Apple Alias Data Handler
      encoder         : Animation
      timecode        : 00:00:00:00
    Stream #0:1(eng): Data: none (tmcd / 0x64636D74), 0 kb/s (default)
    Metadata:
      creation_time   : 2016-12-12T03:17:14.000000Z
      handler_name    : Apple Alias Data Handler
      timecode        : 00:00:00:00
File 'prorestest2.mov' already exists. Overwrite ? [y/N] y
Output #0, mov, to 'prorestest2.mov':
  Metadata:
    major_brand     : qt
    minor_version   : 537199360
    compatible_brands: qt
    encoder         : Lavf57.58.101
    Stream #0:0(eng): Video: prores (prores_ks) (ap4h / 0x68347061), yuva444p10le, 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 25 fps, 12800 tbn, 25 tbc (default)
    Metadata:
      creation_time   : 2016-12-12T03:17:14.000000Z
      handler_name    : Apple Alias Data Handler
      timecode        : 00:00:00:00
      encoder         : Lavc57.66.108 prores_ks
Stream mapping:
  Stream #0:0 -> #0:0 (qtrle (native) -> prores (prores_ks))
Press [q] to stop, [?] for help
frame=  250 fps=5.5 q=-0.0 Lsize=  186122kB time=00:00:09.96 bitrate=153082.2kbits/s speed=0.218x
video:186119kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.001761%

It all looks good, unitl you see the results. Here is a frame from the original, exported as a png (using djv):

…and here is the prores encoded version. There's blockiness and ringing artefacts that are most apparent in areas that have partial alpha:

It's most apparent if you look at the colour channels individually:

I've tried the different -quant_mat settings (in the docs it looks like there should be 6, but if I try -quant_mat 5 I get an out of range error. I also tried using the string version, i.e. -quant_mat 'hq'. Another thing I tried was changing the number of alpha bits using -alpha_bits 16 and -alpha_bits 8. The problem was worse with 16 bit alpha.

So is it something I'm doing or is it the codec not working properly?

Answer 1

Looks to be an encoder deficit, probably due to the rapid transition in the alpha from 1 to 0 in the border regions. Outputting to VP9 with alpha produces a clean result as does outputting to prores without alpha. Outputting with an uniform alpha i.e. whole alpha plane reset to 0 or 0.5 or 1 also produces a clean result. It's not a scaler issue as rawvideo with yuva444p10le also produces clean result.

Looking at the source code, I see this line, which may be relevant:

// todo alpha quantisation for high quants