EDIT: This bug has been fixed. If you're experiencing the problem below, try updating ffmpeg

I need to supply video with alpha as a prores4444 encoded file. I've been trying to use ffmpeg, but for once it seems to be letting me down (I know, gobsmacked!).

I'm using the prores_ks codec and following the intructions on the man page this is my command, testing on a QTRLE encoded file with 8-bit RGBA:

ffmpeg -i '.\alpha test.mov'  -c:v prores_ks -profile:v 4 -quant_mat 'hq' -pix_fmt yuva444p10le 'prorestest2.mov'

The output is:

ffmpeg version N-82664-g801b5c1 Copyright (c) 2000-2016 the FFmpeg developers
  built with gcc 5.4.0 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-dxva2 --enable-libmfx --enable-nvenc --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libfreetype --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-lzma --enable-decklink --enable-zlib
  libavutil      55. 41.101 / 55. 41.101
  libavcodec     57. 66.108 / 57. 66.108
  libavformat    57. 58.101 / 57. 58.101
  libavdevice    57.  2.100 / 57.  2.100
  libavfilter     6. 67.100 /  6. 67.100
  libswscale      4.  3.101 /  4.  3.101
  libswresample   2.  4.100 /  2.  4.100
  libpostproc    54.  2.100 / 54.  2.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '.\alpha test.mov':
    major_brand     : qt
    minor_version   : 537199360
    compatible_brands: qt
    creation_time   : 2016-12-12T03:17:14.000000Z
  Duration: 00:00:10.00, start: 0.000000, bitrate: 406275 kb/s
    Stream #0:0(eng): Video: qtrle (rle  / 0x20656C72), bgra(progressive), 1920x1080, 405436 kb/s, SAR 1:1 DAR 16:9, 25 fps, 25 tbr, 25 tbn, 25 tbc (default)
      creation_time   : 2016-12-12T03:17:14.000000Z
      handler_name    : Apple Alias Data Handler
      encoder         : Animation
      timecode        : 00:00:00:00
    Stream #0:1(eng): Data: none (tmcd / 0x64636D74), 0 kb/s (default)
      creation_time   : 2016-12-12T03:17:14.000000Z
      handler_name    : Apple Alias Data Handler
      timecode        : 00:00:00:00
File 'prorestest2.mov' already exists. Overwrite ? [y/N] y
Output #0, mov, to 'prorestest2.mov':
    major_brand     : qt
    minor_version   : 537199360
    compatible_brands: qt
    encoder         : Lavf57.58.101
    Stream #0:0(eng): Video: prores (prores_ks) (ap4h / 0x68347061), yuva444p10le, 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 25 fps, 12800 tbn, 25 tbc (default)
      creation_time   : 2016-12-12T03:17:14.000000Z
      handler_name    : Apple Alias Data Handler
      timecode        : 00:00:00:00
      encoder         : Lavc57.66.108 prores_ks
Stream mapping:
  Stream #0:0 -> #0:0 (qtrle (native) -> prores (prores_ks))
Press [q] to stop, [?] for help
frame=  250 fps=5.5 q=-0.0 Lsize=  186122kB time=00:00:09.96 bitrate=153082.2kbits/s speed=0.218x
video:186119kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.001761%

It all looks good, unitl you see the results. Here is a frame from the original, exported as a png (using djv): enter image description here

…and here is the prores encoded version. There's blockiness and ringing artefacts that are most apparent in areas that have partial alpha:

prores444 encoded version

It's most apparent if you look at the colour channels individually:

enter image description here

I've tried the different -quant_mat settings (in the docs it looks like there should be 6, but if I try -quant_mat 5 I get an out of range error. I also tried using the string version, i.e. -quant_mat 'hq'. Another thing I tried was changing the number of alpha bits using -alpha_bits 16 and -alpha_bits 8. The problem was worse with 16 bit alpha.

So is it something I'm doing or is it the codec not working properly?

  • Can you provide one frame from the original as a PNG? – Gyan Dec 2 '16 at 5:24
  • I've edited the question with pngs from the movie instead of screenshots. The blockiness is harder to see against white. – stib Dec 4 '16 at 23:14
  • @stib Did you also test prores_aw? It's another ProRes encoder. I didn't try or look at the source code: got to install drywall instead... – llogan Dec 5 '16 at 16:39
  • oh that's easy: cd ~/drywall; ./configure && make && sudo make install – stib Dec 5 '16 at 23:45
  • Gave it a try, no alpha. ffmpeg -i '.\alpha test.mov' -c:v prores_aw -profile:v 3 -pix_fmt yuva444p10le 'prores_aw-test.mov'Incompatible pixel format 'yuva444p10le' for codec 'prores_aw', auto-selecting format 'yuv422p10le' – stib Dec 6 '16 at 0:02

Looks to be an encoder deficit, probably due to the rapid transition in the alpha from 1 to 0 in the border regions. Outputting to VP9 with alpha produces a clean result as does outputting to prores without alpha. Outputting with an uniform alpha i.e. whole alpha plane reset to 0 or 0.5 or 1 also produces a clean result. It's not a scaler issue as rawvideo with yuva444p10le also produces clean result.

Looking at the source code, I see this line, which may be relevant:

// todo alpha quantisation for high quants
  • Should I raise a bug report, or is it redundant, given that it's already a 'todo' item – stib Dec 5 '16 at 6:16
  • Raise a bug report, as that may raise priority. – Gyan Dec 5 '16 at 6:17
  • Link to bug report. – Gyan Dec 7 '16 at 13:36
  • @stib The bug report is marked as fixed. Can you verify? – Gyan Aug 1 '17 at 17:45
  • Thanks for the heads-up. It appears to be working, but I haven't fully tested it yet. – stib Aug 3 '17 at 4:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.