Might be related to this other ticket?
I've been doing some prores tests for both color and alpha. Usually this isn't a big deal since my alpha channels are solid with a 1-2pixel transition to black. However I did some tests with a very gradual change in alpha value over many pixels and noticed some pretty bad alpha channel quantization with the prores_ks encoder. The Apple prores spec says for 4444(XQ) the alpha channel is 16bit and lossless so this is quite different from that. I've attached a comparison image.
ffmpeg -f image2 -framerate 24.0 -i tif/rgba_16bit.%3d.tif -filter_complex "colorspace=all=bt709:range=pc:format=yuv444p12:iall='bt601-6-625':irange=pc[main];alphaextract,format=gray16le[alpha];[main][alpha]alphamerge" -c:v prores_ks -profile:v 5 -pix_fmt yuva444p10le -alpha_bits 16 prores4444XQ_yuva.mov
Image below is actually 8bit due to imgur limitations so the 16bit and 10bit examples aren't really correct but you get the point.
Also it looks like
-alpha_bits 8 and
-alpha_bits 16 produce the same look so maybe its getting bad upstream?
profile:v 4which is regular 4444 and it still looks like that :(