-pix_fmt to a yuv colorspace will use Bayer dithering by default. I don't know how to change that behaviour.
When downsampling with
-pix_fmt to an RGB colorspace like rgb24, no dithering is applied, resulting in ugly banding.
I figured out how to control dithering to YUV colorspace. My issue was just that I misunderstood the ffmpeg filter syntax regarding
format. It's not handled as a filter, but just tells the preceding filter what format to output:
The issue is, zscale seems to be unable to dither at all when downsampling to RGB colorspace. 4 bpc RGB seems to be the exception, where ordered dithering is enforced. And I am unable to control dithering with the scale/swscale filter at all, using a global flag
-sws_dither ed does not work.
colorspace filter can not output to RGB format.
paletteuse seems promising, but I have not figured out how to use it yet, seems like you need to use a filter complex for that.