Those are compression artifacts, where the original video has been compressed and has lost data. Block-based coding for quantisation leads to artifacts like this, as the information depth in the image is too high to be successfully compressed at the level required by the lossy algorithm used. From that Wikipedia page:
Lossy data compression involves ...
You can try adding the original_size filter option, such as:
Specify the size of the original video, the video for which the ASS
file was composed. For the syntax of this option, check the
Video size section in the ffmpeg-utils manual. Due to
a misdesign in ASS aspect ratio arithmetic, this is necessary ...
You can use a combo of the scale and setsar filters.
ffmpeg -i hdv-input.m2t -vf "scale=iw*sar:ih,setsar=1" -crf 28 -c:a aac -b:a 80k -ac 1 -movflags +faststart h-264-out.mp4
The scaler width to set to current width x sample/pixel aspect. If it's already 1, it amounts to new width = old width. I've added a setsar afterwards to ensure the output SAR is 1.
Assuming both MKVs are rips of NTSC DVDs, these are the commands you need:
For 4:3 video
ffmpeg -i "input43.mkv" -vf "scale=640x480,setsar=1,pad=854:480:107:0" [etc..]
For 16:9 video
ffmpeg -i "input169.mkv" -vf "scale=854x480,setsar=1" [etc..]
(Note H.264, the codec you are likely using, needs dimensions to be even, so specifying 853 as width will lead ...
I found some ASS files, like this sample, which contain a Video Aspect Ratio tag but it seems to not make a difference in ffmpeg hardcoding.
The workaround I can think of right now is to create a complex filter where you create a transparent RGBA canvas of 1024x576; burn the subtitles onto them, scale the result to 768x576 or 720x576, depending on your ...