How to specify different flags for inputs with FFMPEG?

Question

You can specify multiple inputs to FFMPEG and mix them into a single output. I understand the basics of this, for example:

ffmpeg -i vid1.mp4 -i vid2.mp4 [commands] output.mp4

What I can't figure out is how I can specify different commands for each input. For example, I have the following I'm trying to do:

 ffmpeg -i vid1.m2t -i vid2.mp4 -map 0:a -filter_complex "[0:v:0][1:v:0]vstack=inputs=2" -crf 24 -c:a aac -b:a 128k -ac 1 -movflags +faststart output.mp4

vid1.m2t is a 1440 anamorphic hdv mpeg stream, and vid2.mp4 is a regular 1920 h.264. Using -filter_complex "[0:v:0][1:v:0]vstack=inputs=2" I'm trying to stack vid1 on top of vid2, but I get the error that says it can't proceed because the two videos are not the same width. Well, yes, except in display the m2t is actually 1920 wide, not 1440.

When I'm converting just the hdv by itself, I use -filter_complex "scale=1920:1080,setsar=1" to ensure square pixels. I know that works well, but how can I incorporate that into this vstack command? If I put it in with the other I get errors. I've tried a variety of different ways, including before the input (thinking that would make ffmpeg apply it only to the input before sending it to the other commands for output), but they all make errors.

Additionally, I wonder if the -crf 24 command can be specified for each input. When converting individually the m2t needs to be at 24 or less, or it looks bad, while the mp4 can be as high as 30. I might be misunderstanding what vstack does, but I would think that if I can tell ffmpeg to give different attention to quality for each input would lessen file size.

---

I have the same issue with hstack. FFMPEG will render the output, but the m2t file comes out 1440 wide, therefore skewed. So the total video width is 3360 wide, instead of 3840.

---

Maybe more specific to this exact problem, how can I use vstack or hstack when the inputs have different dimensions?

Answer 1

Suppose a video stream is 1440x1080 with sample/pixel aspect ratio of 1.333, leading to a display of 1920x1080. Realize that the actual resolution of the stream, and what's stored in the file, is 1440x1080. The aspect ratio flag tells the consuming app (player/editor..) that the intended display proportion is different than the raw ratio of the width to height and that it should use the aspect ratio to rescale the video to produce an accurate display. FFmpeg will not automatically make a video square-pixel since the user may not want that.

The stack filters require that all streams have the same size at the interface that they are joined at - for hstack, that's height, and for vstack, that's width. Only the raw size matters, the aspect ratio is just an informational flag.

So, you'll have to rescale the first input before stacking.

Use

ffmpeg -i vid1.m2t -i vid2.mp4 -filter_complex "[0:v:0]scale=iw*sar:ih,setsar=1[vid1];[vid1][1:v:0]vstack=inputs=2" -map 0:a -crf 24 -ac 1 -c:a aac -b:a 128k -movflags +faststart output.mp4

As far as the encoder is concerned, it is just presented with one stream of 1920x2160 - the historical provenance and composition is lost. Now, if you want to apply different levels of compression to parts of the frame, that's called Region Of Interest encoding. Some initial patches were applied last month to allow this for x264 and x265, but it is not operational yet. I'll update this answer when it becomes so, but for now, you can only assign a single CRF value for the whole frame.