Standard way for handling different resolutions when concatenating videos

Question

I am building a small tool to take the short videos I record from different devices (e.g. iPhone, camcorder, etc), and concatenate them together chronologically. Everything is working up to the point of resizing/padding the videos before concatenation. My solution right now is to just take the max width and max height and then center and pad all the videos accordingly. This doesn't look great though.

For example, I have a 1280x720 stream from my camcorder and a 1920x1080 vertical (i.e. rotated) video from my iPhone. My current solution leads to a lot of padding and a square image.

When I use Shotcut the output looks quite visually appealing: the camcorder video is full screen, and the iPhone video is rotated and then padded. However, it seems as though it must scale the 1280x720 stream to 1920x1080 as the final mp4 file is 1920x1080. I'm not sure I want scaling as the default behavior though. For example, if I had a 4k video and a 1280x720, I imagine the scaling would look terrible.

I know that I can always add selectors to choose what resolution I want, but I found that the production software I looked at (Shotcut, Premiere, etc.) seem to abstract these decisions away.

Is there a optimal way to make this decision? What the standard way of handling these differing resolutions when combining videos in software like Shotcut or Premiere?

Answer 1

I have written various approaches how to fix this below and you can automated the ones from the first two blocks with a clever rule set. But the content-dependent options should be done manually. So if you want to use this, maybe you find a way to tag those videos first by hand, for example by using metedata and your tool can then read the metadata and apply the correct solution. Anyway, there is no real standard way and it's not really common for professional productions to deal with clips in different frame sizes. If so, it is well planned and those scenes are usually filmed in a way that parts of the frame can easily be cropped away afterwards.

Videos with the same aspect ratio

You can downscale to the lowest common resolution if all the videos share the same aspect ratio. As you correctly assume, upscaling would not always look good. Downscaling will not be noticable to the general viewer.

Videos with different aspect ratio

If the videos don't have the same aspect ratio you have to add black bars horizontally or vertically. My recommendation is to use only black bars for 2 different frame sizes, not more. This means some videos will be full screen and all the rest will have the same sized bars. Either keep all bars horizontally or vertically. Anything else might confuse the audience too much. In case you have 3 or more different frame sizes, I recommend to manually crop them before starting your automated solution.

Options depending on content

If it works for your story, you can render mutliple source clips in the same frame, for example take 4 lower resolution clips, put each into a different quarter of the frame, maybe with seperating bars in between. It then resembles like a wall of screens. Generally this works well if you want to show things happening in parallel. One example is filming various instrument players individually while they perform a musical piece. This is also a good solution, when you have vertical videos shot with a smartphone: you could combine 2 or 3 in the same frame, just put them next to each other.

Another option is picture in picture, which is often used for all kinds of presentations, where you have either slides or a shot of the main stage filling the screen and a smaller overlay with the video of the speaker(s) near one of the corners.