I am building a small tool to take the short videos I record from different devices (e.g. iPhone, camcorder, etc), and concatenate them together chronologically. Everything is working up to the point of resizing/padding the videos before concatenation. My solution right now is to just take the max width and max height and then center and pad all the videos accordingly. This doesn't look great though.
For example, I have a 1280x720 stream from my camcorder and a 1920x1080 vertical (i.e. rotated) video from my iPhone. My current solution leads to a lot of padding and a square image.
When I use Shotcut the output looks quite visually appealing: the camcorder video is full screen, and the iPhone video is rotated and then padded. However, it seems as though it must scale the 1280x720 stream to 1920x1080 as the final mp4 file is 1920x1080. I'm not sure I want scaling as the default behavior though. For example, if I had a 4k video and a 1280x720, I imagine the scaling would look terrible.
I know that I can always add selectors to choose what resolution I want, but I found that the production software I looked at (Shotcut, Premiere, etc.) seem to abstract these decisions away.
Is there a optimal way to make this decision? What the standard way of handling these differing resolutions when combining videos in software like Shotcut or Premiere?