... Depends...
There are many concepts that need to be differentiated. Let me explore a bit.
Information
The first one is that we need to think about an image as information. So we need to ask first. Is the information lost? Is the information modified? Is it improved? Keep this concept in mind.
Scaling
And now let me apply this idea to another word. Scaling.
Scaling "per se" does not modify the initial information. It only adapts it to conform to another viewing size or display.
Take your 320x240 video file and do nothing with it. Only play it on any monitor to full screen. The video is exactly the same. If viewed on a large screen you will surely notice the pixels, but that is your perception. The information on the video file is exactly the same.
Now let us actually scale the image so it is no longer a 320x240 file. Normally you would like to use round multiples. 2x, 3x, 4x. If you use a "nearest neighbor" scaling method, the information will be exactly the same.
A 240 P image 4 times would be a 960P image. But scaling it to 1080 has a compromise of how you scale some pixels, the factor is 4.5x so, either some pixels are scaled 4 times and some other 5 times, or you average some values.
Resampling
And here is the key. An operation you are now performing is not scaling, it is resampling. Taking some values, and converting them into other values.
If I say that John has $20 and Mike has $10. I have the exact information.
If I say that the average they have is $15 it destroys the original information. It could be that one has $30 and the other has nothing. Or that the amounts are inverted.
So, resampling is the one that compromises the information.
mainly from stretching pixels.
Here you are probably referring to the aspect ratio. Which is another thing to consider. It can be a resampling maintaining the aspect ratio. Keep that in mind.
I know anecdotally that if you take a 320x240 video file and upscale it to 1920x1080 that the quality is lost
Not really.
First of all, you perceive it as low quality because you are now used to see HD images, so you are adding how you perceive the resulting image.
Again, see that image on a watch, or use it on a nanotechnology screen on a label on a soda can, and you would find it impressive.
Quality is a process, not an end result... necessarily
The algorithm
Our capacity to percive information depends on the information itself.
Our eyes can detect detail if there is contrast. And we percive this contrast on the edges of a shape.
Some resampling algorithms try to enhance the detail on the borders of the shapes. Some try to enhance it, some try to generate new information guessing what values should be there.
At the end there is always a compromise on what do you expect to see, what tools are at our disposal, and how much of the original information you can, or want to keep.
P.S. Downsampling always reduces the ammount of information, but it is only important if the information is relevant.
A 4k video file of a completly white wall will probably not have relevant information, so we can resample safely to SD. But if we have a small text painted there, probably this text will be lost on the downsampling.
