One difference that wasn’t mentioned in other answers is frame downsampling. With a large sensor you have to downsample the image data read from the sensor to the final frame size – for example 1920✕1080 is just around 2 megapixels, whereas the full sensor output is easily ten times as much with recent full-frame cameras.
The problem with downsampling is that the camera is usually not able to do it fast enough while keeping the quality high, it needs to cut corners. As an example, this is a quote from a review of Sony RX-10, a camera with a small 1" sensor:
One of the big selling points on the Cyber-shot DSC-RX10 is its movie
mode. You wouldn't know it from looking at the already impressive spec
sheet, but the way the RX10 creates its videos allows for much greater
quality than most other cameras.
The reason for the higher quality is how video is sampled. Since the
sensors on the vast majority of cameras cannot be 'read out' quickly
enough, they don't read the whole sensor when recording video. Instead
they skip lines of pixels, which means that the amount of data used to
generate is frame is reduced. The gaps between these lines reduce the
resolution they can capture, and can introduce moiré.
The RX10, on the other hand, samples the entire 1"-type sensor, so no
lines are skipped. The results are noticeable both in our test scene
and real world samples.
In other words, smaller sensor can under some circumstances mean less downsampling artifacts and higher image quality.