Does the Sampling Rate affect the syncing process to a video source?

Question

I have recorded interviews using a point and shoot camera for the video and my android phone with Smart Recorder. The camera records in MOV and is uncompress with mono audio. Then I use my smartphone to record myself and the interviewee, since the camera is far away to hear us. The audio is in .wav and I use Premiere Pro to align them. While it does work at the start, sometimes it magically gets out of sync all of the sudden, so I have to cut the audio so that I can catch up.

I am puzzled, and would like to have your insights on how to keep this at a minimum if possible, eliminate it. My theory is that sampling rate is an issue. If this is incorrect, please advice otherwise.

Answer 1

You can tell which one gets out of synch like this:

Set up your camera and your phone like you usually do. Put a clock that shows seconds right in front of the camera. Then, even though this sounds boring, record yourself counting off the seconds.

Synchronize the video and audio in Premiere Pro like you usually do. When they get out of synch, check whether the video lost a second, or whether the audio lost a second. You should be able to tell based on the repetitive rhythm.

Then, as @AJHenderson pointed out, you should replace the piece of equipment that failed. I agree that a camera that can accept external audio input is probably the best option, though it's expensive.

Answer 2

There are any number of possibilities, but the most likely ones all involve getting better equipment. The most severe sync problems with phones tend to come from phones that use variable framerate video or tend to have cheap encoders that tend to drop or delay frames.

When you are recording video, each frame has to be recorded more or less in real time. There is a slight buffer available to store frames while they are processed, but if images are not processed at the rate they are shot, then eventually that buffer fills. If a phone can't keep up, it has two options, either drop a frame outright and skip to the next one or use a file format that lets it alter the frame rate on the fly and record fewer frames per second when it has trouble.

Both of these approaches often destroy any kind of audio sync. The variable frame rate, if well implemented, should limit drift a bit more since the frames should still advance at a fairly consistent rate of time, but they may get delayed and become out of sync. Dropped frames are generally far worse. Since nothing exists for that frame, the next frame is simply used and a slice of time is lost in the video that isn't lost in a separate audio stream that didn't have the dropped frame.

Both of these problems are unfortunately hardware dependent and there is no way to fix them on the hardware you have. (You could try to reduce resolution to make the job easier for the encoder I suppose.) If one of these is the source of the issue, getting another camera with a better encoder is the only other option you really have available if you want to maintain the resolution you are using.

Another possible problem that is less common today, is that the internal timecode clock may simply be fast or slow. For the most part frame timing is pretty consistent with modern technology, but if there is a problem, you can try syncing at the beginning and end of the video and adjust the audio speed accordingly. Check a few times throughout the middle when you do this, because most modern problems will probably not be this simple and the middle will still be out of sync.

An even better solution is to use a camera that can accept an external audio input. By syncing the audio to the video in real time while recording on the same device, you can ensure proper timing is maintained by the encoder. It's still probably a good idea to do an external recording as well, since this will help detect if there were capture problems, but using an external audio input is far simpler than dealing with sync issues after the fact.

Answer 3

A/V sync problems are noticeable at about +/- 0.1 sec. The samplerate clock in your phone and camera would have to be exactly the same speed to within 0.1 sec over 10 mins, or however long your clips are, for you to be able align at only one point.

0.1 / (10 * 60) = .0001. That's about 1 part in 10k. Wikipedia says quartz clocks are often as good as 6 parts per million stability. The stuff I skimmed in that and this indicate that it probably costs a bit extra to get the clock frequency that close out of the factory. (and the 6ppm is the kind of long-term drift you can expect around the starting point, not the absolute accuracy of typical gear.)

Keep in mind, most cameras, esp. ones primarily made for still-image photography, don't need a clock as accurate as a wristwatch, so they probably don't have one. Because that would cost extra.

I saw exactly this kind of drift between the audio from my Panasonic Lumix and audio recorded at the same time on a Dell laptop. See Sync separate audio to video+bad-camera-audio, free NLE recommendations, where I describe a slow way to manually sync clips, and ask for a quicker automated way.

Anyway, this lack of clock sync didn't surprise me at all. It's nothing to do with frame dropping, or bad quality gear. It should work fine to assume that both clocks involved each run at a constant speed, just not at the SAME speed. So you can correct a clip with a linear 2-parameter correction: offset + stretch.

I agree with AJ Henderson's suggestion that it's probably the camera's clock that's less likely to be right. Phones have GPS receivers for timekeeping, and may or may not use that accurate timesource as the clock for their audio input samplerate. And also as AJ points out, need to do wifi, which probably requires calibrated clock sources, unless they can just lock on to a received frequency. Not sure how big a deal accurate clocks are with modern wifi modulations, but probably a big deal.

I would have expected issues recording on any two devices that don't derive their sample clock from the same timesource. Until portable GPS receivers were available, movie studios had a master clock that distributed time to all A/V recording devices, so 1 real hour of footage from one camera would have exactly as many frames as 1 real hour of footage from another camera.