Sync separate audio to video+bad-camera-audio, free NLE recommendations

Question

My use-case, in case the details help people understand what I'm looking for: My brother needs some videos of him singing while playing the piano, to send in as an audition for a dinner theater. For this job, he's already met the musical director, so this isn't a first impression, but obviously we want him to look and especially sound good. I'm not picturing putting in any video effects or titles / credits. (We figure metadata should be meta, like web page title or filename, and stored in the video container metadata, not burned into the video.) So the target for these videos is to be watched on a computer, probably via youtube, not DVD or broadcast.

We used a digital camera to record video, but the audio quality isn't good enough. We recorded audio at the same time with an analog mic hooked up to his laptop (with Audacity), but it needs to be manually synced. More details about the details at the bottom, for anyone that's curious. The camera and laptop clocks don't quite agree on how long a second is: the camera actually records at about 29.981 fps, when it thinks its recording at 30fps (with its audio staying in sync).

So I have 4 videos, 7 to 22 mins long, and 4 audacity projects with better audio for each video (which I can just export to wav or flac), but not synced. Desired output: one file per song, with audio from audacity mangled as little as possible

I need to:

• align the start of each external audio (wav or flac) with the synced but bad-sounding audio that comes in the same file as each vid
• stretch the external-audio or a+v cam clip so they align at the end, too
• discard the audio that came with the video file (or mute it and keep it around for reference so I can go back and fix the alignment if I notice a problem, without starting from scratch.)
• select the time range of the "good take" of each song.
• export these non-overlapping segments of my source videos, one output file per song. (The longer source vids have multiple songs)
• encode each song with ffmpeg / x264 / libfdk-aac and upload to youtube, and/or stick on my website with an HTML5 video tag. Export from the NLE in a lossless format that I can feed to ffmpeg is fine.

I know what I'm doing with ffmpeg / x264 / libfdk-aac / flac / libmp3lame / mp4 container to make final files to put on youtube and/or on my website; I don't need help with that part.

I did the a/v sync part of the problem for the first video, as a proof-of-concept before recording more:

In audacity, I loaded the audio from the camera .MOV, and found I needed to shift the external-mic track left by 5.34563s to align with the camera track. I did that, and cut off the part before zero secs. Then at the end, I found the camera track had to be shifted right 0.43144s to align with the external-mic track at 11m:30s. So I had to make the video 0.431 sec longer to stay synced with the audio. I exported the shifted and clipped external-mic track to a .flac from audacity, after using the noise-reduction filter.

Then I fed that to ffmpeg: ffmpeg -i session2-shifted.flac -an -i P1000669-sess2.MOV -shortest -c:a libmp3lame -q:a 1 -c:v libx264 -preset fast -crf 18 -filter:v "setpts=690.43144*PTS/690,transpose=2:passthrough=portrait" -r "30 * 690 / 690.43144" -movflags +faststart out.mp4

(I only had ffmpeg's built-in not-very-good AAC encoder in the windows build of ffmpeg I put on my brother's laptop, hence the mp3 audio for a test-run.)

So my output is a 29.981 fps mp4, with A/V sync from start to finish. (29.981 is close to NTSC 30/1.001 ~= 29.976, but that's just a coincidence.) I should be able to re-run that with lossless output (or -codec copy), and then chop out the sections I want. Either with a NLE, or note a start/stop time, and use ffmpeg -ss [start] -to [stop] -i session2-synced.mp4 -x264 -blah -blah song1.mp4

I got my brother to clap at the start AND end of each video while the external sound was recording, so we have sync marks that should be visually obvious in the waveform for the last 3 vids. (Aligning the first one was a bit tricky, glad I did one by hand before recording the rest).

I'm looking for recommendations for software to get all that done more quickly / easily than the manual process I used.

I use GNU/Linux, my brother uses Windows. Ideally, there's an NLE that's good for this use-case and is cross-platform, so I can show my brother how to cut segments out of a longer video that he records on his phone or something, with the same software I learn to use. I'm only interested in open source software. I'm not interested in investing time into learning my way around something that isn't open source. (And not going to spend money on it when I can already do what I need with audacity and ffmpeg.) Don't omit a good suggestion just for lack of cross-platform support, though, please.

There are many options on GNU/Linux and it's even been asked on Ask Ubuntu.

I know what a NLE is supposed to be able to do, but haven't used one before. (Just ffmpeg / mencoder with start / stop times covered the minimal stuff I've done before.)

I'm aware that usually people stretch and pitch-correct audio to fit video, but I trust the clock in the laptop more than the clock in the camera. And I want to keep audio mangling to a minimum. Although I already need some pretty serious denoising / 60Hz removal on the external-mic audio, so it's hardly pristine super-high-quality. So I'm willing to consider a solution that stretches/pitch-corrects audio instead of making non-standard fps videos.

I already tried pitivi, but the audio waveform display wasn't very visible, so I'd probably still have to do the syncing in audacity. As installed from the Ubuntu 14.04 package, it can't export with x264, and the only lossless codec is Dirac. (no ffvhuff...) Lossless dirac export is unacceptably slow, so that's another big downside. (I love open source / patent-unencumbered codecs like VP9, but I don't want anyone to have any hassles playing the vids, so I'm going with H.264 video and high-bitrate AAC audio. Or does youtube accept VP9 uploads?)

I've used avidemux, too, but IIRC it doesn't do well outputting to modern containers. It def. doesn't like reading codecs with b frames from modern containers, so it's pretty much a dead end. (h.264 in mkv or mp4, anyone?) It can do audio from an external file, but doesn't show waveforms for syncing.

This question is similar to others that have been asked before:

• How can I record audio & video synced with another audio file?
• How to automatically synchronize two recordings of the same event?
• Free software to synchronize audio and video recorded at the same time
• Non-Linear Video Editor for Free or Cheap That Does These Things
• Video desyncs with the audio over time

The first question has one answer that is sort of what I'm looking for: http://auxmic.com/, but that's Windows only, and it doesn't say anything about how it handles clock drift in the input. I might give it a try, since it's prob. simple enough to work fine with WINE in Linux. I'd still need to use something else to cut out the good take of each song from the videos.

None of the other answers have any useful suggestions, maybe because the questions weren't specific enough.

For those curious about what I actually used:

camera was a Panasonic Lumix FZ28 on a tripod, maybe 2m from the subject, high enough to get a good view of the piano keys and his face. It records 1280x720 or 640x480 MJPEG + 16kHz mono PCM in a MOV container, on SD cards. Video quality is pretty good, given enough light. At 720p, it can go for just under 12mins before hitting the 2GB max file size for FAT32, and stopping. (22Mbit/s or so, obviously needs to be xcoded for final use.) The video scaler in the camera is bad, and introduced some ugly banding from aliasing effects or something in the wall hanging behind my brother when we recorded in 640x480 (11Mb/s MJPEG). We used 640x480 anyway to let the camera run longer, since each separate shot takes manual labour to A/V sync. On top of the time to just select a start/end point and export a file.

The mic was just a cheap Labtec computer desk mic we had lying around. It picked up a lot of 60 Hz interference, even when its cord wasn't near the power cables for the lights and digital piano. (Even with the laptop on battery power.) Audacity's noise removal function does a good job of killing the 60Hz and background hiss. Having the mic sitting right on top of the piano picked up a lot of the sound of physically hitting the piano keys. If we'd had time before he was heading back to school, I would have used something for a mic stand to put the mic farther from the piano keys, and not in physical contact with the body of the digital piano.

I made a separate project in Audacity for each video file, since the A/V offset would be different for each one. I could have just made one big audacity project, but I don't think that would have helped with anything, except maybe being able to tweak noise-reduction / highpass (to maybe reduce the noise of piano keys being hit) filter settings and then apply it to everything at once.

Answer 1

I tried kdenlive, so I'll post my findings about it as an answer. It didn't quite do the job, so I'm not going to mark this as the accepted solution.

kdenlive easily imports my clips in mjpeg+pcm, and flac. And looks like it can export through ffmpeg, which is what I want.

It has a feature to "set audio reference", and for other tracks, "align audio to reference". That's perfect, programmatically aligning audio is much nicer than manually dragging stuff to align waveforms by eye. However, it doesn't seem to handle the clock drift at all. In my 23min shot, it had perfect sync at about 4 mins. It was a touch out of sync at the start, and maybe 0.5s out of sync at the 19 minute mark. (I don't have a good sync point later than that, since that shot hit the 2GB file size limit, xD)

KDENlive's timeline display is in h:mm:ss.frames, not decimal fractions of a second, which confused me at first. I was thinking that it had stretched one track, but no, it didn't.

It's easy to tell there's a problem by leaving both audio tracks unmuted, even though that will reveal problems too small to be noticed visually. (In my longer tracks, like this one, the drift eventually gets big enough to be a problem for visual AV sync.)

I checked manually in audacity: With the start of both tracks aligned, they're about 0.68s apart at the same 19min sync mark. So I could do my aligning in kdenlive manually, but its automatic tool for doing it unfortunately doesn't support clock-drift correction by stretching.

kdenlive does display the audio waveforms nicely, esp. if you go configure->timeline->track height: increase to 80 or 100pix. Nice UI for zooming way in on the timeline.

If I end up doing the AV sync in audacity before exporting, I might well use kdenlive for the rest of the process. I'll have to see how easy it is to make multiple videos from segments of a single timeline.

Answer 2

Try Blender's Video Editing.

If Video and audio where recorded at the same time it will be easy to sync them, just be sure your video frame rate is correctly set in Blender and that you check synced in the video editor (at the bottom beside the time line and play, stop, etc... buttons.

Don't be scared by Blender's complexity, you are not going to use any of the others features, you are just going to use video editing and that is pretty straight forward.

Answer 3

The following response is the methodology of how to sync an external audio recording to your video. This method assumes that both your audio and video recordings are being recorded at the same time rate.

Prior to recording each scene, you need to make a loud clap sound that both the audio input of your video camera, and the external audio recorder can pick up at the same time. Typically, a clapper or hand claps are used for this purpose. A clapper is a framed chalkboard or whiteboard with scene information that is recorded by the video camera, and has an arm that is raised and clapped down to make the needed sound. The clap, when recorded, creates a spike in the audio of both the video camera's audio and the external sound recorder audio. These spikes are illustrated in the sounds waveform view with larger spikes if the sound is loud enough. When you align these two waveform spikes in your time line, the video and external sound are aligned and synced perfectly.

If you record your audio and video in one setting without breaks and without stopping the camera and the audio recorder, there is no need for you to have multiple claps during the recording, a single one will do. However, you will need to insure that when you edit out unwanted portions, you delete both the video and external audio portions at the same time otherwise, your external audio and video will cease to be synced.

Once you have aligned your audio and video; rather than delete the video audio, mute the video's audio track. If the video editing software you are using does not allow you to mute the audio track on your video, then you do have a software problem and you may want to look for a different program. I have been looking at OpenShot.org. It appears to be a full featured open source cross platform video editor which I am currently downloading and have yet to try out.

This should help you refine and simplify your video production.

ST.