My use-case, in case the details help people understand what I'm looking for: My brother needs some videos of him singing while playing the piano, to send in as an audition for a dinner theater. For this job, he's already met the musical director, so this isn't a first impression, but obviously we want him to look and especially sound good. I'm not picturing putting in any video effects or titles / credits. (We figure metadata should be meta, like web page title or filename, and stored in the video container metadata, not burned into the video.) So the target for these videos is to be watched on a computer, probably via youtube, not DVD or broadcast.
We used a digital camera to record video, but the audio quality isn't good enough. We recorded audio at the same time with an analog mic hooked up to his laptop (with Audacity), but it needs to be manually synced. More details about the details at the bottom, for anyone that's curious. The camera and laptop clocks don't quite agree on how long a second is: the camera actually records at about 29.981 fps, when it thinks its recording at 30fps (with its audio staying in sync).
So I have 4 videos, 7 to 22 mins long, and 4 audacity projects with better audio for each video (which I can just export to wav or flac), but not synced. Desired output: one file per song, with audio from audacity mangled as little as possible
I need to:
- align the start of each external audio (wav or flac) with the synced but bad-sounding audio that comes in the same file as each vid
- stretch the external-audio or a+v cam clip so they align at the end, too
- discard the audio that came with the video file (or mute it and keep it around for reference so I can go back and fix the alignment if I notice a problem, without starting from scratch.)
- select the time range of the "good take" of each song.
- export these non-overlapping segments of my source videos, one output file per song. (The longer source vids have multiple songs)
- encode each song with ffmpeg / x264 / libfdk-aac and upload to youtube, and/or stick on my website with an HTML5 video tag. Export from the NLE in a lossless format that I can feed to ffmpeg is fine.
I know what I'm doing with ffmpeg / x264 / libfdk-aac / flac / libmp3lame / mp4 container to make final files to put on youtube and/or on my website; I don't need help with that part.
I did the a/v sync part of the problem for the first video, as a proof-of-concept before recording more:
In audacity, I loaded the audio from the camera .MOV, and found I needed to shift the external-mic track left by 5.34563s to align with the camera track. I did that, and cut off the part before zero secs. Then at the end, I found the camera track had to be shifted right 0.43144s to align with the external-mic track at 11m:30s. So I had to make the video 0.431 sec longer to stay synced with the audio. I exported the shifted and clipped external-mic track to a .flac from audacity, after using the noise-reduction filter.
Then I fed that to ffmpeg:
ffmpeg -i session2-shifted.flac -an -i P1000669-sess2.MOV -shortest -c:a libmp3lame -q:a 1 -c:v libx264 -preset fast -crf 18 -filter:v "setpts=690.43144*PTS/690,transpose=2:passthrough=portrait" -r "30 * 690 / 690.43144" -movflags +faststart out.mp4
(I only had ffmpeg's built-in not-very-good AAC encoder in the windows build of ffmpeg I put on my brother's laptop, hence the mp3 audio for a test-run.)
So my output is a 29.981 fps mp4, with A/V sync from start to finish. (29.981 is close to NTSC 30/1.001 ~= 29.976, but that's just a coincidence.) I should be able to re-run that with lossless output (or
-codec copy), and then chop out the sections I want. Either with a NLE, or note a start/stop time, and use
ffmpeg -ss [start] -to [stop] -i session2-synced.mp4 -x264 -blah -blah song1.mp4
I got my brother to clap at the start AND end of each video while the external sound was recording, so we have sync marks that should be visually obvious in the waveform for the last 3 vids. (Aligning the first one was a bit tricky, glad I did one by hand before recording the rest).
I'm looking for recommendations for software to get all that done more quickly / easily than the manual process I used.
I use GNU/Linux, my brother uses Windows. Ideally, there's an NLE that's good for this use-case and is cross-platform, so I can show my brother how to cut segments out of a longer video that he records on his phone or something, with the same software I learn to use. I'm only interested in open source software. I'm not interested in investing time into learning my way around something that isn't open source. (And not going to spend money on it when I can already do what I need with audacity and ffmpeg.) Don't omit a good suggestion just for lack of cross-platform support, though, please.
I know what a NLE is supposed to be able to do, but haven't used one before. (Just ffmpeg / mencoder with start / stop times covered the minimal stuff I've done before.)
I'm aware that usually people stretch and pitch-correct audio to fit video, but I trust the clock in the laptop more than the clock in the camera. And I want to keep audio mangling to a minimum. Although I already need some pretty serious denoising / 60Hz removal on the external-mic audio, so it's hardly pristine super-high-quality. So I'm willing to consider a solution that stretches/pitch-corrects audio instead of making non-standard fps videos.
I already tried pitivi, but the audio waveform display wasn't very visible, so I'd probably still have to do the syncing in audacity. As installed from the Ubuntu 14.04 package, it can't export with x264, and the only lossless codec is Dirac. (no ffvhuff...) Lossless dirac export is unacceptably slow, so that's another big downside. (I love open source / patent-unencumbered codecs like VP9, but I don't want anyone to have any hassles playing the vids, so I'm going with H.264 video and high-bitrate AAC audio. Or does youtube accept VP9 uploads?)
I've used avidemux, too, but IIRC it doesn't do well outputting to modern containers. It def. doesn't like reading codecs with b frames from modern containers, so it's pretty much a dead end. (h.264 in mkv or mp4, anyone?) It can do audio from an external file, but doesn't show waveforms for syncing.
This question is similar to others that have been asked before:
- How can I record audio & video synced with another audio file?
- How to automatically synchronize two recordings of the same event?
- Free software to synchronize audio and video recorded at the same time
- Non-Linear Video Editor for Free or Cheap That Does These Things
- Video desyncs with the audio over time
The first question has one answer that is sort of what I'm looking for: http://auxmic.com/, but that's Windows only, and it doesn't say anything about how it handles clock drift in the input. I might give it a try, since it's prob. simple enough to work fine with WINE in Linux. I'd still need to use something else to cut out the good take of each song from the videos.
None of the other answers have any useful suggestions, maybe because the questions weren't specific enough.
For those curious about what I actually used:
camera was a Panasonic Lumix FZ28 on a tripod, maybe 2m from the subject, high enough to get a good view of the piano keys and his face. It records 1280x720 or 640x480 MJPEG + 16kHz mono PCM in a MOV container, on SD cards. Video quality is pretty good, given enough light. At 720p, it can go for just under 12mins before hitting the 2GB max file size for FAT32, and stopping. (22Mbit/s or so, obviously needs to be xcoded for final use.) The video scaler in the camera is bad, and introduced some ugly banding from aliasing effects or something in the wall hanging behind my brother when we recorded in 640x480 (11Mb/s MJPEG). We used 640x480 anyway to let the camera run longer, since each separate shot takes manual labour to A/V sync. On top of the time to just select a start/end point and export a file.
The mic was just a cheap Labtec computer desk mic we had lying around. It picked up a lot of 60 Hz interference, even when its cord wasn't near the power cables for the lights and digital piano. (Even with the laptop on battery power.) Audacity's noise removal function does a good job of killing the 60Hz and background hiss. Having the mic sitting right on top of the piano picked up a lot of the sound of physically hitting the piano keys. If we'd had time before he was heading back to school, I would have used something for a mic stand to put the mic farther from the piano keys, and not in physical contact with the body of the digital piano.
I made a separate project in Audacity for each video file, since the A/V offset would be different for each one. I could have just made one big audacity project, but I don't think that would have helped with anything, except maybe being able to tweak noise-reduction / highpass (to maybe reduce the noise of piano keys being hit) filter settings and then apply it to everything at once.