6

My use-case, in case the details help people understand what I'm looking for: My brother needs some videos of him singing while playing the piano, to send in as an audition for a dinner theater. For this job, he's already met the musical director, so this isn't a first impression, but obviously we want him to look and especially sound good. I'm not picturing putting in any video effects or titles / credits. (We figure metadata should be meta, like web page title or filename, and stored in the video container metadata, not burned into the video.) So the target for these videos is to be watched on a computer, probably via youtube, not DVD or broadcast.

We used a digital camera to record video, but the audio quality isn't good enough. We recorded audio at the same time with an analog mic hooked up to his laptop (with Audacity), but it needs to be manually synced. More details about the details at the bottom, for anyone that's curious. The camera and laptop clocks don't quite agree on how long a second is: the camera actually records at about 29.981 fps, when it thinks its recording at 30fps (with its audio staying in sync).

So I have 4 videos, 7 to 22 mins long, and 4 audacity projects with better audio for each video (which I can just export to wav or flac), but not synced. Desired output: one file per song, with audio from audacity mangled as little as possible

I need to:

  • align the start of each external audio (wav or flac) with the synced but bad-sounding audio that comes in the same file as each vid
  • stretch the external-audio or a+v cam clip so they align at the end, too
  • discard the audio that came with the video file (or mute it and keep it around for reference so I can go back and fix the alignment if I notice a problem, without starting from scratch.)
  • select the time range of the "good take" of each song.
  • export these non-overlapping segments of my source videos, one output file per song. (The longer source vids have multiple songs)
  • encode each song with ffmpeg / x264 / libfdk-aac and upload to youtube, and/or stick on my website with an HTML5 video tag. Export from the NLE in a lossless format that I can feed to ffmpeg is fine.

I know what I'm doing with ffmpeg / x264 / libfdk-aac / flac / libmp3lame / mp4 container to make final files to put on youtube and/or on my website; I don't need help with that part.

I did the a/v sync part of the problem for the first video, as a proof-of-concept before recording more:

In audacity, I loaded the audio from the camera .MOV, and found I needed to shift the external-mic track left by 5.34563s to align with the camera track. I did that, and cut off the part before zero secs. Then at the end, I found the camera track had to be shifted right 0.43144s to align with the external-mic track at 11m:30s. So I had to make the video 0.431 sec longer to stay synced with the audio. I exported the shifted and clipped external-mic track to a .flac from audacity, after using the noise-reduction filter.

Then I fed that to ffmpeg: ffmpeg -i session2-shifted.flac -an -i P1000669-sess2.MOV -shortest -c:a libmp3lame -q:a 1 -c:v libx264 -preset fast -crf 18 -filter:v "setpts=690.43144*PTS/690,transpose=2:passthrough=portrait" -r "30 * 690 / 690.43144" -movflags +faststart out.mp4

(I only had ffmpeg's built-in not-very-good AAC encoder in the windows build of ffmpeg I put on my brother's laptop, hence the mp3 audio for a test-run.)

So my output is a 29.981 fps mp4, with A/V sync from start to finish. (29.981 is close to NTSC 30/1.001 ~= 29.976, but that's just a coincidence.) I should be able to re-run that with lossless output (or -codec copy), and then chop out the sections I want. Either with a NLE, or note a start/stop time, and use ffmpeg -ss [start] -to [stop] -i session2-synced.mp4 -x264 -blah -blah song1.mp4

I got my brother to clap at the start AND end of each video while the external sound was recording, so we have sync marks that should be visually obvious in the waveform for the last 3 vids. (Aligning the first one was a bit tricky, glad I did one by hand before recording the rest).

I'm looking for recommendations for software to get all that done more quickly / easily than the manual process I used.

I use GNU/Linux, my brother uses Windows. Ideally, there's an NLE that's good for this use-case and is cross-platform, so I can show my brother how to cut segments out of a longer video that he records on his phone or something, with the same software I learn to use. I'm only interested in open source software. I'm not interested in investing time into learning my way around something that isn't open source. (And not going to spend money on it when I can already do what I need with audacity and ffmpeg.) Don't omit a good suggestion just for lack of cross-platform support, though, please.

There are many options on GNU/Linux and it's even been asked on Ask Ubuntu.

I know what a NLE is supposed to be able to do, but haven't used one before. (Just ffmpeg / mencoder with start / stop times covered the minimal stuff I've done before.)

I'm aware that usually people stretch and pitch-correct audio to fit video, but I trust the clock in the laptop more than the clock in the camera. And I want to keep audio mangling to a minimum. Although I already need some pretty serious denoising / 60Hz removal on the external-mic audio, so it's hardly pristine super-high-quality. So I'm willing to consider a solution that stretches/pitch-corrects audio instead of making non-standard fps videos.

I already tried pitivi, but the audio waveform display wasn't very visible, so I'd probably still have to do the syncing in audacity. As installed from the Ubuntu 14.04 package, it can't export with x264, and the only lossless codec is Dirac. (no ffvhuff...) Lossless dirac export is unacceptably slow, so that's another big downside. (I love open source / patent-unencumbered codecs like VP9, but I don't want anyone to have any hassles playing the vids, so I'm going with H.264 video and high-bitrate AAC audio. Or does youtube accept VP9 uploads?)

I've used avidemux, too, but IIRC it doesn't do well outputting to modern containers. It def. doesn't like reading codecs with b frames from modern containers, so it's pretty much a dead end. (h.264 in mkv or mp4, anyone?) It can do audio from an external file, but doesn't show waveforms for syncing.

This question is similar to others that have been asked before:

The first question has one answer that is sort of what I'm looking for: http://auxmic.com/, but that's Windows only, and it doesn't say anything about how it handles clock drift in the input. I might give it a try, since it's prob. simple enough to work fine with WINE in Linux. I'd still need to use something else to cut out the good take of each song from the videos.

None of the other answers have any useful suggestions, maybe because the questions weren't specific enough.

For those curious about what I actually used:

camera was a Panasonic Lumix FZ28 on a tripod, maybe 2m from the subject, high enough to get a good view of the piano keys and his face. It records 1280x720 or 640x480 MJPEG + 16kHz mono PCM in a MOV container, on SD cards. Video quality is pretty good, given enough light. At 720p, it can go for just under 12mins before hitting the 2GB max file size for FAT32, and stopping. (22Mbit/s or so, obviously needs to be xcoded for final use.) The video scaler in the camera is bad, and introduced some ugly banding from aliasing effects or something in the wall hanging behind my brother when we recorded in 640x480 (11Mb/s MJPEG). We used 640x480 anyway to let the camera run longer, since each separate shot takes manual labour to A/V sync. On top of the time to just select a start/end point and export a file.

The mic was just a cheap Labtec computer desk mic we had lying around. It picked up a lot of 60 Hz interference, even when its cord wasn't near the power cables for the lights and digital piano. (Even with the laptop on battery power.) Audacity's noise removal function does a good job of killing the 60Hz and background hiss. Having the mic sitting right on top of the piano picked up a lot of the sound of physically hitting the piano keys. If we'd had time before he was heading back to school, I would have used something for a mic stand to put the mic farther from the piano keys, and not in physical contact with the body of the digital piano.

I made a separate project in Audacity for each video file, since the A/V offset would be different for each one. I could have just made one big audacity project, but I don't think that would have helped with anything, except maybe being able to tweak noise-reduction / highpass (to maybe reduce the noise of piano keys being hit) filter settings and then apply it to everything at once.

  • BTW, is "clock drift" the right term for the de-sync over time due to sources recorded with clocks of different speeds? I just made up that terminology, but I figure there's probably an established term. – Peter Cordes Jan 13 '15 at 6:09
  • I asked this question on softwarerecs and the answer was also Kdenlive. – sebix Jan 24 '15 at 21:25
  • @sebix, Thanks, but as I said, kdenlive doesn't handle tracks so long that one needs stretching to match the other, as well as alignment at one point, to stay synced over the whole length. That's why I didn't mark my own answer as accepted. :/ – Peter Cordes Jan 24 '15 at 21:53
  • That's also the restriction I am struggling with. Maybe I'm sometime so annoyed 'bout that, that I start to improve the situation... – sebix Jan 24 '15 at 21:57
4

I tried kdenlive, so I'll post my findings about it as an answer. It didn't quite do the job, so I'm not going to mark this as the accepted solution.

kdenlive easily imports my clips in mjpeg+pcm, and flac. And looks like it can export through ffmpeg, which is what I want.

It has a feature to "set audio reference", and for other tracks, "align audio to reference". That's perfect, programmatically aligning audio is much nicer than manually dragging stuff to align waveforms by eye. However, it doesn't seem to handle the clock drift at all. In my 23min shot, it had perfect sync at about 4 mins. It was a touch out of sync at the start, and maybe 0.5s out of sync at the 19 minute mark. (I don't have a good sync point later than that, since that shot hit the 2GB file size limit, xD)

KDENlive's timeline display is in h:mm:ss.frames, not decimal fractions of a second, which confused me at first. I was thinking that it had stretched one track, but no, it didn't.

It's easy to tell there's a problem by leaving both audio tracks unmuted, even though that will reveal problems too small to be noticed visually. (In my longer tracks, like this one, the drift eventually gets big enough to be a problem for visual AV sync.)

I checked manually in audacity: With the start of both tracks aligned, they're about 0.68s apart at the same 19min sync mark. So I could do my aligning in kdenlive manually, but its automatic tool for doing it unfortunately doesn't support clock-drift correction by stretching.

kdenlive does display the audio waveforms nicely, esp. if you go configure->timeline->track height: increase to 80 or 100pix. Nice UI for zooming way in on the timeline.

If I end up doing the AV sync in audacity before exporting, I might well use kdenlive for the rest of the process. I'll have to see how easy it is to make multiple videos from segments of a single timeline.

2

Try Blender's Video Editing.

If Video and audio where recorded at the same time it will be easy to sync them, just be sure your video frame rate is correctly set in Blender and that you check synced in the video editor (at the bottom beside the time line and play, stop, etc... buttons.

Don't be scared by Blender's complexity, you are not going to use any of the others features, you are just going to use video editing and that is pretty straight forward.

  • Thanks, I'll give it a try if you say it does what I'm looking for. I did blender some 3D text like 15 years ago, I remember thinking the UI was neat. :) – Peter Cordes Jan 13 '15 at 6:07
  • Ok, I loaded my vid an external audio in blender, but I didn't see anything yet about automatically aligning them based on audio similarity. Or stretching one track to align with the other given two sync points. This answer doesn't answer any of the specifics of my question, especially not the part about dealing with un-synced clocks. (i.e. the part that makes it non-trivial to sync, even though stuff was recorded at the same time...) Maybe blender can do stuff, I'll keep poking. But this answer didn't point me anywhere useful. – Peter Cordes Jan 27 '15 at 20:46
  • You don't need to stretch anything if you set the right framerate for the video you are going to load. There is no magic, you obviously have to know aproximatle the moment where the audio matches the image, from there you can fine tune the exact frame with the exact audio time. Once you have a single frame synced the whole video is synced (assuming you have set the right video framerate and pushed the sync-av button) – YoMismo Jan 28 '15 at 7:39
  • Reread my question. My two recording devices don't agree on how long a second is (because one is a Lumix camera, not even DSLR). That's the whole point of the question. If I have to use something else to work out the A/V sync offset and stretch, I might as well just plug that into an ffmpeg commandline. (blender could stretch the video by setting a custom FPS, though.) – Peter Cordes Jan 29 '15 at 1:08
  • Also, blender's export-to-h.264 with x264 is very inflexible, so I'd have to export to lossless and transcode. The only option is ABR, not CRF, and there doesn't even seem to be an x264 CPU-usage preset choice (from ultrafast to medium to veryslow). – Peter Cordes Jan 29 '15 at 1:10
1

The following response is the methodology of how to sync an external audio recording to your video. This method assumes that both your audio and video recordings are being recorded at the same time rate.

Prior to recording each scene, you need to make a loud clap sound that both the audio input of your video camera, and the external audio recorder can pick up at the same time. Typically, a clapper or hand claps are used for this purpose. A clapper is a framed chalkboard or whiteboard with scene information that is recorded by the video camera, and has an arm that is raised and clapped down to make the needed sound. The clap, when recorded, creates a spike in the audio of both the video camera's audio and the external sound recorder audio. These spikes are illustrated in the sounds waveform view with larger spikes if the sound is loud enough. When you align these two waveform spikes in your time line, the video and external sound are aligned and synced perfectly.

If you record your audio and video in one setting without breaks and without stopping the camera and the audio recorder, there is no need for you to have multiple claps during the recording, a single one will do. However, you will need to insure that when you edit out unwanted portions, you delete both the video and external audio portions at the same time otherwise, your external audio and video will cease to be synced.

Once you have aligned your audio and video; rather than delete the video audio, mute the video's audio track. If the video editing software you are using does not allow you to mute the audio track on your video, then you do have a software problem and you may want to look for a different program. I have been looking at OpenShot.org. It appears to be a full featured open source cross platform video editor which I am currently downloading and have yet to try out.

This should help you refine and simplify your video production.

ST.

  • When you align these two waveform spikes in your time line, the video and external sound are aligned and synced perfectly. Right, that's the manual process I described. The question was about what software gives you a UI that can do what you describe. (And (preferably in my case) can scale the video frame rate instead of stretching the audio with pitch-correction.) IIRC, I ended up using ffmpeg with an offset and custom frame-rate. This was a one-off for me, helping my brother make an audition video, so I didn't keep looking for other software. – Peter Cordes Dec 19 '16 at 20:02
  • You can try the following programs. Serif MoviePlus starter Edition - freeserifsoftware.com Screen shots for OpenShot at OpenShot.org also appear to have this feature. I am currently downloading it to review. Scott – Scott Tovey Dec 19 '16 at 20:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.