How to automate sync of 80-120 audio clips recorded at home on mobile phones?

Question

I want to produce a virtual choir recording of an amateur choir. I want to give 80-120 members of the choir an accompaniment/click track of some kind. They will record themselves, on their mobile phones, singing along to the accompaniment track. The accompaniment track will play on a second device (say, a laptop), via headphones or ear buds. The send their recordings to me, and I merge them.

How can I synchronise the resulting 80-120 amateur-captured recordings, in an automated way that scales to 120 recordings, with a low budget? What sync aid do I put in the accompaniment track? What do the members need to do as they record themselves? What do I need to do in post-production to accomplish the sync and merge the audio?

I am quite comfortable scripting a tool like ffmpeg or Audacity to align and merge the recordings. I can even write a Python program to read through the samples of an audio recording to detect the audio peak of a clapper, or DTMF tones. What I want to avoid is any task that I have to do manually for each of the contributed recordings. I fear that I won't be able to get through manual tasks 120 times.

I suspect that a challenge for this synchronisation task is that the contributed recordings will be done with a variety of devices, at a variety of resolutions and sample rates. The choir members need to record themselves. They are not sound engineers. They will make some mistakes in production. I am looking for a sync method which is robust in the face of this variation.

I want to focus on audio merge at first. I expect that the contributed recordings will have video as well. Composing all those videos into a grid, maybe with attractive panning or video effects, is nice to have but not essential.

Let's assume for the sake of argument that the contributed videos will be 3-10 minutes long. It would be nice to stretch to 45 minutes, eventually, but not at first.

I don't know how precise the sync needs to be. As a guess, I imagine a variability of at most 0.1 seconds might be enough. 1 second probably is not good enough. 0.001 seconds is probably more precise than we need.

Answer 1

At the start of the accompaniment track, about 15 seconds before the music starts, record a clap, or other loud but brief sound. Have your choir members record this clap sound from the accompaniment track onto their contributed recording, but not record the rest of the accompaniment.

You can do this by having the singers put on their headphones, but unplug the jack from the playback computer. They start recording on their phone, then start playing back the accompaniment track. The sound of the clap will be played out loud, and captured on their recording, because the headphones are unplugged. Immediate after the clap, the singer plugs in their headphones. This stops any more of the accompaniment from being audible on the recording. (Important detail: experiment with how much time you need to leave between the clap and the start of the accompaniment, so that singers can plug in their headphones and get ready to sing.)

When the singers finish the piece, they stop recording, and send you their recordings. You need some system for naming the audio files, and tracking the identity and vocal part of each singer. (The details of this tracking are not germane to the sync question.)

Make a copy of the first few seconds of each contributed recording in .WAV format. Write a Python program to detect when the clap occurs relative to the start of a recording. Do this by reading the value of each audio sample of the .WAV file, and identifying the sample with the largest value. This is the moment of loudest sound. Convert the position of this largest value into an offset of seconds from the start of that recording.

Now, merge the recordings, applying an offset to each so that the clap occurs at the same moment for all all recordings. The recordings should now be in sync. Mix the recordings to taste, and save the mixed recording.

[Note: this is a speculative answer, based on reading Virtual Choir by Jed Scott, and CB Multitrack Editing 101 by Christopher Bill. Most of the virtual choir instructions I found on the web did not ask the contributors to record any sync moment at all. I guess they just have a human align all the parts by ear.]