I'm developing a video recorder application in JavaScript/Electron. The user can talk into the camera, and show pictures, videos, or play audio files. These assets can be superimposed over the video any time. When recording ends, the presentation is all saved into a WebM file.

The JavaScript Media and Streaming API has a feature called MediaRecorder. As the name implies, this can be used to record the contents of a stream, both video and audio. It has annoying limitations though. Most importantly, you can't change streams while recording. If you attempt to replace the current audio stream with another, recording immediately stops. So I had to come up with a workaround, or the user can only show videos without sound.

MediaRecorder only seems to save into WebM with H.256 encoding and Opus audio. There are a few other codecs, but they all unreliable and often crash.

At this point I can record video from the camera, and there's no problem with superimposing short video clips over the camera image, but I can't record the audio of these files. I can't turn off the mic and switch to a new audio stream, and then back, or the API shuts down the recorder.

The solution I came up with is the following:

  1. Record the presentation.
  2. When the user shows a video clip, mark the timestamp, and record the clip's audio stream with a second MediaRecorder instance. (No, I can't take it from the original file, because the user may pause it, seek into it, stop it prematurely, etc.)
  3. When the user hides the video, save the timestamp again, and save the separate audio recording.
  4. After the user finishes recording the presentation, save the "main video".
  5. Using the saved timestamps, assemble an ffmpeg concatlist. Here's an example:
file video.webm
inpoint 00:00:00.000
outpoint 00:00:16.105
file vhP9u_NDj0nEPS6EAtmCf.webm          <-- yes, that's a random filename
file video.webm
inpoint 00:00:21.222
outpoint 00:00:23.688
file mjV66jcoSUUxTq1GbQItH.webm
file video.webm
inpoint 00:00:27.879
outpoint 00:00:31.142
file Ig55EcInUVC8uduqMye8U.webm
file video.webm
inpoint 00:00:35.863
outpoint 00:00:37.991

Now call ffmpeg to put together a new video. Take the main video and the small audio files recorded on the way, and concatenate them into a new audio track:

ffmpeg -f concat -i concatenate.txt -vn -ac 2 concat_audio.webm

Then attach this new audio track to the main video, and save it as a new one:

ffmpeg -i video.webm -i concat_audio.webm -c copy video_final.webm

Sounds good - doesn't work. Looks like WebM doesn't like to be used like this!

The output video is almost always garbled in some way. Timings are off, often by several seconds.

At most times the concatenated audio is shorter than the actual video. This certainly isn't the fault of my timestamps, they're very accurate. (It's easy to verify if they are, as the inserted videos begin exactly at the saved times.)

I tried to convert the audio clips into .wav and the video into .mp4, .mov or .avi before concatenation. In most cases, the video got messed up. In most such cases it pauses at a certain frame, and that's it.

So, my question is: Is there a better way to do this?

How can I make WebM files more "stable"? I know they're meant for streaming. What format and codec should I convert them into for the concatenation to work?

Can ffmpeg replace parts of the audio stream in a video, without having to chop up and reassemble the entire shebang?

Using a video editing tool to put the video together is out of question. It has to be done by the application, and without requiring immense knowledge of video formats and codecs.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.