0

Does webm/mka/mkv/mp4 support discontinuous audio streams?

I'm wondering, can ffmpeg transcode .wav-audio into opus-encapsulated-into-webm, but filter out silent frames - especially full seconds of silence, completely drop them from the stream to save space? (maybe via https://ffmpeg.org/ffmpeg-bitstream-filters.html ?)

The goal of all this is to very efficiently represent near-silence in phone-call-recording files for archival purposes (i.e. by completely omitting it from the stream and having it replaced by digital zeros at audio decoding time).

As far as I understand opus-encapsulated-into-ogg does not support such timestamp discontinuities. opus-encapsulated-into-rtp assumingly does support this, but I'm wondering if more "offline" formats such as webm and whether it's possible to produce them directly without passing by rtp format first.

These frames could then be replaced by zeros at audio decoding time by some form of PLC or DTS gaps replacement by zero signal.

Thanks!


UPD1: Unfortunately, silenceremove filter does not work as I need it. The way I used it below, the audio duration after silenceremove is shortened by the amount of silence. This is not what I am looking for. I am looking to preserve the timestamps, have the silence imputed by digital silence at decoding time and preserve the original audio duration (as reported by ffprobe or audio players)

ffmpeg -y -f lavfi -i sine=f=440:d=5 -f lavfi -i anullsrc=d=3600:cl=mono -f lavfi -i sine=f=880:d=5 -filter_complex "[0][1][2]concat=n=3:v=0:a=1" 5sec_silent1h_5sec.wav

ffmpeg -y -i 5sec_silent1h_5sec.wav -acodec libopus 5sec_silent1h_5sec.opus

ffmpeg -y -i 5sec_silent1h_5sec.wav -acodec libopus -af silenceremove=window=0:detection=peak:stop_mode=all:start_mode=all:stop_periods=-1:stop_threshold=0 5sec_5sec.mkv

The good duration of 5sec_silent1h_5sec.opus and bad duration of 5sec_5sec.mkv are confirmed by ffprobe (1h10s and 10s respectively).

Is there a way to adjust the filters/command line arguments to preserve the audio duration?


UPD2 The following somewhat works, although ~1s of 880hz audio leaks into the first segment (before the long silence segment), no matter how I adjust the between bounds - very strange!

ffmpeg -y -i 5sec_silent1h_5sec.wav -acodec libopus -af "aselect='between(t,0,4)+between(t,3606,3610)'" 5sec_silent1h_5sec_with_leakage.mkv

If this leakage can be fixed, then this can be used together with silencedetect filter.

But of course it would be nicer to get the output in a single invokation of ffmpeg.

The example output file with "leakage": https://1drv.ms/u/s!Apx8USiTtrYmq9Q4QaETF2OJWdKGQA?e=ACTmMg


UPD3: On a recent ffmpeg 6.1 adding timestamp=copy filter option also produces a file with a correct duration, but with leakage ffmpeg -y -i 5sec_silent1h_5sec.wav -acodec libopus -af silenceremove=window=0:detection=peak:stop_mode=all:start_mode=all:stop_periods=-1:stop_threshold=0:timestamp=copy 5sec_1hsilence_5sec_with_leakage.mkv. So it seems that timestamps are still not copied exactly.

1 Answer 1

0

You can have timestamp gaps in an audio stream in Matroska-family or MOV/MP4 format files. That's how Google records and stores WebRTC recordings. There won't be zero signal audio data, rather there will be no packets corresponding to the gap and timestamps of the packets before and after the gap will indicate audio position.

However, it is not the encoder's job or feature in ffmpeg to drop those sections. You will need to use the silenceremove filter with option timestamp=copy (6.1+ or git master > May 2023 needed).

12
  • Would silenceremove produce such output with gaps? Or would it "collapse" the gaps and make the wall-clock-duration shorter? This was my misunderstanding while originally reading its documentation which led my to asking this question. Dec 9, 2023 at 9:52
  • 1
    silenceremove can only remove data; it doesn't consolidate gaps.
    – Gyan
    Dec 9, 2023 at 10:45
  • silenceremove does not affect any timestamps and they stay original (as if no gaps were introduced), right? My goal is to have the encoded file have the same duration as before encoding (and have silence imputed by zeros at decoding time). Do you know if gaps are supported by webm/mka? Am I correct that ogg does not support gaps? Would you have an advice of how to instruct ffmpeg to encode separately .wav streams / in a decoupled mode (for phone call recordings, there are long periods of silence if one considers every channel separately)? Dec 9, 2023 at 10:58
  • 1
    Updated answer. See at the end.
    – Gyan
    Dec 14, 2023 at 16:14
  • 1
    Probably to do with the attack/release used by the filter. Will have to debug when I have time.
    – Gyan
    Dec 15, 2023 at 5:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.