restart ffmpeg job from failure

Question

Is there a common technique or some sort of strategy for restarting a failed transcoding job from where it had left off when it crashed (or stopped for whatever reason)?

For example, if I have a 2 hour transcoding job and it fails/crashes at 1 hour and 59 minutes, is there some way I can just do that last minutes worth of work and use the progress I already had assuming the partially transcoded file is still available?

Answer 1

A direct option to resume encoding is not available.

And as for manually restarting encoding, that's also not possible. Two reasons:

1) when transcoding to a typical container like MP4, the index is written after encoding is completed. There will be no such index in a partial file if the process died unexpectedly, and so the existing file cannot be parsed.

2) If you cannot accurately identify the last uncorrupted frame encoded in the partial file, your join will not be seamless.

---

That said, there is a tedious way this can be done. The method hinges on using the segment muxer. You encode to MPEG-TS format but segment the output as it is being generated. Should the job fail, you examine a few attributes and resume encoding based on that data. Finally, you join all segments together.

Step 1: Encode and segment

For best results, audio and video should be done separately. In fact the audio can be done without segmenting since it will encode much faster

ffmpeg -i in.mp4 -c:a aac -vn job-audio.mp4

And for video

ffmpeg -copyts -start_at_zero -ss 0 -i in.mp4 -c:v libx264 -an
       -f segment -segment_time 0.01 -segment_time_delta 1
       -muxpreload 0 -muxdelay 0 -mpegts_copyts 1
       -segment_start_number 0 -segment_list files.ffcat
job-video%d.ts

Each segment above will contain one GOP, so no frames in a segment rely on frames in other segments for playback.

Step 2 Identify resume point

Suppose the job fails while creating segment #5.

First, you run

ffprobe "in.mp4" -show_entries stream=codec_type,start_time,duration -of compact

This will produce a readout like this

stream|codec_type=audio|start_time=0.000000|duration=356.309333
stream|codec_type=video|start_time=0.040000|duration=356.360000

Note the start time for the video stream.

Now, run it for the first and the last (incomplete) segment

ffprobe "job-video0.ts" -show_entries stream=codec_type,start_time,duration -of compact

stream|codec_type=video|start_time=0.080000|duration=10.000000

You see that there's a positive offset of 0.04 w.r.t the MP4 video stream. Let's call this the initial offset value

Now, for the last stream

ffprobe "job-video5.ts" -show_entries stream=codec_type,start_time,duration -of compact

stream|codec_type=video|start_time=20.080000|duration=4.320000

Step 3 Resume encoding

So, now you run

ffmpeg -copyts -start_at_zero -ss {start_time of last segment - initial offset} -i in.mp4 -c:v libx264 -an
       -f segment -segment_time 0.01 -segment_time_delta 1
       -muxpreload 0 -muxdelay 0 -mpegts_copyts 1 -initial_offset {initial offset}
       -segment_start_number {index number of last segment} -segment_list files-resume.ffcat
job-video%d.ts

Step 4 Join segments

You'll have multiple .ffcat files - from the first run and all resumed runs. Copy over the entries from all secondary runs and append them to the first file. Avoid duplication.

Run

ffmpeg -f concat -i files.ffcat -i job-audio.mp4 -c copy full.mp4

Answer 2

If you know the specific portion of the file that the encoding job chokes on, you can use ffmpeg's --ss and -t options to pass over the choke point. You can read more about these options by looking at the tool's documentation.