Discrepancy in video duration when adjusting speeds using FFmpeg filters

Question

I am working on a project that involves aligning dubbed audio with corresponding subtitles by manipulating video speeds. While I have been using FFmpeg (both directly and indirectly via MoviePy) for video processing, I’ve encountered a peculiar issue.

When using MoviePy to adjust video speeds and concatenate clips, the final video duration aligns perfectly with the dubbed audio duration. Conversely, when employing FFmpeg directly with a complex filter script to manipulate speed and concatenate video parts, the final video tends to be 1-2 seconds shorter than the dubbed audio, leading to misalignment near the end of the video.

Here’s a simplified pseudocode of the process: (MoviePy that works perfectly, but twice as slow)

total_video_duration_sec = get_video_duration(input_video)
durations = get_durations()

if subtitles[0].start.total_seconds() > 0:
    gap_clip = clip.subclip(0, subtitles[0].start.total_seconds()).without_audio()
    processed_clips.append(gap_clip)

for i, (subtitle, audio_durations_sec) in enumerate(zip(subtitles, durations)):

    start_sec = subtitle.start.total_seconds()
    end_sec = subtitle.end.total_seconds()
    next_start_sec = subtitles[i + 1].start.total_seconds() if i < len(subtitles) - 1 else total_video_duration_sec
    original_duration_sec = end_sec - start_sec

    pause_duration_sec = next_start_sec - end_sec

    # Calculate the speed factor
    speed_factor = original_duration_sec / audio_duration_sec

    # Extract the subtitle clip from the original video
    subtitle_clip = clip.subclip(start_sec, end_sec).without_audio()
    
    # Apply speed effect to the subtitle clip
    adjusted_clip = subtitle_clip.fx(speedx, speed_factor)
    
    # Append the adjusted clip to the list
    processed_clips.append(adjusted_clip)


    if pause_duration_sec > 0:
        gap_clip = clip.subclip(end_sec, next_start_sec).without_audio()
        processed_clips.append(gap_clip)

 
# Concatenate all processed clips back together
final_clip = concatenate_videoclips(processed_clips)

and FFmpeg script that works fast, but leads to a slightly shorter video which the gap increasing as length of subtitles increases.

# Fetch the total video duration and frames per second (fps) from the input video
total_video_duration_sec, fps = get_video_info(input_video)

# Fetch the durations of the audio clips associated with each subtitle
durations = get_durations()

# If the first subtitle doesn't start at the beginning of the video, 
# add a segment from the video start to the first subtitle's start
if subtitles[0].start.total_seconds() > 0:
    initial_pause_video_script = f"[0:v]trim=start=0:end={round(subtitles[0].start.total_seconds(), 3)},setpts=PTS-STARTPTS[v0]"
    filter_script.insert(0, initial_pause_video_script)
    inputs.insert(0, "[v0]")

# Iterate over each subtitle and its associated audio duration
for i, (subtitle, audio_duration_sec) in enumerate(zip(subtitles, durations)):
    
    # Determine the starting time of the next subtitle (or use the total video duration if this is the last subtitle)
    next_start_sec = subtitles[i + 1].start.total_seconds() if i < len(subtitles) - 1 else total_video_duration_sec
    start_sec = subtitle.start.total_seconds()
    end_sec = subtitle.end.total_seconds()
    original_duration_sec = end_sec - start_sec
    pause_duration_sec = next_start_sec - end_sec
    
    # Calculate the speed factor based on the audio duration and original subtitle duration
    speed_factor = audio_duration_sec / original_duration_sec 

    # Create the filter script for the subtitle segment of the video
    filter_script.append(f"[0:v]trim=start={start_sec}:end={end_sec},setpts={speed_factor}*(PTS-STARTPTS)[v{i*2+1}]")
    inputs.append(f"[v{i*2+1}]")
    
    # If there's a gap/pause between this subtitle and the next, create a filter script for that gap
    if pause_duration_sec > 0:
        filter_script.append(f"[0:v]trim=start={end_sec}:end={next_start_sec},setpts=PTS-STARTPTS[v{i*2+2}]")
        inputs.append(f"[v{i*2+2}]")

# Combine all filter scripts and inputs to create the final filter script
filter_chain = ";".join(filter_script)
inputs_chain = "".join(inputs)
final_filter_script = f"{filter_chain};{inputs_chain}concat=n={len(inputs)}:v=1,fps={fps}[outv]"

# Write the final filter script to a file
with open(filter_complex_file, "w") as f:
    f.write(final_filter_script)

# Create and run the ffmpeg command to process the video using the generated filter script
cmd = [
    'ffmpeg', '-y', 
    '-i', input_video,  # First input: video file
    '-filter_complex_script', 'filtergraph.txt', 
    '-map', '[outv]',  # Map video stream from filter output
    '-c:v', 'libx264', '-preset', 'veryfast', '-crf', '18',
    final_output_video  # Output file
]
subprocess.run(cmd)

what cou