0

I am working on a project that involves aligning dubbed audio with corresponding subtitles by manipulating video speeds. While I have been using FFmpeg (both directly and indirectly via MoviePy) for video processing, I’ve encountered a peculiar issue.

When using MoviePy to adjust video speeds and concatenate clips, the final video duration aligns perfectly with the dubbed audio duration. Conversely, when employing FFmpeg directly with a complex filter script to manipulate speed and concatenate video parts, the final video tends to be 1-2 seconds shorter than the dubbed audio, leading to misalignment near the end of the video.

Here’s a simplified pseudocode of the process: (MoviePy that works perfectly, but twice as slow)

total_video_duration_sec = get_video_duration(input_video)
durations = get_durations()

if subtitles[0].start.total_seconds() > 0:
    gap_clip = clip.subclip(0, subtitles[0].start.total_seconds()).without_audio()
    processed_clips.append(gap_clip)

for i, (subtitle, audio_durations_sec) in enumerate(zip(subtitles, durations)):

    start_sec = subtitle.start.total_seconds()
    end_sec = subtitle.end.total_seconds()
    next_start_sec = subtitles[i + 1].start.total_seconds() if i < len(subtitles) - 1 else total_video_duration_sec
    original_duration_sec = end_sec - start_sec

    pause_duration_sec = next_start_sec - end_sec

    # Calculate the speed factor
    speed_factor = original_duration_sec / audio_duration_sec

    # Extract the subtitle clip from the original video
    subtitle_clip = clip.subclip(start_sec, end_sec).without_audio()
    
    # Apply speed effect to the subtitle clip
    adjusted_clip = subtitle_clip.fx(speedx, speed_factor)
    
    # Append the adjusted clip to the list
    processed_clips.append(adjusted_clip)


    if pause_duration_sec > 0:
        gap_clip = clip.subclip(end_sec, next_start_sec).without_audio()
        processed_clips.append(gap_clip)

 
# Concatenate all processed clips back together
final_clip = concatenate_videoclips(processed_clips)

and FFmpeg script that works fast, but leads to a slightly shorter video which the gap increasing as length of subtitles increases.


# Fetch the total video duration and frames per second (fps) from the input video
total_video_duration_sec, fps = get_video_info(input_video)

# Fetch the durations of the audio clips associated with each subtitle
durations = get_durations()

# If the first subtitle doesn't start at the beginning of the video, 
# add a segment from the video start to the first subtitle's start
if subtitles[0].start.total_seconds() > 0:
    initial_pause_video_script = f"[0:v]trim=start=0:end={round(subtitles[0].start.total_seconds(), 3)},setpts=PTS-STARTPTS[v0]"
    filter_script.insert(0, initial_pause_video_script)
    inputs.insert(0, "[v0]")

# Iterate over each subtitle and its associated audio duration
for i, (subtitle, audio_duration_sec) in enumerate(zip(subtitles, durations)):
    
    # Determine the starting time of the next subtitle (or use the total video duration if this is the last subtitle)
    next_start_sec = subtitles[i + 1].start.total_seconds() if i < len(subtitles) - 1 else total_video_duration_sec
    start_sec = subtitle.start.total_seconds()
    end_sec = subtitle.end.total_seconds()
    original_duration_sec = end_sec - start_sec
    pause_duration_sec = next_start_sec - end_sec
    
    # Calculate the speed factor based on the audio duration and original subtitle duration
    speed_factor = audio_duration_sec / original_duration_sec 

    # Create the filter script for the subtitle segment of the video
    filter_script.append(f"[0:v]trim=start={start_sec}:end={end_sec},setpts={speed_factor}*(PTS-STARTPTS)[v{i*2+1}]")
    inputs.append(f"[v{i*2+1}]")
    
    # If there's a gap/pause between this subtitle and the next, create a filter script for that gap
    if pause_duration_sec > 0:
        filter_script.append(f"[0:v]trim=start={end_sec}:end={next_start_sec},setpts=PTS-STARTPTS[v{i*2+2}]")
        inputs.append(f"[v{i*2+2}]")

# Combine all filter scripts and inputs to create the final filter script
filter_chain = ";".join(filter_script)
inputs_chain = "".join(inputs)
final_filter_script = f"{filter_chain};{inputs_chain}concat=n={len(inputs)}:v=1,fps={fps}[outv]"

# Write the final filter script to a file
with open(filter_complex_file, "w") as f:
    f.write(final_filter_script)

# Create and run the ffmpeg command to process the video using the generated filter script
cmd = [
    'ffmpeg', '-y', 
    '-i', input_video,  # First input: video file
    '-filter_complex_script', 'filtergraph.txt', 
    '-map', '[outv]',  # Map video stream from filter output
    '-c:v', 'libx264', '-preset', 'veryfast', '-crf', '18',
    final_output_video  # Output file
]
subprocess.run(cmd)

what cou

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.