How to extract frames and saving frame type to filenames with ffmpeg?

Question

I have a research project where I need to apply very, very computationally heavy image filters on a h264 video. Due some restrictions outside my control I have to use proprietary filter that works only on images and not video. Filter has two modes, A mode is very computationally heavy but produces production ready quality and B mode is fast but produces less than stellar quality.

My theory is that given how h264 uses I-frames to get the reference and rest of the frames are "non-essential", if I can find apply the high quality A mode on I-frames and B mode on other frames, I should be able to leverage h264's own compression to get the result without having to use filter on A mode to every frame. Another avenue would be to try with h265 because it's more robust frame prediction capabilities.

What I've figured out so far is that I need to find a way to extract all frames from a video while knowing which ones are I-frames, then do a filter on all the frames, and then re-encode the video back together with minimum quality loss.

So first question is that how can I extract frames with ffmpeg so that it somehow identifies each frame type on the filename?

ffmpeg -i C:\test.mp4 -vf select='eq(pict_type\,I)',setpts='N/(25*TB)' C:\testTemp\%09d.jpg

Above gives you i-frames but I haven't figured out how I can get this to export all the frames and to put the type on the filename?

Second question is that how can I re-encode the video so that it keeps the I-frame positions?

ffmpeg -framerate 24 -i frame-%03d.png output.mp4

Above would make the video out of frames but the I-frame positioning eludes me.

I know this is a complex problem so all help is truly appreciated!

Answer 1

Can't comment on the soundness of your theory, but here's a way to do it. Not directly, but in a roundabout way.

FFmpeg's segment muxer can break up a file at GOP boundaries, so running the command below generates a set of videos which each start with a keyframe but contain no other keyframes.

ffmpeg -i test.mp4 -c copy -f segment -segment_time 0.001 -segment_list list.ffcat -reset_timestamps 1 segs%d.mp4

This will create

segs0.mp4
segs1.mp4
...
segs101.mp4
..etc

Extract each segment to images

ffmpeg -i segN.mp4 outN-%d.png

Apply A on first frame of each image series and B on the rest.

Then for each processed series, run

ffmpeg -framerate X -i procN-%d.png -c:v libx264 -keyint_min 1000 -x264opts stitchable "result\segsN.mp4"

-keyint_min value should be greater than no. of frames in each series.

Hopefully the names of the resulting MP4s are the same as the original split segments, which makes the next step more convenient.

Copy the list.ffcat created in the initial segmenting command to the results folder and run

ffmpeg -f concat -safe 0 -i list.ffcat -c copy processed.mp4