I'm trying to process frames of video in FFMPEG; while processing, I'll crop and resize various areas of the frames in separate streams so that I can perform specific processing.

I've done something similar (as outlined on Super User) in FFMPEG, but as I expand the requirements, I need a different approach.

To recap:

I'm doing something like this:

ffmpeg \
    -i <input> \
    -filter_complex \
        split=2[in1][in2];[in1]<some filter>[out1];[in2]<some other filter>[out2]; \
       [out1][out2]interleave,setpts=N/25/TB[out] \
    -map [out] -r 25 -codec png -f image2pipe pipe:1


  • Get an input from a URL (it will never be loaded from disk, always from a URL)
  • Split multiple streams (the example here has two)
  • For each stream, apply a series of filters to crop, scale, and perform other operations
  • Interleave the streams, so that I can tell which stream is which in the output, which is piped (and read from STDOUT in my application) as a series of PNG images

The reason the above works is that the stream's outputs are both the same size.

However, I find myself now wanting to process a third stream from the original input of a different size, which is causing a problem, as FFMPEG doesn't know how to handle the mismatch in frame sizes.

One option I've considered is not interlacing and supplying a black video frame to use as a delimiter between different streams. I can pick up this video frame (it will be distinct from all others) easily within my application.

While I'm sure I can get this to work, the issue is that I would like to be able to process video on-the-fly with this at some point, and the above won't work for that scenario, as we'd have to wait for the first stream to finish, then the second, etc. etc.

I've considered trying trying to stamp the PNGs with a piece of data, but there doesn't seem to be a way of doing that

Note, I don't necessarily have to interlace the streams; as long as I can tell them apart when processing the PNG frames, that's fine.

Is there a way to do this?

  • FFmpeg cannot handle frames of different sizes within the same stream. They have to be separate streams. If your app can read from named pipes, that's the easiest way out. Do mkfifo for each distinct sequence, and write to those. – Gyan Mar 14 at 4:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.