combine stream and multiple videos to live output, with transitions

Question

I am using FFmpeg to combine a radio stream with a video but my target is to read from a playlist file (own format) and combine multiple videos with the audio in the background. The most important thing is that I want to grab the output and pipe it to VLC or FFplay (I would prefer VLC because somehow sometimes FFplay displays vertical scanlines with some videos where VLC doesn't).

The playlist reading part and so on is already done, so I just need the FFmpeg part combining everything in a FFmpeg command.

Also, a goal is to get it to work locally on a Raspberry Pi so no transcoding should happen because rpi is busy decoding video.

I have the following playlist:

1. video test1.mkv with audio stream (mpeg2)
2. video test2.mpeg with audio stream (same as above)
3. video with audio from video
4. video with audio from mp3
5. ... and so on
6. if needed loop to playlist item 1

So you can see those videos can have a different formats and sizes and the audio can be from the video or from a stream/mp3.

First of all I am wondering if I should actually start to use the library and not do it with the command line, is this even possible with the command line?

the main problems are:

1. Video transitions do not work because stream is playing (doesnt stop)
2. I cannot get the video to pipe to VLC
3. I cannot get the transitions to work properly
4. Should not transcode to work on a RPI
5. Loop 1 item for x or the whole playlist for x or infinite doesn't work

simple map of video and audio stream: ffmpeg -i "SampleVideo_1280x720_1mb.mp4" -i "https://www.mpeg2stream.com/" -map 0:v:0 -map 1:a:0 -f mpegts udp://127.0.0.1:23000

in this i combine the video with a video, this works nice and i grab the output from the stream (meh, i dont want to transcode)

piping: i tried different pipe methods to get this to work but i cannot get it to work well, especially on the windows testbox.

ffmpeg -i "test.mkv" -f mpegts -codec copy - | vlc -

here i still transcode to transportstreams, but also vlc doesnt display it and somehow only a bar is running back and forth.

I gave up on piping and now only put it on an mpegts stream:

-c:v libx264 -preset superfast -tune zerolatency -c:a aac -ar 44100 -f mpegts udp://127.0.0.1:23000

Transitioning: making the transition i do with the following:

ffmpeg -i "SampleVideo_1280x720_1mb.mp4" -i "test1.mkv" -filter_complex "[0]fade=t=out:st=0:d=1:alpha=1,setpts=PTS-STARTPTS[va0]; [1]fade=t=in:st=0:d=1:alpha=1,setpts=PTS-STARTPTS[va1]; [va0][va1]overlay[outv]; [0][1]amix[outa]" -map [outv] -map "[outa]" -crf 10 out.mp4

here i write in an outfile, this should also be piped, i would understand that this does not work without transcoding. isnt there an easier method? also it does not work without scaling to something. so i probably could just use concat without transitioning. Also here i gave up and use just normal concatinating

Concat and scaling:

ffmpeg -i big_buck_bunny.mp4 -i SampleVideo_1280x720_1mb.mp4 -i "file_example_MP4_1920_18MG.mp4" -i "https://teststream.de/stream/"`
-filter_complex `
"[0:v]scale=1280x720:force_original_aspect_ratio=increase[v0]; `
[1:v]scale=1280x720:force_original_aspect_ratio=increase[v1]; `
[2:v]scale=1280x720:force_original_aspect_ratio=increase[v2]; `
[v0][0:a][v1][1:a][v2][3:a]concat=n=3:v=1:a=1[v][a]" `
-map [v] -map [a] `
-c:v libx264 -preset superfast -tune zerolatency -c:a aac -ar 44100 -f mpegts udp://127.0.0.1:23000

the problem with concatinating shows that the first audio plays correctly (from the video) but it does not concat to video 3 because the stream is obviously not finished, how can i concat the stream for video 3 dynamic length? Also i cannot get it to scale automatically to the best resolution without hardcoding it.

Loop: the next thing is i cannot get loop to work. stream-loop doesnt work or is it possible to do this via complex filtering?

any ideas how to get all of this to work or is this not possible with ffmpeg and i should think of another solution?

Cheers and thanks a lot!

Edit 1: Most of the questions now are obvious (some of them are just because i dont know much about video stuff)

So far i have come to create some of it, but i still have some problems, but 2 of them currently stop me the most: https://github.com/martinlierschof/ffmpeg_test/

Problem 1: audio stream starts from beginning every time i amix it. It should always play from the "live" position.

Problem 2: the [videoout] and [audioout] should be able to be looped (after concating) for x amounts or -1 (infinite)

Answer 1

It sounds like you want to live transition from one video playback to another and mix these videos as you go (like a video DJ) and stream them to some kind of website like Twitch. Just use OBS for this and then load all the video files into two different players and then use OBS filters to transition as you go. You will need at least a quad-core (8 threads) machine to handle all the processor overhead, but if you have a decent video card it can assist with the x264 compression for the outbound stream. It would be the shortest distance to what it sounds like you want to do. Because of the command line system of FFMPEG it is hard to have a fluid and smooth interface in a live stream like what you are looking for.