Improve performance on 4 x 4K mosaic with FFmpeg

Question

I'm using a NVIDIA Quadro P4000 and my config is Intel i7 2.4 GHz with 8 Go of RAM. I'm working on Ubuntu 16.04, I've installed NVIDIA drivers using apt repo aswell as CUDA. I'm using FFmpeg in version 3.3.

I've managed to display a mosaic of 4 x 4K videos (same videos for test purpose) with FFmpeg, using hardware acceleration (previous post), using this command line :

ffmpeg \
-hwaccel cuvid -c:v h264_cuvid -i bbb_sunflower_2160p_30fps_normal.mp4 \
-hwaccel cuvid -c:v h264_cuvid -i bbb_sunflower_2160p_30fps_normal.mp4 \
-hwaccel cuvid -c:v h264_cuvid -i bbb_sunflower_2160p_30fps_normal.mp4 \
-hwaccel cuvid -c:v h264_cuvid -i bbb_sunflower_2160p_30fps_normal.mp4 \
-filter_complex "nullsrc=size=2560x1440 [base]; \
[0:v] scale_npp=1280:720, hwdownload, format=nv12 [upperleft]; \
[1:v] scale_npp=1280:720, hwdownload, format=nv12 [upperright]; \
[2:v] scale_npp=1280:720, hwdownload, format=nv12 [lowerleft]; \
[3:v] scale_npp=1280:720, hwdownload, format=nv12 [lowerright]; \
[base][upperleft] overlay=shortest=1:eval=init [tmp1]; \
[tmp1][upperright] overlay=shortest=1:eval=init:x=1280 [tmp2]; \
[tmp2][lowerleft] overlay=shortest=1:eval=init:y=720 [tmp3]; \
[tmp3][lowerright] overlay=shortest=1:eval=init:x=1280:y=720, hwupload_cuda" \
-c:v h264_nvenc -preset llhp -profile:v baseline -f matroska pipe:1 | ffplay -i -

Now my issue is about performance, videos are played at 80% of their original speed, so close to my goal...

As I understand from now, this manipulation has several steps :

1. The four videos are decoded by the GPU.
2. Scaling to 720p is also performed by the GPU.
3. The four flux are downloaded in the CPU and formated in NV12.
4. Mosaic is made by the CPU using overlay filter.
5. Mosaic video flux is uploaded in the GPU.
6. The GPU encodes it in H.264.

After this I'm not sure if the encoded video is retrieved by the CPU and then piped to ffplay or if it's stays in the GPU and directly displayed on the monitor.

Moreover, I'd like ffplay to use hardware acceleration but I didn't manage to force it to use h264_cuvid.

Answer 1

The problem was about the overlay filter which is too slow. I've changed for a combination of hstack and vstack as you can see here :

ffmpeg \
-hwaccel cuvid -c:v h264_cuvid -i bbb_sunflower_2160p_30fps_normal.mp4 \
-hwaccel cuvid -c:v h264_cuvid -i bbb_sunflower_2160p_30fps_normal.mp4 \
-hwaccel cuvid -c:v h264_cuvid -i bbb_sunflower_2160p_30fps_normal.mp4 \
-hwaccel cuvid -c:v h264_cuvid -i bbb_sunflower_2160p_30fps_normal.mp4 \
-filter_complex " \
[0:v] scale_npp=1280:720, hwdownload, format=nv12 [upperleft]; \
[1:v] scale_npp=1280:720, hwdownload, format=nv12 [upperright]; \
[2:v] scale_npp=1280:720, hwdownload, format=nv12 [lowerleft]; \
[3:v] scale_npp=1280:720, hwdownload, format=nv12 [lowerright]; \
[upperleft][upperright] hstack [upperline]; \
[lowerleft][lowerright] hstack [lowerline]; \
[upperline][lowerline] vstack [mosaic]; \
[mosaic] hwupload_cuda" \
-c:v h264_nvenc -preset llhp -profile:v baseline -f matroska pipe:1 | ffplay -i -

It works very well and I even can build a 1080p mosaic using lossless preset and high444p profile.