cuvid - slow performance when used 4 devices

Question

Has 4 nvidia devices:

nvidia-smi topo --matrix
        GPU0    GPU1    GPU2    GPU3    CPU Affinity
GPU0     X      PHB     SYS     SYS     0-5,12-17
GPU1    PHB      X      SYS     SYS     0-5,12-17
GPU2    SYS     SYS      X      PHB     6-11,18-23
GPU3    SYS     SYS     PHB      X      6-11,18-23

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe switches (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing a single PCIe switch
  NV#  = Connection traversing a bonded set of # NVLinks

We want to re-code h264 videos to lower resolution. When we run 2-3 processes per 0 and 1 devices:

ffmpeg -v error -y -hwaccel cuvid -hwaccel_device 0 -c:v:0 h264_cuvid -gpu 0 -i /mnt/worker/6797/be2c96e8f0fd7/objects/133607/fb3/8279fb61a4d0bf66c3f871bdb30e5fb3/1530758904.34570.data -map_metadata -1 -map_chapters -1 -map 0:v:0 -map 0:a:0 -c:v:0 h264_nvenc -gpu 0 -b:v:0 650000 -filter:v:0 scale_npp=-2:360 -c:a:0 aac -gpu 0 -b:a:0 96000 -ac:a:0 2 -max_muxing_queue_size 9999 -movflags +faststart -f mp4 /home/uploads/tmp/out.mp4

all works fine

nvidia-smi dmon -c 1
# gpu   pwr gtemp mtemp    sm   mem   enc   dec  mclk  pclk
# Idx     W     C     C     %     %     %     %   MHz   MHz
    0   102    48     -    20     9    14   100  3304  1189
    1   105    54     -    25     9    17    99  3304  1201
    2    15    27     -     0     0     0     0   405   135
    3    15    27     -     0     0     0     0   405   135

but when works 2-3 processes per each device, then performance gets low. Summary compared to 2 devices:

nvidia-smi dmon
# gpu   pwr gtemp mtemp    sm   mem   enc   dec  mclk  pclk
# Idx     W     C     C     %     %     %     %   MHz   MHz
    0    89    45     -     8     3     5    37  3304  1189
    1    92    50     -     9     3     6    30  3304  1201
    2    76    42     -     9     3     6    29  3304  1201
    3    87    44     -     9     3     6    33  3304  1189

htop,top,iftop,atop show nothing overloaded

Sometimes performance rises higher, or drops below (i`m suspect it depending on input files)

How to I can utilize all decoders ?

Answer 1

NVidia boards has a limit number of concurrent encoding tasks, according number of encode engine chips on board. Look at specifications of boards you are using.

Even if you have a board with unrestricted number of sessions to it, you will have a speed degradation, because of some task pipelining on host side.

Answer 2

Problem was resolved by replace

scale_npp=-2:360

to

scale_npp=-2:360,hwdownload,format=nv12

now decoders fully loaded:

nvidia-smi dmon -c 1
# gpu   pwr gtemp mtemp    sm   mem   enc   dec  mclk  pclk
# Idx     W     C     C     %     %     %     %   MHz   MHz
    0   101    50     -    27     8    15   100  3304  1189
    1   105    54     -    25     8    12   100  3304  1201
    2   104    52     -    48    10    19   100  3304  1316
    3    97    50     -    23     9     8   100  3304  1189