FFMPEG hardware accelerated 10 bit downsampling to H264

Question

I have a home setup where I use a Kubuntu 22 box as my home theater PC (HTPC) to play videos/movies/music in my living room. I also set up a raspberry Pi in the kitchen that receives a video stream (screen grab of the living room TV) and plays it on a monitor on the kitchen with ultra low latency, for when I am cooking.

I recently upgraded my living room TV to a 10 bit capable TV, and enabled it in linux, only to find out that my script to encode via FFMPEG no longer works.

My HTPC is a 12th gen Intel i5. I'm using the iGPU to encode the screen grab. I use FFmpeg to capture the screen with kmsgrab instead of x11 grab, because the CPU overhead needed to x11grab and then convert the video format in software for the encoder was pretty brutal (200-400% CPU). Using the iGPU to capture and encode allows for very light CPU usage, and ultra low latency (~100 ms).

I have been unable to figure out how (preferably via hardware) to down sample the 10-bit color capture to an 8-bit space before handing it off to the h264 vaapi encoder.

I get the following output, which I confirmed is a result of me enabling 10 bit color in Linux. I am also posting my ffmpeg command for reference. I tried sticking a yuv420p command in the middle of the filters, but it didn't work.

sudo ffmpeg -r 30 -f kmsgrab -i - \
               -vaapi_device /dev/dri/renderD128 \
               -vf 'hwmap=derive_device=vaapi,scale_vaapi=w=1920:h=1080:format=nv12' \
               -c:v h264_vaapi -qp 28 -profile 100 \
               -bufsize 0 -f_strict experimental \
               -syncpoints none \
               -f mpegts tcp://fridge-pi:1234

[kmsgrab @ 0x564819d9dac0] Using plane 31 to locate framebuffers.
[kmsgrab @ 0x564819d9dac0] Template framebuffer is 218: 3840x2160 format 30335258 modifier 100000000000001 flags 2.
Input #0, kmsgrab, from 'pipe:':
  Duration: N/A, start: 1676969201.004857, bitrate: N/A
  Stream #0:0: Video: wrapped_avframe, drm_prime, 3840x2160, 29.83 tbr, 1000k tbn, 1000k tbc
Please use -profile:a or -profile:v, -profile is ambiguous
Stream mapping:
  Stream #0:0 -> #0:0 (wrapped_avframe (native) -> h264 (h264_vaapi))
[AVHWFramesContext @ 0x564819e846c0] DRM format not supported by VAAPI.
[Parsed_hwmap_0 @ 0x564819e23dc0] Failed to map frame: -22.
Error while filtering: Invalid argument
Failed to inject frame into filter network: Invalid argument
Error while processing the decoded data for stream #0:0
Conversion failed!

Is there a way to convert the captured stream to 8-bit video and hand it off to the encoder? The raspberry pi and monitor I'm using do not support 10 bit video. The RPi has a hardware accelerator for h264, which I use to get desired frame rates.

If I can't get this working, looks like I'll get to go back to 24-bit color instead of 30.

Edit: I spent several more hours playing around with it and believe the issue is not what I thought it was. It appears that kmsgrab is unable to capture 10 bit color. I found a filter chain that can do tone mapping in VAAPI, but that doesn't work either.

sudo /usr/bin/ffmpeg -r 30 -f x11grab -vsync 1 -i :0.0+0,0 -s 1920x1080 -r 30 \
               -vaapi_device /dev/dri/renderD128 \
               -vf 'hwmap=derive_device=vaapi,scale_vaapi=w=1920:h=1080,tonemap_vaapi=format=nv12:primaries=bt709:t=bt709' \
               -c:v h264_vaapi -qp 28 \
               -f mpegts tcp://fridge-pi:1234