How to convert 10-bit H.265 videos to H.264 without losing quality

Question

I have a MPEG-H Part2/HVEC (H.265) .mkv file (Planar 4:2:0 YUV 10-bit LE). Since my LG PF 1500 beamer does not read H.265 but H.264 I would like to convert it (HVEC -> H.264). I think handbrake is a handy software to do so.

What settings should I use to get a H.264 file with equal quality like the original (HVEC)?

I think I have to chose H.264 10-bit as videoencoder. However I am not sure about the Quality (RF) setting. How do I know what the HVEC quality setting is and how can I pick an equal one for H.264?

This is the info extracted with MediaInfo:

Format : Matroska Format version
: Version 4 File size : 1.48 GiB Duration : 52 min 46 s Overall bit rate : 4 012 kb/s Encoded date
: UTC 2018-05-26 19:19:57 Writing application : mkvmerge v20.0.0 ('I Am The Sun') 64-bit Writing library
: libebml v1.3.5 + libmatroska v1.4.8 Writing frontend
: StaxRip v1.7.0.6

Video ID : 1 Format
: HEVC Format/Info : High Efficiency Video Coding Format profile : Main 10@L4@Main Codec ID : V_MPEGH/ISO/HEVC Duration : 52 min 46 s Bit rate
: 3 529 kb/s Width : 1 920 pixels Height : 1 080 pixels Display aspect ratio : 16:9 Frame rate mode
: Constant Frame rate : 23.976 (24000/1001) FPS Color space : YUV Chroma subsampling : 4:2:0 Bit depth
: 10 bits Bits/(Pixel*Frame) : 0.071 Stream size : 1.30 GiB (88%) Writing library : x265 2.7+346-69aafa6d70ad:[Windows][GCC 7.3.0][64 bit] 10bit Encoding settings : cpuid=1111039 / frame-threads=2 / numa-pools=4 / wpp / no-pmode / no-pme / no-psnr / ssim / log-level=2 / input-csp=1 / input-res=1920x1080 / interlace=0 / total-frames=75917 / level-idc=0 / high-tier=1 / uhd-bd=0 / ref=3 / no-allow-non-conformance / no-repeat-headers / annexb / no-aud / no-hrd / info / hash=0 / no-temporal-layers / open-gop / min-keyint=23 / keyint=250 / gop-lookahead=0 / bframes=8 / b-adapt=2 / b-pyramid / bframe-bias=0 / rc-lookahead=50 / lookahead-slices=6 / scenecut=40 / radl=0 / no-intra-refresh / ctu=64 / min-cu-size=8 / no-rect / no-amp / max-tu-size=32 / tu-inter-depth=1 / tu-intra-depth=1 / limit-tu=0 / rdoq-level=0 / dynamic-rd=0.00 / no-ssim-rd / signhide / no-tskip / nr-intra=0 / nr-inter=0 / no-constrained-intra / no-strong-intra-smoothing / max-merge=3 / limit-refs=3 / no-limit-modes / me=3 / subme=3 / merange=57 / temporal-mvp / weightp / no-weightb / no-analyze-src-pics / deblock=0:0 / sao / no-sao-non-deblock / rd=3 / no-early-skip / no-rskip / no-fast-intra / no-tskip-fast / no-cu-lossless / no-b-intra / no-splitrd-skip / rdpenalty=0 / psy-rd=2.00 / psy-rdoq=0.00 / no-rd-refine / no-lossless / cbqpoffs=0 / crqpoffs=0 / rc=crf / crf=20.0 / qcomp=0.60 / qpstep=4 / stats-write=0 / stats-read=0 / ipratio=1.40 / pbratio=1.30 / aq-mode=3 / aq-strength=1.00 / cutree / zone-count=0 / no-strict-cbr / qg-size=32 / no-rc-grain / qpmax=69 / qpmin=0 / no-const-vbv / sar=0 / overscan=0 / videoformat=5 / range=0 / colorprim=2 / transfer=2 / colormatrix=2 / chromaloc=0 / display-window=0 / max-cll=0,0 / min-luma=0 / max-luma=1023 / log2-max-poc-lsb=8 / vui-timing-info / vui-hrd-info / slices=1 / no-opt-qp-pps / no-opt-ref-list-length-pps / no-multi-pass-opt-rps / scenecut-bias=0.05 / no-opt-cu-delta-qp / no-aq-motion / no-hdr / no-hdr-opt / no-dhdr10-opt / no-idr-recovery-sei / analysis-reuse-level=5 / scale-factor=0 / refine-intra=0 / refine-inter=0 / refine-mv=0 / no-limit-sao / ctu-info=0 / no-lowpass-dct / refine-mv-type=0 / copy-pic=1 / max-ausize-factor=1.0 / no-dynamic-refine / no-single-sei Default
: Yes Forced : No

Tried to find a solution in the book: Video Coding Standards and Video Formats by Rao, however, I could not find a good hint.

Answer 1

It looks like the Video is encoded in 4000 kbit/s, instead of a Rate Factor (RF).
But that isn't the issue here. The issue is: while 264 does support lossless encoding, it not widely compatible! (you would do this by setting the color format to hi444 and -cq 0 (Thanks to Gyan for pointing this out))

If you want the best quality, but also the largest file, you want RF = 1.
If you want very good quality, but still huge files, you want RF <= 16
good quality with big files 16 < RF <=24
mediocre quality with medium filesize 24 < RF <= 28
up to 51 gets worse and smaller.

You might also want to consider using ffmpeg for transcoding video files, that you don't want to edit, since most video editors also loose quality in the process of re-rendering the inputs for the encoder.

ffmpeg stuff

Link: https://ffmpeg.zeranoe.com/builds/
Full Documentation: https://ffmpeg.org/documentation.html
If you want good quality, but long encoding times use this in a CMD that can see the ffmpeg.exe:

ffmpeg -i my_input_file.mkv -map 0:v? -map 0:a? -map 0:s? -c:v libx264 -preset:v medium -profile:v high -rc-lookahead 900 -crf 16 -c:a copy -c:s copy my_output_file.mkv

The parameters mean the following:
-i <file> the file to transcode
-map 0:v? use all video stream(s) from input 0. The question mark means "if they exist"
-map 0:a? use all audio streams
-map 0:s? use all subtitle streams
-map 0:d? use all data
-map 0:t? use all attachments
-c:v libx264 convert the video using the libx264 encoder.
-preset:v medium use medium speed. Alternatives are ultrafast, superfast, veryfast, faster, fast, medium, slow, slower, veryslow, placebo (the faster, the larger the file, slower profile have diminishing results. placebo takes a multiple of the time of veryslow, but results in less than 0.1% smaller size
-profile:v high use high-quality encoding. (other options are just there for backward compatibility)
-rc-lookahead 900 look up to 900 frames ahead to find the best encoding of the current frame. More results in slightly (like placebo) better quality per size, smaller the opposite. Same principle as in -preset applies.
-crf 16 the rate factor. The higher the number, the smaller the size and uglier the picture. About every 6 points, half the file size. Most humans won't notice a difference with -crf 20. Many not even with -crf 24
-c:a copy leave audio as is.
-c:s copy leave subtitles as is.

If you have an nVidia GPU you might want to use the integrated encoding unit. Use this command instead:

ffmpeg -i my_input_file.mkv -map 0:v? -map 0:a? -map 0:s? -c:v h264_nvenc -preset:v medium -profile:v high -rc constqp -qp 16 -c:a copy -c:s copy my_output_file.mkv

Similar paramters. preset is limited to fast, medium, slow, where slow would require additional configuration beyond this answer.
The main change is, that the nvidia encoder doesn't support lookahead and because of that not the crf option. For that reason we use the cq option (Constant Quantization). The difference between these two is, that crf tries to jiggle around the defined value, based on the current and following pictures, while cq stays at that exact value for the entire time.

You can try using constant bitrates, but that is discouraged, because your video quality will be all over the place, because most frames don't have the same difficulty as their predecessors.

Hope this helps

Greetings