I've worked through a similar project over the last 6 months, and performed more tests than I can keep track of now. So this is a little fresh for me. With your questions:
4K camera -> capture card -> Linux PC (compression) -> RTSP
This will work well
Are blackmagic ursa mini 4K and decklink card able to do the first 2
stages respectively?
Yes
Is ffmpeg + libx264 able to perform real-time compression at such a high bitrate?
It's possible, but I'm going to say that whichever way you spin it, you're likely going to arrive at getting a hardware card.
If not, any recommendation for hardware encoding card?
nVidia Quadro cards are excellent. Anything with a Maxwell GM204 chipset or higher. Even a sub $1000 card will open up the world of possibilities, especially when it comes to hardware accelerated resizing, format conversion, and producing multiple outputs.
A note about libx264
I advise care getting advice from anyone who hasn't used it for the purposes you're trying. The theoretical and unit tests don't always account for everything.
libx264 is an excellent encoder. It's well known it produces superior quality encodes than say a hardware accelerated h264 encoder for the same bitrate. It also has a whole heap of specific options that can maximise efficiency and produce a video with multiple 264 layers that is friendly to all types of clients.
The strength of libx264 isn't realtime encoding though. Yes it has zerolatency tuning options, and crf modes to reduce cpu demand, but every small change can make a major impact on encoding time. The difference between doing a 2560x1440 encode and a 3840x2160 encode is drastic. The difference between a yuv420p and yuv444p is also drastic.
Also, importantly, many of the options that make the significant difference in libx264 are lookahead options, these unashamedly prfioritize the end product as opposed to the performance of getting to the end product.
No implementation of x264 that I know of, does threading very well. You can have a rack server with 200GB ram, and 24 actual CPU cores, and max them out very quickly.
One of the "gotcha"'s of libx264 is that you can seemingly have it running so well that it looks like a video player - i.e. - you plug something into the SDI port, and it shows up on the network player. Amazing. You can have it running for days... Then all of a sudden when the input signal becomes very complex (i.e. low lighting, stage effects like haze, fast detailed motion, fades and transitions, etc.), all of a sudden the roof comes tumbling down reveal 35 issues that look like one. You remove the issues one by one, and by the time you have it running a treat, you may decide that x264 isn't going to cut it.
Having said that - running at a higher bitrate, requires less compression, and does run noticeably better than at low bitrates.
Given the cost of hardware cards, the doors they open up, and putting that up against the scope of your project, I would highly recommend getting one, whether you use it or not.
At high bitrates - the benefits of libx264 are negated - in fact - h264_nvenc can do realtime 2pass encoding, with spacial and temporal motion filtering and lookahead. Since you want less than 1s latency, you may not use these, but they are included in some low latency presets.
Blackmagic
The decklink SDK works well with linux. It's simple to configure card options, change inputs, and using the tools provided you build your own capture tools if required. Even a $199 Decklink 4K Mini Recorder PCIe card works well.
Example pipeline
Blackmagic -> Encoder -> RTSP
From that RTSP signal you could then run all of these at the same time:
- RTSP -> Low bitrate preview / wifi stream
- RTSP -> Archive
- RTSP -> Create RTMP stream -> Send to live streaming site
- RTSP -> Create a 1080p stream -> RTSP
- RTSP -> Create an adaptive DASH stream with multiple resolutions
