FFmpeg text overlay add sensor values into video stream

Question

I'am using external camera with ffmpeg to acquire underwater live stream. Now I want to add some dynamic values (text overlay) over video stream and restream that to some external provider like youtube live... My values are from different sensors (temperature, oxygen, salinity etc..) and those values must be embedded into video stream.

What is the righ or "best" or "right" way to do this?

Answer 1

This question a bit similar to I'm trying to add guitar tabs to a music video, do you know any software to do it?

At first, in which format your data, which protocol used from your sensors? I guess, here some closed-source software and proprietary protocol for there sensors? Obviously, you need software which can read this data format/protocol. If you creating own system with micro-controllers, try to find usable and open protocol for data. It very hard to find open protocols with needed functionality.

At this point I guess, you found any appropriate protocol. For example, JSON with custom fields. Now try to find/write software, which will be use it.

One possible way: write web-application, which will display video stream in your browser, loads data from sensors and display funny overlay. Then just grab video from your screen and broadcast to youtube or any other service. This is "dirty" way, but it will work.

Other way: write custom filter to ffmpeg. You can use ass-renderer for reference how to add text on image. Just add reading data from your sensors and your task completed.

---

Writing fast and dirty video filters

Sometimes we need to somehow filter the video, maybe put some text or graphic on top. Write a full application that decompresses the video stream, imbues the image with new elements or changes the image itself, and then compresses the stream anew, somewhat difficult. Today we will learn how to quickly and dirty make such applications without knowing how to encode the video in general.

To begin with, remember that the images consist of rows and columns of points, which are called pixels. Each point consists of 3 or 4 components, usually Red, Green and Blue signals, and sometimes there is an Alpha channel. Therefore, to change 1 point of the image, we need to change 3-4 components (this is usually 3-4 bytes). So, to change the entire image, we need to change all the rows and columns, in each of 3-4 elements and somehow modifying them, save back. For example, if we have a picture of 256x192 pixels in RGB24 format, then it will have 49152 pixels and 147456 color components (bytes).

Therefore, we just need to read pieces of 147456 bytes, somehow modify them and write them back! It's all! No complicated libraries or algorithms are required! Where do we get these bytes? And we can get them with wonderful utility ffmpeg and its equally wonderful rawvideo mode, the exhaust of which we will give to stdout and receive in stdin your application!

Let's write the simplest application that will slightly change the image:

#include <stdio.h>
#include <stdint.h>
#include <stdlib.h>

int main () {

// allocate variables and buffer

uint8_t *pixels = malloc (147456);
uint8_t *pix;
int q;

while (1) {

// read frame
if (fread (pixels, 1,147456, stdin) <= 0) {break;}

// process image
pix = pixels;
for (q = 0; q <49152; q ++) {
*pix++ = *pix * 2; // multiple red to 2
*pix++ = *pix + 120; // shift green channel
*pix++ = *pix + q / 10; // lines in blue channel
}

// write frame back
fwrite (pixels, 1,147456, stdout);
}

return EXIT_SUCCESS;
}

Yes, that's all! Of course, here all the constants are hardcoded, never do so in real projects!

Now, run this and upload a video here:

ffmpeg -i video.mp4 -s 256x192 -f rawvideo -pix_fmt rgb24 - | ./a.out | ffmpeg -s 256x192 -f rawvideo -pix_fmt rgb24 -i - -y out.mp4

And we get the following result:

Let's analyze what parameters in this line participated:

ffmpeg -i video.mp4 -s 256x192 -f rawvideo -pix_fmt rgb24 -

Here we read from the file video.mp4, resize the picture to 256x192 (if your video is already 256x192 in size, then you can not do this), then we specify that it should be converted to rgb24 color space (if it is not already in it, what for can spend quite a lot of resources), and the result we need in the form of uncompressed video rawvideo, the result we record in -, i.e. in stdout.

| ./a.out |

And this is our application. Vertical "tubes" mean that it will take data from the previous application and transfer it to the next

ffmpeg -s 256x192 -f rawvideo -pix_fmt rgb24 -i - -y out.mp4

Here we indicate that the input of this ffmpeg will be uncompressed video in rawvideo format, size of 256x192 and in the rgb24 color space. It is also may need to specify fps, as it is lost from such conversion. We will write output to out.mp4, and even if such file already exists, the -y parameter is responsible for this

Here is a very simple example of changing the image, but you can overlay text, other pictures and anything else. Of course, it's better to write a full-fledged application, or at least patch the already existing filter to the desired state, this will be a more correct solution. But if you need a working solution, while you do not want to write a lot of code, this is the solution for you.