1

I'am using external camera with ffmpeg to acquire underwater live stream. Now I want to add some dynamic values (text overlay) over video stream and restream that to some external provider like youtube live... My values are from different sensors (temperature, oxygen, salinity etc..) and those values must be embedded into video stream.

What is the righ or "best" or "right" way to do this?

  • Since this has to be done on a live stream, use multiple drawtext filters reading from individual (updated) textfiles with the filter set to reload those files. – Gyan Nov 18 '17 at 7:36
  • Why doing it on the hard way 'I didn't test this method yet' but you can use .m3u8 file contins lets say two picture that will force ffmpeg to continusly load these picture , so you can change/replace/update these pictures file and ffpmeg will load them fast – Salem F Oct 17 at 18:50
1

This question a bit similar to I'm trying to add guitar tabs to a music video, do you know any software to do it?

At first, in which format your data, which protocol used from your sensors? I guess, here some closed-source software and proprietary protocol for there sensors? Obviously, you need software which can read this data format/protocol. If you creating own system with micro-controllers, try to find usable and open protocol for data. It very hard to find open protocols with needed functionality.

At this point I guess, you found any appropriate protocol. For example, JSON with custom fields. Now try to find/write software, which will be use it.

One possible way: write web-application, which will display video stream in your browser, loads data from sensors and display funny overlay. Then just grab video from your screen and broadcast to youtube or any other service. This is "dirty" way, but it will work.

Other way: write custom filter to ffmpeg. You can use ass-renderer for reference how to add text on image. Just add reading data from your sensors and your task completed.


Writing fast and dirty video filters

Sometimes we need to somehow filter the video, maybe put some text or graphic on top. Write a full application that decompresses the video stream, imbues the image with new elements or changes the image itself, and then compresses the stream anew, somewhat difficult. Today we will learn how to quickly and dirty make such applications without knowing how to encode the video in general.

To begin with, remember that the images consist of rows and columns of points, which are called pixels. Each point consists of 3 or 4 components, usually Red, Green and Blue signals, and sometimes there is an Alpha channel. Therefore, to change 1 point of the image, we need to change 3-4 components (this is usually 3-4 bytes). So, to change the entire image, we need to change all the rows and columns, in each of 3-4 elements and somehow modifying them, save back. For example, if we have a picture of 256x192 pixels in RGB24 format, then it will have 49152 pixels and 147456 color components (bytes).

Therefore, we just need to read pieces of 147456 bytes, somehow modify them and write them back! It's all! No complicated libraries or algorithms are required! Where do we get these bytes? And we can get them with wonderful utility ffmpeg and its equally wonderful rawvideo mode, the exhaust of which we will give to stdout and receive in stdin your application!

Let's write the simplest application that will slightly change the image:

#include <stdio.h>
#include <stdint.h>
#include <stdlib.h>

int main () {

// allocate variables and buffer

uint8_t *pixels = malloc (147456);
uint8_t *pix;
int q;

while (1) {

// read frame
if (fread (pixels, 1,147456, stdin) <= 0) {break;}

// process image
pix = pixels;
for (q = 0; q <49152; q ++) {
*pix++ = *pix * 2; // multiple red to 2
*pix++ = *pix + 120; // shift green channel
*pix++ = *pix + q / 10; // lines in blue channel
}

// write frame back
fwrite (pixels, 1,147456, stdout);
}

return EXIT_SUCCESS;
}

Yes, that's all! Of course, here all the constants are hardcoded, never do so in real projects!

Now, run this and upload a video here:

ffmpeg -i video.mp4 -s 256x192 -f rawvideo -pix_fmt rgb24 - | ./a.out | ffmpeg -s 256x192 -f rawvideo -pix_fmt rgb24 -i - -y out.mp4

And we get the following result:

enter image description here 2.gif

Let's analyze what parameters in this line participated:

ffmpeg -i video.mp4 -s 256x192 -f rawvideo -pix_fmt rgb24 -

Here we read from the file video.mp4, resize the picture to 256x192 (if your video is already 256x192 in size, then you can not do this), then we specify that it should be converted to rgb24 color space (if it is not already in it, what for can spend quite a lot of resources), and the result we need in the form of uncompressed video rawvideo, the result we record in -, i.e. in stdout.

| ./a.out |

And this is our application. Vertical "tubes" mean that it will take data from the previous application and transfer it to the next

ffmpeg -s 256x192 -f rawvideo -pix_fmt rgb24 -i - -y out.mp4

Here we indicate that the input of this ffmpeg will be uncompressed video in rawvideo format, size of 256x192 and in the rgb24 color space. It is also may need to specify fps, as it is lost from such conversion. We will write output to out.mp4, and even if such file already exists, the -y parameter is responsible for this

Here is a very simple example of changing the image, but you can overlay text, other pictures and anything else. Of course, it's better to write a full-fledged application, or at least patch the already existing filter to the desired state, this will be a more correct solution. But if you need a working solution, while you do not want to write a lot of code, this is the solution for you.

  • I can use JSON or XML to pass my sensor data.. or any other format. The problem I have is only how to pass data to ffmpeg.. – Ferguson Nov 18 '17 at 8:11
  • have found here the example: s55ma.radioamater.si/2017/08/04/… – Ferguson Nov 19 '17 at 10:31
  • Here you must write your text in file every... frame! Yes, this can work, but it pretty dirty for me. I don't like such tmp files and wasting cpu. So, I suggest write a simple filter for it. – bukkojot Nov 19 '17 at 12:09
  • Have tested with Java. Have made a simple Thread process which writes some values into file.. When the file is closed I execute Files.move method with StandardCopyOption.ATOMIC_MOVE.. I know it's dirty but have no much experience on how to make some plugin (If you have some good reference (link) please help me .. The other option is to use OBS studio but I must check If can be executed from command line (terminal) (without GUI).. – Ferguson Nov 19 '17 at 19:06
  • You have any experience with C or C++? You can write own filter on pipes, like ffmpeg -i _SOURCE_ -f rawvideo -pix_fmt rgb32 - | ./your_app | ffmpeg -f rawvideo -pix_fmt rgb32 -i - rtmp://broadcast.to.server/. This will pass frames to your app via stdin, then you can draw on it and pass bitmaps to stdout, where new instance of ffmpeg will encode it and save to file/broadcast. – bukkojot Nov 20 '17 at 1:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.