0

I have separate raw audio and video frames from a camera that I'm trying to merge to a single file.

Video processing

I have JVT NAL sequence, H.264 video, main @ L 31 data, I had the frames separated and I appended (concat) the frames to one file.

here is the ffprobe output of the data:

ffprobe version 4.2.7-0ubuntu0.1 Copyright (c) 2007-2022 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.4.0-1ubuntu1~20.04.1)
  configuration: --prefix=/usr --extra-version=0ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
Input #0, h264, from 'frame_data.h264':
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: h264 (Main), yuvj420p(pc, bt709, progressive), 1280x720, 25 fps, 25 tbr, 1200k tbn, 50 tbc

the file is being played on vlc, however the video is played like it is on fast forward (every 2 seconds of the video should be about 1 second)

The video should run at max 15fps (according to the camera specification)

here is ffprobe output of avi file that contains the raw frames and plays the audio and video correctly after the frames had been processed somehow, This avi file info is for reference only - to compare it with the raw frames to help figure out how to process the frames:

Input #0, avi, from 't.avi':
  Duration: 00:03:29.00, start: 0.000000, bitrate: 1116 kb/s
    Stream #0:0: Video: h264 (Main) (H264 / 0x34363248), yuvj420p(pc, bt709, progressive), 1280x720, 849 kb/s, 30 fps, 15 tbr, 30 tbn, 60 tbc

I tried this command ffmpeg -i frame_data.h264 -filter:v fps=15 test.h264 on the raw frames but the video is still playing like its on fast forward

vlc data before changing fps to 15

vlc screenshot vlc screenshot

vlc data after changing fps to 15

vlc screenshot vlc screenshot

however playing the video via ffplay frame_data.h264 -framerate 15 make it much better, almost synced all the time, however occasionally the video become faster for couple of seconds (maybe it should be synced other way and not by frame rate?)

frames info (raw)

[FRAME]
media_type=video
stream_index=0
key_frame=1
pkt_pts=N/A
pkt_pts_time=N/A
-> pkt_dts=N/A
-> pkt_dts_time=N/A
best_effort_timestamp=N/A
best_effort_timestamp_time=N/A
-> pkt_duration=48000
-> pkt_duration_time=0.040000
pkt_pos=1476573
pkt_size=57677
width=1280
height=720
pix_fmt=yuvj420p
sample_aspect_ratio=N/A
pict_type=I
coded_picture_number=189
display_picture_number=0
interlaced_frame=0
top_field_first=0
repeat_pict=0
color_range=pc
color_space=bt709
color_primaries=bt709
color_transfer=bt709
chroma_location=left
[/FRAME]
[FRAME]
media_type=video
stream_index=0
key_frame=0
pkt_pts=N/A
pkt_pts_time=N/A
-> pkt_dts=N/A
-> pkt_dts_time=N/A
best_effort_timestamp=N/A
best_effort_timestamp_time=N/A
-> pkt_duration=48000
-> pkt_duration_time=0.040000
pkt_pos=1534250
pkt_size=3928
width=1280
height=720
pix_fmt=yuvj420p
sample_aspect_ratio=N/A
pict_type=P
coded_picture_number=190
display_picture_number=0
interlaced_frame=0
top_field_first=0
repeat_pict=0
color_range=pc
color_space=bt709
color_primaries=bt709
color_transfer=bt709
chroma_location=left
[/FRAME]

frame info from working (processed) avi

[FRAME]
media_type=video
stream_index=0
key_frame=0
pkt_pts=N/A
pkt_pts_time=N/A
-> pkt_dts=500
-> pkt_dts_time=16.666667
best_effort_timestamp=500
best_effort_timestamp_time=16.666667
-> pkt_duration=1
-> pkt_duration_time=0.033333
pkt_pos=1772182
pkt_size=3070
width=1280
height=720
pix_fmt=yuvj420p
sample_aspect_ratio=N/A
pict_type=P
coded_picture_number=191
display_picture_number=0
interlaced_frame=0
top_field_first=0
repeat_pict=0
color_range=pc
color_space=bt709
color_primaries=bt709
color_transfer=bt709
chroma_location=left
[/FRAME]

pkt_pts and pkt_dts are missing in the raw data, maybe that is the reason the video plays on fast forward? in the working (processed) avi pkt_dts, pkt_dts_time, pkt_duration_time are not missing. pkt_duration_time is always 0.033333 pkt_dts always jumps by 2 (there is no odd pkt_dts)

Audio processing

I have raw pcm audio of the video:

here is an output of ffplay -f s16le -ar 16k -ac 1 frame_data2.h264 on the raw data

ffplay screenshot5

The raw audio does not playing well using this command.

I guessed the parameters (incorrectly?) from working avi file that contains the pcm data:

Stream #0:1: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 16000 Hz, 1 channels, s16, 256 kb/s

here is frame info from working (processed) avi file

[FRAME]
media_type=audio
stream_index=1
key_frame=1
pkt_pts=4320
pkt_pts_time=0.270000
pkt_dts=4320
pkt_dts_time=0.270000
best_effort_timestamp=4320
best_effort_timestamp_time=0.270000
pkt_duration=480
pkt_duration_time=0.030000
pkt_pos=11670
pkt_size=960
sample_fmt=s16
nb_samples=480
channels=1
channel_layout=unknown
[/FRAME]

I'm trying to understand how should I process the video and the audio so that they will play well (merge them myself with the right av processing to avi/mp4 file)

I also find this extra data that was included for each frame: I suspect the first column is a sequence number maybe related to dts somehow, second column is a timestamp (milliseconds/microseconds?)

600 0xd96533 (audio)
601 0xd9e1dd (audio)
602 0xda4f52 (audio)
603 0xda5a63 (video)
604 0xdacc4b (audio)
605 0xdb39a3 (audio)
606 0xdb5ee9 (video)
607 0xdbb6d8 (audio)
608 0xdc23fe (audio)
609 0xdcb255 (audio)
610 0xdd0e69 (audio)
611 0xdd8b96 (audio)
612 0xdd67d0 (video)
613 0xddf8bd (audio)
614 0xde75fe (audio)
615 0xde6c28 (video)
616 0xdee328 (audio)
617 0xdf6012 (audio)
618 0xdfcd9d (audio)
619 0xe04a7d (audio)
620 0xe07523 (video)
621 0xe0b80c (audio)
622 0xe134db (audio)
623 0xe1798b (video)
624 0xe1a251 (audio)
625 0xe21f5c (audio)
626 0xe28ce1 (audio)
627 0xe27df7 (video)
628 0xe309a7 (audio)
629 0xe376fe (audio)
630 0xe3826e (video)
631 0xe3f3fd (audio)
632 0xe46154 (audio)
633 0xe486e9 (video)
634 0xe4de79 (audio)
635 0xe54bbe (audio)
636 0xe58b79 (video)
637 0xe5c8cb (audio)
638 0xe6361e (audio)
639 0xe6b35b (audio)
640 0xe68fb7 (video)
641 0xe72093 (audio)
642 0xe79db1 (audio)
643 0xe7942f (video)
644 0xe80b09 (audio)
645 0xe887ed (audio)
646 0xe8f546 (audio)
647 0xe97256 (audio)
648 0xe99d2c (video)
649 0xe9dfbd (audio)
650 0xea5ca8 (audio)

Thanks.

Update

I fixed the audio frames, I'm able to play them fine, now I only need help with merging the audio and the video to one file and syncing them with the timestamp (microseconds).

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.