Extract PTS, Frame Number from a TS file using the bytes representation of it

Question

I am trying to write a Python script which will connect the TDT with the PTS in a MPEG-2 TS. So far I have managed to convert the TS file into a .dat file where I have all the hex representation of the 188 bytes on a different line. I have couple of problems:

1. I want to separate the frames and put an incremental number in front of the packets from this frame. For example, all the packets belonging to frame Nr. 1 to have fr%5d in front. I can easily do that using Python as long as I know how I can separate between the frames.
2. I want to decode the PTS from the PCR. So far I have extracted the PTS values using FFMPEG and I have a list for each frame which PTS it has. I have tried to grep the .dat file for the hex representation of the PTS but it doesn't find any matches. So I guess the PTS is coded in a different form, but so far I wasn't able to figure out the syntax of it.

My .dat file has the following content:

b'4700ff1de05ca7686c2b43f5e37e6dafd388761c36900ab37e10409a1dcbdd95dd1b9492ccbf4d75518e357d00cdb41536b54c2e5eb093d46f006835f9ebc39d6e5fe2401f1495cb8d2f9a6190ac3f3d2e519115ca05df63295d94a2c4b4b1c1b921e1a5610e9b7f6ccc69693a142807f4d09ac1ae53b61125f8a0e3efe1f11cb95f0e75dd49231dc0aa02aa3e72045783cb27ecfb591cdae658b04fd75e3f68968da6710f3a08ec3dc84d1eb04fac0e337b21d47560d457b58ad7f6'  b'4700ff1e59ab778d0013a08a07f9733c31daab4375ce1b44b39e9426d26963dff9a87601533d3a7c326849248ee8f0ab6f9677c8b19c198f3da107a76a10bda99dfec628136c5dd7508ec2c7546db32eef7027ed91d8dffe1c64672d41b5d848478bb2ae8c41c62f3305b29005912965a222948a77a81de7cf59cfd745c4a469d595d50432ce05704e7e4e8edadeb250cb792780a410223d4810e72bc67ddba39b7c5d0bf48e6aea508f2dec5c1a2573210610c412f4020d3e0bd130'
b'4700ff1f2b05cbec47587f67c444b52f95d51cf52f2e54ad60cb2089015c1286b88bde1ada2290f7de9a8955c112d122dd83884127a85d60d783d0c7ac62ce7d2176d98f8b7ffa82c45d661394e3f6765ba4226e030c485f709f8cd57da40dfbaceed7332d82f4dc8bce05ca6ed5bc3262399b606f9c16daada74b65691729a48659f63aa847c96d41467a98134ee2eda6edf7e5569f3b5ec04fc274d8026bf182542be6529985159df52fda6db38908853577ac3ecb716126186b3a'
b'4700ff10625a2814a692660f91c9802011f0cfefd21029bc4d574921a5ff61daed60f4865765272d7885a5f6d1f00a6b911566041f9b09d5fb8543286f027e5d1790c883e9fc30f11e67e64b4eba58845563976d5055333c9b5292c28b202dd74457d7900356ce4581e63e7be76ac741685039f37f345adcaab4d77c38b9cc98c9969dbd5016b361c6a641b60f76773f3b0e11fee995bded7012f80fad299ca44c4a7c4715114c52edcaf44521d1bbe12cddb89a4711094a8853920d'
b'4700ff1179fb10469aedae24c5e85cdda33de151c1bd83e8f7573f44d101d3a11b42ba9599c02f134a4bd4795e5afcdade291bc1a3e8a1da4da150cbe94c863e8cd0c04ce686333af6ba25f2f645170c387c4346df8c7e85cda2be546136e09e12b612104ada9990c12d745e1de68a587114c9e268b1e14dd5f68c5037ed2c3acb2aa65e988ac990f2ae1089e962ffd495e5580e8f3525fa3386ed752432d443ff3ca93ce86a20b9114c3d2554b5e701e41d9548c903a3c17498a49b'

So as you can see each packet starts with 0x47 which is the sync byte. I have all kind of PIDs inside the TS. Here I have shown only the VPID 0x00ff. The 0x47 is the sync byte and immediately after it comes the PID.

[EDIT]: I have uploaded the binary file here: wetransfer_link

Answer 1

As per my reading of the demuxer, this is the method to derive the PTS from a base 16 dump.

b'4700ff1de05ca7686c2b43f5e37e6dafd388761c36900ab37...

Skip the first 7 bytes, or more precisely, the first 3 bytes after the PID (00ff). The timestamp is derived from the next 6 bytes.

a7686c2b43f5

There are two components to the PCR - the 90 kHz clock and a 27 Mhz extension.

First left-shift the initial 4 bytes by 1

a7686c2b << 1 = 14ed0d856

Then right-shift the 5th byte by 7 i.e. isolate the high bit.

43 >> 7 = 0

Bitwise-OR the above two results.

14ed0d856 | 0 = 14ed0d856

The above result is the first component C1.

Now, isolate the low bit of the 5th byte and left-shift it by 8

(43 & 1) << 8 = 1

Bitwise-OR this result with the 6th byte

100000000 | f5 = 01f5

This is the 2nd component C2.

The timestamp in the 27 Mhz timebase is C1 * 300 + C2. In seconds, that is

(5617277014 * 300 + 501) / 27000000 = 62414.189063s

You can ignore C2 and just divide C1 by 90000 to get a less precise timestamp.

Answer 2

The PCR is constructed by two components PCR base and PCR extension. In order to extract it you need to get the binary(hex) data between the 7th and 12th bytes (6 in total). So the first 33 bits are constructing the pcr_base, then there are 6 bits which are reserved and 9 bits constructing the pcr_extension. The PCR = pcr_base * 300 + pcr_extension. So in my example for this VPID:

4700ff1de05ca7686c2b43f5e37e6dafd388761c36900ab37e10409a1dcbdd95dd1b9492ccbf4d75518e357d00cdb41536b54c2e5eb093d46f006835f9ebc39d6e5fe2401f1495cb8d2f9a6190ac3f3d2e519115ca05df63295d94a2c4b4b1c1b921e1a5610e9b7f6ccc69693a142807f4d09ac1ae53b61125f8a0e3efe1f11cb95f0e75dd49231dc0aa02aa3e72045783cb27ecfb591cdae658b04fd75e3f68968da6710f3a08ec3dc84d1eb04fac0e337b21d47560d457b58ad7f6

The PCR will be 0xa7686c2b43f5 so when we convert the bytes into binary we have 0b101001110110100001101100001010110100001111110101, so the first 33 bits constitute the pcr_base and the last 9 the pcr_extension, the six bits in between are reserved, so you can ignore them.

pcr_base = 0b101001110110100001101100001010110 = 5617277014
pcr_extension = 0b111110101 = 501 
PCR = 5617277014 * 300 + 501 = 1685183104701