Single IDR frame segment as input

[AlexKordic]

Noticed in ZeroSegments error issue.

The output should be single frame file. Now we have 0-size output.

Sample: 7b328dfa-0d9a-4ab2-97a3-c6c643e1dbdc-954.ts-1647595505425.txt

[yondonfu]

Closed by #313

[thomshutt]

Reopening as the fix for this had to be rolled back

[MikeIndiaAlpha]

I don't have time to work on this (yet), but I was thinking a bit and here's the note to self:

• Most likely case for not getting the data out of single IDR "frame", is that we are not getting single IDR "frame", we are really getting single IDR slice NAL. Now, one of the problems with h.264 is that 1) there is no way of telling if the single slice NAL contains all the macro blocks of the frame without parsing the NAL in question and 2) some decoders cannot stop in the middle of the decoding (they'll reset, wasting time and computations), so they'd rather avoid starting without knowing 100% that all the macro blocks are in the input queue. The way that usually works is to wait for another NAL, and see if it is end of sequence, or slice belonging to another frame (that much can be told from slice header alone) or the like. Or alternatively the flush command.

Now we know that we have problems with flushing (from FFmpeg level) on some HW, and the workaround method is to re-feed first packet. Which in the case of first frame NAL may fail to trigger decoding logic because is again slice NAL of the same frame! Possible workarounds:

• Detect h.264 and use synthetic end of sequence NAL in that case
• Write our own wrapper for HW in question, allowing us to use correct flush/drain mechanisms instead of FFmpeg mechanism that does not work

[MikeIndiaAlpha]

Another note to self:

• It may well be that the problem is having not single IDR frame, but single IDR slice, and that slice not even being a complete information needed for frame decoding. So I guess the bug could really be in segmenter part?

[MikeIndiaAlpha]

Started working on this one today. I want to get to know the problem using version that I know best - that is, ma_refactoring_second branch. When I get to fully understand underlying mechanism I'll consider solving it on earlier version so it can be distributed sooner.

So far it seems to me that the frame passes the whole pipeline (at least I can see that not-so-small video packet gets added to the muxer) BUT the PTS of that packet is huge. And when I open the container file I can see that it has 25+minutes of content, despite no frames showing in ffprobe. So I will continue investigation with hypothesis that timing goes south...but will try to verify it asap.

[MikeIndiaAlpha]

To verify my hypothesis I first did:
./transcoding 7b328dfa-0d9a-4ab2-97a3-c6c643e1dbdc-954.ts P360p30fps4x3 sw
Note that resulting out_sw_0_out.mp4 is not playable by ffplay - nothing is shown or by MacOS "space bar" video preview.
But when I do:
ffmpeg -i out_sw_0_out.mp4 -vcodec copy -an -bsf:v h264_mp4toannexb test.h264
(in effect extracting all video frames into raw h.264 in Annex B form into test.h264)
ffplay test.h264 WILL display the file and the result is as expected - single frame just like when ffplaying original .ts file, but scaled down.
So it seems to me that 1) From video POV frame is processed correctly... 2) ...but it get assigned some crazy PTS while muxing or some such.

[MikeIndiaAlpha]

Ah - ok. Important thing. The above is on ma_refactoring_second branch. On the current master it works 100%. So it looks like first stage of refactoring changes fixed the flushing, but then second part of refactoring screwed muxing a bit, so it doesn't work for all muxing formats (despite video frames being there). Investigation reveals that commit 0c60e3d “Encoder creation logic moved” still works, and 47b07 “Muxer streams refactoring” doesn't.

The task is as follows, then: fix the change between 0x60e3d and 47b07!

[MikeIndiaAlpha]

OK, I found it. There were several errors, but what was doing mp4 in is failure to assign the frame rate. Relevant commit is fixed now and it should work in all cases.