player: open image format [enhancement]

[Xuno]

Can code be tuned for opening image formats like PNG or TIFF as video source?
I want to add a feature for open sequence of images for playback.

i.e. - ffmpeg -r 25 -i %04d.tif

If video source selected as OpenUrl in format like C:\Video\image-seq%04d.tif,
then the player plays image sequence only if non OpenGL rendering.

Sample of playback if OpenGL is used:

Pixel format: rgb48le

Playback if QPainter rendering is used:

--- Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/6891951-player-open-image-format-enhancement?utm_campaign=plugin&utm_content=tracker%2F307703&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F307703&utm_medium=issues&utm_source=github).

[wang-bin]

can you send me 4gb48le image?

[Xuno]

Here are 10 4096x1756 rgb48le .tif frames 51.84MB:

Sequence usage %06d.tif

http://www.xuno.com/content/RED/rgb48le/4096x1756/000001.tif
http://www.xuno.com/content/RED/rgb48le/4096x1756/000002.tif
http://www.xuno.com/content/RED/rgb48le/4096x1756/000003.tif
http://www.xuno.com/content/RED/rgb48le/4096x1756/000004.tif
http://www.xuno.com/content/RED/rgb48le/4096x1756/000005.tif
http://www.xuno.com/content/RED/rgb48le/4096x1756/000006.tif
http://www.xuno.com/content/RED/rgb48le/4096x1756/000007.tif
http://www.xuno.com/content/RED/rgb48le/4096x1756/000008.tif
http://www.xuno.com/content/RED/rgb48le/4096x1756/000009.tif
http://www.xuno.com/content/RED/rgb48le/4096x1756/0000010.tif

[Xuno]

Additional issue with playing 1920x1080 .bmp sequences at 25fps, the queue (as indicated in the log file) becomes empty after about 11 seconds and playback stops. If you then click anywhere on the timeline video plays for a few seconds before stopping again.

Debug: ("C:\Program Files\XunoPlayer\bin\XunoPlaye.exe")
Debug: vo: gl
Debug: 35 x 140
Debug: analyzeFilename :: startFrame : 1
Debug: MainWindow::loadRemoteUrlPresset url: I:/Video/Blackmagic_1920x1080/%06d.bmp, lurl: I:/Video/Blackmagic_1920x1080/%06d.config
Debug: ScreenSaver::retrieveState
Debug: Succeed to enable screen saver (0)
Warning: Queue is still empty
Debug: Succeed to enable screen saver (0)
Warning: Queue is still empty
Debug: sync config to "C:/Users/Private/.QtAV/player.ini"
Warning: ScreenSaver::restoreState: no data, doing nothing

1920x800 .bmp sequences plays OK without stopping, so the issue appears to be a data limitation with the slightly larger 1920x1080 .bmp images.

The same 1920x1080 image sequence plays OK in ffmpeg:

ffplay -i c:\RAW_Frames\Blackmagic_1920x1080%06d.bmp

30 second .bmp test sequences can be generated with the following example:

ffmpeg -i inputfile.mp4 -r 25 -t 30 -s 1920x1080 c:\RAW_Frames%06d.bmp

[wang-bin]

rgb48 is supported now. only desktop gl works. opengl es (ANGLE on windows) does not.

[Xuno]

rgb48le - works , rgb48be - doesn't work
If video is replayed after stopping log shows the following:

Warning: QWindowsGLContext::getProcAddress: Unable to resolve 'glDetachShader'
Warning: QWindowsGLContext::getProcAddress: Unable to resolve 'glDetachShaderARB'
Warning: QWindowsGLContext::getProcAddress: Unable to resolve 'glDetachObject'

[Xuno]

About Warning: Queue is still empty

Bug detected while playing image sequences. If local file system is momentarily overloaded video freezes but timeline slider continues. Two possible user options:

1. Video should skip ahead to catch up with timeline when system resources become available.
2. Video play is slowed down to as fast as the system can read and display the images.

i.e 4K RAW images are heavy on system resources so it can be expected pausing will occur while system catches up, but playback session should not end.

Play can be continued if video seek in done at some pos + or -. No result with pressing play/pause
Question: if image frame drops in timeline, how does it resume playing while skipping to another position?

With the proliferation of 2K and 4K cameras entering the market that can shoot in RAW image format there is a need for user friendly image sequence playback without going through the video compression process.

[wang-bin]

rgb48be: try to change src/utils/OpenGLHelper.cpp line 122

{VideoFormat::Format_B16G16R16, GL_RGB, GL_RGB, GL_UNSIGNED_SHORT },

to

{VideoFormat::Format_B16G16R16, GL_BGR, GL_BGR, GL_UNSIGNED_SHORT },

about another question:

I didn't think about this case. Current ly, player will use a clock it's time increase linearly. Video thread will drop frames if video is too slow compared too the clock. So the progress bar is always updating, but video is not displayed.

So a solution is do not use that clock and ensure no frame will be dropped. It seems not hard. I will add it recently.

[Xuno]

static const fmt_entry pixfmt_gl_entry_common[] = { {VideoFormat::Format_RGBA32, GL_RGBA, GL_RGBA, GL_UNSIGNED_BYTE }, // only tested for osx, win, angle {VideoFormat::Format_ABGR32, GL_RGBA, GL_RGBA, GL_UNSIGNED_BYTE }, // only tested for osx, win, angle {VideoFormat::Format_RGB48, GL_RGB, GL_RGB, GL_UNSIGNED_SHORT }, {VideoFormat::Format_R16G16B16, GL_RGB, GL_RGB, GL_UNSIGNED_SHORT }, {VideoFormat::Format_B16G16R16, GL_BGR, GL_BGR, GL_UNSIGNED_SHORT }, {VideoFormat::Format_RGB24, GL_RGB, GL_RGB, GL_UNSIGNED_BYTE },

it not help, same problem

[wang-bin]

share with me your rgb48be image

[Xuno]

December, 19 was sent to your personal email
now upload to server,
http://www.xuno.com/images/github/outpng-rgb48be/0001.png
...
http://www.xuno.com/images/github/outpng-rgb48be/0014.png

[wang-bin]

I have no idea now. There is another solution. In player's menu Colorspace submenu, change engine to swscale instead of GLSL always works. I see mpv uses software conversion for rgb48be too.

[wang-bin]

I add a clock option in player menu. choose video clock then no frame drop.

rgb48be is a little complex. I had wrong understanding of rgb48le and be. I'm not sure whether my understanding is correct now. In fact, I don't know what's the difference of GL_RGB and GL_BGR.

every component in rgb48 has 2 bytes, so rgb48le is b2b1 g2g1 r2r1 and rgb48be is r1r2 g1g2 b1b2.
so if rgb48le is working in glsl, i think we can use the sample code in c++ for rgb48be, then reorder every color component in glsl. I add a function in rgb.f.glsl

float swap16(float x)
{
    int X = int(x*65535.0);
    int X2 = int(X/256);
    int X1 = X-X2*256;
    return float(X1*256+X2)/65535.0;
}

It change x (for example, value of r1r2) to another endian value (r2r1)

and for rgb48be, I use the following code to get the color

gl_FragColor = u_colorMatrix*vec4(
            swap16(texture2D(u_Texture0, v_TexCoords).b),
            swap16(texture2D(u_Texture0, v_TexCoords).g),
            swap16(texture2D(u_Texture0, v_TexCoords).r),
            1.0) * u_opacity;

It does not work. I don't know why. Do you have any idea?