refactor plugin to use OpenGL fbo rendering

[mbaetgen-wup]

Rendering improvements

• use OpenGL render-to-texture with Frame Buffer Objects (FBO)
• send GL textures downstream - video/x-raw(memory:GLMemory)
• use buffer pool for managing GL textures
• optimized GL thread usage
• real-time rendering optimizations and real-time QoS
• offload rendering tasks from chain function to render buffer (sync and async rendering)

General fixes and improvements

• improved GST event handling (segments, QoS)
• fix some locking issues and race conditions
• fix sync issues with pipeline clock
• auto-detect offline or real-time rendering
• re-usable projectM/gst integration functions
• naming cleanup
• integrate projectM logging api
• use gstreamer-gl to resolve gl function pointers via gst_gl_context_get_proc_address()
• more docs

Requires projectM 4.2 (not yet released, current master branch version, newer than 2/26/26)

Test status
Completed, all known bugs are fixed. Has run for hundreds of hours on Linux.

Linux: Real-time rendering stable / Offline rendering stable
Windows: ?
MacOS: ?

[rootnotez]

Does the /examples directory need to be created? It is confusing with the /test directory, and makes it confusing where the audio/video samples are kept.

[mbaetgen-wup]

that's a good point. The example itself may also not be that useful in it's current state. I will remove it from this PR and add it again in a separate PR later.

[mbaetgen-wup]

gst_pipeline_is_live is a gstreamer function. The function is not supported by gstreamer on Windows at all, so I could see it not being part of some Linux distributions, or older Gstreamer releases.

There is currently a guard excluding the call for Windows, that guard may need to be broader to exclude all situations where that function is not available.

It's not critical, as long as the build succeeds, I would expect a linker error later though. It is used for offline / real-time pipeline auto-detection.
The default is offline rendering (is-live=false) if there is no auto-detection. There is an override property on the gst projectM element, so real-time rendering can be controlled when defining the Gstreamer pipeline via projectm is-live=true ...

[mbaetgen-wup]

try it now..

[rootnotez]

The issue with gst_pipeline_is_live breaking Debian 12 builds, because off Debian packaging an older version of Gstreamer (1.22) was tested and resolved with fb39b13

I just want to put on folks radar that a new stable release of Gstreamer is in the works, but in the interim, note that 1.27 is an (API) unstable release.

https://gstreamer.freedesktop.org/releases/1.28/

[nuess0r]

Hi

Thanks a lot for your work and to continue optimizing it.

I gave it a try again (My last attempt was August 2025) but I ran again into issues. Different ones than in summer so progress! :-)
frontend-SDL2 runs OK

Intel(R) Core(TM) i5-7300U CPU

I'm running Debian SID (unstable):
projectM ceb86ebca77e9d6b5ff033c78eb016467ea68380 (master)
gst-launch-1.0 version 1.28.0
gcc 15
kernel 6.12 (still waiting until my Intel soundwire sound is fixed in ALSA with newer kernels)
mesa 26.0.0
wayland-info.txt

$ rm ~/.cache/gstreamer-1.0/registry.x86_64.bin
$ gst-launch-1.0 -vv audiotestsrc ! queue ! audioconvert ! projectm ! video/x-raw,width=512,height=512,framerate=60/1 ! videoconvert ! xvimagesink sync=false
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
/GstPipeline:pipeline0/GstAudioTestSrc:audiotestsrc0.GstPad:src: caps = audio/x-raw, format=(string)S16LE, layout=(string)interleaved, rate=(int)44100, channels=(int)1, channel-mask=(bitmask)0x0000000000000003
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:sink: caps = audio/x-raw, format=(string)S16LE, layout=(string)interleaved, rate=(int)44100, channels=(int)1, channel-mask=(bitmask)0x0000000000000003
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:src: caps = audio/x-raw, format=(string)S16LE, layout=(string)interleaved, rate=(int)44100, channels=(int)1, channel-mask=(bitmask)0x0000000000000003
/GstPipeline:pipeline0/GstAudioConvert:audioconvert0.GstPad:src: caps = audio/x-raw, format=(string)S16LE, layout=(string)interleaved, rate=(int)44100, channels=(int)1, channel-mask=(bitmask)0x0000000000000003
/GstPipeline:pipeline0/GstProjectM:projectm0.GstPad:src: caps = video/x-raw, width=(int)512, height=(int)512, framerate=(fraction)60/1, format=(string)ABGR
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw, width=(int)512, height=(int)512, framerate=(fraction)60/1, format=(string)ABGR
/GstPipeline:pipeline0/GstVideoConvert:videoconvert0.GstPad:src: caps = video/x-raw, width=(int)512, height=(int)512, framerate=(fraction)60/1, format=(string)BGRx
/GstPipeline:pipeline0/GstXvImageSink:xvimagesink0.GstPad:sink: caps = video/x-raw, width=(int)512, height=(int)512, framerate=(fraction)60/1, format=(string)BGRx
/GstPipeline:pipeline0/GstVideoConvert:videoconvert0.GstPad:sink: caps = video/x-raw, width=(int)512, height=(int)512, framerate=(fraction)60/1, format=(string)ABGR
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw, width=(int)512, height=(int)512, framerate=(fraction)60/1, format=(string)ABGR
Got context from element 'projectm0': gst.gl.GLDisplay=context, gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayWayland\)\ gldisplaywayland0";
ERROR: from element /GstPipeline:pipeline0/GstProjectM:projectm0: Subclass failed to initialize.
Additional debug info:
/home/chrigi/git/gst-projectm/src/gstglbaseaudiovisualizer.c(490): gst_gl_base_audio_visualizer_find_gl_context_unlocked (): /GstPipeline:pipeline0/GstProjectM:projectm0
ERROR: pipeline doesn't want to preroll.
Setting pipeline to NULL ...
ERROR: from element /GstPipeline:pipeline0/GstProjectM:projectm0: Subclass failed to initialize.
Additional debug info:
/home/chrigi/git/gst-projectm/src/gstglbaseaudiovisualizer.c(490): gst_gl_base_audio_visualizer_find_gl_context_unlocked (): /GstPipeline:pipeline0/GstProjectM:projectm0
ERROR: pipeline doesn't want to preroll.
ERROR: from element /GstPipeline:pipeline0/GstProjectM:projectm0: Subclass failed to initialize.
Additional debug info:
/home/chrigi/git/gst-projectm/src/gstglbaseaudiovisualizer.c(490): gst_gl_base_audio_visualizer_find_gl_context_unlocked (): /GstPipeline:pipeline0/GstProjectM:projectm0
ERROR: pipeline doesn't want to preroll.
ERROR: from element /GstPipeline:pipeline0/GstProjectM:projectm0: Subclass failed to initialize.
Additional debug info:
/home/chrigi/git/gst-projectm/src/gstglbaseaudiovisualizer.c(490): gst_gl_base_audio_visualizer_find_gl_context_unlocked (): /GstPipeline:pipeline0/GstProjectM:projectm0
ERROR: pipeline doesn't want to preroll.
ERROR: from element /GstPipeline:pipeline0/GstProjectM:projectm0: Subclass failed to initialize.
Additional debug info:
/home/chrigi/git/gst-projectm/src/gstglbaseaudiovisualizer.c(490): gst_gl_base_audio_visualizer_find_gl_context_unlocked (): /GstPipeline:pipeline0/GstProjectM:projectm0
ERROR: pipeline doesn't want to preroll.
ERROR: from element /GstPipeline:pipeline0/GstAudioTestSrc:audiotestsrc0: Internal data stream error.
Additional debug info:
../libs/gst/base/gstbasesrc.c(3187): gst_base_src_loop (): /GstPipeline:pipeline0/GstAudioTestSrc:audiotestsrc0:
streaming stopped, reason not-negotiated (-4)
Details:
details, flow-return=(int)-4;
ERROR: pipeline doesn't want to preroll.
ERROR: from element /GstPipeline:pipeline0/GstQueue:queue0: Internal data stream error.
Additional debug info:
../plugins/elements/gstqueue.c(1083): gst_queue_handle_sink_event (): /GstPipeline:pipeline0/GstQueue:queue0:
streaming stopped, reason not-negotiated (-4)
Details:
details, 

- [ ] flow-return=(int)-4;

ERROR: pipeline doesn't want to preroll.
Freeing pipeline ...

[mbaetgen-wup]

Hi

Thanks a lot for your work and to continue optimizing it.

I gave it a try again (My last attempt was August 2025) but I ran again into issues. Different ones than in summer so progress! :-) frontend-SDL2 runs OK

Intel(R) Core(TM) i5-7300U CPU

I'm running Debian SID (unstable): projectM ceb86ebca77e9d6b5ff033c78eb016467ea68380 (master) gst-launch-1.0 version 1.28.0 gcc 15 kernel 6.12 (still waiting until my Intel soundwire sound is fixed in ALSA with newer kernels) mesa 26.0.0 wayland-info.txt

$ rm ~/.cache/gstreamer-1.0/registry.x86_64.bin
$ gst-launch-1.0 -vv audiotestsrc ! queue ! audioconvert ! projectm ! video/x-raw,width=512,height=512,framerate=60/1 ! videoconvert ! xvimagesink sync=false
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
/GstPipeline:pipeline0/GstAudioTestSrc:audiotestsrc0.GstPad:src: caps = audio/x-raw, format=(string)S16LE, layout=(string)interleaved, rate=(int)44100, channels=(int)1, channel-mask=(bitmask)0x0000000000000003
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:sink: caps = audio/x-raw, format=(string)S16LE, layout=(string)interleaved, rate=(int)44100, channels=(int)1, channel-mask=(bitmask)0x0000000000000003
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:src: caps = audio/x-raw, format=(string)S16LE, layout=(string)interleaved, rate=(int)44100, channels=(int)1, channel-mask=(bitmask)0x0000000000000003
/GstPipeline:pipeline0/GstAudioConvert:audioconvert0.GstPad:src: caps = audio/x-raw, format=(string)S16LE, layout=(string)interleaved, rate=(int)44100, channels=(int)1, channel-mask=(bitmask)0x0000000000000003
/GstPipeline:pipeline0/GstProjectM:projectm0.GstPad:src: caps = video/x-raw, width=(int)512, height=(int)512, framerate=(fraction)60/1, format=(string)ABGR
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw, width=(int)512, height=(int)512, framerate=(fraction)60/1, format=(string)ABGR
/GstPipeline:pipeline0/GstVideoConvert:videoconvert0.GstPad:src: caps = video/x-raw, width=(int)512, height=(int)512, framerate=(fraction)60/1, format=(string)BGRx
/GstPipeline:pipeline0/GstXvImageSink:xvimagesink0.GstPad:sink: caps = video/x-raw, width=(int)512, height=(int)512, framerate=(fraction)60/1, format=(string)BGRx
/GstPipeline:pipeline0/GstVideoConvert:videoconvert0.GstPad:sink: caps = video/x-raw, width=(int)512, height=(int)512, framerate=(fraction)60/1, format=(string)ABGR
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw, width=(int)512, height=(int)512, framerate=(fraction)60/1, format=(string)ABGR
Got context from element 'projectm0': gst.gl.GLDisplay=context, gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayWayland\)\ gldisplaywayland0";
ERROR: from element /GstPipeline:pipeline0/GstProjectM:projectm0: Subclass failed to initialize.
Additional debug info:
/home/chrigi/git/gst-projectm/src/gstglbaseaudiovisualizer.c(490): gst_gl_base_audio_visualizer_find_gl_context_unlocked (): /GstPipeline:pipeline0/GstProjectM:projectm0
ERROR: pipeline doesn't want to preroll.
Setting pipeline to NULL ...
ERROR: from element /GstPipeline:pipeline0/GstProjectM:projectm0: Subclass failed to initialize.
Additional debug info:
/home/chrigi/git/gst-projectm/src/gstglbaseaudiovisualizer.c(490): gst_gl_base_audio_visualizer_find_gl_context_unlocked (): /GstPipeline:pipeline0/GstProjectM:projectm0
ERROR: pipeline doesn't want to preroll.
ERROR: from element /GstPipeline:pipeline0/GstProjectM:projectm0: Subclass failed to initialize.
Additional debug info:
/home/chrigi/git/gst-projectm/src/gstglbaseaudiovisualizer.c(490): gst_gl_base_audio_visualizer_find_gl_context_unlocked (): /GstPipeline:pipeline0/GstProjectM:projectm0
ERROR: pipeline doesn't want to preroll.
ERROR: from element /GstPipeline:pipeline0/GstProjectM:projectm0: Subclass failed to initialize.
Additional debug info:
/home/chrigi/git/gst-projectm/src/gstglbaseaudiovisualizer.c(490): gst_gl_base_audio_visualizer_find_gl_context_unlocked (): /GstPipeline:pipeline0/GstProjectM:projectm0
ERROR: pipeline doesn't want to preroll.
ERROR: from element /GstPipeline:pipeline0/GstProjectM:projectm0: Subclass failed to initialize.
Additional debug info:
/home/chrigi/git/gst-projectm/src/gstglbaseaudiovisualizer.c(490): gst_gl_base_audio_visualizer_find_gl_context_unlocked (): /GstPipeline:pipeline0/GstProjectM:projectm0
ERROR: pipeline doesn't want to preroll.
ERROR: from element /GstPipeline:pipeline0/GstAudioTestSrc:audiotestsrc0: Internal data stream error.
Additional debug info:
../libs/gst/base/gstbasesrc.c(3187): gst_base_src_loop (): /GstPipeline:pipeline0/GstAudioTestSrc:audiotestsrc0:
streaming stopped, reason not-negotiated (-4)
Details:
details, flow-return=(int)-4;
ERROR: pipeline doesn't want to preroll.
ERROR: from element /GstPipeline:pipeline0/GstQueue:queue0: Internal data stream error.
Additional debug info:
../plugins/elements/gstqueue.c(1083): gst_queue_handle_sink_event (): /GstPipeline:pipeline0/GstQueue:queue0:
streaming stopped, reason not-negotiated (-4)
Details:
details, 

- [ ] flow-return=(int)-4;

ERROR: pipeline doesn't want to preroll.
Freeing pipeline ...

I think the issue is related to the caps filter in your pipeline, the projectM gst plugin (from this branch) pushes GLMemory buffers:
gst-launch-1.0 -vv audiotestsrc ! queue ! audioconvert ! projectm ! 'video/x-raw(memory:GLMemory),width=512,height=512,framerate=60/1' ! glimagesink sync=false

or if you want to stick to xvimagesink, you'd need to add a gldownload to get the buffers into main memory:
gst-launch-1.0 -vv audiotestsrc ! queue ! audioconvert ! projectm ! gldownload ! videoconvert ! video/x-raw,width=512,height=512,framerate=60/1 ! xvimagesink sync=false

There are a couple examples in the README.md on this branch.