Adding a 2D mpl figure to a 3D scene as an overlay or HuD (heads up display)

[OwenDavid]

Hi all,

Can we render a flat mpl figure to the same window as our mesh - but prevent it from being manipulated?

In much the same way the slider or 2Dtext sits on top of the mesh window and has a fixed position. How can we do the same with a mpl figure?

Ideally we would like to add transparency so that mpl figure would appear to be floating on top and not pasted as an image.

The use case would then be to update the mpl figure as the mesh is updated by a callback_timer. Giving the impression of an animated timeseries plot.

I have scoured the examples but cannot seem to identify the correct approach.

What do you think?

[marcomusy]

Yes, check out example
examples/pyplot/embed_matplotlib.py

[OwenDavid]

Thanks for the quick reply! Cool project you have here!

Is it possible to access the object that the mpl figure is rendered into and set the background alpha to 0 - i.e. to make it transparent?

I've tried modifying the mpl fig and ax alpha with no change.

What do you think?

"""Include background images in the rendering scene
(generated by matplotlib)"""
import matplotlib.pyplot as plt
from vedo import *

tmsh = TetMesh(dataurl + "limb.vtu")
msh = tmsh.tomesh().shrink(0.8)

# Create a histogram with matplotlib
fig, ax = plt.subplots()
fig.patch.set_alpha(0.0)
ax.patch.set_alpha(0.0)

show(fig, msh, __doc__, bg="lightgrey", zoom=1.2, axes=1)

[marcomusy]

Thanks Owen! Consider the following option:

import matplotlib.pyplot as plt
from vedo import *

tmsh = TetMesh(dataurl + "limb.vtu")
msh = tmsh.tomesh().shrink(0.8)

# Create a histogram with matplotlib
fig, ax = plt.subplots()
ax.hist(msh.points[:, 2],  color='blue')
img = Image(fig).clone2d("top-right", size=0.4).ontop().alpha(0.75)

show(img, msh, bg="lightgrey", zoom=1.2, axes=1)

[OwenDavid]

I found this example which represents exactly what we are trying to achieve.

Is the same possible with vedo at the moment or would we need to reverse engineer the underlying VTK methods to recreate this in a vedo based window?

https://docs.pyvista.org/examples/02-plot/chart_overlays#chart-overlays-example

[marcomusy]

Sure, what about this:

from vedo import Plotter, TetMesh, Image, dataurl
import matplotlib.pyplot as plt

def func(w, event):
    ax.clear()
    ax.hist(data[data>slider.value], color="blue", alpha=0.7)
    ax.set_xlabel("Some label")
    img = Image(fig).clone2d("top-right", size=0.4).ontop().alpha(0.75)
    img.name = "myhistogram"
    vplt.remove("myhistogram").add(img)

fig, ax = plt.subplots()
fig.tight_layout()
# Load a mesh from vedo's data repository
msh = TetMesh(dataurl + "limb.vtu").tomesh().shrink(0.8)
data = msh.points[:, 2]

vplt = Plotter(bg="lightgrey", size=(800, 800), axes=8)
slider = vplt.add_slider(
    func,
    -800,
    +500,
    value=0,
    title="slider",
    pos=1,
    # delayed=True, # update only when the slider is released
)
img = Image(fig).clone2d("top-right", size=0.4).ontop().alpha(0.75)
img.name = "myhistogram"
vplt.show(msh, img)

[OwenDavid]

Hey thanks Marco!

These are all good suggestions. I am new to this domain and the VEDO and VTK libraries so thanks for your sense of humor!

If we take your last example as a reference we see the mpl hist has a white background - with alpha.

The real question is how can we remove the white background from the mpl plot so the plot would have the blue bars and black axes./text on the same grey background as the slider and mesh.

I suppose essentially - we would like to insert a 2D overlay of an mpl plot which looks similar to the mesh but is locked away from interaction.

I hope the rough copy and paste below makes our intentions clear...

Is there a way to achieve this?

[marcomusy]

Hi Owen, two options:

For the second one (func2) you will need to use the development version:
pip install -U git+https://github.com/marcomusy/vedo.git

from vedo import Plotter, Sphere, Image, settings
from vedo.pyplot import histogram


def func1(w, evt=""):
    d = data[data > w.value]
    if len(d) == 0: return
    hi = histogram(
        d, 
        c="blue4",
        alpha=0.7,
        aspect=16/9,
        title="Histogram",
        xtitle='measured variable',
        ytitle='frequency',    
        label='my histogram',
        mc='red5',
    )
    hi.add_legend(s=1.2, alpha=0.4)
    hi = hi.clone2d("top-right", size=0.7, ontop=False)
    hi.name = "myhistogram"
    vplt.remove("myhistogram").add(hi)


# import matplotlib.pyplot as plt
# plt.rcParams.update({
#     "figure.facecolor":  (1.0, 1.0, 1.0, 0.0),  # white with alpha = 0%
#     "axes.facecolor":    (1.0, 1.0, 1.0, 0.0),  # white with alpha = 0%
#     "savefig.facecolor": (1.0, 1.0, 1.0, 0.0),  # white with alpha = 0%
# })
# fig, ax = plt.subplots()
# fig.tight_layout()
# def func2(w, evt=""):
#     ax.clear()
#     ax.hist(data[data>slider.value], color="blue", alpha=0.7)
#     ax.set_xlabel("Some label")
#     img = Image(fig, channels=4).clone2d("top-right", size=0.4).ontop()
#     img.name = "myhistogram"
#     vplt.remove("myhistogram").add(img)


#############################################################
settings.default_font = "Roboto"

msh = Sphere(res=100).ps(3).c("blue5")
data = msh.points[:, 2]
vplt = Plotter(bg="w", bg2="green9", size=(800, 800), axes=8)
slider = vplt.add_slider(
    func1,
    -1,
    +1,
    value=0,
    title="slider",
    pos=1,
    # delayed=True, # update only when the slider is released
)
vplt.show(msh)

[OwenDavid]

wow! that changes everything!

Thanks Marco

[OwenDavid]

Hi Marco,

Any thoughts on why this approach would not work in the same way for an XY plot?

In this code below the position and size of the XY plot is impacted by the slider position. You may have to zoom out to see it. Could it be due to the changing X axes limits? Perhaps the figure elements are plotted at the same scale as the mesh?

We also find that the xy plot is not overlaid in the same way as the histogram which remains at the same size and position irrespective of zoom, pan, etc..

Is there a way to standardise this so that any mpl figure could be handled in the same way? Or is there a lot of complexity behind this?

What do you think?

from vedo import Plotter, Sphere, Image, settings
from vedo.pyplot import histogram, plot


def func1(w, evt=""):
    d = data[data > w.value]
    if len(d) == 0: return
    
    xy = plot(
        d,
        c="blue4",
        alpha=0.7,
        aspect=16/9,
        title="xy",
        xtitle='point number',
        ytitle='measured variable',
    )
    
    xy.clone2d("top-right", size=0.7, ontop=False)
    xy.name = "myxy"
    vplt.remove("myxy").add(xy)

    hi = histogram(
        d, 
        c="blue4",
        alpha=0.7,
        aspect=16/9,
        title="Histogram",
        xtitle='measured variable',
        ytitle='frequency',    
        label='my histogram',
        mc='red5',
    )
    hi.add_legend(s=1.2, alpha=0.4)
    hi = hi.clone2d("bottom-right", size=0.7, ontop=False)
    hi.name = "myhistogram"
    vplt.remove("myhistogram").add(hi)


# import matplotlib.pyplot as plt
# plt.rcParams.update({
#     "figure.facecolor":  (1.0, 1.0, 1.0, 0.0),  # white with alpha = 0%
#     "axes.facecolor":    (1.0, 1.0, 1.0, 0.0),  # white with alpha = 0%
#     "savefig.facecolor": (1.0, 1.0, 1.0, 0.0),  # white with alpha = 0%
# })
# fig, ax = plt.subplots()
# fig.tight_layout()
# def func2(w, evt=""):
#     ax.clear()
#     ax.hist(data[data>slider.value], color="blue", alpha=0.7)
#     ax.set_xlabel("Some label")
#     img = Image(fig, channels=4).clone2d("top-right", size=0.4).ontop()
#     img.name = "myhistogram"
#     vplt.remove("myhistogram").add(img)


#############################################################
settings.default_font = "Roboto"

msh = Sphere(res=10).ps(3).c("blue5")
data = msh.points[:, 2]
vplt = Plotter(bg="w", bg2="green9", size=(800, 800), axes=8)
slider = vplt.add_slider(
    func1,
    -1,
    +1,
    value=0,
    title="slider",
    pos=1,
    delayed=True, # update only when the slider is released
)
vplt.show(msh)

[marcomusy]

You forgot to assign the xy variable:
xy = xy.clone2d("top-right", size=0.7, ontop=False)

The full code:

from vedo import Plotter, Sphere, settings
from vedo.pyplot import histogram, plot


def func1(w, evt=""):
    d = data[data > w.value]
    if len(d) == 0:
        return

    xy = plot(
        d,
        c="blue4",
        alpha=0.7,
        aspect=16 / 9,
        title="xy",
        xtitle="point number",
        ytitle="measured variable",
    )
    xy = xy.clone2d("top-right", size=0.7, ontop=False)
    xy.name = "myplots"

    hi = histogram(
        d,
        c="blue4",
        alpha=0.7,
        aspect=16 / 9,
        title="Histogram",
        xtitle="measured variable",
        ytitle="frequency",
        label="my histogram",
        mc="red5",
    )
    hi.add_legend(s=1.2, alpha=0.4)
    hi = hi.clone2d("bottom-right", size=0.7, ontop=False)
    hi.name = "myplots"
    vplt.remove("myplots").add(xy, hi)


#############################################################
settings.default_font = "Roboto"

msh = Sphere(res=10).ps(3).c("blue5")
data = msh.points[:, 2]
vplt = Plotter(bg="w", bg2="green9", size=(800, 800), axes=8)
slider = vplt.add_slider(
    func1,
    -1,
    +1,
    value=0,
    title="slider",
    pos=1,
    delayed=True,  # update only when the slider is released
)
vplt.show(msh)

[OwenDavid]

Good catch! Thanks

I'll be sure to pay it forward when I can.

The final result is excellent visually and to my eye represents a great example of what VEDO can achieve.

Would you consider adding this as an example and/or as a method in the API? Perhaps overaly=True or hud_mode = on ?

[marcomusy]

Yes :) I think it deserves to become one reference example in the example/pyplot directory!