Project on plane with texture

[ttsesm]

Hi @marcomusy,

One question, I would like to use the project_on_plane() method in order to project a mesh on a 2D plane but I would like to understand if I can do that with a mesh loaded with texture so that I can keep the texture of the projected surface. Is that possible?

[marcomusy]

Hi, yes I don't see why that should not be possible!

[ttsesm]

ok, it seems to work but for some reason the texture on the projection looks faded... 🤔

Original mesh:

Projected to the z plane:

Any idea, why?

code:

meshes = []
meshes_projected = []
for mesh_file, texture_file in tqdm(zip(mesh_files, texture_files)):
    m = vd.Mesh(mesh_file).texture(texture_file).lighting(style='ambient')
    mx = m.clone().project_on_plane("z")
    meshes.append(m)
    meshes_projected.append(mx)

vd.show(meshes, axes=8, interactive=True).close()
vd.show(meshes_projected, axes=8, interactive=True).close()

[ttsesm]

ok, I found why... in the code of the project_on_plane() you have set the self.alpha(0.1) if I comment this line I am getting the right colors:

Maybe it would be a good idea to pass the transparency as a parameter.

Finally another question, how I can save the projected plane as an image... 🤔
I see there is the vedo.Image() but how to pass the projected mesh to it?

[ttsesm]

Ok, solution for this found as well with the following code:

    # Make a screenshot
    settings.screenshot_transparent_background = True
    x0, x1 = meshes_projected[0].xbounds()
    y0, y1 = meshes_projected[0].ybounds()
    size = ((x1 - x0), (y1 - y0))
    plt = vd.show(meshes_projected[0], size=(1000, 1000), offscreen=True)
    plt.screenshot("img.png")

[ttsesm]

Now I have the following question, if I have multiple meshes:

Left: 3D, Right: projected 2D

which then I am saving individually:

How I can then load the saved images and put them in the same position as before I've saved them individually?

btw, I am saving them both with a size of 1000x1000 and transparent background and I would expect the smaller piece to occupy less pixels in the canvas proportionally to how they look in the 3D space, but I see that they seem of the same size... 🤔

[ttsesm]

@marcomusy ok this is the closest I could get so far (which is almost there):

The problem I think is related to the padding that automatically is added when you map/fit the mesh to a 1000x1000 image. Which I am not sure how to handle correctly in the post-processing, though what I am doing should be working fine. Any idea what I am missing... 🤔

Here is the code as well:

import vedo as vd
from vedo import settings
import numpy as np
import natsort
from wcmatch import glob
from tqdm import tqdm
import copy

import vedo.vtkclasses as vtki


def main():
    folder = '/media/pose_est_rp/repair_gt/3D_Fragments/assembled_objects/RPobj_g7_o0116/'
    mesh_files = natsort.natsorted(glob.glob(folder + '**/*.obj', flags=glob.BRACE | glob.GLOBSTAR))
    texture_files = natsort.natsorted(glob.glob(folder + '**/*.png', flags=glob.BRACE | glob.GLOBSTAR))

    # Load and project meshes
    meshes = []
    meshes_projected = []

    for mesh_file, texture_file in tqdm(zip(mesh_files, texture_files)):
        m = vd.Mesh(mesh_file).texture(texture_file).lighting(style='off').force_opaque()
        mx = m.clone().project_on_plane("z")
        meshes.append(m)
        meshes_projected.append(mx)

    # vd.show(meshes_projected, zoom='tightest', axes=1, interactive=True).close()

    # Create a single renderer/plotter to render all meshes
    settings.screenshot_transparent_background = True
    plotter = vd.Plotter(offscreen=True, size=(1000, 1000))

    # Create a list to store mesh images and their positions
    mesh_images = []

    # Process each mesh separately to create individual images
    for i, mesh in enumerate(meshes_projected):
        # Clear plotter and add only this mesh
        plotter.clear()
        plotter.add(mesh)

        bounds = mesh.bounds()
        x_range = bounds[1] - bounds[0]
        y_range = bounds[3] - bounds[2]

        # Scale factor to fit in 1000x1000 canvas
        max_dimension = max(x_range, y_range)
        scale_factor = 1000 / max_dimension
        scale_factor_x = 1000 / x_range
        scale_factor_y = 1000 / y_range
        # scale_factor = 1000 / ((x_range + y_range) / 2)
        # scale_factor = (scale_factor_x + scale_factor_y) / 2

        # # Set camera to parallel projection for consistent scale
        # plotter.camera.SetParallelProjection(True)
        # plotter.camera.SetParallelScale(max_dimension)
        # plotter.camera.SetPosition(0, 0, 1000)
        # plotter.camera.SetFocalPoint(0, 0, 0)
        # plotter.camera.SetViewUp(0, 1, 0)

        # Calculate mesh position in the unified coordinate system
        center = mesh.center_of_mass()
        # pos_x = (center[0] - bounds[0])# * scale_factor
        # pos_y = (center[1] - bounds[2])# * scale_factor
        pos_x = center[0]
        pos_y = center[1]

        # Render the mesh
        plotter.show(zoom='tightest', interactive=False)

        # Get image data directly from the renderer
        # window_size = plotter.window_size
        rgba_data = plotter.screenshot(asarray=True)
        rgba_data = np.ascontiguousarray(rgba_data)

        # Create a vedo Image from the numpy array
        mesh_image = vd.Image(rgba_data, channels=4)

        # Store the image and its position in the unified space
        mesh_images.append({
            'image': mesh_image,
            'position': (pos_x, pos_y),
            'scale_factor': scale_factor,
            'scale_factor_x': scale_factor_x,
            'scale_factor_y': scale_factor_y,
            # 'dimensions': (min_x, max_x, min_y, max_y)
        })

        # Save the individual image
        mesh_image.write(f"mesh_image_{i}.png")


    # From mapped images back to original
    imgs = []
    for mesh_info in mesh_images:
        # Get the mesh image and extract non-transparent region
        img = mesh_info['image']
        img_array = img.tonumpy()  # Convert vedo Image to numpy array

        # Find non-transparent pixels
        non_transparent = np.where(img_array[:, :, 3] > 0)
        if len(non_transparent[0]) == 0:
            continue  # Skip empty images

        # Extract the bounding box of the non-transparent region
        min_y, max_y = np.min(non_transparent[0]), np.max(non_transparent[0])
        min_x, max_x = np.min(non_transparent[1]), np.max(non_transparent[1])


        # Calculate position for placement
        pos_x, pos_y = mesh_info['position']
        place_x = pos_x - ((max_x - min_x) / 2) / mesh_info['scale_factor_x']
        place_y = pos_y - ((max_y - min_y) / 2) / mesh_info['scale_factor_y']

        LT = vd.LinearTransform()
        # transform = vtki.vtkTransform()
        # transform.Scale(1 / mesh_info['scale_factor_x'], 1 / mesh_info['scale_factor_y'], 1)
        # transform.Translate(place_x, place_y, 0)
        img.scale(1/mesh_info['scale_factor']).apply_transform(LT.translate([place_x, place_y, 0]))
        # img.apply_transform(transform).apply_transform(LT.translate([place_x, place_y, 0]))

        imgs.append(img)

    # Display the unified canvas
    # vd.show(imgs[0], imgs[1].apply_transform(LT.translate([-42, 73, 0])), zoom='tight', axes=1, interactive=True).close()
    vd.show([meshes_projected, imgs,], N=2, zoom='tightest', axes=1, interactive=True).close()

    return 0


if __name__ == "__main__":
    main()

Also in the following link you can find the files: https://filesender.garr.it/?s=download&token=b0832b71-376e-4728-bfbc-49afc472fd03

[marcomusy]

Hi @ttsesm
here is a possible solution:

import natsort
import glob
import vedo

folder = "data/ttsem_files/"
mesh_files = natsort.natsorted(glob.glob(folder + "*.obj"))
texture_files = natsort.natsorted(glob.glob(folder + "*.png"))
print(f"Found {(mesh_files)} meshes and\n{(texture_files)} textures.")

meshes_projected = []
x0m, x1m, y0m, y1m = (1e10, 0, 1e10, 0)
for mesh_file, texture_file in vedo.progressbar(zip(mesh_files, texture_files)):
    m = vedo.Mesh(mesh_file).texture(texture_file).lighting("off")
    mx = m.project_on_plane("z").alpha(1)
    meshes_projected.append(mx)
    x0, x1, y0, y1, _, _ = m.bounds()
    x0m = min(x0m, x0)
    x1m = max(x1m, x1)
    y0m = min(y0m, y0)
    y1m = max(y1m, y1)

box = vedo.Box([x0m, x1m, y0m, y1m, 0, 0]).alpha(0)
sizex = 800
sizey = (y1m - y0m) * sizex / (x1m - x0m)

vedo.settings.screenshot_transparent_background = True
vedo.settings.use_parallel_projection = True

plotter = vedo.Plotter(offscreen=True, size=(sizex, sizey))
plotter.show([box, meshes_projected[0]], zoom="tightest")
plotter.screenshot("mesh0.png")
plotter.remove(meshes_projected[0]).add(meshes_projected[1]).render()
plotter.screenshot("mesh1.png")
plotter.close()
# check if the screenshots are correct
plt = vedo.Plotter(size=(sizex, sizey))
plt.show(vedo.Image("mesh0.png", channels=4), 
         vedo.Image("mesh1.png", channels=4), zoom="tightest").close()

note that you can do m.project_on_plane("z").alpha(1) to modify the transparency.

[ttsesm]

Thanks @marcomusy, as usual really helpful ❤️

Your solution is almost perfect, two additional tweaks though if possible:

1. Is there a direct way to go back to the original coordinate system? Because now the loaded screenshot images are in the provided resolution and I would like optionally to go back in the original coordinates.
   

2. When I project the 3D meshes part of the side/bottom parts (indicated with the red marks in the above image) might be projected as part of the top surface which ideally I would like to avoid. Thus, is there a way to segment only the top surface of the meshes and then project them... 🤔
   I've tried to play a bit with the boundary or the curvature so that I can distinguish the top surface from the rest but I didn't manage to get anything working properly.

Thanks for the hint on the transparency 👍

[marcomusy]

Hi

1. No.. Those are 2 different worlds: pixels vs cartesian coordinates, you need to write your own transformation. You can use LinearTransform().write("mesh1.mat") and then load it back and invert it.
2. you can use .cut_with_plane(normal=[0,0,1]) to get rid of the lower part and/or extract normals (array = mesh.vertex_normals.copy()) and cut off the vertices that point downwards, add it to the mesh with mesh.pointdata["downwards_z"]=array[:,2], then use cut_with_scalar("downwards_z", 0).

..just ideas :)

[ttsesm]

Thanks @marcomusy,

For the 1. here is the code for the transformation that works:

x_range = x1m - x0m
y_range = y1m - y0m

# Scale factor to fit in sizex/sizey canvas
max_dimension = max(x_range, y_range)
scale_factor = sizex / max_dimension

# Create a LinearTransform object for mapping pixels to cartesian coordinates
tform = vedo.LinearTransform()
# Inverse the scale factor to map back to original coordinates
tform.scale(1/scale_factor)

# Calculate position for placement
pos_x, pos_y, _ = box.center_of_mass()
place_x = pos_x - (sizex / 2) / scale_factor
place_y = pos_y - (sizey / 2) / scale_factor
tform.translate([place_x, place_y, 0])

As for the 2. I'll check your suggestions.

[ttsesm]

@marcomusy one question for something that I've noticed, when you clone()/copy() an image how do you maintain the transparency?

For example, if I do the following:

for i, image_file in vedo.progressbar(enumerate(image_files)):
    img = vedo.Image(image_file, channels=4)
    img_transformed = img.clone().apply_transform(tform)

The fourth channel is lost, since if I check in the clone() method I see that you create a vedo.Image() ignoring the fourth channel, so transparency is lost. I've tried to use .alpha(1) but didn't work.

[marcomusy]

i think it's a bug! i'll fix it

[ttsesm]

2. you can use .cut_with_plane(normal=[0,0,1]) to get rid of the lower part and/or extract normals (array = mesh.vertex_normals.copy()) and cut off the vertices that point downwards, add it to the mesh with mesh.pointdata["downwards_z"]=array[:,2], then use cut_with_scalar("downwards_z", 0).

Ciao @marcomusy,

I got the following solution for this task:

def set_cutplane(m, tol_angle=30):
    # Ensure normals are available
    m.compute_normals()

    # Reference normal and tolerance
    ref_normal = np.array([0, 0, 1])
    tolerance_deg = tol_angle
    tolerance_rad = np.deg2rad(tolerance_deg)

    # Get normals and compute angles
    normals = m.cell_normals
    dot_products = normals @ ref_normal
    dot_products = np.clip(dot_products, -1.0, 1.0)
    angles = np.arccos(dot_products)

    # Mask points whose normals are within the angular tolerance
    mask = angles > tolerance_rad
    ids = np.where(mask)[0]
    m.delete_cells(ids)

m = vedo.Mesh(mesh_file).texture(texture_file).lighting("off")
m = set_cutplane(m)

Where I just threshold all faces with a normal tolerance angle of 30 degrees to a reference normal [0, 0, 1] and which works quite nicely as you can see bellow:

However, I have two issues: 1. is some remaining noise pieces which in principle are further away from the main surface remaining mesh (marked with red) and 2. whether I could somehow smooth the edges of the remaining surface, because as you can see by deleting the faces leaves this saw effect on the edges. I've tried to use some providing methods like clean() or smooth() but these have the effect shown here related to the texture.

So, I am trying to figure out whether there is anything that I can do to address these issues without too much hassle. In principle, the smoothing of the edges possibly it can be done also when I create an vedo.Image() if there is any method for it in order to avoid the issue with the texture. Any idea... 🤔

Increasing the tolerance angle m = set_cutplane(m, 50) improves the smoothness on the edges, but I still get some isolated faces here and there: