Visualize any robot

[amjack0]

Minimal PyRoki example to visualize any robot from the list of robot descriptions.

[chungmin99]

Hi!

Could you provide a bit more context on the motivation behind this PR? At first glance, it looks quite similar to the original ViserUrdf example in viser.

It would be great to go beyond just loading the robot from robot-descriptions -- for example, by also supporting:

• loading your own URDF from a provided path, and
• doing inverse kinematics with chosen end-effector(s).

[amjack0]

The goal is to start with a simple, minimal example (pyroki/examples) that can load various robot descriptions.
This allows users to begin with a robot of their choice and gradually build applications on top of it, enabling a flexible and scalable learning path.

About the suggestions you made

loading your own URDF from a provided path, and
doing inverse kinematics with chosen end-effector(s).

I'm happy to contribute this in a separate PR.

[chungmin99]

This allows users to begin with a robot of their choice and gradually build applications on top of it, enabling a flexible and scalable learning path.

I completely agree -- right now the examples don’t really show how users can load in their own robots, and that feels like an important gap.

I'm happy to contribute this in a separate PR.

Great to hear that you’re open to contributing more!

That said, I do feel pretty strongly that these features should be demonstrated together in a single example (ideally in a single PR), since:

1. loading a custom URDF seems core to the idea of showing "any robot," and
2. doing IK -- not just visualization -- feels essential for demonstrating how users can actually use custom robot models with PyRoki.

[amjack0]

I made the necessary code changes, and now I can load custom URDFs. I’ve tested it with a custom arm, legged robot, and humanoid — all work fine, though some improvements are still needed.

Working with inverse kinematics (IK) for custom robot models is a bit tricky, as there are multiple ways to solve IK, such as using solve_ik_with_multiple_targets and solve_ik.

Additionally, base_link and target_link may differ depending on the custom URDF. Currently, I'm able to read base_link = urdf_model.base_link and target_link = all_links[-1], which allows me to solve IK for custom URDFs.

However, it would make more sense to let the user specify the target_link rather than defaulting to the last link in the URDF.

And I am not sure how to define position and wxyz that works for all the custom URDFs.
ik_target = server.scene.add_transform_controls("/ik_target", scale=0.2, position=(0.61, 0.0, 0.56), wxyz=(0, 0, 1, 0))

Aufzeichnung_Custom_URDF_IK.mp4

Aufzeichnung_fanuc_m710ic_IK.mp4

Aufzeichnung_atlas_drc_IK.mp4

[chungmin99]

I made the necessary code changes, and now I can load custom URDFs. I’ve tested it with a custom arm, legged robot, and humanoid — all work fine, though some improvements are still needed.

Yea the videos look great! 😄

Working with inverse kinematics (IK) for custom robot models is a bit tricky, as there are multiple ways to solve IK, such as using solve_ik_with_multiple_targets and solve_ik.

IIRC these two functions are nearly identical -- maybe we can use solve_ik_with_multiple_targets, to support multiple link targets?

Additionally, base_link and target_link may differ depending on the custom URDF. Currently, I'm able to read base_link = urdf_model.base_link and target_link = all_links[-1], which allows me to solve IK for custom URDFs.
However, it would make more sense to let the user specify the target_link rather than defaulting to the last link in the URDF.

I think we can allow users to specify + add new target links, through a snippit like this below:

# server = viser.ViserServer()
target_link_handle_list = []  # for target link names
target_link_pose_handle_list = []  # for target poses for each link

# Add a handle for creating multiple link pose targets.
add_target_link_button = server.gui.add_button(...)
@add_target_link_button.on_click
def _(_):
    _link_dropdown = server.gui.add_dropdown(...)  # with `all_links` as options
    _link_pose = server.scene.add_transform_controls(...)
    target_link_handle_list.append(_link_dropdown)
    target_link_pose_handle_list.append(_link_pose)

And I am not sure how to define position and wxyz that works for all the custom URDFs.
ik_target = server.scene.add_transform_controls("/ik_target", scale=0.2, position=(0.61, 0.0, 0.56), wxyz=(0, 0, 1, 0))

Re; this, a reasonable default might be to use the starting pose (by running the FK at the current joint configuration).

[amjack0]

1. Meanwhile, I did the changes for allow users to specify the target_links. Now we can specify the target_link. What's the big picture or idea behind add new target links ?

2. So now, we use the local path for a custom URDF if provided; otherwise, the built-in robot description is loaded. I am reading the base_link from the URDF as: all_links = [l for l in urdf_model.link_map.keys() if l not in ("world", "base")]
   Here, I ignore links such as world and base so that the base_link refers to the actual physical link of the robot. Feel free to add more non-physical links. (As I tested, Universal Robot has all_links[0] as world/base, and IK did not work in that case.)

3. A good suggestion! I also tried implementing FK for Re; this, a reasonable default might be to use the starting pose (by running the FK at the current joint configuration). I still need to fix the code to make it work.

[amjack0]

I’ve implemented the suggested changes (except add new target links).

The code has been tested with various URDFs, including those using .dae and .stl files —works well. I’ve also added a short README for the script, outlining current limitations.

Feel free to take a look and let me know your feedback!

[amjack0]

Aufzeichnung_spot_IK.mp4