Train UR5e with robotiq gripper

[gtziafas]

Hey guys, thanks for sharing your amazing work!

I was wondering, do you have any suggestions for training with the mujoco_menagerie/universal_robots_ur5e_robotiq?

I'm using the default data generation and training hyper-params as given in the repo for UR5e, and observe a blurry behavior at the gripper region

I've also noticed that the total number of joints here compared to the 6 of just the UR5e arm are 14:

# kinematic_chain.get_joint_parameter_names()
['ur5e_0_shoulder_pan_joint', 'ur5e_0_shoulder_lift_joint', 'ur5e_0_elbow_joint', 'ur5e_0_wrist_1_joint', 'ur5e_0_wrist_2_joint', 'ur5e_0_wrist_3_joint', 'right_driver_joint', 'right_coupler_joint', 'right_spring_link_joint', 'right_follower_joint', 'left_driver_joint', 'left_coupler_joint', 'left_spring_link_joint', 'left_follower_joint']

which means 8 just for the gripper. I guess that makes sense when considering the mesh files for the different parts, but intuitively I would expect just one degree of freedom that determines gripper closure, similar to how we would control the EE.

Is this maybe the issue of blurry gripper rendering? Or it's not really relevant?

Any tips / suggestions for modifying the data-gen script, training params, robot XMLs etc. would be very much appreciated :)

[gtziafas]

Update Managed to get cleaner gripper renders. The issue was mostly the sampled camera params of the current data generation script, which left the gripper out for big percentage of the dataset.

I modified the script to dynamically compute the position of the gripper using forward kinematics and make sure that the lookat of the sampled cameras is there:

In generate_robot_data.py:

[...]
self.chain = build_chain_from_mjcf_path(self.model_xml_path)
self.ee_link_name = 'robotiq_ee' # for UR5e-robotiq
[...]

def find_ee_pos_fk(self, joint_position):
  transforms = self.chain.forward_kinematics(joint_position)
  return transforms[self.ee_link_name].get_matrix()[0,:3,3]

ee_pos = self.find_ee_pos_fk(joint_position)
images, depth_images, intrinsic_matrices, extrinsic_matrices = self.render_images(joint_position, 
                                                                                camera_params,
                                                                                render_depth=True, lookat=ee_pos)

Also seems that the robot is a bit bigger in mujoco_menagerie/universal_robots_ur5e_robotiq so used args.camera_distance_factor=1.25.

PS I would really appreciate if you can still provide some context for the DoF control problem. In your paper, looking at the text2video results section, you illustrate the control vector with just 1 DoF for gripper closure. I assumed you could optimize w.r.t to that DoF instead of the 8 of the xml. How could someone replicate that ?

[alpercanberk]

This is really helpful! I believe that a few others had a similar issue with the Franka gripper. As for the gripper DoF, I actually don't remember, maybe @ruoshiliu knows? The training code assumes that each joint transforms rigidly and independently so I think it's nontrivial to put gripper opening/closure into a single DoF.

We apologize for the inconvenience as this codebase is a cleaner rewrite of our original codebase, which had lots of troublesome dependencies and spaghetti code. In the process of this, some hyperparams changed, resulting in the gripper-related issues.

[Lexbesk]

For the franka gripper, I tried this method but the actors are constantly getting killed and the data generation is very slow when we try to get the position of the gripper. I was unable to solve the problem, so in the end I instead fixed looking at [0, 0, 0.3] for franka, under the same idea here. Also, I increased the distance to 1.9 (a result of my experiments) according to the #8 , and to avoid the distance being too far, instead of using [1.0, 2.0] two distances, I used [1.0, 1.3] (The offset of distance is set to [-0.1, 0.1]. Finally, with this setting, the loss curve is much better and the gripper of franka appeared. However, the gripper is disconnected to the rest of the robot. To fix that, I constructed another smaller (5000) dataset using look at [0, 0, 0.4], trying to capture the wrist better. Also, I used 600 samples with look at pointing to the gripper. After "fine tuning" the former 12000 step checkpoint on this new dataset for 40000 steps, the gripper starts to connect with the body. Then I was confident and trained another 40000 steps on this dataset, finally the robot looks good, phew!