What do we look at to verify the convergence of ASE's LLP?

[jiaweipeng0]

In ASE training on locomotion datasets, I observe Disc_Agent_Acc staying at ~0.95-0.99 even when visual rollouts are clean and episode length is at cap. Is this expected behavior given that the policy optimizes the MI objective rather than trying to fool the discriminator?

I trained an ASE model on just four motions: walk, run, jog, and jump. Here is the result and log file. Also, in the visualization, the jump motion will always be interrupted by other locomotion skills. Can I call this situation Mode Collapse?

Thank you for your help!!!

log.txt

[jiaweipeng0]

Sorry I forgot to mention that I'm using the default hyperparameter setting.

[xbpeng]

It is tricky to determine if methods like ASE has converged, since it is not optimizing a fixed objective, the intrinsic rewards are constantly changing. So often times, even when you don't see much improvements in the rewards or losses, the model could still be improving. The best way is unfortunately to visualize the simulation over time and see if there are noticeable improvements in the behaviors of the model.

It is expected that the discriminator accuracy will be very high, this is normal.

What you are observing could be considered a form of mode collapse. Motions with large flight phases, such as jumping, can pose a difficult exploration problem. So it can be harder for ASE to discover those skills. You can try increasing the sampling weight of the jumping motion in the dataset file:

MimicKit/data/datasets/dataset_humanoid_locomotion.yaml

Line 6 in aeb7de9

weight: 1.0

This might help to encourage the model to replicate that skill.

[jiaweipeng0]

Thank you so much for the detailed answer!! That's very helpful.