Need to verify that VLM provided a label for every atom it's supposed to label #1724
Description
Observed in demo 0 of combo_burger in seed 0 when asking to label 12 training demos:
uncut(lettuce1) is proposed and labeled as True in the initial state, but is just not labeled again the rest of the demo by the VLM. This doesn't happen to be a problem because this predicate is not relevant to anything going on in this particular demo (which might be why the VLM forgets to label it), but we might want to verify that the VLM provides a label for every atom in every state of the demo trajectory. This is not important for predicates that are False in the initial state and remain False, because the way we convert the VLM labeling to a ground atom trajectory preserves that truth label -- but it can be argued that it is a problem for atoms that are True in the initial state. Technically speaking it's fair to set an assumption that if the VLM fails to provide a label, we assume it to be False, but it's better to verify we get a label for every atom at every state.
Two ideas for a fix:
- Ask the VLM to check that it labeled all atoms in the "double check" prompt we have added.
- Relabel (via reprompting) the initial state as many times as we need to until we get a label for every atom, and if future states are missing a label for any atom, when we convert the text response from the VLM into a ground atom abstract state, use previous labels for any atoms that are missing from the labeling at the current state.