Yann LeCun teaches world models to review: Is a robot's failure due to wrong action choices, or just bad luck?
Beating AI News Flash: AMI Labs, founded by Yann LeCun, in collaboration with researchers from Brown University, has published a new paper titled "Counterfactual World Models," enabling the JEPA world model to infer what would have happened if a different action had been taken, based on the actual outcome of one action. This capability is called "counterfactual reasoning."
For example, a boat sailing east is actually pushed off course by 2 meters due to an undercurrent. Although the model does not know how strong the undercurrent is, it can compare the predicted outcomes of sailing east versus sailing west. By adding the difference between the two predictions to the boat's actual arrival position, it can estimate where the boat would have been if it had sailed west. The entire process involves only one real observation of sailing east; the westward result is entirely inferred by the model.
In four physics simulation environments, the team used an additionally trained transformation module for evaluation, and the R² of counterfactual predictions improved from 38%–46% to 86%–96%. In another post-hoc planning test, the proportion of cases where the model found actions that could originally have achieved the goal reached 35%–69%, compared with only 14%–23% for baseline methods.
This method has a key prerequisite: the effects of actions and external disturbances can be roughly superimposed. If objects collide, or motor failures cause different commands to be affected differently, predictions may become inaccurate. Currently, the research is still limited to simulation environments and has not yet verified actual control performance on real robots.