DeFM-WMPFrom terrain perception to quadruped locomotion.

One policy across varied terrain

DeFM-WMP combines DeFM depth representations with the recurrent WMP world model and trains a unified quadruped locomotion policy across an expanded terrain curriculum. The robot adapts its actions from depth observations, motion history and velocity commands without switching controllers for slopes, stairs, bridges or hurdles. Across the 16 level-8 simulated terrains shown here, the system completes continuous traversals, demonstrating one-policy locomotion across varied geometry.

01 / Watch the terrain traversals

Sixteen continuous simulation clips form a 4 × 4 terrain showcase covering slopes, stairs, stepping stones, bridges, planters, climbing and fences. Each preserves 12 seconds of motion at level 8 with a forward command, zero resets and at least six metres of forward displacement.

02 / From depth observations to joint control

The system couples recurrent state estimation with a history-conditioned controller. The world model maintains memory from depth and proprioception; the actor combines that memory with recent observations and velocity commands. RSSM is an intermediate representation, followed by feature encoding, an action policy, and low-level control.

A spatial readout for control

The DeFM visual front end encodes depth observations as spatial tokens. A learned readout combines global and local features, then joins them with encoded proprioception to form the world-model observation representation. This connects terrain geometry with the robot’s posture and joint state, providing a shared basis for recurrent state estimation.

Action-conditioned prediction, observation-conditioned inference

At world-model update k, the RSSM first advances deterministic memory using the previous latent state and the intervening action sequence A. The current observation embedding then conditions the stochastic posterior. Current visual evidence updates z before influencing h through a subsequent recurrent transition. Between depth updates, the controller reuses the latest deterministic state.

During training, prior–posterior KL regularization accompanies depth reconstruction, proprioceptive reconstruction, and reward prediction. These heads train the representation; action selection does not require decoding a depth image.

Beyond RSSM: the controller and the robot

The actor concatenates an encoding of deterministic state h, an encoding of the five-frame observation history H, and the velocity command c. The stochastic state z is not a direct actor input. Twelve action outputs are scaled and offset by a reference pose to produce joint position targets; a PD controller converts them to torques. A separate privileged critic and an AMP discriminator support policy training through value estimation and motion-style rewards.

Visual memory, recent motion feedback, and velocity commands meet in the actor before becoming joint targets. Terrain information therefore reaches motion through a complete chain of state estimation, action selection, and low-level control.

03 / Beyond continuous ground

Narrow beams and stepping stones constrain support. Planters and ramps introduce abrupt changes in height. Fences and low passages demand coordination between leg clearance and body posture. Our training terrain combines these challenges within a shared curriculum, exposing the policy to the transitions between approach, traversal, and continued motion. Increasing difficulty extends learning from basic locomotion to more demanding terrain interactions.