CoRL 2026

MPC-Injection: Biasing Off-Policy Locomotion RL Toward Controller-Induced Behavior Basins

Roy Xing¹ Seyoung Ree² Brian Plancher¹

¹Dartmouth College·²Harvard University

Video Overview

Abstract

Steering RL Towards Behavior Basins

Reinforcement learning (RL) for locomotion frequently converges to locally optimal but undeployable behaviors, such as vibrating limbs or scooting on the torso, that maximize return without producing a usable gait. We present MPC-Injection, a low-overhead method that steers RL toward a designer-preferred gait by inserting transitions into the replay buffer from a model predictive controller solving the same Markov decision process. Unlike reward shaping, MPC-Injection does not require redesigning the task reward, and unlike adversarial imitation learning, it adds no discriminator, no kinematic retargeting, and no auxiliary objective. Instead, the controller's preferred behavior is transferred to the policy purely through the replay state distribution. On a 2D walker in simulation and with sim-to-real evaluation on a Go2 quadruped, we show that MPC-Injection drives the policy into the controller's behavior basin using a one to two-term task reward, producing gaits qualitatively comparable to those of reward shaping with twenty-one tuned terms and of adversarial motion priors without their discriminator and retargeting overhead. We further analyze how the injected transitions bias actor-critic updates toward controller-visited states, allowing the policy to learn behaviors that pure RL may fail to reach under simple reward functions.

Approach

Offline MPC Data Injected During Training

MPC-Injection steers the learner through its replay buffer rather than its reward. It runs in three stages:

MPC data generation: Before any RL run, an MPC is rolled out in an environment matched to the learner's — same Go2 model, 50 Hz control, command ranges, observation noise, and termination thresholds — and its trajectories are saved to disk.
Compute RL rewards for MPC transitions: Rewards for MPC transitions are computed under the RL task reward function rather than the MPC cost, so injected and on-policy transitions share a single reward definition.
Inject during RL: After on-policy transitions are inserted into the replay buffer, MPC transitions are added until the fraction of MPC transitions reaches the injection ratio p (typically p = 0.25).

MPC transitions enter learning only through the replay distribution: there is no separate imitation loss, sample weighting, or auxiliary objective. The training process of the off-policy agent is otherwise unchanged from standard SAC or TD3.

MPC-Injection pipeline: saved MPC trajectories are replayed into the replay buffer alongside the agent's own rollouts, biasing actor-critic updates.

Simulation

Simulation Results

MPC-Injection selects qualitatively different locomotion basins from vanilla RL under identical task rewards, with 25% the injection ratio that most reliably induces the controller's basin.

Walker

Under the same velocity-tracking reward, pure RL converges to a grounded scooting strategy that drags the top of the torso along the ground, while 25% MPC-Injection learns an upright, periodic gait matching the MPC behavior.

0% MPC-Injection (pure RL)

25% MPC-Injection

Quadruped

Pure off-policy RL learns to vibrate the joints, while 25% MPC-Injection learns a structured trot with periodic footstep patterns.

0% MPC-Injection (pure RL)

25% MPC-Injection

Foot trajectories under pure RL.

(a) Pure RL

Foot trajectories under 25% MPC-Injection.

(b) 25% MPC-Injection

(a) Footstep trajectories under pure RL show chaotic, irregular patterns. (b) Footstep trajectories under 25% MPC-Injection show structured, periodic patterns matching a trotting gait.

Barrel Roll

Under the same simple reward function, pure RL never discovers the maneuver, while 25% MPC-Injection completes the roll — a behavior that normally requires reward shaping or assistive forces during curriculum learning.

0% MPC-Injection (pure RL)

25% MPC-Injection

Hardware

Real-World Experiments

Velocity Tracking

All policies tracking a 0.5 m/s velocity command on a Go2.

Reward Shaping

AMP

MPC-Injection (ours)

It took several trials to get AMP to work without falling over immediately, while reward shaping and MPC-Injection consistently trotted.

Perturbation Test

External disturbances applied to each policy under the same command.

Reward Shaping

AMP

MPC-Injection (ours)

It took multiple trials to get AMP to work without falling over, while reward shaping and MPC-Injection consistently worked.

Bounding

Changing only the injected MPC data, under the same simple reward function, biases the policy to a bounding gait.

Citation

BibTeX

@inproceedings{xing2026mpcinjection,
  title         = {MPC-Injection: Biasing Off-Policy Locomotion RL Toward
                   Controller-Induced Behavior Basins},
  author        = {Xing, Roy and Ree, Seyoung and Plancher, Brian},
  booktitle     = {Conference on Robot Learning (CoRL)},
  year          = {2026},
  eprint        = {2606.26392},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO}
}