ETD

• reinforcement learning
• robotics
• world models

World models are an increasingly active line of research: by learning an internal, action-conditioned model of an environment's dynamics, an agent can train its policy through reinforcement learning inside this learned model, i.e. in imagination, rather than through costly or unsafe interaction with the real world.
State-of-the-art world model architectures, however, are typically large and are evaluated almost exclusively on simulated, video-game-style benchmarks. Real-world deployment is becoming more common, but it remains rare on physical platforms, particularly non-rigid ones.
This thesis proposes a lightweight world model for robotic manipulation, combining a discrete VQ-VAE visual representation with an unsupervised masking mechanism, a recurrent (LSTM) dynamics model, and a PPO policy trained in imagination. The masking mechanism is learned end-to-end through a dual-head decoder that separates task-relevant foreground from background, without hand-crafted or frame-difference supervision.
On five MetaWorld manipulation tasks, the proposed model, with 27.7M parameters in its vision-only variant, achieves a higher final success rate than both DreamerV3 configurations evaluated on three of the five tasks: the similarly sized 23.5M configuration and the 165.9M one, which is roughly six times larger. It also achieves a higher final success rate than a model-free PPO baseline on three of the five tasks, with the gap widening as task difficulty increases, while rewards remain broadly comparable across all models. Beyond vision alone, conditioning the dynamics model on proprioception and gripper pressure further improves both reward and success rate on all five tasks, and makes training more stable.
Finally, the pipeline is deployed on a real, three-chamber pneumatically actuated soft robot manipulator, where it is trained directly on the physical platform on a gear-rotation task. We define a complete world model as one combining pixel-based multimodal perception, action-conditioned latent dynamics, and a policy learned in imagination through an iterative, closed training loop; to the best of our knowledge, this is the first deployment of such a system on a soft robotic platform.