Dominic Rigby

VLA Training Revision

Date: 4th November 2026

Related: Pi0.5

RL Token

Key Points

• VLAs are essentially VLMs with action heads stuck on.
• VLMs provides rich interpretation of text and visual data. This introduces ability to follow instructions, intpret scenes and identify objects. Crucial capabilities for intelligent robots.
• VLM processes sensor data, procioceptive data and task in text into tokens which are consumed by VLMs

Action Generator

• Normally produces high level targets for microcontroller rather than low level control
• Either has last token as input or cross attends all tokens
• Often Conditional Flow Matching algorithm (maps Gaussian to real robot movement distribution)
• Generate actions by numerically integrating field equation

RL on VLAs

• Common method:
  1. Add critic and policy heads to VLM
  2. Fine-tune VLA with demonstratioms
  3. Fine-tune action expert with RL
• Key issues for RL
  1. Obtaining prob density:
     • FOten utilise stochastic flow PPO which switches allows us to tractably calculate PD along trajectories
  2. Deciding chunk length… must ensure RL is independent of chunk length
• Different RL VLA algos:
  1. PPO:
     • Pros: straight forward, stable
     • Cons: limited reuse of data make it sample inefficient
  2. SAC:
     • Pros: replay buffer makes it more sample efficient
     • Cons: Q estimation is difficult with flow policies, stability

Different Levels of RL

1. Noise steering policy only: cheap but cannot fix issues in VLA representation
2. Action expert + value head
3. Adapters / LoRA: cheaper full update
4. Backbone and expert fine-tuning: interpretation of instruction is wrong.

Scaling Directions

• Need to scale experience generation, richness of the experience, learner throughput