WAM-Adapter: Efficiently Adapting Video-DiT World Representations to Robot Actions

WAM-Adapter efficiency and performance teaser — **TL;DR:** WAM-Adapter freezes a compact 1.3B Video-DiT, adapts it with LoRA, and directly reuses the multi-layer representations produced by a *single* current+future forward pass. Inference needs no future-video generation and no iterative action denoising, achieving 89.0% success across 50 RoboTwin 2.0 tasks with only 121M trainable parameters.

Abstract

Pretrained Video Diffusion Transformers encode rich spatiotemporal priors from large-scale videos, making them promising backbones for robot control. However, efficiently adapting them into World Action Models remains challenging. Existing approaches reduce video-adaptation or action-prediction costs, but it remains unclear whether representations produced during future-video adaptation can directly support lightweight action prediction.

To address this gap, we propose WAM-Adapter, a parameter- and training-efficient framework that freezes the Video-DiT backbone, adapts it with LoRA, and reuses multi-layer representations for action prediction. A lightweight readout organizes multi-layer K/V representations from current-and-future processing into Current, Early-Future, and Late-Future groups, applying group-specific cross-layer fusion for action-chunk prediction. Inference requires a single Video-DiT forward over the current observation and noisy future slots, without future-video generation or iterative action denoising.

Without embodied pretraining, WAM-Adapter uses only 121M trainable parameters with a compact Wan-1.3B backbone, achieving 89.0% success across 50 RoboTwin 2.0 tasks, 98.5% on LIBERO, and 75.17% on LIBERO-Plus. To our knowledge, this is the smallest reported trainable-parameter budget for directly trained WAMs without embodied pretraining. Under the same setting, WAM-Adapter improves training throughput by 9.66× while using 98% fewer trainable parameters than Fast-WAM. These results show that Video-DiT representations formed during world-model adaptation can be efficiently reused for strong robot control.

Method

A frozen Video-DiT, a single joint current+future forward, and a lightweight temporal-group readout that predicts the whole action chunk directly.

WAM-Adapter architecture — **Overview of WAM-Adapter.** A frozen pretrained Video DiT is adapted with LoRA and processes the clean current observation together with noisy future latent slots in a single forward pass. WAM-Adapter preserves the resulting Current, Early-Future, and Late-Future representations, applies temporal-group-specific cross-layer readout, and uses a lightweight action decoder to directly predict an action chunk.

1. Frozen Video-DiT + LoRA

The pretrained Wan2.1-T2V-1.3B backbone keeps its spatiotemporal priors intact. Rank-64 LoRA adapts it to robot data, so only 121M parameters are trainable — no full backbone fine-tuning, no embodied pretraining.

2. One Joint Current+Future Pass

The clean current frame and a few noisy future latent slots (τ = 1) are processed together in a single Video-DiT forward. There is no future-video denoising loop and no additional action-side Video-DiT forward.

3. Temporal-Group Cross-Layer Readout

The readout keeps the temporal identity of the representations as Current, Early-Future, and Late-Future groups, learns group-specific cross-layer fusion weights, aligns fused keys to the action timeline, and predicts the chunk with a small mixed-context decoder.

Results

Strong control performance with the smallest reported trainable-parameter budget among directly trained WAMs without embodied pretraining.

Main Results

Success rates and efficiency

89.0%

RoboTwin 2.0 (50 tasks)

98.5%

LIBERO (four suites)

75.17%

LIBERO-Plus (zero-shot)

121M

trainable parameters (1.3B frozen backbone)

9.66×

training throughput vs. Fast-WAM

92.05 ms

inference latency (RTX 4090)

3.41 GiB

peak inference memory

Takeaway: WAM-Adapter achieves the highest LIBERO-Plus success rate among the compared methods and reaches 89.0% on RoboTwin 2.0, while training with 98% fewer trainable parameters than Fast-WAM under the same setting.

Analysis

Adaptation, readout and real-world behavior

Backbone adaptation ablation — **Backbone adaptation.** With the backbone frozen and no LoRA, success is 49.4%; rank-64 LoRA (used by WAM-Adapter) reaches 97.4%, close to full fine-tuning (98.5%) that updates the entire Video-DiT.

**Readout analysis.** (a) Layer-wise readout magnitude for each temporal group; (b) attention across predicted action steps — Early-Future representations contribute to early steps, Late-Future to later steps.

**Real-world evaluation.** Beyond simulation, WAM-Adapter consistently outperforms the lightweight WAM baseline across three real-world manipulation tasks under the same evaluation protocol.

**Qualitative prediction and attention.** The model produces accurate future predictions and action chunks, and its attention tracks the manipulated objects across time.

BibTeX

The arXiv release is coming soon; for now please cite the project as:

@article{wamadapter2026,
  title   = {WAM-Adapter: Efficiently Adapting Video-DiT World Representations to Robot Actions},
  author  = {Liu, Dongxu and Li, Zhe and Ke, Zundong and Zhang, Zhipeng},
  journal = {arXiv preprint},
  year    = {2026},
}