Intention abstraction
Condense verbose instructions into compact semantic primitives, generated offline by zero-shot Qwen2-VL-7B.
[waypoint, safezone]See IA evidence ↗3A-VLA in action00:51
A shared intent. Different worlds.01 / OVERVIEW
From pixels to actionable abstractions.
Sparse intent. Structured affordances. Aligned action supervision.
FIGURE 1PDF ↗
Intention and environment abstractions expose task-relevant concepts, such as walls and enemies, to guide action supervision.
Dynamic 3D games confront an agent with dense visual observations and context-dependent actions. 3A-VLA adds two compact auxiliary targets to action learning: intention abstraction (IA), which captures the goal and strategy, and environment semantics abstraction (ESA), which captures task-relevant scene structure. Abstraction alignment reweighting (AAR) uses their relation to adapt the action-imitation loss.
66.4%Game for Peace
Six-task average SR
40.4%Minecraft MCU Mini
Macro-average across three groups
+13.0ppOver Vanilla IL on GP
53.4% → 66.4%
02 / METHOD
Three components. One shared policy.
Qwen2-VL-7B · IA + ESA + AAR
FIGURE 2PDF ↗
One shared causal Transformer learns actions together with IA and ESA targets. AAR reweights action supervision using their relation. At deployment, only the action chunk is decoded.
FIGURE 2 / EXPLORE THE MECHANISM
Follow the thought. Find the action.
Move across the signals or tap a brain region to explore.
INPUTS
Explore the evidence ↗One instruction. The whole scene.
The original image and “Shoot the nearby enemy.” enter one shared model. Keywords and semantic regions are outputs, not inputs.
IA and ESA guide action learning during training. At deployment, the policy decodes actions.
Environment semantics
Pool scene categories into an 8 × 8 grid, retaining top-K tactical classes per cell using an off-the-shelf SegFormer-B2.
person · vehicle · cover · item · otherSee ESA evidence ↗Alignment reweighting
Learn batch-relative action weights from the relation between IA and ESA, with an entropy regularizer and stop-gradient.
Action loss × alignment weightSee AAR evidence ↗TRAIN WITH ABSTRACTIONSAction → IA → ESA↗DEPLOY WITH ACTIONSRGB + instruction → Action
03 / MAIN RESULTS
The full picture, task by task.
TABLE 1
Minecraft: embodied, combat and GUI
Mini: 10 tasks per group
| Model | Size | Embodied steps | Embodied Mini SR | Embodied All SR | Combat steps | Combat Mini SR | Combat All SR | GUI steps | GUI Mini SR | GUI All SR |
|---|---|---|---|---|---|---|---|---|---|---|
| VPT | 248M | 377 | 10.1 ± 2.8 | 6.0 ± 1.4 | 396 | 3.6 ± 1.5 | 3.6 ± 1.2 | 398 | 0.7 ± 0.1 | 0.8 ± 0.3 |
| STEVE-1 | 248M | 384 | 8.4 ± 2.4 | 8.0 ± 1.8 | 395 | 4.9 ± 1.4 | 3.9 ± 1.1 | 391 | 0.0 | 3.2 ± 1.3 |
| ROCKET-1 | 72B | 392 | 19.2 ± 3.5 | 18.9 ± 3.6 | 320 | 29.8 ± 4.2 | 27.9 ± 4.8 | - | - | - |
| JARVIS-VLA | 7B | 305 | 31.0 ± 4.9 | 30.0 ± 5.3 | 352 | 18.3 ± 3.1 | 18.5 ± 3.6 | 339 | 25.3 ± 4.2 | 25.1 ± 4.5 |
| OpenHA | 7B | 287 | 37.0 ± 5.8 | 30.1 ± 4.9 | 316 | 40.0 ± 6.1 | 31.9 ± 5.2 | 314 | 33.3 ± 5.0 | 32.5 ± 5.3 |
| 3A-VLA | 7B | 278 | 40.6 ± 3.2 | 33.5 ± 2.6 | 271 | 45.9 ± 3.5 | 35.4 ± 2.9 | 307 | 34.7 ± 2.7 | 34.8 ± 2.6 |
Embodied steps
Combat steps
GUI stepsCompare success rates and environment steps across Mini and All tasks.
TABLE 2
Game for Peace: six atomic tasks
| Model | Parachuting | Scavenging | Combat | Revival | Vehicle | Rotation | Average | Latency (s) |
|---|---|---|---|---|---|---|---|---|
| GPT-4o | 41.4 | 31.4 | 18.6 | 47.1 | 38.6 | 40.0 | 36.2 | 1.6 |
| Claude-3.7-Sonnet | 48.6 | 38.6 | 24.3 | 57.1 | 41.4 | 45.7 | 42.6 | 2.2 |
| Gemini-2.5-Pro | 55.7 | 42.8 | 25.7 | 58.6 | 47.1 | 51.4 | 46.9 | 1.5 |
| Qwen2-VL-7B | 22.9 | 15.7 | 8.6 | 27.1 | 18.6 | 21.4 | 19.1 | 1.1 |
| Qwen2-VL-7B (Vanilla IL) | 54.3 | 48.6 | 44.3 | 68.6 | 51.4 | 52.9 | 53.4 | 1.1 |
| Qwen2-VL-7B (IL + CoT) | 55.3 | 49.1 | 45.7 | 68.1 | 53.1 | 53.9 | 54.2 | 1.1 |
| Qwen2-VL-7B (IL + Image caption) | 56.0 | 50.4 | 46.3 | 68.7 | 53.4 | 54.3 | 54.9 | 1.1 |
| Qwen2-VL-7B (IL + CoT + Caption) | 56.3 | 50.9 | 46.8 | 69.3 | 53.7 | 55.0 | 55.3 | 1.1 |
| 3A-VLA (Ours) | 66.1 | 60.5 | 67.1 | 70.2 | 68.9 | 65.7 | 66.4 | 1.1 |
Parachuting
Scavenging
Combat
Revival
Vehicle
Rotation
SR (%) for parachuting, scavenging, combat, revival, vehicle acquisition and rotation. Latency is in seconds; the default policy uses zero-shot Qwen2-VL-7B IA and SegFormer-B2 ESA.
The Minecraft and GP policies are fine-tuned separately on 50 h and 200 h of data. Main-table latency is 1.1 s per step on one L40S.
04 / ABLATIONS & ANALYSIS
What makes abstraction useful?
Component controls, weighting controls and visual evidence.
01 / IA
The intent target matters.
Compact keywords outperform a full reasoning trace in the isolated IA comparison.
Revisit this mechanism ↗INSTRUCTION
Find a way to quickly enter the safe zone to escape the hazard zone.
↓ Compact intent targets
waypointsafezone
TABLE 3 · IA
How should intent be represented?
| IA target | MC steps ↓ | MC SR (%) ↑ |
|---|---|---|
| None (baseline) | 332 | 24.6 |
| Full CoT | 310 | 28.3 |
| Keywords | 298 | 30.2 |
IA alternatives, with ESA absent.
02 / ESA
Keep the useful scene structure.
The latent semantic target performs best among these isolated ESA alternatives.
Revisit this mechanism ↗
personcover8 × 8 semantic target
TABLE 3 · ESA
What should the scene target contain?
| ESA target | MC steps ↓ | MC SR (%) ↑ |
|---|---|---|
| None (baseline) | 332 | 24.6 |
| Img. Caption | 320 | 27.0 |
| Img. Pred. | 322 | 26.8 |
| Full Sem. | 318 | 27.4 |
| Latent | 304 | 30.6 |
ESA alternatives, with IA absent.
03 / IA + ESA
Two abstractions. A shared advantage.
Sparse intention and scene targets reach 36.5%; adding AAR brings the full model to 40.4%.
Revisit this mechanism ↗INTENTIONenemywall
↔↔
ENVIRONMENTpersoncover
Alignment → action-loss weight
TABLE 3 · COMBINATION
Bring the abstractions together
| Variant | IA target | ESA target | MC steps ↓ | MC SR (%) ↑ |
|---|---|---|---|---|
| Baseline | - | - | 332 | 24.6 |
| Baseline | Full CoT | Img. Caption | 309 | 32.3 |
| Ours (w/o AAR) | Keywords | Latent | 297 | 36.5 |
| Ours | Keywords | Latent | 285 | 40.4 |
Dense targets, sparse targets, and the complete model.
04 / AAR
Alignment improves the learning signal.
Learned alignment weighting exceeds fixed and scalar weighting controls in this comparison.
Revisit this mechanism ↗TABLE 4 · WEIGHTING
Learned or fixed action weights?
| Weighting scheme | MC steps ↓ | MC SR (%) ↑ |
|---|---|---|
| Ours (w/o AAR) | 297 | 36.5 |
| Fixed w=0.33 | 303 | 36.9 |
| Fixed w=0.67 | 302 | 37.0 |
| Scalar cosine | 299 | 37.1 |
| Scalar Hadamard | 295 | 37.6 |
| AAR (Ours) | 285 | 40.4 |
Compare fixed weights, scalar relations and AAR.
AAR / FURTHER CONTROLS
Test the weighting. Test its parts.
Dense-target controls and component removals complete the comparison.
TABLE 4 · DENSE TARGETS
Does reweighting help dense targets?
| Dense-target configuration | MC steps ↓ | MC SR (%) ↑ |
|---|---|---|
| CoT+Cap. w/o AAR | 309 | 32.3 |
| CoT+Cap. w/ VLM-Score | 307 | 32.8 |
| CoT+Cap. w/ AAR | 301 | 36.0 |
The same CoT + caption targets with three weighting settings.
TABLE 4 · COMPONENTS
Which parts of AAR matter?
| AAR configuration | MC steps ↓ | MC SR (%) ↑ |
|---|---|---|
| Ours (w/o AAR) | 297 | 36.5 |
| AAR w/o H̄ | 291 | 38.2 |
| AAR w/o stop-gradient | 289 | 37.6 |
| AAR (Ours) | 285 | 40.4 |
Remove AAR, entropy regularization or stop-gradient.
FIGURE 3PDF ↗
IA, ESA and AAR progressively concentrate attention on task-relevant regions. The paper’s first example reduces attention entropy from 7.58 to 3.42.
05 / CITATION
Cite this work+
@inproceedings{zhou20263avla,
title={3A-VLA: Abstraction-Aligned Action Learning for Vision-Language Agents in 3D Game Worlds},
author={Zhou, Zheyuan and Du, Liang and Sun, Zixun and Zhou, Xiaoyu and Ye, Ruimin and Chen, Qihao and Chen, Yinda and Qiu, Lemiao},
booktitle={Advances in Neural Information Processing Systems},
year={2026}
}