AutoWorldModel-Bench

Key Numbers

See Main Results for the per-session breakdown.

Games

8 games with diverse entity dynamics, ranging from 5 to 52 entities per frame.

Data Preview

A sample episode from each game, rendered from the raw ECS state that ships in the dataset. Pick a game to play its clip.

Each clip is a short rollout recorded from the game simulator. Every frame shown is directly derivable from the structured entity state stored in the released Parquet files.

Baseline Models

Four diverse architectures covering the major paradigms in generative world modeling.

Evaluation Protocol

Three evaluation modes

Per-horizon composite

Final score

Data splits: test and scenario

Main Results

Per-(game, starter) held-out lift on test and scenario splits, with per-agent columns. Mirrors Table 2 of the paper.

Each (game, starter) row shows the starter score and the agent-best score separately for Opus-4.6 and Codex-5.4, because each agent sees its own starter re-run on the shared host (platform non-determinism; see paper §C.7). $\Delta$ test is weighted $0.1\,c_{1} + 0.2\,c_{10} + 0.7\,c_{20}$, blue (+) / crimson (–). $\Delta$ scenario is the same formula with $c_{h_{\text{end}}}$ in place of $c_{20}$, amber (+) / violet (–).

Opus vs. Codex

Per-task scatter of agent-best composite: Codex on the x-axis, Opus on the y-axis. Points on the diagonal indicate a tie.

Win split at the task level: Codex 19, Opus 13 (paired Wilcoxon p = 0.15).

Long-Horizon Gains

Decomposing the mean test composite across the three rollout horizons. Starter is already strong at $h=1$; almost all of the agent's uplift comes from $h=10$ and $h=20$.

Starter vs. agent-best mean composite at each evaluation horizon.

Per-Game Uplift

Mean $\Delta$ test score across the four starter architectures, per game and agent.

Per-Game Scenario Suite

60 scenario groups across the 8 games (1,860 episodes). Each panel shows per-scenario starter → agent-best composite.

• Asteroids
• Breakout
• Frogger
• Kong
• Platformer
• Pong
• Racer
• Snake

Each scenario constructs a controlled initial state that isolates a specific game mechanic (collision, scoring, terminal transition) and applies a deterministic action sequence.

What Wins? Taxonomy of Winning Edits

Every session winner is classified against the starter by a zero-shot judge. 91% of winners are non-trivial research-style edits (new objective, loss, rollout, architecture, data augmentation, or a combination), not hyperparameter tweaks.

HYPERPARAM / BUGFIX / INFRA are counted as trivial; all other labels are non-trivial. Below: per-label uplift across the 1,335 classified experiments.

Session Progress

Best-so-far validation composite over the course of each 6-hour session, per (game, starter). Curves are step functions that only change when a new best is found. Held-out test scores are computed once at session end on the validation-selected checkpoint.

Compute Budget

Number of experiments each agent completed per 6-hour session, aggregated across the 32 shared tasks. The per-experiment training cap is 600 s; the per-session wall clock is 6 h.

Opus runs more experiments per session because it persists natively across iterations; Codex's exec mode is wrapped in a restart loop (paper §D.2). Both receive the same 6-hour budget.

Dataset Statistics

Each game ships 19K episodes across four splits. Trajectories are stored as Parquet files with binary NumPy tensors (per-entity state, actions, rewards, terminals) plus a per-game meta.json.

Getting Started

# Install
pip install -e .

# Prepare data (downloads from HuggingFace Hub)
python scripts/prepare_data.py --game pong

# Train a single task
cd tasks/pong_dreamer
python train.py

# Evaluate
python score.py