DIAMBRA Arena — How 9 Fighting Games Became Reinforcement Learning Sandboxes

DIAMBRA Arena transforms arcade fighting games into standard reinforcement learning environments — giving any developer with a laptop and a registered account the ability to train RL agents on Dead or Alive, Street Fighter, Tekken, and Mortal Kombat.

Published in October 2022 [1], DIAMBRA Arena wraps game ROMs through a custom Docker-based emulation stack and exposes them via the OpenAI Gym/Gymnasium Python API [2]. The result: 9 fighting games with discrete action spaces, pixel-plus-RAM observations, and built-in support for single-player, two-player, self-play, imitation learning, and human-in-the-loop training.

This post breaks down the architecture, the agent-environment loop, the wrapper system, and what the platform means for game developers who want to experiment with reinforcement learning.

Why Fighting Games?

Fighting games occupy a sweet spot for RL research. They’re more complex than Atari (which DQN mastered in 2015) but more constrained than MOBAs or open-world games. Each match is a well-defined episodic task with:

Discrete actions. Every gamepad button press — punch, kick, block, jump, directional input — maps to a discrete action. The combo system introduces temporal dependencies that reward policies with memory.
Clear reward signals. Health bar deltas are linear and bounded. A round win or loss is unambiguous.
Hidden information. Frame data, recovery animations, and mix-ups create information asymmetry even in a two-player zero-sum game — making them fertile ground for competitive multi-agent RL.
Deterministic simulation. Arcade ROMs run on fixed emulation cycles. Every frame is reproducible, which matters for debugging and evaluation.

DIAMBRA maps these properties into a standard RL framework without requiring game developers to modify ROMs or reverse-engineer memory addresses.

Architecture Overview

The system has three layers:

Layer 1: Docker + Emulation Engine. Each game runs inside a Docker container with a MAME-based emulator. The DIAMBRA engine maintains frame synchronization: it intercepts the emulator’s frame buffer and controller input at each cycle, stopping the emulator between frames until the agent provides its action.

Layer 2: Arena Python Package. The diambra-arena PyPI package provides the Gymnasium-compatible interface. It handles environment creation, action space construction, observation assembly, and wrapper stacking. The package talks to the Docker engine via a gRPC-like client protocol.

Layer 3: Agent Interface. The agent (your Python code) receives observations, returns actions, and collects rewards — exactly like any Gymnasium environment. DIAMBRA adds a CLI (diambra run) that launches the Docker container, mounts the ROM directory, and runs the agent script inside the container network.

                   ┌─────────────┐
                   │   Agent     │
                   │ (your code) │
                   └──────┬──────┘
                          │ obs, reward, done
                          │ actions
                   ┌──────▼──────┐
                   │ diambra-    │
                   │ arena (PyPI)│
                   └──────┬──────┘
                          │ gRPC client
                   ┌──────▼──────┐
                   │  DIAMBRA    │
                   │  Engine     │
                   │  (Docker)   │
                   └──────┬──────┘
                          │ frame buffer
                          │ controller input
                   ┌──────▼──────┐
                   │  MAME       │
                   │  Emulator   │
                   │  (ROM)      │
                   └─────────────┘

Action Spaces

Each game exposes a set of discrete actions corresponding to physical controller buttons. DIAMBRA’s action space is a MultiDiscrete with two components: a movement action (neutral, left, right, up, down, up-left, up-right, down-left, down-right) and an attack action (independent button combinations for punch, kick, block, taunt, etc.).

For Dead or Alive ++ (game ID doapp), the attack space includes 6 buttons — punch, kick, hold, block, tag, and taunt — plus their combinations. The total action count is roughly 9 × (2^6 − 1) = 567 possible combos.

The No Attack Buttons Combinations wrapper simplifies this by restricting attacks to single buttons only, dropping combinations and yielding 9 × 6 = 54 actions. This is the recommended starting point for new agents.

Observation Spaces

Observations come in two flavors:

Screen pixels. Grayscale or RGB frames at native arcade resolution (typically 384×224) downsampled to a configurable working size like 128×128. The frame_shape environment setting controls this — it’s applied at the engine level for speed, not as a Python-side warp.

RAM states. DIAMBRA exposes specific memory values from the emulated arcade hardware: health bars (as percentages 0.0–1.0), character selection, stage identifiers, and side-of-stage. These are packed into a dictionary alongside the frame, giving the agent structured game state without forcing it to learn health-bar reading from pixels.

The Stack Frames wrapper (configurable from 1 to 48 frames) concatenates consecutive frames along the channel dimension, letting the agent perceive velocity and attack animations across time without relying on RNN memory.

Reward Design and the Wrapper System

DIAMBRA ships 12+ built-in wrappers that modify observations, actions, and rewards without changing the core environment. Three matter most for training:

1. Reward Normalization. Health bars vary per game (100 HP in Street Fighter, 240 in Tekken). The normalize_reward wrapper divides the per-frame health delta by a game-aware normalization factor, producing rewards in a consistent range.

2. Reward Clipping. Binary reward signal: +1 for health gain, −1 for health loss, 0 otherwise. This simplifies the reward landscape at the cost of losing magnitude information. Works well with PPO in practice.

3. Frame Dilation. The dilation parameter (paired with stack_frames) controls the gap between stacked frames. A dilation of 4 means frame N, N-4, N-8 are stacked — effectively giving the agent a stroboscopic view that catches fast attacks without the computational cost of 60 FPS processing.

Custom wrappers can be injected via the wrappers parameter as a list of (class, kwargs) pairs, applied after all built-in wrappers.

The Competition Platform

DIAMBRA runs an agent submission platform where developers can upload trained agents and compete in automated tournaments. The pipeline:

Submit a Docker image (or use one of the pre-built agent templates).
The platform runs head-to-head matches on dedicated evaluation hardware.
Episodes stream live on DIAMBRA’s Twitch channel [3].
Agents are ranked on the global leaderboard by Elo score.

The platform also runs special events: a $2000 Street Fighter tournament [4] completed in 2024 and an ongoing LLM Colosseum where language models compete by generating agent code.

This turns DIAMBRA from a research tool into a competitive benchmark — similar to what the VGC (Video Game Championship) series does for Pokémon [5], but automated and running continuously.

Practical Training Setup

The cheapest way to get started with DIAMBRA:

pip install diambra-arena[stable-baselines3]
diambra arena list-roms          # Find your ROM
diambra arena check-roms ./roms/doapp.zip  # Verify SHA256
diambra run -r ./roms/ python train_ppo.py

A PPO agent [6] training on Dead or Alive ++ reaches competent play (beats the built-in CPU at difficulty 3) in about 4 hours on an RTX 3060 — roughly 5 million environment steps.

from stable_baselines3 import PPO
import diambra.arena
from diambra.arena import make, WrapperSettings

settings = WrapperSettings()
settings.normalize_reward = True
settings.clip_reward = True
settings.stack_frames = 4

env = make("doapp", settings=settings)
model = PPO("MultiInputPolicy", env, verbose=1)
model.learn(total_timesteps=5_000_000)

The API supports Ray RLlib for distributed training across multiple GPUs or machines, useful for multi-agent setups where two policies train concurrently via self-play.

What This Means for Game Developers

DIAMBRA matters beyond RL research. Three takeaways:

1. Fighting games are proxy problems. If you can train an agent to play Street Fighter, the techniques — memory-augmented policies, reward shaping, opponent modeling — transfer to any competitive real-time system. Game AI, bot design, difficulty calibration, and automated playtesting all benefit.

2. Emulation-based environments are reproducible. Deterministic tick-by-tick emulation means every training run is reconstructable. This is not true for cloud-dependent game environments (like the ones OpenAI used for Dota 2). Reproducibility matters for both research and debugging.

3. The barrier to entry is low. A $400 GPU, a free DIAMBRA account, and a ROM you already own legally is enough to train competitive agents. The Stable Baselines 3 integration means you don’t need to implement RL algorithms from scratch.

Limitations

DIAMBRA isn’t a silver bullet. The current game library is capped at 9 fighting games from the arcade era — no modern 3D titles, no MOBAs, no open-world environments. The emulation stack adds latency: at step_ratio=1, the environment runs at 60–190 FPS depending on the game, which is 2–6× faster than real-time but well below simulation frameworks like Brax or Isaac Gym.

The observation space doesn’t include audio, which matters in fighting games where players react to sound cues. And the ROM requirement — you must own and verify each game ROM — creates a legal and logistical hurdle.

Key Takeaways

DIAMBRA Arena wraps 9 arcade fighting games as Gymnasium environments, usable with any RL library.
The architecture uses Docker-based MAME emulation with a gRPC client protocol for frame-synchronous agent interaction.
Built-in wrappers handle frame stacking, reward normalization, action simplification, and observation flattening — no custom code needed for standard setups.
A competition platform lets developers submit agents for automated, streamed tournaments.
Training a competent PPO agent costs about $0 in API fees on consumer hardware.
The game library is limited to 2D fighting games, but the underlying approach applies to any emulated title.