Watch a neural network learn from scratch. The agent below starts knowing nothing — it controls a physics simulation by trial and error and, episode after episode, discovers a strategy on its own. No demonstrations, no labelled data, only reward.
Hover any ? icon for a plain definition. Press Start, then use the Agent View to replay any episode and watch a clumsy first try turn into a confident one.
A healthy run shows the cyan reward line climbing toward the target as the amber epsilon line falls — the agent earns more while relying less on random luck.
Deep Q-Networks were introduced by DeepMind in 2015 to play Atari games from raw pixels. This project applies the same algorithm to classic control tasks. Six ideas make it work:
At each step the agent observes the environment's state, chooses an action, and receives a reward plus the next state. RL is the search for an action-picking rule — a policy — that maximises total reward. There are no labelled answers, only this feedback.
A neural network maps the current state to one number per action: the Q-value, its estimate of the total future reward that action leads to. The agent simply picks the action with the highest Q-value.
An agent that always picks its current best guess never discovers anything better. Epsilon (ε) is the chance of acting randomly — it starts at 1.0 (fully random) and decays to 0.05, so the agent explores early and exploits later.
Every transition (state, action, reward, next state) is saved to a 20,000-slot memory buffer. Training samples random batches from it, which breaks the correlation between consecutive steps and lets each experience teach the network many times.
Learning targets come from a frozen copy of the network, re-synced only every 10 episodes. Without it the network would chase a target that shifts every step — like aiming at your own reflection — and training would diverge.
Each step nudges the prediction toward reward + γ × (best Q-value of the next state), where γ = 0.99 discounts future reward. Repeated across millions of steps, this temporal-difference update flows reward back onto the actions that earned it.
Renders the controls and the live SVG chart, and opens one WebSocket per run.
Takes the run config, starts a training session, and streams every episode's metrics.
A background thread runs the DQN training loop episode by episode.
Simulates the physics and returns a reward + next state on every step.