Agentic Racing Research with PPO and Offline World Models

This report documents a reinforcement-learning project that evolved through an autonomous research loop rather than a single fixed implementation. The project began with PPO control for pixel-based racing and later shifted toward an offline recurrent state-space world model when direct control improvements became bottlenecked by representation quality. Throughout the project, the research process itself was agentic: AutoResearch proposed hypotheses, executed targeted configuration changes, screened failures, and promoted only those candidates that survived observation-driven gates.

The PPO phase established a working visual control baseline after early failures such as steering collapse, off-track reward hacking, and no-progress circling were diagnosed and repaired in the training loop. The strongest early AutoResearch single-car result was `run_008`, which reached mean reward `205.46`, mean progress `0.5551`, and off-track rate `0.036` after `49,152` timesteps. A later recursive control branch promoted `g003_c03`, which combined broader brake exploration with a lower corner target speed and reached mean reward `232.58` with mean progress `0.7376`.

The world-model phase was driven by a different question: whether an RSSM could learn a reusable latent dynamics model of racing rather than merely optimize local control. Fixed-architecture experiments from `E1` through `P5` improved corridor persistence, reward faithfulness, and telemetry prediction, culminating in the stable checkpoint `P5_d4_main_telemetry_a100_bs128_e15`. The decisive improvement sequence inside that ladder was the telemetry pivot `P3 -> P4 -> P5`, which converted the project from generic visual continuation into physics-aware prediction. However, the primary unresolved failure mode remained sharp-turn future dynamics. Later data, objective, and structural ablations were therefore valuable mainly as negative evidence: they demonstrated that curated turn-heavy replay, perceptual or progress-only objective tweaks, a wider stochastic latent, and a lightweight world-motion factorization were insufficient to solve turn evolution.

The central contribution of the repository is thus a transparent benchmark record of hypothesis-driven iteration. The strongest result is not a single headline number but a reproducible chronology: PPO became operational through explicit bug repair and AutoResearch-guided reward tuning, while the world-model branch clarified that future progress most likely requires explicit pose evolution rather than additional fixed-objective training on the existing RSSM family.

The project was organized as a repeated Reasoning -> Action -> Observation cycle. In the PPO branch, the cycle was implemented by AutoResearch over candidate training programs and reward-shaping variants. In the world-model branch, the same logic appeared as a sequence of explicit intervention memos, benchmark summaries, and promotion gates. This matters methodologically: the repository is not merely a final model dump, but a record of which hypotheses survived contact with the simulator.

Formally, let the research state at iteration k be s^(k), the proposed code or configuration change be a^(k), and the resulting measured telemetry be o^(k). The agentic loop can be summarized as

The control branch used PPO on `96 x 96` RGB observations with continuous steering, throttle, and brake actions. The maintained local policy implementation used a tanh-squashed Gaussian actor with corrected log-probabilities, together with separate multilayer perceptrons for the policy and value heads. This separation was not a stylistic preference: it followed directly from an early failure analysis in which a shared head allowed value gradients to corrupt policy features.

PPO optimization followed the standard clipped objective,

Returns were estimated with generalized advantage estimation,

The key project lesson from this stage is that apparently small implementation details such as action squashing, bootstrapping across terminal states, and progress computation determined whether PPO behaved as a driver or as a reward hacker. The strongest control results arose only after those low-level pathologies were resolved.

The move from single-car PPO to multi-agent racing was not a simple increase in the number of cars. It required a change in learning topology, data layout, reward semantics, and evaluation. In the maintained torch stack, the relevant paradigm is explicitly `shared_policy_ippo`: one trained network is reused across all cars, each car receives only its own local image observation, and PPO updates are computed from the union of all per-agent trajectories.

That design choice was pragmatic and statistically efficient. The single-agent policy already encoded useful low-level control priors for throttle, steering smoothness, and corner entry. The multi-agent problem therefore did not require a separate "brain" per vehicle. Instead, it required a way to expose one policy to a richer state distribution in which the same local controller sometimes leads, sometimes drafts, sometimes defends position, and sometimes attempts a pass.

Formally, if car i receives observation o_i,t and produces action a_i,t, then the policy is shared: π_θ(a_i,t | o_i,t) for every agent i. The PPO objective is still the clipped surrogate, but now it is aggregated over all agent streams:

where M denotes the total number of active agent streams in the rollout buffer. This is precisely how the implementation behaves: with `n_envs` environments and `num_agents` cars per environment, the rollout buffer is allocated over `n_streams = n_envs * num_agents`, and timesteps are counted per stream rather than per environment.

At the code level, this transition is visible in three places. First, `TorchPPOTrainer` infers a multi-agent observation/action contract from the environment and validates it before training. Second, `_collect_rollout()` converts the environment state into a batched per-agent policy tensor, queries a single policy, and then maps the resulting per-agent actions back into the simulator layout. Third, reward, `done`, and value tensors are flattened into per-agent streams so that PPO updates treat all cars as training data for one shared policy.

This is why the multi-agent section should be understood as a true architectural extension rather than a superficial wrapper. The model is still a single visual actor critic, but the training system around it becomes relational and stream-based.

One of the strongest engineering decisions in the repository was to preserve the parts of the single-agent reward that were already doing useful work. The multi-agent config still keeps the control-first prior: strong progress shaping, explicit off-track termination, corner-entry discipline, steering smoothness, and no-progress failure detection. In other words, competition was added on top of competent driving rather than used as a substitute for it.

This design philosophy is explicit in the maintained multi-agent config, which bootstraps from the promoted single-agent control surface `autoresearch_run_008`, preserves `corner_target_speed = 7.0`, keeps steering-smoothness penalties active, and terminates off-track episodes early. From an ML perspective, this is the correct bias: good racing emerges more reliably when control stability is treated as a prerequisite rather than as an emergent side effect of competitive pressure.

The user-provided design notes are consistent with the final code: the relevant multi-agent shift is not just "add an overtake bonus." It is a move from a progress-only reward to a structured competitive objective in which rank incentives are explicitly filtered by spatial safety. In the implementation, overtake reward is tied to pairwise progress-order changes, then softened or suppressed when spacing is poor.

This matters because racing admits pathological equilibria. A naive competitive reward can teach the agent that contact is a shortcut to position change. The repo's later safety rails therefore penalize collisions by severity, escalate prolonged contact into termination, and explicitly track front-gap and side-clearance telemetry. The reward is not merely asking "did the car move forward?" but "did it improve race position in a way that still looks like driving?"

The evaluation protocol had to change as much as the reward. For single-agent PPO, progress, off-track rate, and steering variance were enough to tell whether the policy was useful. For multi-agent racing, those metrics become incomplete because an apparent gain in reward can come from blocking, hooking, or rubbing rather than from an actual clean pass.

The maintained evaluator therefore records competitive and safety metrics directly from per-agent `info` dictionaries. In addition to reward and progress, it tracks `mean_rank`, `mean_overtakes`, `collision_rate`, `contact_rate`, `hook_contact_rate`, `contact_termination_rate`, `mean_max_contact_steps`, `mean_front_gap`, and `min_front_gap`. This is the correct lens for multi-agent RL: we do not only ask whether the cars are faster, but whether they are interacting in a way that deserves to be called racing.

This evaluation design also explains the shift in mentality visible in the project notes: the correct ordering was control and safety first, competition second. Without that ordering, the system can produce superficially aggressive agents that are merely learning to occupy space destructively. With it, multi-agent traversal becomes a genuine extension of the single-agent controller rather than a collapse of it.

For the broader report, the multi-agent branch therefore serves as the bridge between two research questions. It shows how direct PPO had to become relational and safety-aware before dense racing could be studied seriously, and it motivates why a richer world model later became appealing: once interaction structure matters, mere lane-following is no longer a sufficient representation of the task.

The world-model branch asked a more structural question than the PPO branch. Instead of learning a direct control policy, the model attempted to compress racing into a latent dynamical system that could reconstruct observations, predict rewards, and carry enough physical structure to support imagination. The implementation centered on an RSSM with an image encoder, a deterministic GRU state, stochastic prior and posterior heads, a decoder, and telemetry predictors for speed, progress delta, steer, corner angle, and off-track probability.

Intuitively, the challenge was not simply to redraw a road. The model had to learn the difference between a stable world and a moving camera. A corridor can remain visually plausible while the imagined ego vehicle fails to progress through a turn in a physically meaningful way. That distinction explains why some later runs looked geometrically tidy but still failed on the curated `sharp_turn` clip.

The choice of RSSM was deliberate. For this project, the representation had to satisfy four constraints simultaneously: it had to remain tractable on modest replay budgets, it had to support long-horizon imagination from pixels, it had to expose a latent state that could later feed an actor-critic, and it had to remain debuggable enough for intervention-based iteration. An RSSM is a natural fit for that design space because it explicitly separates a deterministic memory state from a stochastic uncertainty state. The deterministic branch stabilizes medium-horizon geometry; the stochastic branch absorbs ambiguities in future motion and local observation.

The deterministic core was implemented with a GRU rather than a larger transformer-like temporal model because the project's immediate objective was controlled hallucination development, not maximal-scale world-model pretraining. A GRU provides exactly the inductive bias needed for repeated action-conditioned updates on short replay windows: it is sequential, lightweight, and easy to stabilize with LayerNorm. In a racing environment where local continuity matters and data are relatively narrow compared with internet-scale video corpora, that bias was an asset rather than a limitation.

The sequence model therefore followed the familiar latent-dynamics pattern

Training optimized reconstruction, reward, KL, and auxiliary telemetry losses:

From a high-level ML perspective, the GRU should be read as a strategic simplification. The project needed a recurrent memory mechanism that could absorb track continuity, action history, and ego-motion cues without introducing a second difficult scaling problem before the first one was solved. The repository's own intervention history justifies that choice: the main bottleneck was not that the temporal core was too exotic or too small in the abstract, but that pose evolution and progress modeling remained weak even after replay, horizon, and objective refinements.

In other words, the RSSM/GRU choice was not accidental minimalism. It was the right baseline for a project whose immediate goal was to establish faithful hallucination and only then move to policy learning in imagination. The architecture was simple enough to diagnose and modify, yet expressive enough to produce nontrivial gains from `E1` through `P5`.

The present implementation is best understood as closer in spirit to the classical Dreamer lineage than to a full modern Dreamer system. DreamerV3 demonstrated that world-model reinforcement learning can scale across many domains with fixed hyperparameters, and it trains an actor-critic inside latent imagination rather than treating hallucination as an endpoint. Our project adopted the world-model side of that agenda first: recurrent latent state modeling, reconstruction, reward prediction, and action-conditioned rollout analysis. It did not yet complete the full DreamerV3 loop of robust behavior learning inside the imagined latent dynamics.

The more recent Dreamer 4 agenda goes further still. Its emphasis is not merely on learning a usable recurrent latent model, but on learning agents inside a fast and accurate world model at larger scale, including stronger object-interaction fidelity and more scalable imagination training. Relative to that standard, the current repository should be seen as a pre-Dreamer4 stage: it built the hallucination and telemetry foundation, but it did not yet reach the point where latent actor-critic training could be trusted as the main learning engine.

This comparison is useful because it clarifies the roadmap. A more DreamerV3-like next version would tighten the latent actor-critic loop around the current RSSM and improve normalization, replay coverage, and imagination objectives. A more Dreamer4-like next version would likely require a broader shift toward a higher-capacity world model that is better at procedural variation, contact-rich interactions, and large-scale action conditioning, potentially moving beyond the current GRU-centered temporal core.

The final world-model story is more subtle than "the model blurres turns." In the strongest checkpoints, the model frequently preserves the road corridor, keeps the ego car visually plausible, and predicts short-horizon reward and telemetry with genuine competence. The failure is narrower: under the `sharp_turn` probe, the model often continues to render a believable scene without advancing the camera through the turn as a driver would actually experience it.

That pattern explains the progression from data interventions to structural interventions. If the decoder were globally poor, more telemetry or more data would have been enough. Instead, the evidence points toward a representational mismatch. The model appears better at describing "what a road looks like next" than "how the agent's viewpoint moves through a turning road." This is exactly why the later logs reframe the next experiment around a pose-conditioned branch rather than another broad continuation run.

The original ambition of the world-model branch was not merely to produce visually pleasing hallucinations. The intended downstream milestone was latent actor-critic training inside the learned simulator. That downstream goal is already visible in the repository through `world_model_train_control.py`, which freezes the learned RSSM and trains a latent actor and critic on top of it. The fact that this diagnostic did not yet transfer well is therefore not a side note; it is the clearest indicator of what the next phase must solve.

Three critical factors now define the frontier. First, the world model must preserve pose evolution well enough that progress through turns is represented faithfully rather than cosmetically. Second, it must remain accurate under richer procedural variation: more track geometries, more interaction modes, and more diverse replay than the current relatively narrow offline datasets. Third, it must be accurate enough that imagined actor-critic optimization does not exploit model errors faster than it learns driving behavior.

Concretely, a near-term extension would be a more DreamerV3-like phase in which the current RSSM is upgraded from a hallucination benchmark into a real imagination training substrate. That would mean revisiting the latent actor-critic pipeline after improving pose evolution, replay diversity, and reward-faithfulness under imagined rollout. The project already has the scaffolding for this step; what it lacks is a world model accurate enough to justify trusting the imagined gradients.

A more ambitious follow-up would be a Dreamer 4-like rearchitecture aimed at richer procedural generation and stronger interaction modeling. In this report, "procedural" should be read practically: not only more track variation, but also more consistent action-conditioned world evolution across novel layouts and race situations. That level of generality is exactly what would make latent actor-critic training valuable for the next generation of the project, especially once the goal shifts from single-car hallucination toward competitive multi-agent imagination.

The repository should therefore be read as a benchmark record. The positive results are real but scoped. PPO control is operational and documented through a transparent promotion ladder. The world model has crossed the threshold from generic hallucination to meaningful telemetry-aware dynamics, but the evidence does not justify a claim that it yet supports robust imagination through hard turns or reliable downstream control.

This limitation is scientifically useful rather than disappointing. It sharply narrows the next hypothesis: an improved world model will likely require explicit pose evolution, not merely a larger latent or another objective tweak on the current fixed architecture.