We built this because:

• Agents are cheap to run, infinitely patient, and easy to parallelize. These properties make exhaustive monitoring feasible at production scale.

• Teams hate owning observability, ourselves included. Offloading this grunt work to an agent lets us ship faster.

• Strong observability and QA mean fewer bugs, less downtime, and a better experience for Ramp customers.

Ramp Inspect

We chose to build our maintenance system using Ramp Inspect, our internal background coding agent. Each Inspect session spins up a full sandboxed dev environment, allowing the agent to make real API requests, run tests, and reproduce bugs end-to-end against live code. This interactivity is critical, as subtle failure modes are rarely apparent from static code review.

Scheduled auditing

Our first attempt at self-maintenance took the form of a scheduled agentic review. Every night, we automatically spun up an agent to run a QA pass on Ramp Sheets, instructed to:

• Sanity-test core features

• Stress-test recently merged PRs

• Probe existing functionality for latent bugs

This design worked well, and surfaced several real production bugs every day. Further, we designed the workflow to be a system of action: If the agent found a real issue, it would put up a PR to address the root cause. More than once, an engineer shipped a feature with a bug they hadn’t noticed; by morning, the agent had caught the regression and pushed a fix.

However, the nightly agent had serious limitations. With no specific mission, the agent always progressed down the same paths in its QA workflow, reading the same files, investigating the same features, running the same tests. While this pattern proved effective at finding high-radius issues, it could not catch narrow, situational bugs.

Although Ramp Sheets is extensively instrumented on Datadog, our unfocused QA agent struggled to extract nuanced insights from the telemetry. Current frontier models are very capable at a wide range of software engineering tasks, but they cannot synthesize a large codebase with a large observability surface and determine what needs attention. Prioritization at production scale requires a level of intelligence surpassing that of any model available today.

Monitor-driven maintenance

For our next iteration, we used Datadog monitors to direct the agent at specific production issues. These monitors watch metrics and log patterns, firing alerts on error rate spikes, latency regressions, or other deviations from expected behavior. The system works in two steps:

• On PR merge, an agent reads the diff and generates monitors instrumenting the new code.

• When a monitor fires, a Datadog webhook kicks off a new agent with the alert context. The agent reproduces the issue in its sandbox, pushes a fix, and notifies us on Slack.

This pattern allows us to detect and respond to incidents with remarkable speed and granularity. In its first week, the system caught 40 real bugs, each within minutes of a user triggering the issue. In one instance, a user uploaded a spreadsheet with a unique type of embedded image that our existing logic could not handle; the resulting exception set off a monitor and moments later, the agent had alerted us with a fix ready. In another case, an internal user Slacked us about a broken feature - but our system had already flagged the exact issue before his message even landed.

Filtering the noise

The major weakness in this approach was noise. Auto-generated monitors have bad thresholds. Routine user activity triggered a cascade of alerts, most of them false positives. Further, monitors fired repeatedly for the same issue, flooding Slack with duplicate notifications.

To filter the noise, we added a triage step. On every alert, the agent first assesses the scope of the problem:

• If it’s a real issue, the agent pushes a fix and posts to Slack.

• If it’s noise, the agent tunes or deletes the monitor.

To guard against duplicate alerts, we store state on the monitor itself. When an agent pushes a fix, it appends the PR link to the monitor description. Subsequent agents see the link and stand down.

The upshot

In a few weeks, we scaled Ramp Sheets from ten hand-written monitors to over a thousand, one for every 75 lines of code. Our manual monitors were broad-strokes: alert when the frontend crashes, or when an API call times out. The AI-generated ones are far more granular, acting like a tight mesh over the exact shape of the code; whenever that shape drifts from the expected, we are notified.

What we learned

Detect everything, notify selectively. The system should watch every signal, but each alert that reaches a human should mean something. Teams ignore noisy monitors, and they’ll ignore noisy agents too.

Delegate to the agent. Let it scope out the problem, judge impact, make changes, and filter out noise. It’s very good at this, and will get better as models improve.

Sandboxed reproduction improves results. In our system, the agent reproduces the failure against live code and only pushes a fix once that reproduction test passes. This pattern ensures that the issue is real, and that the agent’s proposed fix works in practice.

Model choice matters. Although the GPT-5 model series are very thorough debuggers, we found Opus 4.6 was a more accurate triage evaluator, and specifically better at filtering out noisy alerts.

Tight observability breeds customer empathy. When every slow load or bad output fires a notification, the team feels the product the same way users do. We caught bugs we never would have prioritized on our own.

Keep your existing stack. Auto-generated monitors are powerful but opaque, and not yet reliable enough to be your only line of defense. When things go seriously wrong, you still want instrumentation you wrote and trust. As models improve, that will change.

Want to keep up with our next AI experiments? Subscribe here and follow us on @RampLabs. We’re also hiring across roles at Ramp.

Subscribe now

Watch Claude Code inside RollerCoaster Tycoon on Youtube.

The park rating is climbing. Your flagship coaster is printing money. Guests are happy, for now. But you know what’s coming: the inevitable cascade of breakdowns, the trash piling up by the exits, the queue times spiraling out of control. You could stay glued to the screen, frantically clicking through management windows. Or you could let an AI agent take the wheel.

In a terminal window in the corner of the screen, Claude Code springs to life. It scans 100 data points you haven’t had time to read: park financials, ride breakdowns, guest complaints about bathroom lines. Within seconds, it compiles a task list: build more Drink Stalls near high-traffic areas, hire mechanics before the coasters start failing, and raise the park entrance fee while satisfaction is high.

In this article we’ll tell you why we decided to put Claude Code into RollerCoaster Tycoon, and what lessons it taught us about B2B SaaS.

Why Rollercoaster Tycoon?

At Ramp, we’re building agents across product surfaces and internal operations. Our current approach is small multiples: task-specific agents with tightly scoped system access. This is a security posture as well as an optimization around the current limits of agent intelligence and context windows.

But with each task-specific agent we build, there’s a Promethean urge to create the One Agent with unfettered access to everything. It’s still too early to build this way, but the latest models are pushing this limit; it’s increasingly a security and integrations issue, not an agent limitation. This is why now is the perfect time to start experimenting with broadly scoped agents in toy environments.

There’s plenty of precedent for AI experimentation in videogames; from DeepMind’s work in Atari games and StarCraft to more recent demos in Minecraft and Pokemon. But each of these games is a poor representation of the kind of environment Ramp and our customers exist in. Minecraft is an open sandbox lacking any capitalist structure. Pokemon is about a child too young to open an LLC or apply for a credit card. StarCraft incorporates economics, competition and workforce management, but it’s not very customer-centric.

Ramp needed a game that closely approximates customer-centric business operations and SaaS-powered digital feedback loops. There was simply no other choice. We had to put Claude Code in RollerCoaster Tycoon.

A common misconception about RollerCoaster Tycoon is that it is first and foremost a game about “rollercoaster game”. In reality, the rollercoasters often fade into the background of a sea of park management windows. The game is a Montessori play kit of B2B SaaS interfaces.

And the vibes match is perfect; coding agents like Claude Code, in their humble terminal windows, convey a retro-futurism and Claude in particular brings serious playfulness. It’s a great agent to send back in time so that we can begin to understand how to re-write two decades of business software.

What Claude Can (and Can’t) Do

OpenRCT2 is an open-source re-implementation of RollerCoaster Tycoon 2. We forked the project to add a new kind of window into the game; a terminal running Claude Code.

In order for Claude Code to play the game, we had to create an expansive CLI covering all the actions the user has available in the game, from reviewing guest feedback to changing umbrella prices and everything in between, The CLI is called rctctl and follows the patterns and conventions of kubectl, the expansive CLI behind Kubernetes, one of the most complicated pieces of software ever created.

The CLI replicates every important data point, control and action the user has available, plus some additional commands to review park tiles.

In place of ‘seeing’ the map, Claude can request ASCII grid print-outs of the map at multiple levels of granularity, from ‘hot regions’ to tile grids, to single tiles.

Here’s an example of a dense region of the park.

$ rctctl map area --x 44 --y 38

Map Area
--------
Anchor       : (44, 38) top-left
Span         : 16x16 tiles
    X:44 45 46 47 48 49 50 51 52 53 54
Y 38   R  P  P  P  P  .  .  E  E  .  T
  39   P  P  .  S  P  S  T  P  Q  S  .
  40   P  R  S  R  P  P  .  P  Q  S  .
  41   P  S  S  S  .  P  P  P  P  P  P
  42   P  .  S  S  S  P  .  .  .  .  .
  43   P  S  T  T  S  P  .  .  .  .  .
  44   P  S  S  S  S  P  .  .  .  .  R
  45   P  .  R  R  R  R  R  R  R  R  R
  46   P  R  R  R  P  P  P  P  P  P  P
  47   R  R  R  S  Q  Q  S  Q  Q  Q  Q
  48   R  R  S  S  E  Q  Q  Q  Q  Q  Q

Legend
------
 - R = Ride track/support
 - P = Footpath
 - . = Owned ground
 - E = Ride or park entrance
 - T = Tree or foliage
 - Q = Queue path
 - S = Scenery/building

So Claude is at a pretty steep visuo-spatial disadvantage, and this is most of the intuition you’ll need to appreciate Claude’s relative strengths and weaknesses in the game.

Where Claude excels:

Game knowledge. Claude is surprisingly familiar with all things RCT, and also completely unfazed by the premise that it has been ‘hacked into’ a late-90’s computer game. This was surprising, but fits with Claude’s playful personality and flexible disposition.

Gathering information. Claude excels at trawling through the game’s diverse metrics and observability features. Claude switches between empathizing with aggregate guest thoughts and scrutinizing ride financials. It’s great at creating insightful reports with specific recommendations.

Pulling Digital Levers. Claude is rock solid at adjusting configurations. Opening and closing rides, setting prices, hiring staff (which are automatically placed in RCT2), and starting marketing campaigns. These actions don’t require spatial reasoning, and they’re most similar to traditional CLI operations.

Placing Shops/Stalls and Flat Rides. Nearing the edge of Claude’s in-game competencies are tasks that require a basic spatial understanding of the park. Even finding main pathways can take several iterative steps and cumulative reasoning. Claude can reliably place Bathrooms, Drink Stalls and other stalls. It can even take a swing at flat rides like the Carousel which requires the additional complexities of placing a ride entrance, exit and connective pathways. But flat rides are where things start to feel sketchy and the success rate is closer to 80% with a high risk of small failures with park consequences.

Where Claude struggles:

Pathways and connections. Locating main pathways, routing new paths, and connecting ride entrances/exits are all serious challenges for Claude. The added complexities of obstacles like rides, trees, fences, and terrain slopes can quickly combine to overwhelm Claude’s loose grasp on the spatial reality.

Roller coasters. Placing larger pre-built rollercoasters is another heavily visual task in the game. Ride placement requires spatial reasoning at multiple scales, from relative distance to paths, to individual scenery obstacles, and large barrier structures like hills. Claude often gives up on optimal positioning and just flings rides to greenfield territories far away from main paths, and then fails to build the necessary long pathways to connect these endeavors. It’s an expensive mode of failure, and Claude’s main obstacle to a full playthrough.

Verticality. Claude’s spatial reasoning in the game is already at its limits in two dimensions. The third spatial dimension of inclined ground, underground construction, and any custom rollercoaster design work is basically out of the question. Unfortunately, this is some of the game’s richest gameplay.

The key insight: The informational interfaces of RollerCoaster Tycoon are extremely clean, and Claude is super sharp with them. The digital levers are well within Claude’s capability. And Claude is capable of managing park staff and dropping down simple amenities like Drink Stalls, Restrooms, and Benches. Claude falls short on the big moves, like large new coasters and pathway thoroughfares. But these deficits clearly stem from the awkward spatial interface.

As a mirror to real-world agent design: the limiting factor for general-purpose agents is the legibility of their environments, and the strength of their interfaces. For this reason, we prefer to think of agents as automating diligence, rather than intelligence, for operational challenges.

The Build: Modding Claude Code into RCT

OpenRCT2 is an open-source re-implementation of RollerCoaster Tycoon 2 in C++, extended with fixes and features for nearly a decade by a passionate community.

We don’t know any C++ at all, and we vibe-coded the entire project over a few weeks. The core pieces of the build are….

• A new menu item in the game’s menu bar

• A new window type displaying a terminal mirroring a remote terminal running Claude Code.

• An expansive CLI (rctctl) that Claude uses to interface with the game.

• RPC layer to transmit CLI commands into the game state.

• Tests (or so we think)

We started with a plan from ChatGPT o3-Pro Deep Research (read: ChatGPT on max settings), broke it down into a roadmap with Claude Code, and got to work managing four terminals of Claude Code to execute against the roadmap

After three hours the build was stuck, so we threw the whole thing away and started over.

We started the second iteration with Codex on GPT-5.1-codex. We also broke things down into smaller chunks, used careful planning phases, and kept the context above the ~60% remaining level where coding models perform at their absolute best. The only other notable setback came was an accidental use of the word “revert” which Codex took literally, and ran git revert on a file where 1-2 hours of progress had been accumulating.

Keeping all four agents busy took a lot of mental bandwidth. Each implementation phase produces a summary of the work done, and the planning phases output ~500 words. Research tasks produce lots of prose to critically review, and sometimes we couldn’t remember what the initial questions or work threads were by the time we were reviewing them. The experience overall is less like programming and more like… a management simulation game. It’s vaguely addicting. The speed of progress and looseness of focus amplifies the reward of each new feature shipped.

The full project took ~40 hours over several weeks, which was far more time than we initially expected, but this time included planning, design, and tons of scope creep, all playing out in the terminals. The biggest accelerant to coding progress would have been tighter feedback loops; ways for each coding agent to QA its own work, and not just think that the feature was working as described.

Open Issues

Like any large vibe coded project, it’s very hard to tell what works, and what doesn’t.

In-game Claude was my diligent playtester, and an integrated bug-report tool let it write bug reports directly into the repo for easy review by coding agents. This feedback loop was invaluable for ironing out help text, inconsistencies in the CLI, and observed bugs.

It’s likely that several surfaces have strange or incomplete behaviors. However, at this point, in-game Claude rarely has any bugs to report on, which I take as a moderately reliable signal that the implementation is working.

What We Learned

Building this project reinforced a few principles we’ve seen elsewhere in agent development:

Environment legibility is a key. Claude thrives with the clean and well-structured omniscience of RCT’s built-in monitoring and control surfaces, and clearly struggles with text-based renderings of game space. Inventive text-based representations are fun to imagine, but probably impractical in most cases. This is a boundary for many agent tasks, and is probably best tackled with stuff-as-code arrangements rather than simple text-based representations (like chess notations and markdown-mermaid diagrams).

Coding Agents are a tremendous tailwind; build sails. By putting ‘raw’ Claude Code at the center of the project we automatically and immediately benefited from several Anthropic releases even in the short span of the project. Claude Opus 4.5, fast auto-compaction, custom status lines, and fixes to terminal flickering all came in clutch and ‘for free’ (well, not the Opus upgrade).

Development Loops. Coding agents thrive on feedback loops where they can prove to themselves that the implementation is complete and correct. This project was hampered by a broken iteration loop in which we had to manually QA the in-game terminal experience and functionality. Vibe coding is slow and frustrating when QA is a manual process.

Experience beats study. LLMs may be Big Black Boxes to science, but each of us has in our own head a Bigger Black Box. Anybody with a few hours of experience with LLMs has already begun building an intuitive understanding of LLM capabilities. The best way to continue to develop this intuition is through play.

RollerCoaster Tycoon taught a generation of adults how to mind-meld with a graphical user interface. In this period of transition between GUIs and the future of intelligent computers, RCT makes for an interesting case study of the messy middle we occupy.

The park is running. Claude is at the controls. The guests say they are happy.

Run it yourself

The code is open source. Here’s how to get started.

Prerequisites

• macOS (Sonoma or newer) with Xcode, CMake 3.24+, and Ninja

• RollerCoaster Tycoon 2 (purchase on Steam)

• libvterm for terminal rendering: brew install libvterm pkg-config

Quick Start

# Clone the branch
git clone -b claude https://github.com/jaysobel/OpenRCT2.git
cd OpenRCT2

# Configure
cmake -S . -B build -G Ninja -DOPENRCT2_PREFER_STATIC=ON

# Build everything (game + CLI tools + terminal)
cmake --build build --target agent_bundle -j8

# Run
./build/OpenRCT2.app/Contents/MacOS/OpenRCT2

Once the game launches, click the Claude toolbar button to open the AI terminal.

What Gets Built

The agent_bundle target compiles:

• OpenRCT2 - The game itself with embedded terminal window

• rctctl - CLI tool that Claude uses to control the game via JSON-RPC

• Sprite assets - Graphics and UI elements

How It Works

The terminal spawns Claude Code with access to rctctl, a purpose-built CLI that communicates with the running game over JSON-RPC on localhost:9876. Claude sees the same information a player would—ride stats, guest happiness, park finances—and issues commands through this CLI rather than clicking buttons.

For more details, see:

• SETUP.md - Full setup guide

• CODING_AGENT.md - Technical deep dive

• RCTCTL.md - Complete CLI reference

Requires game assets from a legitimate RCT2 purchase. Copy game files to ~/Library/Application Support/OpenRCT2/ after installation.

Watch the livestream: On Twitch
Read the code: On Github
Join us: At Ramp Labs

We wanted to see how Reinforcement Learning with Verifiable Rewards (RLVR) behaves when trained on datasets with different levels of diversity. Does training on a varied dataset improve transfer learning or does the added noise make learning less stable?

What is Tinker?

Tinker is Thinking Machine Labs’ platform built for efficient fine tuning and inference of open source models. The goal: bring these workflows together so you don’t have to juggle separate systems.

In practice, this means you can:

• Host your model on Tinker

• Send inference calls

• Return rewards from those inferences

• And update your model using the reward

Tinker offers both synchronous and asynchronous APIs. By default you might use sync for simplicity, but for higher throughput you can switch to async. In our experiment we used the async functionality, streaming sampled outputs from Tinker to our reward function as they were generated, rather than waiting for full batches to complete. That allowed immediate feedback, quicker iterations, and higher efficiency.

One Caveat

Tinker doesn’t (yet) let you host custom reward functions unless your “reward” is another model. That’s fine until you try something like SpreadsheetBench, the reward logic was too heavy to run locally, so we had to host it remotely on Modal. It worked, but added lag and friction. We eventually switched to an LLM as a judge reward to simplify the setup and make it easier to see how Tinker actually performs.

The Setup

For the main experiment, we used the Salesforce Webscale-RL dataset.

2,500 Q&A pairs from each math, social sciences, and natural sciences were selected.
2,000 for training, 250 for validation and test sets each.

We trained three domain specific models (Math, Social Science, and Natural Science) and a multi domain model trained on the combined data from all three single domain runs. Every experiment used the same hyperparameters, batch size, and group size to keep comparisons fair. The policy model we fine tuned in all four cases was Qwen-8B with LoRA rank 128.

Because the multi domain model was trained on all three datasets together, we gave it three times as many training iterations while keeping the batch size fixed. This ensured it saw the same data as the individual models combined. We used importance sampling with normalized group advantages to keep gradient updates stable across batches.

For evaluation, we used Qwen3-30B-A3B-Instruct-2507 as an LLM as a judge. It compared each model’s output to the reference answer and assigned a binary reward.

Training

The math specific model showed the fastest and most stable learning. Its reward curve settled quickly, which makes sense given how structured and well defined math reasoning tends to be.

Models trained on social and natural sciences were less steady, with noisier gradients and slower convergence. That probably comes down to their dependence on external knowledge, it’s harder to learn clean patterns when the task itself is full of context and ambiguity.

The multi domain model introduced volatility. Its training trajectory was less stable and often spiky, particularly early in learning. Yet this volatility wasn’t purely harmful: the multi domain model slightly outperformed the math only model on math evaluation. This suggests that exposure to varied reasoning forms (e.g., causal inference or narrative reasoning) may have helped the model develop more generalizable heuristics. Regularization effects likely also contributed, as multi domain exposure reduces overfitting to any single reasoning style.

Head to Head Comparison

To see how the models stacked up, we tested everything on a shared test set: the combined test data from all three single domain runs. Since we wouldn’t know a question’s domain in a real setting, we used an untrained Qwen3-4B-Instruct-2507 to predict the domain and route each question to the corresponding single domain model—functionally similar to an MoE system. The multi domain model just answered everything directly. For evaluation, we used each model’s best validation checkpoint and relied on Qwen3-30B-A3B-Instruct-2507 as an LLM as a judge to generate consistent binary rewards across runs.

Performance was roughly equivalent across systems. The domain classifier’s imperfect accuracy likely degraded the single domain system slightly, but not by much. The main difference was efficiency: the multi domain model took about three times longer to train.

Conclusion

From these experiments, one clear takeaway stands out: segmenting training by domain and running processes in parallel can make post training far more efficient. The single multi domain model didn’t deliver enough benefit from transfer learning to justify its heavy runtime and its learning was less stable overall. For large scale post training, this domain split approach could deliver substantial wall clock savings without sacrificing performance.

So what do we think of Tinker? The system made it remarkably easy to run inference and update models without getting lost in infrastructure work. It let us focus on the research instead of the plumbing. That said, native support for remote reward functions would be a real breakthrough, enabling heavier computational experiments without clunky workarounds. Overall, Tinker feels like a thoughtfully built platform that lowers the barrier to RL research making complex post training workflows accessible to individual researchers and smaller teams alike.

Want to keep up with our next AI experiments? Subscribe here and follow us on @RampLabs. We’re also hiring across roles at Ramp.

Subscribe now

Subscribe now

We built an internal finance agent that could automate hours of tedious accounting work in spreadsheets. But every time someone wanted to use it, they’d spend 30-45 minutes crafting the perfect prompt, uploading the right files, and resolving errors.

We asked: what if the agent could watch us work and write its own prompts?

So we built a second agent called the Architect. This agent watches screen recordings of accountants doing their jobs, extracts the workflow, identifies what external data is needed, and generates detailed prompts, all without human intervention. It then hands off everything to our Doer agent to execute the task.

The result: what used to be dreaded hours wasted on tedious tasks now takes a five-minute screen recording. This is second-order automation: automation that creates automation.

In this post, we’ll break down how we built this two-agent system for finance use cases and share performance benchmarks.

The problem with first-order automation

Most AI agents today have a prompt problem. Our internal finance agent could automate complex accounting workflows, but it required perfectly crafted prompts and the right context files. Here are some sample tasks:

• Monthly revenue reconciliation exercise by consolidating invoice and data warehouse data into Excel to prepare and upload a journal entry into ERP.

• Month-end booking of inter-company fund transfers from several tools into an Excel workbook to create and import a journal entry into ERP.

This creates a paradox where the agent saves time executing tasks but costs time in setup. Accountants had to become prompt engineers trying to describe their process and if they wanted to run the same process next month, they had to re-write the prompt manually.

We realized that the bottleneck wasn’t execution but instruction.

Second-order automation with Architect & Doer agents

Instead of focusing on making the internal finance agent marginally better, we needed another agent to setup the task itself. Our solution was a two-agent system where one agent generates instructions for the other.

The Architect agent watches screen recordings of users completing tasks. Using a large-context multimodal model, it:

• Extracts the step-by-step workflow

• Identifies when users paste external data or reference other files

• Generates detailed, structured prompts for the Doer agent

• Compiles everything into a reusable “Process”

The Doer agent (our internal finance agent) executes these instructions using specialized tools to search, manipulate, and validate spreadsheet operations.

As a result, our two-agent system could take a human-recorded workflow and, after watching it once, create an entirely reusable workflow without additional input.

Deep dive into the two-agent system

The Architect agent solves this blueprint problem. Its role is to watch a human work through a spreadsheet via screen recordings and break the process into clear, specific instructions for our Doer agent to use in the future. The Architect agent also notes when users paste in data from external sources, or understands if other context is required (e.g. the current month) before running the Doer agent.

All of the Architect agent’s outputs are compiled into a Process: a compiled set of steps, required files, and additional information that will eventually be passed along to the Doer agent for each run.

Architect agent flow

• User uploads screen recording of their workflow

• Gemini 2.5 Pro (1M token context window) processes video to extract actions and context

• Agent identifies required files and temporal dependencies (e.g. current month data)

• Generates structured prompt with step-by-step instructions

• Compiles into a Process that can be reused

Doer agent flow

This agent is simple and relies as much as possible on model intelligence to achieve tasks with minimal guide rails.

• Receives process with instructions and uploaded files

• Claude Sonnet 4.5 uses specialized tools to navigate and manipulate spreadsheets

• Executes each instruction with validation checks

• Returns completed spreadsheet

Performance results

Architect agent

While we haven’t yet developed a formal benchmark for the architect agent, we can see its impact in terms of workflow creation from non-technical users. Accountants simply upload a screen recording and come back to a ready-to-use process. The time savings are significant when compared with manual prompting for every run.

Doer agent

We tested against SpreadsheetBench on ~50 randomly selected tasks (3 test cases each):

• Soft-restriction tasks: 49.5% accuracy (vs. OpenAI’s 45.5%)

• Hard-restriction tasks: 32.5% accuracy (vs. GPT-4o’s 13.38%)

Tasks taking 1-2 hours now complete in under 10 minutes (a 6-12x speedup). In cases of partial completion, the agent still saves significant time before human intervention is needed.

The barrier to automation collapsed from weeks of engineering work to simply recording your screen.

Learnings

1. Second-order changes the scaling equation. First-order: Adding workflows scales linearly with human prompt-writing capacity. Second-order: Adding workflows costs near-zero human time once the architect agent exists.

2. Models have complementary strengths. Gemini 2.5 Pro’s 1 million token context window with stellar multimodal support allows us to capture key details from input videos to generate a process, while Anthropic’s Claude Sonnet 4.5 shines when it’s time get work done by being swift, precise, and economic.

3. Building production agents is still hard. Despite powerful models, we spent weeks iterating on system prompts, tool designs, and error handling. There’s no shortcut for testing, debugging, and refinement. The engineering matters as much as the models.

Our combined agent doesn’t just do work, it designs how work gets done. This unlocks a new scaling paradigm: instead of engineers building automations one by one, anyone who can record their screen can generate reusable workflows. The architect-doer pattern transformed automation from an engineering bottleneck into a self-service capability. The next frontier isn’t faster agents, it’s agents that teach themselves what to do.

Want to get early access to future AI experiments? Subscribe below and follow us on X @RampLabs. We’re also hiring across roles at Ramp.

Ramp Labs is the home for AI experiments from Ramp. Our focus is shipping AI tools for the finance community and sharing learnings on building with AI.

Today, we launched Agent Fill, our agentic PDF form filler, in alpha. Read about how we built it below.

Finance teams waste time filling out PDFs

No one dreads filling out PDF forms more than finance teams. Countless hours are wasted on repetitive data entry that requires painstaking attention to detail. That’s why we started to build Agent Fill to help our internal finance team move faster. Now, we’re sharing the tool with the community.

How we built this technically

When filling out a PDF form, humans visually navigate complex pages, interpret questions, and draw from their knowledge to input answers accurately.

A traditional approach to automate this process parses and engages with a form’s file structure directly. While these systems can get the job done in certain ways, they tend to be brittle, and lack the breadth or context required to handle the high structural variance and nuanced content found in financial documents.

To accomplish this in an AI-native way, Agent Fill is equipped with a diverse arsenal of tools.

We start with knowledge extraction from previously filled forms to give our agent the relevant context it needs to fill blank forms. Throwing raw PDFs at Gemini 2.5 Pro takes advantage of its efficacy in producing rich, nuanced output from multimodal data when schema constraints are not enforced.

To help our agent understand the form we want to fill in, we transform it to text with OCR and extract any present PDF widgets with pymupdf. This pool of data is fed to Claude Sonnet 4 for structured extraction. Claude excels with schema fidelity, making it the perfect choice in mapping our data pool to form fields, interpreting the structure of complex financial documents in a format that can be handed off to the agent.

When a human fills out a form on a computer, they click, type, and scroll in their document editor. With Anthropic’s experimental yet formidable computer use tool, Claude Sonnet 4 can do the same. Combining the layered context we generated earlier with screenshots of the form featuring bounding boxes over fillable fields, the agent has everything it needs to fill out the diverse array of PDF forms encountered by finance teams.

Agents are clearly capable, but financial documents can be nebulous and mistakes are inevitable. Our goal is not to squeeze perfection from non-deterministic AI models, but speedy completion of tedious forms. So, we built a simple PDF editor for a human to quickly review and edit the filled form before downloading.

Agent Fill pairs AI automation with human in the loop UX to fill out PDFs in minutes, not hours.

Learnings

1. The power of multimodal context. When we provided the AI agent with partially redundant context in multiple modalities (e.g. screenshot of the form, OCR text, PDF widget data), its effectiveness in executing complex tasks and reasoning improved significantly.

2. Our flexible agent architecture allowed us to optimize each step of the agent’s workflow. We were able to quickly experiment and swap out tools to increase accuracy in results.

3. Despite the effectiveness of AI agents, for critical financial workflows, a human in the loop is still necessary for the end-to-end experience. We built a PDF editor for the human to review the agent’s results.

Try out Agent Fill today: https://labs.ramp.com/agent-fill

Thank you to Shaiyon Hariri, Alex Shevchenko, Alex Stauffer, TK Kong, and Zack Pitcher for contributing to this project.

Want to get early access to future AI experiments? Subscribe below and follow us on X @RampLabs.

Subscribe now