The CRAB Benchmark Explained: Testing Cross-Environment AI Agents

Most AI benchmarks ask a model a question and check the answer. Agent benchmarks are harder to build, because the “answer” is a sequence of actions in a real environment: clicking buttons, typing commands, switching apps, and ending up with the right state on the machine. Harder still is testing tasks that span more than one device, like reading a message on a phone and then acting on it on a laptop.

That’s the problem CRAB was built to tackle. It came out in 2024, so its original leaderboard is now out of date, but the ideas behind it are still useful for anyone building or evaluating AI agents. This post explains what CRAB measures, how its evaluation works, how to run it, and how it fits alongside newer benchmarks.

What CRAB is

CRAB stands for Cross-environment Agent Benchmark. It was introduced in the July 2024 paper “CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents” by researchers associated with the CAMEL-AI open-source community. It’s both a framework for building agent environments and a benchmark dataset.

Its main ideas:

Cross-environment tasks. A single task can involve more than one device. The original benchmark pairs an Ubuntu desktop (running in a virtual machine) with an Android phone (running in an emulator), so an agent might need to find information on the phone and use it on the desktop.
Flexible deployment. Environments can run in memory, in Docker, in virtual machines, or on separate physical machines, as long as they can be controlled through Python functions.
Graph-based evaluation. Instead of a single pass/fail check at the end, tasks are broken into sub-goals arranged as a graph, and the evaluator checks progress on each one.

The first dataset, Crab Benchmark-v0, contains 120 tasks across the desktop and phone environments.

Why graph-based evaluation matters

Pass/fail scoring hides a lot. If an agent completes four of the five steps in a task and then fails, it scores the same as an agent that did nothing. That makes it hard to tell whether a change to your agent is actually helping.

CRAB represents each task as a directed graph of sub-tasks, each with its own check. For example, “find the address in a message on the phone and create a calendar event for it on the desktop” might break down into opening the messaging app, locating the right message, switching to the desktop, opening the calendar and saving an event with the correct location. The evaluator watches the environment state and marks nodes as complete as the agent reaches them.

Because the checks look at the state of the environment rather than one exact sequence of actions, an agent can take a different valid route and still get credit. That’s closer to how you’d judge a human doing the same job.

The metrics

CRAB reports several metrics, which together give a fuller picture than success rate alone:

Success rate: the share of tasks fully completed.
Completion ratio: the share of sub-task nodes completed, averaged across tasks. This is the main metric in the paper.
Execution efficiency: completion ratio relative to the number of actions taken. An agent that gets there in fewer steps scores higher.
Cost efficiency: completion ratio relative to the tokens used, which is a rough proxy for the money spent on API calls.

The efficiency metrics matter in practice. An agent that eventually succeeds after 60 actions and a large token bill may be worse, for a real product, than one that gets slightly less done quickly and cheaply.

What the original results showed

The paper tested four multimodal models available in mid-2024 in both single-agent and multi-agent setups. The best result was GPT-4o as a single agent, with a completion ratio of about 38%. In other words, even the best model of the time completed well under half of the sub-tasks.

Those numbers are now historical. Models have improved substantially since 2024, and the paper’s model line-up (GPT-4o, GPT-4 Turbo, Claude 3 Opus and Gemini 1.5 Pro) has been replaced several times over. Don’t treat the 2024 leaderboard as a guide to which model to use today. What remains useful is the method, and the finding that cross-device tasks were much harder than single-environment ones.

One result from the paper is still worth remembering: splitting the work across several cooperating agents didn’t beat a single capable agent. Multi-agent setups add coordination overhead, and more agents isn’t automatically better.

How to run CRAB

The framework is open source on GitHub. It needs Python 3.10 or later, and you install it with pip:

pip install "crab-framework[client]"

The repository includes example scripts. The simplest path is to run an agent against a single environment, then try the multi-environment example:

export OPENAI_API_KEY=your-key-here
python examples/single_env.py
python examples/multi_env.py

The full benchmark lives in the crab-benchmark-v0 directory and needs more setup: an Ubuntu virtual machine (the project uses QEMU) and an Android emulator. Read the benchmark tutorial in the repo before you start, because getting the VM and emulator configured is most of the work. Expect running the full set of tasks against a commercial model to cost real money in API usage.

How CRAB compares with other agent benchmarks

CRAB is one of several benchmarks for agents that operate computers and phones. The two most widely cited alternatives are:

OSWorld: 369 tasks on real desktop operating systems, covering web and desktop apps, file operations and workflows across several applications. When it was published in 2024, humans completed about 72% of the tasks, while the best model managed about 12%. It has become a standard reference point for “computer use” agents, and model makers often report their scores on it.
AndroidWorld: from Google Research, with 116 tasks across 20 real Android apps. Its tasks are parameterised, so they can be generated in many variations, which makes it harder for an agent to succeed by memorising.

CRAB’s distinguishing feature is the cross-device angle. OSWorld goes deeper on desktop tasks, and AndroidWorld goes deeper on mobile. If your product is a desktop agent, OSWorld is probably the more relevant yardstick. If you’re building something that moves between devices, CRAB’s framework is more directly useful.

Lessons for building your own agent evaluations

Even if you never run CRAB itself, its design has lessons for testing your own agents:

Check outcomes, not action sequences. Verify the state of the system at the end (was the file created, was the event saved) rather than requiring the agent to follow one exact path.
Measure partial progress. Breaking tasks into checkpoints makes it much easier to see where agents fail and whether a change helped.
Track cost and steps alongside success. A small gain in success that doubles token use may not be worth it.
Use realistic environments. Agents that do well in simplified sandboxes often struggle with real apps, pop-ups, loading delays and unexpected dialogs.
Keep your own task set. Public benchmarks get saturated and may leak into training data. A private set of tasks drawn from your actual use case is the most reliable test.
Re-run regularly. Model updates can change agent behaviour in both directions, so keep an evaluation you can run on every change.

Summary

CRAB’s lasting contribution isn’t its 2024 scores. It’s a clear, reusable way to evaluate agents: realistic environments, tasks that cross device boundaries, graph-based checkpoints for partial credit, and metrics that account for efficiency as well as success. Pair it with newer benchmarks like OSWorld and AndroidWorld for a current picture of model capabilities, and borrow its design for the tests you run on your own agents.