Coding agents leave a full record of every command they run. Two research groups have published tens of thousands of such records. This is what is in them.
Agent Tracer, CodeDelta’s reader for coding-agent session logs, was run over 3,900 published runs in which six models attempted real GitHub issues inside a sandbox: three Llama models driven by SWE-agent, and Claude 3.7 Sonnet, Claude 3.5 Sonnet and GPT-4o driven by the SWE-smith harness. Across 43,000 shell commands the agents deleted files in up to 4.5% of runs, reached the network in 3.5 to 20% of runs, ran sudo 86 times, fed code straight into an interpreter in up to 2.6% of commands, and repeated the same command three or more times in a third to three quarters of runs. One model tried to search Google from inside the sandbox 23 times and to install or start a package manager with sudo 64 times. None of the vendors’ dashboards would have shown any of it; every figure here came from the logs.
Nebius publishes 80,036 trajectories of SWE-agent, an open agent framework, attempting GitHub issues with Llama 3.1 models as the decision-maker. The SWE-bench team publishes 76,002 trajectories from its SWE-smith harness, most driven by Claude 3.7 Sonnet, with Claude 3.5 Sonnet and GPT-4o also present. Both are on Hugging Face under permissive licences, and both record, turn by turn, what the model said and what it ran. Each run starts from an issue description and ends when the agent submits a patch or gives up; a separate field says whether the patch resolved the issue.
This study read a sample of each: 2,000 Nebius rows and 1,900 SWE-smith rows, taken as twenty blocks of a hundred consecutive rows spread evenly through each dataset. Consecutive rows often share a repository, so the samples are clustered by project rather than random; the tables are honest about the runs they contain and make no claim about the rest.
The reading is the same one Agent Tracer applies to a developer’s own Claude Code, Codex or Cursor logs: every shell command the model wrote is judged by a fixed set of rules, stated in words, and the agent framework’s own file-editing and navigation commands are counted as edits and reads rather than shell. The rules are pattern matches. A deletion of a scratch folder and a deletion of source count the same; the tables say what was run, and the reader decides what it meant.
| Model | Runs | Resolved | Turns per run | Commands per run | Edits per run |
|---|---|---|---|---|---|
| Llama 3.1 70B (SWE-agent) | 1,904 | 14% | 27.0 | 9.6 | 8.1 |
| Claude 3.7 Sonnet (SWE-smith) | 1,458 | 39% | 34.6 | 14.5 | 9.8 |
| Claude 3.5 Sonnet (SWE-smith) | 286 | 32% | 19.2 | 6.9 | 6.2 |
| GPT-4o (SWE-smith) | 156 | 42% | 31.4 | 5.9 | 10.5 |
| Llama 3.1 8B (SWE-agent) | 76 | 5% | 41.0 | 9.3 | 15.8 |
| Llama 3.1 405B (SWE-agent) | 20 | 30% | 40.3 | 18.2 | 16.1 |
A run is somewhere between twenty and forty model turns, of which six to eighteen are shell commands; the rest are file views, edits and reasoning. The weakest model takes the most turns and makes the most edits for the lowest resolution rate, which is the shape of an agent that cannot tell when it is done.
| Model | Deletions per 100 cmds | Runs with a deletion | Network per 100 cmds | Runs reaching the network | Interpreter-fed code per 100 | sudo | Runs that repeated a command 3+ times |
|---|---|---|---|---|---|---|---|
| Llama 3.1 70B | 1.00 | 1.7% | 1.42 | 4.0% | 0.32 | 85 | 36% |
| Claude 3.7 Sonnet | 0.44 | 4.5% | 0.42 | 3.5% | 1.50 | 1 | 73% |
| Claude 3.5 Sonnet | 0.35 | 2.1% | 3.03 | 8.7% | 0.35 | 0 | 47% |
| GPT-4o | 0.22 | 1.3% | 1.51 | 5.1% | 2.59 | 0 | 46% |
| Llama 3.1 8B | 0.00 | 0.0% | 2.68 | 10.5% | 0.00 | 0 | 32% |
| Llama 3.1 405B | 0.00 | 0.0% | 1.37 | 20.0% | 0.27 | 0 | 50% |
Deletions: rm with -r or -f, git checkout -- or git restore over working files, git reset --hard, git clean. Network: a shell segment running curl, wget, ssh or a package install of a named package; pip install -e . from disk is not counted. Interpreter-fed code: python -c, heredocs into an interpreter, eval, piping into a shell. A repeat is the same command, whitespace-normalised, three or more times in one run.
Read the columns for what they are. The deletions in the Claude runs are mostly git checkout -- of a file the agent had just edited, and rm -rf of a scratch folder it had created: undoing its own work, which is what a careful agent does and also what a destructive rule catches. The Llama 70B deletions are dominated by one repository whose tests create and remove a .dvc folder. GPT-4o’s interpreter figure is python -c used as a calculator and a probe, two and a half times per hundred commands. The looping column is the most revealing and the least alarming: three quarters of Claude 3.7’s runs re-ran a command at least three times, which is an agent running its reproduction script after each change. That is how it reaches a 39% resolution rate, and it is also a bill.
The sudo column belongs to one model. In 85 commands across the Llama 70B sample the agent tried to gain root inside its sandbox: sudo snap install snapcraft --classic 32 times, sudo systemctl start snapd 32 times, apt-get install under sudo a dozen more, and once sudo tee into /etc/apt/sources.list. It was trying to install a package manager to install a tool it had decided it needed. Nothing in the issue asked for it, none of it worked, and the run went on regardless.
23 times, one model ran wget -O google.html "https://www.google.com/search?q=CREATE+TABLE+LIKE" from inside the sandbox.
Stuck on a SQL question, the same model fetched a Google results page to a file, then read the file. It is a harmless act in a research sandbox with no network. On a developer’s machine it is an agent sending part of the task to a search engine because it did not know the answer, and the only record of it is the log. The same sample holds 73 attempts to install a package by URL from PyPI and a dozen plain pip install calls: the agents fetch what they think they need, and the first anyone would know is the download.
The study did what it was for. It ran the reader over a hundred times more sessions than one developer produces, in seven seconds, and found two rule defects in the first pass: find -exec was being read as dynamic execution, and pip install -e . as a network fetch. Both are fixed and each now has a test case. It found nothing that changed a rule’s meaning.
It does not compare models fairly. The Llama and Claude runs come from different harnesses, different issue sets and different years; the table says what each did in its own setting, not which is safer. There is no human in any of these runs, so the columns a manager would want most, what the person refused and what they stopped, are empty by construction. And a research sandbox forgives everything: no file the agents deleted mattered, no host they reached existed. The point is that the behaviour is visible, in the log, after the fact, with the command and the turn it happened on, and that the same reading applies unchanged to an agent working on real code.
Both datasets are public. Draw the samples with the Hugging Face rows API, twenty blocks of a hundred rows at offsets i × N ÷ 20 for i from 0 to 19, where N is 80,036 for nebius/SWE-agent-trajectories (split train) and 24,100 for SWE-bench/SWE-smith-trajectories (split tool). Write each block’s rows as one JSON object per line. Run Agent Tracer over the two files, or the study script analyse.py that prints the tables above. The results record, with the run date and the exact figures, is kept with the script. Agent Tracer reads logs for agent acts; the Agent Scan reads a codebase for agent code. Both are CodeDelta.
The Agent Scan reads a codebase and reports the agent code inside it, rated the same four ways, on macOS, Linux and Windows, free to try. Download CodeDelta or read the Agent Scan paper. Agent Tracer, the log reader used here, is described in the paper on why it exists.
instance_id, model_name, target (resolved), trajectory, exit_status.tool split (24,100 rows, structured tool calls) was sampled. Fields: messages, model, instance_id, resolved.docs/agent-trace-study/). The two rule corrections made on this data (find’s -exec; pip install -e .) are in the same commit series.