--- title: "AISI's Transect Turns 13 Million Tokens of Agent Transcript Into Something You Can Read" pubDatetime: 2026-10-07T18:40:00.000Z description: "The UK AI Security Institute released Transect, an open-source tool that labels long agentic-evaluation transcripts with LLM judges and links every label back to the source. The catch is in the judges." tags: [aisi, evaluations, ai-agents, open-source, research-tools, ai-security, transect, 2026, 2026-q4, 2026-10] --- > [!tldr] TL;DR > AISI and Meridian Labs released [Transect](https://github.com/AI-Safety-Institute/transect), an MIT-licensed Python package that turns long agentic-evaluation transcripts into interactive timeline reports. It reads Inspect `.eval` logs or OpenClaw `.jsonl` exports, uses LLM judges to label transcript segments against a vocabulary you define, and links each label back to the source text so a reviewer can check it. The companion paper reports a case study on an AI research evaluation that produced nearly 13 million tokens, where agents turned out to spend most of their effort on operational work and manuscript-writing rather than sustained hypothesis generation. Worth adopting if you run agentic evals, with the standing caveat that the labels come from LLM judges and need spot-checking. An agentic evaluation that burns 13 million tokens produces one number at the end. That number tells you whether the agent succeeded. It tells you nothing about whether the agent spent its run doing science or doing file management. That is the gap [Transect](https://www.aisi.gov.uk/blog/transect-making-large-scale-agentic-evaluations-easier-to-understand) is built for. The UK AI Security Institute, working with Meridian Labs, released it today as an open-source Python package on top of their existing Inspect Scout framework. AISI frames it as a tool for following an agent's work, picking out what deserves a closer look, and checking your reading of it against the transcript itself. ## How Transect Works You give it transcripts and a vocabulary. The transcripts come from Inspect `.eval` logs or OpenClaw `.jsonl` exports. The vocabulary is yours to define: the categories of behaviour you care about finding. Transect runs LLM judges over the transcript, labels and classifies segments against those categories, and renders the result as an interactive HTML timeline. The design choice worth noting is the link back. Every label points to the span of transcript that produced it. A reviewer who doubts a classification can open the source and read what the agent actually did, rather than taking the judge's word for it. The [GitHub repo](https://github.com/AI-Safety-Institute/transect) shows roughly 30 commits on main and an MIT licence, so this is a young tool. ## The 13-Million-Token Case Study The companion paper, [Transect: Retaining Observability for Long-Horizon LLM Agent Evaluations](https://arxiv.org/abs/2610.08364), went up on arXiv yesterday. Toby D. Pilditch, Konstantinos Voudouris, Alexandra Abbas and Cozmin Ududec frame the problem as an "observability envelope": the window within which a human can still meaningfully follow what an agent did, which long-horizon runs and multi-agent delegation blow straight through. Their case study is an AI research evaluation producing nearly 13 million tokens. Transect's analysis found the agents spent most of their effort on operational work and writing up manuscripts rather than sustained hypothesis generation. If you only had the score, you would have concluded something about research capability. What you had was a measurement of secretarial throughput. That is the argument for this class of tool in one example. Multi-agent delegation makes it worse: work fans out across sub-agents, and the trace a human would need to read grows faster than any human's willingness to read it. A timeline that collapses the run into labelled segments is a way of deciding where to spend attention. ## Who Judges the Judge Transect's core mechanism is LLM-as-judge labelling, and that family of systems has documented reliability problems: verbosity and position bias, self-enhancement bias, results that don't reproduce across runs, and sensitivity to prompt and formatting changes. That literature predates Transect and says nothing about it; read it as skepticism about the method. Closer to home, a separate team published [Automated Transcript Analysis for Detecting Flaws in Agentic Benchmarks](https://arxiv.org/pdf/2607.27518) on 29 July, 70 days ago. They built similar automated-scanner machinery for a similar job, quality-assuring agentic benchmark transcripts, and reported plainly that the scanners miss cases and that performance "varied substantially across benchmarks, criteria and models." That is an adjacent effort, not a rebuttal. It does establish that automated transcript analysis in this space has acknowledged failure rates. To AISI's credit, the architecture concedes the point. The whole reason every label links back to the transcript is that the labels are a starting point for human review, not a verdict. A reviewer who treats the timeline as ground truth and skips the source text is using the tool wrong. Whether evaluators under time pressure will actually click through is a question the design can't answer for them. ## Shared Evaluation Infrastructure Transect is the third layer in a stack: Inspect, then Inspect Scout, now Transect. The stack spans two governments. NIST's Center for AI Standards and Innovation [described its collaboration with UK AISI](https://www.nist.gov/blogs/caisi-research-blog/analyzing-transcripts-ai-agent-evaluations) on Inspect Scout's design back in February, well before Transect existed, so that post discusses the framework generally rather than this tool. Two governments shaping the same transcript-analysis plumbing is a reasonable sign of where evaluation tooling is consolidating. It also tells you something about who the intended user is. A tool whose output is a timeline for a human to audit, rather than a dashboard metric, is built for people whose job is to form a defensible judgement about a model and then defend it. We've written before about [AISI's rebuild of its secure evaluation environment](/posts/aisi-secure-evaluation-environment-rebuild), which came out of the same institute's agentic cyber testing work. For anyone running agentic evals at scale, Transect is free, MIT-licensed and reads the log formats you probably already have. Start with the labels, then read the transcript the labels point at. --- *Sources: [Transect: Making large-scale agentic evaluations easier to understand](https://www.aisi.gov.uk/blog/transect-making-large-scale-agentic-evaluations-easier-to-understand) • [Transect: Retaining Observability for Long-Horizon LLM Agent Evaluations (arXiv:2610.08364)](https://arxiv.org/abs/2610.08364) • [AI-Safety-Institute/transect on GitHub](https://github.com/AI-Safety-Institute/transect)* *Coverage: [NIST/CAISI: Analyzing Transcripts from AI Agent Evaluations](https://www.nist.gov/blogs/caisi-research-blog/analyzing-transcripts-ai-agent-evaluations) • [Automated Transcript Analysis for Detecting Flaws in Agentic Benchmarks](https://arxiv.org/pdf/2607.27518)* *Related on this blog: [[AUTO] HTTP Terminator: Systematic Desync Discovery](/posts/auto-http-terminator) • [GPT-6 Astra Ran Supply-Chain Attacks in 29% of AISI's Simulated Trajectories](/posts/aisi-gpt-6-astra-supply-chain-attacks) • [[AUTO] AI's Predictable Hallucinations Become Supply-Chain Weapons](/posts/auto-hallucinated-packages)*