The AI Reliability Landscape
The open-source projects and startups building a new category
In today’s post, we take a look at what’s trending in AI for science, then dig into two (related) questions: how are LLM-based AI agents actually made more reliable, and can the open-source projects and startups building that tooling become financially viable?
Also: A big hello to everyone who’s joined us from our YouTube channel. We recently passed 500 subscribers, and our most popular interview – featuring MIT’s Jesse Thaler on AI for Physics – is on track to hit 7,000 views. If you’re not yet subscribed on YouTube, come join the fun. Let’s go 🚅
Google AI Leaders Depart to Co-Found A Science Startup
Amid the big-shake-up last week at DeepMind, Google’s AI research lab, four of its top leaders – Jeff Dean, Oriol Vinyals, Sanjay Ghemawat, and Quoc Le – left to-cofound Discovery Loop, a startup established to automate the scientific method. On the menu? Everything from re-engineering machine learning itself, to rethinking medicine and clean energy R&D. An ambitious vision, with echoes of Lila Sciences on the opposite coast in Cambridge, MA.
The perennial question: how does one effectively monetize such an R&D machine, be it through licensing its platform, or competing against industry through a full-stack approach? The full-stack path (i.e. competing against big pharma) is the one Demis Hassabis, the DeepMind co-founder also involved in the recent shake-up, appears to be taking at Isomorphic Labs. For a take on what vertical integration means for the science community, check out my recent exchange with Boltz’s co-founder Gabriele Corso, contrasting Isomorphic with Boltz’s more open approach.
US Genesis Missions Kicks Into Gear
On the heels of announcing the Genesis Mission – a $5 billion US DOE-led initiative unifying 17 National Labs to build a next-generation scientific platform – the DOE just announced the first projects selected under its request for applications. Examples include:
Scalable Agentic Digital Twins for Autonomous Precision Facilities
AI-Driven Discovery of Electrochemical Separation Methods for Rare Earth Elements
AI-Driven Quantum Sensing for Precision Tests of Fundamental Physics
A complete list is available for PDF download here. For a close look at the DOE’s lab for autonomous materials discovery, check out our in-person tour and interview.
Leveraging A.I. To Create A Novel Virus
As per the NYT: “Scientists at Stanford University and the Arc Institute, a research organization in Palo Alto, Calif., taught A.I. to recognize patterns of DNA structure in nature, and then to use that data to write recipes for entirely new viruses. The researchers followed those recipes to create DNA molecules, which they inserted into bacteria. The modified bacteria then produced viruses never seen in nature. The viruses were able to infect other bacteria, demonstrating that they were viable.”
The scientists behind this study leverage the Evo platform, covered here in our interview with Hani Goodarzi of the Arc Institute, which developed it. This achievement also raises pressing questions about bio safety. If that’s of personal interest to you, check out this AIxBiosecurity Research Fellowship Program from Cambridge Boston Alignment Initiative (CBAI).
The Case for AI Reliability Platforms
For all the power of frontier AI models, and their increasingly adept open-source counterparts, they still struggle with a fundamental lack of consistency, owing in large part to their probabilistic architecture. Much like biological systems, these models are “grown” rather than “built.” And just as it’s hard to know why your cat has suddenly declared its own tail its arch-enemy, it can be hard to know why a model capable of crushing the LSAT insists the country of Austria ends with the letter B.
For low-risk use cases where humans are closely in the loop – say, reviewing the draft of an email – these errors are generally no big deal. As we move up the adoption curve into higher-stakes workflows with less human review, however, the equation flips: a once-in-a-hundred hallucination becomes a real liability. Think of an agent issuing an erroneous refund, citing a hallucinated legal case, referring the wrong patient to a clinical trial, or serving as a vehicle for data exfiltration – errors that can end a customer relationship or earn a company a day in court.
Hence the large and growing ecosystem of tools built to “wrangle” these models into consistency: wiring telemetry into agents and observing how they behave, running large-scale evals across their outputs, and/or enlisting other LLMs as judges. All of which sits atop more research-driven, less immediately productizable work to peer inside the “mind” of these models, through techniques like sparse autoencoders (SAEs) and mechanistic interpretability (hat tip, Neel Nanda).
So how do these AI reliability platforms fare? How close can we get AI systems to behavior consistent and reliable enough for meaningful enterprise - and scientific - use cases? And, from a go-to-market perspective, how does a startup in this space capture enough value to sustain an engineering team and build enterprise-specific features?
In this post, I’ll walk you through my outside-in perspective on the (broadly defined) AI reliability space, as uncovered through in-person interviews and secondary research. We’ll cover:
The high-level development flow for AI agents in production, including a product hands-on with a popular platform (Arize Phoenix)
The key users and buyers of reliability platforms, and what each likely cares about
The landscape of existing solutions, and moats startups can target to build reliable, recurring revenue
The AI Agent Development Flow
Step 1: Build and instrument the agent
Building an agentic workflow has never been easier. Open a Claude Cowork session, ask it to build, say, a database of news articles on recent renewable-energy policy, and it will spin up agents that get to work. Fancy a more hands-on approach? Visual builders like Lindy, Gumloop, and n8n have you covered, with varying levels of interoperability with the systems you already work with.
Fancy a go? Here’s my guide on building an agentic workflow with Google Opal.
For enterprise-scale agentic workflows, however, you’re likely writing your agent in code, typically Python. A first pass might mean defining the workflow (here’s one I built for weather lookup), giving it a set of tools (e.g. an API, database retrieval), and attaching telemetry so that each run emits traces of what the agent is doing.
Step 2: Evaluate the agent, offline
Once you’ve built your agent, what happens when you run a dataset through it, ideally one close to what it will see in production? What happens when you swap the model that powers it?
Debugging here means opening a trace tree and walking it to locate the failure, for example linked to retrieval, reasoning, tool execution, or the prompt itself. Rigorous testing requires a rubric or scoring mechanism. At scale, it’s too labor-intensive for humans to score every output, so teams will either constrain agents to easily verifiable outputs (returning a valid JSON, calling the right tool) or enlist an additional LLM as a judge.
This is where the real trade-offs between quality and cost begin, especially as LLM judges are themselves token-hungry. For more on those trade-offs, and the eval types that matter, I highly recommend reading Hamel Husain’s Your AI Product Needs Evals LLMs.
Step 3: Deploy in production and monitor/observe
Once the evals have cleared their quality litmus tests (”gates”), teams will deploy the agent in production and watch it: observing areas like latency, tool error rates, retry loops, token burn.
Offline evals only cover the cases they thought of; real users do things we don’t anticipate. So judges run continuously against a sample of live traffic, sometimes with user-feedback signals (a thumbs-down, an escalation to a human, an abandoned session) as additional labels. Sampling strategy matters, because judging every trace with an LLM gets expensive.
Step 4: Score and iterate
As production traffic gets scored, teams feed what they learned back into both the agent’s design and the QA process itself. Production failures – a botched response, an agent breakdown, even a jailbreak attempt – are dropped into the eval dataset in Step 2, so every future version gets tested against them.
Two things begin to compound. First, the failure dataset generated grows increasingly valuable: it’s specific to the organization and exists nowhere else. Second, iterating well becomes a cross-functional endeavor: engineers sit with domain experts, QA teams and PMs to define what actually counts as a failure, what a better result looks like, and which trade-offs a fix is worth (a pricier but more capable model here, breaking down a data silo there), important skill-sets for future deployments.
Governance, embedded throughout
Threaded through all four steps sits the enterprise wrapper: role-based access controls (RBAC), SSO, audit logs of who changed which prompt, PII detection (and redaction in traces), data residency, and reporting a compliance team can hand to an auditor.
🖐️ Product Hands-On
A popular platform that spans both evaluation and production observability comes from Arize. For a deep dive into what working in one of these platforms actually feels like, check out my hands-on with Phoenix, their open-source toolkit.
Buyer Landscape
When I sold design software at Mural, the personas that came to mind first were the end users themselves, for example: designers building mood boards on a whiteboard, or PMs laying out project sprints. Dig deeper, however, and it was clear that while these users were important champions, the budgets often lay elsewhere – with a CTO leading an enterprise-wide digital transformation, or a strategy team driving how the organization is structured and functions.
AI reliability has similar parallels. The end user might be a product manager trying to get their customer-service chatbot to fire more consistently. Or a lawyer at a BigLaw firm, harnessing a system built on the firm’s proprietary historical data to advise a client on a specific transaction. As with design software, though, no single piece of software is an island. Add to that AI’s nature as a knowledge tool – one that performs better the more it ingests – and it’s exactly the kind of technology that keeps other important stakeholders – like Compliance and Legal – up at night.
As I see it, the key stakeholders in the AI reliability space are as follows:
For builders and sellers in this space, this means realizing that there’s no one-size-fits all to serving this market, and success means carefully weighing and delivering on sometimes competing priorities within the client org itself.
Product Landscape
In SaaS, a freemium model is often a valuable funnel for driving product discovery and adoption. I believe the reliability space is no different, with one asterisk: here, the free tier is usually open source. An engineer can run a pip install and start playing with a toolkit in their own environment within minutes. That said, this comes at a cost: unlike a typical freemium SaaS offering, an open source toolkit run behind the firewall of an enterprise builds little insight into how users are actually leveraging your product.
The dominant strategy in the category seems to be: open source to drive early adoption, traction, and community-based feature development; nudge users into premium enterprise options when their deployment requires it. Scanning multiple vendors, many of the meaningful enterprise “upgrades” seem less product-based and are often directed at the non-user stakeholders in the client’s ecosystem, as discussed above. Take Braintrust’s pricing page:
Non-product-y enterprise offerings appear often to include:
Building A Moat
In a crowded field, with many solid open-source alternatives, how do startups in the AI reliability space differentiate and build a lasting advantage?
The core theme I see is centered around the customers’ usage data, specifically: use the customer’s own data to make the product better for them, and for their peers operating in similar verticals. For example:
Sample efficiency. Tune generic LLM-as-judge metrics to a specific domain with just a handful of annotated examples, and demonstrate immediate accuracy improvements with your secret sauce.
Purpose-built eval models. Deliver accuracy for the specific use case at costs far below off-the-shelf frontier models with a model that only you own.
Automated failure discovery. Build your own engine that clusters similar failures, surfaces root-cause patterns, and recommends fixes at scale, as trained off of prior failure datasets.
Partnering with an outside vendor one could also bring an added stamp of approval to evals, that are harder to replicate:
Validated vertical eval suites. E.g. a clinical-safety suite validated against clinician labels, or a financial-advice suite mapped to FINRA requirements.
Have a different take?
That’s my read on the AI reliability market: how the workflow runs, who buys, where the moats might form. Short of a complete paradigm shift in how powerful, generalizable AI is built and delivered (and perhaps, even then), safely realizing the benefits of AI agents in applications like human clinical trials and investigational new drug applications will require a thriving ecosystem of open and closed source efforts to make these systems more reliable. If you’re building or selling in this space, I’d love to trade notes – and if you think I’ve gotten something wrong, even better. My inbox is open.
Find this newsletter valuable? Subscribe for insights and share it with friends who are passionate about the future of science and technology.
Help us improve. Like a good neural network, this newsletter is only as effective as the (human) input it receives. We read every comment we get.
Catch you on the next one 🤙🙌 – Nabil
