Members-Only
Recent Talks & Demos are for members only
You must be an AI Tinkerers active member to view these talks and demos.
Patch Party: Live-Fixing LLM Agents
See a live demo of patching LLM agents in real-time. This talk shows how to catch and fix agent failures mid-task with precise critiques, boosting success rates without retraining.
This is a live demo of a feedback loop that patches autonomous agents in real time. We’ll show how we built a minimal actor-critic framework that catches step-level agent failures—like hallucinated facts, logic errors, or missed tool calls—and injects precise critiques to recover mid-task.
We’ll walk through:
How we used τ‑Bench and DA-Code to build a taxonomy of agent failure types
How we tagged real traces and analyzed which errors matter most (e.g., reasoning failures > tool bugs)
How we implemented the critic loop using FastAPI + GPT-4o or Claude as the agent, and optional models/humans as critics
How a 1-2 sentence critique can improve success rates by 30%, without replanning or retraining
How to plug this loop into your own stack using just a trace, a tagger, and a hook
We’ll live-debug an agent solving a DA-Code task, show failure in action, and patch it with a model-generated critique. It’s messy, practical, and shows how real-time judgment can boost reliability with minimal overhead.
Compose Email
Loading recent emails...