Experiential Labs: Custom AI Models That Learn From Your Agents

Experiential Labs is a Y Combinator backed AI lab that trains a model on your agent's production traces and serves it back through an OpenAI-compatible API. 40%+ cheaper than frontier models, within 5% of their quality. This Discord is the home base for builders running it in production and engineers who want to see how it works.

What happens inside

Daily conversation about continual learning, LLM cost optimization, distillation, reinforcement learning, supervised fine-tuning, token compression, and per-request model routing. Usually with real numbers attached.

You also get:

• Direct access to the Experiential Labs team for setup help and technical questions
• Early looks at research and releases before they go public
• Discussion around the open source world-model-optimizer repo on GitHub
• Cost and latency benchmarks from other builders (+55% speed and −72% cost vs frontier is the kind of result that gets posted here)

Who this server is for

Founders watching their inference bill, ML engineers exploring fine-tuning and distillation, developers shipping AI agents that call frontier APIs on every request, and researchers following the era of experience. If your agent logs traces, you already have training data. This community shows you what to do with it.

Why Experiential Labs

Every trace your agent produces is a chance to get cheaper, faster, better. Experiential Labs builds a simulation of your production environment from your traces, trains your model against it, and keeps retraining as new data comes in. Each request then routes to the cheapest model that clears your quality bar. Experience compounds.

Join, introduce yourself, and post what you're building. The team actually answers.

Everything in the Experiential Labs Discord comes back to one question: how do you run AI agents in production without paying frontier prices? People here train their own models on traces they already log, then serve them at a fraction of the cost. The conversation gets specific fast, usually with real numbers attached.

Continual learning and model training
This is the core of it. Members talk through fine-tuning workflows, distillation from frontier models like Kimi K3 down to small language models like Qwen-27B, and reinforcement learning that scores an agent's actual outcomes. Booked the cheapest nonstop fare and sent the confirmation? Higher score. Quoted a worse price with a layover and no booking? Lower score. The model learns from the gap. Supervised fine-tuning on gold responses, retraining loops that run on fresh production data every couple of hours, and the question of when a new model version is actually better than the one it replaced all come up constantly. If you have opinions on reward design or catastrophic forgetting, you'll find people to argue with.

Cutting inference costs
LLM cost optimization is the reason most people show up. −72% cost versus frontier is the kind of result that gets posted here, then picked apart. Per-request model routing is a favorite topic: each request routes to the cheapest model that clears your quality bar, and the routing mix shifts as your own model gets smarter. A stack might run 38% your model, 21% Fable, 13% GPT-5.5, then Haiku, GLM-5.2, DeepSeek-V4, and Qwen filling the rest. Token compression is the other big lever, stripping qualifiers, back-references, and filler before inference while keeping every decision, fact, and number intact. −41% tokens with 99% of the answer retained is a real target people chase.

Production traces and evals
Your traces are your training data, so getting them wired up matters. Members compare notes on hooking up Arize, Braintrust, LangChain, or a plain database as a trace source. From there the conversation moves to building a simulation, a digital twin of production stitched together from logged traffic that a model can train against safely. Then evals: how do you score agent behavior, where do you set the quality bar, and how do you keep a −3.2% quality delta from turning into something users actually notice? Benchmarking methodology, eval harnesses, and the gap between offline scores and live behavior all get real discussion.

Agent engineering in the wild
Plenty of talk about the actual building. The endpoint is OpenAI-compatible, so switching your app over is a one-line change instead of a rewrite, and people share how that went. Debugging agents that misbehave in production, handling latency budgets when +55% speed is on the table, structuring tool calls, and deciding which parts of a pipeline should run on a specialized model versus a frontier fallback. Bring a weird agent bug and someone has probably hit it.

Open source and research
The world-model-optimizer repo on GitHub gets its own thread of activity: issues, pull requests, and people running it on their own stacks. Beyond the code, this is a room that follows the research. The era of experience, where recursive self-improvement across domains actually leads, what compounding gains from production data look like over months, and which papers are worth reading this week. It leans technical and it leans forward.

The day-to-day
Underneath the big themes, it's builders comparing what their model just learned this week, founders watching an inference bill drop, and setup questions answered directly by the Experiential Labs team rather than a support macro. New releases and benchmarks tend to land here before they go public. If your agent logs traces, you already have everything you need to start, and this is the room where people figure out what to do with it.