How We Cut Real-Time AI Answer Latency to 2.6s

Resumen

Pixel’s announcement breaks down latency across a real-time interview copilot, measuring 12 recorded questions with real providers. Keeping the model connection open reduced median response time from 3.58 to 2.65 seconds, while experiments with faster speech detection risked splitting questions—showing why measuring accuracy and each pipeline stage matters.

DEV Community

Where 3.6 seconds go in a real-time AI answer, and how we got it to 2.6

We build an interview copilot: it hears a question on a call and starts writing an answer. The number that matters is the time from the end of the question to the first visible words. In our browser call it was a median of 3.6 s. Here is where the time went, measured on 12 recorded questions with real providers. The breakdown End of speech to a final transcript line: about 1.6 s. We ask the speech engine to end a line after 0.45 s of silence, but in practice the final line arrived a median 1.56 s after speech stopped. Detecting the question and sending the request: under 0.05 s. Tokens, the call's lease and building the prompt: 0 to 0.3 s. Opening a connection to the model: 0.73 s, because the connection was closed between questions. Model request to first text: about 1.1 s. The stable part of the prompt was already cached (about 3,000 of 3,400 input tokens). Rendering: under 0.05 s. Streaming the first sentence sooner would have saved nothing (6 ms). What worked Keeping the API's connection to the model open while a call is live. The model stage went from 1.74 s to 1.15 s. The whole path: median 3.58 s to 2.65 s, 17 of 17 questions answered whole. Cost: one tiny keep-alive request every few seconds per server, only while a call runs. What did not (yet) Ending the question sooner from our own silence detection brought the final line to about 0.8 s, but in some runs the transcription then cut questions in two a second after they began. A faster wrong answer is worse than a slower right one, so it stays off until we understand why. Takeaways Measure each stage on real audio before optimising: our guess (the model) was only half the story. Connection set-up is invisible in benchmarks and very visible in live calls. Speech end-of-utterance is the floor you fight next. We're building RoundReply (roundreply.com): live transcripts and real-time interview answers from your resume.
How We Cut Real-Time AI Answer Latency to 2.6s
Powered by Readybot.io

Lo último de Pixel

Incident Response: Look Beyond the Red Alert

Pixel’s DEV Community announcement explores how teams can fixate on one failing service while overlooking the real cause of an outage, such as a full …

Sach: AI Fact-Checks WhatsApp Forwards in India

Pixel’s announcement introduces Sach, a tool that checks forwarded messages in English, Hindi, Telugu, and 14 other Indian languages against official sources. It gives cited …

Java 25 Removes SecurityManager: What to Fix

Pixel’s announcement explains how Java 24’s JEP 486 permanently disabled the SecurityManager, breaking test workarounds that intercept System.exit. It recommends returning exit codes or throwing …