FAULTLINE: An Incident Graph That Learns from Evidence

Summary

Pixel shares FAULTLINE, a prototype built for the HydraDB hackathon that investigates incidents using an evidence-backed graph. It traces failures across services and settings, identifies missing dependencies, and learns verified connections so future investigations can start smarter.

DEV Community

FAULTLINE: Teaching an Incident Graph to Say "I Don't Know" and Then Learn

When checkout fails at 2 a.m., the error almost never names the cause. The failing endpoint is in one place; the setting that broke it sits somewhere else, reached through services nobody remembers wiring together. FAULTLINE is my answer to that gap: a prototype that shows what failed, checks evidence for a missing connection, and saves that connection in a self-run HydraDB graph so the next investigation starts smarter. I built it for the November HydraDB hackathon, Track 1: a graph that improves through evidence-backed investigations. This post covers the problem, the hard parts, how the system works, what the measurements show, and where the claims stop. Live demo: faultline-j7kn.onrender.com Code: github.com/ParthMalhotra07/Faultline Walkthrough video: Loom The problem: the visible error is not the cause Incident response is mostly a search problem. An engineer sees a failing check, such as checkout returning errors, and has to work backwards: which service handles it, what that service depends on, and which recent change touched any of those dependencies. The painful cases involve shared settings. A currency configuration changes. Checkout fails. A service map shows checkout and pricing, but it does not show that pricing reads that setting, because nobody ever recorded the link. So the team rediscovers the same hidden dependency during every incident, and the knowledge disappears again once the incident closes. Two things make this hard for a tool to help with: Missing edges. The map the tool searches is incomplete by definition. A tool that only follows known links cannot explain a failure caused by an unknown one. Trust. If a tool guesses, engineers stop believing it after the first wrong answer. A useful tool must say when it does not know, and show the evidence when it does. This is also the question the hackathon asked: can a graph earn its place over a plain vector store? A graph earns it only if it keeps structure (who calls whom, who reads what) and if it gets better each time it is used. Challenges I faced The interesting engineering was less about drawing a graph and more about making every claim the system makes defensible. 1. Making "I don't know" a first-class answer. A ranking model always returns something. FAULTLINE had to return insufficient evidence when no committed path connects a failing endpoint to a changed setting, and it had to keep abstentions, ambiguity and errors in the scoring denominator so they could not hide behind a flattering accuracy number. 2. Stopping the learner from cheating. In a controlled demo it is easy to leak the answer. I split the system: a private fault controller creates incidents and knows the true origin, but it hands that label only to the evaluator. The learner sees public observations and catalog metadata. Learning data and held-out data follow separate paths, and the protocol is frozen before the final run. 3. Keeping graph writes trustworthy. A graph that learns can also learn wrong things. A claim cannot be committed without a matching evidence ID, source hash and source version. Retries use deterministic IDs so they stay idempotent, and a failed or incomplete inspection cannot quietly improve the graph. Each learning round is also bounded: one source, 10,000 bytes, 100 records and a 1,000 ms budget. 4. Failing loudly when the graph engine misbehaves. The adapter caps requests at 5 seconds and 1,000 returned rows. A timeout, a truncated page or a malformed response becomes an explicit error, never an empty result that looks like "no path found". I run a pinned, open-source HydraDB image and keep its query port on container loopback, so the browser only ever talks to the FAULTLINE server and no arbitrary query endpoint exists. 5. Ambiguity between near-identical causes. Early on, a shorter graph path could win by default. When two changed settings cover the same failed and healthy endpoints, the data alone cannot separate them. I added a guard so the live demo no...
FAULTLINE: An Incident Graph That Learns from Evidence
Powered by Readybot.io

The latest from Pixel

Incident Response: Look Beyond the Red Alert

Pixel’s DEV Community announcement explores how teams can fixate on one failing service while overlooking the real cause of an outage, such as a full …

Sach: AI Fact-Checks WhatsApp Forwards in India

Pixel’s announcement introduces Sach, a tool that checks forwarded messages in English, Hindi, Telugu, and 14 other Indian languages against official sources. It gives cited …

Java 25 Removes SecurityManager: What to Fix

Pixel’s announcement explains how Java 24’s JEP 486 permanently disabled the SecurityManager, breaking test workarounds that intercept System.exit. It recommends returning exit codes or throwing …