Checkout is failing. Pricing is failing. Inventory is failing. There was a payments deployment a moment ago.
Which service would you investigate first?
In Faultline’s demo, payments is the tempting answer. It is also the wrong one. Payment requests are succeeding. The failing pricing and inventory requests both depend on the same product-data service, where a configuration change introduced timeouts.
I built Faultline around that small incident. It is a project for developers and on-call engineers who need to understand what broke, see the evidence, and decide what to do next.
The question I wanted it to answer was simple: can we get from several alerts to one evidence-backed fix, then check that the fix worked?
The problem: an alert tells you where it hurts
An alert from checkout tells you the customer cannot complete a purchase. It does not necessarily tell you which dependency caused the failure. A recent deployment gives you somewhere to look, but timing alone is weak evidence.
The work is in connecting those clues. What does checkout call? Which requests failed? Where do the failing paths meet? Did anything change there? Does the suspicious service have successful requests that contradict our first guess?
I started with a broader idea about connected company data. Narrowing the project to one checkout incident made the goal much clearer. There would be one investigation, one approved action, and a visible recovery check.
A small system that actually fails
Faultline runs five service endpoints inside one local API: checkout, pricing, inventory, payments, and product-data.
The dependency graph looks like this:
checkout ──→ pricing ────→ product-data
├─→ inventory ──→ product-data
└─→ payments
The environment is simulated. The requests execute, the delays run, and the caller deadlines expire. In the default incident, a generated configuration change raises product-data’s simulated upstream delay from 8 milliseconds to 220 milliseconds. Its callers cannot wait that long.
Those executions produce the timeout and dependency-failure records behind the alerts. A generated payments deployment provides the misleading clue. The interface then shows the investigator’s conclusion alongside the records supporting it.
This kept the demo honest: recovery would require changing the configuration and making another successful request.
What I put in FalkorDB
FalkorDB stores the relationships that the investigation needs to follow. Services alone would not be enough; the graph also needs the observations and changes associated with the current incident.
| Node | What it represents | Relationships |
|---|---|---|
IncidentService |
A service in the shopping system | DEPENDS_ON another service |
IncidentRun |
One generated incident | HAS_SPAN and HAS_CHANGE |
IncidentSpan |
An executed request and its result | ON_SERVICE |
IncidentChange |
A configuration, deployment, or rollback event | CHANGED a service |
The run relationships matter. A successful request from an earlier incident should not be used to explain the current failure. Trace and change queries are scoped to the active run.
To follow checkout’s dependencies, the application uses a native Cypher traversal:
MATCH path=(root:IncidentService {name: $root})
-[:DEPENDS_ON*1..4]->(dependency:IncidentService)
RETURN [service IN nodes(path) | service.name] AS services,
length(path) AS hops
ORDER BY hopsWith root set to checkout, this returns the paths the investigator can examine. Returning the hop count explicitly also makes the traversal results easier to inspect.
When several direct branches fail, another query looks for their shared dependencies:
MATCH (root:IncidentService {name: $root})
-[:DEPENDS_ON]->(branch:IncidentService)
-[:DEPENDS_ON*1..3]->(dependency:IncidentService)
WHERE branch.name IN $branches
WITH dependency, collect(DISTINCT branch.name) AS branches
WHERE size(branches) > 1
RETURN dependency.name AS service, branchesFor the pricing and inventory branches, the shared dependency is product-data. The trace records then show direct timeouts there, and a configuration event precedes the failures. Successful payments spans provide evidence against blaming the deployment.
FalkorDB is involved in reaching that answer. The runtime investigator reads paths and evidence from it. If the database is unavailable, the incident workflow cannot complete. The in-memory graph implementation is a test fixture, not a runtime substitute.
Giving the investigator a small set of tools
I exposed four read-only MCP tools:
trace_dependenciesreads service paths.read_trace_evidencereads the incident’s executed request records.read_recent_changesreads the associated changes.find_convergencefinds dependencies shared by failing branches.
The default investigator is deterministic. It uses a fixed sequence of tool calls and checks the returned evidence against a narrow configuration-timeout pattern. There is an optional model-driven mode for selecting tools, but the recorded walkthrough uses the deterministic mode.
That distinction matters. This project demonstrates an explainable investigator for a controlled incident, rather than claiming to diagnose every production outage.
If the graph path or supporting configuration evidence is missing, the investigator must return an inconclusive result and withhold the rollback proposal.
Approval is followed by another request
Once the evidence supports a cause, Faultline proposes restoring the affected service’s previous configuration.
The approval is bound to the incident, the diagnosed service, and the expected configuration revision. An approval from an older incident cannot authorize a change in a new one. The read-only investigation tools cannot perform the rollback themselves.
After approval, the app restores the previous delay value and executes the same checkout request:
GET /api/incident/services/checkout?sku=DEMO-001&quantity=1The initial request returns HTTP 503. After the rollback, that same request returns HTTP 200. Service health is calculated from the executed request spans, so the recovery display follows the result of the check.
The test that changed the project
A convincing explanation can still be a scripted answer. To test that, I kept the dependency graph unchanged and moved the fault to payments.
The investigator now received different traces and a different configuration event. It identified payments, produced a different highlighted path, and proposed a rollback for that service.
That counterexample became an important part of the demo. It checks whether the conclusion follows the evidence instead of merely repeating the first scenario’s story.
The test suite also covers stale approvals, missing graph evidence, MCP argument validation, concurrent operations, and repeat rollback requests. A separate integration test runs both fault locations against a real FalkorDB instance and checks the approved recoveries.
What I learned while building it
A smaller scope made verification possible. Five endpoints were enough to demonstrate a shared dependency, a misleading deployment, and a different diagnosis when the evidence changed. I could inspect the whole chain from fault injection to recovery.
The explanation needs a source. Highlighting a path helps someone understand the dependency chain. The timeout records and configuration event explain why that path matters. Keeping both available made the result easier to assess.
The interface should lead with a decision. Early versions exposed too much of the implementation. I separated the overview, incident investigation, and service directory, then moved raw records and queries into an expandable evidence section. The main incident view now focuses on what failed, the supported cause, and the action awaiting approval.
Verification is part of the action. A rollback response alone does not establish recovery. Repeating the original checkout request gives the user something concrete to compare.
Where it stops today
Faultline is currently a local sandbox. It does not ingest production telemetry or operate a real deployment platform. The configuration fault is a controlled latency change, and the investigator recognizes a deliberately limited failure pattern.
Evidence persists in FalkorDB, while the active run, service configuration, and approval state live in process memory. Restarting the API means starting a new incident.
Before using this approach in production, I would need durable action state, authentication, stronger environment isolation, real telemetry adapters, and broader evaluation across failure types. Multiple simultaneous faults would be a useful next test.
The current project gives me a concrete starting point: a service graph that shapes the diagnosis, records that explain it, and an approved fix checked against the request that originally failed.
Watch the project work
Watch the 1 minute 58 second walkthrough. It includes the default product-data incident, the supporting records, the approved recovery, and the payments counterexample. The narration uses an ElevenLabs synthetic voice.
Faultline was built for Track 01: Agents That Act on Connected Data in Graph Hacks.
AI assistance: I used Codex for implementation, debugging, tests, interface revisions, and demo preparation. This post describes the implemented behavior and its limits; it does not claim production deployment or production incident results.
Further reading: FalkorDB documentation and the Model Context Protocol specification.