Vigil Product tour Buy Vigil
Menu

Incident evidence

Keep what the incident
would otherwise lose.

Vigil preserves the failed observation, the last success and the differences between them. Responders can inspect the facts after the monitor has moved on.

Inspect the evidence example Compare editions

Inside the actual evidence interface

A timeout, with the next clue.

The failed check reports a timeout. The onset diagnostic resolves the hostname, then records a refused connection. Vigil keeps the reported failure separate from what the diagnostic measured.

Illustrative sample using test data. Explore the observations and the diagnostic result below.

Inspect the complete example → · Sample data and provenance →
01 / Compare

Keep the last success beside the first failure. This example shows the response-time change.

02 / Diagnose

Separate the check’s report from the read-only diagnostic result, including its time and probe budget.

03 / Qualify

Show the basis for a conclusion and state when no related failures were found.

A record that outlasts the outage

Preserved once.
Available after recovery.

Evidence stays with an open incident. After resolution, it is retained for one year, even when the original observations have expired.

Related failures, with reasons

Correlation names the shared signal it matched. It gives a responder a lead to investigate, without claiming to know the root cause.

Read the technical documentation →
Technical details and implementation notes
The head of a Vigil incident: critical, identified, Checkout Service is down, auto-opened by the Checkout Service monitor, started at 12:06 AM and open for forty-three minutes.

incident, at onsetthe row the evidence is keyed to; one snapshot per incident, the first one committed

row
incident_evidence, primary key incident_id
written
once, at onset, on conflict do nothing
frozen
Aug 24, 2026, 12:06 AM, and never rewritten afterwards
kept
one year after the incident resolves, never while it is open

The reason it is a row

Everything the answer comes from expires.

A monitoring system can usually reconstruct why something failed, for a while. This exists because that window closes, and it closes faster than anybody writes a postmortem.

So the answer is copied once, at onset, into a row of its own. incident_evidence has incident_id as its primary key and the insert is on conflict do nothing, which puts the idempotency guarantee in Postgres rather than in the application: a retried check, two workers repairing the same incident and a monitor that flapped twice inside one incident all leave exactly one snapshot.

Observations
Pruned at ninety days. The measurement that failed is gone long before the quarter it belongs to is reviewed.
Monitor state
Rewritten by every later check. The monitor's own history no longer holds what it looked like when the incident began.
The fleet
The correlated failure elsewhere has recovered by the time anyone opens the incident to look for it.
Evidence
Kept for a year after the incident resolves, and never pruned at any age while it is still open.

The snapshot

Five things, written once.

  1. The observation that failed

    The check that opened the incident, exactly as it was recorded, sealed against every value the monitor's check type declares secret, in both the stored form and the percent-decoded form the driver actually sends. Bodies are never stored and response headers are an allow-list, so a snapshot cannot become a place credentials accumulate.

  2. The last one that succeeded

    Kept beside it, so the pair can be read rather than hunted for in a table that is about to be pruned.

  3. What changed between them

    Status codes, record counts and certificate issuers are reported on any inequality. Durations need a ratio, because a response time that moved from 130ms to 160ms is what a healthy endpoint does all day. A fact that stopped being reported is a row of its own, and usually the informative one: an http check that stops reporting tlsDaysRemaining stopped completing a handshake.

  4. What four probes found on re-check

    A bounded diagnostic burst: resolve, connect, handshake, request. Read-only, one socket each, and specified below rather than described.

  5. Which other failures share a signal

    Related by rule, not by score: within ten minutes and at least one strong shared signal, which means a hostname, a registrable domain, a resolved address, a failure signature, or in the commercial editions a remote probe location that did not see the target healthy. Each signal carries the value it matched on, so the claim can be checked rather than trusted.

The refusals

What it will not say.

A field an operator learns to distrust is worse than a field that is not there. Every one of these is a decision to stay silent rather than to guess.

  • A bare timeout names no layer. The failing layer is one of dns, tcp, tls, http, application, browser or unknown, and it stays unknown unless something measured it.
  • Every layer carries its basis. measured if a diagnostic re-probed it and it failed, reported if the failure names it, an assertion if the target answered and so the same observation proves the transport. Nothing is stated without saying how it was established.
  • A shared timeout is not a relationship. Neither is a shared HTTP status. They are the two things unrelated monitors have in common every day, so neither counts as a strong signal.
  • There is no repair path on a later check. A snapshot filled in ten minutes late while labelled "at onset" is worse than an absence, so the row is never rewritten.
  • Nothing in it is inferred by a language model. The snapshot is measurement and record. AI drafts postmortems elsewhere, from this material, and is never the source of it.
  • It reaches no public status page. The snapshot is operator-facing only, and it is scoped to the organization inside the query rather than filtered afterwards.

The onset diagnostic

A burst that cannot become an outage.

The one part of this that touches the network again is also the one that could hurt a system already in trouble. It is bounded on every side, it writes nothing, and an installation can switch it off with INCIDENT_EVIDENCE_BURST=false.

It runs after the incident's page has been claimed, never before. Shadow mode refuses it outright, and a monitor on the high-frequency plane never bursts at all.

probes
at most 4
resolve, connect, handshake, request
sockets
1 each
no redirects followed
total budget
5 s
raced against per-step deadlines
concurrency
2
per worker process
stored size
32 KB
byte-accurate cap on the snapshot
side effects
none
writes no observation, moves no monitor status, pages nobody, feeds no objective, triggers no runbook

Where it is read

On the incident, beside the timeline.

The snapshot is not a separate report to go and find. It sits on the incident it belongs to, under the record of what people did about it.

A resolved Vigil incident: the postmortem written into it, and beneath it the append-only timeline of the status changes and updates people posted while it was open.

incident, resolvedthe postmortem and the timeline it was written from

In Vigil Core

Evidence for Core's check types: the failed observation, the last success, the difference, the bounded burst and the related failures. Free, Apache-2.0, no key.

What else Core ships

In the commercial editions

Adds a scripted journey's failed step and timing, with the screenshot referenced by id and hash rather than copied, and what each remote probe saw, read from the frozen round record.

What the $149 covers

Your infrastructure. Your record.

See how Vigil handles an incident.

Explore the product tour Compare editions