Incident management

uptimepage turns a failing check into a first-class operational incident: a tracked lifecycle with acknowledgement, ownership, paging, on-call rotations, escalation, and a retrospective — not just a banner on a status page. This page is for operators running incident response. For the customer-facing surface it publishes to, see Public status page; for the wire-level endpoints see REST API.

The core idea: internal state is not public phase

The single most important distinction is that what your responders see is orthogonal to what your customers see. Conflating the two is the classic incident-tooling bug, so uptimepage keeps three independent axes on one incident:

Acknowledging an incident stops escalation and records who took it — it posts nothing to a status page. Customers see something only when you publish the incident and post a public update. An incident can run its whole internal lifecycle while staying internal.

How an incident opens

A background writer scans every enabled monitor (not only status-page components). When a monitor sustains a bad state — down, error, or degraded — it opens one incident; a sustained recovery to up resolves it automatically (with no human resolver recorded). One open monitor incident at a time; duplicate failures fold into it. A declared incident sits in its own slot, so one left open does not stop the writer opening, paging and counting a real outage under it.

"Sustains" is two gates, not one. A region counts as failing only after alert_confirmations failures in a row from that region alone (default 2), and then the monitor's region policy decides how many failing regions have to agree before the incident opens. The default is majority, meaning more than half of the regions currently reporting results for that monitor; a region that stops reporting drops out of the vote instead of counting as down. Under that default, a monitor probed from several regions does not open an incident because one location had a bad minute. See Multi-region probes for the policy list, or Probe regions on the hosted service.

Closing mirrors it: a failing region counts as recovered only after alert_confirmations passing checks in a row, and the incident ends when the regions still counted as failing drop below the policy. A run shorter than that changes nothing either way, so one stray result neither closes an outage nor extends one. A region that never failed is no evidence of a recovery on its own, so a failing region that stops reporting keeps the incident open until one that was failing passes again. The monitor's recovery period (recovery_period_secs) adds a wait on top: the incident closes only once that recovery has held for the whole period, and a failure that confirms again inside it keeps the same incident open. A flapping service then reads as one incident on the console, in alerts and on the status page. The end is dated to when the last recovery began, so the wait never counts as downtime. While it waits the incident is recovering, marked with recovering_since: the monitor is back up, so escalation and reminders pause, the status page shows the component operational and the incident as Monitoring, and uptime stops counting from then. The console and the dashboard mark it recovering beside its state, and it still takes an ack or a resolve. A failure that confirms again picks the paging back up, and the stretch the monitor was back up in stays on the incident as recovered: it belongs to the incident but never counts as downtime, on the status page or in any uptime figure. An emergency page already sounding keeps going until the incident closes or someone acknowledges it: cancelled for a recovery that then failed, it could not be raised again. The wait passes once the period is over, every check due inside it has passed and no failure is still waiting to confirm. A check due after the period is not waited for, so a monitor checking less often than its period does not stay open a whole extra interval. The period is three minutes by default on an HTTP, TCP, ping, DNS or flow monitor, and off on a heartbeat, manual or expiry monitor, whose next check can be hours away.

Visibility is derived at open time: if the monitor is a component of an enabled status page the incident opens public, otherwise internal. A monitor on no page still gets a fully tracked internal incident.

You can also declare an incident by hand from the console (/incidents/declare) — for a problem a monitor can't see, like a customer report or a partner outage. A manual incident may stand alone or link to a monitor, and always has a title: POST /api/v1/incidents requires title, and amending cannot clear it. A monitor holds at most one open declaration, independently of whatever the writer is doing, so a declaration you forget to close never silences detection on that monitor.

Declaring is quiet by default: the incident opens internal and pages nobody, so you can open one while you are still working out what broke. The form offers both louder options explicitly — publish it to the status pages carrying the linked monitor (or, with no monitor, the pages you pick), and alert the org's channels now. Over the API those are the visibility and notify fields on POST /api/v1/incidents, both off unless set. Alert mail for a declared incident says it was declared by hand, so nobody reads it as a monitor detection.

A declared incident does not dent the monitor's uptime unless you say it should. Uptime is measured from checks, and a declaration has no failing check behind it, so counting one by default would move a number nothing in the check history can explain. The declare form asks the question directly, and the incident stays listed either way with the monitor's incident list marking an excluded one not counted. Switch it on for a real outage the checks could not see, like payments failing while the site answers fine. Over the API this is counts_as_downtime on POST /api/v1/incidents, off unless set, and it can be flipped later with PATCH /api/v1/incidents/{id}. It also decides what the public page paints: see Public status page. An incident a monitor opened always counts: its failed checks are the evidence, and the API refuses to move that with a 422.

Uptime weights the time an incident counts by how bad it was, as Atlassian Statuspage does. A major outage counts in full. A partial outage counts for 30% of its length: some region still reached the service, or a declared incident is of major severity. Degraded performance, or a declaration of minor severity, is not downtime at all. Two incidents over the same minutes count once. The dashboard, the monitor list and page, the uptime API, the MCP tools and the status page all weigh downtime this way. The status page counts only what it shows, so a declaration you never published moves the monitor's own figure but not the page's.

A declaration and a detection on the same monitor

The two are independent objects and hold separate slots, so a monitor can carry one of each at once. Walking the usual sequence:

1. You declare. One manual incident opens, quiet unless you asked it to page, excluded from uptime unless you asked it to count. The writer ignores it from here on.
2. The monitor then goes down for real. The writer sees no monitor incident open for it, so it opens its own, pages on-call, and derives visibility as usual. Both incidents are now open. Only the writer's one moves uptime; yours still reads not counted.
3. The monitor recovers. The writer resolves the incident it opened, stamped at the recovery, and uptime is dented by exactly that window, weighted as above.

Your declaration is left alone in step 3. The writer never resolves what a person opened, so a declaration stays open until someone closes it, and while it is open the org keeps reporting an active incident. Resolve declarations when you are done with them.

Each incident carries a severity (minor / major / critical) and an urgency (high pages on-call, low notifies only; urgency decides how hard it pages once alerting is on). A declared incident takes the severity you choose; an auto-opened one currently defaults to major until an operator changes it.

The console

/incidents is the operator console — a management surface distinct from the dashboard's at-a-glance banner. It lists incidents with severity, state, monitor, assignee, and age, filterable by state. /incidents/{id} is the detail view: header, the action bar, the trigger sample, and the activity log.

The action bar drives the lifecycle:

Acknowledge and resolve prompt for an optional note in the console so you can capture the why at the moment you act. The buttons on an alert take no note.

The activity log

Every lifecycle action writes an append-only event to the incident's internal timeline. Each entry answers who, when, and what: the acting member's email (system-driven transitions show system; an action taken through the MCP server is badged via MCP; one taken from a push notification shows notification, since holding the alert is the only proof and there is no member to name), an exact timestamp, and any note. This is the audit trail — the foundation for tracking response is a healthy habit of leaving notes, and the log makes that habit visible.

Paging and escalation

When an incident opens, the escalation engine pages the responsible channels. Paging reuses the same transports as regular notifications, every channel kind included: Slack, Discord, Teams, Google Chat, Mattermost, Telegram (one-tap linked or bring-your-own bot), WhatsApp, email, SMS, PagerDuty, ntfy, Pushover, Gotify, and webhooks (see Notifications). Pushover emergency-priority pages are receipt-tracked and cancelled on resolve. Telegram rate-limit responses are honoured: a 429 with retry_after pushes the retry out at least that far.

An escalation policy is an ordered ladder of levels. The first level pages as soon as the incident opens; each level pages its targets when it is reached, then waits its delay, and if no one has acknowledged by then the engine pages the next level. After the last level's delay the ladder can be walked again from the top, a configured number of times (at most 10 extra walks). Once the final page is sent the walk ends and the monitor's outage reminders take over, so with a single walk the last level's delay does nothing. Acknowledging the incident halts the walk.

A policy's targets can be:

• a channel — pages that notification channel directly;
• a user — pages the channels that member has chosen to be reached on (see on-call below);
• a schedule — resolves who is on call right now and pages them.

Policies are owner-managed at /settings/escalation: build the ladder, set per-level targets, and pick an org-default policy. Bind a specific policy to a monitor from the monitor's edit form. Resolution at page time is: the monitor's own policy, else the org default, else simple mode — the monitor's bound notification channels are paged directly, with no laddered re-paging.

One notification source. Every down/up notification flows through the incident engine — there is no separate per-monitor alert dispatch, so a monitor can never double-page. A monitor with no policy and no org default pages its bound channels in simple mode.

A monitor inside an active maintenance window with suppress_alerts on pages nobody while the window runs. The incident opens and records normally and the timeline says paging was held; the page itself is held, not dropped, so when the window ends with the incident still open the release sweep pages the channels the hold never reached. An escalation ladder mid-walk parks rather than stopping, and picks up where it left off. An incident that was already paged before the window simply resumes its reminders afterwards — its backoff is left alone, since the on-call already knows about it. An incident an operator declares by hand always pages, window or not: they declared it during the window on purpose.

While an incident stays unacknowledged, the engine re-sends a reminder on the monitor's renotify_interval_secs cadence (default hourly, 0 disables), doubling the gap after each one up to a day; acknowledging or resolving stops both the reminders and any escalation walk. A reminder carries the reminder reason rather than a second opened, so it reads as a reminder in the delivery log and does not re-ping a Slack channel. Failed deliveries retry on exponential backoff and are dead-lettered after the attempt cap. Every attempt is auditable: the incident detail page has a Delivery section, and GET /api/v1/incidents/{id}/notifications returns the same log.

On-call schedules

On-call schedules (owner-managed at /settings/on-call) decide which human a user or schedule target pages.

A schedule has a timezone and one or more layers, in order (layer_order in the API, lowest first, no two the same). A layer is on call at all hours, or only in its hours: weekly windows in the schedule's timezone, each some days of the week from one time to another (windows in the API, such as {"days": ["mon","tue","wed","thu","fri"], "from": "09:00", "to": "17:00"}). A window whose end is at or before its start runs into the next day, so 22:00 to 06:00 covers the night after each day it names. At any moment the first layer on call pages, and later layers fill the hours earlier ones leave: weekday working hours on the first layer and a second layer at all hours covers nights and weekends. A layer on call at all hours must therefore be the last; the API refuses any layer whose hours the layers before it already cover, since it would never page. Hours no layer covers page no one; the calendar and the schedule list show them as "no one". Within a layer, participants rotate in listed order on a cadence:

The editor lists each layer's participants in rotation order; move one up or down to change who follows whom. A layer's first handoff is typed as local time in the schedule's timezone, and a new schedule starts in your browser's timezone with the first handoff next Monday at 09:00. A rotation keeps counting outside its layer's hours: a weekly rotation on weekday working hours gives each person the working days of their week.

Overrides cover a specific window with a chosen person (vacations, swaps) and beat the rotation while active.

The edit page opens on a calendar of the saved schedule, with the rotation below it. Each day names who is on call, with the local time of any handoff that day, and overrides are highlighted over the rotation; its days run midnight to midnight in the schedule's timezone. Build an override there by clicking a start day, then an end day, then choosing who covers. Days already over cannot start one, and the API refuses a window that has ended; one that has already begun starts when it is added, so the calendar keeps who was on call before it. The same person cannot cover two overlapping windows on one schedule; the API refuses the second with ON_CALL_OVERRIDE_OVERLAPS. The overrides still to come are listed under the calendar, where you can remove one; removing one that has begun ends it now instead, so who was on call before stays as it was. The schedule list names who is on call now, until when, and who takes over, and GET /api/v1/on-call/who answers who is on call at any instant.

Resolution at page time, for a given instant: an override covering that instant wins; otherwise the first layer on call at that instant, advanced by its rotation. The result is a set of users.

Your shifts

The on-call page lists your next shifts across every schedule, overrides you cover included. It also gives you a calendar link: an iCalendar feed of your shifts from the last 30 days and the next 90, which a calendar app subscribes to without signing in. Buttons beside it subscribe Google Calendar, Outlook.com, Microsoft 365 or your device's own calendar app in one click, and any other app takes the link by URL. Google subscribes from a computer, not from its phone app. A subscribed calendar picks up swaps and overrides by itself, but Google and Outlook fetch it on their own schedule, which can take up to a day, so the on-call page is the one to trust for tonight. Its past shifts are worked out from the schedules as they stand now, as on the schedule calendar, so editing a rotation changes them too and deleting a schedule drops them. Anyone holding the link can read your shifts, so keep it to yourself. Making a new link, on the page or with POST /api/v1/on-call/my-calendar, stops the old one working, and the link ends when you leave the organization.

Contact channels

A resolved user is paged through the org notification channels they have opted into — each member picks, on the on-call page, which notification channels reach them. A user/schedule target therefore resolves to people, then to their chosen channels; the paging log records the targeted user alongside the channel. If none of a member's chosen channels can deliver (switched off, or an email address not yet confirmed), they resolve but cannot be paged. The schedule list names such members on each schedule, rotation or override alike, and the on-call page warns a member who is on call without one. When someone leaves the organization, removed by an owner or by deleting their account, they come off every rotation, override and escalation level there, and their choice of paging channels is dropped. The channels themselves stay: one bound directly to a monitor or to a level keeps paging wherever it is bound.

Publishing to a status page

Internal incidents never reach customers. Publishing is the explicit gate.

Every public read — the status page, its JSON API, the RSS feed, and the history markers — filters on visibility = 'public', so an internal incident on a public-component monitor never leaks. Monitors that sit on an enabled status page open public automatically; everything else (manual incidents, monitors not on a page) stays internal until you publish, either from the incident detail page or from the declare form itself.

From the incident detail page, publish flips visibility to public (optionally seeding a public title) and unpublish hides it again. A published incident appears on any status page whose components include its monitor.

An incident with no monitor, like a network outage in one data centre that touches every customer, has no component to follow. You pick its pages instead, one by one or all at once, on the declare form or the incident page, and can change the list while it is live. Over the API that is status_page_ids on POST /api/v1/incidents and on POST /api/v1/incidents/{id}/publish. Publish replaces the whole list, so to add or drop one page, read the current list from GET /api/v1/incidents/{id} and send it back changed. Publishing one to no page at all is refused with INCIDENT_STATUS_PAGE_REQUIRED rather than reported as a success nobody can see, and naming pages for an incident that has a monitor is refused with INCIDENT_STATUS_PAGES_WITH_MONITOR, since the monitor already decides where it shows. On those pages it sets the headline from its severity, and its updates reach their subscribers. Narrate it for customers with public updates (the investigating → monitoring → resolved timeline); posting an update is separate from the internal state, exactly as the two-axis model intends.

Postmortems

A resolved incident can carry one postmortem — a retrospective with a summary, root cause, impact, and a list of action items (each with optional owner and a done flag). Write it from the incident detail page (write / edit postmortem).

Publishing a postmortem surfaces it on the public incident page: customers see the summary, root cause, impact, and the action-item text and done state. Internal detail — the action-item owner — is never exposed publicly. A draft stays private until you publish, and publish/unpublish are recorded on the incident's activity timeline with the acting member, so the retrospective's own history is auditable.

Metrics and reporting

/incidents/reports is a metrics dashboard over a trailing window (7 / 30 / 90 days):

• MTTA — mean time to acknowledge (acknowledged_at − started_at).
• MTTR — mean time to resolve (ended_at − started_at).
• Total incidents, counts by severity and by state, auto-resolved vs human-resolved, and the noisiest monitors.

The same numbers are available to automation through the MCP get_incident_metrics tool.

MCP tools

An LLM connected through the MCP server can triage and operate incidents within its granted scopes: read the incident list (open ones, or resolved history in a window) and detail, read metrics, and — with write scope — acknowledge, resolve, publish or unpublish, and post public updates. Customer-supplied incident text is always returned as labelled data, never as instructions. See MCP server for the full tool table and scopes.

Auth and scopes

There is no incident-delete: incidents are resolved, never deleted, to keep the audit trail intact. Owner and member are the only roles — any member can be assigned, put on a schedule, paged, and can operate an incident; owners manage the escalation/on-call configuration.

Configuration

The [escalation] block (env prefix UPTIMEPAGE_ESCALATION__*) controls the engine:

Per-org limits (max_escalation_policies, max_on_call_schedules, on_call_enabled) are plan quotas; see Quotas & rate limits.