Skip to main content

docs Architecture

Architecture

uptimepage is one Rust binary that runs multi-tenant uptime monitoring and public status pages. Postgres holds configuration and control-plane state, ClickHouse holds check results, and extra regions are optional stateless probe processes. This page is the map: what the pieces are, how a request and a check flow through them, and which invariants every feature must respect. Deep dives live in the linked pages.

For an interactive companion to this page, open the flow map: pick any runtime path and watch it light up across every process, surface, service, and store.

Goals

  • Run periodic checks of eight kinds (HTTP, TCP, ping, heartbeat, DNS, TLS certificate, domain expiry, browser flow) against an arbitrary, mutable set of targets, plus manual monitors whose state an operator sets. See Monitor types.
  • Keep every tenant's data isolated by construction, not by convention. See Multi-tenancy.
  • Turn a run of failing checks into a confirmed incident, page the right people, and publish a customer-facing status surface. See Incident management and Public status page.
  • Attribute every result to the region that produced it, so a deployment can probe from many locations with no coordination. See Multi-region probes.
  • Survive hosts that hang (circuit breakers on repeated timeouts), tenant bursts (per-tenant host throttle), and storage flaps (in-process retry and batching), and count every dropped unit of work with a reason.
  • Drain in-flight work on shutdown rather than lose it.

Process modes

The same binary runs in one of three modes; the mode is chosen at startup, before any subsystem is built.

  • Control plane (default). Owns Postgres and ClickHouse, the REST API, the web UI, the MCP server, the marketing host, the scheduler for its own region, the incident writer, the escalation engine, public status pages, and the periodic jobs. Its own region is a normal regions row, not a sentinel.
  • Agent ([agent] enabled = true). A stateless probe: scheduler, registry, worker pool, batcher, sampler, and dispatch long-poller, and nothing else. No database, no web or API, no alerting. It pulls its region's decrypted monitor config from the control plane over an authenticated Bearer token and POSTs results back; region and agent identity are derived from the token server-side, never from the payload. See Multi-region probes.
  • Bootstrap CLI (uptimepage bootstrap-owner --email <addr>). Seeds the first user, org, and owner membership, mints a full-access API token, and exits. Runs against Postgres only. Installs with no terminal use [bootstrap] email instead, which seeds the same owner during control-plane startup and logs a sign-in link rather than a token. See First-run owner.

Module layout

src/
├── main.rs           startup: config load, validation, mode branch, subsystem spawn
├── lib.rs            crate root
├── app.rs            AppState, the composition root (every store + engine as a field)
├── router.rs         router assembly + middleware layer order
├── bootstrap.rs      bootstrap-owner CLI mode + first-run owner seeding
│
│   leaves: know nothing about the app, importable from anywhere
├── domain/           core types: Target, CheckSpec, CheckResult, Incident, on_call,
│                     notification_channel, quota, interpolation; no I/O
├── config/           typed AppConfig split per section + UPTIMEPAGE_ env override
│                     loader; boot.rs holds the refuse-to-start checks
├── error/            AppError -> ApiError envelope, the error code registry,
│                     the narrower public-API envelope
├── security/         AES-GCM envelope crypto, token and MAC hashing, SSRF guard,
│                     abuse deny-lists, redaction, RDAP and certificate probes
├── net/              happy-eyeballs dial (RFC 8305), canonical host forms
├── custom_domains.rs in-memory snapshot of the verified custom domains + host normalization
├── metric_names.rs   every metric name this process emits
├── templates/        askama filters, fingerprinted assets, timestamp formats
├── pagination/       page envelopes + opaque cursors shared by every list surface
├── text.rs           one-line and length-capped text shaping for outbound channels
├── duration.rs       two-unit duration split shared by the console and status page formatters
├── robots.rs         X-Robots-Tag values + the no-store, noindex marking of private responses
├── i18n.rs           status page and subscriber email messages in each page language
│
│   services and stores
├── storage/          Postgres + ClickHouse + in-memory stores behind traits;
│                     admin.rs / operator.rs are the audited tenancy escape hatches;
│                     locks.rs holds the advisory-lock helpers
├── quotas/           effective-plan resolution + governor rate limiter
├── targets/          monitor read models, region defaults, folded status, check validation
├── auth/             sessions, OAuth providers, passkeys, magic links, API tokens,
│                     invitations, login audit
├── billing/          plan moves and the paid lifecycle (provider-driven)
├── app_accounts.rs   who pressed an app's Acknowledge, and the offer to link them
│
│   probe pipeline
├── scheduler/        target registry (full re-list + diff), single-driver timing
│                     heap, gauge sampler
├── worker/           worker pool + per-monitor circuit breaker + host throttle +
│                     one executor per check kind (http/tcp/ping/heartbeat/dns/
│                     tls_cert/domain_expiry/flow/manual)
├── http_client/      poolless probe client: phase-timing connector, DNS cache,
│                     SSRF-guarded dial for the raw-socket checks
├── pipeline/         result batcher (size + timeout flush, bounded, counted drops)
├── ad_hoc_dispatch.rs  check-now / test long-poll to the region's agent
├── agent/            stateless agent-mode entry point
│
│   detection and delivery
├── public_status/    incident writer (poller), status-page aggregator, subscriber dispatch,
│                     the view that publishes a page change and busts the cached page
├── escalation/       paging engine + on-call resolution
├── notifier/         one transport per channel kind + IncidentNotice event
├── email/ telegram/ whatsapp/    transport-specific helpers
├── slack/ discord/   our Slack and Discord apps: signed button presses, replies
├── http_outbound.rs  shared outbound client for webhooks and provider APIs
├── jobs/             periodic jobs: retention, two-store erasure, token/session
│                     cleanups, no-data silence sweep, dead-man snitch
├── observability/    tracing + Prometheus + OTLP, readiness, fleet health gauges
│
│   HTTP surfaces
├── request/          what every surface reads off a request first: caller extractors,
│                     strict JSON bodies, client IP, host parsing, state cookies,
│                     time-range query, metrics and rate-limit layers
├── api/              REST /api/v1 handlers, routes, OpenAPI doc
├── web/              server-rendered operator UI (renders only; mutations call the API)
├── mcp/              in-process MCP server (typed, authorized, audited tools)
├── oauth/            OAuth 2.1 authorization server backing the MCP connector
├── channels/         notification-channel create/repair shared by the API and the UI
├── target_ops/       monitor create, interactive probe and manual state, shared by the
│                     API and MCP
├── maintenance_ops.rs  maintenance window rules shared by the API and MCP
├── analytics.rs      sign-in funnel events for Umami
├── marketing/        apex/www/blog/docs/landing pages; no storage or tenancy imports
└── bin/              the loadtest binary

templates/            askama HTML compiled into the binary
static/               rust-embed bundle: Tailwind 4 output (build.rs) + esbuild JS
migrations/           postgres/NNN_name.{up,down}.sql + clickhouse/*.sql

Dependency rule

Imports point downward through the groups above and never back up. A leaf imports other leaves only. A service or store imports leaves and other services, never a surface module. A handler module imports anything below it, never a sibling surface's handlers (logic two surfaces both need lives in target_ops, maintenance_ops, channels, request or a service), and the only thing above it is app, whose AppState every handler receives. That last edge is the deliberate exception: handlers take &AppState, so app is the one module every handler surface (api, web, mcp, oauth, channels) imports upward. app imports no surface back: the caches that hold a surface's own view models are built by that surface and reach its handlers as an Extension. Splitting AppState into per-surface sub-states would touch most handler files for no behavioural gain, so those upward arrows stay and are the known ones. request is the one slice worth the cost: its extractors and middleware sit under every surface, so they take RequestState, the fields they read, which app builds, and request never imports app.

The rule is what keeps marketing extractable and the agent mode small. tests/marketing_coupling_test.rs scans the marketing tree and fails on any crate:: path outside templates, security, request, custom_domains, http_outbound and net; the agent entry point reaches the probe pipeline, the store traits it needs and the leaves, and nothing from the HTTP surfaces. How the crate got to this shape, and the tool that found the cycles it replaced, is in Uncle Bob's uml-viewer on Rust: 35 dependency cycles down to 3.

Request path

One RouteByHost service (in src/marketing/dispatch.rs) inspects the Host header through the single host parser in src/request/host.rs and routes by class before any handler runs:

  • Marketing (apex and www): the marketing router, which touches no database.
  • App (the operator labels, app and mcp): the full application router. The MCP host is then narrowed to /mcp and /.well-known/* by the isolation middleware below.
  • Tenant public (any other subdomain, and any host the verified-custom-domain snapshot serves): the application router behind a default-deny fence that allows only the public status, subscribe, public API, and static paths. Everything else 404s, so login and operator routes can never appear on a tenant host.
  • Unknown (anything else): a bare 404. Serving the marketing site here would put our pages and their canonical tags on a host somebody else controls.

Inside the application router the middleware order is load-bearing and documented at the top of src/router.rs: http_metrics outermost, then host_isolation, then CSRF. Metrics must observe requests the later guards reject, and a tenant or MCP host must 404 an operator route before CSRF's constant-time compare runs. The /api/v1 stack adds a body limit, API-token auth, and per-org-then-per-user rate limiting. Reordering these changes request semantics, not just style.

Authentication resolves a session cookie or a scoped sm_live_ Bearer token into an authorization extractor (Authorized<Scope>, OwnerAuthorized<Scope>, CurrentOrg, and so on). See Authentication.

Probe path

By default the control plane schedules and runs every check itself. The pipeline from config to stored result:

Postgres (targets)
   │ TargetRegistry.refresh()  full re-list, diffed into added/updated/removed
   ▼
Scheduler        single-driver min-heap keyed by next-due instant; jittered tick;
   │             paused/deleted targets tombstoned in the heap and reaped lazily
   │ dispatch
   ▼
WorkerPool       passive fast path, then in-flight guard, semaphore, circuit
   │             breaker, per-tenant host throttle; each gate that drops work counts it
   │             ├── http / tcp / ping / dns / tls_cert / domain_expiry / flow executors
   │             └── heartbeat + manual (passive, evaluated inline)
   │ CheckResult on a bounded mpsc channel
   ▼
ResultBatcher    flush on size or timeout; buffer bounded, oldest dropped on overflow,
   │             retries exhausted counted
   ▼
ClickHouse       check_results (per-row TTL) + 1-minute and 1-hour rollup views

HTTP checks connect fresh every interval. There is no connection pool, because a monitor probes each target once per interval so a pool would rarely reuse a socket, and connecting fresh is exactly what lets the probe time DNS resolution, TCP connect, and the TLS handshake as separate phases and write them into each result. The connector resolves, applies the SSRF filter, dials with happy eyeballs, and optionally completes TLS, returning per-phase timings. See Monitor types.

On-demand checks (POST /targets/{id}/check-now and POST /targets/test) are dispatched to the target region's agent over a held long-poll, and the request waits for the result. If no agent is serving the region the request returns 503 PROBE_UNAVAILABLE; if every agent there is busy with checks it already claimed, 503 PROBE_BUSY. This path is single-process: the claim and the originating request must land on the same control-plane process.

Detection, incidents, and paging

Results do not page directly. A separate follower turns them into confirmed incidents:

  • Incident writer (src/public_status/incident_writer/) is a poller, not an event listener. On a default 30-second tick it keyset-paginates enabled targets across tenants, reads each target's recent results per a lookback tier, and applies the target's region quorum policy (Any, Majority, All, or Count) to decide up or down. Insert-open and close are race-safe, so exactly one writer pages. This confirmation step is why public status derives from confirmed incidents and never from raw samples.
  • Escalation engine (src/escalation/engine/) always runs and is the single source of down and up notifications: it opens a paging episode, walks the escalation ladder, renotifies, retries with backoff into a dead-letter, and resolves only to the channels paged this episode. On-call is never stored; who is on call is a pure function resolved at page time. A monitor with no policy pages its directly bound channels, just without the ladder. Whether an org may add paging coverage (a policy, schedule, override, paging contact, or binding) is the plan flag plans.on_call_enabled, checked on the write; a self-hosted install is exempt, and the plan's caps decide how many it may keep.
  • No-data detection (src/jobs/silence.rs) handles monitors whose covering regions all went dark, notifying bound channels once per episode. Above a fraction of the fleet it is treated as one infra outage and per-customer notices are suppressed.

Internal incident state (Triggered, Acknowledged, Resolved) and the public communication phase are orthogonal tracks and never share a field. See Incident management and Notifications.

Data model

Two backends, split by access pattern:

  • Postgres holds configuration and control-plane state: tenancy, monitors, incidents, paging and on-call, public-status config, auth, quotas, and ops tables. Low-cardinality, mutated by API operations. Migrations run via sqlx::migrate! at startup. Three high-churn audit tables (org_audit_log, login_attempts, quota_events) are monthly range-partitioned with a boot-time and daily partition maintainer.
  • ClickHouse holds check results only: append-only, high-cardinality, queried by time range. check_results carries a per-row TTL stamped from the org plan; two AggregatingMergeTree views roll it up to one-minute and one-hour grains. Reads for sparklines, buckets, and summaries go through a rollup, never raw. Migrations are idempotent CREATE ... IF NOT EXISTS statements run by a hand-rolled runner; editing a shipped one is a silent no-op on existing volumes, so schema changes are validated against a fresh volume. The result sink retries by re-sending an identical block, which the deduplication window collapses, so mid-batch checkpointing is forbidden.

Erasure spans both stores through a two-step outbox (src/jobs/purge_deleted.rs): the Postgres transaction enqueues and cascades in one shot, and the ClickHouse side drains idempotently and only settles a queue row after a count proves zero rows across all three result tables.

Every tenant-facing storage method takes org: OrgId as its first parameter, every child table carries a denormalised org_id with a parent-match trigger, and the isolation is enforced at the type, schema, script, and test level. The only ways around it are the two audited escape hatches (src/storage/admin.rs, which requires a static reason string, and src/storage/operator.rs, which is global by nature). See Multi-tenancy.

Key design choices

  • Sealed secrets. Channel and variable secrets are sealed with one AES-GCM envelope at the storage edge and redacted on every read. A read-then-write round trip that resubmits the redaction sentinel is rejected, so a UI edit can never overwrite a secret with its own mask. See Variables and secrets.
  • Quota in the insert. Handler-level quota checks exist only for a friendly error. The race-safe guarantee is (count) + 1 <= limit evaluated inside the INSERT under a per-org advisory lock. See Quotas and rate limits.
  • Explicit back-pressure. Every bounded queue and gate that drops a result increments uptimepage_storage_dropped_results_total with a reason label, so a dropped result is always counted, never swallowed. A tick that never ran loses no result and has its own counter: uptimepage_host_throttle_drops_total for the host throttle, uptimepage_checks_skipped_in_flight_total for a probe still running from the previous tick.
  • Per-monitor circuit breakers and a per-tenant host throttle. A monitor that keeps timing out opens its breaker, and its next checks skip the probe instead of holding a worker slot for the full timeout. They repeat its last timeout marked served_stale:, so the monitor keeps its real cause. A target that answers, even with an error, never opens it, and one monitor's breaker never gates another. A separate fail-fast semaphore, keyed per tenant, caps how many in-flight checks one tenant can run against the same host and port, so one customer's burst cannot starve another. Throttle drops are recorded as degraded and do not page; the upstream is fine, the back-pressure is operator-side.
  • Sticky last-good for domain expiry. Each successful RDAP probe writes the expiry, registrar, and last-success time. A subsequent transient failure serves the cached verdict with a served_stale: annotation rather than flipping the monitor; only staleness past a threshold escalates to an alert-eligible error. Concurrent probes for the same domain are collapsed to one outbound request.
  • Self-describing API. utoipa derives an OpenAPI 3.1 document at compile time, served at /api/openapi.json and rendered at /docs. The 4xx/5xx error envelope and the list page envelope are unified across every endpoint. See REST API.
  • Cancellation tokens for shutdown. The root token is cloned into the scheduler, batcher, sampler, jobs, and the graceful axum shutdown. SIGINT or SIGTERM cancels the root and subsystems drain together.

Concurrency model

  • One multi-threaded Tokio runtime.
  • One scheduler driver task owns a min-heap of (due, target) keyed by next-due instant. It sleeps until the earliest due entry, dispatches every due target, and reschedules at interval plus jitter. Memory stays flat in the fleet size rather than one task and timer per target. Paused and deleted targets are tombstoned by sequence number in a side map and reaped lazily on pop, not removed from the heap.
  • WorkerPool spawns a task per dispatch, gated by a semaphore sized to the configured concurrency; the heartbeat check runs inline with no spawn, permit, breaker, or throttle.
  • The batcher, sampler, and each periodic job are single tasks driven by tokio::select! over their trigger and the cancellation token. Every periodic job runs under a Postgres advisory lock keyed by job name, so in a multi-replica deployment only one replica executes it.

Where region partitioning lives

Region assignment is not in the scheduler. It lives in the EnabledTargetSource implementations (RegionTargetSource and HeartbeatTargetSource in src/scheduler/sources.rs, AgentPullSource in src/agent/mod.rs), and the assignment table is target_regions. Results carry their region as a low-cardinality column through both ClickHouse rollups, so reads can slice by region and quorum policies can require agreement across regions. See Multi-region probes.

Multi-tenancy · Multi-region probes · Monitor types · Incident management · Notifications · Public status page · Authentication · Quotas and rate limits · MCP server · Configuration · Development