Monitoring pipeline

Check lag, scrape success, alert delay and series per source.

What it is#

A monitoring system that is itself degraded is the classic blind spot: checks run late, scrapes fail, alerts go out slowly, and nobody notices. Pulsimo's services report their own metrics, and System Health shows them.

Where to find it#

System Health → Monitoring pipeline (admins). The rest of System Health (every component probed, uptime, storage) is in System and database health.

What it shows#

Tile Meaning Turns amber / red
Check lag How far the most overdue check is behind its schedule > 15 s / > 60 s
Checks / min Checks run, with the number of active endpoints
Scrape success Share of successful scrapes in the last 15 minutes, scrapes per minute, samples per second, and sources currently boosted for an incident < 98% / < 90%
Alert delay Average time from event to handled notification over the last hour, notifications sent and failed, and alerts held for deploys in 24 h any failed notification

Chips at the top show whether the Checker, Metrics collector and Notifications services reported in the last 2 minutes. A red chip means the service is down or can't reach the metrics store.

Series per source#

Metrics → Sources → Series shows each source's active series, warns above 10,000, and explains what drives them (Metric sources). One misbehaving exporter can't slow everything down unnoticed.

Pulsimo's metrics#

All have pulsimo_internal="true" and job="pulsimo", with service set to the reporting service. They are instance-wide (not per organization) and are written every 30 seconds.

Series Service
pulsimo_checker_checks_total{result} checker
pulsimo_checker_check_lag_seconds checker
pulsimo_checker_active_endpoints checker
pulsimo_collector_scrapes_total{result} metrics-collector
pulsimo_collector_samples_total metrics-collector
pulsimo_collector_sources metrics-collector
pulsimo_collector_boosted_sources metrics-collector
pulsimo_notifications_total{result} notification
pulsimo_notification_events_total notification
pulsimo_notification_delay_ms_sum notification
pulsimo_notification_held_for_deploy_total notification

Counters restart from zero when a service restarts; increase() and rate() handle that.