System and database health

What it is#

These two pages show whether Pulsimo itself is healthy:

• System Health probes every service, data store and pipeline of the stack, keeps their uptime, and shows the numbers that tell whether each one is doing its job.
• Database Health shows where the data lives, how much there is, how long each kind is kept, and whether the maintenance jobs ran.

Every number on both pages is measured; none is a placeholder. Every address, threshold, retention period and schedule comes from .env through docker compose.

Where to find it#

Sidebar → Observability:

• System Health
• Database Health

Addresses, container names and the configuration panel are shown to admins only.

System Health#

How it works#

The API gateway probes every component every SYSTEM_HEALTH_PROBE_SECONDS (30 s):

• HTTP /health for the services, VictoriaMetrics, VictoriaLogs and vmalert;
• SELECT 1 for PostgreSQL;
• PING for Redis;
• the login page for the web app.

Each probe gives one of four states:

Each probe is written to the metrics store as pulsimo_component_up{component=…} and pulsimo_component_latency_ms{component=…}. That gives real 24-hour uptime, the status bars, and series you can chart or alert on in Dashboards.

Components#

The overall verdict:

• Major outage when a core component is down;
• Partly degraded when any other component is down or slow;
• All systems operational otherwise.

A component you don't deploy (say, no log store) goes in SYSTEM_HEALTH_SKIP=victorialogs, so it is neither shown nor counted.

What each card shows#

Each card shows:

• the status, and how long it has been in it;
• what the component does;
• the probe time;
• 24-hour uptime;
• 96 status bars, one per 15 minutes. Green means up the whole time, amber partly down, red mostly down, grey no data yet;
• the exact error when it is not operational.

It also shows the numbers that tell whether it is doing its job:

For admins each card also has the address it probes and docker logs --tail 100 <container>, both one click to copy.

The top of the page shows:

• the verdict, listing each component that isn't operational and why;
• 24-hour figures: components up, monitored endpoints, checks and their failure share, open incidents (and how many came from logs), alerts sent, and log lines received. Each figure links to its page;
• Monitoring pipeline (Monitoring pipeline): check lag, checks per minute, scrape success and alert delay.

Configuration (admins) lists the settings behind these numbers: probes, retention, concurrency, log limits, incident thresholds. Each shows its current value and the variable that sets it, marked default when it isn't set in .env.

Database Health#

Run cleanup now and Deep cleanup (editors and admins) run those jobs on demand. The page refreshes when they finish.

Retention#

Schedules (cron, UTC)#

Settings#

Change .env, then run docker compose up -d.

Limits#

• Uptime and status bars come from the metrics store. If VictoriaMetrics is down, the cards still show the live state, but uptime shows "–" and the bars are grey for that time.
• The probes run from the API gateway. If the gateway is down, the page itself can't load. That is the "Major outage" you see first.
• Last run of the maintenance jobs resets when the scheduler restarts ("Not run since the scheduler started").