For a long time I believed my homelab was monitored. I had dashboards for the hypervisor, dashboards for the containers, a health script on the VPN gateway, and a metrics hub. Everything was green.
Then three separate things broke for weeks, and nothing told me. The dashboards were fine. The problem was that none of them could actually reach me, and the checks behind them were answering the wrong question.
The False Sense of Safety
A green dashboard tells you that something computed a green value. It does not tell you that a human will find out when the value turns red. Those are two different systems:
- Looking: a dashboard you open when you already suspect a problem
- Paging: something that interrupts you when a check fails, whether or not you are looking
I had built a lot of the first and almost none of the second. And the checks feeding the dashboards were generic ones: is the port open, is the container running, does it answer ping. Those are uniform across every host, which is exactly why they are blind. A reverse proxy, an app server and a database all keep listening while the application above them is broken, migrating, or refusing requests.
Incident 1: The Healthy Container Serving a Maintenance Page
During a patch sweep I updated the container running a file-sharing app and then checked it. Everything I looked at passed:
- no pending updates
- the web, database and cache ports all listening
- the web server running
- zero newly failed systemd units
I called it healthy. Then I opened it in a browser and found a “Maintenance mode” page. The app had been serving nothing else the whole time.
Every check I ran was true, and none of them was the check that mattered. “Ports unchanged, zero failed units” is a much narrower claim than “healthy”, and I had worded it as the second one.
The detail that stung: Uptime Kuma was already watching these services and would have shown the problem immediately. I never opened it. The first question after any change should be “what already watches this?”
Incident 2: The 19-Day Silent Outage
My remote-access gateway has a health check script, and I was fairly proud of it. What it did on failure was write a line to a log file. That was all.
In April 2026 the gateway was broken for 19 days. The health check noticed. It logged a sustained run of warnings, one for every check, every day. Nobody read the log, so nobody knew until I went looking much later.
That is the shape of the whole problem. The signal existed. It just had no path to a human. A health check without alerting is a diary.
Incident 3: The Dashboard That Cried Wolf, and Never Paged
I had adopted Pulse as an infrastructure dashboard and assumed it would also alert me. It did not. Looking back at what it could do:
- it never sent a single notification
- its HTTP check was reachability-only, so it could not assert anything about page content
- it had no push monitors, so it could not listen for a job reporting in
Worse, it showed a permanent false “Critical” for some of my apps. Those apps ship a one-shot init container that checks file ownership and exits 0 by design. Pulse flagged any container whose state was not running, without looking at the exit code. So an app showed as failed while it was healthy and the platform itself reported it as running.
The container stays in the exited state until the next restart, so the alert never cleared. A permanent false alarm is nearly as bad as no alarm, because you learn to ignore the red.
The fix for that specific case is to check the exit code, not the state. An exited container that returned 0 did its job.
Beszel had a similar story in miniature. Its built-in “down” status silently failed to notify anyone for seven weeks.
What Changed
One alerting layer
Uptime Kuma is now the single place that pages me, as of 2026-10-11. Dashboards are for looking. Kuma is for paging, and the other tools stay in their lane:
- Kuma: is a service broken
- Beszel: kept deliberately host-only, with a small timer that checks how recent its last datapoint is, because a hub serving cached data looks identical to a healthy one
- Pulse: a view of the infrastructure, with no job as an alerter
One application-level assertion per service
Each Kuma monitor asserts something that fails when the app is broken. A few examples of the pattern:
| Kind of service | What the check asserts |
|---|---|
| File-sharing app | the status endpoint’s JSON says it is not in maintenance and needs no database upgrade |
| Git hosting | its own health endpoint reports pass |
| Photo library | its ping endpoint answers pong |
| Identity provider | the readiness endpoint, which checks the database and cache (the liveness one does not) |
The file-sharing one is the case that started this. Its status endpoint returns HTTP 200 even while in maintenance mode, with the maintenance flag set to true in the body. A plain status-code check would have stayed green through the whole thing. Only reading the body catches it.
I tested each assertion against both a healthy and a broken response before creating the monitor, so I knew it could actually fail.
Push monitors for jobs
Some things are not services you can poll. For those, the job reports in, and silence is the alert. The gateway’s health script now re-verifies its own state after trying to fix it, then pushes either “up” or “down” with a reason. If it stops pushing for long enough, that alerts too.
I also broke things on purpose to prove the path end to end: I pointed one monitor at a broken URL, and moved aside the push URL file on the gateway. In both cases I checked that the email actually arrived in my inbox, not just that the heartbeat showed red in the database.
The Client’s Screen Is Not Evidence
One related lesson came out of debugging the VPN. My phone’s VPN app showed “Connected” and went warning-free at a moment when the gateway could not reach the phone at all. Byte counters were frozen and pings timed out. At other points it raised warnings while everything worked.
A client app reports its own intent and last known state, not end-to-end reachability. Measure from the side whose behaviour matters. For a VPN, that means asking the gateway, not the phone. For a web app, it means fetching the page, not checking that the process is up.
A Short Checklist
- Pair every generic check (port, ping, container running) with one assertion that fails when the app is broken. Read the response body if a 200 can still be a maintenance page.
- Ask whether a person would hear about a failure. A log file, a dashboard colour or a status page is not a notification.
- Test the alert path by breaking something deliberately and confirming the message lands in your inbox.
- For jobs and cron scripts, use a push monitor so silence raises the alarm.
- Check exit codes, not states, for one-shot containers.
- When reporting a change as done, say what you actually asserted.
- Decide what watches the watcher. A silent alerting tool looks exactly like health.
None of this is sophisticated. The hard part was admitting that “I have monitoring” and “someone will be told” are separate claims, and I had only ever verified the first.