309 lines
10 KiB
Markdown
309 lines
10 KiB
Markdown
# ns8-backup-monitor
|
||
|
||
A lightweight webhook receiver for **NethServer 8** that intercepts Alertmanager backup failure alerts, enriches them with per-module status data from the cluster Redis, optionally checks repository health via `restic`, and delivers a detailed email notification through the NS8 configured mail relay.
|
||
|
||
Unlike solutions that hook into `run-backup` (which only fires on manual UI launches), this service listens to the Alertmanager webhook channel — the same source used by the NS8 monitoring stack — and therefore captures **both manual and scheduled automatic backups**.
|
||
|
||
---
|
||
|
||
## Architecture overview
|
||
|
||
```
|
||
Alertmanager
|
||
│ POST /alert (NsBackupFailed | NsBackupMissing)
|
||
▼
|
||
[receiver.py] HTTP webhook listener (localhost:9099)
|
||
│ waits N seconds for modules to settle
|
||
▼
|
||
[correlator.py] Reads Redis cluster state, classifies outcome
|
||
│ SUCCESS | PARTIAL | REPO_FAILURE
|
||
▼
|
||
[repo_check.py] (only on non-SUCCESS) Probes restic repos via runagent
|
||
▼
|
||
[notifier.py] Builds HTML/text email, sends via ns8-sendmail
|
||
```
|
||
|
||
---
|
||
|
||
## Requirements
|
||
|
||
| Dependency | Notes |
|
||
|---|---|
|
||
| NS8 leader or worker node | Must have access to the cluster Redis socket |
|
||
| `redis-cli` | Included in standard NS8 installations |
|
||
| `runagent` | NS8 binary used to invoke `restic` inside module containers |
|
||
| `ns8-sendmail` | NS8 mail relay script (invoked via `runagent`) |
|
||
| Python 3.8+ | Standard library only — no pip dependencies |
|
||
| Alertmanager | Must be configured to send webhooks to this service |
|
||
|
||
---
|
||
|
||
## File layout
|
||
|
||
```
|
||
ns8-backup-monitor/
|
||
│
|
||
├── README.md ← This file
|
||
│
|
||
├── config/
|
||
│ └── config.yml.example ← Annotated configuration template
|
||
│
|
||
├── deploy/
|
||
│ ├── install.sh ← Interactive installer / uninstaller
|
||
│ └── ns8-backup-monitor.service ← systemd unit file
|
||
│
|
||
└── ns8_backup_monitor/ ← Python package (main application)
|
||
├── __init__.py ← Package marker, exposes version
|
||
├── __main__.py ← CLI entry point (`python3 -m ns8_backup_monitor`)
|
||
├── receiver.py ← HTTP webhook server (Alertmanager → pipeline)
|
||
├── correlator.py ← Redis reader and outcome classifier
|
||
├── repo_check.py ← restic repository health prober
|
||
├── notifier.py ← Email builder and sender
|
||
└── utils.py ← Config loader and logging setup
|
||
```
|
||
|
||
### Runtime paths (after installation)
|
||
|
||
| Path | Purpose |
|
||
|---|---|
|
||
| `/opt/ns8-backup-monitor/` | Application root (Python package) |
|
||
| `/etc/ns8-backup-monitor/config.yml` | Active configuration file |
|
||
| `/etc/systemd/system/ns8-backup-monitor.service` | systemd unit |
|
||
| `/var/log/ns8-backup-monitor/` | Log directory (if file logging is enabled) |
|
||
| `/var/lib/nethserver/cluster/state/redis.sock` | NS8 cluster Redis socket (default) |
|
||
|
||
---
|
||
|
||
## Installation
|
||
|
||
### Quick install (interactive)
|
||
|
||
```bash
|
||
bash <(curl -fsSL https://repo.lelekaos.com/admin/ns8-backup-monitor/raw/branch/main/deploy/install.sh)
|
||
```
|
||
|
||
> **Note:** Use `bash <(curl ...)` rather than `curl ... | bash`.
|
||
> The interactive installer reads answers from your terminal via `read`; piping stdin
|
||
> from curl breaks that interaction.
|
||
|
||
### Non-interactive install (CI / automation)
|
||
|
||
```bash
|
||
curl -fsSL https://repo.lelekaos.com/admin/ns8-backup-monitor/raw/branch/main/deploy/install.sh \
|
||
| bash -s -- \
|
||
--from "backup@example.com" \
|
||
--to "admin@example.com"
|
||
```
|
||
|
||
### What the installer does
|
||
|
||
1. Copies the Python package to `/opt/ns8-backup-monitor/`
|
||
2. Writes `/etc/ns8-backup-monitor/config.yml` from the template
|
||
3. Installs and enables the systemd unit
|
||
4. Prints the Alertmanager webhook receiver URL
|
||
|
||
---
|
||
|
||
## Uninstallation
|
||
|
||
```bash
|
||
bash /opt/ns8-backup-monitor/deploy/install.sh --uninstall
|
||
```
|
||
|
||
The uninstaller stops and removes the systemd unit, then optionally removes the configuration directory.
|
||
|
||
---
|
||
|
||
## Configuration
|
||
|
||
The active configuration file is `/etc/ns8-backup-monitor/config.yml`.
|
||
Edit it directly and restart the service to apply changes.
|
||
|
||
```bash
|
||
nano /etc/ns8-backup-monitor/config.yml
|
||
systemctl restart ns8-backup-monitor
|
||
```
|
||
|
||
See `config/config.yml.example` for a fully annotated reference with all available options.
|
||
|
||
### Key sections
|
||
|
||
```yaml
|
||
# ── Mail settings ─────────────────────────────────────────────
|
||
mail:
|
||
from: "backup@ns02.example.com" # Envelope From address
|
||
to:
|
||
- "admin@example.com" # One or more recipient addresses
|
||
subject_prefix: "[NS8 Backup]" # Prepended to every subject line
|
||
|
||
# ── Webhook receiver ──────────────────────────────────────────
|
||
receiver:
|
||
host: "127.0.0.1" # Bind address (keep localhost unless Alertmanager is remote)
|
||
port: 9099 # Must match the Alertmanager webhook URL
|
||
|
||
# ── Correlator behaviour ─────────────────────────────────────
|
||
correlator:
|
||
wait_seconds: 30 # Seconds to wait after alert before reading Redis
|
||
# (allows slow modules to write their final status)
|
||
recent_window: 3600 # When no backup_id label is present, scan Redis for
|
||
# plan status keys updated within this many seconds
|
||
|
||
# ── Redis connection ─────────────────────────────────────────
|
||
redis:
|
||
socket: "/var/lib/nethserver/cluster/state/redis.sock"
|
||
|
||
# ── Repository health check ──────────────────────────────────
|
||
repo_check:
|
||
enabled: true
|
||
timeout: 60 # Seconds per restic check call
|
||
|
||
# ── Logging ──────────────────────────────────────────────────
|
||
logging:
|
||
level: "INFO" # DEBUG | INFO | WARNING | ERROR
|
||
file: "" # Leave empty to log to stdout (journald captures it)
|
||
```
|
||
|
||
---
|
||
|
||
## Alertmanager integration
|
||
|
||
Add a receiver to your Alertmanager configuration on the NS8 leader node
|
||
(`/etc/alertmanager/alertmanager.yml` or via the NS8 `metrics1` module):
|
||
|
||
```yaml
|
||
receivers:
|
||
- name: ns8-backup-monitor
|
||
webhook_configs:
|
||
- url: "http://127.0.0.1:9099/alert"
|
||
send_resolved: false
|
||
|
||
route:
|
||
receiver: ns8-backup-monitor
|
||
group_by: [alertname]
|
||
group_wait: 10s
|
||
group_interval: 5m
|
||
repeat_interval: 12h
|
||
routes:
|
||
- match_re:
|
||
alertname: "NsBackupFailed|NsBackupMissing"
|
||
receiver: ns8-backup-monitor
|
||
```
|
||
|
||
The service handles two alert names:
|
||
|
||
| Alert name | Meaning |
|
||
|---|---|
|
||
| `NsBackupFailed` | One or more backup modules reported an error |
|
||
| `NsBackupMissing` | Expected backup did not run within the time window |
|
||
|
||
---
|
||
|
||
## Outcome classification
|
||
|
||
After reading per-module Redis keys, the correlator assigns one of three outcomes:
|
||
|
||
| Outcome | Condition | Email subject |
|
||
|---|---|---|
|
||
| `SUCCESS` | All modules succeeded | ✅ Backup completed successfully |
|
||
| `PARTIAL` | Some modules failed, some succeeded | ⚠️ Backup partially failed |
|
||
| `REPO_FAILURE` | All modules failed, or no status found in Redis | ❌ Backup failed – possible repository error |
|
||
|
||
On `PARTIAL` or `REPO_FAILURE`, the repo health check runs automatically and appends
|
||
diagnostic information (restic error output) to the email.
|
||
|
||
---
|
||
|
||
## Redis key structure
|
||
|
||
The correlator reads the following NS8 Redis key patterns:
|
||
|
||
```
|
||
cluster/backup/<backup_id>/status → overall plan status (hash)
|
||
module/<module_id>/backup/<backup_id>/status → per-module status (hash)
|
||
```
|
||
|
||
Hash fields:
|
||
|
||
| Field | Values | Description |
|
||
|---|---|---|
|
||
| `result` | `success` / `error` | Outcome of the backup operation |
|
||
| `timestamp` | ISO 8601 | When the status was last written |
|
||
| `error` | string | Error message, if any |
|
||
| `errors` | integer | Number of module errors (plan-level hash only) |
|
||
|
||
---
|
||
|
||
## Service management
|
||
|
||
```bash
|
||
# Check service status
|
||
systemctl status ns8-backup-monitor
|
||
|
||
# View live logs
|
||
journalctl -u ns8-backup-monitor -f
|
||
|
||
# Restart after config change
|
||
systemctl restart ns8-backup-monitor
|
||
|
||
# Disable on boot
|
||
systemctl disable ns8-backup-monitor
|
||
```
|
||
|
||
---
|
||
|
||
## Troubleshooting
|
||
|
||
### Service fails to start
|
||
|
||
```bash
|
||
journalctl -u ns8-backup-monitor --no-pager -n 50
|
||
```
|
||
|
||
Common causes:
|
||
- `config.yml` not found at the expected path → check `/etc/ns8-backup-monitor/config.yml`
|
||
- Port 9099 already in use → change `receiver.port` in config
|
||
|
||
### No email received after a backup failure
|
||
|
||
1. Verify Alertmanager is firing the webhook:
|
||
```bash
|
||
journalctl -u ns8-backup-monitor -f
|
||
```
|
||
You should see `Received N relevant alert(s)` within a minute of the backup failure.
|
||
|
||
2. Check that `wait_seconds` has elapsed (default 30 s) and look for `Sending notification...` in the log.
|
||
|
||
3. Verify the mail relay works independently:
|
||
```bash
|
||
echo "Test" | runagent ns8-sendmail -s "test" admin@example.com
|
||
```
|
||
|
||
### Correlator finds no modules
|
||
|
||
If the log shows `No recent backup status keys found in Redis`, possible causes:
|
||
- `recent_window` is too short — the backup ran more than 1 hour ago
|
||
- Redis socket path is wrong for your installation
|
||
- The backup plan wrote status to a non-standard key pattern
|
||
|
||
---
|
||
|
||
## Development
|
||
|
||
The application is pure Python 3 with no third-party dependencies.
|
||
|
||
```bash
|
||
# Run locally (requires NS8 Redis socket access)
|
||
python3 -m ns8_backup_monitor --config ./config/config.yml.example
|
||
|
||
# Send a test webhook payload
|
||
curl -s -X POST http://127.0.0.1:9099/alert \
|
||
-H "Content-Type: application/json" \
|
||
-d '{"alerts":[{"status":"firing","labels":{"alertname":"NsBackupFailed","backup_id":"1"}}]}'
|
||
```
|
||
|
||
---
|
||
|
||
## License
|
||
|
||
MIT License — contributions welcome via pull request.
|