On-call adminnight duty for a Proxmox and OPNsense server
For a small business running a Proxmox server and an OPNsense firewall. Night alarms are handled by an on-call program: it fixes known faults by itself, takes a copy before every change, and sends me whatever it cannot settle as a card with options on my phone.
How it works
An alarm without a recipe, or a change that cannot be rolled back: a card goes straight to me, no action taken.
What it does
Since November 2026 the on-call program also handles alarms from the OPNsense firewall and errors from Loki logs.
Rules of the on-call program
- re-checks the alarm against the live metric
- restarts a failed service (except networking, SSH and the Proxmox cluster)
- checks services inside Proxmox machines and on OPNsense
- tests websites, ports and certificates
- heavier steps that may briefly interrupt work
- we set the window together, 02:00-05:00 by default
- server reboot after an update
- limiting machines that eat memory
- anything a recipe does not cover
- does not delete data
- does not change the network or firewall
- does not touch keys or payments
- does not write to people on your behalf
A recipe that failed twice within a day is not repeated: the case comes to me. Every step is written to a log on your server (90 days).
A report every week
On Monday at 7:00 you get a one-page e-mail in Polish. Example:
Dyżurny admin · raport · Firma Przykład · serwer pve1 · 07.11-14.11.2026
W skrócie: 672 przeglądy, alarmów: 3, zamknięte bez Państwa udziału: 2, czeka na decyzję: 1.
Czeka na decyzję:
• ram_available: czeka na decyzję (od 13.11 22:05)
Co się działo:
• 09.11 03:12 systemd_service_unit_failed_state: restart usługi → rozwiązane
• 11.11 14:40 disk_space_usage: kontrola miejsca na dysku → sprawdzone, bez zagrożenia If Netdata stops responding, I get a message straight away and the report says so plainly: a week without checks does not pretend to be a quiet week.
GitCloud Shield: included in the subscription
GitCloud’s own security system, included in the subscription. It checks messages and mail before anyone opens them and scores server security from 0 to 100.
What stays with you
- The on-call program is a systemd service on your server, its Python code lies open in /opt/dyzurny.
- Data, logs and backups stay on your hardware.
- Only e-mails leave the network: a decision request to me and the weekly report to you.
- Settings in one file: /etc/dyzurny/dyzurny.toml.
- Switched off and removed with one command: install.sh --uninstall.
A real example
Environment: one Proxmox server with containers and virtual machines, an OPNsense firewall, Netdata monitoring. As of 12 November 2026:
- A service did not start after an update: restart, metric check, alarm closed without waking anyone.
- Forecast "disk full in a few days": the on-call program checked free space right away. It was a short spike of temporary files and there was enough space. It reported this and deleted nothing.
- A new kernel after an update: host reboot only in the night window and with consent, then a check that the shutdown was clean and services came back.
Price
A fixed price for the scope, agreed in writing before the start. Contract and invoice on request.
How to start
- 1WriteHow many servers you have and what runs on them.
- 2AssessmentI check the environment, price and scope in writing.
- 3SetupOne week: the package installs in one evening, then recipes for your services and alarm tests.
- 4SubscriptionDuty every night, a report every week.
Questions
What if the on-call program does not know the alarm?
It changes nothing and sends me a card. Once solved, I add a recipe, so next time it handles it by itself.
Can the program break something?
The rules are written in the code: it does not delete data or change the network or firewall, heavier steps run only in the window, a server reboot only with consent. Configuration changes I make with a copy and automatic rollback.
What if monitoring itself goes down?
The on-call program checks Netdata before every run. If it does not respond, I get an e-mail straight away and the weekly report shows the gap.
Does it replace backups?
No. The review checks that backups are fresh, and a missing one is an alarm.
Is everything done remotely?
Yes. I do not do hardware repairs or on-site visits.