helmsight

Watch every server from one screen. Install nothing on them.

helmsight is an agentless, self-hosted web dashboard for a fleet of Linux servers. One binary on a central machine connects to your servers over plain SSH, collects metrics and inventory with read-only commands, keeps history in an embedded SQLite database and serves a fast web UI. Nothing is installed, copied or written on the monitored servers.

Status: early development (0.x). helmsight is new. It is tested on Debian, Ubuntu, Rocky, Fedora, openSUSE and Alpine, but it has had little production use and no independent security audit. Configuration options, the API and the database format may change between 0.x releases. Try it on non-critical hosts first, keep a backup of the data directory, and review the security model and the collection script before deploying widely. Collection is read-only; the opt-in Actions feature runs the commands you configure, so enable it with care. helmsight is provided as is, without warranty, under the MIT license. You use it at your own risk.

What you get

• Fleet overview with status, CPU, memory, load, the fullest disk, network, uptime and alerts per host; groups, tags, search and sparklines.
• Host detail with live and historical charts, a per-core CPU heatmap, processes, listening ports, services, containers and system information.
• History for up to 90 days with automatic rollups.
• Drift tracking: what changed since yesterday, host-to-host and baseline comparison of packages, ports, units, kernel and OS.
• Security overview: failed SSH logins, pending security updates, unexpected ports, reboot flags and TLS certificate expiry.
• Alerts with durations and scopes, acknowledgements, silences and notifications by webhook, Slack or email.
• Actions (opt-in): pre-approved commands with confirmation and an append-only audit log.
• Users and roles, TOTP, OpenID Connect single sign-on.
• Wall display for NOC screens, a JSON API with OpenAPI, server-sent events and a Prometheus endpoint.

How it works

1. helmsight keeps one persistent SSH session per host. Every few seconds it runs sh -s on that session and sends a short POSIX shell script on standard input. The script is never written to a file on the remote host and does not appear in its process list.
2. The script only reads: /proc files, df, ps, ss, systemctl, the package database and similar sources. When a tool is missing or not permitted, it reports “n/a” with the reason instead of failing.
3. The output is size-limited, parsed in Rust on the central machine and written to SQLite. Older data is condensed into 1-minute and 5-minute rollups.
4. The web UI is embedded in the binary. The browser only runs code served by helmsight; text coming from remote hosts is never rendered as HTML.

Heavier checks (pending updates, the package inventory) run less often. The security model lists every command and how often it runs.

Design principles

1. Agentless. Nothing is installed or written on monitored hosts.
2. Read-only by default. Only the opt-in Actions feature can change a host, and only with commands an administrator wrote down.
3. No arbitrary command execution. No terminal, no command box.
4. Secure by default. Host keys are verified, the server binds to loopback, secrets are encrypted at rest.
5. Remote output is untrusted. It is size-limited, parsed in Rust and never rendered as HTML.
6. Low overhead. One persistent SSH session and one short shell script per host every few seconds.
7. Portable. Debian, Ubuntu, RHEL/Rocky/Alma, Fedora, openSUSE and Alpine (BusyBox).
8. Single binary. The web UI is embedded; deployment is copying one file.

About this documentation

The documentation is available in English and Turkish; the link in the top bar of every page opens the same page in the other language. The UI itself is in English. Menu and button names are written in bold as they appear in the UI, for example Host keys. Configuration keys, commands and API paths are written as code.

Continue with Installation or jump straight to the Quick start.

Installation

helmsight is a single static binary. It runs on the central machine only; monitored hosts need nothing but an SSH server and a POSIX shell (/bin/sh).

Requirements

Disk usage grows with the number of hosts and the retention periods, which are set in the [retention] section.

Installation script

On a Linux server with systemd, the installation script sets everything up in one go:

curl -fsSLO https://enderkus.github.io/helmsight/install.sh
less install.sh            # read it before running it as root
sudo sh install.sh

It asks a few questions (web UI address, external URL, account on the monitored hosts, this server’s address, first administrator and password), shows what it is going to do and asks for confirmation. Then it:

1. downloads the release archive for this machine’s architecture and verifies its SHA-256 checksum against SHA256SUMS; it stops on a mismatch;
2. installs /usr/local/bin/helmsight and creates the helmsight system user;
3. runs helmsight init, which writes /etc/helmsight/helmsight.toml, creates the SSH key /etc/helmsight/id_ed25519, the data directory /var/lib/helmsight and the first administrator;
4. installs and starts the hardened systemd service;
5. prints the address of the UI and the command that prepares each monitored host (see Monitoring a fleet).

It does not touch the firewall, TLS certificates or a reverse proxy, and it never overwrites an existing configuration, key or database.

With --yes, no password is asked for: the service prints a one-time setup link for the first administrator, and the script shows it at the end. This also works in automation such as cloud-init:

sh install.sh --yes --listen 0.0.0.0:8443 --public-url https://monitor.example.com --from 10.0.0.5

Running the script again on an installed machine upgrades the binary, restarts the service and keeps the configuration and data. Without systemd (for example on Alpine), it installs the binary and the configuration and tells you how to start helmsight.

Prebuilt binaries

Releases provide static Linux binaries for x86_64 and aarch64, with SHA-256 checksums:

curl -LO https://github.com/enderkus/helmsight/releases/latest/download/helmsight-x86_64-unknown-linux-musl.tar.gz
curl -LO https://github.com/enderkus/helmsight/releases/latest/download/SHA256SUMS
sha256sum --check --ignore-missing SHA256SUMS
tar xzf helmsight-x86_64-unknown-linux-musl.tar.gz
sudo install -m 0755 helmsight-x86_64-unknown-linux-musl/helmsight /usr/local/bin/
helmsight --version

On ARM servers (for example AWS Graviton, or a Raspberry Pi 4/5 running 64-bit Linux), replace x86_64 with aarch64.

The sha256sum --check line verifies that the archive is intact and identical to the file on the release page; do not continue until it prints OK.

Container image

docker run -d --name helmsight -p 8443:8080 \
  -v helmsight:/var/lib/helmsight \
  -v /etc/helmsight:/etc/helmsight:ro \
  ghcr.io/enderkus/helmsight

In the container, set listen = "0.0.0.0:8080" and data_dir = "/var/lib/helmsight" in /etc/helmsight/helmsight.toml. Because it listens on a non-loopback address, TLS is enabled automatically: unless you provide a certificate, a self-signed one is generated and its fingerprint is logged at start-up. Open https://<machine>:8443 in the browser.

The image contains only the binary and CA certificates (not even a shell). --local mode is therefore not available in it, and commands run as docker exec helmsight /helmsight <command>, for example:

docker exec -it helmsight /helmsight user add alice --role admin
docker exec -it helmsight /helmsight hosts test --trust

Keep the SSH key and the configuration under /etc/helmsight, mounted read-only, and the database in a named volume (helmsight). The image runs as an unprivileged user with UID 65532; mounted files must be readable by that user.

Building from source

Requirements: Rust (stable), Node.js 22 and npm.

git clone https://github.com/enderkus/helmsight.git
cd helmsight
(cd web && npm ci && npm run build)
cargo build --release
./target/release/helmsight --version

The web UI is embedded into the binary at compile time, so build it in web first. A cargo build without a built UI prints a warning and embeds a simple placeholder page instead.

To build a static Linux binary on another platform (for example macOS), build the container image and copy the binary out of it:

docker build --platform linux/amd64 -t helmsight .
docker create --name hs helmsight
docker cp hs:/helmsight ./helmsight && docker rm hs

Running as a service

The repository ships a hardened systemd unit, examples/helmsight.service:

# A dedicated system user
sudo useradd --system --home-dir /var/lib/helmsight --shell /usr/sbin/nologin helmsight

# Configuration and SSH key: owned by root, readable by the helmsight group
sudo install -d -m 0750 -o root -g helmsight /etc/helmsight
sudo install -m 0640 -o root -g helmsight helmsight.toml /etc/helmsight/
sudo install -m 0640 -o root -g helmsight id_ed25519 /etc/helmsight/

sudo cp helmsight.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now helmsight
journalctl -u helmsight -f

The unit runs helmsight as an unprivileged user with a read-only file system, a private state directory (/var/lib/helmsight) and a restricted set of system calls. Use data_dir = "/var/lib/helmsight" in the configuration.

To create the first user, either open the setup link printed in the service log, or run:

sudo -u helmsight helmsight --config /etc/helmsight/helmsight.toml user add alice --role admin

Upgrading

1. Read the changelog of the new release; 0.x releases may contain incompatible changes.
2. Back up the data directory (see below).
3. Install the new binary over the old one and restart the service: sudo systemctl restart helmsight. The database schema is migrated automatically at start-up when needed.

With the container image, pull the new image and recreate the container with the same volumes.

Backups

The data directory contains:

For a consistent database backup, stop the service briefly and copy the files, or use SQLite’s backup command while it runs:

sqlite3 /var/lib/helmsight/helmsight.db ".backup '/backup/helmsight.db'"

Store secret.key separately from the database: without it, stored secrets cannot be decrypted, and anyone who obtains both can read them.

Uninstalling

helmsight installs nothing on monitored hosts, so uninstalling only touches the central machine: stop the service and delete the binary, /etc/helmsight and the data directory. You can also remove the monitoring account and its authorized_keys line from the hosts.

Quick start

This page shows two ways to get helmsight running in a few minutes: trying it on one machine without SSH, and monitoring real servers over SSH.

Try it on one machine

On a Linux machine, local mode monitors the machine itself by reading /proc directly. No SSH and no configuration file are needed:

./helmsight serve --local

helmsight prints a one-time setup link:

  No users exist yet. Create the first administrator:

    http://127.0.0.1:8080/setup#token=…

Open it, choose a user name and a password (at least 12 characters), and you are in. Data is kept in ~/.local/share/helmsight (or /var/lib/helmsight when running as root).

The token is in the part of the address after #; browsers do not send that part to the server, so it is not written to any log. While no user exists, helmsight prints a new link at every start.

Local mode reads /proc, so it only works on Linux. On other platforms, monitor servers over SSH as described below.

Install on a server

On a Linux server with systemd, the installation script installs helmsight as a service, creates the configuration, the SSH key and the first administrator, and prints the next steps:

curl -fsSLO https://enderkus.github.io/helmsight/install.sh
sudo sh install.sh

The rest of this page shows the same steps by hand.

Monitor servers over SSH

Four steps:

./helmsight init                    # configuration file, key file, first admin
$EDITOR helmsight.toml              # add [[hosts]] entries
./helmsight hosts test --trust      # check SSH and confirm host keys
./helmsight serve

1. init asks a few questions (listen address, data directory, SSH user, path of the SSH private key, name and password of the first administrator). It then writes a commented helmsight.toml and creates the data directory and secret.key, which encrypts stored secrets. It does not generate the SSH key; the next page explains how.

2. Add the hosts to monitor in helmsight.toml:

   [[hosts]]
   name = "web-1"
   address = "10.0.0.11"
   groups = ["web"]
   

3. hosts test --trust connects to every host, shows the fingerprint of host keys it has not seen before and asks you to confirm them, then prints a one-line result per host. Create the monitoring account on each server first, for example with helmsight hosts bootstrap --from <this server's address> | ssh root@web-1 sh; see Monitoring a fleet.

4. serve starts the web server and the collectors.

Open http://127.0.0.1:8080 and sign in with the administrator you created during init. Hosts show as Pending for a few seconds, until their first collection.

Reaching the UI from other machines

helmsight listens on 127.0.0.1 by default, so the UI is only reachable from the same machine. There are two ways to open it to others:

1. Listen directly. Set listen = "0.0.0.0:8443" in [server]. TLS is enabled automatically: provide your own certificate with [server.tls] mode = "files", or let helmsight generate a self-signed one. The fingerprint of a generated certificate is logged at start-up; compare it before accepting the browser’s warning.

[server]
listen = "0.0.0.0:8443"
public_url = "https://monitor.example.com:8443"

[server.tls]
mode = "files"
cert = "/etc/helmsight/tls/fullchain.pem"
key = "/etc/helmsight/tls/privkey.pem"

2. Behind a reverse proxy. helmsight stays on the loopback address and a proxy such as nginx, Caddy or Traefik terminates TLS. Set public_url and trusted_proxies:

[server]
listen = "127.0.0.1:8080"
public_url = "https://monitor.example.com"
trusted_proxies = ["127.0.0.1"]

An example nginx configuration (turning off buffering matters for server-sent events):

server {
    listen 443 ssl;
    server_name monitor.example.com;
    # ssl_certificate ... ; ssl_certificate_key ... ;

    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_http_version 1.1;
        proxy_buffering off;
        proxy_read_timeout 1h;
    }
}

public_url is required for OpenID Connect; it is also used in links in notifications and by the origin check that refuses requests from other sites.

Next steps

• Monitoring a fleet: the monitoring account, the host list and host key approval.
• Alerts and notifications: rules such as disk usage, and Slack or email notifications.
• Security model: exactly what helmsight runs on your servers.

Monitoring a fleet

This page walks through bringing a fleet of servers under monitoring, step by step. In the examples, the machine running helmsight has the address 10.0.0.5 and the monitoring account is called monitor.

1. Create a key for helmsight

If you used the installation script or helmsight init --generate-key, the key already exists (/etc/helmsight/id_ed25519); continue with step 2.

On the machine that runs helmsight (the commands assume the helmsight system user from Installation exists):

sudo install -d -m 0750 -o root -g helmsight /etc/helmsight
sudo ssh-keygen -t ed25519 -f /etc/helmsight/id_ed25519 -N "" -C helmsight
sudo chgrp helmsight /etc/helmsight/id_ed25519
sudo chmod 0640 /etc/helmsight/id_ed25519

Use an ed25519 key dedicated to helmsight. RSA keys work too, but the RSA library in use has a known timing side channel, so prefer ed25519 (details in the security model). Passphrase-protected keys must be loaded into ssh-agent (ssh.use_agent = true).

2. Create the monitoring account on each host

With helmsight hosts bootstrap

helmsight prints a script that does everything below on a host: it creates the account, installs the public key with restrict,from=, unlocks key logins on Alpine and adds the account to the journal group. Running it again is safe.

helmsight hosts bootstrap --from 10.0.0.5 | ssh root@web-1 sh

--from is the helmsight server’s address as the host sees it (several addresses or networks can be separated by commas, e.g. 10.0.0.5,10.1.0.0/16). Without it, the key is accepted from anywhere and a warning is printed. To check the script before running it, save it to a file:

helmsight hosts bootstrap --from 10.0.0.5 > monitor-account.sh

By hand

On every host to monitor, as root:

useradd --create-home --shell /bin/sh monitor
install -d -m 700 -o monitor -g monitor ~monitor/.ssh
echo 'restrict,from="10.0.0.5" ssh-ed25519 AAAA… helmsight' > ~monitor/.ssh/authorized_keys
chown monitor:monitor ~monitor/.ssh/authorized_keys
chmod 600 ~monitor/.ssh/authorized_keys

• Replace ssh-ed25519 AAAA… helmsight with the contents of /etc/helmsight/id_ed25519.pub.
• Replace 10.0.0.5 with the address of the helmsight server as the monitored host sees it; behind NAT, that is the translated address. Thanks to from=, a stolen key cannot be used from another machine.
• restrict disables port, agent and X11 forwarding and terminal (PTY) allocation. helmsight needs none of them.
• The account’s shell must be a working POSIX shell (/bin/sh); nothing can be collected with /usr/sbin/nologin.

On Alpine, create the account with adduser -D -s /bin/sh monitor. Alpine creates accounts locked (!), and OpenSSH refuses key logins to locked accounts. Mark the password as unusable without locking the account:

sed -i 's/^monitor:!/monitor:*/' /etc/shadow

Do not use passwd -u: on BusyBox it can leave the account with an empty password.

For many servers, roll these steps out with the tools you already use, such as Ansible, Salt or cloud-init. An example Ansible task:

- name: helmsight monitoring account
  ansible.builtin.user:
    name: monitor
    shell: /bin/sh
    create_home: true

- name: helmsight key
  ansible.posix.authorized_key:
    user: monitor
    key: "{{ lookup('file', 'files/helmsight.pub') }}"
    key_options: 'restrict,from="10.0.0.5"'
    exclusive: true

Optional group memberships

Some data needs extra permissions. Without them, helmsight shows “n/a” with the reason in the relevant view; nothing else is affected.

usermod -aG systemd-journal monitor

Nothing else is installed on the host.

3. Describe your hosts

In helmsight.toml:

[ssh]
user = "monitor"
identity_files = ["/etc/helmsight/id_ed25519"]

[[hosts]]
name = "web-1"
address = "10.0.0.11"
groups = ["web"]
tags = ["env:prod", "role:web"]

[[hosts]]
name = "web-2"
address = "10.0.0.12"
groups = ["web"]
tags = ["env:prod", "role:web"]
baseline = "web-1"    # compare drift and new ports against web-1

[[hosts]]
name = "db-1"
address = "db-1.internal"
port = 2222
user = "observer"     # a different account for this host
groups = ["db"]

Choosing groups and tags. Groups split the fleet coarsely (web, db, edge) and are used as scopes in filters, alert rules, actions, notification channels and wall display tokens. Tags are free-form and usually written as key:value (env:prod, dc:fra-1, team:payments). A host can have several groups and tags.

The baseline catches differences between servers that do the same job. With baseline = "web-1" on web-2, the Compare view shows the package, port, unit, kernel and OS differences between the two, and the security page lists ports that listen on web-2 but not on web-1.

Keeping the host list separate

Hosts can also live in a separate file (relative to the configuration file):

hosts_file = "hosts.toml"

hosts.toml contains only [[hosts]] entries, which makes it easy to generate the host list from an inventory tool.

Importing from ~/.ssh/config

[ssh_config_import]
path = "~/.ssh/config"
hosts = ["web-*", "db-*"]
tags = ["source:ssh-config"]

HostName, User, Port and IdentityFile are honoured. An explicit [[hosts]] entry with the same name takes precedence. Hosts with ProxyJump are skipped with a warning.

Validate the configuration with helmsight config check. Every error names the file, line and key:

error: helmsight.toml:23:1: `hosts[1].name`: duplicate host name `web-1`

4. Trust the host keys

helmsight refuses unknown host keys. Confirm them once:

helmsight hosts test --trust

HOST        STATUS             DETAIL
web-1       host key unknown   ssh-ed25519 SHA256:ldBw6qzC… ([10.0.0.11]:22)
  Trust this key? Compare with `ssh-keygen -lf` on the host (y/N) [N]: y
  trusted; retrying
web-1       ok                 Debian GNU/Linux 12 (bookworm); user monitor (72 ms)

Before confirming, compare the fingerprint on the host itself:

ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub

Keys can also be approved in the UI under Host keys, where the administrator pastes the fingerprint printed on the host; approval only succeeds if it matches the key presented. A key that changes later stops collection and raises a critical alert until an administrator reviews it.

Where there are many hosts and the network is trusted, ssh.accept_new_host_keys = true trusts keys of new hosts automatically (like OpenSSH’s accept-new). Changed keys are still always refused.

5. Start the server

helmsight serve

Hosts show as Pending for a few seconds, until their first collection. If a host stays Down, Auth failed or Key not trusted, see Troubleshooting.

What helmsight reads

Each tick runs one POSIX shell script over the existing SSH session: /proc files, df, ps, ss, systemctl, the package database and similar read-only sources. The full list and the guarantees are in the security model.

The web interface

The UI follows the operating system’s light or dark preference; switch it in the top bar. Every view shows when its data was last updated, and stale data is dimmed and labelled. The UI is live: host states and alerts update through server-sent events without reloading the page.

What a user sees depends on their role:

Fleet

The fleet overview lists every host with its status, CPU and memory sparklines, load, the fullest filesystem, network throughput, uptime and firing alerts. Use it as a table or as a grid of cards.

• Status chips filter by state. Needs attention collects everything that is not OK, partially collected or stale.
• Search (/) matches names, addresses, operating systems, groups and tags; group:web and tag:env:prod match exactly.
• The statuses tell different problems apart:

This distinction is deliberate: “down” is a network or hardware problem, “auth failed” is an account or key problem, and “key changed” is a possible security incident. Each needs a different response, often from a different team. See Troubleshooting for fixes.

Host detail

Click a host in the fleet list to open it. The tabs:

• Overview: headline values, then charts for CPU (user, system, iowait, steal), a per-core heatmap, memory and swap, load, disk usage per mount, disk I/O per device, network per interface and TCP connections. Pick a range from 15 minutes to 30 days or a custom one; crosshairs are synchronized across charts. Every chart has a table view.
• Processes: top processes by CPU and by memory.
• Network: listening ports with owning processes, interface rates and TCP states.
• Services: failed units first, then running services.
• Containers: Docker or Podman containers with state and usage.
• System: OS, kernel, hardware, logged-in users and collection details.
• Changes: what changed in packages, ports, units, kernel and OS.
• Security: failed SSH logins, pending updates and ports not present on the baseline.

Reading the charts

• In the CPU chart, iowait is time spent waiting for disks and steal is time the hypervisor gave to other guests. High steal usually means the physical host under the virtual machine is overloaded.
• Memory “used” excludes reclaimable cache, so Linux using free memory for caching does not show up as “memory full”.
• Load averages should be read against the number of cores; the load1_per_core metric does that for alert rules.
• Periods when a host was unreachable show as gaps; no line is drawn across them.
• Long ranges are drawn from the finest resolution that covers them (raw samples, 1-minute or 5-minute averages). On a 30-day chart, spikes of a few seconds are therefore smoothed out; pick a narrower range to examine short events.

Compare

Compare a host with another host, or with its configured baseline. Only differences are shown: packages (and their versions), listening ports, enabled units, kernel and OS release. It is the quickest way to catch servers with the same role drifting apart over time (configuration drift).

Security

The fleet-wide security page brings together a chart of failed SSH logins across all hosts over time and the count per host for the last 24 hours, pending and security updates, reboot-required flags, ports not on each host’s baseline and TLS certificates checked from the helmsight server.

Changes

A timeline of inventory changes across the fleet, filterable by kind and host. This is where “what changed last night?” gets answered: installed or upgraded packages, opened or closed ports, enabled services, a new kernel.

Administration pages

Administrators also see:

• Users: local users, roles and TOTP status.
• Host keys: pending and changed host keys.
• Wall display: wall display tokens.
• Audit log: the tamper-evident record of actions and administrative changes.

Every user manages their own password and TOTP enrolment on the Account page in the user menu at the top right. When actions are configured, operators and administrators also see the Actions page in the navigation.

Keyboard shortcuts

Alerts and notifications

Rules

Rules live in the configuration file:

[[alerts.rules]]
id = "disk-full"
expr = "disk_used_pct > 90 for 10m"
severity = "critical"
summary = "Filesystem almost full"
tags = ["env:prod"]          # optional scope: hosts, groups, tags

expr is <metric> <operator> <threshold> [for <duration>]. A rule fires when the condition holds for the whole duration and resolves when it no longer holds. Per-instance metrics such as disk_used_pct fire once per mount point. The configuration reference lists every metric; the UI shows them under Alerts → Rules.

Writing rules

• Operators: >, >=, <, <=, ==, !=.
• Threshold units: K, M, G, T (powers of 1000) and Ki, Mi, Gi, Ti (powers of 1024). For example, net_rx_bytes > 100M means 100 MB per second.
• Duration (for) avoids alerts on short spikes. Without a duration, the rule fires as soon as the condition holds.
• Scope: hosts that match any entry in hosts, groups or tags are included. Without any of them, the rule applies to every host.
• Severity: info, warning (default) or critical.

While a host is unreachable, its metric alerts keep their state instead of resolving, so a “disk full” alert does not close and reopen during an outage.

Example rules

# CPU above 90% for 15 minutes
[[alerts.rules]]
id = "cpu-high"
expr = "cpu_pct > 90 for 15m"

# Load above 2 per core
[[alerts.rules]]
id = "load-high"
expr = "load1_per_core > 2 for 10m"

# Memory pressure
[[alerts.rules]]
id = "mem-high"
expr = "mem_used_pct > 95 for 5m"
severity = "critical"

# Inode exhaustion (blocks file creation regardless of free space)
[[alerts.rules]]
id = "inodes"
expr = "disk_inodes_used_pct > 90 for 10m"

# CPU steal on virtual machines (overloaded hypervisor)
[[alerts.rules]]
id = "steal"
expr = "cpu_steal_pct > 10 for 15m"

# Production hosts with pending security updates
[[alerts.rules]]
id = "security-updates"
expr = "pending_security_updates > 0"
severity = "info"
tags = ["env:prod"]

# Hosts that need a reboot
[[alerts.rules]]
id = "reboot"
expr = "reboot_required == 1 for 1d"
severity = "info"

# Brute-force SSH attempts
[[alerts.rules]]
id = "ssh-attempts"
expr = "ssh_failed_logins_1h > 50"

# Clock skew (breaks TLS, Kerberos and log ordering)
[[alerts.rules]]
id = "clock-skew"
expr = "clock_skew_secs > 30"

# Stopped containers
[[alerts.rules]]
id = "containers"
expr = "containers_not_running > 0 for 5m"
groups = ["app"]

Built-in rules

Built-in rules need no configuration:

Working with alerts

• Acknowledge (operator) an alert to show that someone is on it; reminders stop until it resolves.
• Silence (operator) a rule, a host or both for a period, with a reason. Silenced alerts are still recorded but not notified. Useful for planned maintenance.
• Resolved alerts stay visible for seven days in the UI and for retention.events in the database.

Notifications

[[notify]]
id = "ops"
type = "slack"                 # webhook, slack or email
url = "secret:slack-webhook"   # stored encrypted: helmsight secret set slack-webhook
min_severity = "warning"

Notifications go out when an alert fires, when it resolves, and as reminders every alerts.repeat_interval (4 hours by default) while it is unacknowledged. Deliveries are retried and recorded with each alert. Admins can send a test notification with POST /api/v1/notify/<id>/test.

Channel examples

Slack (or tools that accept Slack-compatible webhooks, such as Mattermost and Rocket.Chat):

helmsight secret set slack-webhook     # paste the webhook URL

[[notify]]
id = "slack-ops"
type = "slack"
url = "secret:slack-webhook"
min_severity = "warning"

Email:

[[notify]]
id = "mail-oncall"
type = "email"
smtp_host = "smtp.example.com"
smtp_port = 587
smtp_security = "starttls"
smtp_username = "helmsight@example.com"
smtp_password = "secret:smtp"
from = "helmsight@example.com"
to = ["oncall@example.com"]
min_severity = "critical"

Generic webhook (your own automation, incident management tools):

[[notify]]
id = "hook"
type = "webhook"
url = "https://hooks.example.com/helmsight"
headers = { Authorization = "secret:hook-token" }
groups = ["db"]                # only alerts of the db group

The JSON webhook payload is documented in the configuration reference.

When a channel has hosts, groups or tags, only alerts of those hosts are sent to it. Alerts that do not belong to a host, such as certificate alerts, only go to unscoped channels.

Actions

Actions let operators perform a small set of pre-approved tasks, such as restarting nginx, without shell access. They are disabled until you configure them, and the actions UI is hidden entirely when none exist.

Actions are the only feature of helmsight that can change monitored hosts, so read this whole page before enabling them.

[[actions]]
id = "restart-nginx"
label = "Restart nginx"
description = "Restarts the web server. Takes about two seconds."
command = "sudo -n /usr/bin/systemctl restart nginx.service"
groups = ["web"]       # required: hosts, groups and/or tags
role = "operator"      # operator or admin
timeout = "60s"

• The command is fixed. The UI can only choose an action and a host; it cannot pass arguments, and commands have no templates or variables.
• Every action must name its target hosts, groups or tags explicitly; there is no “all hosts” default.
• Before running, a dialog shows the exact command and host and asks for confirmation.
• The output (up to 64 KiB) is shown in the UI; its first 4 KiB are also written to the audit log.
• Privileged commands must start with sudo -n, with a sudoers rule for exactly that command on each host:

Cmnd_Alias HELMSIGHT_ACTIONS = /usr/bin/systemctl restart nginx.service
monitor ALL=(root) NOPASSWD: HELMSIGHT_ACTIONS

Add the rule with visudo -f /etc/sudoers.d/helmsight; visudo refuses to save a file with syntax errors. Example file: examples/sudoers.

The -n option stops sudo from asking for a password: without a matching rule, the command fails immediately instead of waiting.

Designing safe actions

• Never allow a shell (/bin/sh, bash), an editor (vi, nano), a pager (less), wildcards (*) or ALL in sudoers. Each of them means full root access.
• Write commands with full paths and fixed arguments, such as systemctl restart nginx.service. A rule like systemctl restart * allows restarting every service.
• Turn reversible tasks with limited impact into actions: restarts, cache flushes, status queries. Use your existing change management for tasks that delete data or change configuration.
• Use role = "admin" for risky actions.
• Give the monitoring account sudo rights only on hosts where actions are defined. Hosts that are only monitored need no sudoers rule.

An example action that needs no privileges:

[[actions]]
id = "disk-usage"
label = "Show disk usage"
description = "Shows how full the filesystems are."
command = "df -h"
groups = ["web", "db"]

Audit log

Every run is written to the audit log when it starts and when it finishes, with the user, host, command, exit status and the first 4 KiB of output. Administrators read it under Audit log. User, host key, silence, display token and secret changes are recorded too.

The log is append-only: the database rejects updates and deletes, and every entry carries a SHA-256 hash chained to the previous one. Verify integrity in the UI, or helmsight audit verify, detects tampering.

The chain makes it noticeable when someone with direct access to the database file silently changes or deletes a past entry; it cannot stop someone with root on the central server from rewriting the whole log. If your audit requirements are strict, export the log regularly (for example through the API).

Wall display

The wall display is a full-screen, read-only view for TV and NOC screens: large status tiles grouped by host group, firing alerts and a clock, without navigation. When the fleet does not fit, it rotates through the groups every 20 seconds. Status is shown with text and icons, not only colour, so it stays readable from a distance and for people with colour vision deficiencies.

Open /display while signed in, or use a display token so the screen does not need an account:

1. As an administrator, open Wall display and create a token, choosing the groups the screen may see (or all hosts).
2. Copy the address, which is shown only once, for example https://monitor.example.com/display#token=hsd_…, and open it on the screen.

The token is in the URL fragment (the part after #), so it is never sent to web servers or written to their logs. It grants read access to the chosen groups only, and can be revoked at any time; the screen then shows that the token is no longer valid.

Tokens do not expire on their own: the same address keeps working after the screen’s browser restarts. If a screen is stolen or changes hands, revoke only that screen’s token; other screens are not affected. Creating one token per screen is therefore a good habit.

Options

Append options to the fragment with &:

Example: https://monitor.example.com/display#token=hsd_…&rotate=30&theme=dark

Tips for setting up a screen

• Open the browser in kiosk mode, for example chromium --kiosk --noerrdialogs "https://monitor.example.com/display#token=…".
• Turn off the screen’s sleep mode and screen saver.
• With a self-signed certificate, make the screen’s browser trust it; otherwise the warning page appears after every restart.
• When the connection drops, the screen shows that its data is stale and recovers on its own when the connection returns.

Configuration reference

helmsight reads one TOML file. The first of these is used:

1. --config <path>
2. $HELMSIGHT_CONFIG
3. /etc/helmsight/helmsight.toml
4. ./helmsight.toml

Türkçe sürüm

examples/helmsight.toml is a commented example of every option. Validate a file with helmsight config check; errors show the file, line, column and key:

error: helmsight.toml:23:1: `hosts[1].name`: duplicate host name `web-1`
error: helmsight.toml:41:8: `alerts.rules[0].expr`: unknown operator `=>` (use > >= < <= == !=)

Unknown keys are errors, so typos never pass silently.

Durations are strings such as "500ms", "30s", "5m", "1h30m", "7d", "2w" (ms milliseconds, s seconds, m minutes, h hours, d days, w weeks).

Secrets may be literal strings, but references are preferred:

env: works well with systemd’s EnvironmentFile= or container environment variables, and file: with Docker or Kubernetes secret files (/run/secrets/...).

Changes to the configuration take effect after a restart.

A complete example

# Top-level keys must come before any [section] header.
hosts_file = "hosts.toml"

[server]
listen = "127.0.0.1:8080"
data_dir = "/var/lib/helmsight"
public_url = "https://monitor.example.com"
trusted_proxies = ["127.0.0.1"]

[auth]
require_totp_for_admins = true

[ssh]
user = "monitor"
identity_files = ["/etc/helmsight/id_ed25519"]

[[alerts.rules]]
id = "disk-full"
expr = "disk_used_pct > 90 for 10m"
severity = "critical"

[[notify]]
id = "ops"
type = "slack"
url = "secret:slack-webhook"

[[certs]]
endpoint = "www.example.com:443"

In TOML, every key written after a [section] header belongs to that section. Write top-level keys such as hosts_file at the very top of the file; in the wrong place, config check reports them as unknown keys.

[server]

trusted_proxies matters for identifying clients correctly: sign-in rate limiting works per client address. Behind a proxy without this setting, every user appears to come from the proxy’s address.

[server.tls]

With a Let’s Encrypt certificate, point cert at fullchain.pem and key at privkey.pem, and restart helmsight after renewal (for example with certbot’s --deploy-hook).

[auth]

[auth.oidc]

Register <public_url>/api/v1/auth/oidc/callback as the redirect URI. Roles are updated from the claim at every sign-in, so changing a user’s groups at the identity provider changes their role at their next sign-in.

Example (Keycloak):

[auth.oidc]
issuer = "https://sso.example.com/realms/ops"
client_id = "helmsight"
client_secret = "secret:oidc"
role_claim = "realm_access.roles"
role_map = { "helmsight-admin" = "admin", "sre" = "operator", "dev" = "viewer" }
label = "Sign in with company account"

Keeping a local administrator account for emergencies is recommended; use disable_local_login with care so that you do not lose access to the UI when SSO is down.

[ssh]

[collect]

Failed hosts are retried with backoff: exponential up to 5 minutes when unreachable, 1 to 5 minutes after authentication failures (to avoid triggering intrusion prevention) and every minute for host key problems.

Lowering interval increases chart resolution, but also increases the load on the hosts and the database proportionally. For large fleets, 10 to 15 seconds is a reasonable starting point.

[retention]

Charts pick the finest resolution that covers the requested range. Longer retention grows the database proportionally. Expired data is deleted automatically every hour; rollups are computed every minute.

Hosts

[[hosts]]

hosts_file

A top-level hosts_file = "hosts.toml" loads additional [[hosts]] from another file (relative to the configuration file).

[ssh_config_import]

HostName, User, Port and IdentityFile are honoured, with OpenSSH “first value wins” semantics across matching Host blocks. Explicit [[hosts]] with the same name take precedence. Hosts with ProxyJump are skipped with a warning; Match and Include are not evaluated.

Alerts

[alerts]

Built-in rules: host_unreachable (critical; also for authentication failures), host_key_changed (critical), host_key_unknown (warning), failed_units (warning, one alert per unit), cert_expiry (warning or critical by days left; warning when the check fails).

[[alerts.rules]]

Operators: >, >=, <, <=, ==, !=. Thresholds accept the suffixes K, M, G, T (powers of 1000) and Ki, Mi, Gi, Ti (powers of 1024). Per-instance metrics produce one alert per instance (for example per mount point). While a host is unreachable, its metric alerts keep their state instead of resolving.

Metrics available to rules

[[notify]]

Notifications are sent when an alert fires, when it resolves (if the firing notification was sent) and as reminders. Silenced alerts are recorded but not notified. Failed deliveries are retried three times and listed with the alert. Send a test from the API: POST /api/v1/notify/<id>/test.

Webhook payload:

{
  "version": 1,
  "source": "helmsight",
  "status": "firing",
  "alert": {
    "id": 42,
    "rule": "disk-full",
    "host": "web-1",
    "instance": "/var",
    "severity": "critical",
    "summary": "disk_used_pct on /var is 93.1% (> 90.0%)",
    "value": 93.1,
    "started_at": "2026-09-30T16:56:12Z",
    "resolved_at": null,
    "acknowledged_by": null
  },
  "url": "https://monitor.example.com/alerts?id=42"
}

status is firing, resolved, reminder or test.

[[certs]]

The certificate is read even when it does not validate; trust problems are shown separately from connection failures. Checks run from the helmsight server, so only endpoints that helmsight can reach can be checked.

[[actions]]

[prometheus]

scrape_configs:
  - job_name: helmsight
    scheme: https
    authorization:
      credentials_file: /etc/prometheus/helmsight.token
    static_configs:
      - targets: ["monitor.example.com"]

Exposed metrics are prefixed with helmsight_: host_up, host_alerts_firing, cpu_percent{mode}, memory_*_bytes, load1/5/15, filesystem_{size,used,avail}_bytes{mount}, disk_{read,written}_bytes_per_second{device}, network_{receive,transmit}_bytes_per_second{interface}, tcp_connections{state}, pending_updates, pending_security_updates, reboot_required and more, all labelled with host.

Command line

Global options: --config <path>, --data-dir <dir> (overrides server.data_dir; without a configuration file, commands use the local-mode default ~/.local/share/helmsight, or /var/lib/helmsight for root), --log-format text|json.

When no user exists, serve prints a one-time setup link (/setup#token=...) for creating the first administrator in the browser.

Environment variables

HTTP API

The web UI uses the same versioned JSON API under /api/v1. The OpenAPI document is served at /api/v1/openapi.json. Authenticate with the session cookie from POST /api/v1/auth/login; state-changing requests need the X-CSRF-Token header returned by GET /api/v1/auth/me. Live updates are available as server-sent events at /api/v1/events. Wall displays call /api/v1/display/state with Authorization: Bearer <display token>. /healthz returns ok without authentication.

Command line

helmsight [--config <path>] [--data-dir <dir>] [--log-format text|json] <command>

Run helmsight <command> --help for detailed help on any command.

init options

init asks for every value that is not given as an option. With --non-interactive it asks nothing and uses the defaults for the rest; the first administrator is then only created with --password-stdin.

The global --data-dir sets the data directory written to the configuration. In interactive mode, when the key file does not exist, init offers to create it.

printf '%s\n' "$PASSWORD" | helmsight --config /etc/helmsight/helmsight.toml \
  --data-dir /var/lib/helmsight init --non-interactive \
  --ssh-key /etc/helmsight/id_ed25519 --generate-key --admin alice --password-stdin

Finding the configuration file

The configuration file is found via --config, $HELMSIGHT_CONFIG, /etc/helmsight/helmsight.toml or ./helmsight.toml. Without a configuration file (local mode), --data-dir or the local-mode default directory (~/.local/share/helmsight, or /var/lib/helmsight for root) is used.

Common examples

# Create a user with the password supplied by a script (e.g. automation)
printf '%s\n' "$PASSWORD" | helmsight user add alice --role operator --password-stdin

# A user who lost their TOTP device
helmsight user reset-password alice --reset-totp

# Test only two hosts
helmsight hosts test web-1 db-1

# Prepare a new monitored host
helmsight hosts bootstrap --from 10.0.0.5 | ssh root@web-3 sh

# Store the Slack webhook URL encrypted
helmsight secret set slack-webhook

# Read a secret from a file
helmsight secret set smtp --stdin < /run/secrets/smtp

# Run temporarily on another address
helmsight serve --listen 127.0.0.1:9090

--password-stdin and --stdin keep passwords out of the shell history and the process list; on purpose, there is no way to pass a password as a command-line argument.

Exit status

Commands exit with 0 on success and a non-zero status on errors. hosts test fails when at least one host cannot be reached, config check when the configuration has errors, and audit verify when the chain is broken, so these commands can be used as checks in scripts and CI pipelines.

Logging

Logging is controlled with HELMSIGHT_LOG, for example HELMSIGHT_LOG=debug or HELMSIGHT_LOG=info,server=debug. --log-format json writes logs as JSON lines, suitable for a log collection system. Logs never contain secrets, passwords, session tokens or collected output.

API and integrations

JSON API

The web UI is built on the same versioned API under /api/v1, so everything visible in the UI is also available through the API. The OpenAPI document, generated from the code, is served at /api/v1/openapi.json and works with Swagger UI, Postman or a client generator.

Authentication uses the session cookie set by POST /api/v1/auth/login. State-changing requests must send the X-CSRF-Token header returned by GET /api/v1/auth/me:

curl -c cookies -H 'Content-Type: application/json' \
  -d '{"username":"alice","password":"…"}' https://monitor.example.com/api/v1/auth/login
curl -b cookies https://monitor.example.com/api/v1/hosts

Create a separate viewer user (or operator if needed) for automation; do not use personal accounts in scripts.

Frequently used endpoints:

See the OpenAPI document for the full list and the schemas.

Example: list critical alerts

curl -sb cookies https://monitor.example.com/api/v1/alerts \
  | jq '.[] | select(.severity == "critical" and .state == "firing") | {host, rule_id, summary}'

See the alerts schema in the OpenAPI document for the full response format.

Server-sent events

GET /api/v1/events streams host events (a host summary whenever it changes) and alerts, changes and hostkeys events that signal clients to refetch. A resync event means the client fell behind and should reload.

curl -Nb cookies https://monitor.example.com/api/v1/events

Behind a reverse proxy, turn off buffering for this path (proxy_buffering off in nginx); otherwise events arrive late.

Prometheus

[prometheus]
enabled = true
token = "secret:prometheus"

/metrics then exposes the collected host metrics in the Prometheus text format to requests with Authorization: Bearer <token>. Metric names are listed in the configuration reference.

This makes helmsight a data source for an existing Prometheus and Grafana setup: basic system metrics reach Prometheus without installing node_exporter on the hosts, and long-term storage and dashboards can live there.

Webhooks

Webhook channels receive a JSON document for every notification; see the configuration reference for the payload.

Security

This document describes what helmsight does on monitored hosts, what it cannot do, how to run it with least privilege, and how to report a vulnerability.

Türkçe sürüm

Threat model

helmsight has three parties:

1. The central server runs the helmsight binary. It holds SSH private keys, the database (metrics, inventory, users, sessions, audit log) and a key file that encrypts stored secrets.
2. Monitored hosts are reached over SSH. Their output is untrusted: a compromised host, or an unprivileged local user who controls a process name, a log line or a file name, may try to feed hostile data back.
3. Users reach the web UI over HTTP(S) with a role: viewer, operator or admin. Wall displays use revocable read-only tokens.

helmsight is designed so that:

• A compromised monitored host cannot attack the browser or the server. Output is size-limited (16 MiB per execution) and parsed by Rust code that never panics on malformed input. This is checked by fuzz-style tests with truncated and corrupted real output. Values are rendered as text, never as HTML. The browser only executes code served by the binary, and the Content-Security-Policy forbids inline scripts, inline styles, framing and third-party origins. Section markers in the script output include a random per-run nonce that is never visible in the remote process list, so a process or log line cannot forge a section.
• A compromised monitored host cannot reach other hosts. Each host has its own SSH session; nothing is forwarded (no agent forwarding, no port forwarding).
• A UI user cannot run arbitrary commands. There is no terminal, file manager or command input. Actions are fixed command strings from the configuration file; the API accepts only an action id and a host name, and checks that the user’s role and the host are permitted.
• A network attacker cannot impersonate a host. SSH host keys are verified with OpenSSH known_hosts semantics. Unknown keys are refused until an admin confirms the fingerprint (or accept_new_host_keys is enabled). A changed key always stops collection, raises a critical alert and requires an admin to verify the new fingerprint.

Out of scope: an attacker with root on the central server can read the SSH keys and the database. Protect that machine accordingly.

What this means in practice

• If helmsight’s monitoring key is stolen, the attacker can only log in as the unprivileged monitor account and (with from=) only from the helmsight server’s address. Do not skip the restrict,from= options.
• If a monitored host is compromised, the worst outcome is false metrics for that host; the attacker cannot use it to move on to helmsight or to other hosts.
• The central server has read access to the whole fleet; protect it as carefully as a bastion host: keep it updated, open only the ports it needs and restrict access to the UI.

What helmsight runs on monitored hosts

Every tick, helmsight opens a channel on the host’s persistent SSH session and runs sh -s, sending the collection script on standard input. The script is in crates/collect/src/remote.sh; review it before deploying. It contains only reads:

Properties of the script:

• POSIX sh only; it runs under dash, bash and BusyBox ash.
• Every output redirection targets /dev/null (enforced by a unit test).
• Missing or forbidden tools produce “n/a” with a reason, never an error.
• Long-running commands are bounded with timeout where available.
• The script is never saved on the remote host; it is read from standard input and appears in the process list only as sh -s.

What gets written on the host

helmsight’s commands write nothing. The integration test suite logs in as an unprivileged user on Debian, Ubuntu, Rocky, Fedora, openSUSE and Alpine, runs every collection group and asserts that no file outside /proc, /sys, /dev, /run and /var/log changed.

What still happens as a side effect of logging in over SSH, as with any SSH login:

• The host’s login bookkeeping records the session (wtmp, lastlog, the journal or syslog).
• On Ubuntu, pam_motd creates ~/.cache/motd.legal-displayed in the monitoring account’s home directory on its first login.

Exceptions you control:

• dnf/yum: dnf and yum always write their own log files and, for unprivileged users, a cache under /var/tmp. Pending updates on RHEL-family hosts are therefore only listed when collect.dnf_updates = true.
• apt as root: if you monitor with the root account (not recommended), apt-get -s may refresh its binary package cache in /var/cache/apt.
• Actions run exactly the commands an administrator configured, which may change the host. They are disabled unless configured.

Creating the monitoring account

helmsight hosts bootstrap --from 10.0.0.5 prints a script that does the following on a host (and unlocks key logins on Alpine); pipe it to ssh root@<host> sh or review it first. To do it by hand, on every monitored host:

# Unprivileged account with a home directory for authorized_keys.
useradd --create-home --shell /bin/sh monitor
install -d -m 700 -o monitor -g monitor ~monitor/.ssh

# Restrict the key: no PTY, no forwarding, only from the helmsight server.
echo 'restrict,from="10.0.0.5" ssh-ed25519 AAAA... helmsight' \
  > ~monitor/.ssh/authorized_keys
chown monitor:monitor ~monitor/.ssh/authorized_keys
chmod 600 ~monitor/.ssh/authorized_keys

restrict disables port, agent and X11 forwarding and PTY allocation. helmsight needs none of them. Replace 10.0.0.5 with the address of the helmsight server.

Do not give the account a password; it should only log in with the key. Setting PasswordAuthentication no in sshd_config on your hosts is also recommended.

Optional group memberships, each adding visibility:

Without these, helmsight shows “n/a” with the reason in the relevant view.

The SSH key used by helmsight should be dedicated to it and of type ed25519 (ssh-keygen -t ed25519 -f /etc/helmsight/id_ed25519 -N ""). RSA keys work, but the RSA implementation used has a known timing side channel in private-key operations (RUSTSEC-2023-0071), so prefer ed25519. Keys can be loaded from files (ssh.identity_files, readable only by the helmsight user) or from ssh-agent. Passphrase-protected keys must be loaded into ssh-agent.

Trusting host keys

New hosts are refused until their key is trusted. Either:

• run helmsight hosts test --trust, which shows each fingerprint and asks for confirmation; or
• approve the key in the UI under Host keys. The administrator must paste the fingerprint printed by ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub on the host, and approval only succeeds if it matches the key presented.

Approved keys are stored in <data_dir>/known_hosts. Existing OpenSSH files can be consulted read-only with ssh.known_hosts_files. If you already distribute your hosts’ keys with a configuration management tool, pointing helmsight at that file is the safest option.

Actions

Actions are the only feature that changes monitored hosts, and they are off unless configured.

• Commands are written by an administrator in the configuration file. The UI cannot pass arguments and there is no templating.
• Every action must list its target hosts, groups or tags explicitly.
• The UI shows the exact command and host in a confirmation dialog.
• Privileged commands must use sudo -n. Grant exactly those commands in sudoers (examples/sudoers); never grant shells, editors, wildcards or ALL.
• Every run is written to the audit log twice: before it starts and with its exit status and truncated output.
• The audit log is append-only. Database triggers reject updates and deletes, and each entry carries a SHA-256 hash chained to the previous one; helmsight audit verify (or Verify integrity in the UI) detects tampering.

The central server

• Run helmsight as a dedicated system user with the hardened systemd unit in examples/helmsight.service.
• The data directory is created with mode 0700; the database, key file and self-signed TLS key with mode 0600. helmsight refuses to use a key file readable by other users.
• Secrets in the configuration can be references (secret:<name>, env:<VAR>, file:<path>) instead of plain text. Stored secrets and TOTP seeds are encrypted with XChaCha20-Poly1305 using <data_dir>/secret.key, created on first run. Back the key file up separately from the database.
• helmsight never logs secrets, passwords, session tokens or command output of collections.

Web application security

• Binds to 127.0.0.1 by default. On other addresses, TLS is enabled automatically, with provided certificates or a self-signed one whose SHA-256 fingerprint is logged at start-up. Plain HTTP on a public address requires server.tls.allow_insecure_http = true, meant for deployments behind a TLS-terminating reverse proxy (set server.public_url to its https:// URL and server.trusted_proxies to its address).
• Passwords are hashed with Argon2id (19 MiB, 2 iterations); unknown users take the same time as wrong passwords, so user names cannot be guessed from response times. The minimum length is 12 characters.
• Optional TOTP (RFC 6238); codes cannot be reused. Admins can be required to enrol (auth.require_totp_for_admins).
• OpenID Connect uses the authorization code flow with PKCE, state and nonce. Roles come from a configurable claim; users without a mapped role are denied unless default_role is set.
• Session cookies are random 256-bit tokens, stored hashed, HttpOnly, SameSite=Strict, Secure with TLS and __Host- prefixed. Sessions expire after auth.session_ttl and after auth.idle_timeout of inactivity.
• State-changing requests need the per-session X-CSRF-Token header and are rejected when the Origin header names another site.
• Sign-in is rate limited per user name and per client address.
• Responses carry Content-Security-Policy (no inline code, no framing), X-Content-Type-Options, Referrer-Policy: no-referrer, Cross-Origin-Opener-Policy, Permissions-Policy and HSTS when TLS is used. API responses are not cached.
• Wall display tokens are stored hashed, grant read access to a fixed set of groups only, are passed in the URL fragment (never sent to servers or logs) and can be revoked at any time.
• The Prometheus endpoint is disabled unless a bearer token is configured.

Security checklist before deployment

• helmsight runs as a dedicated system user, with the systemd unit or in a container.
• The monitoring key is an ed25519 key used only by helmsight, and its file is not readable by others.
• The authorized_keys line has restrict,from="…".
• The monitoring account is not in the docker group (or was added knowing that this is equivalent to root).
• Host keys were approved after comparing fingerprints; accept_new_host_keys is only enabled on a trusted network.
• The UI is on loopback behind a TLS proxy, or served directly with TLS; public_url is set.
• TOTP is required for administrators (auth.require_totp_for_admins = true), or sign-in uses SSO.
• Secrets are written as secret:, env: or file: references; the configuration file contains no plain-text passwords.
• secret.key is backed up separately from the database.
• If actions are used, sudoers only allows the exact commands.

Reporting a vulnerability

Please report security problems privately via GitHub’s Report a vulnerability button on the repository’s Security tab. Do not open a public issue. Include the version (helmsight --version), a description, and steps to reproduce. Reports can be written in English or Turkish.

We aim to acknowledge reports within three working days and to publish a fix and advisory as soon as possible. We credit reporters unless they prefer otherwise.

FAQ

Why not an agent? Agents must be installed, updated, configured and trusted on every host, and they widen the attack surface. SSH is already there, already hardened and already audited. helmsight adds one read-only login session.

How much load does it put on a host? One short shell script every 5 seconds that reads files under /proc and runs a few lightweight tools: a few milliseconds of CPU. Expensive checks such as pending updates run every 6 hours. The intervals can be changed in the [collect] section.

How many resources does the central server need? helmsight is a single process; it keeps one persistent SSH session per host and compact rows of metrics. A few hundred hosts on a small VM are expected to be fine.

Does it need root? No. Use a dedicated unprivileged account. A few details need group memberships (journal access for failed logins, the Docker socket for containers); helmsight shows “n/a” with the reason for anything it cannot read.

Does it really write nothing on the hosts? helmsight’s commands write nothing; an integration test checks this on six distributions. SSH logins are recorded by the host as usual (wtmp, lastlog, the journal, and on Ubuntu a one-time pam_motd marker in the account’s home directory). dnf and yum always write log files, so pending updates on RHEL-family hosts are only listed with collect.dnf_updates = true.

Which distributions are supported? Debian, Ubuntu, RHEL and its rebuilds (Rocky, Alma), Fedora, openSUSE and Alpine. Any Linux with an SSH server and a POSIX shell should work; missing tools degrade to “n/a”.

Can I monitor non-Linux systems (BSD, macOS, Windows)? No. The collection script relies on Linux’s /proc file system.

Can I use a jump host (bastion)? Not yet. Hosts with ProxyJump in an imported SSH configuration are skipped with a warning.

How many hosts can it handle? Collection is concurrent with one persistent session per host; the number of hosts collected at the same time is limited by collect.max_parallel. Storage uses one compact row per host per sample.

Where is the data? In the data directory: helmsight.db (SQLite), secret.key, known_hosts and tls/. Back up the key file separately from the database; stored secrets cannot be decrypted without it. See Installation → Backups.

Can I export the data? Yes: there is a JSON API with an OpenAPI document, a Prometheus endpoint and webhooks for alerts. See API and integrations.

Is there a dark mode? Yes. The UI follows the OS preference, and you can override it in the top bar.

Is the UI available in other languages? Not yet; the UI is in English. The documentation is available in English and Turkish.

Can I use it in production? helmsight is at version 0.x: it is tested, but it has had little production use and no independent security audit. Try it on non-critical hosts first, read the security model, and keep in mind that you use it at your own risk.

I found a bug or a vulnerability. Report bugs through GitHub Issues. Report vulnerabilities privately, without opening a public issue, as described in the security model.

Troubleshooting

Start with helmsight config check and helmsight hosts test. The first validates the configuration, the second connects to every host and prints one line per host with the same states the UI shows.

A host shows “Down”

helmsight could not open a TCP connection or complete the SSH handshake within ssh.connect_timeout.

• From the helmsight server, run ssh -p <port> monitor@<address> true.
• Check firewalls between the two machines and the address and port of the host.
• Unreachable hosts are retried with exponential backoff up to 5 minutes, so a recovered host can take a few minutes to come back.

A host shows “Auth failed”

The SSH server rejected every key helmsight offered.

• Is the public key in ~monitor/.ssh/authorized_keys on the host, with mode 600 and owned by the account?
• Does the from="…" option match the address the host sees? Behind NAT this is the translated address.
• Is the account locked or its shell /usr/sbin/nologin? helmsight needs a working POSIX shell (/bin/sh). Accounts created on Alpine with adduser -D are locked; see Monitoring a fleet.
• The SSH server’s log usually names the reason: journalctl -u ssh -u sshd --since "10 min ago", /var/log/auth.log or /var/log/secure.
• Can the helmsight process read the key file? With the systemd unit, the file must be readable by the helmsight group.
• After authentication failures helmsight waits 1 to 5 minutes before retrying, so it does not trigger fail2ban or similar tools.

“Key not trusted” or “Key changed”

• Key not trusted: the host is new. Run helmsight hosts test --trust or approve the key under Host keys in the UI, after comparing the fingerprint with ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub on the host.
• Key changed: collection stopped on purpose. If the host was reinstalled or its keys were rotated, verify the new fingerprint on the host and approve it under Host keys. If you cannot explain the change, treat it as a possible man-in-the-middle attack.

A view shows “n/a”

“n/a” always comes with the reason reported by the host. Common ones:

Listening ports show process owners only for processes of the monitoring account; that is a kernel restriction for unprivileged users.

Pending updates are empty on Rocky, Alma, RHEL or Fedora

dnf and yum write log files even for read-only queries, so helmsight does not run them unless you set collect.dnf_updates = true. Updates are checked every 6 hours (collect.updates_interval).

I lost the setup link

As long as no user exists, helmsight serve prints a new setup link at every start. Alternatively, create the administrator from the command line:

helmsight user add alice --role admin

I forgot a password or lost a TOTP device

helmsight user reset-password alice --reset-totp

This sets a new password, removes TOTP enrolment and ends the user’s sessions.

Sign-in fails with “Cross-origin request refused”

The browser’s Origin does not match the address helmsight sees. Behind a reverse proxy, set server.public_url to the exact external URL (for example https://monitor.example.com) and keep the proxy’s Host header.

Single sign-on fails

• server.public_url must be set, and <public_url>/api/v1/auth/oidc/callback must be registered as the redirect URI at the identity provider.
• A user without a matching value in role_map is denied unless default_role is set. Check the claim named by role_claim in the identity provider’s token.

Address already in use

If start-up fails because the address is in use, another process (or an older helmsight still running) holds the port:

sudo ss -ltnp 'sport = :8080'

Stop that process or choose another port with --listen or server.listen.

Getting more detail

Run the server with more logging:

HELMSIGHT_LOG=debug helmsight serve

Logs never contain secrets or collected output. When reporting a problem, include helmsight --version, the distribution of the affected host and the output of helmsight hosts test <name>.

Contributing

Thank you for helping. This document covers the development setup, the checks every change must pass and the rules that keep helmsight safe.

Türkçe sürüm

Questions, bug reports and suggestions are welcome in English or Turkish on GitHub Issues. Code, code comments and commit messages are written in English.

Development setup

Requirements: Rust stable (see rust-toolchain.toml), Node.js 22, npm and, for integration tests, Docker.

(cd web && npm ci && npm run build)   # the binary embeds web/dist
cargo build
./target/debug/helmsight serve --local --data-dir /tmp/helmsight-dev

For UI work, run the backend and the Vite dev server side by side; Vite proxies /api to http://127.0.0.1:8080 (override with HELMSIGHT_DEV_BACKEND):

cargo run -- serve --config dev.toml
(cd web && npm run dev)

Without a built UI, cargo build embeds a placeholder page and prints a warning, so Rust-only work does not need Node.js.

Project layout

Checks

Every commit must pass:

cargo fmt --all --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace
cargo deny check
(cd web && npm run check && npm run lint && npm test && npm run build)

Integration tests start sshd containers and verify collection end to end, including that nothing is written on the monitored host:

HELMSIGHT_DOCKER_TESTS=1 cargo test -p transport --test docker
# all six distributions:
HELMSIGHT_DOCKER_TESTS=1 HELMSIGHT_DOCKER_DISTROS="debian ubuntu rocky fedora opensuse alpine" \
  cargo test -p transport --test docker

Parser fixtures

crates/collect/tests/fixtures holds real script output from Debian, Ubuntu, Rocky, Fedora, openSUSE and Alpine, captured as root and as an unprivileged user. After changing remote.sh, regenerate them:

scripts/capture-fixtures.sh            # all distributions
scripts/capture-fixtures.sh alpine     # just one

Review the diff of the fixtures; they are part of the test suite.

Rules for code

• Remote output is untrusted. Parsers must accept arbitrary bytes, never panic (the collect crate denies unwrap, panic and indexing), cap the size of what they keep and fall back to “n/a” with a reason. Add a test with real output and let the property tests in tests/fuzz.rs cover it.
• The remote script is read-only. POSIX sh only, no temporary files, every redirection to /dev/null (a unit test enforces this), every tool optional. Check it with shellcheck -s sh.
• No arbitrary commands. Never add an API that accepts a command, arguments or a path to run on a host.
• Never render remote data as HTML. No {@html}, no innerHTML; ESLint rejects both. Keep the UI compatible with the strict CSP: no inline scripts or style="..." attributes (use classes or style: directives).
• Never log secrets, passwords, tokens or collected output.
• UI copy is short, factual and technical. Errors say what happened and what to do next. Status is never conveyed by color alone.

Commits and pull requests

• Use Conventional Commits: feat(collect): parse /proc/pressure, fix(ui): keep table header sticky.
• Keep commits small and buildable; every commit passes the checks above.
• Update CHANGELOG.md under “Unreleased” for user-visible changes, and the documentation when behaviour or configuration changes. The documentation exists in English and Turkish (README.tr.md, docs/tr/, site/book-tr); update both if you can, or mention in the pull request that the Turkish text needs updating.

Renaming the project

The product name lives in three places: the binary crate name in crates/helmsight/Cargo.toml (and its [[bin]] name), PRODUCT_NAME in crates/common/src/lib.rs (cookie names, database file name, prefixes of Prometheus metrics) and PRODUCT_NAME in web/src/lib/brand.ts.

Reporting security issues

Please do not open public issues for vulnerabilities; see docs/security.md.

Changelog

Release notes are also published on the GitHub releases page. The format follows Keep a Changelog, and version numbers follow Semantic Versioning. The configuration, the API and the database format may change between 0.x releases; read the notes of a release before upgrading.

Unreleased

0.2.0 - 2026-10-01

Added

• An installation script (install.sh, published on the website and with each release) that verifies the download, installs the binary and a hardened systemd service, and creates the configuration, an SSH key and the first administrator. Running it again upgrades an installation; --uninstall removes it and keeps the data.
• helmsight hosts bootstrap prints a script that creates the monitoring account on a host, with the public key restricted by restrict,from=.
• helmsight init accepts its answers as options (--listen, --public-url, --ssh-user, --ssh-key, --admin), runs without questions with --non-interactive, reads the administrator’s password with --password-stdin and creates an ed25519 key with --generate-key.
• Turkish documentation: README.tr.md, docs/tr/ and a Turkish website and documentation under /tr/, with language links between them.
• A troubleshooting page in the documentation.
• More detailed documentation in both languages: requirements, upgrades, backups, reverse proxy setup, example alert rules and notification channels, safe action design and a security checklist.

Fixed

• The README quick start now changes into the directory unpacked from the release archive before running the binary.

0.1.1 - 2026-10-01

Fixed

• Sign-in and every other state-changing request were refused as cross-origin when the web UI was served over TLS with HTTP/2.

0.1.0 - 2026-10-01

Added

• Agentless collection over SSH with one persistent session per host and a single read-only POSIX shell script per tick, tested on Debian, Ubuntu, Rocky, Fedora, openSUSE and Alpine.
• Local mode (serve --local) that monitors the machine it runs on.
• Fleet overview with groups, tags, search, sparklines and distinct states for unreachable hosts, authentication failures, host key problems and partial collections.
• Host detail with live and historical charts, per-core CPU heatmap, top processes, listening ports, services, containers and system information.
• History in SQLite with 1-minute and 5-minute rollups and configurable retention.
• Inventory snapshots, change timeline and host-to-host or baseline comparison.
• Security overview: failed SSH logins, pending and security updates, reboot-required flags, ports not on the baseline and TLS certificate expiry.
• Alert rules with durations and scopes, built-in rules, acknowledgements, silences, reminders, and webhook, Slack and email notifications.
• Opt-in actions with confirmation and a hash-chained, append-only audit log.
• Viewer, operator and admin roles; Argon2id passwords; TOTP; OpenID Connect single sign-on with role mapping.
• Wall display mode with revocable, group-scoped display tokens.
• Versioned JSON API with OpenAPI document, server-sent events and a Prometheus endpoint.
• TLS with provided certificates or a generated self-signed certificate.
• Secrets encrypted at rest with XChaCha20-Poly1305.
• CLI: serve, init, user, hosts test, config check, secret, audit verify.