Skip to content

Remote hosts

With -h, -g or --limit, sadame runs a recipe on other machines over SSH. Nothing has to be installed on them first: sadame copies its own binary and the recipe with your system's scp, runs it with ssh, and shows each host's output as it arrives, prefixed with the host's name. This guide covers how hosts are named and reached, what is copied where, sudo, jump hosts, running many hosts at once, and what to do when a connection fails.

Quick example

cd examples/08-multi-role-inventory
sadame plan site.rb -h deploy@db1.example.com:2222

$ sadame plan site.rb -h deploy@db1.example.com:2222
[db1.example.com] sadame 0.5.1 plan on db1.example.com (Debian GNU/Linux 12 (bookworm))
[db1.example.com]
[db1.example.com] Plan: 0 to create, 0 to update, 0 to remove, 0 to run, 6 unchanged.

Hosts
  db1.example.com  ok

plan on 1 host: 1 ok. Finished in 0.21s.

The targets need a POSIX shell, tar with gzip support, sha256sum (or shasum), and Linux or macOS on x86_64 or ARM64. The machine running sadame needs the OpenSSH client (ssh and scp).

Naming a host: [user@]host[:port]

Form Meaning
web1.example.com user and port from ~/.ssh/config, or ssh's defaults
deploy@web1.example.com log in as deploy
db1.example.com:2222 connect to port 2222
admin@db1.example.com:2222 both
root@[2001:db8::1]:22 IPv6 addresses go in brackets

A name that is also in the inventory keeps the options from there; the user and port given with -h replace those. Several hosts are separated by commas: -h web1.example.com,web2.example.com.

SSH configuration and keys

Sadame runs the system's ssh and scp, so everything in ~/.ssh/config applies: HostName, User, Port, IdentityFile, ProxyJump, and the SSH agent. It adds these options to every connection:

Option Why
-o BatchMode=yes never stop to ask for a password; use keys or an agent
-o ConnectTimeout=10 an unreachable host fails after 10 seconds
-o ControlMaster=auto, ControlPath, ControlPersist=60 the several commands of a push share one connection

A host's ssh_opts from hosts.rb come before them, so they can override them; ssh uses the first value it sees for an option. They are passed to scp too, so use options both accept, such as -i, -F, -J and -o.

The first connection to a new host checks its host key against ~/.ssh/known_hosts as usual. With BatchMode, an unknown key is not accepted interactively: connect once with ssh yourself, add the key with ssh-keyscan, or set StrictHostKeyChecking accept-new for those hosts in ~/.ssh/config.

Non-standard ports

Any of these works; hosts.rb is the usual place, as for db1 in examples/08-multi-role-inventory/hosts.rb:

host "db1.example.com", port: 2222    # in hosts.rb

sadame plan site.rb -h db1.example.com:2222

Host db1.example.com                  # in ~/.ssh/config
  Port 2222

ssh takes the port with -p and scp with -P; sadame passes the right one to each.

Jump hosts

A host that is only reachable through a bastion needs one ProxyJump line in ~/.ssh/config; nothing in sadame changes:

Host app1.internal
  User deploy
  ProxyJump bastion.example.com

To keep that configuration with the recipe instead, put it in a file and pass it with -F through ssh_opts. With -F, ssh reads only that file, so it must also name the key and known hosts file:

host "app1.internal", ssh_opts: ["-F", "ssh_config"]

Sadame's own tests reach a host on a private network through a bastion this way; see test/integration/push/ssh_config and the internal group in test/integration/push/recipe/hosts.rb.

sudo

Most recipes need root on the target. Log in as a user with sudo rights and let sadame use sudo, either per host or for every target of a run:

host "web1.example.com", user: "deploy", sudo: true    # in hosts.rb

sadame apply site.rb -g web --sudo                    # every selected host

sudo must not ask for a password, because there is no terminal to type it into. Sadame checks that with sudo -n true before it copies anything. A line like this in /etc/sudoers.d/deploy (edit it with visudo -f) gives the deploy user passwordless sudo:

deploy ALL=(ALL) NOPASSWD: ALL

What is copied where

What Where When
The sadame binary for the target's system ~/.sadame/bin/SHA256/sadame on the target only when that checksum is not there yet; it is checked after copying
The recipe's directory, without .git, vars.rb and hosts.rb ~/.sadame/run/RUN_ID/ on the target every run; removed when sadame finishes there
The target's vars, already merged ~/.sadame/run/RUN_ID/.sadame-vars.rb every run; removed with the directory
Binaries downloaded for other systems ~/.sadame/dist/VERSION/ on the machine running sadame the first time a system is needed

vars.rb and hosts.rb are evaluated on the machine running sadame, so ENV and files they read are those of that machine. The target receives the result. Templates and the recipe itself run on the target.

To clean a target, remove ~/.sadame.

Where the binary comes from

Sadame needs a binary for each target's system (uname -sm on the target). It looks in this order:

  1. $SADAME_BINARY_DIR/sadame-linux-amd64 (and so on), if that variable is set; useful for development builds and machines without internet access: SADAME_BINARY_DIR=dist;
  2. itself, when it is the binary for that system (a Linux x86_64 machine pushing to Linux x86_64 targets copies itself);
  3. ~/.sadame/dist/VERSION/, from an earlier download;
  4. the GitHub release of the same version, checked against its SHA256SUMS; $SADAME_DOWNLOAD_URL points at a mirror instead.

Many hosts at once

By default every selected host runs at the same time, in its own process. Lines from different hosts interleave, each with its host's name in front, and a table at the end shows each host's result:

$ sadame drift site.rb --limit 'web*'
[web2.example.com] sadame 0.5.1 drift on web2.example.com (Debian GNU/Linux 12 (bookworm))
[web2.example.com]
[web1.example.com] sadame 0.5.1 drift on web1.example.com (Debian GNU/Linux 12 (bookworm))
[web1.example.com]
[web2.example.com] ~ file[/etc/demo-app/web.conf]
[web2.example.com]     --- /etc/demo-app/web.conf
[web2.example.com]     +++ /etc/demo-app/web.conf (desired)
[web2.example.com]     @@ -1,3 +1,3 @@
[web2.example.com]      # Managed by sadame.
[web2.example.com]      environment = production
[web2.example.com]     -workers     = 12
[web2.example.com]     +workers     = 8
[web2.example.com]
[web1.example.com] No drift: 6 resources match the recipe.
[web2.example.com] Drift: 1 of 6 resources differs from the recipe.

Hosts
  web1.example.com  ok
  web2.example.com  drift

drift on 2 hosts: 1 ok, 1 drift. Finished in 0.22s.

$ echo $?
1
Result Meaning Counts as exit status
ok sadame finished with status 0 0
drift drift found a difference 1
failed apply had a failed resource 1
error sadame could not run, or exited with 2 2
unreachable ssh could not connect 2
skipped not run, because an earlier batch failed (--serial) none

The exit status of the whole run is the highest of its hosts. A host that cannot be reached does not stop the others:

$ sadame plan site.rb -g db -h deploy@db2.example.com
[db1.example.com] sadame 0.5.1 plan on db1.example.com (Debian GNU/Linux 12 (bookworm))
[db1.example.com]
[db1.example.com] Plan: 0 to create, 0 to update, 0 to remove, 0 to run, 6 unchanged.

Hosts
  db1.example.com  ok
  db2.example.com  unreachable  connect failed: ssh: Could not resolve hostname db2.example.com: Name or service not known

plan on 2 hosts: 1 ok, 1 unreachable. Finished in 0.22s.

$ echo $?
2
Full output of sadame apply site.rb -g all on three fresh servers 
$ sadame apply site.rb -g all
[db1.example.com] copying sadame-linux-arm64 to ~/.sadame/bin/dcf637ae7d2f79587cd36cc4fb47eff845444424933647a5e75565917475c21d/
[web2.example.com] sadame 0.5.1 apply on web2.example.com (Debian GNU/Linux 12 (bookworm))
[web1.example.com] sadame 0.5.1 apply on web1.example.com (Debian GNU/Linux 12 (bookworm))
[web1.example.com]
[web2.example.com]
[web2.example.com] changed    file[/etc/motd]
[web2.example.com]            --- /etc/motd
[web2.example.com]            +++ /etc/motd (desired)
[web1.example.com] changed    file[/etc/motd]
[web2.example.com]            @@ -1,7 +1,3 @@
[web2.example.com]            -
[web2.example.com]            -The programs included with the Debian GNU/Linux system are free software;
[web2.example.com]            -the exact distribution terms for each program are described in the
[web1.example.com]            --- /etc/motd
[web2.example.com]            -individual files in /usr/share/doc/*/copyright.
[web2.example.com]            -
[web2.example.com]            -Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
[web2.example.com]            -permitted by applicable law.
[web2.example.com]            +web2.example.com (production)
[web2.example.com]            +Groups: web, all
[web2.example.com]            +Managed by sadame. Local changes will be overwritten.
[web1.example.com]            +++ /etc/motd (desired)
[web1.example.com]            @@ -1,7 +1,3 @@
[web1.example.com]            -
[web1.example.com]            -The programs included with the Debian GNU/Linux system are free software;
[web1.example.com]            -the exact distribution terms for each program are described in the
[web1.example.com]            -individual files in /usr/share/doc/*/copyright.
[web1.example.com]            -
[web1.example.com]            -Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
[web1.example.com]            -permitted by applicable law.
[web1.example.com]            +web1.example.com (production)
[web1.example.com]            +Groups: web, all
[web1.example.com]            +Managed by sadame. Local changes will be overwritten.
[db1.example.com] sadame 0.5.1 apply on db1.example.com (Debian GNU/Linux 12 (bookworm))
[db1.example.com]
[db1.example.com] changed    file[/etc/motd]
[db1.example.com]            --- /etc/motd
[db1.example.com]            +++ /etc/motd (desired)
[db1.example.com]            @@ -1,7 +1,3 @@
[db1.example.com]            -
[db1.example.com]            -The programs included with the Debian GNU/Linux system are free software;
[db1.example.com]            -the exact distribution terms for each program are described in the
[db1.example.com]            -individual files in /usr/share/doc/*/copyright.
[db1.example.com]            -
[db1.example.com]            -Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
[db1.example.com]            -permitted by applicable law.
[db1.example.com]            +db1.example.com (production)
[db1.example.com]            +Groups: db, all
[db1.example.com]            +Managed by sadame. Local changes will be overwritten.
[web2.example.com] changed    package[nginx]
[web2.example.com]            state: absent -> installed
[web2.example.com] changed    directory[/etc/demo-app]
[web2.example.com]            state: absent -> present
[web2.example.com]            mode: 0755
[web2.example.com] changed    file[/etc/demo-app/web.conf]
[web2.example.com]            state: absent -> present
[web2.example.com]            mode: 0644
[web2.example.com]            --- /dev/null
[web2.example.com]            +++ /etc/demo-app/web.conf (desired)
[web2.example.com]            @@ -0,0 +1,3 @@
[web2.example.com]            +# Managed by sadame.
[web2.example.com]            +environment = production
[web2.example.com]            +workers     = 8
[web2.example.com] changed    file[/etc/nginx/conf.d/demo-app.conf]
[web2.example.com]            state: absent -> present
[web2.example.com]            mode: 0644
[web2.example.com]            --- /dev/null
[web2.example.com]            +++ /etc/nginx/conf.d/demo-app.conf (desired)
[web2.example.com]            @@ -0,0 +1,8 @@
[web2.example.com]            +# Managed by sadame.
[web2.example.com]            +server {
[web2.example.com]            +  listen 8080;
[web2.example.com]            +  server_name www.example.com;
[web2.example.com]            +  location / {
[web2.example.com]            +    return 200 "production\n";
[web2.example.com]            +  }
[web2.example.com]            +}
[web2.example.com] changed    service[nginx]
[web2.example.com]            state: stopped -> running
[web2.example.com] notified   service[nginx]: reload (delayed, from file[/etc/nginx/conf.d/demo-app.conf])
[web2.example.com]
[web2.example.com] Summary
[web2.example.com]   changed      6  file[/etc/motd]
[web2.example.com]                   package[nginx]
[web2.example.com]                   directory[/etc/demo-app]
[web2.example.com]                   file[/etc/demo-app/web.conf]
[web2.example.com]                   file[/etc/nginx/conf.d/demo-app.conf]
[web2.example.com]                   service[nginx]
[web2.example.com]   unchanged    0
[web2.example.com]   failed       0
[web2.example.com]   skipped      0
[web2.example.com]   notified     1  service[nginx]: reload
[web2.example.com]
[web2.example.com] Finished in 7.27s.
[db1.example.com] changed    package[sqlite3]
[db1.example.com]            state: absent -> installed
[web1.example.com] changed    package[nginx]
[web1.example.com]            state: absent -> installed
[web1.example.com] changed    directory[/etc/demo-app]
[web1.example.com]            state: absent -> present
[web1.example.com]            mode: 0755
[web1.example.com] changed    file[/etc/demo-app/web.conf]
[web1.example.com]            state: absent -> present
[web1.example.com]            mode: 0644
[web1.example.com]            --- /dev/null
[web1.example.com]            +++ /etc/demo-app/web.conf (desired)
[web1.example.com]            @@ -0,0 +1,3 @@
[web1.example.com]            +# Managed by sadame.
[web1.example.com]            +environment = production
[web1.example.com]            +workers     = 4
[web1.example.com] changed    file[/etc/nginx/conf.d/demo-app.conf]
[web1.example.com]            state: absent -> present
[web1.example.com]            mode: 0644
[web1.example.com]            --- /dev/null
[web1.example.com]            +++ /etc/nginx/conf.d/demo-app.conf (desired)
[web1.example.com]            @@ -0,0 +1,8 @@
[web1.example.com]            +# Managed by sadame.
[web1.example.com]            +server {
[web1.example.com]            +  listen 8080;
[web1.example.com]            +  server_name www.example.com;
[web1.example.com]            +  location / {
[web1.example.com]            +    return 200 "production\n";
[web1.example.com]            +  }
[web1.example.com]            +}
[web1.example.com] changed    service[nginx]
[web1.example.com]            state: stopped -> running
[web1.example.com] notified   service[nginx]: reload (delayed, from file[/etc/nginx/conf.d/demo-app.conf])
[web1.example.com]
[web1.example.com] Summary
[web1.example.com]   changed      6  file[/etc/motd]
[web1.example.com]                   package[nginx]
[web1.example.com]                   directory[/etc/demo-app]
[web1.example.com]                   file[/etc/demo-app/web.conf]
[web1.example.com]                   file[/etc/nginx/conf.d/demo-app.conf]
[web1.example.com]                   service[nginx]
[web1.example.com]   unchanged    0
[web1.example.com]   failed       0
[web1.example.com]   skipped      0
[web1.example.com]   notified     1  service[nginx]: reload
[web1.example.com]
[web1.example.com] Finished in 11.10s.
[db1.example.com] changed    package[cron]
[db1.example.com]            state: absent -> installed
[db1.example.com] changed    user[demo-db]
[db1.example.com]            state: absent -> present
[db1.example.com]            home: /var/lib/demo-db
[db1.example.com]            shell: /usr/sbin/nologin
[db1.example.com] changed    directory[/var/lib/demo-db]
[db1.example.com]            state: absent -> present
[db1.example.com]            mode: 0750
[db1.example.com]            owner: demo-db
[db1.example.com]            group: demo-db
[db1.example.com] changed    file[/etc/cron.d/demo-db-backup]
[db1.example.com]            state: absent -> present
[db1.example.com]            mode: 0644
[db1.example.com]            --- /dev/null
[db1.example.com]            +++ /etc/cron.d/demo-db-backup (desired)
[db1.example.com]            @@ -0,0 +1 @@
[db1.example.com]            +0 3 * * * demo-db sqlite3 /var/lib/demo-db/app.db .dump > /var/lib/demo-db/backup.sql
[db1.example.com]
[db1.example.com] Summary
[db1.example.com]   changed      6  file[/etc/motd]
[db1.example.com]                   package[sqlite3]
[db1.example.com]                   package[cron]
[db1.example.com]                   user[demo-db]
[db1.example.com]                   directory[/var/lib/demo-db]
[db1.example.com]                   file[/etc/cron.d/demo-db-backup]
[db1.example.com]   unchanged    0
[db1.example.com]   failed       0
[db1.example.com]   skipped      0
[db1.example.com]   notified     0
[db1.example.com]
[db1.example.com] Finished in 14.63s.

Hosts
  web1.example.com  ok
  web2.example.com  ok
  db1.example.com   ok

apply on 3 hosts: 3 ok. Finished in 14.87s.

With --json, the run prints one document with each host's result and its full report; see JSON output.

Fewer at a time: --forks

--forks N works on at most N hosts at once; the next one starts as soon as one finishes. Use it when the machine running sadame, or a jump host between it and the servers, should not open a connection to every server at once. Through a jump host, keep it at 10 or below: past 10 connections that are still logging in, OpenSSH's sshd refuses new ones at random (MaxStartups).

sadame apply site.rb -g all --forks 10

Rolling changes: --serial

--serial N runs the hosts in batches of N, in the order they were selected, one batch after another; --serial 25% makes each batch a quarter of the hosts. When a host of a batch ends as failed, error or unreachable, sadame stops after that batch, and the hosts of later batches are skipped: a change that breaks one server does not reach the others. drift results do not stop the run.

Here web2 has a file where the recipe wants a directory, so its apply fails and db1, in the third batch, is never touched:

$ sadame apply site.rb -g all --serial 1
Batch 1 of 3: web1.example.com
[web1.example.com] sadame 0.5.1 apply on web1.example.com (Debian GNU/Linux 12 (bookworm))
[web1.example.com]
[web1.example.com] unchanged  file[/etc/motd]
[web1.example.com] unchanged  package[nginx]
[web1.example.com] unchanged  directory[/etc/demo-app]
[web1.example.com] unchanged  file[/etc/demo-app/web.conf]
[web1.example.com] unchanged  file[/etc/nginx/conf.d/demo-app.conf]
[web1.example.com] unchanged  service[nginx]
[web1.example.com]
[web1.example.com] Summary
[web1.example.com]   changed      0
[web1.example.com]   unchanged    6
[web1.example.com]   failed       0
[web1.example.com]   skipped      0
[web1.example.com]   notified     0
[web1.example.com]
[web1.example.com] Finished in 0.01s.

Batch 2 of 3: web2.example.com
[web2.example.com] sadame 0.5.1 apply on web2.example.com (Debian GNU/Linux 12 (bookworm))
[web2.example.com]
[web2.example.com] unchanged  file[/etc/motd]
[web2.example.com] unchanged  package[nginx]
[web2.example.com] failed     directory[/etc/demo-app]
[web2.example.com]            error: directory[/etc/demo-app]: /etc/demo-app is a regular file, not a directory
[web2.example.com] skipped    file[/etc/demo-app/web.conf]
[web2.example.com] skipped    file[/etc/nginx/conf.d/demo-app.conf]
[web2.example.com] skipped    service[nginx]
[web2.example.com]
[web2.example.com] Summary
[web2.example.com]   changed      0
[web2.example.com]   unchanged    2
[web2.example.com]   failed       1  directory[/etc/demo-app]
[web2.example.com]   skipped      3  file[/etc/demo-app/web.conf]
[web2.example.com]                   file[/etc/nginx/conf.d/demo-app.conf]
[web2.example.com]                   service[nginx]
[web2.example.com]   notified     0
[web2.example.com]
[web2.example.com] Finished in 0.00s.

Stopped after batch 2 of 3: web2.example.com did not succeed; 1 host not run.

Hosts
  web1.example.com  ok
  web2.example.com  failed
  db1.example.com   skipped      not run: batch 2 failed

apply on 3 hosts: 1 ok, 1 failed, 1 skipped. Finished in 0.42s.

$ echo $?
1

--forks limits each batch too: --serial 10 --forks 5 runs batches of ten hosts, five at a time.

Troubleshooting

Symptom in the summary Cause Fix
connect failed: USER@HOST: Permission denied (publickey). no key the server accepts for that user check the user name; ssh-copy-id USER@HOST; set IdentityFile or ssh_opts: ["-i", "..."]
connect failed: Host key verification failed. the host's key is unknown or changed if the change is expected, ssh-keygen -R HOST and connect once; if not, find out why first
connect failed: ssh: connect to host HOST port 22: Connection refused wrong port, or sshd not running give the port: HOST:2222, port: 2222 or Port in ~/.ssh/config
connect failed: ssh: Could not resolve hostname HOST DNS or a typo check the name; set address: in hosts.rb or HostName in ~/.ssh/config
sudo failed: sudo: a password is required sudo asks for a password give the user passwordless sudo, or run without sudo
/PATH/sadame-linux-arm64 does not exist (SADAME_BINARY_DIR is set) or no sadame-linux-arm64 for this development version no binary for the target's system build it (rake build:linux-arm64) or use a released version
sadame has no binary for Linux riscv64 the target's CPU is not supported run sadame from source on that machine

The first three, as sadame reports them:

$ sadame plan site.rb -h nobody@web1.example.com

Hosts
  web1.example.com  unreachable  connect failed: nobody@web1.example.com: Permission denied (publickey).

plan on 1 host: 1 unreachable. Finished in 0.16s.

$ sadame plan site.rb -h web1.example.com

Hosts
  web1.example.com  unreachable  connect failed: Host key verification failed.

plan on 1 host: 1 unreachable. Finished in 0.04s.

$ sadame plan site.rb -h deploy@db1.example.com:22

Hosts
  db1.example.com  unreachable  connect failed: ssh: connect to host db1.example.com port 22: Connection refused

plan on 1 host: 1 unreachable. Finished in 0.01s.

And a user whose sudo asks for a password:

$ sadame plan site.rb -h ops@web1.example.com --sudo

Hosts
  web1.example.com  error        sudo failed: sudo: a password is required

plan on 1 host: 1 error. Finished in 0.20s.

Common mistakes

  • Expecting password prompts. Sadame never asks for SSH or sudo passwords. Use keys and passwordless sudo.
  • Recipes in a large directory. The recipe's whole directory is copied to every host on every run. Keep recipes in their own directory, not at the top of a large repository.
  • Reading local files in a recipe. A recipe runs on the target; a path like ~/.ssh/id_rsa.pub there means the target's file. Put files the recipe needs next to it and use source:.
  • Pushing to a target with a different sadame version installed. That does not matter: sadame always runs the binary it copied, identified by its checksum.

See also