Skip to content

Drift in CI

Servers drift: someone edits a file by hand during an incident, a package upgrade rewrites a configuration, a service gets stopped and stays stopped. This guide sets up a scheduled CI job that runs sadame drift --json against every server in the inventory, fails the pipeline when one no longer matches its recipe, and keeps the JSON report as an artifact, so the failure tells you exactly what changed and where. Examples are given for GitHub Actions and GitLab CI.

How it works

The CI job installs sadame, loads an SSH key from the CI secrets, and runs:

sadame drift site/site.rb -g all --json > drift.json

Sadame connects to every host of the all group, copies itself there the first time, compares each host with the recipe without changing anything, and writes one JSON document with every host's report (see Remote hosts). Its exit status decides the job:

Exit status Meaning Pipeline
0 every server matches the recipe passes
1 at least one server differs from the recipe fails
2 a server could not be checked: unreachable, a recipe error, or a state sadame could not read fails

On the console, the same run shows each host's differences prefixed with its name, and a summary:

$ sadame drift site.rb --limit 'web*'
[web2.example.com] sadame 0.5.1 drift on web2.example.com (Debian GNU/Linux 12 (bookworm))
[web2.example.com]
[web1.example.com] sadame 0.5.1 drift on web1.example.com (Debian GNU/Linux 12 (bookworm))
[web1.example.com]
[web2.example.com] ~ file[/etc/demo-app/web.conf]
[web2.example.com]     --- /etc/demo-app/web.conf
[web2.example.com]     +++ /etc/demo-app/web.conf (desired)
[web2.example.com]     @@ -1,3 +1,3 @@
[web2.example.com]      # Managed by sadame.
[web2.example.com]      environment = production
[web2.example.com]     -workers     = 12
[web2.example.com]     +workers     = 8
[web2.example.com]
[web1.example.com] No drift: 6 resources match the recipe.
[web2.example.com] Drift: 1 of 6 resources differs from the recipe.

Hosts
  web1.example.com  ok
  web2.example.com  drift

drift on 2 hosts: 1 ok, 1 drift. Finished in 0.22s.

$ echo $?
1

Before you start

  • The recipe and its hosts.rb in the repository, here in site/.
  • A key pair for CI. Its public half goes in authorized_keys of the users named in hosts.rb; its private half is the SSH_PRIVATE_KEY secret.
  • SSH_KNOWN_HOSTS: the output of ssh-keyscan for every server, so the job verifies host keys instead of trusting whatever answers.
  • Passwordless sudo for those users, when the recipe needs root (most do; drift reads files only root can read).

The CI job installs sadame with install.sh, which downloads the latest release. To pin a version, set SADAME_VERSION (see Install).

GitHub Actions

Copy this to .github/workflows/drift.yml and add the two secrets under Settings > Secrets and variables > Actions:

examples/ci/github-actions-drift.yml
# Checks the servers in the inventory for drift every morning and fails the
# workflow when one differs from the recipe. The run's summary page shows
# what differs, and the JSON report is kept as an artifact either way.
#
# Copy to .github/workflows/drift.yml in the repository that holds your
# recipe (here in site/, with site/hosts.rb). It needs repository secrets
# SSH_PRIVATE_KEY (a key the hosts.rb users accept) and SSH_KNOWN_HOSTS (the
# output of `ssh-keyscan` for every server), and passwordless sudo on the
# servers if the recipe needs root.
name: Drift

on:
  schedule:
    - cron: "0 6 * * *"
  workflow_dispatch:

jobs:
  drift:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7

      - name: Install sadame
        run: curl -fsSL https://raw.githubusercontent.com/enderkus/sadame/main/install.sh | sh

      - name: Set up SSH
        run: |
          install -d -m 0700 ~/.ssh
          printf '%s\n' "${{ secrets.SSH_PRIVATE_KEY }}" > ~/.ssh/id_ed25519
          chmod 0600 ~/.ssh/id_ed25519
          printf '%s\n' "${{ secrets.SSH_KNOWN_HOSTS }}" > ~/.ssh/known_hosts

      # Exit status: 0 no drift, 1 drift on a server, 2 a server could not be
      # checked. Anything but 0 fails this step and the workflow.
      - name: Check for drift
        run: sadame drift site/site.rb -g all --json > drift.json

      - name: Summarize
        if: always()
        run: sadame report drift.json >> "$GITHUB_STEP_SUMMARY"

      - name: Keep the drift report
        if: always()
        uses: actions/upload-artifact@v7
        with:
          name: drift-report
          path: drift.json

sadame report drift.json turns the JSON into Markdown for the run's summary page: a table of hosts, and for each host that differs the resources with their diffs. if: always() on the last two steps is what keeps the summary and the report when the drift step fails, which is when you need them. The report appears under the run's Artifacts as drift-report.

sadame init --ci github writes this workflow, with the path of your recipe, to .github/workflows/sadame-drift.yml.

GitLab CI

Add this job to .gitlab-ci.yml, add the two values as CI/CD variables, and create a pipeline schedule under Build > Pipeline schedules:

examples/ci/gitlab-ci-drift.yml
# Checks the servers in the inventory for drift in scheduled pipelines and
# fails the job when one differs from the recipe. The pipeline's test report
# shows what differs, and the JSON report is kept as an artifact either way.
#
# Add this job to .gitlab-ci.yml in the project that holds your recipe (here
# in site/, with site/hosts.rb) and create a pipeline schedule. It needs
# CI/CD variables SSH_PRIVATE_KEY and SSH_KNOWN_HOSTS (masked, protected),
# and passwordless sudo on the servers if the recipe needs root.
drift:
  image: alpine:3.20
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"
    - if: $CI_PIPELINE_SOURCE == "web"
  before_script:
    - apk add --no-cache openssh-client
    - wget -qO- https://raw.githubusercontent.com/enderkus/sadame/main/install.sh | sh
    - install -d -m 0700 ~/.ssh
    - printf '%s\n' "$SSH_PRIVATE_KEY" > ~/.ssh/id_ed25519
    - chmod 0600 ~/.ssh/id_ed25519
    - printf '%s\n' "$SSH_KNOWN_HOSTS" > ~/.ssh/known_hosts
  script:
    # Exit status: 0 no drift, 1 drift on a server, 2 a server could not be checked.
    - sadame drift site/site.rb -g all --json > drift.json
  after_script:
    - sadame report drift.json --format junit > drift.xml
  artifacts:
    when: always
    paths:
      - drift.json
    reports:
      junit: drift.xml
    expire_in: 30 days

after_script turns the JSON into JUnit XML with sadame report drift.json --format junit, and reports: junit: hands it to GitLab, which shows each host as a test suite and each drifted resource as a failed test, with its diff, on the pipeline's Tests tab. artifacts: when: always keeps drift.json on failed jobs too.

sadame init --ci gitlab writes this job, with the path of your recipe, to .gitlab/sadame-drift.yml, to include from .gitlab-ci.yml.

Reading the report

sadame report prints the summary the workflows above put on the run's page:

$ sadame report drift.json
### sadame drift on 2 hosts: 1 ok, 1 drift

| Host | Result | Resources |
|---|---|---|
| web1.example.com | ok | 6 unchanged |
| web2.example.com | **drift** | 1 of 6 differ |

<details>
<summary>web2.example.com: 1 of 6 differ</summary>

**~ file[/etc/demo-app/web.conf]**

```diff
--- current
+++ desired
@@ -1,3 +1,3 @@
 # Managed by sadame.
 environment = production
-workers     = 12
+workers     = 8

and with `--format junit`, JUnit XML for any CI system that reads test
reports (Jenkins, GitLab, Azure Pipelines and others):

```text
$ sadame report drift.json --format junit
<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="sadame drift" tests="12" failures="1" errors="0">
  <testsuite name="web1.example.com" tests="6" failures="0" errors="0">
    <testcase classname="web1.example.com" name="file[/etc/motd]"/>
    <testcase classname="web1.example.com" name="package[nginx]"/>
    <testcase classname="web1.example.com" name="directory[/etc/demo-app]"/>
    <testcase classname="web1.example.com" name="file[/etc/demo-app/web.conf]"/>
    <testcase classname="web1.example.com" name="file[/etc/nginx/conf.d/demo-app.conf]"/>
    <testcase classname="web1.example.com" name="service[nginx]"/>
  </testsuite>
  <testsuite name="web2.example.com" tests="6" failures="1" errors="0">
    <testcase classname="web2.example.com" name="file[/etc/motd]"/>
    <testcase classname="web2.example.com" name="package[nginx]"/>
    <testcase classname="web2.example.com" name="directory[/etc/demo-app]"/>
    <testcase classname="web2.example.com" name="file[/etc/demo-app/web.conf]">
      <failure message="differs from the recipe" type="update">--- current
+++ desired
@@ -1,3 +1,3 @@
 # Managed by sadame.
 environment = production
-workers     = 12
+workers     = 8</failure>
    </testcase>
    <testcase classname="web2.example.com" name="file[/etc/nginx/conf.d/demo-app.conf]"/>
    <testcase classname="web2.example.com" name="service[nginx]"/>
  </testsuite>
</testsuites>

The JSON has one entry per host with its status and full report; the format is in the JSON output reference. With jq, this lists the resources that drifted, per host:

jq -r '.hosts[] | .name as $host | (.report.resources // [])[]
       | select(.status != "unchanged") | "\($host): \(.id) \(.status)"' drift.json

and this the hosts that could not be checked:

jq -r '.hosts[] | select(.status == "error" or .status == "unreachable") | "\(.name): \(.error)"' drift.json

Sadame's own CI runs sadame drift --json on its runner on every push, edits a managed file, and expects exit status 1; see .github/workflows/ci.yml in the repository.

Common mistakes

  • Unguarded exec resources. They count as drift on every run, so the job never passes. Give each one a guard or action: :nothing.
  • No sudo. Without it, files only root can read fail to compare and the job exits 2 instead of telling you about drift.
  • Losing the report on failure. Without if: always() (GitHub) or when: always (GitLab), the upload step is skipped exactly when drift is found.
  • Trusting any host key. StrictHostKeyChecking=no lets anyone who can answer for a server's name read your recipe and its vars. Pin the keys with SSH_KNOWN_HOSTS.
  • Applying from the drift job. Keep the scheduled job read-only and review the drift first; apply on purpose, after looking at plan.

See also