From 436fa25b8b5b6a5f5875ca71b26ece0ff604133b Mon Sep 17 00:00:00 2001 From: Erik Hjelm Fjeldheim Date: Thu, 1 Oct 2026 13:27:56 -0600 Subject: [PATCH 1/2] fix: #27 stop the deploy job failing and silently skipping the API restart Two bugs, both reported by rachelks after #36 merged and the deploy went red. 1. `systemctl list-unit-files | grep -q '^crate-api.service'` took the else branch even when the unit was installed. `grep -q` exits at the first match and closes the pipe, `systemctl` gets SIGPIPE and dies, and with `pipefail` the pipeline reports failure so the `if` falls through to "not installed yet, skipping". Whether it matched or not depended on how much output `systemctl` had flushed before grep closed, which is why it was intermittent. The log said: Failed to print table: Broken pipe crate-api.service not installed yet (arrives in M2). Skipping. Replaced with `[ -f /etc/systemd/system/crate-api.service ]`. No pipe, no SIGPIPE, no pipefail interaction. 2. `sudo systemctl restart crate-api` fails because the self-hosted runner has no TTY and no passwordless rule: sudo: A terminal is required to authenticate Needs a NOPASSWD rule scoped to exactly these commands. That is a change on the VM, not in this file, so it is called out in the step comment and in the PR. Also gated the restart on `server/` actually being in the repository. The unit on the VM still points at prototype code from 23 Sep, and `server/` does not exist on main until M2. Restarting unconditionally would bring that stale process back. Now it skips with a stated reason while M1 is the current state, and fails loudly if `server/` exists but was never built. Reported by rachelks. Co-Authored-By: opencode --- .github/workflows/ci.yml | 46 +++++++++++++++++++++++++++++----------- 1 file changed, 34 insertions(+), 12 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index e7cb803..b64e12f 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -114,18 +114,40 @@ jobs: chmod -R a+rX /var/www/html/project2 curl -fsS -o /dev/null -w 'GET /project2/ : %{http_code}\n' http://localhost/project2/ - # crate-api only exists from M2 onward. Fail the deploy loudly if the - # unit is present but down; skip quietly if it has not been created yet. - - name: Restart the GraphQL API if present + # Restarting the GraphQL API is M2 work. Until `server/` exists in this + # repository there is nothing to deploy, and restarting the unit would + # just bring back whatever was last installed on the VM by hand. + # + # Two things here are deliberately not what the first version did: + # + # - The unit check is `[ -f ... ]`, not `systemctl list-unit-files | grep`. + # `grep -q` exits on the first match and closes the pipe, `systemctl` + # gets SIGPIPE and dies, and with `pipefail` the whole pipeline reports + # failure. So the `if` took the else branch and skipped a service that + # was definitely there. The log showed "Failed to print table: Broken + # pipe". Reported by rachelks. + # + # - The restart needs `sudo`, and the runner has no TTY. That needs a + # NOPASSWD rule scoped to exactly these commands. Without it the step + # dies with "sudo: A terminal is required to authenticate". + - name: Restart the GraphQL API if the server is deployed run: | set -euo pipefail - if systemctl list-unit-files | grep -q '^crate-api.service'; then - sudo systemctl restart crate-api - sleep 2 - curl -fsS -o /dev/null -w 'POST /graphql : %{http_code}\n' \ - -X POST http://localhost/graphql \ - -H 'content-type: application/json' \ - -d '{"query":"{ __typename }"}' - else - echo "crate-api.service not installed yet (arrives in M2). Skipping." + if [ ! -f /etc/systemd/system/crate-api.service ]; then + echo "crate-api.service is not installed. Skipping API restart." + exit 0 fi + if [ ! -d server ]; then + echo "No server/ in this repository yet (arrives in M2). Skipping API restart." + exit 0 + fi + if [ ! -f server/dist/index.js ]; then + echo "::error::server/ exists but server/dist/index.js does not. Was 'pnpm build' run?" + exit 1 + fi + sudo /usr/bin/systemctl restart crate-api + sleep 2 + curl -fsS -o /dev/null -w 'POST /graphql : %{http_code}\n' \ + -X POST http://localhost/graphql \ + -H 'content-type: application/json' \ + -d '{"query":"{ __typename }"}' From 1f6172cf0f2e1579a56f93069e838e9026ef6164 Mon Sep 17 00:00:00 2001 From: Erik Hjelm Fjeldheim Date: Thu, 1 Oct 2026 13:39:49 -0600 Subject: [PATCH 2/2] fix: #27 poll for the API instead of sleeping two seconds The health check after `systemctl restart crate-api` returned 503 through the Apache proxy on a real run even though the service was active. `Type=simple` makes systemd report active as soon as the process is spawned, well before Node has bound the port. Measured three restarts on the VM: the API answers again after 1878, 1850 and 1864 ms. A fixed `sleep 2` therefore leaves about 120 ms of margin, which is not enough to be reliable. Replaced with a poll: up to 30 attempts, one second apart, each with a two second curl timeout. Reports 200 as soon as it is up, and fails with the last status code if it never is. Verified by running the step against the live unit on the VM. Co-Authored-By: opencode --- .github/workflows/ci.yml | 28 +++++++++++++++++++++++----- 1 file changed, 23 insertions(+), 5 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index b64e12f..5d8c3df 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -146,8 +146,26 @@ jobs: exit 1 fi sudo /usr/bin/systemctl restart crate-api - sleep 2 - curl -fsS -o /dev/null -w 'POST /graphql : %{http_code}\n' \ - -X POST http://localhost/graphql \ - -H 'content-type: application/json' \ - -d '{"query":"{ __typename }"}' + + # `Type=simple` makes systemd report active as soon as the process is + # spawned, well before Node has bound the port. Measured on the VM, + # the API answers again after about 1850 ms, so a fixed `sleep 2` + # leaves a margin of roughly 120 ms. That was enough to return 503 + # through the Apache proxy on at least one run. Poll instead. + printf 'Waiting for the API' + for _ in $(seq 1 30); do + code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 2 \ + -X POST http://localhost/graphql \ + -H 'content-type: application/json' \ + -d '{"query":"{ __typename }"}' 2>/dev/null || echo 000) + if [ "$code" = "200" ]; then + echo + echo "POST /graphql : 200" + exit 0 + fi + printf '.' + sleep 1 + done + echo + echo "::error::crate-api restarted but /graphql never returned 200 (last code ${code})." + exit 1