From 61ddb765230cc3269b59786cf3c33a9e30f1d75c Mon Sep 17 00:00:00 2001 From: Thor Tveito Date: Thu, 15 Oct 2026 11:13:05 +0200 Subject: [PATCH] Docs: Document how to roll back a bad deploy Add a rollback section to docs/ci-cd.md. Re-running the deploy of the last good commit gets the site working within minutes, and reverting the merge on main stops the next merge from deploying the bad code again. The section also lists what a rollback does not undo: the database, the VM's .env and one-time setup, and pages already open in a browser. Closes #69 Co-Authored-By: Claude Opus 5.5 --- docs/ci-cd.md | 67 ++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 64 insertions(+), 3 deletions(-) diff --git a/docs/ci-cd.md b/docs/ci-cd.md index c51d575..b04df4b 100644 --- a/docs/ci-cd.md +++ b/docs/ci-cd.md @@ -7,7 +7,7 @@ This page explains how our continuous integration (CI) and deployment (CD) work. - **CI is automated.** A GitHub Actions workflow checks every pull request and every push to `main`. - **CD is automated.** Every push to `main` that passes the checks is deployed - to the VM at http://it2810-25.idi.ntnu.no/project2/. + to the VM at . ## Continuous integration @@ -155,5 +155,66 @@ sudoers files if it does not. `/srv/ufound-it/backend/.env` does not match the one MySQL has for that user. Fix the file, then run `sudo systemctl restart ufound-it-backend`. The backend only reads `.env` when it starts. -- **Rolling back.** Revert the merge commit on `main` with a pull request. The - revert is deployed like any other merge. +- **The site is broken after a deploy,** even though the run is green. See + [Rolling back a bad deploy](#rolling-back-a-bad-deploy). + +### Rolling back a bad deploy + +The site check only tests that the backend starts and can read from the +database, so a merge can pass it and still break a page. Rolling back puts the +last good version back on the VM. There are two ways, and you usually do both: +the first to get the site working within minutes, the second so the next merge +does not deploy the broken code again. + +**1. Redeploy the last good commit (fast).** + +1. Open _Deployments_ on the repository page, or the `production` environment. + It lists every deploy with its commit. Find the last one before the bad + merge. +2. Open the `CI/CD` run on `main` for that commit in the _Actions_ tab (runs + before #73 are listed as `CI`) and click _Re-run all jobs_. +3. A re-run builds and deploys the commit of the original run, not the latest + `main`. When the `deploy` job is green, the old version is live again. + +This does not change `main`, so the bad commit is still there and the next +merge deploys it again. Always follow up with step 2. GitHub only re-runs runs +from the last 30 days; for anything older, use step 2 alone. + +**2. Revert the merge on `main` (permanent).** + +1. Open the merged pull request that caused the problem and click _Revert_. + GitHub creates a new branch and a pull request that undoes the merge. +2. Review and merge it like any other pull request. The merge is deployed as + usual. + +Without the button, the same can be done from the command line. `-m 1` tells +Git to undo the merge relative to `main` before it: + +```bash +git switch main && git pull +git switch -c revert- +git revert -m 1 +git push -u origin revert- +``` + +To bring the change back later, fix it on a new branch and merge that. Merging +the old branch again does nothing, because Git considers its commits already +merged; revert the revert commit first if you want to reuse them. + +**What a rollback does not undo.** Only the code on the VM changes. These stay +as the bad version left them and have to be fixed by hand if they were part of +the problem: + +- **The database.** Rows the bad version wrote stay, and schema changes run by + hand on the VM are not reverted. If the old code no longer matches the + schema, undo the schema change or fix forward instead of rolling back. +- **`/srv/ufound-it/backend/.env`** and the one-time setup (service, sudoers, + Apache). CI never changes these. +- **Pages already open in a browser.** They keep the old JavaScript until + reloaded and may fail to load a page they have not visited yet, because the + deploy replaced its file. A reload fixes it. + +**Check the rollback.** The newest entry under _Deployments_ should be the +commit you rolled back to, and the site should work again. If the `deploy` job +itself fails or never starts, the runner is the problem, not the code; see +[If a CI run never starts](#the-self-hosted-runner).