Docs: Document how to roll back a bad deploy

docs/ci-cd.md

@@ -7,7 +7,7 @@ This page explains how our continuous integration (CI) and deployment (CD) work.
 - **CI is automated.** A GitHub Actions workflow checks every pull request and
   every push to `main`.
 - **CD is automated.** Every push to `main` that passes the checks is deployed
-  to the VM at http://it2810-25.idi.ntnu.no/project2/.
+  to the VM at <http://it2810-25.idi.ntnu.no/project2/>.
 
 ## Continuous integration
 
@@ -155,5 +155,66 @@ sudoers files if it does not.
   `/srv/ufound-it/backend/.env` does not match the one MySQL has for that user.
   Fix the file, then run `sudo systemctl restart ufound-it-backend`. The backend
   only reads `.env` when it starts.
-- **Rolling back.** Revert the merge commit on `main` with a pull request. The
-  revert is deployed like any other merge.
+- **The site is broken after a deploy,** even though the run is green. See
+  [Rolling back a bad deploy](#rolling-back-a-bad-deploy).
+
+### Rolling back a bad deploy
+
+The site check only tests that the backend starts and can read from the
+database, so a merge can pass it and still break a page. Rolling back puts the
+last good version back on the VM. There are two ways, and you usually do both:
+the first to get the site working within minutes, the second so the next merge
+does not deploy the broken code again.
+
+**1. Redeploy the last good commit (fast).**
+
+1. Open _Deployments_ on the repository page, or the `production` environment.
+   It lists every deploy with its commit. Find the last one before the bad
+   merge.
+2. Open the `CI/CD` run on `main` for that commit in the _Actions_ tab (runs
+   before #73 are listed as `CI`) and click _Re-run all jobs_.
+3. A re-run builds and deploys the commit of the original run, not the latest
+   `main`. When the `deploy` job is green, the old version is live again.
+
+This does not change `main`, so the bad commit is still there and the next
+merge deploys it again. Always follow up with step 2. GitHub only re-runs runs
+from the last 30 days; for anything older, use step 2 alone.
+
+**2. Revert the merge on `main` (permanent).**
+
+1. Open the merged pull request that caused the problem and click _Revert_.
+   GitHub creates a new branch and a pull request that undoes the merge.
+2. Review and merge it like any other pull request. The merge is deployed as
+   usual.
+
+Without the button, the same can be done from the command line. `-m 1` tells
+Git to undo the merge relative to `main` before it:
+
+```bash
+git switch main && git pull
+git switch -c revert-<pr-number>
+git revert -m 1 <merge-commit-sha>
+git push -u origin revert-<pr-number>
+```
+
+To bring the change back later, fix it on a new branch and merge that. Merging
+the old branch again does nothing, because Git considers its commits already
+merged; revert the revert commit first if you want to reuse them.
+
+**What a rollback does not undo.** Only the code on the VM changes. These stay
+as the bad version left them and have to be fixed by hand if they were part of
+the problem:
+
+- **The database.** Rows the bad version wrote stay, and schema changes run by
+  hand on the VM are not reverted. If the old code no longer matches the
+  schema, undo the schema change or fix forward instead of rolling back.
+- **`/srv/ufound-it/backend/.env`** and the one-time setup (service, sudoers,
+  Apache). CI never changes these.
+- **Pages already open in a browser.** They keep the old JavaScript until
+  reloaded and may fail to load a page they have not visited yet, because the
+  deploy replaced its file. A reload fixes it.
+
+**Check the rollback.** The newest entry under _Deployments_ should be the
+commit you rolled back to, and the site should work again. If the `deploy` job
+itself fails or never starts, the runner is the problem, not the code; see
+[If a CI run never starts](#the-self-hosted-runner).