diff --git a/README.md b/README.md index 7f60b44..da9eb6d 100644 --- a/README.md +++ b/README.md @@ -183,7 +183,97 @@ parameters, timing and query plan, plus client events like search debouncing. It talks to Postgres directly and is a development tool only, not part of the app. For raw SQL: `psql -d t13`. -## Deploying the frontend +## Deployment + +Everything below runs directly on the group VM (`it2810-13.idi.ntnu.no`), no +Docker. `sudo` is required for the one-time steps and is only available to +members who were given it on the VM. + +### Database and backend (once per VM, plus whenever the backend changes) + +1. PostgreSQL and `unzip` (skip if already installed): + + ```bash + sudo apt install postgresql unzip + sudo -u postgres createuser --createdb + createdb t13 # if it already exists under someone else's role, see below + ``` + + Two gotchas on a shared VM, both safe to fix since `t13` has no + production data to lose: + + - **If `t13` already exists under another teammate's role**, you won't + have `CREATE` on its `public` schema (Postgres 15+ default) and + `npm run import` fails with `permission denied for schema public`: + `sudo -u postgres psql -d t13 -c "GRANT ALL ON SCHEMA public TO ;"` + - **The backend's default `DATABASE_URL` (`postgres://localhost/t13`) + connects over TCP**, which needs a Postgres password this role doesn't + have (`createuser` makes a passwordless role) and fails with a cryptic + `SASL: SCRAM-SERVER-FIRST-MESSAGE` error. Connect over the local unix + socket instead (peer-authenticated as your OS user, no password), in + `backend/.env`: + `DATABASE_URL=postgresql:///t13?host=/var/run/postgresql` + +2. Node.js >= 22.18, **via [nvm](https://github.com/nvm-sh/nvm)**, not + `apt install nodejs`: Ubuntu's `nodejs` package is built without + TypeScript type-stripping support (run + `node -e "console.log(process.config.variables.node_use_amaro)"` to + check — `false` means it's missing), which this backend's no-build-step + setup depends on, and it doesn't include `npm` either. + + ```bash + curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.1/install.sh | bash + # reopen the shell, or: export NVM_DIR="$HOME/.nvm" && . "$NVM_DIR/nvm.sh" + nvm install 22 + ``` + +3. Clone the repo on the VM. The VM has no other access to git.ntnu.no, so + add an SSH key for it first — either a repo-scoped read-only [deploy + key](../../settings/keys) (needs someone with admin on the repo) or, if + that's not available, a personal key added to your own account's [SSH + keys](https://git.ntnu.no/settings/keys) (`gh ssh-key add`), scoped to + just this clone with `core.sshCommand`: + + ```bash + ssh-keygen -t ed25519 -f ~/.ssh/t13_deploy_key -N "" -C "t13-project2-vm-deploy" + cat ~/.ssh/t13_deploy_key.pub # add this as a deploy key or personal SSH key, see above + + git clone git@git.ntnu.no:IT2810-H26/T13-Project-2.git + cd T13-Project-2 + git config core.sshCommand 'ssh -i ~/.ssh/t13_deploy_key -o IdentitiesOnly=yes' + npm ci + npm run import + ``` + + This downloads both Kaggle datasets itself and loads Postgres; nothing + needs to be copied over separately. It must report **17,698 recipes, + 5,653 with photo** — `backend/scripts/import.ts` checks the zips' SHA-256 + and these counts itself and fails loudly if either dataset has changed + since this script was written, rather than silently importing different + data. + + A verified copy of both raw zips (~228 MB) is backed up outside the repo + at `~davidhne/dataset-backup/` on the VM, in case Kaggle ever removes or + changes them; `npm run import` doesn't read from there automatically, so + copy them into `data/raw/` first if Kaggle is unreachable. + +4. Install and start the backend as a systemd service, so it survives + reboots and restarts if it crashes. Edit `User`, `WorkingDirectory` and + the `ExecStart` node path in the unit file first if your clone isn't + under `davidhne`'s home directory, or your nvm Node is a different + version (`nvm which 22` prints the exact path): + + ```bash + sudo cp deploy/systemd/t13-backend.service /etc/systemd/system/ + sudo systemctl daemon-reload + sudo systemctl enable --now t13-backend + sudo systemctl status t13-backend # check it's "active (running)" + ``` + + To deploy a backend change later: `git pull`, then + `sudo systemctl restart t13-backend`. + +### Frontend The frontend is served by Apache on the VM from `/var/www/html/project2` (http://it2810-13.idi.ntnu.no/project2/). @@ -196,17 +286,36 @@ The frontend is served by Apache on the VM from `/var/www/html/project2` ``` 2. Once per VM: install the Apache config from - [`deploy/apache/project2.conf`](deploy/apache/project2.conf). Without it, - reloading any page other than the home page (e.g. `/project2/recipes/2`) - gives Apache's 404, because those routes exist only in React Router. On the - VM, from a clone of this repo: + [`deploy/apache/project2.conf`](deploy/apache/project2.conf). It serves + `index.html` for any React Router route so reloading e.g. + `/project2/recipes/2` doesn't 404, proxies `/project2/graphql` to the + backend on port 3001 so the browser never has to reach that port directly + (same origin as the frontend, so no CORS either), and serves + `data/images/` (the recipe photos `npm run import` builds) with a long + cache lifetime. Edit the hardcoded paths in the file first if the repo + isn't cloned under `davidhne`'s home directory. On the VM, from a clone of + this repo: ```bash + sudo a2enmod proxy proxy_http expires sudo cp deploy/apache/project2.conf /etc/apache2/conf-available/ sudo a2enconf project2 sudo apache2ctl configtest sudo systemctl reload apache2 ``` - Check: `curl -o /dev/null -w "%{http_code}\n" http://it2810-13.idi.ntnu.no/project2/fridge` - prints `200`. + If the repo is cloned under a home directory (as above), Apache + (`www-data`) also needs permission to traverse into it to reach + `data/images/` — a bare home directory is usually `drwx------`, which + gives `403 Forbidden` even though the files underneath are readable: + + ```bash + chmod o+x ~ # adds traverse, not read/list, on the home dir itself + ``` + + Check: + `curl -o /dev/null -w "%{http_code}\n" http://it2810-13.idi.ntnu.no/project2/fridge` + and + `curl -o /dev/null -w "%{http_code}\n" http://it2810-13.idi.ntnu.no/project2/graphql` + (GET on the GraphQL endpoint; Apollo Server answers `400`, not `502`/`404`) + both print something other than Apache's default 404. diff --git a/backend/scripts/import.ts b/backend/scripts/import.ts index 2350787..c780939 100644 --- a/backend/scripts/import.ts +++ b/backend/scripts/import.ts @@ -9,10 +9,11 @@ import { Client } from 'pg' import sharp from 'sharp' import { parse } from 'csv-parse/sync' import { execFileSync } from 'node:child_process' +import { createHash } from 'node:crypto' import { Readable } from 'node:stream' import type { ReadableStream as NodeWebReadableStream } from 'node:stream/web' import { pipeline } from 'node:stream/promises' -import { createWriteStream } from 'node:fs' +import { createReadStream, createWriteStream } from 'node:fs' import fs from 'node:fs/promises' import path from 'node:path' import { buildCatalog } from './transform.ts' @@ -29,10 +30,17 @@ const foodDir = path.join(rawDir, 'food') const imagesDir = path.join(repoRoot, 'data/images') const foodImagesDir = path.join(foodDir, 'Food Images', 'Food Images') +// Both datasets are downloaded fresh from Kaggle on every machine that runs +// `npm run import` (the zips are gitignored, see README.md "Data"). Kaggle +// hosts them, not us, so a changed or removed dataset would otherwise give a +// different database with no warning. The recorded hash pins the exact bytes +// we built and tested against; the recipe/photo counts below pin the content +// after parsing, in case Kaggle ever re-zips the same files differently. const DOWNLOADS = [ { url: 'https://www.kaggle.com/api/v1/datasets/download/hugodarwood/epirecipes', dest: path.join(rawDir, 'epirecipes.zip'), + sha256: '2b52b8b5ab6683d78eae5644376eee90f91e58c707f4156efb50af7a89668672', }, { url: 'https://www.kaggle.com/api/v1/datasets/download/pes12017000148/food-ingredients-and-recipe-dataset-with-images', @@ -40,9 +48,13 @@ const DOWNLOADS = [ rawDir, 'food-ingredients-and-recipe-dataset-with-images.zip', ), + sha256: 'ef958f0e2bf38576c27550f2dcd35ad44aa64323852942ee454d624760785ea0', }, ] +const EXPECTED_RECIPE_COUNT = 17698 +const EXPECTED_PHOTO_COUNT = 5653 + async function exists(p: string): Promise { try { await fs.access(p) @@ -52,27 +64,48 @@ async function exists(p: string): Promise { } } -async function downloadIfMissing(url: string, dest: string) { +async function fileSha256(filePath: string): Promise { + const hash = createHash('sha256') + await pipeline(createReadStream(filePath), hash) + return hash.digest('hex') +} + +async function downloadIfMissing(url: string, dest: string, sha256: string) { if (await exists(dest)) { - console.log(`have ${path.basename(dest)}, skipping download`) - return + console.log(`have ${path.basename(dest)}, verifying checksum`) + } else { + console.log(`downloading ${path.basename(dest)}`) + const res = await fetch(url) + if (!res.ok || !res.body) { + throw new Error( + `download failed for ${url}: ${res.status} ${res.statusText}`, + ) + } + await fs.mkdir(path.dirname(dest), { recursive: true }) + // fetch's global ReadableStream and stream/web's don't structurally match in TS. + // Write to .part and rename when done, so an interrupted download isn't + // mistaken for a complete one on the next run. + await pipeline( + Readable.fromWeb( + res.body as unknown as NodeWebReadableStream, + ), + createWriteStream(`${dest}.part`), + ) + await fs.rename(`${dest}.part`, dest) } - console.log(`downloading ${path.basename(dest)}`) - const res = await fetch(url) - if (!res.ok || !res.body) { + + // Checked on every run (not only right after downloading), so a zip that + // got swapped out on Kaggle or corrupted on disk is caught before it's + // extracted and silently gives a different catalog. + const actual = await fileSha256(dest) + if (actual !== sha256) { throw new Error( - `download failed for ${url}: ${res.status} ${res.statusText}`, + `${path.basename(dest)} checksum mismatch: expected ${sha256}, got ${actual}. ` + + `The Kaggle dataset has likely changed since this script was written. Inspect ` + + `the new file before trusting it, then update the recorded hash (and the ` + + `expected recipe/photo counts below) in import.ts if the change is intentional.`, ) } - await fs.mkdir(path.dirname(dest), { recursive: true }) - // fetch's global ReadableStream and stream/web's don't structurally match in TS. - // Write to .part and rename when done, so an interrupted download isn't - // mistaken for a complete one on the next run. - await pipeline( - Readable.fromWeb(res.body as unknown as NodeWebReadableStream), - createWriteStream(`${dest}.part`), - ) - await fs.rename(`${dest}.part`, dest) } async function extractIfMissing(zipPath: string, destDir: string) { @@ -87,8 +120,8 @@ async function extractIfMissing(zipPath: string, destDir: string) { execFileSync('unzip', ['-oq', zipPath, '-d', destDir]) } -for (const { url, dest } of DOWNLOADS) { - await downloadIfMissing(url, dest) +for (const { url, dest, sha256 } of DOWNLOADS) { + await downloadIfMissing(url, dest, sha256) } await extractIfMissing(path.join(rawDir, 'epirecipes.zip'), epiDir) @@ -120,6 +153,23 @@ const images: ImageRaw[] = csvRows.map((row) => ({ console.log('building catalog') const { recipes, report } = buildCatalog(epi, images) +// The zip checksums pin the exact bytes, but Kaggle could in principle +// re-zip the same files with different internal formatting. Pin the +// parsed content too, so a silently different catalog is never mistaken +// for a successful import. +const withPhotoCount = recipes.filter((r) => r.imageName).length +if ( + recipes.length !== EXPECTED_RECIPE_COUNT || + withPhotoCount !== EXPECTED_PHOTO_COUNT +) { + throw new Error( + `dataset content has changed: expected ${EXPECTED_RECIPE_COUNT} recipes ` + + `(${EXPECTED_PHOTO_COUNT} with a photo), got ${recipes.length} ` + + `(${withPhotoCount} with a photo). Verify the new data before updating ` + + `the expected counts in import.ts.`, + ) +} + console.log('converting images') await fs.rm(imagesDir, { recursive: true, force: true }) await fs.mkdir(imagesDir, { recursive: true }) diff --git a/deploy/apache/project2.conf b/deploy/apache/project2.conf index df56af2..722cbd9 100644 --- a/deploy/apache/project2.conf +++ b/deploy/apache/project2.conf @@ -1,5 +1,5 @@ -# Apache config for the frontend at /project2 on the group VM. -# Install: see README.md "Deploying the frontend". +# Apache config for the frontend and API at /project2 on the group VM. +# Install: see README.md "Deployment". # # Routes like /project2/recipes/2 exist only in React Router, not as files. # Without a fallback, reloading such a page gives Apache's 404. FallbackResource @@ -9,3 +9,23 @@ FallbackResource /project2/index.html + +# The browser can only reach the backend through Apache (port 3001 is not +# exposed), and proxying it under the same origin as the frontend avoids +# CORS entirely. Needs: sudo a2enmod proxy proxy_http + + ProxyPass http://localhost:3001/ + ProxyPassReverse http://localhost:3001/ + + +# Recipe photos built by `npm run import` (backend/scripts/import.ts), served +# directly rather than through Node since they're static files. Long cache +# lifetime because filenames are stable recipe IDs that never change content. +# Needs: sudo a2enmod expires. Edit the path below if the repo isn't cloned +# under davidhne's home directory. +Alias /project2/images /home/davidhne/T13-Project-2/data/images + + Require all granted + ExpiresActive On + ExpiresDefault "access plus 1 year" + diff --git a/deploy/systemd/t13-backend.service b/deploy/systemd/t13-backend.service new file mode 100644 index 0000000..c5e0626 --- /dev/null +++ b/deploy/systemd/t13-backend.service @@ -0,0 +1,27 @@ +# systemd unit for the GraphQL backend on the group VM. +# Install: see README.md "Deployment". Edit WorkingDirectory, User and the +# node path below for the account the repo is actually cloned under before +# installing. +# +# Calls node directly instead of `npm start`: the backend needs no build step +# (Node's type stripping runs the .ts files as-is). Uses an nvm-installed +# node, not /usr/bin/node or /usr/bin/npm: the VM's apt `nodejs` package is +# built without TypeScript type-stripping support (amaro), which this +# backend depends on, and doesn't include npm either. Installed with: +# curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.1/install.sh | bash +# nvm install 22 +[Unit] +Description=T13 Project 2 GraphQL backend +After=network.target postgresql.service +Wants=postgresql.service + +[Service] +Type=simple +User=davidhne +WorkingDirectory=/home/davidhne/T13-Project-2/backend +ExecStart=/home/davidhne/.nvm/versions/node/v22.23.3/bin/node --env-file-if-exists=.env src/index.ts +Restart=on-failure +RestartSec=5 + +[Install] +WantedBy=multi-user.target