From 5aad9c580979d0f29c048111153fc7d0712f1892 Mon Sep 17 00:00:00 2001 From: dhn06 Date: Thu, 15 Oct 2026 11:21:16 +0200 Subject: [PATCH 1/4] chore(vm): protect import against dataset drift, add backend systemd+proxy - import.ts now records the SHA-256 of both Kaggle zips and the expected recipe/photo counts (17698 / 5653), and fails loudly instead of silently importing different data if either changes - deploy/apache/project2.conf: proxy /project2/graphql to the backend and serve data/images/ with long cache headers - deploy/systemd/t13-backend.service: run the backend as a systemd service - README: Deployment section documenting the remaining manual VM steps (Postgres role, Node/npm, cloning, systemd, Apache) Part of #14; the VM provisioning itself (sudo-gated steps) is still manual. Co-Authored-By: Claude Sonnet 5 --- README.md | 81 ++++++++++++++++++++++++--- backend/scripts/import.ts | 88 +++++++++++++++++++++++------- deploy/apache/project2.conf | 24 +++++++- deploy/systemd/t13-backend.service | 22 ++++++++ 4 files changed, 187 insertions(+), 28 deletions(-) create mode 100644 deploy/systemd/t13-backend.service diff --git a/README.md b/README.md index 7f60b44..2a05c43 100644 --- a/README.md +++ b/README.md @@ -183,7 +183,64 @@ parameters, timing and query plan, plus client events like search debouncing. It talks to Postgres directly and is a development tool only, not part of the app. For raw SQL: `psql -d t13`. -## Deploying the frontend +## Deployment + +Everything below runs directly on the group VM (`it2810-13.idi.ntnu.no`), no +Docker. `sudo` is required for the one-time steps and is only available to +members who were given it on the VM. + +### Database and backend (once per VM, plus whenever the backend changes) + +1. PostgreSQL and `unzip` (skip if already installed): + + ```bash + sudo apt install postgresql unzip + sudo -u postgres createuser --createdb + createdb t13 + ``` + +2. Node.js >= 22.18. The VM's `nodejs` package does not pull in `npm`, so + install it too: + + ```bash + sudo apt install nodejs npm + ``` + +3. Clone the repo on the VM (needs a deploy key added to this repo's "Deploy + keys", since the VM has no other access to git.ntnu.no) and build the + database: + + ```bash + git clone git@git.ntnu.no:IT2810-H26/T13-Project-2.git + cd T13-Project-2 + npm ci + npm run import + ``` + + This downloads both Kaggle datasets itself and loads Postgres; nothing + needs to be copied over separately. It must report **17,698 recipes, + 5,653 with photo** — `backend/scripts/import.ts` checks the zips' SHA-256 + and these counts itself and fails loudly if either dataset has changed + since this script was written, rather than silently importing different + data. Also keep a copy of the raw zips (`data/raw/*.zip`, ~228 MB) + somewhere outside Kaggle (e.g. a backup folder on the VM outside the repo, + or the team's shared drive) in case Kaggle ever removes or changes them. + +4. Install and start the backend as a systemd service, so it survives + reboots and restarts if it crashes. Edit `User` and `WorkingDirectory` in + the unit file first if your clone isn't under `davidhne`'s home directory: + + ```bash + sudo cp deploy/systemd/t13-backend.service /etc/systemd/system/ + sudo systemctl daemon-reload + sudo systemctl enable --now t13-backend + sudo systemctl status t13-backend # check it's "active (running)" + ``` + + To deploy a backend change later: `git pull`, then + `sudo systemctl restart t13-backend`. + +### Frontend The frontend is served by Apache on the VM from `/var/www/html/project2` (http://it2810-13.idi.ntnu.no/project2/). @@ -196,17 +253,27 @@ The frontend is served by Apache on the VM from `/var/www/html/project2` ``` 2. Once per VM: install the Apache config from - [`deploy/apache/project2.conf`](deploy/apache/project2.conf). Without it, - reloading any page other than the home page (e.g. `/project2/recipes/2`) - gives Apache's 404, because those routes exist only in React Router. On the - VM, from a clone of this repo: + [`deploy/apache/project2.conf`](deploy/apache/project2.conf). It serves + `index.html` for any React Router route so reloading e.g. + `/project2/recipes/2` doesn't 404, proxies `/project2/graphql` to the + backend on port 3001 so the browser never has to reach that port directly + (same origin as the frontend, so no CORS either), and serves + `data/images/` (the recipe photos `npm run import` builds) with a long + cache lifetime. Edit the hardcoded paths in the file first if the repo + isn't cloned under `davidhne`'s home directory. On the VM, from a clone of + this repo: ```bash + sudo a2enmod proxy proxy_http expires sudo cp deploy/apache/project2.conf /etc/apache2/conf-available/ sudo a2enconf project2 sudo apache2ctl configtest sudo systemctl reload apache2 ``` - Check: `curl -o /dev/null -w "%{http_code}\n" http://it2810-13.idi.ntnu.no/project2/fridge` - prints `200`. + Check: + `curl -o /dev/null -w "%{http_code}\n" http://it2810-13.idi.ntnu.no/project2/fridge` + and + `curl -o /dev/null -w "%{http_code}\n" http://it2810-13.idi.ntnu.no/project2/graphql` + (GET on the GraphQL endpoint; Apollo Server answers `400`, not `502`/`404`) + both print something other than Apache's default 404. diff --git a/backend/scripts/import.ts b/backend/scripts/import.ts index 2350787..c780939 100644 --- a/backend/scripts/import.ts +++ b/backend/scripts/import.ts @@ -9,10 +9,11 @@ import { Client } from 'pg' import sharp from 'sharp' import { parse } from 'csv-parse/sync' import { execFileSync } from 'node:child_process' +import { createHash } from 'node:crypto' import { Readable } from 'node:stream' import type { ReadableStream as NodeWebReadableStream } from 'node:stream/web' import { pipeline } from 'node:stream/promises' -import { createWriteStream } from 'node:fs' +import { createReadStream, createWriteStream } from 'node:fs' import fs from 'node:fs/promises' import path from 'node:path' import { buildCatalog } from './transform.ts' @@ -29,10 +30,17 @@ const foodDir = path.join(rawDir, 'food') const imagesDir = path.join(repoRoot, 'data/images') const foodImagesDir = path.join(foodDir, 'Food Images', 'Food Images') +// Both datasets are downloaded fresh from Kaggle on every machine that runs +// `npm run import` (the zips are gitignored, see README.md "Data"). Kaggle +// hosts them, not us, so a changed or removed dataset would otherwise give a +// different database with no warning. The recorded hash pins the exact bytes +// we built and tested against; the recipe/photo counts below pin the content +// after parsing, in case Kaggle ever re-zips the same files differently. const DOWNLOADS = [ { url: 'https://www.kaggle.com/api/v1/datasets/download/hugodarwood/epirecipes', dest: path.join(rawDir, 'epirecipes.zip'), + sha256: '2b52b8b5ab6683d78eae5644376eee90f91e58c707f4156efb50af7a89668672', }, { url: 'https://www.kaggle.com/api/v1/datasets/download/pes12017000148/food-ingredients-and-recipe-dataset-with-images', @@ -40,9 +48,13 @@ const DOWNLOADS = [ rawDir, 'food-ingredients-and-recipe-dataset-with-images.zip', ), + sha256: 'ef958f0e2bf38576c27550f2dcd35ad44aa64323852942ee454d624760785ea0', }, ] +const EXPECTED_RECIPE_COUNT = 17698 +const EXPECTED_PHOTO_COUNT = 5653 + async function exists(p: string): Promise { try { await fs.access(p) @@ -52,27 +64,48 @@ async function exists(p: string): Promise { } } -async function downloadIfMissing(url: string, dest: string) { +async function fileSha256(filePath: string): Promise { + const hash = createHash('sha256') + await pipeline(createReadStream(filePath), hash) + return hash.digest('hex') +} + +async function downloadIfMissing(url: string, dest: string, sha256: string) { if (await exists(dest)) { - console.log(`have ${path.basename(dest)}, skipping download`) - return + console.log(`have ${path.basename(dest)}, verifying checksum`) + } else { + console.log(`downloading ${path.basename(dest)}`) + const res = await fetch(url) + if (!res.ok || !res.body) { + throw new Error( + `download failed for ${url}: ${res.status} ${res.statusText}`, + ) + } + await fs.mkdir(path.dirname(dest), { recursive: true }) + // fetch's global ReadableStream and stream/web's don't structurally match in TS. + // Write to .part and rename when done, so an interrupted download isn't + // mistaken for a complete one on the next run. + await pipeline( + Readable.fromWeb( + res.body as unknown as NodeWebReadableStream, + ), + createWriteStream(`${dest}.part`), + ) + await fs.rename(`${dest}.part`, dest) } - console.log(`downloading ${path.basename(dest)}`) - const res = await fetch(url) - if (!res.ok || !res.body) { + + // Checked on every run (not only right after downloading), so a zip that + // got swapped out on Kaggle or corrupted on disk is caught before it's + // extracted and silently gives a different catalog. + const actual = await fileSha256(dest) + if (actual !== sha256) { throw new Error( - `download failed for ${url}: ${res.status} ${res.statusText}`, + `${path.basename(dest)} checksum mismatch: expected ${sha256}, got ${actual}. ` + + `The Kaggle dataset has likely changed since this script was written. Inspect ` + + `the new file before trusting it, then update the recorded hash (and the ` + + `expected recipe/photo counts below) in import.ts if the change is intentional.`, ) } - await fs.mkdir(path.dirname(dest), { recursive: true }) - // fetch's global ReadableStream and stream/web's don't structurally match in TS. - // Write to .part and rename when done, so an interrupted download isn't - // mistaken for a complete one on the next run. - await pipeline( - Readable.fromWeb(res.body as unknown as NodeWebReadableStream), - createWriteStream(`${dest}.part`), - ) - await fs.rename(`${dest}.part`, dest) } async function extractIfMissing(zipPath: string, destDir: string) { @@ -87,8 +120,8 @@ async function extractIfMissing(zipPath: string, destDir: string) { execFileSync('unzip', ['-oq', zipPath, '-d', destDir]) } -for (const { url, dest } of DOWNLOADS) { - await downloadIfMissing(url, dest) +for (const { url, dest, sha256 } of DOWNLOADS) { + await downloadIfMissing(url, dest, sha256) } await extractIfMissing(path.join(rawDir, 'epirecipes.zip'), epiDir) @@ -120,6 +153,23 @@ const images: ImageRaw[] = csvRows.map((row) => ({ console.log('building catalog') const { recipes, report } = buildCatalog(epi, images) +// The zip checksums pin the exact bytes, but Kaggle could in principle +// re-zip the same files with different internal formatting. Pin the +// parsed content too, so a silently different catalog is never mistaken +// for a successful import. +const withPhotoCount = recipes.filter((r) => r.imageName).length +if ( + recipes.length !== EXPECTED_RECIPE_COUNT || + withPhotoCount !== EXPECTED_PHOTO_COUNT +) { + throw new Error( + `dataset content has changed: expected ${EXPECTED_RECIPE_COUNT} recipes ` + + `(${EXPECTED_PHOTO_COUNT} with a photo), got ${recipes.length} ` + + `(${withPhotoCount} with a photo). Verify the new data before updating ` + + `the expected counts in import.ts.`, + ) +} + console.log('converting images') await fs.rm(imagesDir, { recursive: true, force: true }) await fs.mkdir(imagesDir, { recursive: true }) diff --git a/deploy/apache/project2.conf b/deploy/apache/project2.conf index df56af2..722cbd9 100644 --- a/deploy/apache/project2.conf +++ b/deploy/apache/project2.conf @@ -1,5 +1,5 @@ -# Apache config for the frontend at /project2 on the group VM. -# Install: see README.md "Deploying the frontend". +# Apache config for the frontend and API at /project2 on the group VM. +# Install: see README.md "Deployment". # # Routes like /project2/recipes/2 exist only in React Router, not as files. # Without a fallback, reloading such a page gives Apache's 404. FallbackResource @@ -9,3 +9,23 @@ FallbackResource /project2/index.html + +# The browser can only reach the backend through Apache (port 3001 is not +# exposed), and proxying it under the same origin as the frontend avoids +# CORS entirely. Needs: sudo a2enmod proxy proxy_http + + ProxyPass http://localhost:3001/ + ProxyPassReverse http://localhost:3001/ + + +# Recipe photos built by `npm run import` (backend/scripts/import.ts), served +# directly rather than through Node since they're static files. Long cache +# lifetime because filenames are stable recipe IDs that never change content. +# Needs: sudo a2enmod expires. Edit the path below if the repo isn't cloned +# under davidhne's home directory. +Alias /project2/images /home/davidhne/T13-Project-2/data/images + + Require all granted + ExpiresActive On + ExpiresDefault "access plus 1 year" + diff --git a/deploy/systemd/t13-backend.service b/deploy/systemd/t13-backend.service new file mode 100644 index 0000000..9cec447 --- /dev/null +++ b/deploy/systemd/t13-backend.service @@ -0,0 +1,22 @@ +# systemd unit for the GraphQL backend on the group VM. +# Install: see README.md "Deployment". Edit WorkingDirectory and User for the +# account the repo is actually cloned under before installing. +# +# Calls node directly instead of `npm start`: the backend needs no build step +# (Node's type stripping runs the .ts files as-is), and npm itself is not +# installed alongside the VM's nodejs package, only the node binary. +[Unit] +Description=T13 Project 2 GraphQL backend +After=network.target postgresql.service +Wants=postgresql.service + +[Service] +Type=simple +User=davidhne +WorkingDirectory=/home/davidhne/T13-Project-2/backend +ExecStart=/usr/bin/node --env-file-if-exists=.env src/index.ts +Restart=on-failure +RestartSec=5 + +[Install] +WantedBy=multi-user.target From 955711b086abaa90b7f4cbdc52bef3b1d2017868 Mon Sep 17 00:00:00 2001 From: dhn06 Date: Thu, 15 Oct 2026 11:22:03 +0200 Subject: [PATCH 2/4] docs(vm): point to the dataset zip backup on the VM Co-Authored-By: Claude Sonnet 5 --- README.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 2a05c43..be8f2ef 100644 --- a/README.md +++ b/README.md @@ -222,9 +222,12 @@ members who were given it on the VM. 5,653 with photo** — `backend/scripts/import.ts` checks the zips' SHA-256 and these counts itself and fails loudly if either dataset has changed since this script was written, rather than silently importing different - data. Also keep a copy of the raw zips (`data/raw/*.zip`, ~228 MB) - somewhere outside Kaggle (e.g. a backup folder on the VM outside the repo, - or the team's shared drive) in case Kaggle ever removes or changes them. + data. + + A verified copy of both raw zips (~228 MB) is backed up outside the repo + at `~davidhne/dataset-backup/` on the VM, in case Kaggle ever removes or + changes them; `npm run import` doesn't read from there automatically, so + copy them into `data/raw/` first if Kaggle is unreachable. 4. Install and start the backend as a systemd service, so it survives reboots and restarts if it crashes. Edit `User` and `WorkingDirectory` in From ea5e6ae2b885c5b2ebef98affe318396142ab8ee Mon Sep 17 00:00:00 2001 From: dhn06 Date: Thu, 15 Oct 2026 11:45:43 +0200 Subject: [PATCH 3/4] chore(vm): use nvm Node instead of apt's (no TypeScript stripping support) The VM's apt `nodejs` package is built without amaro (TypeScript type-stripping), which the no-build-step backend depends on, and has no npm either. Verified on the VM: node_use_amaro is false for apt's 22.22.1, true for nvm's 22.23.3. Point the systemd unit and README at an nvm-installed Node instead. Also corrects the clone step: a repo-scoped deploy key needs admin on the repo, which isn't always available, so document the personal-SSH-key fallback actually used to bootstrap the VM clone. Co-Authored-By: Claude Sonnet 5 --- README.md | 31 ++++++++++++++++++++++-------- deploy/systemd/t13-backend.service | 15 ++++++++++----- 2 files changed, 33 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index be8f2ef..80b0f8a 100644 --- a/README.md +++ b/README.md @@ -199,20 +199,33 @@ members who were given it on the VM. createdb t13 ``` -2. Node.js >= 22.18. The VM's `nodejs` package does not pull in `npm`, so - install it too: +2. Node.js >= 22.18, **via [nvm](https://github.com/nvm-sh/nvm)**, not + `apt install nodejs`: Ubuntu's `nodejs` package is built without + TypeScript type-stripping support (`node -e +"console.log(process.config.variables.node_use_amaro)"` prints `false`), + which this backend's no-build-step setup depends on, and it doesn't + include `npm` either. ```bash - sudo apt install nodejs npm + curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.1/install.sh | bash + # reopen the shell, or: export NVM_DIR="$HOME/.nvm" && . "$NVM_DIR/nvm.sh" + nvm install 22 ``` -3. Clone the repo on the VM (needs a deploy key added to this repo's "Deploy - keys", since the VM has no other access to git.ntnu.no) and build the - database: +3. Clone the repo on the VM. The VM has no other access to git.ntnu.no, so + add an SSH key for it first — either a repo-scoped read-only [deploy + key](../../settings/keys) (needs someone with admin on the repo) or, if + that's not available, a personal key added to your own account's [SSH + keys](https://git.ntnu.no/settings/keys) (`gh ssh-key add`), scoped to + just this clone with `core.sshCommand`: ```bash + ssh-keygen -t ed25519 -f ~/.ssh/t13_deploy_key -N "" -C "t13-project2-vm-deploy" + cat ~/.ssh/t13_deploy_key.pub # add this as a deploy key or personal SSH key, see above + git clone git@git.ntnu.no:IT2810-H26/T13-Project-2.git cd T13-Project-2 + git config core.sshCommand 'ssh -i ~/.ssh/t13_deploy_key -o IdentitiesOnly=yes' npm ci npm run import ``` @@ -230,8 +243,10 @@ members who were given it on the VM. copy them into `data/raw/` first if Kaggle is unreachable. 4. Install and start the backend as a systemd service, so it survives - reboots and restarts if it crashes. Edit `User` and `WorkingDirectory` in - the unit file first if your clone isn't under `davidhne`'s home directory: + reboots and restarts if it crashes. Edit `User`, `WorkingDirectory` and + the `ExecStart` node path in the unit file first if your clone isn't + under `davidhne`'s home directory, or your nvm Node is a different + version (`nvm which 22` prints the exact path): ```bash sudo cp deploy/systemd/t13-backend.service /etc/systemd/system/ diff --git a/deploy/systemd/t13-backend.service b/deploy/systemd/t13-backend.service index 9cec447..c5e0626 100644 --- a/deploy/systemd/t13-backend.service +++ b/deploy/systemd/t13-backend.service @@ -1,10 +1,15 @@ # systemd unit for the GraphQL backend on the group VM. -# Install: see README.md "Deployment". Edit WorkingDirectory and User for the -# account the repo is actually cloned under before installing. +# Install: see README.md "Deployment". Edit WorkingDirectory, User and the +# node path below for the account the repo is actually cloned under before +# installing. # # Calls node directly instead of `npm start`: the backend needs no build step -# (Node's type stripping runs the .ts files as-is), and npm itself is not -# installed alongside the VM's nodejs package, only the node binary. +# (Node's type stripping runs the .ts files as-is). Uses an nvm-installed +# node, not /usr/bin/node or /usr/bin/npm: the VM's apt `nodejs` package is +# built without TypeScript type-stripping support (amaro), which this +# backend depends on, and doesn't include npm either. Installed with: +# curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.1/install.sh | bash +# nvm install 22 [Unit] Description=T13 Project 2 GraphQL backend After=network.target postgresql.service @@ -14,7 +19,7 @@ Wants=postgresql.service Type=simple User=davidhne WorkingDirectory=/home/davidhne/T13-Project-2/backend -ExecStart=/usr/bin/node --env-file-if-exists=.env src/index.ts +ExecStart=/home/davidhne/.nvm/versions/node/v22.23.3/bin/node --env-file-if-exists=.env src/index.ts Restart=on-failure RestartSec=5 From 67a55ba9d4942b41f024b43460e5103040ed8b8c Mon Sep 17 00:00:00 2001 From: dhn06 Date: Thu, 15 Oct 2026 11:54:34 +0200 Subject: [PATCH 4/4] docs(vm): document the deployment gotchas found while bringing #14 up All discovered while actually bringing the VM up end-to-end: - shared t13 database owned by another role -> GRANT ALL ON SCHEMA public - default DATABASE_URL needs a password this role doesn't have -> connect over the unix socket instead (peer auth) - a bare home directory blocks Apache from traversing into data/images/ (403) even though the files underneath are readable -> chmod o+x ~ Co-Authored-By: Claude Sonnet 5 --- README.md | 34 +++++++++++++++++++++++++++++----- 1 file changed, 29 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 80b0f8a..da9eb6d 100644 --- a/README.md +++ b/README.md @@ -196,15 +196,30 @@ members who were given it on the VM. ```bash sudo apt install postgresql unzip sudo -u postgres createuser --createdb - createdb t13 + createdb t13 # if it already exists under someone else's role, see below ``` + Two gotchas on a shared VM, both safe to fix since `t13` has no + production data to lose: + + - **If `t13` already exists under another teammate's role**, you won't + have `CREATE` on its `public` schema (Postgres 15+ default) and + `npm run import` fails with `permission denied for schema public`: + `sudo -u postgres psql -d t13 -c "GRANT ALL ON SCHEMA public TO ;"` + - **The backend's default `DATABASE_URL` (`postgres://localhost/t13`) + connects over TCP**, which needs a Postgres password this role doesn't + have (`createuser` makes a passwordless role) and fails with a cryptic + `SASL: SCRAM-SERVER-FIRST-MESSAGE` error. Connect over the local unix + socket instead (peer-authenticated as your OS user, no password), in + `backend/.env`: + `DATABASE_URL=postgresql:///t13?host=/var/run/postgresql` + 2. Node.js >= 22.18, **via [nvm](https://github.com/nvm-sh/nvm)**, not `apt install nodejs`: Ubuntu's `nodejs` package is built without - TypeScript type-stripping support (`node -e -"console.log(process.config.variables.node_use_amaro)"` prints `false`), - which this backend's no-build-step setup depends on, and it doesn't - include `npm` either. + TypeScript type-stripping support (run + `node -e "console.log(process.config.variables.node_use_amaro)"` to + check — `false` means it's missing), which this backend's no-build-step + setup depends on, and it doesn't include `npm` either. ```bash curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.1/install.sh | bash @@ -289,6 +304,15 @@ The frontend is served by Apache on the VM from `/var/www/html/project2` sudo systemctl reload apache2 ``` + If the repo is cloned under a home directory (as above), Apache + (`www-data`) also needs permission to traverse into it to reach + `data/images/` — a bare home directory is usually `drwx------`, which + gives `403 Forbidden` even though the files underneath are readable: + + ```bash + chmod o+x ~ # adds traverse, not read/list, on the home dir itself + ``` + Check: `curl -o /dev/null -w "%{http_code}\n" http://it2810-13.idi.ntnu.no/project2/fridge` and