diff --git a/README.md b/README.md index 144b34b..40eeaef 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ We are building a web application that helps gymnastics coaches plan their gymna ## Status -The first delivery was a specification. There is no working application yet, but the catalog data has been extracted from the Code of Points and is in [`data/`](data/), and the database schema is in place: the Prisma schema and its migrations create all the tables of the data model in PostgreSQL. The seed script, the GraphQL server and the search are the next steps. +The first delivery was a specification. There is no working application yet, but the catalog data has been extracted from the Code of Points and is in [`data/`](data/), and the database schema is in place: the Prisma schema and its migrations create all the tables of the data model in PostgreSQL. The seed script loads `data/` into the database. The GraphQL server and the search are the next steps. The code in `src/` is the Vite starter project with one example test. The tooling is set up: TypeScript, ESLint, Prettier and Vitest. @@ -23,7 +23,7 @@ The code in `src/` is the Vite starter project with one example test. The toolin ## The dataset -The catalog data comes from the FIG Women's Artistic Gymnastics Code of Points 2025–2028, which is published as a PDF. We have extracted the apparatus, element groups, elements and vaults from the PDF with a Python script in [`scripts/extract-cop/`](scripts/extract-cop/), and the result is in [`data/`](data/). [docs/EXTRACTION.md](docs/EXTRACTION.md) explains how it was done and what has been checked by hand. The data will be loaded into the database with a seed script, which is not written yet. +The catalog data comes from the FIG Women's Artistic Gymnastics Code of Points 2025–2028, which is published as a PDF. We have extracted the apparatus, element groups, elements and vaults from the PDF with a Python script in [`scripts/extract-cop/`](scripts/extract-cop/), and the result is in [`data/`](data/). [docs/EXTRACTION.md](docs/EXTRACTION.md) explains how it was done and what has been checked by hand. The seed script in `backend/src/seed/` loads the data into the database (see "Seeding" below). The data is used for educational purposes. All rights to the Code of Points belong to FIG (Fédération Internationale de Gymnastique). @@ -63,13 +63,13 @@ In the repository now: - React, TypeScript and Vite for the frontend - A backend package in `backend/` with TypeScript and Prisma, set up as an npm workspace - PostgreSQL 18, installed directly on our own machines (not in Docker), with the Prisma schema and migrations for all the tables of the data model +- A seed script that loads the catalog from `data/` into the database - ESLint and Prettier for linting and formatting, for both the frontend and the backend - Vitest for testing Planned: - A GraphQL backend in Node and TypeScript -- A seed script that loads `data/` into the database - PostgreSQL 18 on the VM, set up the same way as on our own machines ### Why PostgreSQL @@ -173,6 +173,7 @@ From the repository root: npm ci npm run db:generate npm run db:migrate +npm run db:seed ``` `npm run db:migrate` creates the tables in your database by applying the migrations in `backend/prisma/migrations/`. Run it again whenever someone has added a new migration. @@ -182,6 +183,7 @@ npm run db:migrate | `npm run db:generate` | Generates the Prisma client from `backend/prisma/schema.prisma` | | `npm run db:migrate` | Creates a migration from changes in the schema and applies it to your database | | `npm run db:deploy` | Applies existing migrations without creating new ones. Used on the server | +| `npm run db:seed` | Loads the catalog from `data/` into the database. See "Seeding" below | | `npm run db:studio` | Opens Prisma Studio in the browser, to look at the data in the database | ### 5. Check the connection @@ -217,6 +219,43 @@ To check that the backend can reach your database: If the output says "Following migration have not yet been applied", the connection works but your tables are missing or out of date. Go back to the repository root and run `npm run db:migrate`. If the connection fails, Prisma prints an error such as "Can't reach database server" or "Authentication failed" instead. +### 6. Seeding + +The seed script loads the catalog (apparatus, element groups, variant groups, elements and vaults) from the JSON files in `data/` into your database, and rebuilds the search tables. Run it from the repository root: + +```bash +npm run db:seed +``` + +Run it again: + +- **after migrations**, when you have run `npm run db:migrate` or `npm run db:deploy` +- **whenever `data/` changes**, for example after a new run of the extraction script + +It is safe to run as often as you like: + +- **Everything happens in one transaction.** If something fails, nothing is changed. +- **A second run with the same data changes nothing.** It prints a table of what was inserted, updated, marked as removed and deleted. +- **It checks the data first.** If something in `data/` breaks a rule in `docs/DATA_MODEL.md`, for example a variant group with different codes or a missing image, it stops with a message that names the row and the rule, and changes nothing. +- **It never touches user data.** Coaches, gymnasts, routines and their lists are left alone. +- **Removed elements are kept while they are in use.** An element or vault that is no longer in `data/` is marked as removed. It is deleted only if no routine or gymnast list uses it. + +The code is in `backend/src/seed/`. The rules for search words, including the written-out abbreviations such as `handstand` for `hstd`, are in `backend/src/catalog/words.ts`. The search uses the same rules. + +### 7. Look at the data + +To see what is in your database, open Prisma Studio from the repository root: + +```bash +npm run db:studio +``` + +The terminal prints the address, for example "Prisma Studio is running at: http://localhost:51212". The port can change between runs. Open the address in your browser if it does not open by itself. The page has a list of all the tables. Click a table to see its rows, filter and sort them, and follow the relations, for example from an element to its variant group or from a routine to its items. Stop it with Ctrl+C in the terminal. + +Prisma Studio uses the connection in `backend/.env`, so there is nothing more to set up. You can also edit and delete rows there, so be careful: changes to the catalog are overwritten the next time you run `npm run db:seed`, and changes to user data are not undone. + +To write your own SQL queries, connect with `psql -h localhost -U cop -d cop` (on Windows, use _SQL Shell (psql)_) and enter the password from `backend/.env`. The tables have snake_case names in the database, such as `catalog_entries` and `element_group_id`. `\dt` lists the tables, `\d elements` shows the columns of a table, and `\q` quits. + ## Project structure | Folder | Contents | diff --git a/backend/package.json b/backend/package.json index 29582f7..dd0d4ac 100644 --- a/backend/package.json +++ b/backend/package.json @@ -8,6 +8,7 @@ "prisma:generate": "prisma generate", "prisma:migrate": "prisma migrate dev", "prisma:deploy": "prisma migrate deploy", + "prisma:seed": "prisma db seed", "prisma:studio": "prisma studio" }, "engines": { diff --git a/backend/prisma.config.ts b/backend/prisma.config.ts index 04dbf49..ff3dce0 100644 --- a/backend/prisma.config.ts +++ b/backend/prisma.config.ts @@ -7,6 +7,8 @@ export default defineConfig({ schema: 'prisma/schema.prisma', migrations: { path: 'prisma/migrations', + // `prisma db seed` (npm run db:seed) runs the seed script. + seed: 'tsx src/seed/seed.ts', }, datasource: { url: process.env['DATABASE_URL'], diff --git a/backend/src/catalog/abbreviations.test.ts b/backend/src/catalog/abbreviations.test.ts new file mode 100644 index 0000000..241cfcb --- /dev/null +++ b/backend/src/catalog/abbreviations.test.ts @@ -0,0 +1,67 @@ +// Checks that every abbreviation in the catalog descriptions is either written +// out (ABBREVIATIONS) or left out on purpose (ABBREVIATIONS_LEFT_OUT). +// +// A word in a description is treated as an abbreviation when it has at least +// two letters and: +// - is all uppercase, like "HB" and "LB", or +// - has no vowels (a, e, i, o, u, y), like "fwd", "bwd" and "hstd", or +// - is followed by a dot that does not end a sentence, like "sec." in +// "(2 sec.)" and "min." in "min. ¾ twist". A dot ends a sentence when it is +// the last character, or when the next word starts with an uppercase letter. +// Single letters such as the "L" in "L grip" are shapes, not abbreviations. + +import { readFileSync } from 'node:fs' +import { describe, expect, it } from 'vitest' +import { + ABBREVIATIONS, + ABBREVIATIONS_LEFT_OUT, + normalizeWords, +} from './words.js' + +type CatalogItem = { id: string; description: string } + +const dataDir = new URL('../../../data/', import.meta.url) +const read = (file: string): CatalogItem[] => + JSON.parse(readFileSync(new URL(file, dataDir), 'utf8')) +const items = [...read('elements.json'), ...read('vaults.json')] + +function findAbbreviations(description: string): string[] { + const found: string[] = [] + const text = description.trimEnd() + for (const match of text.matchAll(/[A-Za-z]+/g)) { + const word = match[0] + const end = match.index + word.length + const next = text.slice(end + 1).trimStart()[0] + const dotInside = + text[end] === '.' && next !== undefined && next === next.toLowerCase() + const isAbbreviation = + word.length >= 2 && + (word === word.toUpperCase() || !/[aeiouy]/i.test(word) || dotInside) + if (isAbbreviation) found.push(...normalizeWords(word)) + } + return found +} + +describe('abbreviations in the catalog descriptions', () => { + it('finds the abbreviations it is meant to find', () => { + expect(findAbbreviations('Swing fwd on HB to hstd (2 sec.)')).toEqual([ + 'fwd', + 'hb', + 'hstd', + 'sec', + ]) + expect(findAbbreviations('Salto in L hang. Then a tuck.')).toEqual([]) + }) + + it('are all written out or left out on purpose', () => { + const unknown = new Map() + for (const item of items) { + for (const word of findAbbreviations(item.description)) { + if (!(word in ABBREVIATIONS) && !(word in ABBREVIATIONS_LEFT_OUT)) + unknown.set(word, item.id) + } + } + // Shows each unknown abbreviation with an id where it is used. + expect(Object.fromEntries(unknown)).toEqual({}) + }) +}) diff --git a/backend/src/catalog/words.test.ts b/backend/src/catalog/words.test.ts new file mode 100644 index 0000000..41ed336 --- /dev/null +++ b/backend/src/catalog/words.test.ts @@ -0,0 +1,91 @@ +import { describe, expect, it } from 'vitest' +import { normalizeWords, searchWords } from './words.js' + +describe('normalizeWords', () => { + it('lowercases', () => { + expect(normalizeWords('Salto BWD Tucked')).toEqual([ + 'salto', + 'bwd', + 'tucked', + ]) + }) + + it('removes accents', () => { + expect(normalizeWords('Fouetté Röhm Šimić')).toEqual([ + 'fouette', + 'rohm', + 'simic', + ]) + }) + + it('keeps a code such as 3.304 as one word', () => { + expect(normalizeWords('3.304 and 1.22')).toEqual(['3.304', 'and', '1.22']) + }) + + it('removes punctuation', () => { + expect( + normalizeWords('Clear hip circle (Weiler-kip), hop-grip; “false-pop”.'), + ).toEqual([ + 'clear', + 'hip', + 'circle', + 'weiler', + 'kip', + 'hop', + 'grip', + 'false', + 'pop', + ]) + }) + + it('does not keep a dot at the end of a word', () => { + expect(normalizeWords('to hang on HB. (2 sec.)')).toEqual([ + 'to', + 'hang', + 'on', + 'hb', + '2', + 'sec', + ]) + }) + + it('splits fractions and degrees into numbers', () => { + expect(normalizeWords('½ turn (180°), 1/1 turn')).toEqual([ + '1', + '2', + 'turn', + '180', + '1', + '1', + 'turn', + ]) + }) + + it('returns no words for an empty text or only punctuation', () => { + expect(normalizeWords('')).toEqual([]) + expect(normalizeWords(' – (…) ')).toEqual([]) + }) + + it('never returns % or _, which are wildcards in LIKE', () => { + expect(normalizeWords('%sal_to%')).toEqual(['sal', 'to']) + }) +}) + +describe('searchWords', () => { + it('adds the written-out form of abbreviations', () => { + expect([...searchWords('Cast to hstd on LB')]).toEqual([ + 'cast', + 'to', + 'hstd', + 'handstand', + 'on', + 'lb', + 'low', + 'bar', + ]) + }) + + it('keeps each word once', () => { + expect([...searchWords('fwd and fwd')]).toEqual(['fwd', 'forward', 'and']) + }) +}) diff --git a/backend/src/catalog/words.ts b/backend/src/catalog/words.ts new file mode 100644 index 0000000..95e2c11 --- /dev/null +++ b/backend/src/catalog/words.ts @@ -0,0 +1,60 @@ +// The words the catalog search looks in. The seed script uses these functions to +// fill CatalogWord, and the search (issue #24) must use normalizeWords on the +// search text, so that both sides follow exactly the same rules. + +/** + * Splits a text into search words: lowercase, without accents, and without + * punctuation. A code such as "3.304" is kept as one word. + * + * normalizeWords('Clear hip circle to hstd (Weiler-kip), 1/1 turn') returns + * ['clear', 'hip', 'circle', 'to', 'hstd', 'weiler', 'kip', '1', '1', 'turn']. + */ +export function normalizeWords(text: string): string[] { + const plain = text + .normalize('NFKD') // "é" -> "e" + accent, "½" -> "1⁄2" + .replace(/\p{M}/gu, '') // remove the accents + .toLowerCase() + // A code: digits with dots between them. Otherwise: letters and digits. + return plain.match(/\d+(?:\.\d+)+|[a-z0-9]+/g) ?? [] +} + +/** + * Abbreviations used in the descriptions of the Code of Points, and the words + * they stand for. The key is the abbreviation as normalizeWords returns it. + * The seed adds the written-out words next to the abbreviation, so a search + * for "handstand" finds an element whose description says "hstd". + */ +export const ABBREVIATIONS: Readonly> = { + fwd: 'forward', + bwd: 'backward', + swd: 'sideward', + hstd: 'handstand', + hdst: 'handstand', // misspelling of hstd in two descriptions + hb: 'high bar', + lb: 'low bar', + sec: 'seconds', // "(2 sec.)" + min: 'minimum', // "min. ¾ twist" +} + +/** + * Abbreviations in the descriptions that are left out on purpose, with the + * reason. The abbreviation test fails on any abbreviation that is in neither + * list. + */ +export const ABBREVIATIONS_LEFT_OUT: Readonly> = { + etc: 'not something anyone searches for', +} + +/** + * The search words for a text: its own words, and the written-out form of + * every abbreviation in it. Each word appears once. + */ +export function searchWords(text: string): Set { + const words = new Set() + for (const word of normalizeWords(text)) { + words.add(word) + const writtenOut = ABBREVIATIONS[word] + if (writtenOut) normalizeWords(writtenOut).forEach((w) => words.add(w)) + } + return words +} diff --git a/backend/src/db.ts b/backend/src/db.ts new file mode 100644 index 0000000..308cfb3 --- /dev/null +++ b/backend/src/db.ts @@ -0,0 +1,13 @@ +import 'dotenv/config' +import { PrismaPg } from '@prisma/adapter-pg' +import { PrismaClient } from './generated/prisma/client.js' + +/** A Prisma client connected to DATABASE_URL (from backend/.env). */ +export function createPrismaClient(): PrismaClient { + const connectionString = process.env['DATABASE_URL'] + if (!connectionString) + throw new Error( + 'DATABASE_URL is not set. See "Database (local setup)" in the README.', + ) + return new PrismaClient({ adapter: new PrismaPg({ connectionString }) }) +} diff --git a/backend/src/seed/catalogData.ts b/backend/src/seed/catalogData.ts new file mode 100644 index 0000000..b8431d2 --- /dev/null +++ b/backend/src/seed/catalogData.ts @@ -0,0 +1,225 @@ +// Reads the catalog from data/*.json, checks it, and converts it to the shape +// of the database. Nothing here touches the database. + +import { existsSync, readFileSync } from 'node:fs' +import { join } from 'node:path' + +/** Sort order of the apparatus. Not in data/apparatus.json. */ +export const APPARATUS_POSITION: Readonly> = { + VT: 1, + UB: 2, + BB: 3, + FX: 4, +} + +type ApparatusJson = { id: string; name: string } +type GroupJson = { + id: string + apparatusId: string + number: number + name: string +} +type VariantGroupJson = { id: string; variantIds: string[] } +type ItemJson = { + id: string + elementGroupId: string + code: string + description: string + name: string | null + image: string +} +type ElementJson = ItemJson & { value: number } +type VaultJson = ItemJson & { dValue: number } + +/** One element or vault, converted: the value is kept as text with one decimal. */ +export type CatalogItem = { + kind: 'element' | 'vault' + id: string + elementGroupId: string + variantGroupId: string | null + code: string + value: string + description: string + name: string | null + image: string +} + +export type CatalogData = { + apparatus: { id: string; name: string; position: number }[] + groups: GroupJson[] + variantGroups: { id: string }[] + items: CatalogItem[] +} + +/** The data does not follow a rule. The message names the row and the rule. */ +export class CatalogDataError extends Error {} + +function readJson(dataDir: string, file: string): T { + return JSON.parse(readFileSync(join(dataDir, file), 'utf8')) as T +} + +/** A number with at most one decimal, as text ("0.1", "2.6"). */ +function oneDecimal(value: unknown, row: string, field: string): string { + if (typeof value !== 'number' || !Number.isFinite(value)) { + throw new CatalogDataError(`${row}: ${field} is not a number`) + } + const text = String(value) + if (!/^-?\d+(\.\d)?$/.test(text)) { + throw new CatalogDataError( + `${row}: ${field} ${text} has more than one decimal. The database stores one decimal and would round it.`, + ) + } + return text.includes('.') ? text : `${text}.0` +} + +/** + * Reads data/ and checks it against catalog constraints 1–5 in + * docs/DATA_MODEL.md and the shape the database needs. Throws a + * CatalogDataError at the first problem. + */ +export function loadCatalogData(dataDir: string): CatalogData { + const apparatusJson = readJson(dataDir, 'apparatus.json') + const groups = readJson(dataDir, 'elementGroups.json') + const variantGroupsJson = readJson( + dataDir, + 'variantGroups.json', + ) + const elements = readJson(dataDir, 'elements.json') + const vaults = readJson(dataDir, 'vaults.json') + + const apparatus = apparatusJson.map((a) => { + const position = APPARATUS_POSITION[a.id] + if (position === undefined) + throw new CatalogDataError( + `apparatus ${a.id}: unknown apparatus, no sort order`, + ) + return { id: a.id, name: a.name, position } + }) + const apparatusIds = new Set(apparatus.map((a) => a.id)) + const groupById = new Map(groups.map((g) => [g.id, g])) + for (const g of groups) { + if (!apparatusIds.has(g.apparatusId)) { + throw new CatalogDataError( + `element group ${g.id}: apparatus ${g.apparatusId} does not exist`, + ) + } + } + + // Variant groups: constraints 1 (at least two ids that exist) and 3 (at + // most one group per item) are checked here, 2 and 4 below. + const itemIds = new Set([...elements, ...vaults].map((i) => i.id)) + const variantGroupOf = new Map() + for (const vg of variantGroupsJson) { + if (vg.variantIds.length < 2) { + throw new CatalogDataError( + `variant group ${vg.id}: has fewer than two variants (constraint 1)`, + ) + } + for (const id of vg.variantIds) { + if (!itemIds.has(id)) + throw new CatalogDataError( + `variant group ${vg.id}: ${id} does not exist (constraint 1)`, + ) + const other = variantGroupOf.get(id) + if (other) + throw new CatalogDataError( + `${id}: is in variant groups ${other} and ${vg.id} (constraint 3)`, + ) + variantGroupOf.set(id, vg.id) + } + } + + const items: CatalogItem[] = [] + const seen = new Set() + const convert = ( + kind: CatalogItem['kind'], + raw: ElementJson | VaultJson, + ): CatalogItem => { + const row = `${kind} ${raw.id}` + if (seen.has(raw.id)) + throw new CatalogDataError(`${row}: the id is used more than once`) + seen.add(raw.id) + const group = groupById.get(raw.elementGroupId) + if (!group) + throw new CatalogDataError( + `${row}: element group ${raw.elementGroupId} does not exist`, + ) + if ((group.apparatusId === 'VT') !== (kind === 'vault')) { + throw new CatalogDataError( + `${row}: is in group ${group.id}, which is for ${kind === 'vault' ? 'elements' : 'vaults'}`, + ) + } + const field = kind === 'element' ? 'value' : 'dValue' + const value = oneDecimal( + kind === 'element' + ? (raw as ElementJson).value + : (raw as VaultJson).dValue, + row, + field, + ) + const number = Number(value) + if (kind === 'element' && (number < 0.1 || number > 1)) { + throw new CatalogDataError(`${row}: value ${value} is outside 0.1–1.0`) + } + if (kind === 'vault' && number <= 0) + throw new CatalogDataError( + `${row}: dValue ${value} is not greater than 0`, + ) + if (!raw.description?.trim()) + throw new CatalogDataError(`${row}: has no description`) + // Constraint 5: every element and vault has an image. + if (!raw.image) + throw new CatalogDataError(`${row}: has no image (constraint 5)`) + if (!existsSync(join(dataDir, 'images', group.apparatusId, raw.image))) { + throw new CatalogDataError( + `${row}: image ${group.apparatusId}/${raw.image} does not exist (constraint 5)`, + ) + } + return { + kind, + id: raw.id, + elementGroupId: raw.elementGroupId, + variantGroupId: variantGroupOf.get(raw.id) ?? null, + code: raw.code, + value, + description: raw.description, + name: raw.name, + image: raw.image, + } + } + elements.forEach((e) => items.push(convert('element', e))) + vaults.forEach((v) => items.push(convert('vault', v))) + + // Constraints 2 (one type per group) and 4 (same code and value). + const itemById = new Map(items.map((i) => [i.id, i])) + for (const vg of variantGroupsJson) { + const members = vg.variantIds.map((id) => itemById.get(id)!) + const first = members[0]! + for (const m of members) { + if (m.kind !== first.kind) + throw new CatalogDataError( + `variant group ${vg.id}: mixes elements and vaults (constraint 2)`, + ) + if (m.code !== first.code || m.value !== first.value) { + throw new CatalogDataError( + `variant group ${vg.id}: ${m.id} has code ${m.code} and value ${m.value}, but ${first.id} has code ${first.code} and value ${first.value} (constraint 4)`, + ) + } + } + } + + return { + apparatus, + groups, + variantGroups: variantGroupsJson.map((vg) => ({ id: vg.id })), + items, + } +} + +/** The code with each part padded to four digits, so "2.1" sorts before "10.1". */ +export function codeSort(code: string): string { + return code + .split('.') + .map((part) => part.padStart(4, '0')) + .join('.') +} diff --git a/backend/src/seed/seed.ts b/backend/src/seed/seed.ts new file mode 100644 index 0000000..c1b3c1e --- /dev/null +++ b/backend/src/seed/seed.ts @@ -0,0 +1,324 @@ +// Seeds the catalog from data/*.json and rebuilds the search tables. +// +// Run with `npm run db:seed` from the repository root. It is safe to run more +// than once: everything happens in one transaction, and a second run with the +// same data changes nothing in the catalog. User data (coaches, gymnasts, +// routines and their lists) is never touched. Rows whose id starts with "SYN-" +// are synthetic test data from another script and are left alone. +// +// Options: --data reads the JSON files from another folder than data/. + +import { fileURLToPath } from 'node:url' +import { parseArgs } from 'node:util' +import { createPrismaClient } from '../db.js' +import { searchWords } from '../catalog/words.js' +import { Prisma } from '../generated/prisma/client.js' +import { + type CatalogData, + type CatalogItem, + CatalogDataError, + codeSort, + loadCatalogData, +} from './catalogData.js' + +type Tx = Prisma.TransactionClient +type Counts = { + inserted: number + updated: number + removed: number + deleted: number +} + +const NOT_SYNTHETIC = { NOT: { id: { startsWith: 'SYN-' } } } +const DEFAULT_DATA_DIR = fileURLToPath( + new URL('../../../data/', import.meta.url), +) + +const counts = (): Counts => ({ + inserted: 0, + updated: 0, + removed: 0, + deleted: 0, +}) + +async function seedApparatusAndGroups( + tx: Tx, + data: CatalogData, + summary: Record, +) { + const apparatus = new Map( + (await tx.apparatus.findMany({ where: NOT_SYNTHETIC })).map((a) => [ + a.id, + a, + ]), + ) + for (const a of data.apparatus) { + const old = apparatus.get(a.id) + if (!old) { + await tx.apparatus.create({ data: a }) + summary.apparatus!.inserted++ + } else if (old.name !== a.name || old.position !== a.position) { + await tx.apparatus.update({ where: { id: a.id }, data: a }) + summary.apparatus!.updated++ + } + } + + const groups = new Map( + (await tx.elementGroup.findMany({ where: NOT_SYNTHETIC })).map((g) => [ + g.id, + g, + ]), + ) + for (const g of data.groups) { + const old = groups.get(g.id) + if (!old) { + await tx.elementGroup.create({ data: g }) + summary.elementGroups!.inserted++ + } else if ( + old.apparatusId !== g.apparatusId || + old.number !== g.number || + old.name !== g.name + ) { + await tx.elementGroup.update({ where: { id: g.id }, data: g }) + summary.elementGroups!.updated++ + } + } + + const variantGroups = new Set( + (await tx.variantGroup.findMany({ where: NOT_SYNTHETIC })).map( + (vg) => vg.id, + ), + ) + const newGroups = data.variantGroups.filter((vg) => !variantGroups.has(vg.id)) + if (newGroups.length) await tx.variantGroup.createMany({ data: newGroups }) + summary.variantGroups!.inserted += newGroups.length +} + +/** Upserts the elements or the vaults, and marks the ones missing from the data as removed. */ +async function seedItems( + tx: Tx, + kind: CatalogItem['kind'], + items: CatalogItem[], + summary: Counts, + now: Date, +) { + const table = kind === 'element' ? tx.element : tx.vault + type Row = { + id: string + elementGroupId: string + variantGroupId: string | null + code: string + description: string + name: string | null + image: string + removedAt: Date | null + value?: Prisma.Decimal + dValue?: Prisma.Decimal + } + const existing = new Map( + ( + (await (table as typeof tx.element).findMany({ + where: NOT_SYNTHETIC, + })) as Row[] + ).map((r) => [r.id, r]), + ) + const toRow = (i: CatalogItem) => ({ + id: i.id, + elementGroupId: i.elementGroupId, + variantGroupId: i.variantGroupId, + code: i.code, + description: i.description, + name: i.name, + image: i.image, + removedAt: null, + ...(kind === 'element' ? { value: i.value } : { dValue: i.value }), + }) + + const inserts = [] + for (const item of items) { + const old = existing.get(item.id) + if (!old) { + inserts.push(toRow(item)) + continue + } + const oldValue = kind === 'element' ? old.value : old.dValue + const changed = + old.elementGroupId !== item.elementGroupId || + old.variantGroupId !== item.variantGroupId || + old.code !== item.code || + old.description !== item.description || + old.name !== item.name || + old.image !== item.image || + old.removedAt !== null || // it was removed, and is back in the data + !oldValue?.equals(item.value) + if (changed) { + if (kind === 'element') + await tx.element.update({ + where: { id: item.id }, + data: toRow(item) as Prisma.ElementUncheckedUpdateInput, + }) + else + await tx.vault.update({ + where: { id: item.id }, + data: toRow(item) as Prisma.VaultUncheckedUpdateInput, + }) + summary.updated++ + } + } + if (inserts.length) { + if (kind === 'element') + await tx.element.createMany({ + data: inserts as Prisma.ElementCreateManyInput[], + }) + else + await tx.vault.createMany({ + data: inserts as Prisma.VaultCreateManyInput[], + }) + } + summary.inserted += inserts.length + + // Soft delete: in the database, but no longer in the data. + const inData = items.map((i) => i.id) + const where = { ...NOT_SYNTHETIC, id: { notIn: inData }, removedAt: null } + const removed = + kind === 'element' + ? await tx.element.updateMany({ where, data: { removedAt: now } }) + : await tx.vault.updateMany({ where, data: { removedAt: now } }) + summary.removed += removed.count + + // Clean-up: removed items that no routine and no gymnast list uses. + const unused = { + ...NOT_SYNTHETIC, + removedAt: { not: null }, + routineElements: { none: {} }, + gymnastElements: { none: {} }, + } + const deleted = + kind === 'element' + ? await tx.element.deleteMany({ where: unused }) + : await tx.vault.deleteMany({ where: unused }) + summary.deleted += deleted.count +} + +/** Deletes variant groups with fewer than two members, and unlinks their last member. */ +async function cleanVariantGroups(tx: Tx, summary: Counts) { + const groups = await tx.variantGroup.findMany({ + where: NOT_SYNTHETIC, + include: { _count: { select: { elements: true, vaults: true } } }, + }) + for (const group of groups) { + if (group._count.elements + group._count.vaults >= 2) continue + await tx.element.updateMany({ + where: { variantGroupId: group.id }, + data: { variantGroupId: null }, + }) + await tx.vault.updateMany({ + where: { variantGroupId: group.id }, + data: { variantGroupId: null }, + }) + await tx.variantGroup.delete({ where: { id: group.id } }) + summary.deleted++ + } +} + +/** + * Rebuilds CatalogEntry and CatalogWord from the catalog. One entry per code, + * from the first variant that is not removed. The words come from the code, + * name and description of all variants of the code that are not removed. + */ +async function rebuildSearchTables(tx: Tx, data: CatalogData) { + const deleted = await tx.catalogEntry.deleteMany({ where: NOT_SYNTHETIC }) // the words are deleted with them + + const apparatusOfGroup = new Map( + data.groups.map((g) => [g.id, g.apparatusId]), + ) + // Everything in the data is in the catalog and not removed after the seed. + const byCode = new Map() + for (const item of data.items) { + const key = `${apparatusOfGroup.get(item.elementGroupId)}-${item.code}` + byCode.set(key, [...(byCode.get(key) ?? []), item]) + } + + const entries: Prisma.CatalogEntryCreateManyInput[] = [] + const words: Prisma.CatalogWordCreateManyInput[] = [] + for (const variants of byCode.values()) { + variants.sort((a, b) => a.id.localeCompare(b.id)) + const first = variants[0]! + entries.push({ + id: first.id, + type: first.kind === 'element' ? 'ELEMENT' : 'VAULT', + code: first.code, + name: first.name, + description: first.description, + difficulty: first.value, + apparatusId: apparatusOfGroup.get(first.elementGroupId)!, + elementGroupId: first.elementGroupId, + hasName: first.name !== null, + nameSort: first.name?.toLowerCase() ?? '', + codeSort: codeSort(first.code), + }) + const entryWords = new Set() + for (const v of variants) { + for (const text of [v.code, v.name ?? '', v.description]) + searchWords(text).forEach((w) => entryWords.add(w)) + } + entryWords.forEach((word) => words.push({ entryId: first.id, word })) + } + await tx.catalogEntry.createMany({ data: entries }) + await tx.catalogWord.createMany({ data: words }) + return { + deletedEntries: deleted.count, + entries: entries.length, + words: words.length, + } +} + +export async function seed(dataDir: string) { + const data = loadCatalogData(dataDir) // throws before anything is written + const prisma = createPrismaClient() + const summary: Record = { + apparatus: counts(), + elementGroups: counts(), + variantGroups: counts(), + elements: counts(), + vaults: counts(), + } + try { + const now = new Date() + const search = await prisma.$transaction( + async (tx) => { + await seedApparatusAndGroups(tx, data, summary) + const elements = data.items.filter((i) => i.kind === 'element') + const vaults = data.items.filter((i) => i.kind === 'vault') + await seedItems(tx, 'element', elements, summary.elements!, now) + await seedItems(tx, 'vault', vaults, summary.vaults!, now) + await cleanVariantGroups(tx, summary.variantGroups!) + return rebuildSearchTables(tx, data) + }, + { timeout: 120_000, maxWait: 10_000 }, + ) + + console.log(`Seeded the catalog from ${dataDir}`) + console.table(summary) + console.log( + `Search tables rebuilt: ${search.entries} catalog entries and ${search.words} words (${search.deletedEntries} old entries replaced).`, + ) + } finally { + await prisma.$disconnect() + } +} + +if (process.argv[1] === fileURLToPath(import.meta.url)) { + const { values } = parseArgs({ options: { data: { type: 'string' } } }) + seed(values.data ?? DEFAULT_DATA_DIR).catch((error: unknown) => { + if (error instanceof CatalogDataError) { + console.error(`The seed stopped. Nothing was changed.\n${error.message}`) + } else { + console.error( + 'The seed failed. Nothing was changed, because everything runs in one transaction.', + ) + console.error(error) + } + process.exit(1) + }) +} diff --git a/docs/AI_USAGE.md b/docs/AI_USAGE.md index 3047163..0d9dd3d 100644 --- a/docs/AI_USAGE.md +++ b/docs/AI_USAGE.md @@ -130,3 +130,18 @@ task description that listed the tables, the decisions above and the checks to run, and continued with the team's decisions and the review comments as they came. + +## Seed script (`backend/src/seed/`, `backend/src/catalog/`, README) + +- **Tool:** Claude Code +- **What the team did:** Wrote the task description with the rules the seed + must follow: convert the old shape of `data/`, check the data before + writing, upsert in one transaction, soft delete and clean-up, leave user data + and synthetic test rows alone, and rebuild the search tables. Asked to see + the list of abbreviations before it was finalized, and approved it. +- **What Claude did:** Went through the descriptions to find the + abbreviations and proposed the list, wrote the seed script, the word + normalization and the tests, and the "Seeding" section in the README. Ran + the checks on a local database: two runs in a row with no changes, the row + counts against the JSON files, broken copies of the data, the soft delete + and clean-up, and that user data is untouched. diff --git a/docs/API.md b/docs/API.md index 3b6929c..807dbc3 100644 --- a/docs/API.md +++ b/docs/API.md @@ -245,7 +245,7 @@ The element and vault tables stay as the data model describes them and are the s **Sort without `null`.** Most items have no name, so sorting by name cannot put `null` last, because a cursor does not accept `nulls` placement. The sort is `hasName` descending, then `nameSort`, then `id`. All three are plain fields and no value is `null`, so named items come first in both directions and the cursor works. -**The word index.** A second table, `CatalogWord`, has one row for each distinct word of each entry: the entry id and the word in lowercase. The seed script fills it by splitting the code, the name and the description of **all the variants** of the code into words (letters and digits; a code like `3.304` is kept as one word). A word that is only in variant `-b` therefore still finds the code. The seed script also adds abbreviations from the Code of Points written out as words, for example `handstand` for `hstd`. It has a B-tree index on `word`. The search is a Prisma filter with one condition per search word: +**The word index.** A second table, `CatalogWord`, has one row for each distinct word of each entry: the entry id and the word in lowercase. The seed script fills it by splitting the code, the name and the description of **all the variants** of the code into words (letters and digits, without accents; a code like `3.304` is kept as one word). A word that is only in variant `-b` therefore still finds the code. The seed script also adds abbreviations from the Code of Points written out as words, for example `handstand` for `hstd` and `low bar` for `LB`. The list is `ABBREVIATIONS` in `backend/src/catalog/words.ts`, and a test fails if the data contains an abbreviation that is neither in the list nor left out on purpose. It has a B-tree index on `word`. The search is a Prisma filter with one condition per search word: ```ts where: { @@ -261,7 +261,7 @@ where: { } ``` -- The search text is split into words on spaces. Each word is cleaned to lowercase letters, digits and `.` before use. Empty words are dropped. +- The search text is split into words with `normalizeWords` in `backend/src/catalog/words.ts`: lowercase, accents removed, and only letters and digits kept, except that a code like `3.304` is kept as one word. The seed script uses the same function for the stored words, so the search term and the words always follow the same rules. Empty words are dropped. - **The cleaning is the only protection against wildcards.** Prisma does not escape `%` and `_` in `startsWith`: a search for `%` becomes `LIKE '%%'` and matches every word (in a test, all 225 000 entries in about 460 ms). The cleaning removes them, so it must always run before the query, and it must have tests. - **The cleaning is also what makes the search case-insensitive.** The words are stored in lowercase, so an uppercase term such as "SAL" finds nothing unless it is lowercased first. - **All words must match.** An item is found only if it has, for every search word, a word that starts with it. "giant turn" finds giant circles with a turn, and not every giant circle and every turn. diff --git a/package.json b/package.json index 748a1d5..3fd9cf4 100644 --- a/package.json +++ b/package.json @@ -16,6 +16,7 @@ "db:generate": "npm run prisma:generate -w backend", "db:migrate": "npm run prisma:migrate -w backend", "db:deploy": "npm run prisma:deploy -w backend", + "db:seed": "npm run prisma:seed -w backend", "db:studio": "npm run prisma:studio -w backend" }, "dependencies": {