23 seed the catalog

README.md

@@ -6,7 +6,7 @@ We are building a web application that helps gymnastics coaches plan their gymna
 
 ## Status
 
-The first delivery was a specification. There is no working application yet, but the catalog data has been extracted from the Code of Points and is in [`data/`](data/), and the database schema is in place: the Prisma schema and its migrations create all the tables of the data model in PostgreSQL. The seed script, the GraphQL server and the search are the next steps.
+The first delivery was a specification. There is no working application yet, but the catalog data has been extracted from the Code of Points and is in [`data/`](data/), and the database schema is in place: the Prisma schema and its migrations create all the tables of the data model in PostgreSQL. The seed script loads `data/` into the database. The GraphQL server and the search are the next steps.
 
 The code in `src/` is the Vite starter project with one example test. The tooling is set up: TypeScript, ESLint, Prettier and Vitest.
 
@@ -23,7 +23,7 @@ The code in `src/` is the Vite starter project with one example test. The toolin
 
 ## The dataset
 
-The catalog data comes from the FIG Women's Artistic Gymnastics Code of Points 2025–2028, which is published as a PDF. We have extracted the apparatus, element groups, elements and vaults from the PDF with a Python script in [`scripts/extract-cop/`](scripts/extract-cop/), and the result is in [`data/`](data/). [docs/EXTRACTION.md](docs/EXTRACTION.md) explains how it was done and what has been checked by hand. The data will be loaded into the database with a seed script, which is not written yet.
+The catalog data comes from the FIG Women's Artistic Gymnastics Code of Points 2025–2028, which is published as a PDF. We have extracted the apparatus, element groups, elements and vaults from the PDF with a Python script in [`scripts/extract-cop/`](scripts/extract-cop/), and the result is in [`data/`](data/). [docs/EXTRACTION.md](docs/EXTRACTION.md) explains how it was done and what has been checked by hand. The seed script in `backend/src/seed/` loads the data into the database (see "Seeding" below).
 
 The data is used for educational purposes. All rights to the Code of Points belong to FIG (Fédération Internationale de Gymnastique).
 
@@ -63,13 +63,13 @@ In the repository now:
 - React, TypeScript and Vite for the frontend
 - A backend package in `backend/` with TypeScript and Prisma, set up as an npm workspace
 - PostgreSQL 18, installed directly on our own machines (not in Docker), with the Prisma schema and migrations for all the tables of the data model
+- A seed script that loads the catalog from `data/` into the database
 - ESLint and Prettier for linting and formatting, for both the frontend and the backend
 - Vitest for testing
 
 Planned:
 
 - A GraphQL backend in Node and TypeScript
-- A seed script that loads `data/` into the database
 - PostgreSQL 18 on the VM, set up the same way as on our own machines
 
 ### Why PostgreSQL
@@ -173,6 +173,7 @@ From the repository root:
 npm ci
 npm run db:generate
 npm run db:migrate
+npm run db:seed
 ```
 
 `npm run db:migrate` creates the tables in your database by applying the migrations in `backend/prisma/migrations/`. Run it again whenever someone has added a new migration.
@@ -182,6 +183,7 @@ npm run db:migrate
 | `npm run db:generate` | Generates the Prisma client from `backend/prisma/schema.prisma`                |
 | `npm run db:migrate`  | Creates a migration from changes in the schema and applies it to your database |
 | `npm run db:deploy`   | Applies existing migrations without creating new ones. Used on the server      |
+| `npm run db:seed`     | Loads the catalog from `data/` into the database. See "Seeding" below          |
 | `npm run db:studio`   | Opens Prisma Studio in the browser, to look at the data in the database        |
 
 ### 5. Check the connection
@@ -217,6 +219,43 @@ To check that the backend can reach your database:
 
 If the output says "Following migration have not yet been applied", the connection works but your tables are missing or out of date. Go back to the repository root and run `npm run db:migrate`. If the connection fails, Prisma prints an error such as "Can't reach database server" or "Authentication failed" instead.
 
+### 6. Seeding
+
+The seed script loads the catalog (apparatus, element groups, variant groups, elements and vaults) from the JSON files in `data/` into your database, and rebuilds the search tables. Run it from the repository root:
+
+```bash
+npm run db:seed
+```
+
+Run it again:
+
+- **after migrations**, when you have run `npm run db:migrate` or `npm run db:deploy`
+- **whenever `data/` changes**, for example after a new run of the extraction script
+
+It is safe to run as often as you like:
+
+- **Everything happens in one transaction.** If something fails, nothing is changed.
+- **A second run with the same data changes nothing.** It prints a table of what was inserted, updated, marked as removed and deleted.
+- **It checks the data first.** If something in `data/` breaks a rule in `docs/DATA_MODEL.md`, for example a variant group with different codes or a missing image, it stops with a message that names the row and the rule, and changes nothing.
+- **It never touches user data.** Coaches, gymnasts, routines and their lists are left alone.
+- **Removed elements are kept while they are in use.** An element or vault that is no longer in `data/` is marked as removed. It is deleted only if no routine or gymnast list uses it.
+
+The code is in `backend/src/seed/`. The rules for search words, including the written-out abbreviations such as `handstand` for `hstd`, are in `backend/src/catalog/words.ts`. The search uses the same rules.
+
+### 7. Look at the data
+
+To see what is in your database, open Prisma Studio from the repository root:
+
+```bash
+npm run db:studio
+```
+
+The terminal prints the address, for example "Prisma Studio is running at: http://localhost:51212". The port can change between runs. Open the address in your browser if it does not open by itself. The page has a list of all the tables. Click a table to see its rows, filter and sort them, and follow the relations, for example from an element to its variant group or from a routine to its items. Stop it with Ctrl+C in the terminal.
+
+Prisma Studio uses the connection in `backend/.env`, so there is nothing more to set up. You can also edit and delete rows there, so be careful: changes to the catalog are overwritten the next time you run `npm run db:seed`, and changes to user data are not undone.
+
+To write your own SQL queries, connect with `psql -h localhost -U cop -d cop` (on Windows, use _SQL Shell (psql)_) and enter the password from `backend/.env`. The tables have snake_case names in the database, such as `catalog_entries` and `element_group_id`. `\dt` lists the tables, `\d elements` shows the columns of a table, and `\q` quits.
+
 ## Project structure
 
 | Folder                 | Contents                                                                                                                                               |

backend/package.json

@@ -8,6 +8,7 @@
     "prisma:generate": "prisma generate",
     "prisma:migrate": "prisma migrate dev",
     "prisma:deploy": "prisma migrate deploy",
+    "prisma:seed": "prisma db seed",
     "prisma:studio": "prisma studio"
   },
   "engines": {

backend/prisma.config.ts

@@ -7,6 +7,8 @@ export default defineConfig({
   schema: 'prisma/schema.prisma',
   migrations: {
     path: 'prisma/migrations',
+    // `prisma db seed` (npm run db:seed) runs the seed script.
+    seed: 'tsx src/seed/seed.ts',
   },
   datasource: {
     url: process.env['DATABASE_URL'],

backend/src/catalog/abbreviations.test.ts

@@ -0,0 +1,67 @@
+// Checks that every abbreviation in the catalog descriptions is either written
+// out (ABBREVIATIONS) or left out on purpose (ABBREVIATIONS_LEFT_OUT).
+//
+// A word in a description is treated as an abbreviation when it has at least
+// two letters and:
+// - is all uppercase, like "HB" and "LB", or
+// - has no vowels (a, e, i, o, u, y), like "fwd", "bwd" and "hstd", or
+// - is followed by a dot that does not end a sentence, like "sec." in
+//   "(2 sec.)" and "min." in "min. ¾ twist". A dot ends a sentence when it is
+//   the last character, or when the next word starts with an uppercase letter.
+// Single letters such as the "L" in "L grip" are shapes, not abbreviations.
+
+import { readFileSync } from 'node:fs'
+import { describe, expect, it } from 'vitest'
+import {
+  ABBREVIATIONS,
+  ABBREVIATIONS_LEFT_OUT,
+  normalizeWords,
+} from './words.js'
+
+type CatalogItem = { id: string; description: string }
+
+const dataDir = new URL('../../../data/', import.meta.url)
+const read = (file: string): CatalogItem[] =>
+  JSON.parse(readFileSync(new URL(file, dataDir), 'utf8'))
+const items = [...read('elements.json'), ...read('vaults.json')]
+
+function findAbbreviations(description: string): string[] {
+  const found: string[] = []
+  const text = description.trimEnd()
+  for (const match of text.matchAll(/[A-Za-z]+/g)) {
+    const word = match[0]
+    const end = match.index + word.length
+    const next = text.slice(end + 1).trimStart()[0]
+    const dotInside =
+      text[end] === '.' && next !== undefined && next === next.toLowerCase()
+    const isAbbreviation =
+      word.length >= 2 &&
+      (word === word.toUpperCase() || !/[aeiouy]/i.test(word) || dotInside)
+    if (isAbbreviation) found.push(...normalizeWords(word))
+  }
+  return found
+}
+
+describe('abbreviations in the catalog descriptions', () => {
+  it('finds the abbreviations it is meant to find', () => {
+    expect(findAbbreviations('Swing fwd on HB to hstd (2 sec.)')).toEqual([
+      'fwd',
+      'hb',
+      'hstd',
+      'sec',
+    ])
+    expect(findAbbreviations('Salto in L hang. Then a tuck.')).toEqual([])
+  })
+
+  it('are all written out or left out on purpose', () => {
+    const unknown = new Map<string, string>()
+    for (const item of items) {
+      for (const word of findAbbreviations(item.description)) {
+        if (!(word in ABBREVIATIONS) && !(word in ABBREVIATIONS_LEFT_OUT))
+          unknown.set(word, item.id)
+      }
+    }
+    // Shows each unknown abbreviation with an id where it is used.
+    expect(Object.fromEntries(unknown)).toEqual({})
+  })
+})

backend/src/catalog/words.test.ts

@@ -0,0 +1,91 @@
+import { describe, expect, it } from 'vitest'
+import { normalizeWords, searchWords } from './words.js'
+
+describe('normalizeWords', () => {
+  it('lowercases', () => {
+    expect(normalizeWords('Salto BWD Tucked')).toEqual([
+      'salto',
+      'bwd',
+      'tucked',
+    ])
+  })
+
+  it('removes accents', () => {
+    expect(normalizeWords('Fouetté Röhm Šimić')).toEqual([
+      'fouette',
+      'rohm',
+      'simic',
+    ])
+  })
+
+  it('keeps a code such as 3.304 as one word', () => {
+    expect(normalizeWords('3.304 and 1.22')).toEqual(['3.304', 'and', '1.22'])
+  })
+
+  it('removes punctuation', () => {
+    expect(
+      normalizeWords('Clear hip circle (Weiler-kip), hop-grip; “false-pop”.'),
+    ).toEqual([
+      'clear',
+      'hip',
+      'circle',
+      'weiler',
+      'kip',
+      'hop',
+      'grip',
+      'false',
+      'pop',
+    ])
+  })
+
+  it('does not keep a dot at the end of a word', () => {
+    expect(normalizeWords('to hang on HB. (2 sec.)')).toEqual([
+      'to',
+      'hang',
+      'on',
+      'hb',
+      '2',
+      'sec',
+    ])
+  })
+
+  it('splits fractions and degrees into numbers', () => {
+    expect(normalizeWords('½ turn (180°), 1/1 turn')).toEqual([
+      '1',
+      '2',
+      'turn',
+      '180',
+      '1',
+      '1',
+      'turn',
+    ])
+  })
+
+  it('returns no words for an empty text or only punctuation', () => {
+    expect(normalizeWords('')).toEqual([])
+    expect(normalizeWords(' – (…) ')).toEqual([])
+  })
+
+  it('never returns % or _, which are wildcards in LIKE', () => {
+    expect(normalizeWords('%sal_to%')).toEqual(['sal', 'to'])
+  })
+})
+
+describe('searchWords', () => {
+  it('adds the written-out form of abbreviations', () => {
+    expect([...searchWords('Cast to hstd on LB')]).toEqual([
+      'cast',
+      'to',
+      'hstd',
+      'handstand',
+      'on',
+      'lb',
+      'low',
+      'bar',
+    ])
+  })
+
+  it('keeps each word once', () => {
+    expect([...searchWords('fwd and fwd')]).toEqual(['fwd', 'forward', 'and'])
+  })
+})

backend/src/catalog/words.ts

@@ -0,0 +1,60 @@
+// The words the catalog search looks in. The seed script uses these functions to
+// fill CatalogWord, and the search (issue #24) must use normalizeWords on the
+// search text, so that both sides follow exactly the same rules.
+
+/**
+ * Splits a text into search words: lowercase, without accents, and without
+ * punctuation. A code such as "3.304" is kept as one word.
+ *
+ * normalizeWords('Clear hip circle to hstd (Weiler-kip), 1/1 turn') returns
+ * ['clear', 'hip', 'circle', 'to', 'hstd', 'weiler', 'kip', '1', '1', 'turn'].
+ */
+export function normalizeWords(text: string): string[] {
+  const plain = text
+    .normalize('NFKD') // "é" -> "e" + accent, "½" -> "1⁄2"
+    .replace(/\p{M}/gu, '') // remove the accents
+    .toLowerCase()
+  // A code: digits with dots between them. Otherwise: letters and digits.
+  return plain.match(/\d+(?:\.\d+)+|[a-z0-9]+/g) ?? []
+}
+
+/**
+ * Abbreviations used in the descriptions of the Code of Points, and the words
+ * they stand for. The key is the abbreviation as normalizeWords returns it.
+ * The seed adds the written-out words next to the abbreviation, so a search
+ * for "handstand" finds an element whose description says "hstd".
+ */
+export const ABBREVIATIONS: Readonly<Record<string, string>> = {
+  fwd: 'forward',
+  bwd: 'backward',
+  swd: 'sideward',
+  hstd: 'handstand',
+  hdst: 'handstand', // misspelling of hstd in two descriptions
+  hb: 'high bar',
+  lb: 'low bar',
+  sec: 'seconds', // "(2 sec.)"
+  min: 'minimum', // "min. ¾ twist"
+}
+
+/**
+ * Abbreviations in the descriptions that are left out on purpose, with the
+ * reason. The abbreviation test fails on any abbreviation that is in neither
+ * list.
+ */
+export const ABBREVIATIONS_LEFT_OUT: Readonly<Record<string, string>> = {
+  etc: 'not something anyone searches for',
+}
+
+/**
+ * The search words for a text: its own words, and the written-out form of
+ * every abbreviation in it. Each word appears once.
+ */
+export function searchWords(text: string): Set<string> {
+  const words = new Set<string>()
+  for (const word of normalizeWords(text)) {
+    words.add(word)
+    const writtenOut = ABBREVIATIONS[word]
+    if (writtenOut) normalizeWords(writtenOut).forEach((w) => words.add(w))
+  }
+  return words
+}

backend/src/db.ts

@@ -0,0 +1,13 @@
+import 'dotenv/config'
+import { PrismaPg } from '@prisma/adapter-pg'
+import { PrismaClient } from './generated/prisma/client.js'
+
+/** A Prisma client connected to DATABASE_URL (from backend/.env). */
+export function createPrismaClient(): PrismaClient {
+  const connectionString = process.env['DATABASE_URL']
+  if (!connectionString)
+    throw new Error(
+      'DATABASE_URL is not set. See "Database (local setup)" in the README.',
+    )
+  return new PrismaClient({ adapter: new PrismaPg({ connectionString }) })
+}