24 add indexes and search

README.md

@@ -6,7 +6,7 @@ We are building a web application that helps gymnastics coaches plan their gymna
 
 ## Status
 
-The first delivery was a specification. There is no working application yet, but the catalog data has been extracted from the Code of Points and is in [`data/`](data/), and the database schema is in place: the Prisma schema and its migrations create all the tables of the data model in PostgreSQL. The seed script loads `data/` into the database. The GraphQL server and the search are the next steps.
+The first delivery was a specification. There is no working application yet, but the catalog data has been extracted from the Code of Points and is in [`data/`](data/), and the database schema is in place: the Prisma schema and its migrations create all the tables of the data model in PostgreSQL. The seed script loads `data/` into the database, and the catalog search is written as a backend function. The GraphQL server, which will call it, is the next step.
 
 The code in `src/` is the Vite starter project with one example test. The tooling is set up: TypeScript, ESLint, Prettier and Vitest.
 
@@ -50,7 +50,7 @@ GraphQL is required by the assignment, but it also suits this application. The f
 - **The data is nested.** A page often needs an apparatus with its groups and the items in them, or a gymnast with its routines and their elements. With REST that means several requests, or one endpoint for each page. With GraphQL the client asks for exactly the nested fields it needs in one request.
 - **Less data over the network.** A list of search results only needs a code, a description and a value, while a detail page needs more. The client chooses the fields, so we do not send what is not shown. This is part of our sustainability choices.
 - **One typed contract.** The schema describes every query, argument and field. We can generate TypeScript types from it, so a change in the API shows up as a type error in the client and not as a bug at runtime.
-- **Search, filter, sort and pagination in one query.** `searchCatalog` takes all of them as typed arguments and runs on the server over the whole data set. Pages are fetched with cursors, which keeps every page fast whatever the size of the catalog.
+- **Search, filter, sort and pagination in one query.** `searchCatalog` takes all of them as typed arguments and runs on the server over the whole data set. Pages are fetched with cursors, so a deep page does not read the earlier pages again (measured on a large catalog in a later sub-issue).
 
 The costs are that HTTP caching is harder than with REST, because all requests go to one endpoint, and that the server must guard against expensive queries. We handle the last one with an upper limit on the page size, and we avoid repeated database queries for nested fields with DataLoader.
 
@@ -64,6 +64,7 @@ In the repository now:
 - A backend package in `backend/` with TypeScript and Prisma, set up as an npm workspace
 - PostgreSQL 18, installed directly on our own machines (not in Docker), with the Prisma schema and migrations for all the tables of the data model
 - A seed script that loads the catalog from `data/` into the database
+- The catalog search, with Prisma only and cursor pagination. It is not yet reachable through a GraphQL server
 - ESLint and Prettier for linting and formatting, for both the frontend and the backend
 - Vitest for testing
 
@@ -256,11 +257,19 @@ Prisma Studio uses the connection in `backend/.env`, so there is nothing more to
 
 To write your own SQL queries, connect with `psql -h localhost -U cop -d cop` (on Windows, use _SQL Shell (psql)_) and enter the password from `backend/.env`. The tables have snake_case names in the database, such as `catalog_entries` and `element_group_id`. `\dt` lists the tables, `\d elements` shows the columns of a table, and `\q` quits.
 
+## Catalog search
+
+The catalog search is in `backend/src/catalog/search.ts`. It finds elements and vaults by the start of their words, with filters, sorting and cursor pagination, using Prisma only. How it works and why is described in "Search without raw SQL" in [docs/API.md](docs/API.md#search-without-raw-sql).
+
+The tests in `backend/src/catalog/` run with `npm test`. The search tests use your local database and need the seeded catalog, so run `npm run db:seed` first. They only read from the database.
+
+**Without `DATABASE_URL` in `backend/.env`, the database tests are skipped.** Vitest then reports them as skipped and the run still passes, so a passing run without a database does not mean the search was tested. Check that the output says 0 skipped.
+
 ## Project structure
 
 | Folder                 | Contents                                                                                                                                               |
 | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
-| `backend/`             | The backend package: TypeScript, Prisma and the draft GraphQL schema                                                                                   |
+| `backend/`             | The backend package: TypeScript, Prisma, the seed script, the catalog search and the draft GraphQL schema                                              |
 | `data/`                | The catalog data extracted from the Code of Points: JSON files and one image per element                                                               |
 | `docs/`                | The specification: user interface plans, data model, example data and AI usage. [docs/EXTRACTION.md](docs/EXTRACTION.md) explains how `data/` was made |
 | `public/`              | Static files served as they are, such as the favicon                                                                                                   |

backend/src/catalog/cursor.ts

@@ -0,0 +1,76 @@
+// The cursor of the catalog search: a text that says where the next page
+// starts. It is base64url-encoded JSON with the sort field and direction, the
+// sort values of the last item on the page, and its id.
+//
+// The client should treat it as opaque, but it is not secret: the catalog is
+// public, so a changed or hand-made cursor can only lead to another page of
+// public data. What matters is that it is checked strictly when it is decoded,
+// before any value reaches Prisma.
+
+import { BadUserInputError } from '../errors.js'
+import {
+  SORTS,
+  type SortDirection,
+  type SortField,
+  type SortValues,
+  isText,
+} from './sorts.js'
+
+/** What a cursor points at: the sort values and the id of the last item. */
+export type CursorPosition = { values: SortValues; id: string }
+
+const INVALID =
+  'The cursor is not valid. Use the endCursor from the previous page.'
+
+export function encodeCursor(
+  field: SortField,
+  direction: SortDirection,
+  position: CursorPosition,
+): string {
+  const payload = { field, direction, id: position.id, ...position.values }
+  return Buffer.from(JSON.stringify(payload), 'utf8').toString('base64url')
+}
+
+/**
+ * Decodes and checks a cursor: it must be for the same sort field and
+ * direction as the current request, have exactly the expected keys, and every
+ * value must have the right type. Anything else gives BAD_USER_INPUT.
+ */
+export function decodeCursor(
+  cursor: string,
+  field: SortField,
+  direction: SortDirection,
+): CursorPosition {
+  let payload: unknown
+  try {
+    payload = JSON.parse(Buffer.from(cursor, 'base64url').toString('utf8'))
+  } catch {
+    throw new BadUserInputError(INVALID)
+  }
+  if (
+    typeof payload !== 'object' ||
+    payload === null ||
+    Array.isArray(payload)
+  ) {
+    throw new BadUserInputError(INVALID)
+  }
+
+  const {
+    field: f,
+    direction: d,
+    id,
+    ...values
+  } = payload as Record<string, unknown>
+  if (f !== field || d !== direction) {
+    throw new BadUserInputError(
+      `The cursor was made for another sort (${String(f)} ${String(d)}), not ${field} ${direction}. Start again without a cursor.`,
+    )
+  }
+  const sort = SORTS[field]
+  const sameKeys =
+    Object.keys(values).sort().join() === [...sort.keys].sort().join()
+  if (!sameKeys || !isText(id) || id === '' || !sort.isValid(values)) {
+    throw new BadUserInputError(INVALID)
+  }
+  return { values: values as SortValues, id }
+}