24 add indexes and search

[anettkva]

Summary

Adds the catalog search function that the searchCatalog resolver will call. There is no GraphQL server on dev yet, so nothing calls it yet.

What it does

• Searches CatalogEntry and CatalogWord with Prisma only. A new lint rule fails the build if someone uses raw SQL
• Every search word must match the start of a word: "giant circle" finds giant circles, and "handstand" also finds descriptions that say "hstd"
• Filters by apparatus, element group, difficulty values and named items
• Sorts by code, difficulty or name, with cursor pagination (keyset, not offset)
• Lists the element groups of one apparatus
• Counts all matches only when asked for (totalCount)

Decisions

• The search uses the same word rules as the seed (normalizeWords), so "hop-grip", "sec." and "1/1" find what is stored. The search only adds removing a trailing "s".
• The cursor is plain base64url JSON, not encrypted. The catalog is public, so a changed cursor can only lead to another page of public data. It is checked strictly when it is decoded.
• Sorting by difficulty needs an apparatus filter, because element values and vault D-values can't be compared.
• Unknown filter values give BAD_USER_INPUT, not an empty list, so a typo in the client is noticed.

The reasons for each are in "Search without raw SQL" in docs/API.md.

How to test

1. Run npm run db:seed so the catalog is in your database
2. Run npm test
3. Check that the output says 0 skipped. Without DATABASE_URL in backend/.env, the search tests are skipped, and the run still passes

Closes #24

🤖 Generated with Claude Code

[ronjalh]

A couple of things to think about after Claude and I ran scalability tests:

• High: Which limits for the search? Claude's suggestion is 10 words and 100 characters. Now: none.
• Should text that gives no words (such as «ø») give an empty result, BAD_USER_INPUT, or still everything?
• Should whole numbers be exact matches (the point is mentioned in later messages), or do we keep prefix?
• High: (referred to elsewhere as priority 1 or point 1) Deep pages get slower the deeper you go (more about this in a separate comment)
• Med: two extra database calls per search page: (more about this in a separate point)
• The issue requires testing against a large catalog, no proof that it has been done.
• Difficulty filter with a vault group without an apparatus gives silent 0 hits.
• The rule «never mode: 'insensitive'» is not automatically tested. It has only been checked by hand. Can be enforced with a lint rule (no-restricted-syntax) or a test that checks the SQL.
• Text that gives no words returns everything. «ø», «ß», «日本語» and «😀» all give all entries, not none. Does not apply to the current catalog (only ½ ¾ ° –), but is surprising.
• The lint rule is missing $queryRawTyped (Prisma's TypedSQL).
• errors.ts is under catalog/ but is needed by all modules (NOT_FOUND, UNAUTHENTICATED …). Move to src/errors.ts.
• The sort logic is duplicated in four functions (orderBy, cursorValues, afterCursor, the keys in cursor). One table per sort makes it easier to add a sort and to fix point 1 in one place.

[ronjalh]

1: Deep pages get slower the deeper you go (name and difficulty). The documentation says «a page costs the same on page 1 and page 10 000». Measured: name ascending 2 ms at the start, 12 ms after 20 000, 48 ms after 150 000. Code is constant (2–3 ms). The cause is in EXPLAIN: the id condition ends up as a filter and not as an index bound (Rows Removed by Filter: 115 141). Rows that share the same sort value (all without a name have nameSort = '', and difficulty has only 10 values) are read from the start of the group every time. In the real catalog 69 % have no name.

• It is not noticeable with thousands of entries, but it contradicts the requirement «any amount of data», and issue Add indexes and search #24 has a checkbox for exactly this that is not documented.
• Solutions without raw SQL: (a) split the condition into two Prisma queries (equal values with id >, and then strictly greater values) and merge them, (b) store one unique sort key per sort in CatalogEntry, so the cursor becomes a single simple column (requires a change in the migration and seed).

[ronjalh]

No limit on the number of search words. 100 words took 0.7 seconds, and 300 words took 14 seconds in a single query. A single request can load the database. Needs a cap (for example 10 words and 100 characters) that gives BAD_USER_INPUT, and it must be in the documentation.

[ronjalh]

Two extra database calls per search page. Each page with an apparatus and group filter makes 2 lookups (3 queries instead of 1). The catalog is static, so the lookups can be cached in memory or done once.

[ronjalh]

I want to bring this up again in connection with search. I seem to remember that this was a deliberate decision, but I am just double-checking that we did this on purpose. If so, we should update the UI to have an indication that there are several variants, which we do not have now.
Background

In the Code of Points some elements have the same code and the same value, but several ways of performing them. Code UB-1.206 has two:

• UB-1.206-a: «Free straddle jump over LB to hang on HB» (with straight legs apart)
• UB-1.206-b: «Hecht jump (legs together) with hand repulsion over LB to hang on HB» (with the legs together)

In the database they are two separate rows, but in the search list only one row per code is shown, and it is always -a. The other variant is only shown when you open the detail page.

[ronjalh]

see commetns

[anettkva]

Thanks for the review! Here is what's done with each point, and why.

Changed

• Deep pages: the rows after a cursor are now read with separate queries, for example "same code, higher id" and then "higher code" (afterParts in sorts.ts). With one OR condition, PostgreSQL checked id as a filter and reread every row with the same sort value on every page (Rows Removed by Filter: 223 on one page). That's common here: 69 % of the items have no name, and there are only ten difficulty values. Now id is in the Index Cond, and a test checks that there is no OR.
• Search text limits: at most 80 characters and 8 words, otherwise BAD_USER_INPUT. Each word adds a query condition. The median description is 12 words, so normal typing never hits the limit.
• Text without searchable words (%, ø, s) now returns nothing instead of the whole catalog. Empty text still means no text filter.
• Apparatus and groups are cached in memory for 10 minutes, so the filter checks need no database call. The catalog is read-only, and the TTL lets a new seed show up without a restart.
• Vault group + difficulty filter now gives BAD_USER_INPUT instead of 0 results without saying why, the same as apparatusId: "VT".
• Lint: $queryRawTyped and mode: 'insensitive' are now forbidden in the backend.
• Structure: BadUserInputError moved to backend/src/errors.ts, so the resolvers can use it too. Each sort's logic is now in one table in sorts.ts, so adding a sort means changing one place.

Kept as it is

Numbers match by prefix
"3" finds 3.101, 3.2 and so on. That is the same rule as for words, and it lets the user search by a group number. An exact match for numbers would be a second rule to explain.

Not in this PR

Proof on a large catalog
I checked the query plans on the seeded catalog. Measuring with a large amount of data is sub-issue 5, so this PR does not add large test data.

Showing variants in the search list
This is a frontend change. variants is already in the API, so no backend change is needed. Showing one row per code was decided in #22, and the user picks the variant on the detail page. I suggest writing a separate issue for it.

Docs

docs/API.md now covers the limits, text without words, the cache, the vault group rule and the split keyset queries, each with the reason. The README and docs/AI_USAGE.md are updated too.

[ronjalh]

changes seems addressed

[ronjalh]

lgtm