Skip to content

Conversation

ronjalh
Copy link
Member

@ronjalh ronjalh commented Oct 17, 2026

Summary

Adds a separate script that fills the catalog with fake items, so the search can be tested at a size where indexes matter. Closes #25.

What's in it

  • npm run db:synthetic -- add [--count N] adds 100 000 items by default (75 % elements, 25 % vaults), with SYN- ids in synthetic groups of their own (SYN-UB-1001 and so on), and the search tables for them.
  • npm run db:synthetic -- remove deletes them again. It only touches rows with a SYN- id.
  • The normal seed never runs it, and leaves SYN- rows alone (it already did).
  • add runs in one transaction and refuses to run when synthetic data already exists.
  • The generator (syntheticData.ts) copies descriptions from the real catalog, so the word distribution is realistic, and gives the same rows every time. It has tests without a database.
  • README section "Synthetic test data and measurements" with the search times with and without indexes, deep paging, and EXPLAIN (ANALYZE, BUFFERS) results.

Measured (100 562 entries, 1.4 million words, PostgreSQL 18)

Page of 20 with indexes: 1–2 ms for browsing, filters and one-letter searches, 10 ms for "kip" together with a filter. Without indexes: 12–52 ms. A page 501 into a result costs the same as the first page with indexes (about 2 ms). The full tables are in the README.

Testing

  • npm run lint, tsc --noEmit and the 52 tests pass against a database (0 skipped).
  • Added 100 000 and 50 000 items to a throwaway PostgreSQL 18 database, checked that a second add is refused, that the seed leaves the rows alone, and that remove brings the tables back to the real catalog (562 entries, 8 896 words).

Notes

  • The measurements are from one laptop and one run; use them to compare, not as promises.
  • The AI-usage entry is in docs/AI_USAGE.md.

🤖 Generated with Claude Code

npm run db:synthetic -- add [--count N] adds fake items with SYN- ids in
groups of their own (100 000 by default), and -- remove deletes them
again. The normal seed never runs it and leaves the rows alone. The README
has the search times with and without indexes, and what EXPLAIN shows.
@ronjalh ronjalh requested a review from a team as a code owner October 17, 2026 07:51
@ronjalh ronjalh self-assigned this Oct 17, 2026
Sign in to join this conversation on GitHub.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant