test(e2e): add automated end-to-end tests

.gitignore

@@ -35,3 +35,7 @@ scripts/extract-cop/en_1.1.-WAG-Code-of-Points-2025-2028.pdf
 scripts/extract-cop/review.html
 scripts/extract-cop/.venv
 __pycache__
+
+# Playwright (end-to-end tests)
+playwright-report/
+test-results/

README.md

@@ -512,6 +512,47 @@ The tables take 63 MB (`catalog_entries`) and 222 MB (`catalog_words`) with thei
 
 Run `ANALYZE` after adding or removing many rows, or PostgreSQL plans from old statistics (`npm run db:synthetic` does not run it). To repeat the measurement, call `searchCatalog` from a small script and use `EXPLAIN (ANALYZE, BUFFERS)` on the queries Prisma logs (`log: [{ emit: 'event', level: 'query' }]`). The application code never uses raw SQL, but `EXPLAIN` is typed by hand in `psql`.
 
+## End-to-end tests
+
+The assignment asks for "a form of automated end-to-end testing" of "a longer sequence of interactions", and for testing of the API. The component tests (Vitest) and the API tests in `backend/src/server/` already exist; the end-to-end tests drive the **whole application in a real browser**, against the real backend and the real database, the way a user does. They are in `e2e/` and use [Playwright](https://playwright.dev), with [axe-core](https://github.com/dequelabs/axe-core-npm/tree/develop/packages/playwright) for the accessibility checks.
+
+| File                       | What it tests                                                                                                                                                                                                               |
+| -------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `e2e/elements-flow.e2e.ts` | One long flow: search, filter by apparatus and group, "Load more", sort, open an element, go back, "Clear filters". It checks the address, the results, the title and the focus at each step.                               |
+| `e2e/vault.e2e.ts`         | The vault difficulty bands, and sorting vault by its D-score in both directions.                                                                                                                                            |
+| `e2e/atypical.e2e.ts`      | Atypical use: an address with filters that do not exist, a search without a result, the 80-character limit, a structure that fails to load (the request is made to fail), the disabled group filter, and keyboard-only use. |
+| `e2e/accessibility.e2e.ts` | axe-core on the page in its main states: filtered and sorted, a menu open, the reason for a disabled filter shown, vault, no result, dark mode, a phone (375 px) and reflow at 320 px. A violation fails the test.          |
+| `e2e/api.e2e.ts`           | The GraphQL API over HTTP: search, filter, sort, cursors, the error codes and `Apparatus.difficultyValues`. The same requests can be sent to the virtual machine.                                                           |
+
+### How to run them
+
+They need the database to be running and seeded (see "Database (local setup)" above), and a browser:
+
+```bash
+npm ci
+npm run generate                  # the Prisma and GraphQL types, once
+npx playwright install chromium   # the browser, once (about 150 MB)
+npm run e2e
+```
+
+`npm run e2e` starts the backend and the frontend itself (and uses them if they are already running), runs the tests, and prints the result. `npm run e2e:report` opens the report with a screenshot and a trace of every test that failed. The tests only read the catalog, so they can be run again and again.
+
+Settings, all optional, as environment variables:
+
+| Variable                                | Meaning                                                                                                                                                                                                        |
+| --------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `E2E_BASE_URL`                          | An address that is already running, for example `http://it2810-29.idi.ntnu.no/project2`. Nothing is started, and the same tests check the real deployment (including the Apache proxy of `/project2/graphql`). |
+| `E2E_FRONTEND_PORT`, `E2E_BACKEND_PORT` | The ports that the tests start the app on, 5173 and 3001 by default. Change them if another program uses them, or the tests would use that program.                                                            |
+| `E2E_CHANNEL`                           | A browser that is already installed, for example `msedge`, instead of the Chromium that Playwright downloads.                                                                                                  |
+| `E2E_API_URL`                           | The GraphQL address of `e2e/api.e2e.ts`, if it is not the one that follows from the others.                                                                                                                    |
+
+### Choices and limits
+
+- **Playwright, not Cypress.** It is made for end-to-end tests, drives real Chromium, waits for the page by itself (a test says what it expects, and Playwright retries until it is true), and has an official axe-core integration. The tests are TypeScript like the rest of the project, and the files end in `.e2e.ts` so that Vitest does not pick them up.
+- **The tests depend on the seeded catalog** (the data in `data/`). They avoid exact numbers where a relation is enough ("fewer than the whole catalog", "every code starts with the number of the group"), but they do assume that uneven bars, floor and vault exist with their groups and difficulty values.
+- **The accessibility tests are automated checks, not a full audit.** axe-core finds what a program can find. What needs a person, such as a screen reader (NVDA, VoiceOver) and whether the order of the focus makes sense, has to be tested by hand. `region` is turned off for a state with a menu open: the menus are in a portal (the Base UI default), which axe reports as best practice, not as a WCAG failure.
+- **They are not run by a server on every pull request.** The course git server has no runners (see issue #76), so they are run by the person who makes the change, and before each delivery against the virtual machine with `E2E_BASE_URL`.
+
 ## Project structure
 
 | Folder                 | Contents                                                                                                                                               |
@@ -521,6 +562,7 @@ Run `ANALYZE` after adding or removing many rows, or PostgreSQL plans from old s
 | `docs/`                | The specification: user interface plans, data model, example data and AI usage. [docs/EXTRACTION.md](docs/EXTRACTION.md) explains how `data/` was made |
 | `public/`              | Static files served as they are, such as the favicon                                                                                                   |
 | `scripts/extract-cop/` | The Python script that extracts `data/` from the Code of Points PDF                                                                                    |
+| `e2e/`                 | The end-to-end tests (Playwright): whole flows in a real browser, accessibility checks and the API over HTTP. See "End-to-end tests"                   |
 | `src/`                 | The frontend code: the pages, the Apollo client, the components and their tests                                                                        |
 
 ## Working in this repository

docs/AI_USAGE.md

@@ -507,3 +507,10 @@
   - **Accessibility check of the page** with axe-core, Lighthouse, keyboard-only use, contrast, target sizes, reflow, text spacing and reduced motion. Fixed what was in this pull request (focus lost after "Clear filters", the reason for a disabled filter not visible on touch, the page title) and listed the rest. A screen reader was not used: that has to be done by a person.
   - **Fixed the review point:** when the structure of the filters cannot be loaded and the address has filters, the page shows the error with "Try again" and does not search.
 - **What was not done by Claude:** nothing was posted or replied to on git without being asked, and the choice to offer bands for vault, the wording of the sort list and which findings to fix were the team's.
+
+## End-to-end tests (`e2e/`, `playwright.config.ts`)
+
+- **Tool:** Claude Code
+- **What the team did:** Read the requirement of the assignment ("a form of automated end-to-end testing ... a longer sequence of interactions, and testing of the API") and wrote issue #139. Decided that the tests are run against the virtual machine too, and that atypical use and accessibility are part of them.
+- **What Claude did:** Checked what the requirement means and what the repository already had (component tests and API tests, but no test that drives the whole application). Chose Playwright (ready-made, waits for the page by itself) with the official axe-core integration, and wrote the configuration so that the tests start the app themselves or run against any address (`E2E_BASE_URL`). Wrote one long flow, the vault bands, atypical use (wrong addresses, no result, a failing request, keyboard only, the disabled filter), axe-core checks of the page in its main states, and tests of the API over HTTP. Ran them against the real backend and the real catalog with Playwright's own Chromium and with Edge, and broke the application on purpose in four places to check that a test fails each time. Wrote the README section, including what the tests do not cover (a screen reader, and a run on every pull request, because the course server has no runners).
+- **What was found:** The first run had 4 failing tests, all in the tests and not in the application: Playwright does not click a control that is `aria-disabled`, one expectation was wrong, and a check ran before the new results had arrived (the page keeps the old results on the screen until the new ones are there).

docs/API.md

@@ -493,7 +493,7 @@ Vitest runs the API tests against a test database, with queries sent to the serv
 
 **Decision: the test files run one at a time (`fileParallelism: false` in `vite.config.ts`).** The database tests share one local database, and some of them write to it (the login tests) while others check that the row counts do not change, so two files at the same time would break each other. The price is that the frontend tests also run one file at a time. We keep it because the suite is still small and fast (the frontend and non-database tests finish in about a second), so the loss is small for now. Running only the database tests one at a time (for example as a separate Vitest project, or with a database per test file) would remove the price, but it is more setup than the gain is worth now. Revisit it when the suite gets slow.
 
-The end-to-end tests (a long sequence of actions in the browser) are described separately in the README when they exist.
+The end-to-end tests (a long sequence of actions in the browser, and the API over HTTP against any address, also the virtual machine) are in `e2e/` and are described in "End-to-end tests" in the README.
 
 ## Open questions
 

e2e/accessibility.e2e.ts

@@ -0,0 +1,148 @@
+// axe-core checks the Elements page against WCAG 2.0, 2.1 and 2.2 level A and AA, in the
+// states where something is different: filtered and sorted, a menu open, the reason for a
+// disabled filter shown, dark mode and a phone. A violation fails the test. What axe cannot
+// decide (a screen reader, the order of the focus) is described in the README.
+
+import AxeBuilder from '@axe-core/playwright'
+import { expect, test, type Page } from '@playwright/test'
+import { openMenu, resultsLoaded } from './helpers.ts'
+
+const TAGS = ['wcag2a', 'wcag2aa', 'wcag21a', 'wcag21aa', 'wcag22aa']
+
+/**
+ * `region` is axe's best-practice rule that all content is in a landmark. The menus are in
+ * a portal at the end of the document (the Base UI default, so that they are not clipped),
+ * so it is turned off for a state with a menu open. It is not a WCAG failure.
+ */
+async function violations(page: Page, options: { menuOpen?: boolean } = {}) {
+  const builder = new AxeBuilder({ page }).withTags(TAGS)
+  if (options.menuOpen) builder.disableRules(['region'])
+  const { violations } = await builder.analyze()
+  return violations.map((v) => ({
+    rule: v.id,
+    impact: v.impact,
+    nodes: v.nodes.map((n) => n.target.join(' ')),
+  }))
+}
+
+const FILTERED =
+  'elements?apparatus=UB&group=UB-6&difficulty=A,B&sort=difficulty&dir=desc'
+
+test.describe('light mode', () => {
+  test('the page with the whole catalog', async ({ page }) => {
+    await page.goto('elements')
+    await resultsLoaded(page)
+    expect(await violations(page)).toEqual([])
+  })
+
+  test('filtered and sorted', async ({ page }) => {
+    await page.goto(FILTERED)
+    await resultsLoaded(page)
+    expect(await violations(page)).toEqual([])
+  })
+
+  test('with a filter menu open', async ({ page }) => {
+    await page.goto(FILTERED)
+    await resultsLoaded(page)
+    await openMenu(page, /^Difficulty/)
+    expect(await violations(page, { menuOpen: true })).toEqual([])
+  })
+
+  test('with the sort menu open', async ({ page }) => {
+    await page.goto(FILTERED)
+    await resultsLoaded(page)
+    await openMenu(page, /Sort by/)
+    expect(await violations(page, { menuOpen: true })).toEqual([])
+  })
+
+  test('with the reason for the disabled group filter shown', async ({
+    page,
+  }) => {
+    await page.goto('elements')
+    await resultsLoaded(page)
+    await page
+      .getByRole('button', { name: 'Element group' })
+      .click({ force: true })
+    await expect(
+      page.getByText('Choose an apparatus to choose a group.').first(),
+    ).toBeVisible()
+    expect(await violations(page)).toEqual([])
+  })
+
+  test('vault, with the bands', async ({ page }) => {
+    await page.goto('elements?apparatus=VT&difficulty=2,3')
+    await resultsLoaded(page)
+    expect(await violations(page)).toEqual([])
+  })
+
+  test('a search without a result', async ({ page }) => {
+    await page.goto('elements?search=zzzqqqxxx')
+    await expect(page.getByText(/^No elements match/)).toBeVisible()
+    expect(await violations(page)).toEqual([])
+  })
+})
+
+test.describe('dark mode', () => {
+  test.beforeEach(async ({ page }) => {
+    // The app keeps the choice of theme here (App.tsx).
+    await page.addInitScript(() => localStorage.setItem('theme', 'dark'))
+  })
+
+  test('filtered and sorted', async ({ page }) => {
+    await page.goto(FILTERED)
+    await resultsLoaded(page)
+    await expect(page.locator('html')).toHaveAttribute('data-theme', 'dark')
+    expect(await violations(page)).toEqual([])
+  })
+
+  test('with a menu open', async ({ page }) => {
+    await page.goto(FILTERED)
+    await resultsLoaded(page)
+    await openMenu(page, /^Difficulty/)
+    expect(await violations(page, { menuOpen: true })).toEqual([])
+  })
+})
+
+test.describe('on a phone', () => {
+  test.use({ viewport: { width: 375, height: 800 }, hasTouch: true })
+
+  test('filtered and sorted, without sideways scrolling', async ({ page }) => {
+    await page.goto(FILTERED)
+    await resultsLoaded(page)
+    expect(await violations(page)).toEqual([])
+    const overflow = await page.evaluate(
+      () =>
+        document.documentElement.scrollWidth >
+        document.documentElement.clientWidth,
+    )
+    expect(overflow).toBe(false)
+  })
+
+  test('the controls are at least 40 px tall, so they can be pressed with a finger', async ({
+    page,
+  }) => {
+    await page.goto('elements')
+    await resultsLoaded(page)
+    const heights = await page
+      .locator('[role=group] button, button[aria-haspopup]')
+      .evaluateAll((buttons) =>
+        buttons.map((b) => Math.round(b.getBoundingClientRect().height)),
+      )
+    expect(heights.length).toBeGreaterThanOrEqual(4)
+    expect(Math.min(...heights)).toBeGreaterThanOrEqual(40)
+  })
+})
+
+test('the page reflows at 320 px wide (400 % zoom) without sideways scrolling', async ({
+  page,
+}) => {
+  await page.setViewportSize({ width: 320, height: 700 })
+  await page.goto(FILTERED)
+  await resultsLoaded(page)
+  const overflow = await page.evaluate(
+    () =>
+      document.documentElement.scrollWidth >
+      document.documentElement.clientWidth,
+  )
+  expect(overflow).toBe(false)
+})