diff --git a/AGENTS.md b/AGENTS.md index 1b53aa6..6a71e04 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,22 +4,22 @@ - **Name:** UFOund it - **What it is:** UFOund it, is a webapp that displays UFO sightings. -- **Dataset:** [NUFORC UFO Sightings (Kaggle)](https://www.kaggle.com/datasets/NUFORC/ufo-sightings), CSV, approx 80,000 reports, approx 15mb. +- **Dataset:** [NUFORC](https://nuforc.org/) UFO sightings, CSV in `backend/data/ufo_sightings_scrubbed.csv` (the Kaggle page is gone), approx 80,000 reports, approx 15mb. - **Current state:** The backend is an Apollo Server with `sightings`, `sighting` and `createSighting`, backed by MySQL on the VM. The search and detail pages fetch sightings through Apollo Client, with search, sort and "load more" pagination done in the database. A temporary footer shows whether the backend is reachable. - **Course context:** NTNU IT2810 project. Must be reproducible, accessible and sustainable. ## Tech stack -| Area | Choice | -| -------------------- | ------------------------------------------------------------------------- | -| Runtime | Node.js 22 or newer (`engines` in package.json) | -| Frontend | React, TypeScript, Vite, React Router | -| API | GraphQL (custom backend, Node + TypeScript, Apollo Server) | -| GraphQL client | Apollo Client | -| Database | MySQL (leaning, not final), run directly on the VM | -| Testing | Vitest + React Testing Library (components), e2e tool and API testing TBD | -| Linting / formatting | ESLint, Prettier | -| UI libraries | Leaflet with react-leaflet (OpenStreetMap tiles, no API key) | +| Area | Choice | +| -------------------- | ------------------------------------------------------------------------------------- | +| Runtime | Node.js 22 or newer (`engines` in package.json) | +| Frontend | React, TypeScript, Vite, React Router | +| API | GraphQL (custom backend, Node + TypeScript, Apollo Server) | +| GraphQL client | Apollo Client | +| Database | MySQL, run directly on the VM | +| Testing | Vitest + React Testing Library (components), API tests with node --test, e2e tool TBD | +| Linting / formatting | ESLint, Prettier | +| UI libraries | Leaflet with react-leaflet (OpenStreetMap tiles, no API key) | ## Commands @@ -32,6 +32,13 @@ npm install # Start frontend dev server (proxies /project2/graphql to the backend) npm run dev +# Set up the database on the VM (once). Installs MySQL, creates the tables and +# imports the dataset. Full steps are in README.md. +bash backend/db-installation/my-sql-download.sh +python3 backend/data/cleanAndInsertDatatoDb.py verify +python3 backend/data/cleanAndInsertDatatoDb.py clean +python3 backend/data/cleanAndInsertDatatoDb.py insert + # Start backend dev server (GraphQL on http://localhost:3001, restarts on save) npm run dev:backend @@ -73,7 +80,7 @@ npm run test:watch ``` frontend/ src/ - components/ Reusable UI (Banner, SearchBar, SightingList, SightingMap, BackendStatus) + components/ Reusable UI (Banner, SearchBar, SightingList, SightingMap, LocationPicker, BackendStatus) pages/ Route-level components (SearchPage, SightingDetailPage, SubmitSightingPage) hooks/ Custom hooks (debounce, sighting search, single sighting) utils/ Pure helpers (formatting) @@ -85,9 +92,15 @@ backend/ Node + TypeScript GraphQL server (Apollo Server 5) index.ts Starts the server on PORT (default 3001) schema.ts GraphQL type definitions and resolvers validate-sighting.ts Server-side validation for createSighting + schema.test.ts, schema.db.test.ts API tests (faked and real database) db/ MySQL connection pool and sighting queries + db-design/ create-tables.sql, the database schema + db-installation/ Script that installs MySQL on the VM + data/ NUFORC CSV and the script that cleans and imports it dist/ Build output from `npm run build` (git-ignored) -# docs/ +.github/workflows/ CI/CD (checks, then deploy on merge to main) +deploy/ Apache, systemd and sudoers files for the VM +docs/ ci-cd.md: CI, deploy, VM setup and rollback ``` ## Code style @@ -144,6 +157,8 @@ Never: - **Backend port:** 3001 - **GraphQL endpoint:** The frontend calls `/project2/graphql` on its own origin. Apache must proxy it to `http://localhost:3001/graphql`, like the Vite dev proxy does locally. `VITE_GRAPHQL_URL` overrides the URL at build time if needed. - **How to deploy:** Automatic. The `deploy` job in CI deploys every push to `main` that passes `checks`, and GitHub tracks it as the `production` environment. The frontend goes to `/var/www/html/project2`, the backend runs from `/srv/ufound-it` as the systemd service `ufound-it-backend`. One-time VM setup is in `docs/ci-cd.md`. +- **Backend settings on the VM:** `/srv/ufound-it/backend/.env`, readable with `sudo` (variables in `backend/.env.example`). CI never changes it, and it must never be committed. +- **Bad deploy:** roll back as described in `docs/ci-cd.md`. ## AI usage documentation diff --git a/README.md b/README.md index 855c55d..d00f268 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,7 @@ Repository for team IT2810-H26-T25 in IT2810-H26. 1. [Running the Project](#running-the-project) 2. [Working in this repository](#working-in-this-repository) 3. [Project Description](#project-description) -4. [Planned Functionality](#planned-functionality) +4. [Functionality](#functionality) 5. [GraphQL API Design](#graphql-api-design) 6. [Stored User-Generated Data](#stored-user-generated-data) 7. [Technology Choices](#technology-choices) @@ -15,20 +15,108 @@ Repository for team IT2810-H26-T25 in IT2810-H26. ## Running the Project -To get a feel for how the application will look, you can run the project -locally. You need [Node.js](https://nodejs.org/) and npm installed. From the -repository root: +The project has three parts: a MySQL database, a GraphQL backend and a React +frontend. The live version runs at http://it2810-25.idi.ntnu.no/project2/. + +### Requirements + +- [Node.js](https://nodejs.org/) 22 or newer, with npm. +- For the database: MySQL 8 and Python 3 on a Linux machine (we use the course + VM). + +### 1. Clone and install ```bash +git clone +cd npm install +``` + +### 2. Set up the database (once, on the VM) + +The scripts need `sudo` and the `mysql` client. Run them from the repository +root: + +```bash +# Installs MySQL, creates the database project_2_db and the user project_2_app +# with a random password, and creates the tables. The password is saved in +# ~/.config/project_2/mysql.cnf. +bash backend/db-installation/my-sql-download.sh + +# Checks and cleans the NUFORC CSV (about 80,000 rows), then imports it. +python3 backend/data/cleanAndInsertDatatoDb.py verify +python3 backend/data/cleanAndInsertDatatoDb.py clean +python3 backend/data/cleanAndInsertDatatoDb.py insert +``` + +`insert` reads the password from `~/.config/project_2/mysql.cnf`. Use +`--mysql-config` to point it at another file. + +### 3. Start the backend + +The backend reads its database settings from `backend/.env`, which is not +committed. Copy `backend/.env.example` to `backend/.env` and set `DB_PASSWORD` +to the password from `~/.config/project_2/mysql.cnf`. Then: + +```bash +npm run dev:backend +``` + +GraphQL is now served on http://localhost:3001/graphql. To run the built +version, as on the VM, use `npm run build` and then `npm run start -w backend`. + +**Running locally against the VM's database.** Instead of installing MySQL, +open an SSH tunnel and keep the terminal open (needs the NTNU network or VPN): + +```bash +ssh -L 3306:localhost:3306 @it2810-25.idi.ntnu.no +``` + +Then use `DB_HOST=localhost` in `backend/.env` as usual. + +### 4. Start the frontend + +In a second terminal, from the repository root: + +```bash npm run dev ``` -Then open the address Vite prints in the terminal, usually -http://localhost:5173. For this first submission there is no backend or -database to set up, because the app runs on example data stored in the -frontend. Other useful commands are `npm run build`, `npm run lint` and -`npm run test`. +Open the address Vite prints, usually http://localhost:5173/project2/. Vite +forwards `/project2/graphql` to the backend, so no CORS setup is needed. The +footer shows "Backend: unavailable" if the backend is not running or is on +another port. If port 3001 is taken, run `PORT=3002 npm run dev:backend` and +`BACKEND_PORT=3002 npm run dev`. + +### Other commands + +| Command | What it does | +| ---------------------- | ---------------------------------------------------------- | +| `npm run build` | Type-checks and builds the frontend and the backend. | +| `npm run lint` | ESLint for the frontend and the backend. | +| `npm run format:check` | Prettier check for the whole repository. | +| `npm run test` | Frontend and backend tests. No backend or database needed. | + +The API tests that run against the real database are described in +[AGENTS.md](AGENTS.md#commands). + +### Starting the backend on the VM + +Everything needed to start the backend is in this repository and on the VM: + +- The backend runs as the systemd service `ufound-it-backend` from + `/srv/ufound-it`, on port 3001. Apache forwards + `/project2/graphql` to it. Check it with + `sudo systemctl status ufound-it-backend` and + `journalctl -u ufound-it-backend -n 50`. +- The database settings are in `/srv/ufound-it/backend/.env` (readable with + `sudo`). The variables are listed in `backend/.env.example`. The database + password is also in `~/.config/project_2/mysql.cnf` for the account that ran + the install script. +- To start the backend by hand instead: + `cd /srv/ufound-it/backend && /srv/ufound-it/node --env-file=.env dist/index.js`. +- How the VM is set up, and how every merge to `main` is deployed, is in + [docs/ci-cd.md](docs/ci-cd.md). ## Working in this repository @@ -57,10 +145,10 @@ UFOund it is a web application for exploring reported UFO sightings. You can see where people have observed UFOs, what the objects looked like and what the observers wrote about them. -The data comes from the -[UFO Sightings dataset on Kaggle](https://www.kaggle.com/datasets/NUFORC/ufo-sightings), -which contains about 80,000 reports collected by the National UFO Reporting -Center (NUFORC). Each report has a date and time, city, state, country, the +The data is a dataset of about 80,000 reports collected by the National UFO +Reporting Center ([NUFORC](https://nuforc.org/)). We got it from Kaggle, but +the dataset page has been removed, so the CSV file we use is kept in this +repository as `backend/data/ufo_sightings_scrubbed.csv`. Each report has a date and time, city, state, country, the shape of the object, how long the sighting lasted, a free text comment from the observer, the date it was posted and the latitude and longitude of the location. @@ -72,42 +160,39 @@ shapes and dates are natural things to sort and filter on. With 80,000 rows the dataset is also large enough that searching, sorting and paging have to happen on the server, which is what the course asks for. -For this first submission the application runs on a small set of hand written -example sightings stored in the frontend. There is no backend or database yet. +The sightings are stored in a MySQL database on the course VM and served +through a GraphQL API. Search, sorting and paging happen in the database, so +the browser only loads what it shows. -## Planned Functionality - -**Featured sighting.** The landing page shows a featured sighting of the day -with its location on a map, its shape and the observer's comment. +## Functionality **Search.** A search field finds sightings by city, shape or words in the comment. The search is case insensitive and waits until the user stops typing -before it runs, so it does not send a request for every keystroke. This already -works on the example data. +before it runs, so it does not send a request for every keystroke. **List view.** Search results are shown as a list where each sighting has a -small map, the location, date, shape, duration and comment. Clicking anywhere on -a sighting opens its detail page. Since a search can match thousands of -sightings, the list will either be split into pages or load more results as the -user scrolls. We have not decided which yet. +small map, the location, date, shape, duration and comment. The list loads 20 +sightings at a time, and a "Load more" button fetches the next page. Clicking +anywhere on a sighting opens its detail page. **Detail view.** Each sighting has its own page with a larger map that can be dragged and zoomed, along with all the information about the report. The page has its own URL, so a sighting can be shared or bookmarked. -**Sorting and filtering.** Results can be sorted by newest or oldest first, or -alphabetically by city or shape. The search and sort choices are kept in the -URL, so they are still there when the user returns from a detail page or -refreshes. +**Sorting.** Results can be sorted by newest or oldest first, or +alphabetically by city or shape. The sorting runs in the database, so it covers +all results and not only the ones loaded. The search and sort choices are kept +in the URL, so they are still there when the user returns from a detail page +or refreshes. -**Adding sightings.** Users will be able to report their own sightings, which -are then stored in the database and shown alongside the ones from the dataset. +**Adding sightings.** Users can report their own sightings with a form that +includes a map to pick the location. The server validates the input, and the +sighting is stored in the database and shown alongside the ones from the +dataset. -Right now the search, sorting and filtering run in the browser on the example -data, so we can try out how it feels. In the finished version this will be -replaced by GraphQL queries against the backend, so that sorting and filtering -always cover the whole dataset and not only what has been loaded in the -browser. +**Not done yet.** The API can filter by shape, country, state and date range, +but the interface has no filter controls yet. End-to-end tests are also still +missing. ## GraphQL API Design @@ -131,15 +216,13 @@ what it shows, and Apollo Client can merge the pages into one list. The trade-of ## Stored User-Generated Data -The main data users create are their own sighting reports. A report will have +The main data users create are their own sighting reports. A report has the same fields as the sightings in the dataset, which means location, date and time, shape, duration and a comment. These are stored permanently in the database. -We have not yet decided whether users will log in. Without accounts, reports -are stored without being tied to a user. With accounts, users could see and -manage their own reports, and we could later add comments or up votes on -sightings, which is an idea we would like to explore if there is time. +There are no user accounts, so reports are stored without being tied to a +user. They are marked as user submitted, to tell them apart from the dataset. (You cannot tell this in the frontend yet) ## Technology Choices @@ -148,32 +231,17 @@ navigation between the search page and the detail pages. Maps are made with Leaflet through react-leaflet, using map tiles from OpenStreetMap, which is free and needs no API key. -The backend will be a Node.js server written in TypeScript with a GraphQL API. -We plan to use Apollo Server on the backend and Apollo Client in the frontend, -since Apollo is well documented and handles caching of query results for us. -For the database we are leaning towards MySQL, but this is not final. The -backend and database will run directly on the course virtual machine, without -Docker. +The backend is a Node.js server written in TypeScript with a GraphQL API. We +use Apollo Server on the backend and Apollo Client in the frontend, since +Apollo is well documented and handles caching of query results for us. The data +is stored in MySQL. The backend and database run directly on the course virtual machine. -We will test components with Vitest and React Testing Library, and add end to -end tests and API tests as the application grows. ESLint and Prettier keep the +We test components with Vitest and React Testing Library, and the GraphQL API +with Node's built-in test runner, both against a faked database and against the +real one. End to end tests are still to come. ESLint and Prettier keep the code consistent, and every pull request is reviewed by another group member before it is merged into main. -## Running the backend against the VM database - -The backend reads its database settings from `backend/.env`, which is not -committed. To run it locally against the database on the course VM: - -1. Open an SSH tunnel and keep the terminal open. `ssh -L 3306:localhost:3306 @it2810-25.idi.ntnu.no` - (needs NTNU network or VPN). -2. Copy `backend/.env.example` to `backend/.env` and fill in `DB_PASSWORD` and - `DB_NAME` -3. Run `npm run dev:backend`. - -On the VM the same `.env` file is used with `DB_HOST=localhost`, and the -backend is started with `npm run start -w backend`. - ## AI Usage We use AI coding assistants, mainly Claude Code, to generate code, fix bugs and diff --git a/docs/ci-cd.md b/docs/ci-cd.md index b04df4b..db7c714 100644 --- a/docs/ci-cd.md +++ b/docs/ci-cd.md @@ -33,7 +33,7 @@ The workflow has one job, `checks`. It runs on our | `npm ci` | Installs exactly the versions in `package-lock.json`. | | `npm run lint` | ESLint for the frontend and the backend. | | `npm run format:check` | Prettier formatting for the whole repository. | -| `npm test` | Vitest unit and component tests. | +| `npm test` | Frontend (Vitest) and backend (node --test) tests. | | `npm run build` | Type-checks and builds the frontend and the backend. | ### The self-hosted runner @@ -57,10 +57,10 @@ workflow uses `actions/setup-node`. Old builds pile up in ### What it does not do yet -- There are no end-to-end tests or API tests, because we have not chosen the - tools yet. The backend has no `test` script, so `npm test` skips it. -- It does not run against a database. The Python import script in - `backend/data` is not tested by CI. +- There are no end-to-end tests, because we have not chosen the tool yet. +- It does not run against a database. The API tests that need one + (`backend/src/schema.db.test.ts`) skip themselves in CI, and the Python import + script in `backend/data` is not tested by CI. ## Merging @@ -160,8 +160,9 @@ sudoers files if it does not. ### Rolling back a bad deploy -The site check only tests that the backend starts and can read from the -database, so a merge can pass it and still break a page. Rolling back puts the +The site check only tests that the backend answers, can read from the database +and that the start page loads, so a merge can pass it and still break another +page. Rolling back puts the last good version back on the VM. There are two ways, and you usually do both: the first to get the site working within minutes, the second so the next merge does not deploy the broken code again. @@ -171,14 +172,19 @@ does not deploy the broken code again. 1. Open _Deployments_ on the repository page, or the `production` environment. It lists every deploy with its commit. Find the last one before the bad merge. -2. Open the `CI/CD` run on `main` for that commit in the _Actions_ tab (runs - before #73 are listed as `CI`) and click _Re-run all jobs_. +2. Open the `CI/CD` run on `main` for that commit in the _Actions_ tab and + click _Re-run all jobs_. 3. A re-run builds and deploys the commit of the original run, not the latest `main`. When the `deploy` job is green, the old version is live again. -This does not change `main`, so the bad commit is still there and the next -merge deploys it again. Always follow up with step 2. GitHub only re-runs runs -from the last 30 days; for anything older, use step 2 alone. +This only works for deploys from run #73 onward. Earlier runs (listed as `CI`) +have no `deploy` job and no `production` deployment, and runs before #70 did +not deploy at all, so there is nothing to re-run. For anything older, use step +2 alone. + +Re-running does not change `main`, so the bad commit is still there and the +next merge deploys it again. Always follow up with step 2. GitHub also only +re-runs runs from the last 30 days; for anything older, use step 2 alone. **2. Revert the merge on `main` (permanent).**