Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -247,3 +247,6 @@ sketch

# End of https://www.toptal.com/developers/gitignore/api/node,react,intellij+all
# Thanks for reviewing

# To allow for local versions of the dataset CSV files, without adding it to Git
**.csv
218 changes: 218 additions & 0 deletions Database/AI-context.md

Large diffs are not rendered by default.

15 changes: 15 additions & 0 deletions Database/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# Database

The `Database` folder holds the Discogs-derived dataset used by the project:

| Item | What it is |
|---|---|
| `schema.sql` | PostgreSQL schema for the subset: 10 tables (releases, tracks, artists, aliases, members, groups, genres/styles, name variations) |
| `dataset-subset.sql` | Generated INSERT file: the catalogues of 15 major artists (The Beatles, David Bowie, Queen, Pink Floyd, Miles Davis, ...), deduplicated to one release per album, preferring the original vinyl |
| `Notes.md` | Full documentation: CSV parsing pitfalls, artist selection, dedup rules, ID generation, validation results |
| `scripts/` | The pipeline. One command from dump to database: `bash scripts/make_subset.sh` (see Notes.md §9.5) |
| `discogs_20251201_*.csv` | The raw Discogs dump (~35 GB, gitignored — not in the repo, needed only to regenerate) |

The subset covers 15 artists, ~12,500 releases (one pressing per album, original vinyl preferred), ~268,000 tracks, and is validated against the schema with foreign keys enabled before every commit of the file. Track positions are verbatim Discogs positions (`A1`, `1`, `CD1-03`, ...).

Later on in the project, we will de-duplicate even further, but for now this will do.
Loading