IT2810-H26 · egilhp · Oct 16, 2026 · Oct 16, 2026 · Oct 16, 2026 · Oct 16, 2026
@@ -249,4 +249,7 @@ sketch
 # Thanks for reviewing
 
 # To allow for local versions of the dataset CSV files, without adding it to Git
-**.csv
+**.csv
+
+# Keeping docker data out of the github repo
+Database/dev-env/postgres-data
@@ -2,6 +2,10 @@
 
 The `Database` folder holds the Discogs-derived dataset used by the project:
 
+## Dataset
+
+The dataset is derived as follows
+
 | Item | What it is |
 |---|---|
 | `schema.sql` | PostgreSQL schema for the subset: 10 tables (releases, tracks, artists, aliases, members, groups, genres/styles, name variations) |
@@ -13,3 +17,7 @@ The `Database` folder holds the Discogs-derived dataset used by the project:
 The subset covers 15 artists, ~12,500 releases (one pressing per album, original vinyl preferred), ~268,000 tracks, and is validated against the schema with foreign keys enabled before every commit of the file. Track positions are verbatim Discogs positions (`A1`, `1`, `CD1-03`, ...).
 
 Later on in the project, we will de-duplicate even further, but for now this will do.
+
+## dev-env
+
+To allow for local testing, a `docker-compose.yml` file is present, allowing for easy creation of a PostgreSQL database matching the one in production.
@@ -151,7 +151,7 @@ Aliases/groups/members were included as potential GraphQL graph edges even thoug
 
 - **Extraartist-credited appearances are excluded** by design (see 4.2). A "Beatles member solo + related productions" dataset would need a second pass matching `extraartists_artist_id` (column 11) instead of column 1 — `scan_seq.pl` needs only the prefilter regex and check-column changed, plus the zipping in `jsonlify.pl`.
 - Labels file: `scan_labels.pl`-style pass was implemented and produced `labels.csv` (44,868 of 2,569,333 label rows, from `release_labels_label_id` col 24 + `companies_company_id` col 7) but the file was deleted per user decision. The approach is documented here in case labels are wanted later.
-- No deduplication was applied — if the dump contains duplicate releases, they are preserved.
+- No deduplication was applied — if the dump contains duplicate releases, they are preserved. (This refers to the raw extracted CSVs/JSONL; the SQL subset is deduplicated as described in section 9. Remaining SQL-level twins that title rules cannot safely merge are listed in section 9.3a: renamed reissues like "Kid A Mnesia", combined reissues like "Kid A / Amnesiac", numeric edition suffixes like "Thriller 25th", and abbreviation variants like "Vol.1" vs "Volume 1". Titles in non-Latin scripts collapse to their ASCII/digit remnants in the dedup key (e.g. a Greek and a Bulgarian compilation both key as "1"), which the per-artist credit dedupe keeps from becoming visible duplicates.)
 - Sorting: records are in original dump order (ascending `release_id` is *not* guaranteed across the file).
 - Broken UTF-8 fallback (5.1) may pass through invalid bytes in rare cells; the full JSON re-parse in validation covers structural validity, not charset validity.
 
@@ -168,16 +168,17 @@ It runs, in order: `scan_seq.pl` (releases, ~25 min), `scan_seq.pl` (masters, ~3
 
 When changing the artist list, the 15 IDs must be updated in `scan_seq.pl`, `scan_artists.pl`, and `subset_to_sql.pl`. Run from the folder containing the dump CSVs (paths are relative; `make_subset.sh` does this itself).
 
-## 9. dataset-subset.sql generation (2026-10-09)
+## 9. dataset-subset.sql generation (2026-10-09; dedup improved 2026-10-16 — see 9.1 and 9.3a)
 
 `dataset-subset.sql` (19 MB, in this folder) loads the 10 tables of `schema.sql` with a reduced, relational form of the same data. Generated by `scripts/subset_to_sql.pl` from the extracted CSVs (section 8 outputs); no JSONL step needed. The original extracted CSVs are regenerated on demand since `**.csv` is gitignored — `make_subset.sh` handles this.
 
 ### 9.1 Subset decisions (user-approved)
 
 - **One release per album**, via two stages inside `subset_to_sql.pl`:
   1. *Per master*: each release group contributes its best pressing (218,947 → 13,578; the count exceeds the 8,171 masters of the 15 artists because split/compilation releases can carry `master_id`s outside the artists' own masters).
-  2. *Per normalized title*: the stage-1 winners are grouped per artist by normalized title — lowercase, leading "The " dropped, punctuation removed, and **edition-type parentheticals ignored** ((Remaster), (Deluxe Edition), (25th Anniversary), ...). Content-bearing parentheticals ((Live), (Remixes), (Part 2), soundtrack names) stay in the key, so "Album (Live)" never merges with "Album". Titles could not be grouped directly across all releases: release titles are far messier than master titles, and direct grouping exploded into 73,865 groups.
-- **Winner within a title group**, in order: vinyl format > not tagged reissue/repress/remaster > format tagged "Album" (beats EPs/singles, e.g. the White Album over same-titled EPs) > earliest year > master's main release > lowest `release_id`. Verified cases: The Dark Side Of The Moon → 1973-03-24 original (release 1873013); The Beatles → 1968-11-22 White Album (456663); Station To Station → 1976-01-23 original. Known casualty of title dedup: Queen has two masters titled exactly "Greatest Hits"; the 1980 vinyl (3318494) wins over the 1981-10-26 UK one (376152) because it is the earliest vinyl.
+  2. *Per normalized title*: the stage-1 winners are grouped per artist by normalized title — lowercase, leading "The " dropped, punctuation removed, and **edition-type qualifiers ignored**. A bracketed segment (`(...)` or `[...]`) is dropped when it is a pure edition/format marker: the classic set ((Remaster), (Deluxe Edition), (25th Anniversary), (Reissue), (Repress), packaging like Digipak/Gatefold/SACD/180g), a year-only `(2011)`, format markers with track/song counts ((Sampler CD), (Dvd 1), (3 Track Promotional Sample), (Promo Only), (2016 Mix), (Bonus Cuts), (Mono)/(Vinyl), ...), or anything mentioning "broadcast". A **bare trailing edition word** is also dropped ("X Sampler", "X - 2011 Remaster", "Vogue Anniversary"). Content-bearing segments stay in the key — (Live), (Remixes), (Vol. 2), (Part 2), venue+year, soundtrack names, year ranges like "[2002–2016]" — so "Album (Live)" never merges with "Album" and distinct series volumes stay apart. Bare trailing "deluxe"/"mix" are deliberately NOT stripped ("Best Deluxe" and "The Michael Jackson Mix" are different releases, not editions). Titles could not be grouped directly across all releases: release titles are far messier than master titles, and direct grouping exploded into 73,865 groups.
+- **Winner within a title group**, in order: not a promo sampler (format/title tagged "Sampler" — a sampler must never beat the full release it promotes) > vinyl format > not tagged reissue/repress/remaster > format tagged "Album" (beats EPs/singles, e.g. the White Album over same-titled EPs) > earliest year > master's main release > lowest `release_id`. Verified cases: The Dark Side Of The Moon → 1973-03-24 original (release 1873013); The Beatles → 1968-11-22 White Album (456663); Station To Station → 1976-01-23 original; The Last Waltz → 2008 full 58-track release over the 1978 vinyl promo sampler; Capitol Albums Vol. 2 → 2006 full 104-track box over the 18-track sampler. Known casualty of title dedup: Queen has two masters titled exactly "Greatest Hits"; the 1980 vinyl (3318494) wins over the 1981-10-26 UK one (376152) because it is the earliest vinyl.
+- **One credit per (artist, album)** (added 2026-10-16): a release that won ANOTHER artist's title group is inserted once but is only credited to an artist whose title group it actually won. Without this, multi-artist compilations/splits showed up as same-title twins under every credited artist (e.g. two "Starman" entries for David Bowie, two "Heroes" entries). The output now satisfies the invariant *at most one release per (artist, normalized title)* — verified 0 violations on the final SQL.
 - **Track positions kept verbatim** (user decision): `A1`, `1`, `CD1-03`, ... unchanged. No vinyl-format conversion; vinyl releases already use vinyl positions. Tracks with no position get `''`; tracklist entries with no title are dropped (`tracks.title` is NOT NULL).
 - **release_artists restricted to the 15 artists** (user decision): co-credited main artists (e.g. Tony Sheridan on Beatles releases) are NOT in the `artists` table, so no links to them are emitted — satisfies the FK to `artists(id)`. `artists.name` is always populated; `real_name` is NULL when the dump has none (groups).
 - **sub_tracks dropped** (no parent linkage, no table in the schema). Masters have no table in the schema; `masters.csv` is used only for the `main_release_id` mapping.
@@ -187,7 +188,7 @@ When changing the artist list, the 15 IDs must be updated in `scan_seq.pl`, `sca
 
 The dump has no ids for genres, styles, or name variations: `genre_id`/`style_id` are assigned alphabetically over the distinct names in the kept releases (15 genres, ~360 styles — 1:1 with names, verified); `name_variation_id` is a per-artist 1..n sequence. `track_id` is a global 1..N sequence. Alias/member/group ids come from the dump (no fallbacks were needed for the 15 artists).
 
-### 9.3 Output stats (validated)
+### 9.3 Output stats (validated, 2026-10-16)
 
 | Table | Rows |
 |---|---|
@@ -196,17 +197,28 @@ The dump has no ids for genres, styles, or name variations: `genre_id`/`style_id
 | members | 63 |
 | groups | 49 |
 | artist_name_variations | 544 |
-| releases | 12,505 |
-| release_artists | 13,490 |
-| tracks | 268,159 |
-| release_genres | 22,748 |
-| release_styles | 27,406 |
+| releases | 12,483 |
+| release_artists | 13,450 |
+| tracks | 267,851 |
+| release_genres | 22,703 |
+| release_styles | 27,368 |
 
-Per-artist kept releases range from 237 (Aphex Twin) to 1,703 (The Beatles).
+Per-artist kept releases range from 236 (Aphex Twin) to 1,701 (The Beatles). `release_artists` rows equal the number of title groups (13,450) — a direct consequence of the one-credit-per-(artist, album) rule.
+
+### 9.3a The 2026-10-16 dedup improvement (v2 title normalization + credit dedupe)
+
+User report: the database still contained "twin" albums — same album, different editions (e.g. four Radiohead entries matching "Kid A"). Analysis of the 2026-10-09 output found two leak mechanisms: (1) edition qualifiers that escaped the v1 normalization — `[...]` brackets, format markers like (Sampler CD)/(Dvd 1)/(Promo Only)/(CD Edition), year-only parentheticals, bare trailing words ("X Sampler"), (Repress), (Broadcast); (2) cross-artist artifacts — multi-artist compilations that won one artist's title group while also being credited to another, leaving same-title twins (e.g. "Starman" ×2 for Bowie). The v1→v2 audit over the 13,578 stage-A winners: 183 title groups hold ≥2 distinct titles; 22 of them are newly merged by v2. Applied changes and measured effect on the final SQL (2026-10-09 → 2026-10-16):
+
+- 61 releases removed, 39 full-release counterparts added → 12,505 → 12,483 releases. Every removed release is an edition variant (sampler/promo/mix/format decoration) or a sampler-tagged pressing that lost stage A to the full pressing of the same master; no studio album or distinct compilation was lost (checked by diffing the release sets).
+- 18 duplicate (release, artist) credits dropped by the credit dedupe; release_artists 13,490 → 13,450.
+- User-visible fixes: "Best Of Bowie" 5 entries → 1 (1980 original); "Kid A (The World Premier Broadcast)" merged into "Kid A"; "Heroes" 2 → 1 for Bowie (1977 original kept); "Help!" 8 → 7; Queen "Greatest Hits (Longplay CD)" merged; samplers no longer beat full releases (The Last Waltz, Capitol Albums Vol. 2).
+- Deliberately NOT merged (kept separate): "Kid A / Amnesiac" and "Kid A Mnesia" (combined/renamed reissues — different or renamed content; would need tracklist-similarity clustering), "Thriller 25th" (numeric edition suffix), "The Capitol Albums Volume 1 Sampler" vs "The Capitol Albums Vol.1" (Vol/Volume abbreviation mismatch), "Madonna Mix"/"The Michael Jackson Mix" (megamixes are distinct releases, not editions), "Best Deluxe" vs "Best" and "Jazz Deluxe" vs "Jazz" (different compilations sharing a word).
 
 ### 9.4 Validation
 
-`scripts/validate_sql.mjs` (Node ≥ 22, uses built-in `node:sqlite`, no dependencies) loads `schema.sql` + `dataset-subset.sql` into an in-memory SQLite database with `PRAGMA foreign_keys = ON` and checks: load succeeds, `foreign_key_check` = 0 violations, `integrity_check` = ok, every release has ≥ 1 release_artists row, no release_artists rows outside the 15, no duplicate (release_id, genre/style) pairs, genre/style ids 1:1 with names. Run: `node scripts/validate_sql.mjs` from this folder. All checks passed on 2026-10-09. Read-only sanity helpers used while tuning the dedup rule: `analyze_dupes.mjs`, `analyze_groups.mjs`, `analyze_parens.mjs`, `inspect_formats.pl`.
+`scripts/validate_sql.mjs` (Node ≥ 22, uses built-in `node:sqlite`, no dependencies) loads `schema.sql` + `dataset-subset.sql` into an in-memory SQLite database with `PRAGMA foreign_keys = ON` and checks: load succeeds, `foreign_key_check` = 0 violations, `integrity_check` = ok, every release has ≥ 1 release_artists row, no release_artists rows outside the 15, no duplicate (release_id, genre/style) pairs, genre/style ids 1:1 with names. Run: `node scripts/validate_sql.mjs` from this folder. All checks passed on 2026-10-09 and again on 2026-10-16 after the dedup improvement. Read-only sanity helpers used while tuning the dedup rule: `analyze_dupes.mjs` (now ports the production v2 `norm_title` and reports the one-release-per-(artist, album) invariant), `analyze_groups.mjs`, `analyze_parens.mjs`, `inspect_formats.pl`.
+
+The invariant check (0 releases sharing an (artist, normalized-title) key, 0 same-title twins per artist) was run as a one-off script against the final SQL on 2026-10-16; `analyze_dupes.mjs` reproduces the normalized-title half of it.
 
 ### 9.5 Regeneration (single command)