chore(import): document import workflow and apply Round 2 contact deduplications

README.md

@@ -1,31 +1,95 @@
-# GerbilManager
+# GerbilManager
-
+
-- `GerbilManagerWebAPI/` — ASP.NET Core backend (Swagger unter `/swagger`)
+A manager application for gerbil breeding data featuring a .NET Aspire backend and a React-based frontend.
-- `gerbil-manager-web/` — React-Frontend (Vite + TypeScript), siehe `gerbil-manager-web/README.md`
+
-
+- `GerbilManagerWebAPI/` — ASP.NET Core backend (Swagger under `/swagger`)
-# Requirements
+- `gerbil-manager-web/` — React Frontend (Vite + TypeScript), see `gerbil-manager-web/README.md`
-Install ef:
+- `tools/import/` — Python import pipeline scripts and database schemas
-
+
-dotnet tool install --global dotnet-ef
+## Requirements
-
+To manage database migrations, install the EF Core CLI tool globally:
-# Used cmd´s
+```bash
-## Start containers
+dotnet tool install --global dotnet-ef
-docker compose up
+```
-## Create the initial database
+
-dotnet ef database update
+## Running the Application
-## Create an migration script
+To run the full stack (PostgreSQL database, WebAPI, and Frontend Vite app) in one command:
-dotnet ef migrations add <name>
+```bash
-
+dotnet run --project GerbilManager.AppHost
-## Docker
+```
-### force compose rebuild
+
-docker-compose build --no-cache
+---
-
+
-## Frontend (gerbil-manager-web)
+## 📥 Data Import & Normalization Pipeline
-cd gerbil-manager-web
+
-npm install
+The application features a Python-based import pipeline that compiles gerbil breeding records, lineage, photos, and contact information from source files (markdown logs, Excel spreadsheets, Word documents) into a unified JSON format, and ingests them into the PostgreSQL database.
-npm run dev
+
-
+### 1. Structure of the Pipeline
-
+
-TODO:
+*   **Source Data**: Located at `C:\Users\gulum\dev\Wurfchronik_Bilder` (contains markdown files, `.xlsx` pedigree trees, and detail logs).
-- Backup docker volume database 
+*   **Compilation**: [merge_and_resolve.py](file:///C:/Users/gulum/dev/GerbilManager/tools/import/merge_and_resolve.py) processes the source files, extracts breeding details, resolves lineage links, maps colors/genotypes, and normalizes contact names.
-- run "dotnet ef database update" if no database exist.
+*   **Normalization Mapping**: `merge_and_resolve.py` contains a `norm_map` dictionary of contact name normalizations. This maps various spelling variations and abbreviations to a single canonical name.
+
+### 2. How to Run the Import
+
+#### Automatic Way (All-in-One)
+The PowerShell script in `tools/import` boots the Aspire stack, waits for the API to become ready, and automatically triggers the ingest:
+```powershell
+powershell -ExecutionPolicy Bypass -File tools/import/run_app_and_import.ps1
+```
+
+#### Manual Way
+1.  **Recompile the import data**:
+    Run the compiler script from the root directory to parse raw sources and output a fresh `resolved_import.json`:
+    ```bash
+    python tools/import/merge_and_resolve.py
+    ```
+2.  **Ingest into the running database**:
+    Ensure the application is running, then send an HTTP `POST` request to the API:
+    ```bash
+    Invoke-RestMethod -Uri "http://localhost:5179/import/ingest-resolved" -Method Post
+    ```
+
+---
+
+## 👥 Contact Deduplication Workflow
+
+When the database is wiped and re-imported, you can detect and resolve duplicate contacts (e.g. spelling mistakes, initials vs full names) using the deduplication checklist workflow.
+
+### 1. Search for Duplicate Contacts
+You can run duplicate checks using two different similarity parameter models:
+*   **Round 1 (Clever matching on first 4 characters)**:
+    ```bash
+    python tools/import/find_duplicates_v5.py
+    ```
+    Creates: `tools/import/output/potential_duplicates_decision.md`
+*   **Round 2 (Broader matching on initials, matching last names, phones, or emails)**:
+    ```bash
+    python tools/import/find_duplicates_round2.py
+    ```
+    Creates: `tools/import/output/potential_duplicates_decision_round2.md`
+
+### 2. Fill out the Checklist
+Open the generated markdown checklist file and edit the checkboxes:
+*   **Combine/Merge**: Put an `x` in the checkbox `[x]` of the name you want to choose as the **canonical name**. Leave the checkboxes empty `[ ]` for any names you want to **merge into the canonical name**.
+*   **Standalone**: Put an `o` in the checkbox `[o]` of any name that should remain a separate, standalone contact.
+*   **New Name**: Put an `x` next to `Custom name:` and type your new desired name inside the backticks.
+*   **Keep all separate**: Leave all checkboxes in a Choice empty `[ ]`.
+
+*Example:*
+```markdown
+- [x] Max Mustermann   <-- canonical name
+- [ ] Max M.           <-- merges into Max Mustermann
+- [o] Moritz Mustermann <-- remains a separate standalone contact
+```
+
+### 3. Apply the Decisions
+Once you have saved your decisions in the checklist file, run the parser:
+```bash
+python tools/import/parse_decisions.py
+```
+This automatically parses your selections, resolves the merges, and updates the `norm_map` inside `merge_and_resolve.py` so they are **permanently saved** in the codebase.
+Run the compilation pipeline again to produce the final deduplicated import JSON:
+```bash
+python tools/import/merge_and_resolve.py
+```

tools/import/merge_and_resolve.py

@@ -60,8 +60,10 @@ def get_normalized_contact_name(name):
         
     norm_map = {
         "alessandrab": "Alessandra Bartoletti",
+        "alexsandrab": "Alessandra Bartoletti",
         "andreafey": "Andrea und Stefanie Fey",
         "andreastefaniefey": "Andrea und Stefanie Fey",
+        "anettw": "Annett Wernecke",
         "angelachristoph": "Angela und Ekki Christoph",
         "angelaekkichristoph": "Angela und Ekki Christoph",
         "angie": "Angie Reichert-Cambeis",
@@ -76,6 +78,7 @@ def get_normalized_contact_name(name):
         "arturaleksandraundpolamundrzynski": "Artur, Aleksandra und Pola Mundrzynski",
         "astridr": "Astrid Rohmann",
         "astridroh": "Astrid Rohmann",
+        "babarastehle": "Barbara Stehle",
         "barbarar": "Barbara Riegler",
         "biancab": "Bianca Bernhardt",
         "biancam": "Bianca Mayer + Alex Wachten",
@@ -83,6 +86,7 @@ def get_normalized_contact_name(name):
         "birgitk": "Birgit Knopp",
         "birgitkropp": "Birgit Knopp",
         "birgittabüskens": "Birgitta Müller-Büskens",
+        "birgittamüllerbüskens": "Brigitta Müller-Büskens",
         "blackforestgv": "Black Forest",
         "brigittast": "Brigitta Struve",
         "buntefellnasen": "bunten Fellnasen",
@@ -116,6 +120,7 @@ def get_normalized_contact_name(name):
         "inahübner": "Ina Hübner + Nicolai Thome",
         "inapeines": "Ina Pleines",
         "janab": "Jana Beikert",
+        "jasiminweber": "Jasmin Weber und Ronny Reichelt",
         "jasmin": "Jasmin Weber",
         "jasminweber": "Jasmin Weber und Ronny Reichelt",
         "jasminweberromyreichelt": "Jasmin Weber und Ronny Reichelt",
@@ -151,16 +156,20 @@ def get_normalized_contact_name(name):
         "melaniefeythorbenmeier": "Melanie Fey und Thorben Meier",
         "michaeldavidschmitz": "Michael und David Schmitz",
         "michaelschmitz": "Michael und David Schmitz",
+        "nataliereitz": "Nathalie Reitz",
         "natascham": "Natascha Marienfeld",
+        "nicolefischler": "Nicole Tischler",
         "nicolel": "Nicole Lannert",
         "nicolen": "Nicole Nuzzo",
         "nicolet": "Nicole Tischler",
         "nicolew": "Nicole Webersinn",
         "nicost": "Nico Stamm",
+        "nielsh": "Nils H.",
         "noelstrahbach": "Noel Strohbach",
         "nora": "Nora Rudersdorf geb. Holzbach",
         "norah": "Nora Holzbach",
         "norarh": "Nora Rudersdorf",
+        "norarudersdorf": "Nora Rudersdorf geb. Holzbach",
         "oflennylengo": "Lenny Lengo",
         "pascale": "Pascale Diefenbach",
         "pascaledießenbach": "Pascale Diefenbach",
@@ -178,7 +187,10 @@ def get_normalized_contact_name(name):
         "ranialößler": "Rania Löffler",
         "rominahubrich": "Romina Milde / Hubrich",
         "rominamilde": "Romina Milde / Hubrich",
+        "romonag": "Ramona Gömpel",
+        "ronialöffler": "Rania Löffler",
         "sandrak": "Sandra Kubas",
+        "sarahlöser": "Sarah Löwer",
         "sarahlöuer": "Sarah Löwer",
         "sarahz": "Sarah Zitzer",
         "saskiaw": "Saskia Wucher",
@@ -186,6 +198,7 @@ def get_normalized_contact_name(name):
         "schlossmäuse": "Schlossmäuse",
         "schlossmäusen": "Schlossmäuse",
         "sebastiansch": "Sebastian Schmitt",
+        "serinaberg": "Selina Berg",
         "silkewolfganghintze": "Silke und Wolfgang Hintze",
         "smilla": "Smilla H.",
         "steffi": "Steffi K.",