chore(import): document import workflow and apply Round 2 contact deduplications

README.md

@@ -1,31 +1,95 @@
-# GerbilManager
-
-- `GerbilManagerWebAPI/` — ASP.NET Core backend (Swagger unter `/swagger`)
-- `gerbil-manager-web/` — React-Frontend (Vite + TypeScript), siehe `gerbil-manager-web/README.md`
-
-# Requirements
-Install ef:
-
-dotnet tool install --global dotnet-ef
-
-# Used cmd´s
-## Start containers
-docker compose up
-## Create the initial database
-dotnet ef database update
-## Create an migration script
-dotnet ef migrations add <name>
-
-## Docker
-### force compose rebuild
-docker-compose build --no-cache
-
-## Frontend (gerbil-manager-web)
-cd gerbil-manager-web
-npm install
-npm run dev
-
-
-TODO:
-- Backup docker volume database 
-- run "dotnet ef database update" if no database exist.
+# GerbilManager
+
+A manager application for gerbil breeding data featuring a .NET Aspire backend and a React-based frontend.
+
+- `GerbilManagerWebAPI/` — ASP.NET Core backend (Swagger under `/swagger`)
+- `gerbil-manager-web/` — React Frontend (Vite + TypeScript), see `gerbil-manager-web/README.md`
+- `tools/import/` — Python import pipeline scripts and database schemas
+
+## Requirements
+To manage database migrations, install the EF Core CLI tool globally:
+```bash
+dotnet tool install --global dotnet-ef
+```
+
+## Running the Application
+To run the full stack (PostgreSQL database, WebAPI, and Frontend Vite app) in one command:
+```bash
+dotnet run --project GerbilManager.AppHost
+```
+
+---
+
+## 📥 Data Import & Normalization Pipeline
+
+The application features a Python-based import pipeline that compiles gerbil breeding records, lineage, photos, and contact information from source files (markdown logs, Excel spreadsheets, Word documents) into a unified JSON format, and ingests them into the PostgreSQL database.
+
+### 1. Structure of the Pipeline
+
+*   **Source Data**: Located at `C:\Users\gulum\dev\Wurfchronik_Bilder` (contains markdown files, `.xlsx` pedigree trees, and detail logs).
+*   **Compilation**: [merge_and_resolve.py](file:///C:/Users/gulum/dev/GerbilManager/tools/import/merge_and_resolve.py) processes the source files, extracts breeding details, resolves lineage links, maps colors/genotypes, and normalizes contact names.
+*   **Normalization Mapping**: `merge_and_resolve.py` contains a `norm_map` dictionary of contact name normalizations. This maps various spelling variations and abbreviations to a single canonical name.
+
+### 2. How to Run the Import
+
+#### Automatic Way (All-in-One)
+The PowerShell script in `tools/import` boots the Aspire stack, waits for the API to become ready, and automatically triggers the ingest:
+```powershell
+powershell -ExecutionPolicy Bypass -File tools/import/run_app_and_import.ps1
+```
+
+#### Manual Way
+1.  **Recompile the import data**:
+    Run the compiler script from the root directory to parse raw sources and output a fresh `resolved_import.json`:
+    ```bash
+    python tools/import/merge_and_resolve.py
+    ```
+2.  **Ingest into the running database**:
+    Ensure the application is running, then send an HTTP `POST` request to the API:
+    ```bash
+    Invoke-RestMethod -Uri "http://localhost:5179/import/ingest-resolved" -Method Post
+    ```
+
+---
+
+## 👥 Contact Deduplication Workflow
+
+When the database is wiped and re-imported, you can detect and resolve duplicate contacts (e.g. spelling mistakes, initials vs full names) using the deduplication checklist workflow.
+
+### 1. Search for Duplicate Contacts
+You can run duplicate checks using two different similarity parameter models:
+*   **Round 1 (Clever matching on first 4 characters)**:
+    ```bash
+    python tools/import/find_duplicates_v5.py
+    ```
+    Creates: `tools/import/output/potential_duplicates_decision.md`
+*   **Round 2 (Broader matching on initials, matching last names, phones, or emails)**:
+    ```bash
+    python tools/import/find_duplicates_round2.py
+    ```
+    Creates: `tools/import/output/potential_duplicates_decision_round2.md`
+
+### 2. Fill out the Checklist
+Open the generated markdown checklist file and edit the checkboxes:
+*   **Combine/Merge**: Put an `x` in the checkbox `[x]` of the name you want to choose as the **canonical name**. Leave the checkboxes empty `[ ]` for any names you want to **merge into the canonical name**.
+*   **Standalone**: Put an `o` in the checkbox `[o]` of any name that should remain a separate, standalone contact.
+*   **New Name**: Put an `x` next to `Custom name:` and type your new desired name inside the backticks.
+*   **Keep all separate**: Leave all checkboxes in a Choice empty `[ ]`.
+
+*Example:*
+```markdown
+- [x] Max Mustermann   <-- canonical name
+- [ ] Max M.           <-- merges into Max Mustermann
+- [o] Moritz Mustermann <-- remains a separate standalone contact
+```
+
+### 3. Apply the Decisions
+Once you have saved your decisions in the checklist file, run the parser:
+```bash
+python tools/import/parse_decisions.py
+```
+This automatically parses your selections, resolves the merges, and updates the `norm_map` inside `merge_and_resolve.py` so they are **permanently saved** in the codebase.
+Run the compilation pipeline again to produce the final deduplicated import JSON:
+```bash
+python tools/import/merge_and_resolve.py
+```