diff --git a/README.md b/README.md index ee137b9..4b6fa1a 100644 --- a/README.md +++ b/README.md @@ -1,31 +1,95 @@ -# GerbilManager - -- `GerbilManagerWebAPI/` — ASP.NET Core backend (Swagger unter `/swagger`) -- `gerbil-manager-web/` — React-Frontend (Vite + TypeScript), siehe `gerbil-manager-web/README.md` - -# Requirements -Install ef: - -dotnet tool install --global dotnet-ef - -# Used cmd´s -## Start containers -docker compose up -## Create the initial database -dotnet ef database update -## Create an migration script -dotnet ef migrations add - -## Docker -### force compose rebuild -docker-compose build --no-cache - -## Frontend (gerbil-manager-web) -cd gerbil-manager-web -npm install -npm run dev - - -TODO: -- Backup docker volume database -- run "dotnet ef database update" if no database exist. \ No newline at end of file +# GerbilManager + +A manager application for gerbil breeding data featuring a .NET Aspire backend and a React-based frontend. + +- `GerbilManagerWebAPI/` — ASP.NET Core backend (Swagger under `/swagger`) +- `gerbil-manager-web/` — React Frontend (Vite + TypeScript), see `gerbil-manager-web/README.md` +- `tools/import/` — Python import pipeline scripts and database schemas + +## Requirements +To manage database migrations, install the EF Core CLI tool globally: +```bash +dotnet tool install --global dotnet-ef +``` + +## Running the Application +To run the full stack (PostgreSQL database, WebAPI, and Frontend Vite app) in one command: +```bash +dotnet run --project GerbilManager.AppHost +``` + +--- + +## 📥 Data Import & Normalization Pipeline + +The application features a Python-based import pipeline that compiles gerbil breeding records, lineage, photos, and contact information from source files (markdown logs, Excel spreadsheets, Word documents) into a unified JSON format, and ingests them into the PostgreSQL database. + +### 1. Structure of the Pipeline + +* **Source Data**: Located at `C:\Users\gulum\dev\Wurfchronik_Bilder` (contains markdown files, `.xlsx` pedigree trees, and detail logs). +* **Compilation**: [merge_and_resolve.py](file:///C:/Users/gulum/dev/GerbilManager/tools/import/merge_and_resolve.py) processes the source files, extracts breeding details, resolves lineage links, maps colors/genotypes, and normalizes contact names. +* **Normalization Mapping**: `merge_and_resolve.py` contains a `norm_map` dictionary of contact name normalizations. This maps various spelling variations and abbreviations to a single canonical name. + +### 2. How to Run the Import + +#### Automatic Way (All-in-One) +The PowerShell script in `tools/import` boots the Aspire stack, waits for the API to become ready, and automatically triggers the ingest: +```powershell +powershell -ExecutionPolicy Bypass -File tools/import/run_app_and_import.ps1 +``` + +#### Manual Way +1. **Recompile the import data**: + Run the compiler script from the root directory to parse raw sources and output a fresh `resolved_import.json`: + ```bash + python tools/import/merge_and_resolve.py + ``` +2. **Ingest into the running database**: + Ensure the application is running, then send an HTTP `POST` request to the API: + ```bash + Invoke-RestMethod -Uri "http://localhost:5179/import/ingest-resolved" -Method Post + ``` + +--- + +## 👥 Contact Deduplication Workflow + +When the database is wiped and re-imported, you can detect and resolve duplicate contacts (e.g. spelling mistakes, initials vs full names) using the deduplication checklist workflow. + +### 1. Search for Duplicate Contacts +You can run duplicate checks using two different similarity parameter models: +* **Round 1 (Clever matching on first 4 characters)**: + ```bash + python tools/import/find_duplicates_v5.py + ``` + Creates: `tools/import/output/potential_duplicates_decision.md` +* **Round 2 (Broader matching on initials, matching last names, phones, or emails)**: + ```bash + python tools/import/find_duplicates_round2.py + ``` + Creates: `tools/import/output/potential_duplicates_decision_round2.md` + +### 2. Fill out the Checklist +Open the generated markdown checklist file and edit the checkboxes: +* **Combine/Merge**: Put an `x` in the checkbox `[x]` of the name you want to choose as the **canonical name**. Leave the checkboxes empty `[ ]` for any names you want to **merge into the canonical name**. +* **Standalone**: Put an `o` in the checkbox `[o]` of any name that should remain a separate, standalone contact. +* **New Name**: Put an `x` next to `Custom name:` and type your new desired name inside the backticks. +* **Keep all separate**: Leave all checkboxes in a Choice empty `[ ]`. + +*Example:* +```markdown +- [x] Max Mustermann <-- canonical name +- [ ] Max M. <-- merges into Max Mustermann +- [o] Moritz Mustermann <-- remains a separate standalone contact +``` + +### 3. Apply the Decisions +Once you have saved your decisions in the checklist file, run the parser: +```bash +python tools/import/parse_decisions.py +``` +This automatically parses your selections, resolves the merges, and updates the `norm_map` inside `merge_and_resolve.py` so they are **permanently saved** in the codebase. +Run the compilation pipeline again to produce the final deduplicated import JSON: +```bash +python tools/import/merge_and_resolve.py +``` \ No newline at end of file diff --git a/tools/import/merge_and_resolve.py b/tools/import/merge_and_resolve.py index 05f29e8..9bedb11 100644 --- a/tools/import/merge_and_resolve.py +++ b/tools/import/merge_and_resolve.py @@ -60,8 +60,10 @@ def get_normalized_contact_name(name): norm_map = { "alessandrab": "Alessandra Bartoletti", + "alexsandrab": "Alessandra Bartoletti", "andreafey": "Andrea und Stefanie Fey", "andreastefaniefey": "Andrea und Stefanie Fey", + "anettw": "Annett Wernecke", "angelachristoph": "Angela und Ekki Christoph", "angelaekkichristoph": "Angela und Ekki Christoph", "angie": "Angie Reichert-Cambeis", @@ -76,6 +78,7 @@ def get_normalized_contact_name(name): "arturaleksandraundpolamundrzynski": "Artur, Aleksandra und Pola Mundrzynski", "astridr": "Astrid Rohmann", "astridroh": "Astrid Rohmann", + "babarastehle": "Barbara Stehle", "barbarar": "Barbara Riegler", "biancab": "Bianca Bernhardt", "biancam": "Bianca Mayer + Alex Wachten", @@ -83,6 +86,7 @@ def get_normalized_contact_name(name): "birgitk": "Birgit Knopp", "birgitkropp": "Birgit Knopp", "birgittabüskens": "Birgitta Müller-Büskens", + "birgittamüllerbüskens": "Brigitta Müller-Büskens", "blackforestgv": "Black Forest", "brigittast": "Brigitta Struve", "buntefellnasen": "bunten Fellnasen", @@ -116,6 +120,7 @@ def get_normalized_contact_name(name): "inahübner": "Ina Hübner + Nicolai Thome", "inapeines": "Ina Pleines", "janab": "Jana Beikert", + "jasiminweber": "Jasmin Weber und Ronny Reichelt", "jasmin": "Jasmin Weber", "jasminweber": "Jasmin Weber und Ronny Reichelt", "jasminweberromyreichelt": "Jasmin Weber und Ronny Reichelt", @@ -151,16 +156,20 @@ def get_normalized_contact_name(name): "melaniefeythorbenmeier": "Melanie Fey und Thorben Meier", "michaeldavidschmitz": "Michael und David Schmitz", "michaelschmitz": "Michael und David Schmitz", + "nataliereitz": "Nathalie Reitz", "natascham": "Natascha Marienfeld", + "nicolefischler": "Nicole Tischler", "nicolel": "Nicole Lannert", "nicolen": "Nicole Nuzzo", "nicolet": "Nicole Tischler", "nicolew": "Nicole Webersinn", "nicost": "Nico Stamm", + "nielsh": "Nils H.", "noelstrahbach": "Noel Strohbach", "nora": "Nora Rudersdorf geb. Holzbach", "norah": "Nora Holzbach", "norarh": "Nora Rudersdorf", + "norarudersdorf": "Nora Rudersdorf geb. Holzbach", "oflennylengo": "Lenny Lengo", "pascale": "Pascale Diefenbach", "pascaledießenbach": "Pascale Diefenbach", @@ -178,7 +187,10 @@ def get_normalized_contact_name(name): "ranialößler": "Rania Löffler", "rominahubrich": "Romina Milde / Hubrich", "rominamilde": "Romina Milde / Hubrich", + "romonag": "Ramona Gömpel", + "ronialöffler": "Rania Löffler", "sandrak": "Sandra Kubas", + "sarahlöser": "Sarah Löwer", "sarahlöuer": "Sarah Löwer", "sarahz": "Sarah Zitzer", "saskiaw": "Saskia Wucher", @@ -186,6 +198,7 @@ def get_normalized_contact_name(name): "schlossmäuse": "Schlossmäuse", "schlossmäusen": "Schlossmäuse", "sebastiansch": "Sebastian Schmitt", + "serinaberg": "Selina Berg", "silkewolfganghintze": "Silke und Wolfgang Hintze", "smilla": "Smilla H.", "steffi": "Steffi K.",