chore(import): document import workflow and apply Round 2 contact deduplications
This commit is contained in:
126
README.md
126
README.md
@@ -1,31 +1,95 @@
|
||||
# GerbilManager
|
||||
|
||||
- `GerbilManagerWebAPI/` — ASP.NET Core backend (Swagger unter `/swagger`)
|
||||
- `gerbil-manager-web/` — React-Frontend (Vite + TypeScript), siehe `gerbil-manager-web/README.md`
|
||||
|
||||
# Requirements
|
||||
Install ef:
|
||||
|
||||
dotnet tool install --global dotnet-ef
|
||||
|
||||
# Used cmd´s
|
||||
## Start containers
|
||||
docker compose up
|
||||
## Create the initial database
|
||||
dotnet ef database update
|
||||
## Create an migration script
|
||||
dotnet ef migrations add <name>
|
||||
|
||||
## Docker
|
||||
### force compose rebuild
|
||||
docker-compose build --no-cache
|
||||
|
||||
## Frontend (gerbil-manager-web)
|
||||
cd gerbil-manager-web
|
||||
npm install
|
||||
npm run dev
|
||||
|
||||
|
||||
TODO:
|
||||
- Backup docker volume database
|
||||
- run "dotnet ef database update" if no database exist.
|
||||
# GerbilManager
|
||||
|
||||
A manager application for gerbil breeding data featuring a .NET Aspire backend and a React-based frontend.
|
||||
|
||||
- `GerbilManagerWebAPI/` — ASP.NET Core backend (Swagger under `/swagger`)
|
||||
- `gerbil-manager-web/` — React Frontend (Vite + TypeScript), see `gerbil-manager-web/README.md`
|
||||
- `tools/import/` — Python import pipeline scripts and database schemas
|
||||
|
||||
## Requirements
|
||||
To manage database migrations, install the EF Core CLI tool globally:
|
||||
```bash
|
||||
dotnet tool install --global dotnet-ef
|
||||
```
|
||||
|
||||
## Running the Application
|
||||
To run the full stack (PostgreSQL database, WebAPI, and Frontend Vite app) in one command:
|
||||
```bash
|
||||
dotnet run --project GerbilManager.AppHost
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📥 Data Import & Normalization Pipeline
|
||||
|
||||
The application features a Python-based import pipeline that compiles gerbil breeding records, lineage, photos, and contact information from source files (markdown logs, Excel spreadsheets, Word documents) into a unified JSON format, and ingests them into the PostgreSQL database.
|
||||
|
||||
### 1. Structure of the Pipeline
|
||||
|
||||
* **Source Data**: Located at `C:\Users\gulum\dev\Wurfchronik_Bilder` (contains markdown files, `.xlsx` pedigree trees, and detail logs).
|
||||
* **Compilation**: [merge_and_resolve.py](file:///C:/Users/gulum/dev/GerbilManager/tools/import/merge_and_resolve.py) processes the source files, extracts breeding details, resolves lineage links, maps colors/genotypes, and normalizes contact names.
|
||||
* **Normalization Mapping**: `merge_and_resolve.py` contains a `norm_map` dictionary of contact name normalizations. This maps various spelling variations and abbreviations to a single canonical name.
|
||||
|
||||
### 2. How to Run the Import
|
||||
|
||||
#### Automatic Way (All-in-One)
|
||||
The PowerShell script in `tools/import` boots the Aspire stack, waits for the API to become ready, and automatically triggers the ingest:
|
||||
```powershell
|
||||
powershell -ExecutionPolicy Bypass -File tools/import/run_app_and_import.ps1
|
||||
```
|
||||
|
||||
#### Manual Way
|
||||
1. **Recompile the import data**:
|
||||
Run the compiler script from the root directory to parse raw sources and output a fresh `resolved_import.json`:
|
||||
```bash
|
||||
python tools/import/merge_and_resolve.py
|
||||
```
|
||||
2. **Ingest into the running database**:
|
||||
Ensure the application is running, then send an HTTP `POST` request to the API:
|
||||
```bash
|
||||
Invoke-RestMethod -Uri "http://localhost:5179/import/ingest-resolved" -Method Post
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 👥 Contact Deduplication Workflow
|
||||
|
||||
When the database is wiped and re-imported, you can detect and resolve duplicate contacts (e.g. spelling mistakes, initials vs full names) using the deduplication checklist workflow.
|
||||
|
||||
### 1. Search for Duplicate Contacts
|
||||
You can run duplicate checks using two different similarity parameter models:
|
||||
* **Round 1 (Clever matching on first 4 characters)**:
|
||||
```bash
|
||||
python tools/import/find_duplicates_v5.py
|
||||
```
|
||||
Creates: `tools/import/output/potential_duplicates_decision.md`
|
||||
* **Round 2 (Broader matching on initials, matching last names, phones, or emails)**:
|
||||
```bash
|
||||
python tools/import/find_duplicates_round2.py
|
||||
```
|
||||
Creates: `tools/import/output/potential_duplicates_decision_round2.md`
|
||||
|
||||
### 2. Fill out the Checklist
|
||||
Open the generated markdown checklist file and edit the checkboxes:
|
||||
* **Combine/Merge**: Put an `x` in the checkbox `[x]` of the name you want to choose as the **canonical name**. Leave the checkboxes empty `[ ]` for any names you want to **merge into the canonical name**.
|
||||
* **Standalone**: Put an `o` in the checkbox `[o]` of any name that should remain a separate, standalone contact.
|
||||
* **New Name**: Put an `x` next to `Custom name:` and type your new desired name inside the backticks.
|
||||
* **Keep all separate**: Leave all checkboxes in a Choice empty `[ ]`.
|
||||
|
||||
*Example:*
|
||||
```markdown
|
||||
- [x] Max Mustermann <-- canonical name
|
||||
- [ ] Max M. <-- merges into Max Mustermann
|
||||
- [o] Moritz Mustermann <-- remains a separate standalone contact
|
||||
```
|
||||
|
||||
### 3. Apply the Decisions
|
||||
Once you have saved your decisions in the checklist file, run the parser:
|
||||
```bash
|
||||
python tools/import/parse_decisions.py
|
||||
```
|
||||
This automatically parses your selections, resolves the merges, and updates the `norm_map` inside `merge_and_resolve.py` so they are **permanently saved** in the codebase.
|
||||
Run the compilation pipeline again to produce the final deduplicated import JSON:
|
||||
```bash
|
||||
python tools/import/merge_and_resolve.py
|
||||
```
|
||||
@@ -60,8 +60,10 @@ def get_normalized_contact_name(name):
|
||||
|
||||
norm_map = {
|
||||
"alessandrab": "Alessandra Bartoletti",
|
||||
"alexsandrab": "Alessandra Bartoletti",
|
||||
"andreafey": "Andrea und Stefanie Fey",
|
||||
"andreastefaniefey": "Andrea und Stefanie Fey",
|
||||
"anettw": "Annett Wernecke",
|
||||
"angelachristoph": "Angela und Ekki Christoph",
|
||||
"angelaekkichristoph": "Angela und Ekki Christoph",
|
||||
"angie": "Angie Reichert-Cambeis",
|
||||
@@ -76,6 +78,7 @@ def get_normalized_contact_name(name):
|
||||
"arturaleksandraundpolamundrzynski": "Artur, Aleksandra und Pola Mundrzynski",
|
||||
"astridr": "Astrid Rohmann",
|
||||
"astridroh": "Astrid Rohmann",
|
||||
"babarastehle": "Barbara Stehle",
|
||||
"barbarar": "Barbara Riegler",
|
||||
"biancab": "Bianca Bernhardt",
|
||||
"biancam": "Bianca Mayer + Alex Wachten",
|
||||
@@ -83,6 +86,7 @@ def get_normalized_contact_name(name):
|
||||
"birgitk": "Birgit Knopp",
|
||||
"birgitkropp": "Birgit Knopp",
|
||||
"birgittabüskens": "Birgitta Müller-Büskens",
|
||||
"birgittamüllerbüskens": "Brigitta Müller-Büskens",
|
||||
"blackforestgv": "Black Forest",
|
||||
"brigittast": "Brigitta Struve",
|
||||
"buntefellnasen": "bunten Fellnasen",
|
||||
@@ -116,6 +120,7 @@ def get_normalized_contact_name(name):
|
||||
"inahübner": "Ina Hübner + Nicolai Thome",
|
||||
"inapeines": "Ina Pleines",
|
||||
"janab": "Jana Beikert",
|
||||
"jasiminweber": "Jasmin Weber und Ronny Reichelt",
|
||||
"jasmin": "Jasmin Weber",
|
||||
"jasminweber": "Jasmin Weber und Ronny Reichelt",
|
||||
"jasminweberromyreichelt": "Jasmin Weber und Ronny Reichelt",
|
||||
@@ -151,16 +156,20 @@ def get_normalized_contact_name(name):
|
||||
"melaniefeythorbenmeier": "Melanie Fey und Thorben Meier",
|
||||
"michaeldavidschmitz": "Michael und David Schmitz",
|
||||
"michaelschmitz": "Michael und David Schmitz",
|
||||
"nataliereitz": "Nathalie Reitz",
|
||||
"natascham": "Natascha Marienfeld",
|
||||
"nicolefischler": "Nicole Tischler",
|
||||
"nicolel": "Nicole Lannert",
|
||||
"nicolen": "Nicole Nuzzo",
|
||||
"nicolet": "Nicole Tischler",
|
||||
"nicolew": "Nicole Webersinn",
|
||||
"nicost": "Nico Stamm",
|
||||
"nielsh": "Nils H.",
|
||||
"noelstrahbach": "Noel Strohbach",
|
||||
"nora": "Nora Rudersdorf geb. Holzbach",
|
||||
"norah": "Nora Holzbach",
|
||||
"norarh": "Nora Rudersdorf",
|
||||
"norarudersdorf": "Nora Rudersdorf geb. Holzbach",
|
||||
"oflennylengo": "Lenny Lengo",
|
||||
"pascale": "Pascale Diefenbach",
|
||||
"pascaledießenbach": "Pascale Diefenbach",
|
||||
@@ -178,7 +187,10 @@ def get_normalized_contact_name(name):
|
||||
"ranialößler": "Rania Löffler",
|
||||
"rominahubrich": "Romina Milde / Hubrich",
|
||||
"rominamilde": "Romina Milde / Hubrich",
|
||||
"romonag": "Ramona Gömpel",
|
||||
"ronialöffler": "Rania Löffler",
|
||||
"sandrak": "Sandra Kubas",
|
||||
"sarahlöser": "Sarah Löwer",
|
||||
"sarahlöuer": "Sarah Löwer",
|
||||
"sarahz": "Sarah Zitzer",
|
||||
"saskiaw": "Saskia Wucher",
|
||||
@@ -186,6 +198,7 @@ def get_normalized_contact_name(name):
|
||||
"schlossmäuse": "Schlossmäuse",
|
||||
"schlossmäusen": "Schlossmäuse",
|
||||
"sebastiansch": "Sebastian Schmitt",
|
||||
"serinaberg": "Selina Berg",
|
||||
"silkewolfganghintze": "Silke und Wolfgang Hintze",
|
||||
"smilla": "Smilla H.",
|
||||
"steffi": "Steffi K.",
|
||||
|
||||
Reference in New Issue
Block a user