95 lines
4.3 KiB
Markdown
95 lines
4.3 KiB
Markdown
# GerbilManager
|
|
|
|
A manager application for gerbil breeding data featuring a .NET Aspire backend and a React-based frontend.
|
|
|
|
- `GerbilManagerWebAPI/` — ASP.NET Core backend (Swagger under `/swagger`)
|
|
- `gerbil-manager-web/` — React Frontend (Vite + TypeScript), see `gerbil-manager-web/README.md`
|
|
- `tools/import/` — Python import pipeline scripts and database schemas
|
|
|
|
## Requirements
|
|
To manage database migrations, install the EF Core CLI tool globally:
|
|
```bash
|
|
dotnet tool install --global dotnet-ef
|
|
```
|
|
|
|
## Running the Application
|
|
To run the full stack (PostgreSQL database, WebAPI, and Frontend Vite app) in one command:
|
|
```bash
|
|
dotnet run --project GerbilManager.AppHost
|
|
```
|
|
|
|
---
|
|
|
|
## 📥 Data Import & Normalization Pipeline
|
|
|
|
The application features a Python-based import pipeline that compiles gerbil breeding records, lineage, photos, and contact information from source files (markdown logs, Excel spreadsheets, Word documents) into a unified JSON format, and ingests them into the PostgreSQL database.
|
|
|
|
### 1. Structure of the Pipeline
|
|
|
|
* **Source Data**: Located at `C:\Users\gulum\dev\Wurfchronik_Bilder` (contains markdown files, `.xlsx` pedigree trees, and detail logs).
|
|
* **Compilation**: [merge_and_resolve.py](file:///C:/Users/gulum/dev/GerbilManager/tools/import/merge_and_resolve.py) processes the source files, extracts breeding details, resolves lineage links, maps colors/genotypes, and normalizes contact names.
|
|
* **Normalization Mapping**: `merge_and_resolve.py` contains a `norm_map` dictionary of contact name normalizations. This maps various spelling variations and abbreviations to a single canonical name.
|
|
|
|
### 2. How to Run the Import
|
|
|
|
#### Automatic Way (All-in-One)
|
|
The PowerShell script in `tools/import` boots the Aspire stack, waits for the API to become ready, and automatically triggers the ingest:
|
|
```powershell
|
|
powershell -ExecutionPolicy Bypass -File tools/import/run_app_and_import.ps1
|
|
```
|
|
|
|
#### Manual Way
|
|
1. **Recompile the import data**:
|
|
Run the compiler script from the root directory to parse raw sources and output a fresh `resolved_import.json`:
|
|
```bash
|
|
python tools/import/merge_and_resolve.py
|
|
```
|
|
2. **Ingest into the running database**:
|
|
Ensure the application is running, then send an HTTP `POST` request to the API:
|
|
```bash
|
|
Invoke-RestMethod -Uri "http://localhost:5179/import/ingest-resolved" -Method Post
|
|
```
|
|
|
|
---
|
|
|
|
## 👥 Contact Deduplication Workflow
|
|
|
|
When the database is wiped and re-imported, you can detect and resolve duplicate contacts (e.g. spelling mistakes, initials vs full names) using the deduplication checklist workflow.
|
|
|
|
### 1. Search for Duplicate Contacts
|
|
You can run duplicate checks using two different similarity parameter models:
|
|
* **Round 1 (Clever matching on first 4 characters)**:
|
|
```bash
|
|
python tools/import/find_duplicates_v5.py
|
|
```
|
|
Creates: `tools/import/output/potential_duplicates_decision.md`
|
|
* **Round 2 (Broader matching on initials, matching last names, phones, or emails)**:
|
|
```bash
|
|
python tools/import/find_duplicates_round2.py
|
|
```
|
|
Creates: `tools/import/output/potential_duplicates_decision_round2.md`
|
|
|
|
### 2. Fill out the Checklist
|
|
Open the generated markdown checklist file and edit the checkboxes:
|
|
* **Combine/Merge**: Put an `x` in the checkbox `[x]` of the name you want to choose as the **canonical name**. Leave the checkboxes empty `[ ]` for any names you want to **merge into the canonical name**.
|
|
* **Standalone**: Put an `o` in the checkbox `[o]` of any name that should remain a separate, standalone contact.
|
|
* **New Name**: Put an `x` next to `Custom name:` and type your new desired name inside the backticks.
|
|
* **Keep all separate**: Leave all checkboxes in a Choice empty `[ ]`.
|
|
|
|
*Example:*
|
|
```markdown
|
|
- [x] Max Mustermann <-- canonical name
|
|
- [ ] Max M. <-- merges into Max Mustermann
|
|
- [o] Moritz Mustermann <-- remains a separate standalone contact
|
|
```
|
|
|
|
### 3. Apply the Decisions
|
|
Once you have saved your decisions in the checklist file, run the parser:
|
|
```bash
|
|
python tools/import/parse_decisions.py
|
|
```
|
|
This automatically parses your selections, resolves the merges, and updates the `norm_map` inside `merge_and_resolve.py` so they are **permanently saved** in the codebase.
|
|
Run the compilation pipeline again to produce the final deduplicated import JSON:
|
|
```bash
|
|
python tools/import/merge_and_resolve.py
|
|
``` |