FEAT-8b: spreadsheet import tooling (stages 1-2) + review report

tools/import/ (Python, zero-dep migration tooling, not product code):
- xlsx_util.py: dependency-free .xlsx reader (shared strings, cells, drawing anchors)
- genotype.py: notation -> frozen 8-locus mapping + verbatim rawGenotype + unmappedTokens; '-' -> '?'
- extract.py: 10 Stammbaum charts + Wurfchronik -> animals.json/litters.json + anchor-mapped photos;
  dedup on normalise(name)+DOB -> German review-report.md (no DB load)

Run: 889 raw -> 587 unique animals, 24 conflicts, 310 ambiguous, 123 photos, 752 litters.
Output gitignored except review-report.md. Re-runnable per file (Wurfchronik Teil2+).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-06 00:40:54 +02:00
parent 016b5a1521
commit 1b776cd994
6 changed files with 1058 additions and 0 deletions

58
tools/import/README.md Normal file
View File

@@ -0,0 +1,58 @@
# GerbilManager import tooling (FEAT-8b)
One-off **migration tooling** (Python, no third-party deps) that turns Julian's
wife's hand-built spreadsheets into normalised JSON for review and, later, import.
This is *not* product code — it lives outside the app and is run manually.
See the format analysis in `FEAT-8a-format-spec.md` (Pam's hive workspace).
## What it does
`extract.py` runs **stages 12** of the pipeline:
1. **Extract (stage 1)**
- 10 *Stammbaum* pedigree charts → animals (name, DOB, death, Farbschlag,
genotype, breeder, positionally-reconstructed parent links, photos).
- *Wurfchronik* litter chronicle → litters (date, dam, sire, Wurfstärke,
sex breakdown, Zuchtnummer, notes). Columns are read **by header row** because
the two sheets use different schemas.
- Embedded photos (`xl/media`) → `output/photos/<animal-slug>/`, mapped to the
animal by drawing anchor position.
2. **Dedup + review (stage 2)**
- Merge animals on `normalise(name) + DOB` (corroborated by DOB+genotype).
- Emit a German-language `output/review-report.md` for the breeder to verify
(merges, **conflicts**, ambiguous/incomplete entries, unmapped genotype tokens).
- **Nothing is loaded into the database** — stage 3 (API load) is separate and
waits on DATA-2 + FEAT-1b phase 2.
Genotypes are mapped to the frozen 8-locus contract (A C D E G P Sp Re) while
preserving everything: `genotype.mapped8locus`, `genotype.rawGenotype` (verbatim),
`genotype.unmappedTokens` (e.g. the `Uw` locus, markers `WFNZ/WP/DP`). A `-`
(unknown second allele) maps to `?`.
## Run
```sh
cd tools/import
python extract.py # uses the default source paths
python extract.py --stammbaeume "<dir>" --wurfchronik "<file.xlsx>"
```
Requires Python 3. **Re-runnable / idempotent** — re-run when more files arrive
(Wurfchronik `Teil2+`, or new charts).
## Output (`tools/import/output/`, git-ignored except the report)
| File | Contents |
|---|---|
| `animals.json` | deduped animals with genotype, parentRefs, photos, sourceFiles |
| `litters.json` | litters from the Wurfchronik |
| `photos/<slug>/…` | extracted, anchor-mapped images |
| `review-report.md` | **human review deliverable** (committed) |
## Files
- `xlsx_util.py` — dependency-free `.xlsx` reader (zip + XML): shared strings,
cells by reference, image/drawing anchors.
- `genotype.py` — genotype notation parser → 8-locus mapping + raw + unmapped.
- `extract.py` — the pipeline (stages 12).