Gulum
1b776cd994
tools/import/ (Python, zero-dep migration tooling, not product code): - xlsx_util.py: dependency-free .xlsx reader (shared strings, cells, drawing anchors) - genotype.py: notation -> frozen 8-locus mapping + verbatim rawGenotype + unmappedTokens; '-' -> '?' - extract.py: 10 Stammbaum charts + Wurfchronik -> animals.json/litters.json + anchor-mapped photos; dedup on normalise(name)+DOB -> German review-report.md (no DB load) Run: 889 raw -> 587 unique animals, 24 conflicts, 310 ambiguous, 123 photos, 752 litters. Output gitignored except review-report.md. Re-runnable per file (Wurfchronik Teil2+). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
59 lines
2.5 KiB
Markdown
59 lines
2.5 KiB
Markdown
# GerbilManager import tooling (FEAT-8b)
|
||
|
||
One-off **migration tooling** (Python, no third-party deps) that turns Julian's
|
||
wife's hand-built spreadsheets into normalised JSON for review and, later, import.
|
||
This is *not* product code — it lives outside the app and is run manually.
|
||
|
||
See the format analysis in `FEAT-8a-format-spec.md` (Pam's hive workspace).
|
||
|
||
## What it does
|
||
|
||
`extract.py` runs **stages 1–2** of the pipeline:
|
||
|
||
1. **Extract (stage 1)**
|
||
- 10 *Stammbaum* pedigree charts → animals (name, DOB, death, Farbschlag,
|
||
genotype, breeder, positionally-reconstructed parent links, photos).
|
||
- *Wurfchronik* litter chronicle → litters (date, dam, sire, Wurfstärke,
|
||
sex breakdown, Zuchtnummer, notes). Columns are read **by header row** because
|
||
the two sheets use different schemas.
|
||
- Embedded photos (`xl/media`) → `output/photos/<animal-slug>/`, mapped to the
|
||
animal by drawing anchor position.
|
||
2. **Dedup + review (stage 2)**
|
||
- Merge animals on `normalise(name) + DOB` (corroborated by DOB+genotype).
|
||
- Emit a German-language `output/review-report.md` for the breeder to verify
|
||
(merges, **conflicts**, ambiguous/incomplete entries, unmapped genotype tokens).
|
||
- **Nothing is loaded into the database** — stage 3 (API load) is separate and
|
||
waits on DATA-2 + FEAT-1b phase 2.
|
||
|
||
Genotypes are mapped to the frozen 8-locus contract (A C D E G P Sp Re) while
|
||
preserving everything: `genotype.mapped8locus`, `genotype.rawGenotype` (verbatim),
|
||
`genotype.unmappedTokens` (e.g. the `Uw` locus, markers `WFNZ/WP/DP`). A `-`
|
||
(unknown second allele) maps to `?`.
|
||
|
||
## Run
|
||
|
||
```sh
|
||
cd tools/import
|
||
python extract.py # uses the default source paths
|
||
python extract.py --stammbaeume "<dir>" --wurfchronik "<file.xlsx>"
|
||
```
|
||
|
||
Requires Python 3. **Re-runnable / idempotent** — re-run when more files arrive
|
||
(Wurfchronik `Teil2+`, or new charts).
|
||
|
||
## Output (`tools/import/output/`, git-ignored except the report)
|
||
|
||
| File | Contents |
|
||
|---|---|
|
||
| `animals.json` | deduped animals with genotype, parentRefs, photos, sourceFiles |
|
||
| `litters.json` | litters from the Wurfchronik |
|
||
| `photos/<slug>/…` | extracted, anchor-mapped images |
|
||
| `review-report.md` | **human review deliverable** (committed) |
|
||
|
||
## Files
|
||
|
||
- `xlsx_util.py` — dependency-free `.xlsx` reader (zip + XML): shared strings,
|
||
cells by reference, image/drawing anchors.
|
||
- `genotype.py` — genotype notation parser → 8-locus mapping + raw + unmapped.
|
||
- `extract.py` — the pipeline (stages 1–2).
|