Neuer „Datenherkunft"-Button in der Rennmausakte öffnet einen Dialog, der zeigt,
aus welchen Quellen der Eintrag erzeugt wurde: Quelldateien (Stammbäume +
Wurfchronik), Anzahl zusammengeführter Datensätze, Eltern-Herleitung
(Methode/Konfidenz) und Hinweise (z. B. „per manueller Entscheidung zugeordnet",
„Konflikt gelöst", „aus N Datensätzen zusammengeführt").
Import: build_provenance() in merge_and_resolve.py sammelt die Herkunft über alle
deduplizierten Datensätze und schreibt sie als Provenance-JSON je Tier in
resolved_import.json. Backend: Gerbil.Provenance (nullable text) + Migration
AddGerbilProvenance, gemappt im Ingest und im GerbilDto zurückgegeben. Frontend:
ProvenanceDialog + Typen + Strings.
Tests: Ingest-Round-trip (vorhanden/abwesend), e2e provenance.spec.ts.
dotnet(212)/vitest(129)/playwright/tsc/eslint grün.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
NpgsqlRetryingExecutionStrategy rejects user-initiated transactions: SaveChanges
inside a BeginTransactionAsync block triggers OnFirstExecution which throws
InvalidOperationException. Fix: CreateExecutionStrategy().ExecuteAsync wraps the
entire tx block; mutable state (counters, contactByNorm, change tracker) reset at
lambda top for idempotent retry. Logic extracted to RunLoopAsync local function
shared by dry-run and execute paths.
Regression test (Test 7): FakeRetryingStrategy with MaxRetryCount=1 reproduces
the OnFirstExecution check in CI without a live Npgsql instance.
165/165 tests, ef has-pending=No, no schema change.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
FIX-1 DEDUP (P0): Two same-name siblings in one litter produced identical
ExternalRefs ('docx-{ws}-{name}-{dob}'), crashing execute on IX_Gerbils_ExternalRef.
Fix: ordinal counter per base-ref within the batch → first occurrence keeps the
base ref, subsequent ones get -2, -3 … suffix. Deterministic (JSON-order) → idempotent
re-runs find existing rows via ExternalRef path (PATH 1). HashSet guard added as a
belt-and-suspenders check.
FIX-2 TRANSACTION: execute now opens a single BeginTransactionAsync before the loop
and commits after the final SaveChangesAsync. Eager contact saves (within the tx) and
the gerbil batch save are fully atomic — crash → full rollback, no partial state.
FIX-3 LITTER LOOKUP: docx WsCode is a litter-size fraction ('4/4', '/5') — NOT a
PairingCode like 'G01/ZdkC'. The previous WsCode→PairingCode lookup was always a
no-op (hence litterLinked=0 in every dry-run). Fix: look up DB litters by LitterDob
(±5 days); link only when exactly ONE candidate exists (unambiguous, no false links).
P0 REGRESSION TEST (Test 6): SQLite + EnsureCreated → unique index enforced.
Two animals same name+litter → execute succeeds, both created, distinct ExternalRefs,
re-run = 0 new. This test would have caught the live crash.
Gate: 164/164 tests, has-pending=No, no schema change.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Root cause: synth litters (from chart parentRefs) were added with FatherId/MotherId pointing
to gerbils that were also [Added] in the same SaveChanges batch. With 337 derived litters from
41 pedigrees, multi-generational references created a true dependency cycle:
Gerbil [Added] ← FK{MotherId} Litter [Added] ← FK{LitterId} Gerbil [Added]
EF's topo-sort threw InvalidOperationException; smaller datasets (REIMPORT-1/2) didn't
trigger it because they had fewer chart-derived litters.
Fix: add synth litters with FatherId=null/MotherId=null (deferred), track intended FKs
in synthLitterPendingParents, apply them in a separate SaveChanges after all gerbils are
persisted. Single-direction Gerbil→Litter dependency in the main save; no cycle possible.
No schema change, no migration.
Regression test: artificial A←B←A cycle in SQLite (FK-enforced) must not throw.
Also verifies deferred FK update sets litter parent correctly after execute.
GATE: 158/158 C# tests, has-pending-model-changes=No
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
DB-4 (medium): ICU German collation de-x-icu auf deutschen Textspalten
Gerbil.Name, Gerbil.NameSearch, Gerbil.OriginBreeder, ColorVariety.Name,
Contact.Name. Korrekte ä/ö/ü-Sortierung + locale-aware lower() in Postgres.
Nur bei Npgsql (Database.ProviderName check) — SQLite-Testhost kennt keine
eigenen Collation-Namen und wuerde EnsureCreated fail lassen.
Migration AddGermanCollation (ALTER COLUMN ... TYPE text COLLATE de-x-icu).
DB-5 (low): Litter.ExternalRef + filtered unique index + Importer-Anbindung
Litter.ExternalRef (string?, nullable) = stable import source id (sl.Id).
Filtered unique index WHERE ExternalRef IS NOT NULL (wie DB-1 fuer Gerbil).
Migration AddLitterExternalRef.
Importer (ImportService.cs): setzt ExternalRef = sl.Id beim Litter-Insert;
prueft existingLitterExtRefSet als PRIMAEREN Idempotenz-Key (ExternalRef),
Name+Date-Key als Fallback fuer Wuerfe ohne ExternalRef aus frueheren Laeufen.
+Test DB5_Litter_ExternalRef_set_and_used_for_idempotency (mutiert den Namen
nach erstem Import, prueft dass Re-Import via ExternalRef erkennt).
GATE: 140/140 C#, has-pending=No.
CR-9 (major): gidByNameDob TryGetValue + ExternalRef-Fallback
Verhindert KeyNotFoundException wenn Name/DOB zwischen zwei Laeufen driftet
(z.B. correctDob-Remap oder UI-Umbenennung). Fallback: ExternalRef-Dict-Lookup;
bei Miss: sauberes Ueberspringen + Note statt 500. +Test CR9_NameDOB_drift.
CR-11 (major): Farbschlag aus Genotyp ableiten (fill-NULL-only)
Deep-Band-Tiere (gen>=2, kein Farbschlag-Feld) landen nicht laenger mit null
ColorVariety. GenotypePotentiallyMatches() vergleicht locus-pair-weise (??=wildcard,
case-insensitive). Nur vollstaendige Genotypen (8 Loci, kein ??) loesen Ableitung aus.
Plan-Loop: fuellt colorVarietyId bei null + vollstaendigem Genotyp.
Post-Sweep: bestehende DB-Tiere mit null ColorVarietyId werden nachgefuellt.
AnimalSummary.FarbschlagDerivedFromGenotype = Zaehler. +Test CR11_ColorVariety_from_geno.
CR-10 (major, Python): malformed Override-Genotyp wird nicht angewendet
apply_conflict_decisions validiert mapped8locus nach gt.parse(). Leeres Ergebnis
= Genotyp unveraendert + decisionWarning statt stillem Blanken. Konflikt wird
trotzdem aufgeloest (Entscheidung gilt, nur Genotyp-Override ausgelassen).
+5 Python-Tests (CR-10-Block in test_extract.py).
DB-1 (high): filtered unique index auf Gerbil.ExternalRef
WHERE ExternalRef IS NOT NULL — verhindert doppelten Import bei Race-Conditions
oder Lauf-Ueberschneidungen. Migration UniqueExternalRef. SQLite-Testhost:
HasFilter() wird via EnsureCreated appliziert (SQLite unterstuetzt Partial-Indexes).
PartialUpdateTests externalRef-Assertion auf NotNull geaendert (name-hash unique).
GATE: 139/139 C# + Python ALL PASS, ef has-pending=No.
1) TOLERANTER KC-MATCHER (extract.py norm_zucht): trailing \b nach '.' greift nicht
zwischen two non-word-chars (z.B. nach '.' vor ' '). Fix: r\bv\.\s?d\. (kein \b
am Ende). Alle Schreibvarianten ('Zucht der Kleinen Chaoten', 'kleinen Chaoten',
'v.d. Kleinen Chaoten', '[ZdkC]') -> canon 'kleinechaote'. +is_clan_zucht() helper.
+11 Python-Tests (Regression + neue Varianten). FIX-1-Entscheidungs-Matching
unberuehrt (canon_pair loest v.d. bereits via split_name_zucht).
2) HERKUNFT-BACKFILL (ImportService.cs, fill-NULL-only): POST-Sweep ueber alle
DB-Tiere mit OriginBreeder==null && IsResident==true. Leitet OriginBreeder aus
dem Elterntier (FatherId/MotherId -> OriginBreeder) ab; Fallback 'Zucht der
Kleinen Chaoten'. NIEMALS ueberschreibt nicht-leeren OriginBreeder. Trocken-Lauf
zaehlt HerkunftBackfilled; Execute schreibt. Fixt Tier 'C' (OriginBreeder null).
3) FARBSCHLAG-WOULD-REBACKFILL (no-op Zaehler): zaehlt bereits-importierte Tiere,
bei denen der aktuelle Extraktor einen anderen Farbschlag liefert als in der DB.
Kein Overwrite (Sicherheitsmechanismus ausstehend auf god/Julian-Freigabe);
Optionen A/B/C an god gemeldet, Empfehlung: Option B (Timestamp-basiert).
4) ResidencySummary um HerkunftBackfilled + FarbschlagWouldRebackfill erweitert.
ImportReport-Notizen fuer beide neuen Zaehler.
GATE: 133/133 C#-Tests, 57 Python-Tests, ef has-pending=No.
31 undatierte Wurfchronik-Eintraege (leeres Datumsfeld) wurden bei jedem Lauf
als littersCreated++ gezaehlt, weil ihr Idempotenz-Key (kein Datum) nie in
existingLitterKeySet stand -- aber kein DB-Insert folgte, da date is DateOnly d
false war. Das verfaelschte Arithmetik und Julians Bericht (752 vs 723).
Fix: undatierte Eintraege werden am Schleifenanfang uebersprungen
(littersWithoutDate++, continue) bevor sie in litterIdMap oder den
Created-Zaehler einfliessen. Execute-Block ohne redundante date-Pruefung.
Neues Feld LitterSummary.WithoutDate; Report-Notiz wenn WithoutDate > 0.
Gate: 126/126 C#-Tests, has-pending-model-changes = No.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
FIX-1 ZUCHT: apply_conflict_decisions/apply_dob_remaps matchen jetzt
auf das VOLLE canon_pair-Tupel (nameCanon, zuchtCanon, dob) wenn die
Decision eine Zucht traegt; Fallback name-only wenn keine Zucht.
C3-Regel gewahrt: gleicher Name+DOB, andere Zucht -> kein Hit.
Neuer Regression-Test: Luna ZdkC-Decision trifft nur luna-kc,
nicht luna-bf (andere Zucht).
FIX-2 MERGE: dedup() waehlt das spezifischste Genotyp (fewest '?'
alleles) als sekundaeren Tiebreaker nach locus-count. CC schlaegt C-,
Gg schlaegt G- unabhaengig von der Reihenfolge. 3 neue Merge-Tests
(C- first/CC first/G-vsGg).
FIX-3 BACKFILL allDbNormToGid: ResolveParentForBackfill prueft jetzt
BEIDE Quellen: (a) createdAnimalByName (aktiver Lauf) und
(b) allDbNormToGid (alle DB-Tiere). Decktt den kritischen Fall:
Elterntier in fruehrem Lauf geladen, in diesem Lauf absent vom
Extract. Neuer 3-Lauf-SQLite-Test: Lauf 1 null-Vater, Lauf 2 laedt
Vater, Lauf 3 backfillt via allDbNormToGid.
Gate: 125/125 C#-Tests, Python ALL PASS, has-pending-model-changes=No.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
FIX-1 decision-matching: apply_conflict_decisions/apply_dob_remaps
nutzen jetzt canon_pair(name)[0] als Match-Key (Dedup-Identitaet:
call-name ohne Zucht, v.d.<->von den gefaltet). Workaround-Spelling
v.d. in Victoria Welbys Decision bleibt erhalten; beide Formen
matchen jetzt. Kommentar im decision-Eintrag aktualisiert.
FIX-2 specific-wins: _alleles_compatible aendert '? vs x = False'
-> '? vs x = True' (spezifischer Wert gewinnt). C- vs CC, G- vs Gg,
P? vs PP sind kein Konflikt mehr. Echte Wert-Widersprueche (DD vs Dd,
Ee vs ee, PP vs Pp) bleiben Konflikte. Loest Enya, Ella, Zac
automatisch (Konflikte 8->5 erwartet). 2 bestehende Tests angepasst,
7 neue Tests.
FIX-3 parent-FK backfill: nach dem Wurfchronik-Rueckverknuepfungs-
Block iteriert ImportService.RunAsync ueber bereits importierte
Wuerfe mit null Father/MotherId und setzt fehlende FKs wenn das
Elterntier jetzt ladbar ist. Trockenlauf zaehlt, Execute schreibt.
LitterSummary.ParentFksBackfilled + 2 neue C#-Tests (SQLite).
FIX-4 Skarlett-Artefakt: parse_detail() strippt trailing / +YEAR
aus dem Genotyp-Tail (re.sub). Sterbejahr bleibt als death-Date
erhalten -> Skarlett erscheint als reiner Sterbedatum-Konflikt.
2 neue Python-Tests.
Gate: 124/124 C#-Tests, Python test_extract/test_genotype ALL PASS,
has-pending-model-changes = No.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
god maintains tools/import/conflict-decisions.json as Julian/his wife answer
the D-conflicts. extract.py now consumes it (apply_conflict_decisions): for an
animal matching normalize(name)+dob, it clears the conflict, marks
resolvedByDecision, and — when the decision carries a `genotype` (breeder
notation, parsed via genotype.py) and/or `farbschlag` — treats those as
AUTHORITATIVE. Tolerates a missing/empty/garbled file. Genuinely-unresolved
conflicts stay quarantined.
Loader (ImportService): SourceAnimal.ResolvedByDecision flows through; the
report surfaces Animals.ConflictsResolvedByDecision + a German note.
Result on real data: the 2 current decisions (Firefly D-/PP, WildFire PP)
un-quarantine → Konflikte 21 → 19. As god appends entries the count grows;
nothing else needed from me.
Tests: python test_extract (decision clears conflict + genotype authoritative
+ removes from conflicts list + tolerates missing file) and a C# loader test
(a resolved animal loads and is counted). Folded into the EXTRACT-BANDS branch
so the next re-extract applies band-aware Farbschlag + these decisions in one
pass. No schema change (JSON DTO fields). python + dotnet 121/121 green;
has-pending clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
BUG (blocked the supervised live re-import): /import/execute threw
Npgsql 23503 FK_Litters_Gerbils_FatherId and rolled back. Root cause was
INSERT ORDERING — synthesized derived litters were SaveChanges()'d BEFORE
the parent gerbils (created later in the animal loop), so the litter's
Father/MotherId pointed at rows that didn't exist yet. The dry-run and the
EF in-memory test provider don't enforce FKs, so it slipped through.
Fix:
- Stage synthesized litters in the context but DON'T save them early; the
single SaveChanges after the animal loop lets EF order parents → litters →
offspring (all FKs nullable). Saving litters first was the fault.
- FK-integrity guard (god's spec): compute the persisted set (existing DB +
this run's loadable) and null out any litter parent FK not in it; SKIP a
derived litter whose BOTH parents are unresolvable (offspring loads with
LitterId=null). Quarantined parents already resolve to null via
ResolveParentGid; this is defense-in-depth + makes the invariant explicit.
- Report litterParentFksDropped + derivedLittersSkipped (LitterSummary) +
a German note — so a green dry-run (0/0) GUARANTEES execute won't FK-fault.
Tests: two SQLite-backed regressions (SQLite enforces FKs, unlike the
in-memory provider) — a derived litter with NEW chart parents executes
without throwing, and a quarantined parent leaves that FK null. 120 C#
tests + python green; no schema change (has-pending clean).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per hive/agents/god/OWNERSHIP-residency.md (Julian): "Clan kleine Chaoten"
is the wife's own Zucht. Classify each animal:
(a) RESIDENT if zuchtCanon contains 'kleinechaote' (separator/declension-
insensitive: "v.d. Kleinen Chaoten", "Clan-Kleine-Chaoten", …); OR
(b) PARENT EXCEPTION — a parent of a Clan offspring is resident even if its
own Zuchtname is foreign (you can only breed a Clan litter if the parents
were in your care). Runs AFTER the parentRefs→litter linking.
Otherwise EXTERNAL (pedigree ancestor, not living stock — kept, not deleted).
Stored as a dedicated Gerbil.IsResident bool (distinct from Status/Abgabe —
origin/identity, not current location), defaulting true (own stock unless
marked external; DB HasDefaultValue(true)). Additive migration
AddGerbilResidency; has-pending-model-changes clean. Round-trips in
GerbilDto/GerbilInput and is Gridify-filterable (isResident==true) so Kevin
can build a "nur Bestand" filter. ImportService computes it at (re-)import
and refreshes existing rows on the idempotent sweep. ResidencySummary added
to the import report.
Projected on real data: 306 loadable → 227 resident (188 rule a, +39 via
parent exception), 79 external ancestors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Structural fix (god, Julian-reported via 'C'): the loader ignored
SourceAnimal.ParentRefs, so animals whose ancestry exists only as
Stammbaum chart-position refs loaded with LitterId=null ("unbekannt").
ImportService now synthesizes/reuses a derived litter from parentRefs:
resolves father+mother via name+DOB, groups siblings (same parents+dob)
into one litter, sets Father/Mother + offspring LitterId, dates it to the
offspring DOB, and tags Notes "aus Stammbaum-Diagramm abgeleitet
(Konfidenz: …)" so it's transparent/reversible. Existing animals that
become linkable are re-linked on re-run (sweep-idempotent). Dry-run counts
included. Projected: ~124 loadable animals gain a parent link.
Box-colour = sex (Julian): blue box = male, white box = female. All 11
pedigrees encode this as a solid theme-8 (accent5/blue) fill vs no fill.
xlsx_util.cell_fill_sex reads it; extract.py sets animal.gender from the
box; ImportService.InferGender prefers it over sire/dam name inference.
Result: 306/306 loadable animals now sexed (154♂/152♀).
Extractor noise fix (god): reject Farbschlag values that are actually a
parent NAME bled across cells (contain v.d./von/of/gen.) — cleared phantom
conflicts (e.g. Chayton). Combined with GEN-3 Uw→G: Konflikte 32→21.
Also skip Excel "~$" lock files in the glob.
GEN-3a contract (Kevin): ComposeGenotype appends "Slsl" for WP/Sls
carriers (wild-type sl/sl omitted) so 8-locus strings stay unchanged.
Importer-only. The live re-import into Julian's DB stays a separate
supervised gated step. 95 C# tests + python genotype tests green;
has-pending-model-changes clean (no schema change on this branch).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
genotype.py:
- Uw/uw aliased to G/g (same locus) so the D2 conflict group + pure-Uw
cases stop being conflicts (Gg == Uwuw).
- Sls/WP recognized as a SECOND spotting locus (S(l)s(l)=WP het); carried
into mapped8locus alongside Sp (Sp+Sls = Superschecke).
- dea/Dea/taub/hörend -> hearing/deaf phenotype FLAG (not a locus).
- WFNZ/RV/GV/DP -> provenance/breeding tags (not genotype, not conflicts).
- test_genotype.py: zero-dep unit tests for all four.
extract.py: surface deaf+tags on animals; dedup conflict detection now
compares the NORMALIZED genotype key (mapped8locus) instead of the raw
string, so Uw=G no longer triggers a conflict. Result: Konflikte 32 -> 27,
Zucht-Splits stays 0. Dedup identity = name + DOB + Zucht.
Backend: Gerbil.IsDeaf (bool?) + additive migration AddGerbilDeafFlag
(has-pending-model-changes clean) + GerbilDto/GerbilInput round-trip.
ImportService sets IsDeaf from animal.deaf and preserves Sls + tags + deaf
in RawImportData (kept out of the 8-locus compact Genotype contract until
GEN-3a adopts them).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Gerbil.NameSearch (lowercase, strip whitespace/.-_) auto-synced in ApplicationContext
.SaveChanges; Gridify-filterable so 'clan kleine chaoten' matches 'Clan-Kleine-Chaoten'
(client strips separators the same way). GerbilSearch.Normalize shared helper.
- Gerbil.OriginBreeder (free-text Herkunft) on DTO+Input, set by the FEAT-8 import from the
source Zucht (imported animals have no OriginContact). Gridify-filterable.
- GET /gerbils/breeders: distinct non-empty OriginBreeder values for the Herkunft dropdown.
- Migration AddSearchFields (+ NameSearch backfill SQL for existing rows).
