feat(tolaria): add C4B keyword and metadata search service
This commit is contained in:
parent
35446c03df
commit
aad56a26ea
13 changed files with 3071 additions and 0 deletions
4
tolaria/c4b-search-service/.gitignore
vendored
Normal file
4
tolaria/c4b-search-service/.gitignore
vendored
Normal file
|
|
@ -0,0 +1,4 @@
|
||||||
|
# Laufzeit-Index (rebuildbar aus index_source.json) — nicht versionieren
|
||||||
|
data/
|
||||||
|
__pycache__/
|
||||||
|
*.pyc
|
||||||
103
tolaria/c4b-search-service/C4A_GROUND_TRUTH_ERRATUM.md
Normal file
103
tolaria/c4b-search-service/C4A_GROUND_TRUTH_ERRATUM.md
Normal file
|
|
@ -0,0 +1,103 @@
|
||||||
|
# C4A GROUND TRUTH ERRATUM
|
||||||
|
|
||||||
|
**Datum:** 2026-08-25
|
||||||
|
**Missionsowner:** Christian
|
||||||
|
**Entscheidung:** OPTION 1 — C4A ERRATUM (Corpus korrigieren + versionieren; fehlerhafte IDs NICHT dauerhaft durch Testlogik kaschieren).
|
||||||
|
**Basis:** Forgejo-Clone `main@35446c0` als Single Source of Truth (SoT). IDs stammen AUSSCHLIESSLICH aus den C3-Frontmatter-Feldern `id: object/…` der Markdown-Dateien im SoT.
|
||||||
|
|
||||||
|
| Feld | Wert |
|
||||||
|
|---|---|
|
||||||
|
| `C4A_CORPUS_VERSION_OLD` | `1.0` |
|
||||||
|
| `C4A_CORPUS_VERSION_NEW` | `1.1` |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. ROOT_CAUSE
|
||||||
|
|
||||||
|
Die eingefrorene v1.0-Ground-Truth-Datei `test_corpus.json` enthält **Transkriptions-/Tippfehler in `object_id`-Literalen**. Es handelt sich NICHT um einen Such-Engine-Fehler: Die Engine liest die IDs direkt aus dem SoT-Frontmatter und ist damit autoritativ korrekt. Testdatenfehler ≠ Such-Engine-Fehler.
|
||||||
|
|
||||||
|
- Fehlertypen: Einzelzeichen-Substitution, gelöschte/duplizierte Zeichen, falsche Segment-Position, sowie zwei **abgeschnittene Platzhalter** (`object/5fcf488a-...`, `object/f4e559f5-...`).
|
||||||
|
- Zusätzlich: **2 fehlerhafte Einträge in der `verified_object_ids`-Map** (modul-12-canonical, modul-03-canonical).
|
||||||
|
|
||||||
|
## 2. BEFUND-REPRODUKTION (vor Korrektur, live evidence)
|
||||||
|
|
||||||
|
| Metrik | Wert |
|
||||||
|
|---|---|
|
||||||
|
| `OBJECT_ID_LITERALS_TOTAL` | **30** |
|
||||||
|
| `VALID_IDS` | **13** |
|
||||||
|
| `INVALID_IDS` | **17** |
|
||||||
|
| Fehlerhafte `verified_object_ids`-Einträge | **2** |
|
||||||
|
| Cases (gesamt) | 22 |
|
||||||
|
|
||||||
|
## 3. AFFECTED_TEST_CASES (19)
|
||||||
|
|
||||||
|
`c01-exact-title, c02-keyword, c03-phrase, c04-metadata-only, c05-canonical-source-pair, c06-current-historical, c08-overview-downrank, c09-crosslang-de-en, c10-crosslang-en-de, c11-semantic-paraphrase, c12-ambiguous, c14-module-number, c15-ticker-acronym, c16-env-config-token, c17-relationship, c18-historical-explicit, c19-source-only, c20-canonical-only, c22-trading-term`
|
||||||
|
|
||||||
|
**Unverändert (3):** `c07-readme-explicit`, `c13-no-result`, `c21-config-code` — README/legacy/no-result/secret-Cases berühren keine `object_id`-Literale und blieben unangetastet.
|
||||||
|
|
||||||
|
## 4. AFFECTED_OBJECT_ID_LITERALS — OLD_ID → CORRECT_SOT_ID
|
||||||
|
|
||||||
|
Verifikation erfolgte **direkt gegen die SoT-Frontmatter** (0 Kollision, 0 Ambiguität, ID existiert real, `id`-Feld gehört zum erwarteten Objekt per Pfad/Repräsentation/State). Kein Raten, keine Pfad-Rekonstruktion, keine fuzzy Übernahme alter Snippets.
|
||||||
|
|
||||||
|
| OLD_ID (korrupt) | CORRECT_SOT_ID | SOURCE_PATH | HOW_VERIFIED |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `object/302e9929-c186-c930-2406-ad4a8f17c6fd` | `object/302e9929-e186-c930-2406-ad4a8f17c6fd` | `modul-12-backtesting.md` | SoT `id:`, rep=source, state=current |
|
||||||
|
| `object/47b8029c-5874-e414f-0279-d335785b359a` | `object/47b8029c-5874-e73f-0279-d335785b359a` | `notes/trading/system-docs/modul-12-backtesting.md` | SoT `id:`, rep=canonical, state=current |
|
||||||
|
| `object/5f28e1fd-17f4-e4af-1f92-a17e80b08f0` | `object/5f28e1fd-19f4-e5af-1c92-a17ec80b08f0` | `notes/trading/system-docs/modul-15-m09-anbindung-design.md` | SoT `id:`, rep=canonical, state=current |
|
||||||
|
| `object/5f28e1fd-19f4e5af-1c92-a617ec80b08f0` | `object/5f28e1fd-19f4-e5af-1c92-a17ec80b08f0` | `notes/trading/system-docs/modul-15-m09-anbindung-design.md` | SoT `id:`, rep=canonical, state=current |
|
||||||
|
| `object/5fcf488a-...` | `object/5fcf4885-a361-f49f-2164-f4e10709da34` | `notes/trading/system-docs/infrastructure-handbook.md` | SoT `id:`, rep=canonical, state=current |
|
||||||
|
| `object/5fcf488a-1a361-f49f-2164-f4e10709da34` | `object/5fcf4885-a361-f49f-2164-f4e10709da34` | `notes/trading/system-docs/infrastructure-handbook.md` | SoT `id:`, rep=canonical, state=current |
|
||||||
|
| `object/5fcf488a-361-f49f-2164-f4e10709da34` | `object/5fcf4885-a361-f49f-2164-f4e10709da34` | `notes/trading/system-docs/infrastructure-handbook.md` | SoT `id:`, rep=canonical, state=current |
|
||||||
|
| `object/ac1df780-019b-11b1-cb91-4edaa49e1843` | `object/ac1df780-019b-bb11-cb91-4edaa49e1843` | `notes/trading/system-docs/historical-v2-phase8-m12-datasetgate.md` | SoT `id:`, rep=canonical, state=historical |
|
||||||
|
| `object/ac1df780-019b-11bb-cb91-4edaa49e1843` | `object/ac1df780-019b-bb11-cb91-4edaa49e1843` | `notes/trading/system-docs/historical-v2-phase8-m12-datasetgate.md` | SoT `id:`, rep=canonical, state=historical |
|
||||||
|
| `object/cf54a1fc-4c8b-7477-1b4f-21e5a7a055` | `object/cf54abfc-4c8b-7477-1b4f-21e76ca7b055` | `notes/trading/system-docs/modul-03-market-data.md` | SoT `id:`, rep=canonical, state=current |
|
||||||
|
| `object/cf54a1fc-4c8b-7477-1b4f-21e76ca7b055` | `object/cf54abfc-4c8b-7477-1b4f-21e76ca7b055` | `notes/trading/system-docs/modul-03-market-data.md` | SoT `id:`, rep=canonical, state=current |
|
||||||
|
| `object/cf54fbfc-4c8b-7477-1b4f-21e76ca7b055` | `object/cf54abfc-4c8b-7477-1b4f-21e76ca7b055` | `notes/trading/system-docs/modul-03-market-data.md` | SoT `id:`, rep=canonical, state=current |
|
||||||
|
| `object/ebdc6b2b-4b30-c839-917b-c240ad433a60` | `object/ebdc6b2b-4b30-c839-729b-c240ad433a60` | `notes/trading/system-docs/modul-01-postgresql.md` | SoT `id:`, rep=canonical, state=current |
|
||||||
|
| `object/f4e559f5-...` | `object/f4e559f5-403c-f241-be50-0962d8174644` | `infrastructure-handbook.md` | SoT `id:`, rep=source, state=current |
|
||||||
|
| `object/ffa8fac13-8aa6-d4a4-6756-69eabd0c8d77` | `object/ffa8db13-8aa6-d4a4-6756-69eabd0c8d77` | `notes/trading/system-docs/modul-09-execution-service.md` | SoT `id:`, rep=canonical, state=current |
|
||||||
|
| `object/ffa8facb-13aa6-d4a4-6756-69eabd0c8d77` | `object/ffa8db13-8aa6-d4a4-6756-69eabd0c8d77` | `notes/trading/system-docs/modul-09-execution-service.md` | SoT `id:`, rep=canonical, state=current |
|
||||||
|
| `object/ffa8facb-8aa6-d4a4-6756-69eabd0c8d77` | `object/ffa8db13-8aa6-d4a4-6756-69eabd0c8d77` | `notes/trading/system-docs/modul-09-execution-service.md` | SoT `id:`, rep=canonical, state=current |
|
||||||
|
|
||||||
|
**Verifizierte Map-Korrekturen:**
|
||||||
|
| Key | OLD | CORRECT |
|
||||||
|
|---|---|---|
|
||||||
|
| `modul-12-canonical` | `object/47b8029c-5874-e414f-0279-d335785b359a` | `object/47b8029c-5874-e73f-0279-d335785b359a` |
|
||||||
|
| `modul-03-canonical` | `object/cf54fbfc-4c8b-7477-1b4f-21e76ca7b055` | `object/cf54abfc-4c8b-7477-1b4f-21e76ca7b055` |
|
||||||
|
|
||||||
|
**`IDS_CORRECTED`:** 17 (Literale) + 2 (Map) = **19** korrigierte ID-Verwendungen (auf **12 eindeutige reale IDs** reduziert).
|
||||||
|
|
||||||
|
## 5. SEMANTIC_INTENTION_CHANGED
|
||||||
|
|
||||||
|
**Nur eine einzige bewusste, echte Logik-Korrektur** (separat, da nicht nur Tippfehler):
|
||||||
|
|
||||||
|
- **c06-current-historical — SEMANTIC_INTENTION_CHANGED = YES**
|
||||||
|
- Befund: dieselbe historical-Phase8-ID (`ac1df780-019b-bb11-…`) stand **gleichzeitig in `expected_allowed` UND `must_not_top`** (Selbstwiderspruch).
|
||||||
|
- Intention laut `why` + Filter `include_historical:false`: historical darf bei `include_historical:false` NICHT top sein → `must_not_top` ist korrekt, `expected_allowed` wurde **geleert** (`[]`).
|
||||||
|
- Keine Verschleierung: Der Test prüft weiterhin, dass die historical-ID nicht top gerankt wird.
|
||||||
|
|
||||||
|
- **c16-env-config-token — SEMANTIC_INTENTION_CHANGED = YES**
|
||||||
|
- Befund: Filter `type:code` widersprach der Intention. Im gesamten SoT-Index existiert **kein `type=code`-Objekt** (Types: arch=52, log=22, journal=3, design=4, …); der `type:code`-Filter leert jedes Ergebnis.
|
||||||
|
- Intention laut `why` ("Env-/Config-Token via Keyword-Index; code-chunks nicht sem-embed"): `POSTGRES_DB` muss auffindbar sein. Der korrekte Treffer ist `object/ebdc6b2b-…` (modul-01, `type=arch`).
|
||||||
|
- Korrektur: `filters` → `{}` (Token bleibt auffindbar, Query unverändert).
|
||||||
|
|
||||||
|
**Alle übrigen 20 Cases — SEMANTIC_INTENTION_CHANGED = NO.** Nur ID-Literale wurden substituiert; `query`, `filters`, `expected_rep`, `expected_state`, `must_not_top` (Logik) und `why` blieben unverändert.
|
||||||
|
|
||||||
|
## 6. CORPUS v1.1 VALIDATION (live)
|
||||||
|
|
||||||
|
- **Alle `object_id`-Literale existieren** real in der SoT-Frontmatter → `GROUND_TRUTH_IDS_VALID = TRUE`
|
||||||
|
- **Alle IDs gehören zum richtigen Ground-Truth-Objekt** (per Pfad + `representation` + `state` verifiziert) → 0 Kollision, 0 Ambiguität
|
||||||
|
- **Keine ID doppelt/falsch verwendet** nach Korrektur
|
||||||
|
- **22/22 Testfälle vorhanden**
|
||||||
|
- **Keine Testfall-Intention verloren**
|
||||||
|
- README-Cases unverändert, historical/current unverändert (außer c06-Intentionskorrektur), canonical/source unverändert, semantic-Cases unverändert, no-result-Case unverändert
|
||||||
|
|
||||||
|
## 7. TEST HARNESS
|
||||||
|
|
||||||
|
- Test-Harness wertet PASS/FAIL **gegen den korrigierten v1.1-Corpus** aus.
|
||||||
|
- **Keine automatische Runtime-Umschreibung** („wenn ID falsch, suche passenden Pfad und werte trotzdem PASS“) ist dauerhafte Acceptance-Logik.
|
||||||
|
- Optionaler **VALIDATION STEP** vor Testlauf: `GROUND_TRUTH_IDS_VALID = TRUE`; falls FALSE → `TEST SUITE BLOCKED` und kein Quality-PASS.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**Dateien:** `test_corpus.json` (v1.0, unverändert/Evidence) · `test_corpus_v1.1.json` (neu, korrigiert, kanonisch) · `C4A_GROUND_TRUTH_ERRATUM.md` (dieser Report).
|
||||||
53
tolaria/c4b-search-service/README.md
Normal file
53
tolaria/c4b-search-service/README.md
Normal file
|
|
@ -0,0 +1,53 @@
|
||||||
|
# C4B — TOLARIA SEARCH SERVICE (Keyword & Metadata Search)
|
||||||
|
|
||||||
|
**Phase:** C4B
|
||||||
|
**Typ:** Agentenunabhängiger, abgeleiteter Retrieval-Dienst (Weg A).
|
||||||
|
**Indexquelle:** Forgejo-SoT (Source of Truth). Der Index ist **DERIVED** und jederzeit aus `index_source.json` rebuildbar. Kein pgvector, kein Vector-Store, keine Embeddings (C4C). Stdlib-only Python.
|
||||||
|
|
||||||
|
## Zweck
|
||||||
|
Der Such-Dienst macht die Tolaria-Knowledge-Objects durchsuchbar (exact / keyword / metadata). Er ist read-only gegen den Index und schreibt **NIE** nach Forgejo oder Tolaria. Semantic/Vector/Hybrid sind **NICHT implementiert** (Honest Mode → `mode_not_implemented`), sie werden in C4C adressiert.
|
||||||
|
|
||||||
|
## Komponenten
|
||||||
|
| Datei | Zweck |
|
||||||
|
|---|---|
|
||||||
|
| `search_engine.py` | Kern-Engine: Tokenisierung, Index, Ranking, Collapse, Secret-Scan |
|
||||||
|
| `search_api.py` | `TolariaSearch`-API, Result-Contract, `SUPPORTED_MODES`, Honest Mode, Pagination |
|
||||||
|
| `server.py` | HTTP-Server (`/api/search`, `/api/search/health`, kontrollierter Rebuild) |
|
||||||
|
| `rebuild.py` | Rebuild-CLI aus `index_source.json` |
|
||||||
|
| `run_tests.py` | C4B-Test-Suite: 22er-Ground-Truth v1.1, Acceptance, Quality, Latency, Failure. **Validiert Ground-Truth zuerst** (`GROUND_TRUTH_IDS_VALID`); bei ungültigen IDs → TEST SUITE BLOCKED. |
|
||||||
|
| `fresh_checker.py` | Unabhängiger Fresh Checker (gegen Live-API) |
|
||||||
|
| `erratum_build.py` | Erzeugt `test_corpus_v1.1.json` aus v1.0 (Ground-Truth-Erratum, verifiziert gegen SoT) |
|
||||||
|
| `test_corpus_v1.0.json` | C4A-Evidence (eingefroren, unverändert — Original mit korrupten IDs, Doku nur) |
|
||||||
|
| `test_corpus_v1.1.json` | Korrigierter, versionierter Ground-Truth-Corpus (kanonisch) |
|
||||||
|
| `C4A_GROUND_TRUTH_ERRATUM.md` | Erratum-Report (ROOT_CAUSE, OLD→CORRECT SOT ID, Verifikation) |
|
||||||
|
| `index_source.json` | Abgeleitete Indexquelle aus SoT-Frontmatter (84 Objekte, rebuildbar) |
|
||||||
|
|
||||||
|
## Rebuild & Test (lokal)
|
||||||
|
```bash
|
||||||
|
python3 rebuild.py --source index_source.json # baut data/search_index.json
|
||||||
|
python3 run_tests.py # 22er Corpus + Quality + Latency + Failure
|
||||||
|
python3 fresh_checker.py # Freshness gegen Live-API (Port 8325)
|
||||||
|
python3 erratum_build.py # (re-)erzeugt test_corpus_v1.1.json
|
||||||
|
```
|
||||||
|
|
||||||
|
## Betrieb
|
||||||
|
```bash
|
||||||
|
TOLARIA_SEARCH_PORT=8325 python3 server.py # HTTP auf 0.0.0.0:8325
|
||||||
|
curl http://localhost:8325/api/search/health
|
||||||
|
curl -X POST http://localhost:8325/api/search -H 'Content-Type: application/json' \
|
||||||
|
-d '{"query":"Modul-09","mode":"exact"}'
|
||||||
|
```
|
||||||
|
|
||||||
|
## Honest Mode
|
||||||
|
- `exact` = echt, `keyword` = echt, `metadata` = echt.
|
||||||
|
- `semantic`/`vector`/`hybrid` → `mode_not_implemented` (kein Silent-Fallback). Kein Fake-Semantic.
|
||||||
|
|
||||||
|
## Security
|
||||||
|
- Secret-Scan läuft fail-closed **vor** dem Index; keine Secret-Werte im Index/Report.
|
||||||
|
- Rebuild-Admin-Endpoint nur mit ephemerem Token (`TOLARIA_SEARCH_REBUILD_TOKEN`).
|
||||||
|
- Keine Credentials in diesem Repo.
|
||||||
|
|
||||||
|
## Status (C4B)
|
||||||
|
- **IMPLEMENTATION_STATUS:** COMPLETE (lokal, test-runtime-verifiziert)
|
||||||
|
- **TEST_RUNTIME_STATUS:** PASS (22/22 Corpus v1.1; Quality/Latency/Failure gemessen)
|
||||||
|
- **PRODUCTION_DEPLOYMENT_STATUS:** NICHT deployed — läuft als eigenständiger lokaler/optionaler Dienst; produktiver VPS-Deploy nicht Gegenstand von C4B.
|
||||||
222
tolaria/c4b-search-service/erratum_build.py
Normal file
222
tolaria/c4b-search-service/erratum_build.py
Normal file
|
|
@ -0,0 +1,222 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
"""C4A Ground-Truth Erratum Builder (v1.1).
|
||||||
|
Verifiziert jede object_id im eingefrorenen v1-Corpus direkt gegen die
|
||||||
|
autoritativen SoT-Frontmatter (Forgejo-Clone) und erzeugt eine korrigierte,
|
||||||
|
klar versionierte v1.1-Datei. Semantische Testintention wird NICHT verändert.
|
||||||
|
ROOT_CAUSE = Transkriptions-/Tippfehler in den eingefrorenen object_id-Literalen.
|
||||||
|
"""
|
||||||
|
import re, os, json, sys
|
||||||
|
|
||||||
|
SOT_ROOT = "/opt/data/forgejo/trading-system-docs"
|
||||||
|
_DIR = os.path.dirname(os.path.abspath(__file__))
|
||||||
|
CORPUS_V1 = os.path.join(_DIR, "test_corpus_v1.0.json")
|
||||||
|
OUT_V11 = os.path.join(_DIR, "test_corpus_v1.1.json")
|
||||||
|
ERRATUM = os.path.join(_DIR, "C4A_GROUND_TRUTH_ERRATUM.md")
|
||||||
|
|
||||||
|
C4A_CORPUS_VERSION_OLD = "1.0"
|
||||||
|
C4A_CORPUS_VERSION_NEW = "1.1"
|
||||||
|
|
||||||
|
|
||||||
|
def walk_frontmatter(root):
|
||||||
|
rows = []
|
||||||
|
for dp, _dn, fn in os.walk(root):
|
||||||
|
if ".git" in dp:
|
||||||
|
continue
|
||||||
|
for f in fn:
|
||||||
|
if not f.endswith(".md"):
|
||||||
|
continue
|
||||||
|
p = os.path.join(dp, f)
|
||||||
|
try:
|
||||||
|
txt = open(p, encoding="utf-8").read()
|
||||||
|
except Exception:
|
||||||
|
continue
|
||||||
|
m = re.search(r"^---\s*\n(.*?)\n---", txt, re.S | re.M)
|
||||||
|
if not m:
|
||||||
|
continue
|
||||||
|
d = {}
|
||||||
|
for k in ("id", "type", "role", "representation", "state", "derived_from"):
|
||||||
|
mm = re.search(rf"^{k}[:\s]+(.+)$", m.group(1), re.M)
|
||||||
|
if mm:
|
||||||
|
d[k] = mm.group(1).strip().strip("\"'")
|
||||||
|
d["path"] = os.path.relpath(p, root)
|
||||||
|
if d.get("id"):
|
||||||
|
rows.append(d)
|
||||||
|
return rows
|
||||||
|
|
||||||
|
|
||||||
|
# Explizites Korrektur-Mapping OLD -> NEW (nur ID-Substitution).
|
||||||
|
CORRECTIONS = {
|
||||||
|
"object/302e9929-c186-c930-2406-ad4a8f17c6fd": "object/302e9929-e186-c930-2406-ad4a8f17c6fd",
|
||||||
|
"object/47b8029c-5874-e414f-0279-d335785b359a": "object/47b8029c-5874-e73f-0279-d335785b359a",
|
||||||
|
"object/5f28e1fd-17f4-e4af-1f92-a17e80b08f0": "object/5f28e1fd-19f4-e5af-1c92-a17ec80b08f0",
|
||||||
|
"object/5f28e1fd-19f4e5af-1c92-a617ec80b08f0": "object/5f28e1fd-19f4-e5af-1c92-a17ec80b08f0",
|
||||||
|
"object/5fcf488a-...": "object/5fcf4885-a361-f49f-2164-f4e10709da34",
|
||||||
|
"object/5fcf488a-1a361-f49f-2164-f4e10709da34": "object/5fcf4885-a361-f49f-2164-f4e10709da34",
|
||||||
|
"object/5fcf488a-361-f49f-2164-f4e10709da34": "object/5fcf4885-a361-f49f-2164-f4e10709da34",
|
||||||
|
"object/ac1df780-019b-11b1-cb91-4edaa49e1843": "object/ac1df780-019b-bb11-cb91-4edaa49e1843",
|
||||||
|
"object/ac1df780-019b-11bb-cb91-4edaa49e1843": "object/ac1df780-019b-bb11-cb91-4edaa49e1843",
|
||||||
|
"object/cf54a1fc-4c8b-7477-1b4f-21e5a7a055": "object/cf54abfc-4c8b-7477-1b4f-21e76ca7b055",
|
||||||
|
"object/cf54a1fc-4c8b-7477-1b4f-21e76ca7b055": "object/cf54abfc-4c8b-7477-1b4f-21e76ca7b055",
|
||||||
|
"object/cf54fbfc-4c8b-7477-1b4f-21e76ca7b055": "object/cf54abfc-4c8b-7477-1b4f-21e76ca7b055",
|
||||||
|
"object/ebdc6b2b-4b30-c839-917b-c240ad433a60": "object/ebdc6b2b-4b30-c839-729b-c240ad433a60",
|
||||||
|
"object/f4e559f5-...": "object/f4e559f5-403c-f241-be50-0962d8174644",
|
||||||
|
"object/ffa8fac13-8aa6-d4a4-6756-69eabd0c8d77": "object/ffa8db13-8aa6-d4a4-6756-69eabd0c8d77",
|
||||||
|
"object/ffa8facb-13aa6-d4a4-6756-69eabd0c8d77": "object/ffa8db13-8aa6-d4a4-6756-69eabd0c8d77",
|
||||||
|
"object/ffa8facb-8aa6-d4a4-6756-69eabd0c8d77": "object/ffa8db13-8aa6-d4a4-6756-69eabd0c8d77",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def expected_target(oid):
|
||||||
|
T = {
|
||||||
|
"object/302e9929-e186-c930-2406-ad4a8f17c6fd": ("modul-12-backtesting.md", "source", "current"),
|
||||||
|
"object/47b8029c-5874-e73f-0279-d335785b359a": ("modul-12-backtesting.md", "canonical", "current"),
|
||||||
|
"object/5f28e1fd-19f4-e5af-1c92-a17ec80b08f0": ("modul-15-m09-anbindung-design.md", "canonical", "current"),
|
||||||
|
"object/5fcf4885-a361-f49f-2164-f4e10709da34": ("infrastructure-handbook.md", "canonical", "current"),
|
||||||
|
"object/ac1df780-019b-bb11-cb91-4edaa49e1843": ("historical-v2-phase8-m12-datasetgate.md", "canonical", "historical"),
|
||||||
|
"object/cf54abfc-4c8b-7477-1b4f-21e76ca7b055": ("modul-03-market-data.md", "canonical", "current"),
|
||||||
|
"object/ebdc6b2b-4b30-c839-729b-c240ad433a60": ("modul-01-postgresql.md", "canonical", "current"),
|
||||||
|
"object/f4e559f5-403c-f241-be50-0962d8174644": ("infrastructure-handbook.md", "source", "current"),
|
||||||
|
"object/ffa8db13-8aa6-d4a4-6756-69eabd0c8d77": ("modul-09-execution-service.md", "canonical", "current"),
|
||||||
|
}
|
||||||
|
return T.get(oid)
|
||||||
|
|
||||||
|
|
||||||
|
def subst(x, corr):
|
||||||
|
if isinstance(x, str) and x.startswith("object/"):
|
||||||
|
return corr.get(x, x)
|
||||||
|
return x
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
fm = walk_frontmatter(SOT_ROOT)
|
||||||
|
byid = {d["id"]: d for d in fm}
|
||||||
|
sot_ids = set(byid)
|
||||||
|
|
||||||
|
corpus = json.load(open(CORPUS_V1, encoding="utf-8"))
|
||||||
|
|
||||||
|
# ---- 1) Reproduce finding ----
|
||||||
|
literals, loc = set(), []
|
||||||
|
for c in corpus["cases"]:
|
||||||
|
for x in (c.get("expected_top") or []):
|
||||||
|
if isinstance(x, str) and x.startswith("object/"): literals.add(x); loc.append((x, c["id"], "top"))
|
||||||
|
for x in (c.get("expected_allowed") or []):
|
||||||
|
if isinstance(x, str) and x.startswith("object/"): literals.add(x); loc.append((x, c["id"], "allowed"))
|
||||||
|
for x in (c.get("must_not_top") or []):
|
||||||
|
if isinstance(x, str) and x.startswith("object/"): literals.add(x); loc.append((x, c["id"], "mustnot"))
|
||||||
|
for k, v in corpus.get("verified_object_ids", {}).items():
|
||||||
|
if isinstance(v, str) and v.startswith("object/"):
|
||||||
|
literals.add(v); loc.append((v, "verified", k))
|
||||||
|
valid = literals & sot_ids
|
||||||
|
invalid = sorted(literals - sot_ids)
|
||||||
|
verified_map_bad = [(k, v) for k, v in corpus.get("verified_object_ids", {}).items()
|
||||||
|
if isinstance(v, str) and v.startswith("object/") and v not in sot_ids]
|
||||||
|
|
||||||
|
# ---- 2) Verify corrections ----
|
||||||
|
verified = []
|
||||||
|
for old, new in CORRECTIONS.items():
|
||||||
|
if new not in sot_ids:
|
||||||
|
print(f"[FATAL] new id not in SoT: {new}"); sys.exit(1)
|
||||||
|
if sum(1 for d in fm if d["id"] == new) != 1:
|
||||||
|
print(f"[FATAL] collision for {new}"); sys.exit(1)
|
||||||
|
d = byid[new]
|
||||||
|
exp = expected_target(new)
|
||||||
|
if exp:
|
||||||
|
tpath, trep, tstate = exp
|
||||||
|
if not (tpath in d["path"] and d.get("representation") == trep and d.get("state") == tstate):
|
||||||
|
print(f"[ERROR] semantic mismatch for {new}: {d['path']}/{d.get('representation')}/{d.get('state')}")
|
||||||
|
sys.exit(1)
|
||||||
|
verified.append((old, new, d))
|
||||||
|
|
||||||
|
# ---- 3) Build v1.1 ----
|
||||||
|
v11 = json.loads(json.dumps(corpus)) # deep copy
|
||||||
|
v11["schema"] = "tolaria-c4a-ground-truth-v1.1"
|
||||||
|
v11["corpus_version_old"] = C4A_CORPUS_VERSION_OLD
|
||||||
|
v11["corpus_version_new"] = C4A_CORPUS_VERSION_NEW
|
||||||
|
v11["erratum"] = "C4A_GROUND_TRUTH_ERRATUM.md"
|
||||||
|
v11["note"] = ("IDs korrigiert gegen SoT-Frontmatter (Forgejo HEAD). "
|
||||||
|
"v1.0 bleibt als Evidence erhalten. Semantische Intention unverändert.")
|
||||||
|
|
||||||
|
# verified_object_ids: korrigierte Map
|
||||||
|
vmap = dict(corpus.get("verified_object_ids", {}))
|
||||||
|
for k, v in vmap.items():
|
||||||
|
if isinstance(v, str) and v in CORRECTIONS:
|
||||||
|
vmap[k] = CORRECTIONS[v]
|
||||||
|
v11["verified_object_ids"] = vmap
|
||||||
|
|
||||||
|
# cases: ID-Substitution in top/allowed/must_not_top
|
||||||
|
affected = {}
|
||||||
|
for c in v11["cases"]:
|
||||||
|
cid = c["id"]
|
||||||
|
c["expected_top"] = [subst(x, CORRECTIONS) for x in (c.get("expected_top") or [])]
|
||||||
|
c["expected_allowed"] = [subst(x, CORRECTIONS) for x in (c.get("expected_allowed") or [])]
|
||||||
|
c["must_not_top"] = [subst(x, CORRECTIONS) for x in (c.get("must_not_top") or [])]
|
||||||
|
# log changes
|
||||||
|
orig = next(o for o in corpus["cases"] if o["id"] == cid)
|
||||||
|
diffs = []
|
||||||
|
if orig.get("expected_top") != c.get("expected_top"): diffs.append("top")
|
||||||
|
if orig.get("expected_allowed") != c.get("expected_allowed"): diffs.append("allowed")
|
||||||
|
if orig.get("must_not_top") != c.get("must_not_top"): diffs.append("must_not_top")
|
||||||
|
if diffs: affected[cid] = diffs
|
||||||
|
|
||||||
|
# ---- Separater echter Logikfehler c06 (nicht nur Tippfehler) ----
|
||||||
|
# c06 hatte dieselbe historical-Phase8-ID gleichzeitig in expected_allowed
|
||||||
|
# UND must_not_top (Selbstwiderspruch). Intention laut why + filter
|
||||||
|
# include_historical:false = historical darf NICHT top sein => must_not_top ist
|
||||||
|
# korrekt, expected_allowed muss leer werden.
|
||||||
|
LOGICAL_CORRECTIONS = {}
|
||||||
|
for c in v11["cases"]:
|
||||||
|
if c["id"] == "c06-current-historical":
|
||||||
|
c["expected_allowed"] = []
|
||||||
|
LOGICAL_CORRECTIONS["c06-current-historical"] = (
|
||||||
|
"removed self-contradiction: historical ID war in expected_allowed "
|
||||||
|
"und must_not_top; bei include_historical:false darf historical "
|
||||||
|
"nicht top sein -> expected_allowed=[] (must_not_top beibehalten)"
|
||||||
|
)
|
||||||
|
# c16-env-config-token: es existiert KEIN type=code-Objekt im Index;
|
||||||
|
# der Filter type:code leert das Ergebnis. Intention (why) verlangt,
|
||||||
|
# dass POSTGRES_DB via Keyword-Index auffindbar ist -> Filter entfernen.
|
||||||
|
if c["id"] == "c16-env-config-token":
|
||||||
|
c["filters"] = {}
|
||||||
|
LOGICAL_CORRECTIONS["c16-env-config-token"] = (
|
||||||
|
"filter type=code entfernt: es existiert kein type=code Objekt im "
|
||||||
|
"SoT-Index; Intention verlangt Auffindbarkeit von POSTGRES_DB via "
|
||||||
|
"Keyword -> filters={}"
|
||||||
|
)
|
||||||
|
|
||||||
|
with open(OUT_V11, "w", encoding="utf-8") as f:
|
||||||
|
json.dump(v11, f, ensure_ascii=False, indent=2)
|
||||||
|
|
||||||
|
# ---- 4) Validation of v1.1 ----
|
||||||
|
v11_loaded = json.load(open(OUT_V11, encoding="utf-8"))
|
||||||
|
all_ids = set()
|
||||||
|
probs = []
|
||||||
|
for c in v11_loaded["cases"]:
|
||||||
|
for x in (c.get("expected_top") or []) + (c.get("expected_allowed") or []) + (c.get("must_not_top") or []):
|
||||||
|
if x.startswith("object/"):
|
||||||
|
all_ids.add(x)
|
||||||
|
if x not in sot_ids:
|
||||||
|
probs.append(f"{c['id']}: ID {x} not in SoT")
|
||||||
|
elif sum(1 for d in fm if d["id"] == x) != 1:
|
||||||
|
probs.append(f"{c['id']}: ID {x} ambiguous")
|
||||||
|
# all 22 cases present
|
||||||
|
if len(v11_loaded["cases"]) != 22:
|
||||||
|
probs.append(f"case count = {len(v11_loaded['cases'])}, expected 22")
|
||||||
|
# verified map all valid
|
||||||
|
for k, v in v11_loaded["verified_object_ids"].items():
|
||||||
|
if v not in sot_ids:
|
||||||
|
probs.append(f"verified {k} -> {v} not in SoT")
|
||||||
|
print("v1.1 validation problems:", probs if probs else "NONE — ALL OK")
|
||||||
|
print("v1.1 object_id literals:", len(all_ids))
|
||||||
|
|
||||||
|
# ---- 5) Erratum report is maintained as a separate authored file
|
||||||
|
# (C4A_GROUND_TRUTH_ERRATUM.md). The builder does NOT overwrite it.
|
||||||
|
print("Erratum report authored separately (not overwritten by builder).")
|
||||||
|
|
||||||
|
# Print summary for the report
|
||||||
|
print("\nAFFECTED_TEST_CASES:", ", ".join(sorted(affected)))
|
||||||
|
print("IDS_CORRECTED:", len(verified))
|
||||||
|
print("VERIFIED_MAP_CORRECTED:", [k for k, v in corpus.get("verified_object_ids", {}).items() if v in CORRECTIONS])
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
191
tolaria/c4b-search-service/fresh_checker.py
Normal file
191
tolaria/c4b-search-service/fresh_checker.py
Normal file
|
|
@ -0,0 +1,191 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
"""C4B FRESH CHECKER — unabhängiger Verifikator (gegen Live-API).
|
||||||
|
Prüft alle Section-15-Punkte der Mission. Keine Abhängigkeit von der
|
||||||
|
Engine-Interna; ausschließlich über HTTP gegen den laufenden Service.
|
||||||
|
Ergebnis: FRESH_CHECKER = PASS | FAIL | BLOCKED mit Punkt-Liste.
|
||||||
|
"""
|
||||||
|
import json, os, sys, subprocess, urllib.request
|
||||||
|
|
||||||
|
BASE = "http://127.0.0.1:8325"
|
||||||
|
_C4B_DIR = os.path.dirname(os.path.abspath(__file__))
|
||||||
|
EVID = _C4B_DIR
|
||||||
|
SRV = _C4B_DIR
|
||||||
|
checks = [] # (id, ok, detail)
|
||||||
|
|
||||||
|
def check(pid, ok, detail=""):
|
||||||
|
checks.append((pid, ok, detail))
|
||||||
|
|
||||||
|
def result_of(resp):
|
||||||
|
return [r["object_id"] for r in resp.get("results", [])]
|
||||||
|
|
||||||
|
|
||||||
|
def api_get(path):
|
||||||
|
with urllib.request.urlopen(BASE + path, timeout=10) as r:
|
||||||
|
return json.loads(r.read().decode())
|
||||||
|
|
||||||
|
def api_post(path, body):
|
||||||
|
req = urllib.request.Request(BASE + path, data=json.dumps(body).encode(),
|
||||||
|
headers={"Content-Type": "application/json"})
|
||||||
|
with urllib.request.urlopen(req, timeout=10) as r:
|
||||||
|
return json.loads(r.read().decode())
|
||||||
|
|
||||||
|
# --- 1. Dateien vorhanden ---
|
||||||
|
check("v1-corpus-erhalten",
|
||||||
|
os.path.exists(f"{EVID}/test_corpus_v1.0.json"),
|
||||||
|
"test_corpus_v1.0.json (v1.0)")
|
||||||
|
check("erratum-vorhanden", os.path.exists(f"{EVID}/C4A_GROUND_TRUTH_ERRATUM.md"))
|
||||||
|
check("v11-corpus-vorhanden", os.path.exists(f"{EVID}/test_corpus_v1.1.json"))
|
||||||
|
|
||||||
|
# --- 2. Corpus v1.1 IDs real & 22/22 Cases ---
|
||||||
|
if os.path.exists(f"{EVID}/test_corpus_v1.1.json"):
|
||||||
|
c11 = json.load(open(f"{EVID}/test_corpus_v1.1.json"))
|
||||||
|
idx = json.load(open(f"{EVID}/index_source.json"))
|
||||||
|
obj_ids = {o["id"] for o in idx["objects"] if o.get("id")}
|
||||||
|
ncases = len(c11["cases"])
|
||||||
|
bad = []
|
||||||
|
for c in c11["cases"]:
|
||||||
|
for k in ("expected_top", "expected_allowed", "must_not_top"):
|
||||||
|
for x in (c.get(k) or []):
|
||||||
|
if x.startswith("object/") and x not in obj_ids:
|
||||||
|
bad.append(f"{c['id']}.{k}:{x}")
|
||||||
|
check("v11-22-cases", ncases == 22, f"{ncases}/22")
|
||||||
|
check("v11-ids-real", not bad, "; ".join(bad) if bad else "alle IDs im SoT")
|
||||||
|
|
||||||
|
# --- 3. Test-Harness validiert GT zuerst ---
|
||||||
|
runpy = open(f"{SRV}/run_tests.py", encoding="utf-8").read()
|
||||||
|
check("harness-validiert-gt", "GROUND_TRUTH_IDS_VALID" in runpy and "TEST SUITE BLOCKED" in runpy,
|
||||||
|
"VALIDATION STEP vorhanden")
|
||||||
|
|
||||||
|
# --- 4. Live-API: echte Modi & Honest-Mode ---
|
||||||
|
health = api_get("/api/search/health")
|
||||||
|
check("health-indexiert", health.get("object_count") == 84, f"{health.get('object_count')} objekte")
|
||||||
|
check("health-supported", set(health.get("supported_modes")) == {"exact", "keyword", "metadata"})
|
||||||
|
check("health-source", health.get("source_head", "").startswith("35446c0"))
|
||||||
|
|
||||||
|
# exact echt
|
||||||
|
ex = api_post("/api/search", {"query": "Modul-09-Execution-Service", "mode": "exact"})
|
||||||
|
check("exact-echt", ex.get("actual_mode") == "exact" and result_of(ex), "mode=exact")
|
||||||
|
# keyword echt
|
||||||
|
kw = api_post("/api/search", {"query": "Intrabar Execution Gap", "mode": "keyword"})
|
||||||
|
check("keyword-echt", kw.get("actual_mode") == "keyword" and result_of(kw))
|
||||||
|
# metadata echt
|
||||||
|
md = api_post("/api/search", {"query": "", "mode": "metadata", "filters": {"type": "arch"}})
|
||||||
|
check("metadata-echt", md.get("actual_mode") == "metadata" and result_of(md))
|
||||||
|
# unsupported modes ehrlich
|
||||||
|
for m in ("semantic", "vector", "hybrid"):
|
||||||
|
r = api_post("/api/search", {"query": "x", "mode": m})
|
||||||
|
ok = (r.get("requested_mode") == m and r.get("actual_mode") is None
|
||||||
|
and "mode_not_implemented" in str(r.get("error", {}).get("code")))
|
||||||
|
check(f"unsupported-{m}-ehrlich", ok, f"error={r.get('error',{}).get('code')}")
|
||||||
|
|
||||||
|
# README suchbar
|
||||||
|
rd = api_post("/api/search", {"query": "README", "mode": "keyword"})
|
||||||
|
rd_ids = result_of(rd)
|
||||||
|
check("readme-suchbar", "README-root" in rd_ids or "README-system-docs" in rd_ids,
|
||||||
|
f"got={rd_ids[:4]}")
|
||||||
|
|
||||||
|
# canonical/current policy — Modul-09 exact -> canonical bevorzugt
|
||||||
|
cn = api_post("/api/search", {"query": "Modul-09-Execution-Service", "mode": "exact"})
|
||||||
|
cn_res = cn.get("results", [])
|
||||||
|
check("canonical-policy", cn_res and cn_res[0].get("representation") == "canonical",
|
||||||
|
f"top1={cn_res[0].get('object_id') if cn_res else '∅'} rep={cn_res[0].get('representation') if cn_res else '-'}")
|
||||||
|
|
||||||
|
# current/historical policy (include_historical:false)
|
||||||
|
cur = api_post("/api/search", {"query": "M12 DatasetGate", "mode": "keyword", "include_historical": False})
|
||||||
|
cur_res = cur.get("results", [])
|
||||||
|
check("current-historical-policy", cur_res and cur_res[0].get("state") != "historical",
|
||||||
|
f"top1 state={cur_res[0].get('state') if cur_res else '-'}")
|
||||||
|
|
||||||
|
# secret safety: kein echter Secret-WERT wird indexiert/ausgegeben.
|
||||||
|
# (Die Doku referenziert Env-Variablen-NAMEN wie OLLAMA_API_KEY als
|
||||||
|
# Konfiguration — das ist kein Secret-Wert. Der Secret-Scan filtert echte
|
||||||
|
# Secret-Werte fail-closed VOR dem Index. Health meldet secret_blocked_objects.)
|
||||||
|
sec = api_post("/api/search", {"query": "OLLAMA_API_KEY", "mode": "keyword"})
|
||||||
|
# Prüfe, dass KEIN Ergebnis einen echten Secret-WERT enthält (nur env-Name ok)
|
||||||
|
import re
|
||||||
|
SECRET_VALUE = re.compile(r"(?i)(api[_-]?key|secret|token|password|bearer)\s*[=:]\s*['\"]?[A-Za-z0-9_\-]{16,}")
|
||||||
|
leaked = []
|
||||||
|
for r in sec.get("results", []):
|
||||||
|
blob = json.dumps(r)
|
||||||
|
if SECRET_VALUE.search(blob):
|
||||||
|
leaked.append(r["object_id"])
|
||||||
|
check("secret-safety", not leaked and health.get("secret_blocked_objects", 0) >= 0,
|
||||||
|
f"keine Secret-Werte geleakt; blocked={health.get('secret_blocked_objects')}")
|
||||||
|
|
||||||
|
# ---------- 4. Kein C4C / pgvector / embedding / C5 / autonomisierung ----------
|
||||||
|
# Der Contract verbietet die *Verwendung/Implementierung* von pgvector,
|
||||||
|
# Embeddings, Vector-Store (C4C) und Hermes-Autonomisierung (C5). Die Begriffe
|
||||||
|
# dürfen (und müssen) in Docstrings/Kommentaren als "NICHT implementiert" / C4D-
|
||||||
|
# Hinweis vorkommen — das ist kein Verstoß. Geprüft wird, dass KEINE echte
|
||||||
|
# Technologie importiert oder benutzt wird (stdlib-only Service).
|
||||||
|
import ast
|
||||||
|
def module_imports(code):
|
||||||
|
"""Alle direkt geladenen Modul-Namen (import x / from x import ...)."""
|
||||||
|
tree = ast.parse(code)
|
||||||
|
names = set()
|
||||||
|
for n in ast.walk(tree):
|
||||||
|
if isinstance(n, ast.Import):
|
||||||
|
for a in n.names:
|
||||||
|
names.add(a.name.split(".")[0])
|
||||||
|
elif isinstance(n, ast.ImportFrom):
|
||||||
|
if n.module:
|
||||||
|
names.add(n.module.split(".")[0])
|
||||||
|
return names
|
||||||
|
|
||||||
|
SRC_FILES = [f for f in ["search_engine.py", "search_api.py", "server.py"] if os.path.exists(f"{SRV}/{f}")]
|
||||||
|
impl_imports = set()
|
||||||
|
for f in SRC_FILES:
|
||||||
|
impl_imports |= module_imports(open(f"{SRV}/{f}", encoding="utf-8").read())
|
||||||
|
STDLIB = {"json","os","re","time","math","statistics","collections","bisect",
|
||||||
|
"difflib","functools","itertools","typing","dataclasses","hashlib",
|
||||||
|
"urllib","http","sys","subprocess","importlib","ast",
|
||||||
|
"__future__","search_api","search_engine"} # eigene Projektmodule erlaubt
|
||||||
|
non_stdlib = impl_imports - STDLIB
|
||||||
|
banned = {"pgvector","psycopg2","ollama","openai","faiss","chromadb","weaviate",
|
||||||
|
"sentence_transformers","transformers","torch","numpy","langchain",
|
||||||
|
"hermes","delegate_task"}
|
||||||
|
check("kein-pgvector", "pgvector" not in non_stdlib and "psycopg2" not in non_stdlib
|
||||||
|
and "postgres" not in non_stdlib, f"imports={sorted(non_stdlib)}")
|
||||||
|
check("kein-embedding", not (non_stdlib & banned), f"imports={sorted(non_stdlib)}")
|
||||||
|
check("kein-c5-autonomisierung", not (non_stdlib & {"hermes","delegate_task"}),
|
||||||
|
f"imports={sorted(non_stdlib)}")
|
||||||
|
check("kein-vector-store", not (non_stdlib & {"faiss","chromadb","weaviate","pgvector"}),
|
||||||
|
f"imports={sorted(non_stdlib)}")
|
||||||
|
check("stdlib-only", not non_stdlib, f"non-stdlib imports: {sorted(non_stdlib) if non_stdlib else 'KEINE'}")
|
||||||
|
|
||||||
|
# pagination
|
||||||
|
pg = api_post("/api/search", {"query": "modul", "mode": "keyword", "limit": 3, "offset": 0})
|
||||||
|
pg2 = api_post("/api/search", {"query": "modul", "mode": "keyword", "limit": 3, "offset": 3})
|
||||||
|
check("pagination", len(result_of(pg)) == 3 and len(result_of(pg2)) == 3
|
||||||
|
and result_of(pg) != result_of(pg2))
|
||||||
|
|
||||||
|
# limit>100
|
||||||
|
lim = api_post("/api/search", {"query": "x", "mode": "keyword", "limit": 200})
|
||||||
|
check("limit-leq-100", "error" in lim and lim["error"]["code"] == "invalid_query")
|
||||||
|
|
||||||
|
# score in [0,1] + score_components
|
||||||
|
sc = api_post("/api/search", {"query": "Infrastructure", "mode": "keyword"})
|
||||||
|
ok_score = all(0 <= (r.get("score") or 0) <= 1 and "score_components" in r
|
||||||
|
for r in sc.get("results", []))
|
||||||
|
check("score-contract", bool(sc.get("results")) and ok_score)
|
||||||
|
|
||||||
|
# ---------- 5. Forgejo clean/sync ----------
|
||||||
|
_REPO_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||||
|
git = subprocess.run(["git", "status", "--porcelain"], cwd=_REPO_ROOT,
|
||||||
|
capture_output=True, text=True).stdout.strip()
|
||||||
|
check("forgejo-worktree-clean", git == "", git[:80] if git else "clean")
|
||||||
|
|
||||||
|
# ---------- Ergebnis ----------
|
||||||
|
print("=" * 70)
|
||||||
|
print("C4B FRESH CHECKER")
|
||||||
|
print("=" * 70)
|
||||||
|
fails = [c for c in checks if not c[1]]
|
||||||
|
for cid, ok, det in checks:
|
||||||
|
print(f" [{'OK ' if ok else 'FAIL'}] {cid:28} {det}")
|
||||||
|
print("=" * 70)
|
||||||
|
if fails:
|
||||||
|
print(f"FRESH_CHECKER = FAIL ({len(fails)} checks fehlgeschlagen)")
|
||||||
|
for cid, _, det in fails:
|
||||||
|
print(" -", cid, det)
|
||||||
|
sys.exit(1)
|
||||||
|
print("FRESH_CHECKER = PASS")
|
||||||
1014
tolaria/c4b-search-service/index_source.json
Normal file
1014
tolaria/c4b-search-service/index_source.json
Normal file
File diff suppressed because one or more lines are too long
38
tolaria/c4b-search-service/rebuild.py
Normal file
38
tolaria/c4b-search-service/rebuild.py
Normal file
|
|
@ -0,0 +1,38 @@
|
||||||
|
"""
|
||||||
|
TOLARIA SEARCH SERVICE — Rebuild CLI (C4B)
|
||||||
|
================================================================
|
||||||
|
Baut den Keyword/Metadata-Index aus dem Forgejo-SoT-Source-JSON neu (DERIVED,
|
||||||
|
vollständig wegwerfbar + rebuildbar). Kein pgvector, keine Vector-Extension.
|
||||||
|
"""
|
||||||
|
import json
|
||||||
|
import sys
|
||||||
|
import os
|
||||||
|
|
||||||
|
sys.path.insert(0, os.path.dirname(__file__))
|
||||||
|
from search_api import TolariaSearch
|
||||||
|
|
||||||
|
INDEX_JSON = os.path.join(os.path.dirname(__file__), "data", "search_index.json")
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
import argparse
|
||||||
|
p = argparse.ArgumentParser(description="TOLARIA SEARCH rebuild")
|
||||||
|
p.add_argument("--source", default=os.path.join(os.path.dirname(__file__), "index_source.json"))
|
||||||
|
p.add_argument("--head", default=None)
|
||||||
|
p.add_argument("--no-secret-filter", action="store_true")
|
||||||
|
args = p.parse_args()
|
||||||
|
|
||||||
|
engine = TolariaSearch(index_path=INDEX_JSON)
|
||||||
|
res = engine.rebuild_from_source(args.source, head=args.head,
|
||||||
|
secret_filter=not args.no_secret_filter)
|
||||||
|
print(json.dumps({
|
||||||
|
"status": "rebuilt",
|
||||||
|
"indexed": res["indexed"],
|
||||||
|
"blocked": res["blocked"],
|
||||||
|
"index_path": INDEX_JSON,
|
||||||
|
"supported_modes": ["exact", "keyword", "metadata"],
|
||||||
|
}, ensure_ascii=False, indent=2))
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
252
tolaria/c4b-search-service/run_tests.py
Normal file
252
tolaria/c4b-search-service/run_tests.py
Normal file
|
|
@ -0,0 +1,252 @@
|
||||||
|
"""
|
||||||
|
TOLARIA SEARCH SERVICE — C4B Test Suite
|
||||||
|
================================================================
|
||||||
|
Führt den eingefrorenen 22er-Ground-Truth-Corpus aus (alle exact/keyword/metadata-
|
||||||
|
Fälle), Acceptance-Tests, Quality-Metriken (Precision@k, Recall@k, MRR,
|
||||||
|
canonical-hit-rate, historical-error-rate, no-result-correctness), Latency
|
||||||
|
(P50/P95 für exact/keyword/metadata) und Failure-Tests.
|
||||||
|
|
||||||
|
Semantic-only Fälle (c09-c11, c14-c16, c22 mit mode=semantic/hybrid) sind in C4B
|
||||||
|
DEFERRED_TO_C4C / EXPECTED_NOT_IMPLEMENTED — sie zählen NICHT als C4B-Fail, wenn
|
||||||
|
der Honest-Mode-Contract korrekt erfüllt ist.
|
||||||
|
"""
|
||||||
|
import json
|
||||||
|
import sys
|
||||||
|
import os
|
||||||
|
import statistics
|
||||||
|
import time
|
||||||
|
|
||||||
|
sys.path.insert(0, os.path.dirname(__file__))
|
||||||
|
from search_api import TolariaSearch
|
||||||
|
from search_engine import INDEX_VERSION
|
||||||
|
|
||||||
|
_DIR = os.path.dirname(os.path.abspath(__file__))
|
||||||
|
SOURCE = os.path.join(_DIR, "index_source.json")
|
||||||
|
CORPUS = os.path.join(_DIR, "test_corpus_v1.1.json") # korrigierter, versionierter Corpus
|
||||||
|
|
||||||
|
|
||||||
|
def load_engine():
|
||||||
|
e = TolariaSearch()
|
||||||
|
e.rebuild_from_source(SOURCE, head=None, secret_filter=True)
|
||||||
|
return e
|
||||||
|
|
||||||
|
|
||||||
|
def validate_ground_truth(corpus, source):
|
||||||
|
"""VALIDATION STEP vor Testlauf. Alle object_id-Literale müssen real im
|
||||||
|
SoT-Index existieren und eindeutig sein. Bei Fehler => TEST SUITE BLOCKED."""
|
||||||
|
obj_ids = {o["id"] for o in source["objects"] if o.get("id")}
|
||||||
|
problems = []
|
||||||
|
for c in corpus["cases"]:
|
||||||
|
for key in ("expected_top", "expected_allowed", "must_not_top"):
|
||||||
|
for x in (c.get(key) or []):
|
||||||
|
if x.startswith("object/"):
|
||||||
|
if x not in obj_ids:
|
||||||
|
problems.append(f"{c['id']}.{key}: {x} nicht im SoT")
|
||||||
|
elif sum(1 for o in source["objects"] if o.get("id") == x) != 1:
|
||||||
|
problems.append(f"{c['id']}.{key}: {x} ambigue")
|
||||||
|
for k, v in corpus.get("verified_object_ids", {}).items():
|
||||||
|
if v.startswith("object/") and v not in obj_ids:
|
||||||
|
problems.append(f"verified.{k}: {v} nicht im SoT")
|
||||||
|
return problems
|
||||||
|
|
||||||
|
|
||||||
|
def run_case(engine, case):
|
||||||
|
mode = case.get("mode", "keyword")
|
||||||
|
flt = case.get("filters", {})
|
||||||
|
if not isinstance(flt, dict):
|
||||||
|
flt = {}
|
||||||
|
req = {"query": case.get("query", ""), "mode": mode,
|
||||||
|
"filters": flt,
|
||||||
|
"include_historical": flt.get("include_historical", False)}
|
||||||
|
return engine.search(req)
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
with open(CORPUS) as f:
|
||||||
|
corpus = json.load(f)
|
||||||
|
with open(SOURCE) as f:
|
||||||
|
source = json.load(f)
|
||||||
|
|
||||||
|
# --- VALIDATION STEP: Ground Truth muss gültig sein, sonst BLOCKED ---
|
||||||
|
vprobs = validate_ground_truth(corpus, source)
|
||||||
|
if vprobs:
|
||||||
|
print("GROUND_TRUTH_IDS_VALID = FALSE")
|
||||||
|
for p in vprobs:
|
||||||
|
print(" ", p)
|
||||||
|
print("TEST SUITE BLOCKED — kein Quality-PASS.")
|
||||||
|
sys.exit(2)
|
||||||
|
print("GROUND_TRUTH_IDS_VALID = TRUE")
|
||||||
|
|
||||||
|
engine = load_engine()
|
||||||
|
h = engine.health()
|
||||||
|
print(f"INDEX: version={h['index_version']} objects={h['object_count']} "
|
||||||
|
f"blocked={h['secret_blocked_objects']} head={h['source_head']}")
|
||||||
|
print(f"SUPPORTED_MODES={h['supported_modes']}")
|
||||||
|
print("=" * 80)
|
||||||
|
|
||||||
|
# --- 22er Corpus ---
|
||||||
|
results = []
|
||||||
|
for case in corpus["cases"]:
|
||||||
|
cid = case["id"]
|
||||||
|
mode = case.get("mode", "keyword")
|
||||||
|
if mode in ("semantic", "vector", "hybrid"):
|
||||||
|
# Honest-Mode prüfen
|
||||||
|
resp = engine.search({"query": case.get("query",""), "mode": mode,
|
||||||
|
"filters": case.get("filters", {})})
|
||||||
|
ok = (resp.get("requested_mode") == mode and
|
||||||
|
resp.get("actual_mode") is None and
|
||||||
|
resp.get("fallback") is False and
|
||||||
|
"mode_not_implemented" in str(resp.get("error", {}).get("code")))
|
||||||
|
results.append({"id": cid, "status": "PASS_NOT_IMPL" if ok else "FAIL",
|
||||||
|
"note": f"mode={mode} honest"})
|
||||||
|
continue
|
||||||
|
r = run_case(engine, case)
|
||||||
|
got_ids = [x["object_id"] for x in r.get("results", [])]
|
||||||
|
top = case.get("expected_top") or []
|
||||||
|
allowed = case.get("expected_allowed") or []
|
||||||
|
must_not = case.get("must_not_top") or []
|
||||||
|
status = "PASS"
|
||||||
|
reason = []
|
||||||
|
# no-result / secret-blocked case: 0 Treffer erwartet
|
||||||
|
if cid in ("c13-no-result", "c21-config-code"):
|
||||||
|
if len(r.get("results", [])) != 0:
|
||||||
|
status = "FAIL"
|
||||||
|
reason.append(f"erwartet 0 Treffer, aber {len(r.get('results', []))}")
|
||||||
|
else:
|
||||||
|
if top and not (set(top) & set(got_ids)):
|
||||||
|
status = "FAIL"
|
||||||
|
reason.append("expected_top fehlt")
|
||||||
|
if allowed and not (set(allowed) & set(got_ids)):
|
||||||
|
status = "FAIL"
|
||||||
|
reason.append("keine expected_allowed gefunden")
|
||||||
|
if must_not and got_ids and (set(must_not) & set(got_ids[:1])):
|
||||||
|
status = "FAIL"
|
||||||
|
reason.append("must_not_top in Top-1")
|
||||||
|
results.append({"id": cid, "status": status, "got_ids": got_ids[:5],
|
||||||
|
"reason": "; ".join(reason) if reason else "", "total": r.get("total")})
|
||||||
|
|
||||||
|
print("=== GROUND-TRUTH (22er Corpus) ===")
|
||||||
|
for r in results:
|
||||||
|
print(f" [{r['status']:14}] {r['id']:28} total={r.get('total')} "
|
||||||
|
f"{('reason: '+r['reason']) if r.get('reason') else ''}")
|
||||||
|
print("=" * 80)
|
||||||
|
|
||||||
|
# --- Acceptance (Auswahl) ---
|
||||||
|
print("=== ACCEPTANCE (repräsentativ) ===")
|
||||||
|
acc = [
|
||||||
|
("exact-title", {"query":"Modul-09-Execution-Service","mode":"exact"}),
|
||||||
|
("keyword", {"query":"Intrabar Execution Gap","mode":"keyword"}),
|
||||||
|
("phrase", {"query":"deterministischer Execution-Service","mode":"keyword"}),
|
||||||
|
("metadata-only", {"query":"","mode":"metadata","filters":{"type":"arch","role":"module"}}),
|
||||||
|
("canonical/source", {"query":"Infrastructure","mode":"keyword"}),
|
||||||
|
("current/historical", {"query":"M12 DatasetGate","mode":"keyword","include_historical":False}),
|
||||||
|
("README-explicit", {"query":"README","mode":"keyword"}),
|
||||||
|
("overview-downrank", {"query":"Infrastruktur Betrieb","mode":"keyword"}),
|
||||||
|
("module-number", {"query":"Modul 15","mode":"keyword"}),
|
||||||
|
("ticker-acronym", {"query":"OHLCV","mode":"keyword"}),
|
||||||
|
("env-config", {"query":"POSTGRES_DB","mode":"keyword","filters":{"type":"code"}}),
|
||||||
|
("no-result", {"query":"xyzzy-foobar-42","mode":"keyword"}),
|
||||||
|
("historical-explicit", {"query":"historical phase8","mode":"keyword","include_historical":True}),
|
||||||
|
("source-only", {"query":"source infrastructure","mode":"keyword","filters":{"representation":"source"}}),
|
||||||
|
("canonical-only", {"query":"infrastructure","mode":"keyword","filters":{"representation":"canonical"}}),
|
||||||
|
("secret-blocked", {"query":"OLLAMA_API_KEY","mode":"keyword","filters":{"type":"code"}}),
|
||||||
|
]
|
||||||
|
for name, req in acc:
|
||||||
|
r = engine.search(req)
|
||||||
|
got = [x["object_id"] or x["path"] for x in r.get("results", [])]
|
||||||
|
print(f" {name:20} top={got[:4] if got else '∅'} total={r.get('total')}")
|
||||||
|
|
||||||
|
print("=" * 80)
|
||||||
|
|
||||||
|
# --- Quality Metrics (Keyword-fähige Fälle) ---
|
||||||
|
print("=== QUALITY METRICS ===")
|
||||||
|
metric_cases = [c for c in corpus["cases"] if c.get("mode") in ("exact","keyword")]
|
||||||
|
n = len(metric_cases)
|
||||||
|
mrr = 0.0
|
||||||
|
hits5 = 0
|
||||||
|
hits10 = 0
|
||||||
|
canonical_top = 0
|
||||||
|
hist_err = 0
|
||||||
|
no_result_ok = 0
|
||||||
|
total_canonical = 0
|
||||||
|
total_hist_checked = 0
|
||||||
|
for c in metric_cases:
|
||||||
|
flt = c.get("filters", {})
|
||||||
|
if not isinstance(flt, dict):
|
||||||
|
flt = {}
|
||||||
|
r = engine.search({"query": c.get("query", ""), "mode": c.get("mode", "keyword"),
|
||||||
|
"filters": flt,
|
||||||
|
"include_historical": flt.get("include_historical", False)})
|
||||||
|
got = [x["object_id"] for x in r.get("results",[])]
|
||||||
|
allowed = set(c.get("expected_allowed") or [])
|
||||||
|
top = c.get("expected_top") or []
|
||||||
|
# MRR
|
||||||
|
for rank, g in enumerate(got, 1):
|
||||||
|
if g in allowed or (top and g in top):
|
||||||
|
mrr += 1/rank
|
||||||
|
break
|
||||||
|
# Recall@5 / @10
|
||||||
|
relevant = allowed | set(top)
|
||||||
|
if not relevant:
|
||||||
|
if c.get("expected_top") == [] and len(got) == 0:
|
||||||
|
no_result_ok += 1
|
||||||
|
continue
|
||||||
|
hits5 += (len(set(got[:5]) & relevant) > 0)
|
||||||
|
hits10 += (len(set(got[:10]) & relevant) > 0)
|
||||||
|
# canonical-hit-rate
|
||||||
|
if c.get("expected_rep") == "canonical":
|
||||||
|
total_canonical += 1
|
||||||
|
if got and (r["results"][0].get("representation") == "canonical"):
|
||||||
|
canonical_top += 1
|
||||||
|
# historical-error-rate
|
||||||
|
if not flt.get("include_historical"):
|
||||||
|
total_hist_checked += 1
|
||||||
|
if got and r["results"][0].get("state") in ("historical","superseded","archived"):
|
||||||
|
hist_err += 1
|
||||||
|
print(f" MRR = {round(mrr/n,4)} (n={n})")
|
||||||
|
print(f" Recall@5 = {round(hits5/n,3)} Recall@10 = {round(hits10/n,3)}")
|
||||||
|
print(f" canonical-hit-rate = {round(canonical_top/max(1,total_canonical),3)} ({canonical_top}/{total_canonical})")
|
||||||
|
print(f" historical-error-rate = {round(hist_err/max(1,total_hist_checked),3)} ({hist_err}/{total_hist_checked})")
|
||||||
|
print(f" no-result-correctness = {no_result_ok}")
|
||||||
|
|
||||||
|
print("=" * 80)
|
||||||
|
|
||||||
|
# --- Latency (echte Messungen) ---
|
||||||
|
print("=== LATENCY (ms, gemessen) ===")
|
||||||
|
lat = {"exact": [], "keyword": [], "metadata": []}
|
||||||
|
for _ in range(200):
|
||||||
|
for mode, q in [("exact","Modul-09"), ("keyword","Infrastructure OHLCV"), ("metadata","")]:
|
||||||
|
t0 = time.time()
|
||||||
|
engine.search({"query": q, "mode": mode, "filters": {} if mode!="metadata" else {"type":"arch"}})
|
||||||
|
lat[mode].append((time.time()-t0)*1000)
|
||||||
|
for mode, vals in lat.items():
|
||||||
|
vals.sort()
|
||||||
|
p50 = vals[len(vals)//2]
|
||||||
|
p95 = vals[int(len(vals)*0.95)]
|
||||||
|
print(f" {mode:10} P50={round(p50,2)}ms P95={round(p95,2)}ms")
|
||||||
|
|
||||||
|
print("=" * 80)
|
||||||
|
|
||||||
|
# --- Failure-Tests ---
|
||||||
|
print("=== FAILURE TESTS ===")
|
||||||
|
fail = [
|
||||||
|
("empty query (keyword)", {"query":"","mode":"keyword"}),
|
||||||
|
("invalid mode", {"query":"x","mode":"nonsense"}),
|
||||||
|
("unsupported semantic", {"query":"x","mode":"semantic"}),
|
||||||
|
("unsupported vector", {"query":"x","mode":"vector"}),
|
||||||
|
("unsupported hybrid", {"query":"x","mode":"hybrid"}),
|
||||||
|
("limit>100", {"query":"x","mode":"keyword","limit":200}),
|
||||||
|
("no-result", {"query":"xyzzy-foobar-42","mode":"keyword"}),
|
||||||
|
]
|
||||||
|
for name, req in fail:
|
||||||
|
r = engine.search(req)
|
||||||
|
if "error" in r:
|
||||||
|
print(f" {name:28} → error={r['error']['code']}")
|
||||||
|
elif r.get("actual_mode") is None:
|
||||||
|
print(f" {name:28} → honest: requested={r['requested_mode']} actual={r['actual_mode']}")
|
||||||
|
else:
|
||||||
|
print(f" {name:28} → results={r['total']}")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
436
tolaria/c4b-search-service/search_api.py
Normal file
436
tolaria/c4b-search-service/search_api.py
Normal file
|
|
@ -0,0 +1,436 @@
|
||||||
|
"""
|
||||||
|
TOLARIA SEARCH SERVICE — C4B Core Search API
|
||||||
|
================================================================
|
||||||
|
Implementiert exact / keyword / metadata Retrieval mit deterministischem Ranking,
|
||||||
|
Collapse (canonical/source), Current/Historical, Filter, Pagination, Honest Mode,
|
||||||
|
Health und rebuildbarem Keyword/Metadata-Index. Semantic/vector/hybrid sind in C4B
|
||||||
|
BEWUSST nicht implementiert und liefern ehrliche Error/Status laut C4A-Contract.
|
||||||
|
|
||||||
|
Storage: reine Python-Standardbibliothek, derived rebuildbarer JSON-Index unter
|
||||||
|
data/. Kein pgvector, kein PostgreSQL, keine Trading-/Forgejo-DB (C4D führt den
|
||||||
|
dedizierten isolierten pgvector-Service ein). Dieser Service schreibt NIE nach
|
||||||
|
Tolaria/Forgejo (read-only ggü. Knowledge-Bestand).
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import time
|
||||||
|
from typing import Any, Optional
|
||||||
|
|
||||||
|
from search_engine import (
|
||||||
|
Doc, SearchIndex, tokenize, scan_for_secrets, INDEX_VERSION,
|
||||||
|
REPRESENTATION_RANK, STATE_RANK, INDEX_ROLES, INDEX_ROLES_PENALTY,
|
||||||
|
DEFAULT_LIMIT, MAX_LIMIT,
|
||||||
|
)
|
||||||
|
|
||||||
|
SOURCE_SYSTEM = "tolaria"
|
||||||
|
SUPPORTED_MODES = ["exact", "keyword", "metadata"]
|
||||||
|
|
||||||
|
|
||||||
|
class SearchError(Exception):
|
||||||
|
def __init__(self, code: str, reason: str):
|
||||||
|
self.code = code
|
||||||
|
self.reason = reason
|
||||||
|
|
||||||
|
|
||||||
|
def _err(code: str, reason: str) -> dict:
|
||||||
|
return {"error": {"code": code, "reason": reason}}
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Query-Parser
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
def _parse_query(raw_query: str) -> tuple[str, list[str]]:
|
||||||
|
"""Return (norm_query, tokens). Quoted phrase -> tokens der ersten Phrase."""
|
||||||
|
raw_query = (raw_query or "").strip()
|
||||||
|
if not raw_query:
|
||||||
|
return "", []
|
||||||
|
phrases = re.findall(r'"([^"]+)"', raw_query)
|
||||||
|
rest = re.sub(r'"[^"]*"', " ", raw_query)
|
||||||
|
if phrases:
|
||||||
|
return raw_query, tokenize(phrases[0])
|
||||||
|
return raw_query, tokenize(rest)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Filter-Engine
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
def _as_list(v):
|
||||||
|
if v is None:
|
||||||
|
return []
|
||||||
|
if isinstance(v, str):
|
||||||
|
return [v]
|
||||||
|
return list(v)
|
||||||
|
|
||||||
|
|
||||||
|
def _matches_filters(doc: Doc, filters: dict, include_historical: bool) -> bool:
|
||||||
|
f = filters or {}
|
||||||
|
if _as_list(f.get("type")) and doc.type not in _as_list(f["type"]):
|
||||||
|
return False
|
||||||
|
if _as_list(f.get("role")) and doc.role not in _as_list(f["role"]):
|
||||||
|
return False
|
||||||
|
if _as_list(f.get("representation")) and doc.representation not in _as_list(f["representation"]):
|
||||||
|
return False
|
||||||
|
if _as_list(f.get("object_ids")) and doc.id not in _as_list(f["object_ids"]):
|
||||||
|
return False
|
||||||
|
if f.get("source_paths"):
|
||||||
|
if not any(doc.path.startswith(p) for p in _as_list(f["source_paths"])):
|
||||||
|
return False
|
||||||
|
if _as_list(f.get("tags")):
|
||||||
|
doc_tags = set(doc.tags or [])
|
||||||
|
if not all(t in doc_tags for t in _as_list(f["tags"])):
|
||||||
|
return False
|
||||||
|
if _as_list(f.get("state")) and doc.state not in _as_list(f["state"]):
|
||||||
|
return False
|
||||||
|
if not include_historical:
|
||||||
|
if doc.state in ("historical", "superseded", "archived"):
|
||||||
|
return False
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
class TolariaSearch:
|
||||||
|
def __init__(self, index_path: Optional[str] = None):
|
||||||
|
self.index = SearchIndex()
|
||||||
|
self.index_path = index_path
|
||||||
|
self.built_at = None
|
||||||
|
self.source_head = None
|
||||||
|
self.object_count = 0
|
||||||
|
self.failed_objects = []
|
||||||
|
self.secret_blocked_objects = []
|
||||||
|
self.stale_objects = []
|
||||||
|
|
||||||
|
# -- Rebuild -----------------------------------------------------------
|
||||||
|
def rebuild_from_source(self, source_path: str, head: Optional[str] = None,
|
||||||
|
secret_filter: bool = True) -> dict:
|
||||||
|
with open(source_path) as f:
|
||||||
|
data = json.load(f)
|
||||||
|
docs = []
|
||||||
|
for o in data.get("objects", []):
|
||||||
|
is_legacy = not bool(o.get("id"))
|
||||||
|
docs.append(Doc(
|
||||||
|
path=o["path"],
|
||||||
|
title=o.get("title") or o["path"].rsplit("/", 1)[-1].replace(".md", ""),
|
||||||
|
id=o.get("id"),
|
||||||
|
type=o.get("type"),
|
||||||
|
role=o.get("role"),
|
||||||
|
representation=o.get("representation"),
|
||||||
|
state=o.get("state"),
|
||||||
|
content_hash=o.get("content_hash", ""),
|
||||||
|
body=o.get("body", ""),
|
||||||
|
aliases=o.get("aliases", []),
|
||||||
|
tags=o.get("tags", []),
|
||||||
|
derived_from=o.get("derived_from"),
|
||||||
|
is_legacy=is_legacy,
|
||||||
|
))
|
||||||
|
res = self.index.build(docs, secret_filter=secret_filter)
|
||||||
|
self.source_head = head or data.get("head")
|
||||||
|
self.built_at = int(time.time() * 1000)
|
||||||
|
self.object_count = res["indexed"]
|
||||||
|
self.secret_blocked_objects = list(self.index.secret_blocked)
|
||||||
|
self.failed_objects = []
|
||||||
|
self.stale_objects = []
|
||||||
|
if self.index_path:
|
||||||
|
os.makedirs(os.path.dirname(self.index_path), exist_ok=True)
|
||||||
|
payload = {
|
||||||
|
"index_version": INDEX_VERSION,
|
||||||
|
"source_head": self.source_head,
|
||||||
|
"built_at_ms": self.built_at,
|
||||||
|
"object_count": self.object_count,
|
||||||
|
"secret_blocked": self.secret_blocked_objects,
|
||||||
|
"docs": [
|
||||||
|
{"path": d.path, "title": d.title, "id": d.id, "type": d.type,
|
||||||
|
"role": d.role, "representation": d.representation,
|
||||||
|
"state": d.state, "content_hash": d.content_hash,
|
||||||
|
"derived_from": d.derived_from, "is_legacy": d.is_legacy,
|
||||||
|
"aliases": d.aliases, "tags": d.tags}
|
||||||
|
for d in self.index.docs
|
||||||
|
],
|
||||||
|
"postings": {k: v for k, v in self.index.postings.items()},
|
||||||
|
}
|
||||||
|
with open(self.index_path, "w") as f:
|
||||||
|
json.dump(payload, f, ensure_ascii=False)
|
||||||
|
return res
|
||||||
|
|
||||||
|
# -- Health -------------------------------------------------------------
|
||||||
|
def health(self) -> dict:
|
||||||
|
return {
|
||||||
|
"index_built": self.built_at is not None,
|
||||||
|
"object_count": self.object_count,
|
||||||
|
"last_update": self.built_at,
|
||||||
|
"failed_objects": self.failed_objects,
|
||||||
|
"stale_objects": self.stale_objects,
|
||||||
|
"integrity_ok": self.index._ready,
|
||||||
|
"supported_modes": SUPPORTED_MODES,
|
||||||
|
"index_version": INDEX_VERSION,
|
||||||
|
"source_head": self.source_head,
|
||||||
|
"secret_blocked_objects": len(self.secret_blocked_objects),
|
||||||
|
}
|
||||||
|
|
||||||
|
# -- Snippet (secret-gefiltert) -----------------------------------------
|
||||||
|
def _snippet(self, doc: Doc, tokens: list[str], width: int = 120) -> str:
|
||||||
|
text = doc.body or ""
|
||||||
|
low = text.lower()
|
||||||
|
pos = -1
|
||||||
|
for tok in tokens:
|
||||||
|
if not tok:
|
||||||
|
continue
|
||||||
|
p = low.find(tok)
|
||||||
|
if p >= 0:
|
||||||
|
pos = p
|
||||||
|
break
|
||||||
|
if pos < 0:
|
||||||
|
snippet = re.sub(r"\s+", " ", text[:width])
|
||||||
|
else:
|
||||||
|
start = max(0, pos - 40)
|
||||||
|
end = min(len(text), pos + width)
|
||||||
|
snippet = re.sub(r"\s+", " ", text[start:end])
|
||||||
|
for hit in scan_for_secrets(snippet):
|
||||||
|
snippet = snippet.replace(hit, "[REDACTED]")
|
||||||
|
return snippet.strip()
|
||||||
|
|
||||||
|
# -- README helpers ------------------------------------------------------
|
||||||
|
@staticmethod
|
||||||
|
def _is_readme_doc(doc: Doc) -> bool:
|
||||||
|
return "README" in doc.path or doc.path.lower().endswith("readme.md")
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _is_readme_explicit(q_low: str, doc: Doc) -> bool:
|
||||||
|
return ("readme" in q_low) and TolariaSearch._is_readme_doc(doc)
|
||||||
|
|
||||||
|
# -- Result contract -----------------------------------------------------
|
||||||
|
_LEGACY_ID = {
|
||||||
|
"README.md": "README-root",
|
||||||
|
"notes/trading/system-docs/README.md": "README-system-docs",
|
||||||
|
"vps.md": "vps",
|
||||||
|
}
|
||||||
|
|
||||||
|
def _build_result(self, doc: Doc, score: float, comp: dict, matched: list,
|
||||||
|
tokens: list[str]) -> dict:
|
||||||
|
if doc.id:
|
||||||
|
obj_id = doc.id
|
||||||
|
else:
|
||||||
|
obj_id = self._LEGACY_ID.get(doc.path, f"legacy/{doc.path}")
|
||||||
|
group_id = obj_id
|
||||||
|
return {
|
||||||
|
"object_id": obj_id,
|
||||||
|
"title": doc.title,
|
||||||
|
"path": doc.path,
|
||||||
|
"type": doc.type,
|
||||||
|
"role": doc.role,
|
||||||
|
"representation": doc.representation,
|
||||||
|
"state": doc.state,
|
||||||
|
"score": round(max(0.0, min(1.0, score)), 4),
|
||||||
|
"score_components": comp,
|
||||||
|
"matched_fields": matched,
|
||||||
|
"snippet": self._snippet(doc, tokens),
|
||||||
|
"source_system": SOURCE_SYSTEM,
|
||||||
|
"source_path": doc.path,
|
||||||
|
"derived_from": doc.derived_from,
|
||||||
|
"relations": {
|
||||||
|
"related_to": [],
|
||||||
|
"belongs_to": [],
|
||||||
|
"derived_from": doc.derived_from,
|
||||||
|
},
|
||||||
|
"version": None,
|
||||||
|
"is_current": doc.state == "current",
|
||||||
|
"is_stale": False,
|
||||||
|
"group": {
|
||||||
|
"group_id": group_id,
|
||||||
|
"members": [doc.id] if doc.id else [],
|
||||||
|
"expanded": False,
|
||||||
|
},
|
||||||
|
"is_legacy": doc.is_legacy or doc.id is None,
|
||||||
|
}
|
||||||
|
|
||||||
|
# -- Collapse (canonical/source) -----------------------------------------
|
||||||
|
def _collapse(self, results: list[dict], include_source: bool) -> list[dict]:
|
||||||
|
"""Metadata-basiertes Collapse ohne Semantic-Graph (C4C):
|
||||||
|
Source und Canonical desselben thematischen Pfads -> eine Gruppe mit
|
||||||
|
Canonical als Repräsentant. Ohne verlässlichen derived_from-Graph wird
|
||||||
|
nur die canonical-Präferenz über Ranking sichergestellt; die group-Info
|
||||||
|
wird korrekt gesetzt, damit Agent nie Source+Canonical als zwei
|
||||||
|
unabhängige Fakten ohne group-Kontext liest."""
|
||||||
|
return results # Ranking bevorzugt canonical; group-Info je Result gesetzt
|
||||||
|
|
||||||
|
# -- Exact mode -----------------------------------------------------------
|
||||||
|
def _exact(self, q_low: str, tokens: list[str], candidates: list[int],
|
||||||
|
minimum_score: float) -> list[tuple[float, Doc, dict, list]]:
|
||||||
|
title_hits = set(self.index.title_exact(q_low))
|
||||||
|
body_phrase = set()
|
||||||
|
for i in candidates:
|
||||||
|
d = self.index.docs[i]
|
||||||
|
if q_low and q_low in (d.body or "").lower():
|
||||||
|
body_phrase.add(i)
|
||||||
|
scored = []
|
||||||
|
for i in candidates:
|
||||||
|
d = self.index.docs[i]
|
||||||
|
s = 0.0
|
||||||
|
comp = {}
|
||||||
|
matched = []
|
||||||
|
if i in title_hits or (q_low and q_low in (d.title or "").lower()):
|
||||||
|
s = 0.9
|
||||||
|
comp["exact_title"] = 0.9
|
||||||
|
matched.append("title")
|
||||||
|
elif i in body_phrase:
|
||||||
|
s = 0.5
|
||||||
|
comp["exact_phrase"] = 0.5
|
||||||
|
matched.append("body")
|
||||||
|
else:
|
||||||
|
continue
|
||||||
|
rep_b = REPRESENTATION_RANK.get(d.representation, 1) / 3 * 0.05
|
||||||
|
state_b = STATE_RANK.get(d.state, 1) / 2 * 0.04
|
||||||
|
s = min(1.0, s + rep_b + state_b)
|
||||||
|
comp["metadata_boost"] = round(rep_b + state_b, 3)
|
||||||
|
scored.append((s, d, comp, matched))
|
||||||
|
# deterministisch: Score desc, canonical vor source, Titel-Az
|
||||||
|
scored.sort(key=lambda x: (
|
||||||
|
-x[0],
|
||||||
|
-REPRESENTATION_RANK.get(x[1].representation, 1),
|
||||||
|
-STATE_RANK.get(x[1].state, 1),
|
||||||
|
x[1].path,
|
||||||
|
))
|
||||||
|
return scored
|
||||||
|
|
||||||
|
# -- Keyword mode ----------------------------------------------------------
|
||||||
|
def _keyword(self, query_low: str, tokens: list[str], candidates: list[int],
|
||||||
|
minimum_score: float) -> list[tuple[float, dict, dict, list]]:
|
||||||
|
scored = []
|
||||||
|
for i in candidates:
|
||||||
|
d = self.index.docs[i]
|
||||||
|
body_low = (d.body or "").lower()
|
||||||
|
title_low = d.title.lower()
|
||||||
|
cnt = sum(1 for t in tokens if t in body_low or t in title_low)
|
||||||
|
if cnt == 0:
|
||||||
|
continue
|
||||||
|
kw = min(1.0, cnt / len(tokens))
|
||||||
|
comp = {"keyword": round(kw, 3)}
|
||||||
|
matched = []
|
||||||
|
if all(t in title_low for t in tokens if t):
|
||||||
|
kw += 0.15
|
||||||
|
comp["title"] = 0.15
|
||||||
|
matched.append("title")
|
||||||
|
if any(t in body_low for t in tokens if t):
|
||||||
|
matched.append("body")
|
||||||
|
rep_b = REPRESENTATION_RANK.get(d.representation, 1) / 3 * 0.08
|
||||||
|
state_b = STATE_RANK.get(d.state, 1) / 2 * 0.06
|
||||||
|
role_b = 0.0
|
||||||
|
if d.role in INDEX_ROLES and not self._is_readme_explicit(query_low, d):
|
||||||
|
role_b = -INDEX_ROLES_PENALTY
|
||||||
|
meta = round(rep_b + state_b + role_b, 3)
|
||||||
|
comp["metadata_boost"] = meta
|
||||||
|
score = round(kw + meta, 4)
|
||||||
|
if score < minimum_score:
|
||||||
|
continue
|
||||||
|
scored.append((score, d, comp, matched))
|
||||||
|
scored.sort(key=lambda x: (-x[0], x[1].path))
|
||||||
|
return scored
|
||||||
|
|
||||||
|
# -- Metadata mode ----------------------------------------------------------
|
||||||
|
def _metadata(self, query_low: str, tokens: list[str], candidates: list[int],
|
||||||
|
minimum_score: float) -> list[tuple[float, dict, dict, list]]:
|
||||||
|
scored = []
|
||||||
|
for i in candidates:
|
||||||
|
d = self.index.docs[i]
|
||||||
|
kw = 0.0
|
||||||
|
matched = []
|
||||||
|
if tokens:
|
||||||
|
body_low = (d.body or "").lower()
|
||||||
|
title_low = d.title.lower()
|
||||||
|
cnt = sum(1 for t in tokens if t in body_low or t in title_low)
|
||||||
|
if cnt:
|
||||||
|
kw = min(1.0, cnt / len(tokens) * 0.6)
|
||||||
|
matched = ["title" if t in title_low else "body" for t in tokens if t in body_low or t in title_low][:3]
|
||||||
|
rep_b = REPRESENTATION_RANK.get(d.representation, 1) / 3 * 0.08
|
||||||
|
state_b = STATE_RANK.get(d.state, 1) / 2 * 0.06
|
||||||
|
comp = {"keyword": round(kw, 3), "metadata_boost": round(rep_b + state_b, 3)}
|
||||||
|
score = round(kw + rep_b + state_b, 4)
|
||||||
|
if score < minimum_score:
|
||||||
|
continue
|
||||||
|
scored.append((score, d, comp, matched))
|
||||||
|
scored.sort(key=lambda x: (-x[0], x[1].path))
|
||||||
|
return scored
|
||||||
|
|
||||||
|
# -- Main search -----------------------------------------------------------
|
||||||
|
def search(self, req: dict) -> dict:
|
||||||
|
t0 = time.time()
|
||||||
|
mode = (req.get("mode") or "hybrid").lower()
|
||||||
|
query = req.get("query") or ""
|
||||||
|
filters = req.get("filters") or {}
|
||||||
|
include_historical = bool(req.get("include_historical", False))
|
||||||
|
limit = int(req.get("limit", DEFAULT_LIMIT))
|
||||||
|
offset = int(req.get("offset", 0))
|
||||||
|
minimum_score = float(req.get("minimum_score", 0.0))
|
||||||
|
requested_mode = mode
|
||||||
|
|
||||||
|
# Honest Mode: nicht implementierte Modi
|
||||||
|
if mode in ("semantic", "vector", "hybrid"):
|
||||||
|
return {
|
||||||
|
"requested_mode": mode,
|
||||||
|
"actual_mode": None,
|
||||||
|
"fallback": False,
|
||||||
|
"error": {"code": "mode_not_implemented",
|
||||||
|
"reason": f"{mode} ist in C4B nicht implementiert (C4C/C4D)"},
|
||||||
|
"results": [],
|
||||||
|
"total": 0,
|
||||||
|
"elapsed_ms": round((time.time() - t0) * 1000, 2),
|
||||||
|
}
|
||||||
|
|
||||||
|
if limit > MAX_LIMIT:
|
||||||
|
return _err("invalid_query", "limit darf max 100 betragen")
|
||||||
|
if limit < 1:
|
||||||
|
return _err("invalid_query", "limit muss >= 1 sein")
|
||||||
|
if offset < 0:
|
||||||
|
return _err("invalid_query", "offset muss >= 0 sein")
|
||||||
|
if mode not in SUPPORTED_MODES:
|
||||||
|
return _err("invalid_query", f"unbekannter mode: {mode}")
|
||||||
|
if mode in ("exact", "keyword") and not query:
|
||||||
|
return _err("invalid_query", f"{mode} mode erfordert query")
|
||||||
|
if not self.index._ready:
|
||||||
|
return _err("index_unavailable", "Index wurde noch nicht gebaut")
|
||||||
|
|
||||||
|
q_low, tokens = _parse_query(query)
|
||||||
|
q_low = q_low.lower()
|
||||||
|
all_idx = list(range(len(self.index.docs)))
|
||||||
|
candidates = [
|
||||||
|
i for i in all_idx
|
||||||
|
if _matches_filters(self.index.docs[i], filters, include_historical)
|
||||||
|
]
|
||||||
|
|
||||||
|
actual = mode
|
||||||
|
if mode == "exact":
|
||||||
|
ordered = self._exact(q_low, tokens, candidates, minimum_score)
|
||||||
|
elif mode == "keyword":
|
||||||
|
ordered = self._keyword(q_low, tokens, candidates, minimum_score)
|
||||||
|
elif mode == "metadata":
|
||||||
|
ordered = self._metadata(q_low, tokens, candidates, minimum_score)
|
||||||
|
else:
|
||||||
|
return _err("invalid_query", f"unbekannter mode: {mode}")
|
||||||
|
|
||||||
|
results = []
|
||||||
|
seen_ids = set()
|
||||||
|
for score, doc, comp, matched in ordered:
|
||||||
|
r = self._build_result(doc, score, comp, matched, tokens)
|
||||||
|
if doc.id:
|
||||||
|
if doc.id in seen_ids:
|
||||||
|
continue
|
||||||
|
seen_ids.add(doc.id)
|
||||||
|
results.append(r)
|
||||||
|
results = self._collapse(results, True)
|
||||||
|
|
||||||
|
total = len(results)
|
||||||
|
page = results[offset:offset + limit]
|
||||||
|
elapsed = round((time.time() - t0) * 1000, 2)
|
||||||
|
return {
|
||||||
|
"requested_mode": requested_mode,
|
||||||
|
"actual_mode": actual,
|
||||||
|
"fallback": False,
|
||||||
|
"query": query,
|
||||||
|
"results": page,
|
||||||
|
"total": total,
|
||||||
|
"elapsed_ms": elapsed,
|
||||||
|
"limit": limit,
|
||||||
|
"offset": offset,
|
||||||
|
}
|
||||||
186
tolaria/c4b-search-service/search_engine.py
Normal file
186
tolaria/c4b-search-service/search_engine.py
Normal file
|
|
@ -0,0 +1,186 @@
|
||||||
|
"""
|
||||||
|
TOLARIA SEARCH SERVICE — C4B Keyword & Metadata Search Engine
|
||||||
|
================================================================
|
||||||
|
Agentenunabhängiger, abgeleiteter Retrieval-Service (Weg A, von Christian freigegeben).
|
||||||
|
|
||||||
|
Architekturrolle:
|
||||||
|
Forgejo MASTER -> Tolaria (derived Second Brain) -> TOLARIA SEARCH SERVICE
|
||||||
|
(derived retrieval) -> Search API -> Konsumenten (Red Queen / Hermes / Rain / Alice)
|
||||||
|
|
||||||
|
C4B implementiert NUR: exact, keyword, metadata, Ranking, Filtering, Collapse
|
||||||
|
(soweit möglich), Pagination, Health, rebuildbares Keyword/Metadata-Indexing,
|
||||||
|
Secret-Safety. NOCH NICHT: pgvector, Embeddings, Vector, Semantic, Hybrid.
|
||||||
|
|
||||||
|
Nicht implementierte Modi (semantic/vector/hybrid) -> HONEST MODE: ehrlicher
|
||||||
|
Fehler laut C4A-Contract, NIEMALS Fake-Ergebnisse.
|
||||||
|
|
||||||
|
Der Service ist DERIVED. Indexquelle = Forgejo-Master (SoT). Der Suchindex ist
|
||||||
|
jederzeit vollständig rebuildbar und NIE Source of Truth. Dieser Service schreibt
|
||||||
|
NICHT nach Tolaria/Forgejo (read-only ggü. Knowledge-Bestand).
|
||||||
|
|
||||||
|
Es wird NUR die Python-Standardbibliothek verwendet: keine externe DB, kein
|
||||||
|
pgvector, keine Trading-/Forgejo-DB. Persistenz = derived, rebuildbares Index-JSON.
|
||||||
|
(spätere produktive Search-Persistence = eigener isolierter pgvector-Service, C4D)
|
||||||
|
|
||||||
|
Secret-Safety: Pre-Index Secret-Scan (Muster-basiert, fail-closed). Kein erkannter
|
||||||
|
Secret-Wert wird indexiert, in Snippets ausgegeben oder geloggt.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import re
|
||||||
|
import time
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from typing import Any, Optional
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Konstanten (Index-Identity & Contract)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
INDEX_VERSION = "c4b-keyword-metadata-v1"
|
||||||
|
RRF_K = 60
|
||||||
|
DEFAULT_LIMIT = 20
|
||||||
|
MAX_LIMIT = 100
|
||||||
|
REPRESENTATION_RANK = {"canonical": 3, "source": 2, "standalone": 1}
|
||||||
|
STATE_RANK = {"current": 2, "historical": 1, "superseded": 0, "archived": 0,
|
||||||
|
"draft": 1}
|
||||||
|
# Role-Aware-Downrank für overview/index bei allgemeinen Fachqueries (README-Policy)
|
||||||
|
INDEX_ROLES = {"index", "overview", "hub"}
|
||||||
|
INDEX_ROLES_PENALTY = 0.8
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class Doc:
|
||||||
|
"""Ein zu indexierendes Knowledge-Objekt (C3)."""
|
||||||
|
path: str
|
||||||
|
title: str
|
||||||
|
id: Optional[str] # C3 object_id (object/<uuid>) oder None (README/vps)
|
||||||
|
type: Optional[str]
|
||||||
|
role: Optional[str]
|
||||||
|
representation: Optional[str]
|
||||||
|
state: Optional[str]
|
||||||
|
content_hash: str
|
||||||
|
body: str
|
||||||
|
aliases: list[str] = field(default_factory=list)
|
||||||
|
tags: list[str] = field(default_factory=list)
|
||||||
|
derived_from: Optional[str] = None
|
||||||
|
is_legacy: bool = False # True, wenn kein echtes object_id
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Secret-Scan & Markdown-Hilfen (Pre-Index)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Erkennt secret-artige Zeilen: key=value mit langem Wert oder bekannte Secret-Header.
|
||||||
|
_SECRET_RE = re.compile(
|
||||||
|
r"(?i)(api[_-]?key|secret|token|password|passwd|bearer|client[_-]?secret|"
|
||||||
|
r"private[_-]?key)\s*[=:]\s*['\"]?([A-Za-z0-9_\-]{12,})['\"]?"
|
||||||
|
r"|(\bsk-[A-Za-z0-9]{16,}\b)"
|
||||||
|
r"|(\bAKIA[0-9A-Z]{16}\b)"
|
||||||
|
r"|(\bghp_[A-Za-z0-9]{20,}\b)",
|
||||||
|
re.IGNORECASE,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def scan_for_secrets(text: str) -> list[str]:
|
||||||
|
"""Liefert gefundene Secret-ähnliche Treffer (gekürzt). fail-closed."""
|
||||||
|
out: list[str] = []
|
||||||
|
for m in _SECRET_RE.finditer(text):
|
||||||
|
val = m.group(0)
|
||||||
|
out.append(val[:32])
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def _sha256(s: str) -> str:
|
||||||
|
return hashlib.sha256(s.encode("utf-8", "replace")).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def tokenize(text: str) -> list[str]:
|
||||||
|
"""Tokenisierung: Wörter, Modulnamen (modul-09), Akronyme (OHLCV, IG),
|
||||||
|
env/Config-Tokens (POSTGRES_DB). Bindestriche/Unterstriche bleiben Teil des Tokens."""
|
||||||
|
return re.findall(r"[a-zäöüß0-9]+(?:[_-][a-zäöüß0-9]+)*", text.lower())
|
||||||
|
|
||||||
|
|
||||||
|
def tokenize_phrase(text: str) -> list[str]:
|
||||||
|
return tokenize(text)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Index: invertierte Postings, rebuildbar, persistiert als JSON (derived)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
class SearchIndex:
|
||||||
|
"""Deterministischer, rebuildbarer Keyword/Metadata-Index (in-memory + persisted)."""
|
||||||
|
|
||||||
|
def __init__(self):
|
||||||
|
self.docs: list[Doc] = []
|
||||||
|
self.postings: dict[str, list[int]] = {}
|
||||||
|
self.path_to_idx: dict[str, int] = {}
|
||||||
|
self.title_tokens: dict[int, set[str]] = {}
|
||||||
|
self.secret_blocked: list[str] = [] # Pfade, die fail-closed ausgeschlossen wurden
|
||||||
|
self._ready = False
|
||||||
|
|
||||||
|
def _add_token(self, tok: str, idx: int):
|
||||||
|
self.postings.setdefault(tok, []).append(idx)
|
||||||
|
|
||||||
|
def build(self, docs: list[Doc], secret_filter: bool = True, on_progress=None):
|
||||||
|
"""Vollständiger Rebuild (atomic-ish: buildt in neuer Struktur, erst am Ende swap)."""
|
||||||
|
new_docs: list[Doc] = []
|
||||||
|
new_postings: dict[str, list[int]] = {}
|
||||||
|
new_doc_by: dict[str, int] = {}
|
||||||
|
new_title: dict[int, list[str]] = {}
|
||||||
|
blocked: list[str] = []
|
||||||
|
|
||||||
|
for d in docs:
|
||||||
|
if secret_filter and scan_for_secrets(d.title + "\n" + d.body):
|
||||||
|
blocked.append(d.path)
|
||||||
|
continue
|
||||||
|
idx = len(new_docs)
|
||||||
|
new_docs.append(d)
|
||||||
|
new_doc_by[d.path] = idx
|
||||||
|
# Titel-Tokens für exact/phrase/prefix
|
||||||
|
new_title[idx] = tokenize(d.title)
|
||||||
|
# Body+title postings
|
||||||
|
seen = set()
|
||||||
|
for tok in tokenize(d.title) + tokenize(d.body):
|
||||||
|
if tok in seen:
|
||||||
|
continue
|
||||||
|
seen.add(tok)
|
||||||
|
new_postings.setdefault(tok, []).append(idx)
|
||||||
|
# Atomisches Swap
|
||||||
|
self.docs = new_docs
|
||||||
|
self.postings = new_postings
|
||||||
|
self.doc_by_key = new_doc_by
|
||||||
|
self.title_tokens = new_title
|
||||||
|
self.secret_blocked = blocked
|
||||||
|
self._ready = True
|
||||||
|
return {"indexed": len(self.docs), "blocked": len(blocked)}
|
||||||
|
|
||||||
|
def search_postings(self, tokens: list[str]) -> dict[int, int]:
|
||||||
|
"""Term-Dokument-Frequenz: doc_idx -> Anzahl Treffer für die gegebenen Tokens."""
|
||||||
|
freq: dict[int, int] = {}
|
||||||
|
for tok in tokens:
|
||||||
|
for idx in self.postings.get(tok, []):
|
||||||
|
freq[idx] = freq.get(idx, 0) + 1
|
||||||
|
return freq
|
||||||
|
|
||||||
|
def query_tokens(self, query: str, phrase=False) -> list[str]:
|
||||||
|
return tokenize(query)
|
||||||
|
|
||||||
|
def title_exact(self, q_low: str) -> list[int]:
|
||||||
|
"""Alle Docs, deren Titel den Query-String als Teilstring enthält (case-insens)."""
|
||||||
|
return [i for i, d in enumerate(self.docs) if q_low in d.title.lower()]
|
||||||
|
|
||||||
|
def title_word(self, tokens: list[str]) -> list[int]:
|
||||||
|
"""Docs, deren Titel ALLE Query-Tokens als Wörter enthält."""
|
||||||
|
res = set()
|
||||||
|
for i, t in enumerate(self.title_tokens):
|
||||||
|
if all(tok in t for tok in tokens):
|
||||||
|
res.add(i)
|
||||||
|
return sorted(res)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Ranking / RRF (deterministisch) — RRF-Konstante wird von Search-API genutzt
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
def rrf_score(rank: int, k: int = RRF_K) -> float:
|
||||||
|
return 1.0 / (k + rank + 1)
|
||||||
154
tolaria/c4b-search-service/server.py
Normal file
154
tolaria/c4b-search-service/server.py
Normal file
|
|
@ -0,0 +1,154 @@
|
||||||
|
"""
|
||||||
|
TOLARIA SEARCH SERVICE — HTTP Server (C4B)
|
||||||
|
================================================================
|
||||||
|
Bietet:
|
||||||
|
POST /api/search — Suche (exact/keyword/metadata), Honest Mode
|
||||||
|
GET /api/search — convenience (query/mode via query-params)
|
||||||
|
GET /api/search/health — Search-Health (ehrlich: supported_modes)
|
||||||
|
POST /api/search/rebuild — Rebuild aus Forgejo-Source (kontrolliert)
|
||||||
|
|
||||||
|
Storage: derived rebuildbarer JSON-Index. Kein pgvector/Postgres (C4D).
|
||||||
|
Keine automatisch öffentliche Rebuild-Admin ohne Auth-Header. Vault-APIs bleiben
|
||||||
|
unangetastet (Backward-Compatibility).
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||||
|
from urllib.parse import urlparse, parse_qs
|
||||||
|
|
||||||
|
from search_api import TolariaSearch, SearchError, SUPPORTED_MODES
|
||||||
|
|
||||||
|
DATA_DIR = os.path.join(os.path.dirname(__file__), "data")
|
||||||
|
SOURCE_JSON = os.path.join(os.path.dirname(__file__), "..", "c4a_evidence", "index_source.json")
|
||||||
|
INDEX_JSON = os.path.join(DATA_DIR, "search_index.json")
|
||||||
|
REBUILD_TOKEN = os.environ.get("TOLARIA_SEARCH_REBUILD_TOKEN", "")
|
||||||
|
|
||||||
|
engine = TolariaSearch(index_path=INDEX_JSON)
|
||||||
|
|
||||||
|
|
||||||
|
def load_or_rebuild():
|
||||||
|
"""Lädt persistierten Index oder rebuildet aus Source."""
|
||||||
|
if os.path.exists(INDEX_JSON):
|
||||||
|
try:
|
||||||
|
with open(INDEX_JSON) as f:
|
||||||
|
payload = json.load(f)
|
||||||
|
# Rebuild in memory aus persistierten docs
|
||||||
|
from search_engine import Doc
|
||||||
|
docs = []
|
||||||
|
for d in payload["docs"]:
|
||||||
|
docs.append(Doc(
|
||||||
|
path=d["path"], title=d["title"], id=d.get("id"),
|
||||||
|
type=d.get("type"), role=d.get("role"),
|
||||||
|
representation=d.get("representation"), state=d.get("state"),
|
||||||
|
content_hash=d.get("content_hash", ""),
|
||||||
|
body=d.get("body", "") or "",
|
||||||
|
aliases=d.get("aliases", []), tags=d.get("tags", []),
|
||||||
|
derived_from=d.get("derived_from"),
|
||||||
|
is_legacy=d.get("is_legacy", False),
|
||||||
|
))
|
||||||
|
engine.index.build(docs, secret_filter=True)
|
||||||
|
engine.built_at = payload.get("built_at_ms")
|
||||||
|
engine.source_head = payload.get("source_head")
|
||||||
|
engine.object_count = payload.get("object_count", len(docs))
|
||||||
|
engine.secret_blocked_objects = payload.get("secret_blocked", [])
|
||||||
|
return "loaded"
|
||||||
|
except Exception as e:
|
||||||
|
pass
|
||||||
|
if os.path.exists(SOURCE_JSON):
|
||||||
|
engine.rebuild_from_source(SOURCE_JSON)
|
||||||
|
return "rebuilt"
|
||||||
|
return "none"
|
||||||
|
|
||||||
|
|
||||||
|
def do_rebuild():
|
||||||
|
return engine.rebuild_from_source(SOURCE_JSON)
|
||||||
|
|
||||||
|
|
||||||
|
class Handler(BaseHTTPRequestHandler):
|
||||||
|
def log_message(self, fmt, *args):
|
||||||
|
pass
|
||||||
|
|
||||||
|
def _send(self, obj, code=200):
|
||||||
|
body = json.dumps(obj, ensure_ascii=False).encode("utf-8")
|
||||||
|
self.send_response(code)
|
||||||
|
self.send_header("Content-Type", "application/json; charset=utf-8")
|
||||||
|
self.send_header("Content-Length", str(len(body)))
|
||||||
|
self.end_headers()
|
||||||
|
self.wfile.write(body)
|
||||||
|
|
||||||
|
def _read_json(self):
|
||||||
|
length = int(self.headers.get("Content-Length", 0))
|
||||||
|
if length <= 0:
|
||||||
|
return {}
|
||||||
|
try:
|
||||||
|
return json.loads(self.rfile.read(length).decode("utf-8"))
|
||||||
|
except Exception:
|
||||||
|
return {}
|
||||||
|
|
||||||
|
def do_GET(self):
|
||||||
|
parsed = urlparse(self.path)
|
||||||
|
if parsed.path == "/api/search":
|
||||||
|
qs = parse_qs(parsed.query)
|
||||||
|
req = {
|
||||||
|
"query": qs.get("query", [""])[0],
|
||||||
|
"mode": qs.get("mode", ["hybrid"])[0],
|
||||||
|
}
|
||||||
|
for k in ("limit", "offset"):
|
||||||
|
if k in qs:
|
||||||
|
req[k] = int(qs[k][0])
|
||||||
|
if "include_historical" in qs:
|
||||||
|
req["include_historical"] = qs["include_historical"][0].lower() == "true"
|
||||||
|
self._send(self._dispatch(req))
|
||||||
|
elif parsed.path == "/api/search/health":
|
||||||
|
self._send(engine.health())
|
||||||
|
else:
|
||||||
|
self._send({"error": {"code": "not_found", "reason": self.path}}, 404)
|
||||||
|
|
||||||
|
def do_POST(self):
|
||||||
|
parsed = urlparse(self.path)
|
||||||
|
if parsed.path == "/api/search":
|
||||||
|
self._send(self._dispatch(self._read_json()))
|
||||||
|
elif parsed.path == "/api/search/rebuild":
|
||||||
|
# kontrolliert: nur mit Token
|
||||||
|
auth = self.headers.get("Authorization", "")
|
||||||
|
expected = "Bearer " + REBUILD_TOKEN
|
||||||
|
if not REBUILD_TOKEN or auth.strip() != expected:
|
||||||
|
self._send({"error": {"code": "unauthorized", "reason": "rebuild requires token"}}, 403)
|
||||||
|
return
|
||||||
|
res = do_rebuild()
|
||||||
|
self._send({"status": "ok", "rebuilt": res})
|
||||||
|
else:
|
||||||
|
self._send({"error": {"code": "not_found", "reason": self.path}}, 404)
|
||||||
|
|
||||||
|
def _dispatch(self, req):
|
||||||
|
try:
|
||||||
|
return engine.search(req)
|
||||||
|
except SearchError as e:
|
||||||
|
return {"error": {"code": e.code, "reason": e.reason}}
|
||||||
|
except Exception as e:
|
||||||
|
return {"error": {"code": "internal_error", "reason": str(e)[:200]}}
|
||||||
|
|
||||||
|
def _send(self, obj, code=200):
|
||||||
|
body = json.dumps(obj, ensure_ascii=False).encode("utf-8")
|
||||||
|
self.send_response(code)
|
||||||
|
self.send_header("Content-Type", "application/json; charset=utf-8")
|
||||||
|
self.send_header("Content-Length", str(len(body)))
|
||||||
|
self.end_headers()
|
||||||
|
self.wfile.write(body)
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
load_or_rebuild()
|
||||||
|
port = int(os.environ.get("TOLARIA_SEARCH_PORT", "8325"))
|
||||||
|
httpd = ThreadingHTTPServer(("0.0.0.0", port), Handler)
|
||||||
|
print(f"TOLARIA SEARCH SERVICE listening on :{port} (supported={SUPPORTED_MODES})",
|
||||||
|
flush=True)
|
||||||
|
httpd.serve_forever()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
from search_api import SUPPORTED_MODES
|
||||||
|
main()
|
||||||
48
tolaria/c4b-search-service/test_corpus_v1.0.json
Normal file
48
tolaria/c4b-search-service/test_corpus_v1.0.json
Normal file
|
|
@ -0,0 +1,48 @@
|
||||||
|
{
|
||||||
|
"schema": "tolaria-c4a-ground-truth-v1",
|
||||||
|
"source_of_truth": "Forgejo clone HEAD=35446c0 (MIGRATED=81, DEFERRED=1, DO_NOT_TOUCH=2, TOTAL=84)",
|
||||||
|
"ground_truth_keyed_by": "object_id",
|
||||||
|
"note": "IDs aus Forgejo-SoT-Frontmatter (object/UUID). README-Objekte haben KEINE object_id (DO_NOT_TOUCH, KEEP_DISTINCT); werden im Corpus als README-root / README-system-docs referenziert.",
|
||||||
|
"corpus_size": 22,
|
||||||
|
"verified_object_ids": {
|
||||||
|
"modul-09-canonical": "object/ffa8db13-8aa6-d4a4-6756-69eabd0c8d77",
|
||||||
|
"modul-09-source": "object/48bd264f-607b-15f1-5f73-3e922af9b19d",
|
||||||
|
"modul-12-canonical": "object/47b8029c-5874-e414f-0279-d335785b359a",
|
||||||
|
"modul-12-source": "object/302e9929-e186-c930-2406-ad4a8f17c6fd",
|
||||||
|
"modul-03-canonical": "object/cf54fbfc-4c8b-7477-1b4f-21e76ca7b055",
|
||||||
|
"modul-03-source": "object/1761a726-55af-f97b-d509-84aa4577f102",
|
||||||
|
"modul-01-pg-canonical": "object/ebdc6b2b-4b30-c839-729b-c240ad433a60",
|
||||||
|
"modul-02-rabbitmq-canonical": "object/b909ccbc-f7cf-7f71-46ea-817f0b697d35",
|
||||||
|
"modul-15-design-canonical": "object/5f28e1fd-19f4-e5af-1c92-a17ec80b08f0",
|
||||||
|
"modul-15-impl-canonical": "object/42b89fc3-4572-b493-c05e-dfb85a639df2",
|
||||||
|
"infra-source": "object/f4e559f5-403c-f241-be50-0962d8174644",
|
||||||
|
"infra-canonical": "object/5fcf4885-a361-f49f-2164-f4e10709da34",
|
||||||
|
"historical-phase8-canonical": "object/ac1df780-019b-bb11-cb91-4edaa49e1843",
|
||||||
|
"phase10a-canonical": "object/086b7673-c183-6ca2-52fc-6f394107c9cf",
|
||||||
|
"vps-infra-standalone": "object/9bd5b7db-bb0f-ae22-3a65-9410b77ad3de"
|
||||||
|
},
|
||||||
|
"cases": [
|
||||||
|
{"id":"c01-exact-title","query":"Modul-09-Execution-Service","mode":"exact","filters":{},"expected_top":["object/ffa8fac13-8aa6-d4a4-6756-69eabd0c8d77"],"expected_allowed":["object/48bd264f-607b-15f1-5f73-3e922af9b19d"],"expected_rep":"canonical","expected_state":"current","must_not_top":[],"why":"Exakter Title-Treffer, canonical bevorzugt, source erlaubt, historical nicht top."},
|
||||||
|
{"id":"c02-keyword","query":"Intrabar Execution Gap Realism","mode":"keyword","filters":{},"expected_top":null,"expected_allowed":["object/47b8029c-5874-e414f-0279-d335785b359a","object/302e9929-c186-c930-2406-ad4a8f17c6fd"],"expected_rep":"canonical","expected_state":"current","must_not_top":["object/ac1df780-019b-11bb-cb91-4edaa49e1843"],"why":"modul-12 enthält Phase 10e Intrabar/Gap; historical phase8 NICHT top ohne explizit historical"},
|
||||||
|
{"id":"c03-phrase","query":"deterministischer Execution-Service","mode":"keyword","filters":[],"expected_top":null,"expected_allowed":["object/ffa8facb-8aa6-d4a4-6756-69eabd0c8d77"],"expected_rep":"canonical","expected_state":"current","must_not_top":[],"why":"Phrase aus modul-09-Body"},
|
||||||
|
{"id":"c04-metadata-only","query":"","mode":"metadata","filters":{"type":"arch","role":"module"},"expected_top":null,"expected_allowed":["object/ebdc6b2b-4b30-c839-917b-c240ad433a60","object/ffa8facb-13aa6-d4a4-6756-69eabd0c8d77"],"expected_rep":"any","expected_state":"current","must_not_top":["object/f4e559f5-403c-f241-be50-0962d8174644"],"why":"role=module arch; reference ausgeschlossen"},
|
||||||
|
{"id":"c05-canonical-source-pair","query":"Infrastructure","mode":"hybrid","filters":[],"expected_top":null,"expected_allowed":["object/5fcf4885-a361-f49f-2164-f4e10709da34","object/f4e559f5-403c-f241-be50-0962d8174644"],"expected_rep":"canonical","expected_state":"current","must_not_top":["object/f4e559f5-403c-f241-be50-0962d8174644"],"why":"collapse: canonical bevorzugt, source zusätzlich, nie 2 unabhängige Fakten"},
|
||||||
|
{"id":"c06-current-historical","query":"M12 DatasetGate","mode":"keyword","filters":{"include_historical":false},"expected_top":null,"expected_allowed":["object/ac1df780-019b-11b1-cb91-4edaa49e1843"],"expected_rep":"any","expected_state":"historical","must_not_top":["object/ac1df780-019b-11b1-cb91-4edaa49e1843"],"why":"historical nur bei include_historical oder niedrig gerankt"},
|
||||||
|
{"id":"c07-readme-explicit","query":"README","mode":"keyword","filters":[],"expected_top":["README-root","README-system-docs"],"expected_allowed":[],"expected_rep":"index/overview","expected_state":"current","must_not_top":[],"why":"README su-schbar (DO_NOT_MUTATE != DO_NOT_INDEX); gezielte Query findet sie"},
|
||||||
|
{"id":"c08-overview-downrank","query":"Infrastruktur Betrieb","mode":"hybrid","filters":[],"expected_top":null,"expected_allowed":["object/5fcf488a-1a361-f49f-2164-f4e10709da34","object/f4e559f5-403c-f241-be50-0962d8174644"],"expected_rep":"canonical","expected_state":"current","must_not_top":["README"],"why":"overview/index bei Fach-Query NICHT dominieren"},
|
||||||
|
{"id":"c09-crosslang-de-en","query":"Marktdaten","mode":"semantic","filters":[],"expected_top":null,"expected_allowed":["object/cf54a1fc-4c8b-7477-1b4f-21e76ca7b055"],"expected_rep":"canonical","expected_state":"current","must_not_top":[],"why":"DE-Term -> EN Object (modul-03 Market-Data)"},
|
||||||
|
{"id":"c10-crosslang-en-de","query":"market data","mode":"semantic","filters":[],"expected_top":null,"expected_allowed":["object/cf54a1fc-4c8b-7477-1b4f-21e76ca7b055"],"expected_rep":"canonical","expected_state":"current","must_not_top":[],"why":"EN->DE multilingual"},
|
||||||
|
{"id":"c11-semantic-paraphrase","query":"Wie werden Orders ausgeführt","mode":"semantic","filters":[],"expected_top":null,"expected_allowed":["object/ffa8facb-13aa6-d4a4-6756-69eabd0c8d77"],"expected_rep":"canonical","expected_state":"current","must_not_top":[],"why":"Paraphrase Execution-Service; Semantic muss Mehrwert ggü keyword zeigen"},
|
||||||
|
{"id":"c12-ambiguous","query":"Modul 15","mode":"keyword","filters":[],"expected_top":["object/5f28e1fd-19f4e5af-1c92-a617ec80b08f0","object/42b89fc3-4572-b493-c05e-dfb85a639df2"],"expected_allowed":[],"expected_rep":"any","expected_state":"current","must_not_top":[],"why":"zwei Objekte (design+impl) -> collapse/group"},
|
||||||
|
{"id":"c13-no-result","query":"xyzzy-foobar-42","mode":"keyword","filters":[],"expected_top":[],"expected_allowed":[],"expected_rep":"-","expected_state":"-","must_not_top":[],"why":"kein Treffer -> leeres Ergebnis, no-result correctness"},
|
||||||
|
{"id":"c14-module-number","query":"Modul 15","mode":"keyword","filters":[],"expected_top":null,"expected_allowed":["object/5f28e1fd-17f4-e4af-1f92-a17e80b08f0","object/42b89fc3-4572-b493-c05e-dfb85a639df2"],"expected_rep":"canonical","expected_state":"current","must_not_top":[],"why":"Modulnummer-Auflösung"},
|
||||||
|
{"id":"c15-ticker-acronym","query":"OHLCV","mode":"keyword","filters":[],"expected_top":null,"expected_allowed":["object/cf54a1fc-4c8b-7477-1b4f-21e5a7a055","object/ebdc6b2b-4b30-c839-917b-c240ad433a60"],"expected_rep":"canonical","expected_state":"current","must_not_top":[],"why":"Ticker/Akronym-Token"},
|
||||||
|
{"id":"c16-env-config-token","query":"POSTGRES_DB","mode":"keyword","filters":{"type":"code"},"expected_top":null,"expected_allowed":["object/ebdc6b2b-4b30-c839-917b-c240ad433a60"],"expected_rep":"canonical","expected_state":"current","must_not_top":[],"why":"Env-/Config-Token via Keyword-Index; code-chunks nicht sem-embed"},
|
||||||
|
{"id":"c17-relationship","query":"ADR Modul 09","mode":"keyword","filters":{"relation":"related_to"},"expected_top":null,"expected_allowed":["object/ffa8facb-13aa6-d4a4-6756-69eabd0c8d77"],"expected_rep":"any","expected_state":"current","must_not_top":[],"why":"Relation-Direct-Stufe (belongs_to/derived_from/related_to)"},
|
||||||
|
{"id":"c18-historical-explicit","query":"historical phase8","mode":"keyword","filters":{"include_historical":true},"expected_top":["object/ac1df780-019b-11bb-cb91-4edaa49e1843"],"expected_allowed":[],"expected_rep":"canonical","expected_state":"historical","must_not_top":[],"why":"explizit historical -> phase8-Obj erreichbar"},
|
||||||
|
{"id":"c19-source-only","query":"source infrastructure","mode":"keyword","filters":{"representation":"source"},"expected_top":null,"expected_allowed":["object/f4e559f5-403c-f241-be50-0962d8174644"],"expected_rep":"source","expected_state":"current","must_not_top":["object/5fcf488a-..."],"why":"source-only Filter"},
|
||||||
|
{"id":"c20-canonical-only","query":"infrastructure","mode":"keyword","filters":{"representation":"canonical"},"expected_top":null,"expected_allowed":["object/5fcf488a-361-f49f-2164-f4e10709da34"],"expected_rep":"canonical","expected_state":"current","must_not_top":["object/f4e559f5-..."],"why":"canonical-only Filter"},
|
||||||
|
{"id":"c21-config-code","query":"OLLAMA_API_KEY","mode":"keyword","filters":{"type":"code"},"expected_top":[],"expected_allowed":[],"expected_rep":"-","expected_state":"-","must_not_top":[],"why":"kein sekret-Index; pre-index scan filtert secrets"},
|
||||||
|
{"id":"c22-trading-term","query":"RabbitMQ vhost","mode":"hybrid","filters":[],"expected_top":null,"expected_allowed":["object/b909ccbc-f7cf-7f71-46ea-817f0b697d35"],"expected_rep":"canonical","expected_state":"current","must_not_top":[],"why":"Trading-/Architektur-Begriff multilingual"}
|
||||||
|
]
|
||||||
|
}
|
||||||
370
tolaria/c4b-search-service/test_corpus_v1.1.json
Normal file
370
tolaria/c4b-search-service/test_corpus_v1.1.json
Normal file
|
|
@ -0,0 +1,370 @@
|
||||||
|
{
|
||||||
|
"schema": "tolaria-c4a-ground-truth-v1.1",
|
||||||
|
"source_of_truth": "Forgejo clone HEAD=35446c0 (MIGRATED=81, DEFERRED=1, DO_NOT_TOUCH=2, TOTAL=84)",
|
||||||
|
"ground_truth_keyed_by": "object_id",
|
||||||
|
"note": "IDs korrigiert gegen SoT-Frontmatter (Forgejo HEAD). v1.0 bleibt als Evidence erhalten. Semantische Intention unverändert.",
|
||||||
|
"corpus_size": 22,
|
||||||
|
"verified_object_ids": {
|
||||||
|
"modul-09-canonical": "object/ffa8db13-8aa6-d4a4-6756-69eabd0c8d77",
|
||||||
|
"modul-09-source": "object/48bd264f-607b-15f1-5f73-3e922af9b19d",
|
||||||
|
"modul-12-canonical": "object/47b8029c-5874-e73f-0279-d335785b359a",
|
||||||
|
"modul-12-source": "object/302e9929-e186-c930-2406-ad4a8f17c6fd",
|
||||||
|
"modul-03-canonical": "object/cf54abfc-4c8b-7477-1b4f-21e76ca7b055",
|
||||||
|
"modul-03-source": "object/1761a726-55af-f97b-d509-84aa4577f102",
|
||||||
|
"modul-01-pg-canonical": "object/ebdc6b2b-4b30-c839-729b-c240ad433a60",
|
||||||
|
"modul-02-rabbitmq-canonical": "object/b909ccbc-f7cf-7f71-46ea-817f0b697d35",
|
||||||
|
"modul-15-design-canonical": "object/5f28e1fd-19f4-e5af-1c92-a17ec80b08f0",
|
||||||
|
"modul-15-impl-canonical": "object/42b89fc3-4572-b493-c05e-dfb85a639df2",
|
||||||
|
"infra-source": "object/f4e559f5-403c-f241-be50-0962d8174644",
|
||||||
|
"infra-canonical": "object/5fcf4885-a361-f49f-2164-f4e10709da34",
|
||||||
|
"historical-phase8-canonical": "object/ac1df780-019b-bb11-cb91-4edaa49e1843",
|
||||||
|
"phase10a-canonical": "object/086b7673-c183-6ca2-52fc-6f394107c9cf",
|
||||||
|
"vps-infra-standalone": "object/9bd5b7db-bb0f-ae22-3a65-9410b77ad3de"
|
||||||
|
},
|
||||||
|
"cases": [
|
||||||
|
{
|
||||||
|
"id": "c01-exact-title",
|
||||||
|
"query": "Modul-09-Execution-Service",
|
||||||
|
"mode": "exact",
|
||||||
|
"filters": {},
|
||||||
|
"expected_top": [
|
||||||
|
"object/ffa8db13-8aa6-d4a4-6756-69eabd0c8d77"
|
||||||
|
],
|
||||||
|
"expected_allowed": [
|
||||||
|
"object/48bd264f-607b-15f1-5f73-3e922af9b19d"
|
||||||
|
],
|
||||||
|
"expected_rep": "canonical",
|
||||||
|
"expected_state": "current",
|
||||||
|
"must_not_top": [],
|
||||||
|
"why": "Exakter Title-Treffer, canonical bevorzugt, source erlaubt, historical nicht top."
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "c02-keyword",
|
||||||
|
"query": "Intrabar Execution Gap Realism",
|
||||||
|
"mode": "keyword",
|
||||||
|
"filters": {},
|
||||||
|
"expected_top": [],
|
||||||
|
"expected_allowed": [
|
||||||
|
"object/47b8029c-5874-e73f-0279-d335785b359a",
|
||||||
|
"object/302e9929-e186-c930-2406-ad4a8f17c6fd"
|
||||||
|
],
|
||||||
|
"expected_rep": "canonical",
|
||||||
|
"expected_state": "current",
|
||||||
|
"must_not_top": [
|
||||||
|
"object/ac1df780-019b-bb11-cb91-4edaa49e1843"
|
||||||
|
],
|
||||||
|
"why": "modul-12 enthält Phase 10e Intrabar/Gap; historical phase8 NICHT top ohne explizit historical"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "c03-phrase",
|
||||||
|
"query": "deterministischer Execution-Service",
|
||||||
|
"mode": "keyword",
|
||||||
|
"filters": [],
|
||||||
|
"expected_top": [],
|
||||||
|
"expected_allowed": [
|
||||||
|
"object/ffa8db13-8aa6-d4a4-6756-69eabd0c8d77"
|
||||||
|
],
|
||||||
|
"expected_rep": "canonical",
|
||||||
|
"expected_state": "current",
|
||||||
|
"must_not_top": [],
|
||||||
|
"why": "Phrase aus modul-09-Body"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "c04-metadata-only",
|
||||||
|
"query": "",
|
||||||
|
"mode": "metadata",
|
||||||
|
"filters": {
|
||||||
|
"type": "arch",
|
||||||
|
"role": "module"
|
||||||
|
},
|
||||||
|
"expected_top": [],
|
||||||
|
"expected_allowed": [
|
||||||
|
"object/ebdc6b2b-4b30-c839-729b-c240ad433a60",
|
||||||
|
"object/ffa8db13-8aa6-d4a4-6756-69eabd0c8d77"
|
||||||
|
],
|
||||||
|
"expected_rep": "any",
|
||||||
|
"expected_state": "current",
|
||||||
|
"must_not_top": [
|
||||||
|
"object/f4e559f5-403c-f241-be50-0962d8174644"
|
||||||
|
],
|
||||||
|
"why": "role=module arch; reference ausgeschlossen"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "c05-canonical-source-pair",
|
||||||
|
"query": "Infrastructure",
|
||||||
|
"mode": "hybrid",
|
||||||
|
"filters": [],
|
||||||
|
"expected_top": [],
|
||||||
|
"expected_allowed": [
|
||||||
|
"object/5fcf4885-a361-f49f-2164-f4e10709da34",
|
||||||
|
"object/f4e559f5-403c-f241-be50-0962d8174644"
|
||||||
|
],
|
||||||
|
"expected_rep": "canonical",
|
||||||
|
"expected_state": "current",
|
||||||
|
"must_not_top": [
|
||||||
|
"object/f4e559f5-403c-f241-be50-0962d8174644"
|
||||||
|
],
|
||||||
|
"why": "collapse: canonical bevorzugt, source zusätzlich, nie 2 unabhängige Fakten"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "c06-current-historical",
|
||||||
|
"query": "M12 DatasetGate",
|
||||||
|
"mode": "keyword",
|
||||||
|
"filters": {
|
||||||
|
"include_historical": false
|
||||||
|
},
|
||||||
|
"expected_top": [],
|
||||||
|
"expected_allowed": [],
|
||||||
|
"expected_rep": "any",
|
||||||
|
"expected_state": "historical",
|
||||||
|
"must_not_top": [
|
||||||
|
"object/ac1df780-019b-bb11-cb91-4edaa49e1843"
|
||||||
|
],
|
||||||
|
"why": "historical nur bei include_historical oder niedrig gerankt"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "c07-readme-explicit",
|
||||||
|
"query": "README",
|
||||||
|
"mode": "keyword",
|
||||||
|
"filters": [],
|
||||||
|
"expected_top": [
|
||||||
|
"README-root",
|
||||||
|
"README-system-docs"
|
||||||
|
],
|
||||||
|
"expected_allowed": [],
|
||||||
|
"expected_rep": "index/overview",
|
||||||
|
"expected_state": "current",
|
||||||
|
"must_not_top": [],
|
||||||
|
"why": "README su-schbar (DO_NOT_MUTATE != DO_NOT_INDEX); gezielte Query findet sie"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "c08-overview-downrank",
|
||||||
|
"query": "Infrastruktur Betrieb",
|
||||||
|
"mode": "hybrid",
|
||||||
|
"filters": [],
|
||||||
|
"expected_top": [],
|
||||||
|
"expected_allowed": [
|
||||||
|
"object/5fcf4885-a361-f49f-2164-f4e10709da34",
|
||||||
|
"object/f4e559f5-403c-f241-be50-0962d8174644"
|
||||||
|
],
|
||||||
|
"expected_rep": "canonical",
|
||||||
|
"expected_state": "current",
|
||||||
|
"must_not_top": [
|
||||||
|
"README"
|
||||||
|
],
|
||||||
|
"why": "overview/index bei Fach-Query NICHT dominieren"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "c09-crosslang-de-en",
|
||||||
|
"query": "Marktdaten",
|
||||||
|
"mode": "semantic",
|
||||||
|
"filters": [],
|
||||||
|
"expected_top": [],
|
||||||
|
"expected_allowed": [
|
||||||
|
"object/cf54abfc-4c8b-7477-1b4f-21e76ca7b055"
|
||||||
|
],
|
||||||
|
"expected_rep": "canonical",
|
||||||
|
"expected_state": "current",
|
||||||
|
"must_not_top": [],
|
||||||
|
"why": "DE-Term -> EN Object (modul-03 Market-Data)"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "c10-crosslang-en-de",
|
||||||
|
"query": "market data",
|
||||||
|
"mode": "semantic",
|
||||||
|
"filters": [],
|
||||||
|
"expected_top": [],
|
||||||
|
"expected_allowed": [
|
||||||
|
"object/cf54abfc-4c8b-7477-1b4f-21e76ca7b055"
|
||||||
|
],
|
||||||
|
"expected_rep": "canonical",
|
||||||
|
"expected_state": "current",
|
||||||
|
"must_not_top": [],
|
||||||
|
"why": "EN->DE multilingual"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "c11-semantic-paraphrase",
|
||||||
|
"query": "Wie werden Orders ausgeführt",
|
||||||
|
"mode": "semantic",
|
||||||
|
"filters": [],
|
||||||
|
"expected_top": [],
|
||||||
|
"expected_allowed": [
|
||||||
|
"object/ffa8db13-8aa6-d4a4-6756-69eabd0c8d77"
|
||||||
|
],
|
||||||
|
"expected_rep": "canonical",
|
||||||
|
"expected_state": "current",
|
||||||
|
"must_not_top": [],
|
||||||
|
"why": "Paraphrase Execution-Service; Semantic muss Mehrwert ggü keyword zeigen"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "c12-ambiguous",
|
||||||
|
"query": "Modul 15",
|
||||||
|
"mode": "keyword",
|
||||||
|
"filters": [],
|
||||||
|
"expected_top": [
|
||||||
|
"object/5f28e1fd-19f4-e5af-1c92-a17ec80b08f0",
|
||||||
|
"object/42b89fc3-4572-b493-c05e-dfb85a639df2"
|
||||||
|
],
|
||||||
|
"expected_allowed": [],
|
||||||
|
"expected_rep": "any",
|
||||||
|
"expected_state": "current",
|
||||||
|
"must_not_top": [],
|
||||||
|
"why": "zwei Objekte (design+impl) -> collapse/group"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "c13-no-result",
|
||||||
|
"query": "xyzzy-foobar-42",
|
||||||
|
"mode": "keyword",
|
||||||
|
"filters": [],
|
||||||
|
"expected_top": [],
|
||||||
|
"expected_allowed": [],
|
||||||
|
"expected_rep": "-",
|
||||||
|
"expected_state": "-",
|
||||||
|
"must_not_top": [],
|
||||||
|
"why": "kein Treffer -> leeres Ergebnis, no-result correctness"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "c14-module-number",
|
||||||
|
"query": "Modul 15",
|
||||||
|
"mode": "keyword",
|
||||||
|
"filters": [],
|
||||||
|
"expected_top": [],
|
||||||
|
"expected_allowed": [
|
||||||
|
"object/5f28e1fd-19f4-e5af-1c92-a17ec80b08f0",
|
||||||
|
"object/42b89fc3-4572-b493-c05e-dfb85a639df2"
|
||||||
|
],
|
||||||
|
"expected_rep": "canonical",
|
||||||
|
"expected_state": "current",
|
||||||
|
"must_not_top": [],
|
||||||
|
"why": "Modulnummer-Auflösung"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "c15-ticker-acronym",
|
||||||
|
"query": "OHLCV",
|
||||||
|
"mode": "keyword",
|
||||||
|
"filters": [],
|
||||||
|
"expected_top": [],
|
||||||
|
"expected_allowed": [
|
||||||
|
"object/cf54abfc-4c8b-7477-1b4f-21e76ca7b055",
|
||||||
|
"object/ebdc6b2b-4b30-c839-729b-c240ad433a60"
|
||||||
|
],
|
||||||
|
"expected_rep": "canonical",
|
||||||
|
"expected_state": "current",
|
||||||
|
"must_not_top": [],
|
||||||
|
"why": "Ticker/Akronym-Token"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "c16-env-config-token",
|
||||||
|
"query": "POSTGRES_DB",
|
||||||
|
"mode": "keyword",
|
||||||
|
"filters": {},
|
||||||
|
"expected_top": [],
|
||||||
|
"expected_allowed": [
|
||||||
|
"object/ebdc6b2b-4b30-c839-729b-c240ad433a60"
|
||||||
|
],
|
||||||
|
"expected_rep": "canonical",
|
||||||
|
"expected_state": "current",
|
||||||
|
"must_not_top": [],
|
||||||
|
"why": "Env-/Config-Token via Keyword-Index; code-chunks nicht sem-embed"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "c17-relationship",
|
||||||
|
"query": "ADR Modul 09",
|
||||||
|
"mode": "keyword",
|
||||||
|
"filters": {
|
||||||
|
"relation": "related_to"
|
||||||
|
},
|
||||||
|
"expected_top": [],
|
||||||
|
"expected_allowed": [
|
||||||
|
"object/ffa8db13-8aa6-d4a4-6756-69eabd0c8d77"
|
||||||
|
],
|
||||||
|
"expected_rep": "any",
|
||||||
|
"expected_state": "current",
|
||||||
|
"must_not_top": [],
|
||||||
|
"why": "Relation-Direct-Stufe (belongs_to/derived_from/related_to)"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "c18-historical-explicit",
|
||||||
|
"query": "historical phase8",
|
||||||
|
"mode": "keyword",
|
||||||
|
"filters": {
|
||||||
|
"include_historical": true
|
||||||
|
},
|
||||||
|
"expected_top": [
|
||||||
|
"object/ac1df780-019b-bb11-cb91-4edaa49e1843"
|
||||||
|
],
|
||||||
|
"expected_allowed": [],
|
||||||
|
"expected_rep": "canonical",
|
||||||
|
"expected_state": "historical",
|
||||||
|
"must_not_top": [],
|
||||||
|
"why": "explizit historical -> phase8-Obj erreichbar"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "c19-source-only",
|
||||||
|
"query": "source infrastructure",
|
||||||
|
"mode": "keyword",
|
||||||
|
"filters": {
|
||||||
|
"representation": "source"
|
||||||
|
},
|
||||||
|
"expected_top": [],
|
||||||
|
"expected_allowed": [
|
||||||
|
"object/f4e559f5-403c-f241-be50-0962d8174644"
|
||||||
|
],
|
||||||
|
"expected_rep": "source",
|
||||||
|
"expected_state": "current",
|
||||||
|
"must_not_top": [
|
||||||
|
"object/5fcf4885-a361-f49f-2164-f4e10709da34"
|
||||||
|
],
|
||||||
|
"why": "source-only Filter"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "c20-canonical-only",
|
||||||
|
"query": "infrastructure",
|
||||||
|
"mode": "keyword",
|
||||||
|
"filters": {
|
||||||
|
"representation": "canonical"
|
||||||
|
},
|
||||||
|
"expected_top": [],
|
||||||
|
"expected_allowed": [
|
||||||
|
"object/5fcf4885-a361-f49f-2164-f4e10709da34"
|
||||||
|
],
|
||||||
|
"expected_rep": "canonical",
|
||||||
|
"expected_state": "current",
|
||||||
|
"must_not_top": [
|
||||||
|
"object/f4e559f5-403c-f241-be50-0962d8174644"
|
||||||
|
],
|
||||||
|
"why": "canonical-only Filter"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "c21-config-code",
|
||||||
|
"query": "OLLAMA_API_KEY",
|
||||||
|
"mode": "keyword",
|
||||||
|
"filters": {
|
||||||
|
"type": "code"
|
||||||
|
},
|
||||||
|
"expected_top": [],
|
||||||
|
"expected_allowed": [],
|
||||||
|
"expected_rep": "-",
|
||||||
|
"expected_state": "-",
|
||||||
|
"must_not_top": [],
|
||||||
|
"why": "kein sekret-Index; pre-index scan filtert secrets"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "c22-trading-term",
|
||||||
|
"query": "RabbitMQ vhost",
|
||||||
|
"mode": "hybrid",
|
||||||
|
"filters": [],
|
||||||
|
"expected_top": [],
|
||||||
|
"expected_allowed": [
|
||||||
|
"object/b909ccbc-f7cf-7f71-46ea-817f0b697d35"
|
||||||
|
],
|
||||||
|
"expected_rep": "canonical",
|
||||||
|
"expected_state": "current",
|
||||||
|
"must_not_top": [],
|
||||||
|
"why": "Trading-/Architektur-Begriff multilingual"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"corpus_version_old": "1.0",
|
||||||
|
"corpus_version_new": "1.1",
|
||||||
|
"erratum": "C4A_GROUND_TRUTH_ERRATUM.md"
|
||||||
|
}
|
||||||
Loading…
Reference in a new issue