feat(tolaria): add C5 Forgejo polling and change detection
This commit is contained in:
parent
183afed1d6
commit
091c1828c7
4 changed files with 1691 additions and 21 deletions
|
|
@ -1,32 +1,142 @@
|
||||||
# C5A — CONTRACT & STATE MACHINE v1
|
# C5 SYNC SERVICE — CONTRACT & STATE MACHINE (C5A) + FORGEJO POLLING & CHANGE DETECTION (C5B)
|
||||||
|
|
||||||
**Phase:** C5A (erste Phase des C5 Sync Service)
|
**Phase:** C5A (Contract & State Machine) + C5B (Forgejo Polling & Change Detection)
|
||||||
**Status:** DONE (Contract & State Machine) — **INAKTIV, NO-WRITE**
|
**Status:** C5A = DONE · C5B = DONE (Polling & Change Detection) — **INAKTIV, NO-WRITE**
|
||||||
**Architektur:** FORGEJO MASTER → C5 SYNC SERVICE → TOLARIA DERIVED → SEARCH FULL REBUILD
|
**Architektur:** FORGEJO MASTER → C5 SYNC SERVICE → TOLARIA DERIVED → SEARCH FULL REBUILD
|
||||||
|
|
||||||
C5A implementiert **ausschließlich** den Contract + die Sync State Machine + die
|
C5A implementiert **ausschließlich** den Contract + die Sync State Machine + die
|
||||||
persistente Progress-State-Struktur. Es ist eine **deterministische, inaktive
|
persistente Progress-State-Struktur. C5B baut darauf die **Forgejo Polling &
|
||||||
Python-Library + CLI + Testsuite** — es führt **keine** externen Writes aus.
|
Change Detection Engine** (read-only). Beide sind **deterministische, inaktive
|
||||||
|
Python-Libraries + CLI + Testsuite** — sie führen **keine** externen Writes aus.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## NO-WRITE GUARANTEE (verbindlich)
|
## NO-WRITE GUARANTEE (verbindlich, C5A + C5B)
|
||||||
|
|
||||||
C5A kann **NOCH NICHT**:
|
C5A + C5B können **NOCH NICHT**:
|
||||||
- Tolaria schreiben
|
- Tolaria schreiben
|
||||||
- Search rebuilden
|
- Search rebuilden
|
||||||
- Forgejo schreiben
|
- Forgejo schreiben
|
||||||
- produktiv pollen
|
- produktiv pollen (kein Poll-Daemon)
|
||||||
- Netzwerk-Mutationen ausführen
|
- Netzwerk-Mutationen ausführen
|
||||||
|
|
||||||
Es gibt **keine** Netzwerk-/HTTP-/Socket-/Subprocess-Mutationsfunktionen in C5A.
|
**C5A** hat **keine** Netzwerk-/HTTP-/Socket-/Subprocess-Mutationsfunktionen
|
||||||
Die No-Write-Guarantee wird statisch per `assert_no_write_guarantee()` bewiesen
|
(`assert_no_write_guarantee()`).
|
||||||
(keine `requests`/`urllib`/`http`/`socket`/`subprocess`-Imports) und durch den
|
|
||||||
Test `test_no_write_guarantee` abgesichert.
|
**C5B** führt **ausschließlich read-only git-Befehle** gegen den lokalen Clone aus
|
||||||
|
(strikte Whitelist: `rev-parse`, `log`, `show`, `diff-tree`, `ls-tree`,
|
||||||
|
`cat-file`, `merge-base`, `rev-list`, `diff`). Jeder Write-Befehl
|
||||||
|
(`push`/`commit`/`add`/`reset`/`checkout`/`merge`/`rebase`) wird mit
|
||||||
|
`GitWriteBlockedError` blockiert. Die No-Downstream-Write-Guarantee wird statisch
|
||||||
|
per `assert_no_downstream_write()` bewiesen (keine HTTP-Mutationsimporte, keine
|
||||||
|
`/api/vault/save`-/`/api/search/rebuild`-Endpoints im Code, keine Git-Write-Befehle
|
||||||
|
in der Whitelist) und durch den Test `test_no_downstream_write` abgesichert.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## State Machine
|
## C5B — FORGEJO POLLING & CHANGE DETECTION ENGINE
|
||||||
|
|
||||||
|
### FORGEJO_READ_METHOD
|
||||||
|
**A) lokale Git-Mirror/Clone-Analyse** (bevorzugt).
|
||||||
|
- **AUTH_REQUIREMENT:** KEIN (read-only gegen lokalen Clone; kein Write-Token)
|
||||||
|
- **NETWORK_DEPENDENCY:** KEIN (lokale `.git`-Objekte; kein HTTP)
|
||||||
|
|
||||||
|
### POLLING_MODEL
|
||||||
|
- **Default:** 60 Sekunden
|
||||||
|
- **Environment-konfigurierbar:** `C5_POLL_INTERVAL_SECONDS` (z.B. `=120`)
|
||||||
|
- C5B implementiert die **Polling-Logik**, startet aber **KEINEN** dauerhaften
|
||||||
|
Produktions-Daemon. Tests führen einzelne kontrollierte `poll_once`-Zyklen aus.
|
||||||
|
|
||||||
|
### HEAD_DISCOVERY
|
||||||
|
`get_current_head()` → Commit-SHA + parent/timestamp/message. Bei nicht
|
||||||
|
erreichbarem Forgejo: `RC_FORGEJO_UNAVAILABLE`, kein State-Fortschritt.
|
||||||
|
|
||||||
|
### COMMIT_RANGE_MODEL
|
||||||
|
Aus `last_seen_commit` (C5B-Baseline) bis HEAD deterministisch alle fehlenden
|
||||||
|
Commits in chronologischer (parent-korrekter) Reihenfolge. Kein Überspringen,
|
||||||
|
keine Doppelerfassung. Merge-Commits werden explizit behandelt (alle Parents).
|
||||||
|
Divergenz/History-Rewrite wird erkannt.
|
||||||
|
|
||||||
|
> **Wichtig:** C5B-Baseline ist `last_seen_commit`. `last_applied_commit` bleibt
|
||||||
|
> laut C5A-Contract für die spätere Propagation (C5C) reserviert und wird von C5B
|
||||||
|
> **NIE** fortgeschrieben.
|
||||||
|
|
||||||
|
### HISTORY_SAFETY
|
||||||
|
Wenn `last_seen_commit` kein Ancestor von HEAD ist: **FAIL CLOSED**
|
||||||
|
(`RC_OUT_OF_ORDER_COMMIT`). Keine automatische Rebase-/Reset-Logik.
|
||||||
|
|
||||||
|
### DIFF_MODEL
|
||||||
|
Für jeden neuen Commit: `git diff-tree -r --name-status -M -C --root` →
|
||||||
|
`ADDED` / `MODIFIED` / `DELETED` / `RENAMED` / `MOVED`. Bei unsicherer
|
||||||
|
Rename-Detection wird **nicht geraten** — Ambiguität wird sauber klassifiziert
|
||||||
|
(Human Review).
|
||||||
|
|
||||||
|
### KNOWLEDGE_SCOPE_MODEL
|
||||||
|
Nicht jede Forgejo-Datei ist automatisch Tolaria-Knowledge:
|
||||||
|
- **IN_SCOPE:** Markdown im Repo-Root ODER unter `notes/trading/system-docs/`
|
||||||
|
- **OUT_OF_SCOPE:** `tolaria/`, `a2-5/`, `red-queen-architecture/`,
|
||||||
|
`notion-safety-brain/`, `notion-command-center/`, `backup_patches/`,
|
||||||
|
`notes/reference/`, Nicht-Markdown, Build-Artefakte
|
||||||
|
- **LEGACY_SPECIAL:** `README.md` (x2, KEEP_DISTINCT) + `vps.md` (DEFERRED)
|
||||||
|
- **HUMAN_REVIEW:** Knowledge-Objekt ohne gültige `object_id`
|
||||||
|
|
||||||
|
Keine Secrets/Build-Artefakte in Sync-Change-Sets. Nicht blind `.md` = Sync.
|
||||||
|
|
||||||
|
### OBJECT_ID_MODEL
|
||||||
|
`object_id` wird **aus dem Frontmatter** gelesen (`id: object/<FULL_UUID>`),
|
||||||
|
nicht aus dem Pfad erzeugt, nicht neu generiert, nicht geraten. Ohne gültige ID:
|
||||||
|
`UNKNOWN_OBJECT_ID` / `UNKNOWN_LEGACY_OBJECT` → Human Gate.
|
||||||
|
|
||||||
|
### HASH_MODEL
|
||||||
|
- **`content_hash`:** SHA-256 des fachlichen Bodies (nach Frontmatter)
|
||||||
|
- **`metadata_hash`:** SHA-256 der relevanten Knowledge-Schema-Metadaten
|
||||||
|
(`id`, `type`, `role`, `representation`, `state`, `derived_from`, `tags`,
|
||||||
|
`knowledge_schema`). Pfad ist separat.
|
||||||
|
- Deterministisch und testbar.
|
||||||
|
|
||||||
|
### CHANGE_CLASSIFICATION
|
||||||
|
Aus Before/After deterministisch: `CREATE`, `CONTENT_UPDATE`,
|
||||||
|
`METADATA_UPDATE`, `STATE_UPDATE`, `RENAME`, `MOVE`,
|
||||||
|
`SOURCE_CANONICAL_RELATION_UPDATE`, `TAGS_UPDATE`, `SUPERSEDE`,
|
||||||
|
`DELETE_REQUEST`. Bei mehreren geänderten Aspekten in EINEM Objekt werden
|
||||||
|
**mehrere ObjectChange-Einträge** erzeugt (keine Information verloren).
|
||||||
|
|
||||||
|
### RENAME_MOVE_MODEL
|
||||||
|
Rename/Move nur automatisch, wenn **gleiche `object_id`** + eindeutige
|
||||||
|
Git-/Content-Evidence. ID bleibt stabil. Rename = Dateiname geändert (gleiches
|
||||||
|
Verzeichnis); Move = Verzeichnis geändert. Bei Ambiguität: Human Gate.
|
||||||
|
|
||||||
|
### DELETE_MODEL
|
||||||
|
Git-Delete wird **NIEMALS** zu automatischem Hard Delete. C5B erzeugt
|
||||||
|
`DELETE_REQUEST` → C5A führt es in Human Review. Keine Tolaria-Aktion.
|
||||||
|
|
||||||
|
### PAIR_MODEL
|
||||||
|
Wenn Root/source + canonical gemeinsam verändert werden, erkennt C5B die
|
||||||
|
Pair-Beziehung (`derived_from`). Ordnung für spätere C5C: Source zuerst,
|
||||||
|
Canonical danach. Noch nicht propagiert — Output enthält genug Information für
|
||||||
|
C5C.
|
||||||
|
|
||||||
|
### SECRET_SAFETY
|
||||||
|
C5B schreibt **keine** Secret-Inhalte in Logs/SQLite/Evidence. Bei
|
||||||
|
Secret-/Forbidden-Dateien: nur sichere Metadaten (`path`, `reason_code`,
|
||||||
|
`commit_sha`). Enthält ein Knowledge-Dokument einen möglichen Secret-Wert:
|
||||||
|
`RC_SECRET_DETECTED` FAIL CLOSED. Kein Wert wird ausgegeben.
|
||||||
|
|
||||||
|
### C5A_INTEGRATION
|
||||||
|
C5B speist gefundene Commits + ObjectChanges in den C5A Store ein (nur
|
||||||
|
State/Persistence, kein Network Write downstream). Flow: Poll once → discover
|
||||||
|
commits → classify changes → register commit → add object changes → Status
|
||||||
|
`DISCOVERED`/`VALIDATING` (bzw. `HUMAN_REVIEW_REQUIRED`). **Nicht** weiter zu
|
||||||
|
`PROPAGATING_TOLARIA`.
|
||||||
|
|
||||||
|
### OBSERVABILITY
|
||||||
|
C5B erweitert sichere Statusdaten: `last_seen_commit`, `poll_timestamp`,
|
||||||
|
`commits_discovered`, `objects_discovered`, `out_of_scope_count`,
|
||||||
|
`last_poll_status`, `last_poll_error_code`. Keine Content-Logs.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## State Machine (C5A)
|
||||||
|
|
||||||
### Normalzustände
|
### Normalzustände
|
||||||
`DISCOVERED → VALIDATING → READY → PROPAGATING_TOLARIA → VERIFYING_TOLARIA → UPDATING_SEARCH → VERIFYING_SEARCH → APPLIED`
|
`DISCOVERED → VALIDATING → READY → PROPAGATING_TOLARIA → VERIFYING_TOLARIA → UPDATING_SEARCH → VERIFYING_SEARCH → APPLIED`
|
||||||
|
|
@ -169,6 +279,10 @@ C5A führt **keine echten externen Health-Probes** aus (rein State-Machine) —
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
export C5A_DB=/path/to/c5a.db # default: ./c5a.db
|
export C5A_DB=/path/to/c5a.db # default: ./c5a.db
|
||||||
|
export C5_REPO=/path/to/forgejo-clone # default: .
|
||||||
|
export C5_POLL_INTERVAL_SECONDS=60 # default: 60
|
||||||
|
|
||||||
|
# C5A
|
||||||
python3 rq_c5a_cli.py health
|
python3 rq_c5a_cli.py health
|
||||||
python3 rq_c5a_cli.py bootstrap-state
|
python3 rq_c5a_cli.py bootstrap-state
|
||||||
python3 rq_c5a_cli.py bootstrap-transition --to RECONCILING
|
python3 rq_c5a_cli.py bootstrap-transition --to RECONCILING
|
||||||
|
|
@ -178,30 +292,52 @@ python3 rq_c5a_cli.py process-commit --json <file>
|
||||||
python3 rq_c5a_cli.py list-commits [--status <s>]
|
python3 rq_c5a_cli.py list-commits [--status <s>]
|
||||||
python3 rq_c5a_cli.py commit-status --sha <sha>
|
python3 rq_c5a_cli.py commit-status --sha <sha>
|
||||||
python3 rq_c5a_cli.py no-write-check
|
python3 rq_c5a_cli.py no-write-check
|
||||||
|
|
||||||
|
# C5B (rq_c5_cli.py)
|
||||||
|
python3 rq_c5_cli.py poll-once # ein kontrollierter Poll-Zyklus (read-only)
|
||||||
|
python3 rq_c5_cli.py show-pending # ausstehende Commits anzeigen
|
||||||
|
python3 rq_c5_cli.py show-commit <sha> # Details eines Commits anzeigen
|
||||||
|
python3 rq_c5_cli.py dry-run-diff <sha> # Diff eines Commits anzeigen (read-only)
|
||||||
|
python3 rq_c5_cli.py health # Health-Contract anzeigen
|
||||||
|
python3 rq_c5_cli.py no-write-check # No-Downstream-Write-Guarantee pruefen
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**Kein** `daemon`/`start-forever`-Befehl in C5B — der produktive Poll-Daemon kommt
|
||||||
|
erst beim Deployment (C5).
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Tests
|
## Tests
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
python3 test_c5a.py # 25 PASS / 0 FAIL
|
python3 test_c5a.py # 25 PASS / 0 FAIL
|
||||||
|
python3 -m unittest test_c5b -v # 39 PASS / 0 FAIL
|
||||||
```
|
```
|
||||||
|
|
||||||
Abgedeckte Fälle: normal commit lifecycle, multi-object commit, duplicate commit,
|
**C5A** abgedeckte Fälle: normal commit lifecycle, multi-object commit, duplicate
|
||||||
already-applied, out-of-order commit, missing predecessor, retry progression,
|
commit, already-applied, out-of-order commit, missing predecessor, retry
|
||||||
max retry → DEAD, human gate transition, restart/reload persistence, state
|
progression, max retry → DEAD, human gate transition, restart/reload persistence,
|
||||||
corruption handling, idempotency, delete request → human gate, dangling relation
|
state corruption handling, idempotency, delete request → human gate, dangling
|
||||||
→ human gate, unexpected drift → human gate, no-write guarantee, bootstrap
|
relation → human gate, unexpected drift → human gate, no-write guarantee, bootstrap
|
||||||
lifecycle, baseline-only-from-BASELINE_READY, reason codes closed set, operations
|
lifecycle, baseline-only-from-BASELINE_READY, reason codes closed set, operations
|
||||||
closed set, health contract fields, last_applied-only-after-full-pass, retry
|
closed set, health contract fields, last_applied-only-after-full-pass, retry
|
||||||
available bound, backoff sequence, invalid reason code rejected.
|
available bound, backoff sequence, invalid reason code rejected.
|
||||||
|
|
||||||
|
**C5B** abgedeckte Fälle: head discovery, single commit, multi commit, added
|
||||||
|
knowledge object, content update, metadata update, state update, tags update,
|
||||||
|
relation update, rename, move, delete request, out-of-scope file, legacy object,
|
||||||
|
unknown ID, secret detection, duplicate poll, lost poll, ordering, Forgejo
|
||||||
|
unavailable, history divergence, C5A persistence integration, no downstream write,
|
||||||
|
poll interval config.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Dateien
|
## Dateien
|
||||||
|
|
||||||
- `rq_c5a.py` — Kern-Library (Store + State Machine + No-Write-Check)
|
- `rq_c5a.py` — C5A Kern-Library (Store + State Machine + No-Write-Check)
|
||||||
- `rq_c5a_cli.py` — CLI
|
- `rq_c5a_cli.py` — C5A CLI
|
||||||
- `test_c5a.py` — Testsuite (25 Tests)
|
- `test_c5a.py` — C5A Testsuite (25 Tests)
|
||||||
|
- `rq_c5b.py` — C5B Kern-Library (GitReader + Scope + Classifier + Poller + No-Downstream-Write-Check)
|
||||||
|
- `rq_c5_cli.py` — C5 CLI (C5A + C5B: poll-once, show-pending, show-commit, dry-run-diff, health, no-write-check)
|
||||||
|
- `test_c5b.py` — C5B Testsuite (39 Tests)
|
||||||
- `README.md` — diese Datei
|
- `README.md` — diese Datei
|
||||||
|
|
|
||||||
139
tolaria/c5-sync-service/rq_c5_cli.py
Normal file
139
tolaria/c5-sync-service/rq_c5_cli.py
Normal file
|
|
@ -0,0 +1,139 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Red Queen — C5 CLI (C5A + C5B): kontrollierte Inspektion & Dry-Run.
|
||||||
|
|
||||||
|
KEIN produktiver Poll-Daemon. KEINE daemon/start-forever-Funktion.
|
||||||
|
Nur explizite, begrenzte Befehle:
|
||||||
|
* poll-once — ein kontrollierter Poll-Zyklus (read-only)
|
||||||
|
* show-pending — ausstehende Commits anzeigen
|
||||||
|
* show-commit — Details eines Commits anzeigen
|
||||||
|
* dry-run-diff — Diff eines Commits anzeigen (read-only)
|
||||||
|
* health — Health-Contract anzeigen
|
||||||
|
* no-write-check — No-Downstream-Write-Guarantee pruefen
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
from typing import Any, Dict, List, Optional
|
||||||
|
|
||||||
|
from rq_c5a import C5AStore
|
||||||
|
from rq_c5b import (
|
||||||
|
C5BPoller,
|
||||||
|
GitReader,
|
||||||
|
assert_no_downstream_write,
|
||||||
|
DEFAULT_POLL_INTERVAL_SECONDS,
|
||||||
|
ENV_POLL_INTERVAL,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _store(db_path: str) -> C5AStore:
|
||||||
|
return C5AStore(db_path)
|
||||||
|
|
||||||
|
|
||||||
|
def _poller(store: C5AStore, repo_path: str, interval: Optional[int]) -> C5BPoller:
|
||||||
|
return C5BPoller(store, repo_path, poll_interval_seconds=interval)
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_poll_once(args: argparse.Namespace) -> int:
|
||||||
|
store = _store(args.db)
|
||||||
|
poller = _poller(store, args.repo, args.interval)
|
||||||
|
result = poller.poll_once()
|
||||||
|
print(json.dumps(result, ensure_ascii=False, indent=2))
|
||||||
|
store.close()
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_show_pending(args: argparse.Namespace) -> int:
|
||||||
|
store = _store(args.db)
|
||||||
|
pending = store.list_commits()
|
||||||
|
print(json.dumps(pending, ensure_ascii=False, indent=2))
|
||||||
|
store.close()
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_show_commit(args: argparse.Namespace) -> int:
|
||||||
|
store = _store(args.db)
|
||||||
|
commit = store.get_commit(args.sha)
|
||||||
|
if commit is None:
|
||||||
|
print(f"Commit nicht gefunden: {args.sha}")
|
||||||
|
store.close()
|
||||||
|
return 1
|
||||||
|
print(json.dumps(commit, ensure_ascii=False, indent=2))
|
||||||
|
objs = store.list_object_changes(args.sha)
|
||||||
|
if objs:
|
||||||
|
print("--- Object Changes ---")
|
||||||
|
print(json.dumps(objs, ensure_ascii=False, indent=2))
|
||||||
|
store.close()
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_dry_run_diff(args: argparse.Namespace) -> int:
|
||||||
|
reader = GitReader(args.repo)
|
||||||
|
changes = reader.diff_name_status(args.sha)
|
||||||
|
print(json.dumps(changes, ensure_ascii=False, indent=2))
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_health(args: argparse.Namespace) -> int:
|
||||||
|
store = _store(args.db)
|
||||||
|
print(json.dumps(store.health(), ensure_ascii=False, indent=2))
|
||||||
|
store.close()
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_no_write_check(args: argparse.Namespace) -> int:
|
||||||
|
result = assert_no_downstream_write()
|
||||||
|
print(json.dumps(result, ensure_ascii=False, indent=2))
|
||||||
|
return 0 if result["no_downstream_write"] else 1
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv: Optional[List[str]] = None) -> int:
|
||||||
|
parser = argparse.ArgumentParser(
|
||||||
|
prog="rq_c5_cli",
|
||||||
|
description="Red Queen C5 CLI (C5A + C5B) — kontrollierte Inspektion & Dry-Run",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--db", default=os.environ.get("C5A_DB", "c5a.db"),
|
||||||
|
help="Pfad zur C5A-SQLite-DB (Env C5A_DB, Default c5a.db)",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--repo", default=os.environ.get("C5_REPO", "."),
|
||||||
|
help="Pfad zum Forgejo-Clone (Env C5_REPO, Default .)",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--interval", type=int, default=None,
|
||||||
|
help=f"Poll-Intervall in Sekunden (Env {ENV_POLL_INTERVAL}, Default {DEFAULT_POLL_INTERVAL_SECONDS})",
|
||||||
|
)
|
||||||
|
|
||||||
|
sub = parser.add_subparsers(dest="command", required=True)
|
||||||
|
|
||||||
|
p = sub.add_parser("poll-once", help="Ein kontrollierter Poll-Zyklus (read-only)")
|
||||||
|
p.set_defaults(func=cmd_poll_once)
|
||||||
|
|
||||||
|
p = sub.add_parser("show-pending", help="Ausstehende Commits anzeigen")
|
||||||
|
p.set_defaults(func=cmd_show_pending)
|
||||||
|
|
||||||
|
p = sub.add_parser("show-commit", help="Details eines Commits anzeigen")
|
||||||
|
p.add_argument("sha", help="Commit-SHA")
|
||||||
|
p.set_defaults(func=cmd_show_commit)
|
||||||
|
|
||||||
|
p = sub.add_parser("dry-run-diff", help="Diff eines Commits anzeigen (read-only)")
|
||||||
|
p.add_argument("sha", help="Commit-SHA")
|
||||||
|
p.set_defaults(func=cmd_dry_run_diff)
|
||||||
|
|
||||||
|
p = sub.add_parser("health", help="Health-Contract anzeigen")
|
||||||
|
p.set_defaults(func=cmd_health)
|
||||||
|
|
||||||
|
p = sub.add_parser("no-write-check", help="No-Downstream-Write-Guarantee pruefen")
|
||||||
|
p.set_defaults(func=cmd_no_write_check)
|
||||||
|
|
||||||
|
args = parser.parse_args(argv)
|
||||||
|
return args.func(args)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
856
tolaria/c5-sync-service/rq_c5b.py
Normal file
856
tolaria/c5-sync-service/rq_c5b.py
Normal file
|
|
@ -0,0 +1,856 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Red Queen — C5B: FORGEJO POLLING & CHANGE DETECTION ENGINE v1 (read-only).
|
||||||
|
|
||||||
|
VERBINDLICHER RAHMEN
|
||||||
|
====================
|
||||||
|
C5B ist die zweite Phase des C5 Sync Service (FORGEJO MASTER -> C5 SYNC SERVICE ->
|
||||||
|
TOLARIA DERIVED -> SEARCH FULL REBUILD). C5B baut AUSSCHLIESSLICH:
|
||||||
|
* FORGEJO READ (lokale Git-Clone-Analyse, read-only)
|
||||||
|
* COMMIT DISCOVERY (HEAD + Commit-Range)
|
||||||
|
* COMMIT ORDERING (chronologisch, parent-korrekt)
|
||||||
|
* DIFF / CHANGE DETECTION (ADDED/MODIFIED/DELETED/RENAMED/MOVED)
|
||||||
|
* OBJECT CHANGE CONTRACT (BEFORE/AFTER, Hashes, Operationen)
|
||||||
|
* Uebergabe an den C5A State Store (nur State/Persistence)
|
||||||
|
|
||||||
|
NOCH KEINE PROPAGATION. C5B propagiert NICHT nach Tolaria, rebuildet NICHT den
|
||||||
|
Search, schreibt NICHT nach Forgejo, startet KEINEN produktiven Poll-Daemon.
|
||||||
|
|
||||||
|
ABSOLUTES WRITE-VERBOT (C5B):
|
||||||
|
* KEIN Forgejo-Write / kein git push / kein git commit / kein git add.
|
||||||
|
* KEIN Tolaria-Write / kein POST /api/vault/save.
|
||||||
|
* KEIN Search-Rebuild / kein POST /api/search/rebuild.
|
||||||
|
* KEIN Container-Deploy, KEINE Netzwerk-Aenderung, KEINE Hermes-Rechte.
|
||||||
|
* KEIN produktiver Polling-Daemon (nur kontrollierte poll_once-Zyklen in Tests).
|
||||||
|
* KEIN C5C / C5D / C5-Deployment.
|
||||||
|
|
||||||
|
FORGEJO_READ_METHOD = A) lokale Git-Mirror/Clone-Analyse (bevorzugt).
|
||||||
|
* AUTH_REQUIREMENT = KEIN (read-only gegen lokalen Clone; kein Write-Token).
|
||||||
|
* NETWORK_DEPENDENCY = KEIN (lokale .git-Objekte; kein HTTP).
|
||||||
|
* Der GitReader fuehrt NUR read-only git-Befehle aus (strikte Whitelist).
|
||||||
|
|
||||||
|
SECURITY BOUNDARY (dokumentiert, NICHT implementiert):
|
||||||
|
* Forgejo credential: READ ONLY (C5B haelt KEINEN Forgejo-Write).
|
||||||
|
* Tolaria write: nur spaeter C5 Service (C5B schreibt NIE nach Tolaria).
|
||||||
|
* Search rebuild: nur C5 Service Token (C5B ruft NIE Search-Rebuild).
|
||||||
|
* Agents: keine direkten Tolaria-/Search-Admin-Writes.
|
||||||
|
* Netzwerk-Haertung wird spaeter beim Deployment umgesetzt, NICHT jetzt.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import subprocess
|
||||||
|
import time
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any, Dict, List, Optional, Tuple
|
||||||
|
|
||||||
|
from rq_c5a import (
|
||||||
|
C5AStore,
|
||||||
|
ST_DISCOVERED,
|
||||||
|
ST_VALIDATING,
|
||||||
|
ST_WAITING_FOR_PREDECESSOR,
|
||||||
|
ST_HUMAN_REVIEW_REQUIRED,
|
||||||
|
ST_APPLIED,
|
||||||
|
OP_CREATE,
|
||||||
|
OP_CONTENT_UPDATE,
|
||||||
|
OP_METADATA_UPDATE,
|
||||||
|
OP_STATE_UPDATE,
|
||||||
|
OP_RENAME,
|
||||||
|
OP_MOVE,
|
||||||
|
OP_SOURCE_CANONICAL_RELATION_UPDATE,
|
||||||
|
OP_TAGS_UPDATE,
|
||||||
|
OP_SUPERSEDE,
|
||||||
|
OP_DELETE_REQUEST,
|
||||||
|
RC_FORGEJO_UNAVAILABLE,
|
||||||
|
RC_OUT_OF_ORDER_COMMIT,
|
||||||
|
RC_UNKNOWN_OBJECT_ID,
|
||||||
|
RC_UNKNOWN_LEGACY_OBJECT,
|
||||||
|
RC_SECRET_DETECTED,
|
||||||
|
RC_INVALID_SCHEMA,
|
||||||
|
RC_DANGLING_DERIVED_FROM,
|
||||||
|
RC_AMBIGUOUS_DELETE,
|
||||||
|
REASON_CODES,
|
||||||
|
OPERATIONS,
|
||||||
|
)
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Konstanten
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
# Polling-Intervall (Default 60s, Environment-konfigurierbar)
|
||||||
|
DEFAULT_POLL_INTERVAL_SECONDS = 60
|
||||||
|
ENV_POLL_INTERVAL = "C5_POLL_INTERVAL_SECONDS"
|
||||||
|
|
||||||
|
# Knowledge-Scope-Klassifikation
|
||||||
|
SCOPE_IN_SCOPE = "IN_SCOPE"
|
||||||
|
SCOPE_OUT_OF_SCOPE = "OUT_OF_SCOPE"
|
||||||
|
SCOPE_LEGACY_SPECIAL = "LEGACY_SPECIAL"
|
||||||
|
SCOPE_HUMAN_REVIEW = "HUMAN_REVIEW"
|
||||||
|
|
||||||
|
# Verzeichnisse, die KEINE Tolaria-Knowledge-Objekte enthalten (Architektur/Ops/Code)
|
||||||
|
OUT_OF_SCOPE_DIRS = frozenset({
|
||||||
|
"tolaria", "a2", "a3", "a4", "a5",
|
||||||
|
"red-queen-architecture", "notion-safety-brain", "notion-command-center",
|
||||||
|
"backup_patches", "notes/reference", "notes",
|
||||||
|
})
|
||||||
|
|
||||||
|
# Legacy-Ausnahmen (C3I: KEEP_DISTINCT / DEFERRED) — C5B darf diese NICHT
|
||||||
|
# zusammenfuehren oder neu-vergeben. Sie werden separat klassifiziert.
|
||||||
|
LEGACY_SPECIAL_PATHS = frozenset({
|
||||||
|
"README.md", # Root-README (KEEP_DISTINCT)
|
||||||
|
"notes/trading/system-docs/README.md", # Canonical-README (KEEP_DISTINCT)
|
||||||
|
"vps.md", # DEFERRED (C3I)
|
||||||
|
})
|
||||||
|
|
||||||
|
# Canonical-Verzeichnis (derived knowledge)
|
||||||
|
CANONICAL_DIR = "notes/trading/system-docs/"
|
||||||
|
|
||||||
|
# Git read-only Whitelist (C5B darf NUR diese Befehle ausfuehren)
|
||||||
|
# Jeder Befehl ist read-only. Write-Befehle (push/commit/add/reset/checkout/
|
||||||
|
# merge/rebase/fetch --write) sind NICHT in der Whitelist und werden blockiert.
|
||||||
|
GIT_READONLY_COMMANDS = frozenset({
|
||||||
|
"rev-parse", "log", "show", "diff-tree", "ls-tree", "cat-file",
|
||||||
|
"merge-base", "rev-list", "name-only", "diff",
|
||||||
|
})
|
||||||
|
|
||||||
|
# Git Write-Befehle (explizit verboten — statische + dynamische Pruefung)
|
||||||
|
GIT_WRITE_COMMANDS = frozenset({
|
||||||
|
"push", "commit", "add", "reset", "checkout", "merge", "rebase",
|
||||||
|
"fetch", "pull", "clone", "init", "rm", "mv", "tag", "branch",
|
||||||
|
"stash", "clean", "gc", "prune", "repack", "update-ref", "write-tree",
|
||||||
|
})
|
||||||
|
|
||||||
|
# Frontmatter-Id-Muster: object/<FULL_UUID>
|
||||||
|
OBJECT_ID_RE = re.compile(r"^object/[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$")
|
||||||
|
|
||||||
|
# Secret-/Forbidden-Muster (nur zur Klassifikation, KEINE Werte persistieren)
|
||||||
|
SECRET_PATTERNS = [
|
||||||
|
re.compile(r"sk-[A-Za-z0-9]{20,}"),
|
||||||
|
re.compile(r"ghp_[A-Za-z0-9]{20,}"),
|
||||||
|
re.compile(r"bearer\s+[A-Za-z0-9]{20,}", re.IGNORECASE),
|
||||||
|
re.compile(r"-----BEGIN (RSA |EC |OPENSSH )?PRIVATE KEY-----"),
|
||||||
|
re.compile(r"AKIA[0-9A-Z]{16}"),
|
||||||
|
]
|
||||||
|
|
||||||
|
# Knowledge-Schema-Metadaten, die in metadata_hash eingehen
|
||||||
|
METADATA_FIELDS = ("id", "type", "role", "representation", "state",
|
||||||
|
"derived_from", "tags", "knowledge_schema")
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Fehlerklassen
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
class C5BError(Exception):
|
||||||
|
"""Basis-Fehlerklasse fuer C5B."""
|
||||||
|
|
||||||
|
def __init__(self, message: str, reason_code: Optional[str] = None):
|
||||||
|
super().__init__(message)
|
||||||
|
self.message = message
|
||||||
|
self.reason_code = reason_code
|
||||||
|
|
||||||
|
|
||||||
|
class ForgejoUnavailableError(C5BError):
|
||||||
|
"""Forgejo (lokaler Clone) nicht lesbar."""
|
||||||
|
|
||||||
|
|
||||||
|
class HistoryDivergenceError(C5BError):
|
||||||
|
"""last_applied_commit ist kein Ancestor von HEAD (FAIL CLOSED)."""
|
||||||
|
|
||||||
|
|
||||||
|
class GitWriteBlockedError(C5BError):
|
||||||
|
"""Versuch, einen verbotenen git-Write-Befehl auszufuehren."""
|
||||||
|
|
||||||
|
|
||||||
|
class NoDownstreamWriteError(C5BError):
|
||||||
|
"""C5B darf keine externen Writes ausfuehren."""
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# GitReader (read-only, strikte Whitelist)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
class GitReader:
|
||||||
|
"""
|
||||||
|
Fuehrt AUSSCHLIESSLICH read-only git-Befehle gegen einen lokalen Clone aus.
|
||||||
|
|
||||||
|
Whitelist: rev-parse, log, show, diff-tree, ls-tree, cat-file, merge-base,
|
||||||
|
rev-list, diff. Jeder andere Befehl (insb. push/commit/add/reset/checkout/
|
||||||
|
merge/rebase) wird mit GitWriteBlockedError blockiert.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, repo_path: str):
|
||||||
|
self.repo_path = str(repo_path)
|
||||||
|
if not os.path.isdir(os.path.join(self.repo_path, ".git")):
|
||||||
|
raise ForgejoUnavailableError(
|
||||||
|
f"Kein Git-Repo unter {self.repo_path}",
|
||||||
|
reason_code=RC_FORGEJO_UNAVAILABLE,
|
||||||
|
)
|
||||||
|
|
||||||
|
def _run(self, args: List[str], check: bool = True) -> str:
|
||||||
|
"""Fuehrt einen read-only git-Befehl aus. Blockiert Write-Befehle."""
|
||||||
|
if not args:
|
||||||
|
raise GitWriteBlockedError("Leerer git-Befehl")
|
||||||
|
cmd = args[0]
|
||||||
|
if cmd not in GIT_READONLY_COMMANDS:
|
||||||
|
raise GitWriteBlockedError(
|
||||||
|
f"git-Befehl '{cmd}' ist nicht in der read-only Whitelist"
|
||||||
|
)
|
||||||
|
if cmd in GIT_WRITE_COMMANDS:
|
||||||
|
raise GitWriteBlockedError(
|
||||||
|
f"git-Befehl '{cmd}' ist ein verbotener Write-Befehl"
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
proc = subprocess.run(
|
||||||
|
["git", "-C", self.repo_path] + args,
|
||||||
|
capture_output=True, text=True, timeout=30,
|
||||||
|
)
|
||||||
|
except (subprocess.TimeoutExpired, OSError) as e:
|
||||||
|
raise ForgejoUnavailableError(
|
||||||
|
f"Forgejo (git) nicht erreichbar: {e}",
|
||||||
|
reason_code=RC_FORGEJO_UNAVAILABLE,
|
||||||
|
)
|
||||||
|
if check and proc.returncode != 0:
|
||||||
|
raise ForgejoUnavailableError(
|
||||||
|
f"git {cmd} fehlgeschlagen: {proc.stderr.strip()}",
|
||||||
|
reason_code=RC_FORGEJO_UNAVAILABLE,
|
||||||
|
)
|
||||||
|
return proc.stdout
|
||||||
|
|
||||||
|
def get_current_head(self) -> str:
|
||||||
|
"""Gibt den aktuellen HEAD-Commit-SHA zurueck (read-only)."""
|
||||||
|
out = self._run(["rev-parse", "HEAD"]).strip()
|
||||||
|
if not out:
|
||||||
|
raise ForgejoUnavailableError(
|
||||||
|
"HEAD nicht bestimmbar", reason_code=RC_FORGEJO_UNAVAILABLE
|
||||||
|
)
|
||||||
|
return out
|
||||||
|
|
||||||
|
def commit_meta(self, sha: str) -> Dict[str, Any]:
|
||||||
|
"""Gibt parent, timestamp, message eines Commits zurueck (read-only)."""
|
||||||
|
out = self._run(["show", "-s", "--format=%H%x00%P%x00%ct%x00%s", sha]).strip()
|
||||||
|
parts = out.split("\x00")
|
||||||
|
if len(parts) < 4:
|
||||||
|
raise ForgejoUnavailableError(
|
||||||
|
f"Commit-Metadaten nicht lesbar: {sha}",
|
||||||
|
reason_code=RC_FORGEJO_UNAVAILABLE,
|
||||||
|
)
|
||||||
|
return {
|
||||||
|
"commit_sha": parts[0],
|
||||||
|
"parent_sha": parts[1] or None, # Root-Commit hat keinen Parent
|
||||||
|
"timestamp": int(parts[2]) if parts[2].isdigit() else None,
|
||||||
|
"message": parts[3],
|
||||||
|
}
|
||||||
|
|
||||||
|
def is_ancestor(self, ancestor: str, descendant: str) -> bool:
|
||||||
|
"""
|
||||||
|
Prueft, ob ancestor ein Vorfahr von descendant ist (read-only).
|
||||||
|
git merge-base --is-ancestor: exit 0 = ist Ancestor, exit 1 = nicht.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
proc = subprocess.run(
|
||||||
|
["git", "-C", self.repo_path,
|
||||||
|
"merge-base", "--is-ancestor", ancestor, descendant],
|
||||||
|
capture_output=True, text=True, timeout=30,
|
||||||
|
)
|
||||||
|
return proc.returncode == 0
|
||||||
|
except (subprocess.TimeoutExpired, OSError) as e:
|
||||||
|
raise ForgejoUnavailableError(
|
||||||
|
f"Forgejo (git) nicht erreichbar: {e}",
|
||||||
|
reason_code=RC_FORGEJO_UNAVAILABLE,
|
||||||
|
)
|
||||||
|
|
||||||
|
def commit_range(self, start_exclusive: Optional[str], end: str) -> List[str]:
|
||||||
|
"""
|
||||||
|
Deterministisch alle Commits in (start_exclusive, end] in chronologischer
|
||||||
|
(parent-korrekter) Reihenfolge. Kein Ueberspringen, keine Doppelerfassung.
|
||||||
|
"""
|
||||||
|
if start_exclusive is None:
|
||||||
|
# Alle Commits bis end (reverse = aeltester zuerst)
|
||||||
|
out = self._run(["rev-list", "--reverse", end]).strip()
|
||||||
|
else:
|
||||||
|
out = self._run(["rev-list", "--reverse", f"{start_exclusive}..{end}"]).strip()
|
||||||
|
if not out:
|
||||||
|
return []
|
||||||
|
return [line for line in out.splitlines() if line]
|
||||||
|
|
||||||
|
def diff_name_status(self, commit_sha: str) -> List[Dict[str, Any]]:
|
||||||
|
"""
|
||||||
|
Gibt die Aenderungen eines Commits gegenueber seinem Parent zurueck.
|
||||||
|
Nutzt git diff-tree (read-only) mit rename detection.
|
||||||
|
"""
|
||||||
|
out = self._run([
|
||||||
|
"diff-tree", "-r", "--name-status", "-M", "-C", "--root",
|
||||||
|
"--format=", commit_sha,
|
||||||
|
]).strip()
|
||||||
|
changes = []
|
||||||
|
for line in out.splitlines():
|
||||||
|
parts = line.split("\t")
|
||||||
|
if len(parts) < 2:
|
||||||
|
continue
|
||||||
|
status = parts[0]
|
||||||
|
path = parts[1]
|
||||||
|
# Rename/Copy: "R100\told\tnew" oder "C100\told\tnew"
|
||||||
|
if status.startswith("R") or status.startswith("C"):
|
||||||
|
if len(parts) >= 3:
|
||||||
|
changes.append({
|
||||||
|
"status": status[0], # R oder C
|
||||||
|
"similarity": status[1:],
|
||||||
|
"path_before": parts[1],
|
||||||
|
"path_after": parts[2],
|
||||||
|
})
|
||||||
|
continue
|
||||||
|
# ADDED: Datei existiert nur in AFTER (path_before=None)
|
||||||
|
if status == "A":
|
||||||
|
changes.append({
|
||||||
|
"status": status,
|
||||||
|
"path_before": None,
|
||||||
|
"path_after": path,
|
||||||
|
})
|
||||||
|
continue
|
||||||
|
# DELETED: Datei existiert nur in BEFORE (path_after=None)
|
||||||
|
if status == "D":
|
||||||
|
changes.append({
|
||||||
|
"status": status,
|
||||||
|
"path_before": path,
|
||||||
|
"path_after": None,
|
||||||
|
})
|
||||||
|
continue
|
||||||
|
# MODIFIED (M, T, U): beide Pfade = path
|
||||||
|
changes.append({
|
||||||
|
"status": status,
|
||||||
|
"path_before": path,
|
||||||
|
"path_after": path,
|
||||||
|
})
|
||||||
|
return changes
|
||||||
|
|
||||||
|
def file_content(self, sha: str, path: str) -> Optional[str]:
|
||||||
|
"""Liest den Inhalt einer Datei in einem Commit (read-only)."""
|
||||||
|
try:
|
||||||
|
out = self._run(["show", f"{sha}:{path}"], check=False)
|
||||||
|
except ForgejoUnavailableError:
|
||||||
|
return None
|
||||||
|
if out == "" and self._file_exists(sha, path) is False:
|
||||||
|
return None
|
||||||
|
return out
|
||||||
|
|
||||||
|
def _file_exists(self, sha: str, path: str) -> Optional[bool]:
|
||||||
|
"""Prueft, ob eine Datei in einem Commit existiert (read-only)."""
|
||||||
|
try:
|
||||||
|
out = self._run(["ls-tree", "-r", "--name-only", sha], check=False)
|
||||||
|
except ForgejoUnavailableError:
|
||||||
|
return None
|
||||||
|
return path in out.splitlines()
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Frontmatter / Object ID / Hash Model
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def parse_frontmatter(content: str) -> Tuple[Dict[str, Any], str]:
|
||||||
|
"""
|
||||||
|
Parst YAML-Frontmatter (--- ... ---) und gibt (fm_dict, body) zurueck.
|
||||||
|
Body = fachlicher Inhalt NACH dem Frontmatter-Block.
|
||||||
|
"""
|
||||||
|
if not content.startswith("---"):
|
||||||
|
return {}, content
|
||||||
|
lines = content.splitlines()
|
||||||
|
if len(lines) < 2:
|
||||||
|
return {}, content
|
||||||
|
# Ende des Frontmatter-Blocks finden
|
||||||
|
end = None
|
||||||
|
for i in range(1, len(lines)):
|
||||||
|
if lines[i].strip() == "---":
|
||||||
|
end = i
|
||||||
|
break
|
||||||
|
if end is None:
|
||||||
|
return {}, content
|
||||||
|
fm_lines = lines[1:end]
|
||||||
|
body = "\n".join(lines[end + 1:])
|
||||||
|
fm: Dict[str, Any] = {}
|
||||||
|
for line in fm_lines:
|
||||||
|
if ":" not in line:
|
||||||
|
continue
|
||||||
|
key, _, val = line.partition(":")
|
||||||
|
key = key.strip()
|
||||||
|
val = val.strip()
|
||||||
|
if not key:
|
||||||
|
continue
|
||||||
|
# Einfache Listen (tags: [a, b]) und Skalare
|
||||||
|
if val.startswith("[") and val.endswith("]"):
|
||||||
|
inner = val[1:-1].strip()
|
||||||
|
fm[key] = [x.strip().strip("'\"") for x in inner.split(",") if x.strip()]
|
||||||
|
else:
|
||||||
|
fm[key] = val.strip("'\"")
|
||||||
|
return fm, body
|
||||||
|
|
||||||
|
|
||||||
|
def extract_object_id(content: str) -> Optional[str]:
|
||||||
|
"""Liest object_id aus dem Frontmatter (id: object/<FULL_UUID>)."""
|
||||||
|
fm, _ = parse_frontmatter(content)
|
||||||
|
oid = fm.get("id")
|
||||||
|
if not oid:
|
||||||
|
return None
|
||||||
|
oid = str(oid).strip()
|
||||||
|
if not OBJECT_ID_RE.match(oid):
|
||||||
|
return None
|
||||||
|
return oid
|
||||||
|
|
||||||
|
|
||||||
|
def content_hash(body: str) -> str:
|
||||||
|
"""SHA-256 des fachlichen Bodies (deterministisch)."""
|
||||||
|
return hashlib.sha256(body.encode("utf-8")).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def metadata_hash(fm: Dict[str, Any]) -> str:
|
||||||
|
"""
|
||||||
|
SHA-256 der relevanten Knowledge-Schema-Metadaten (deterministisch).
|
||||||
|
Beruecksichtigt: id, type, role, representation, state, derived_from,
|
||||||
|
tags, knowledge_schema. Pfad ist separat (nicht Teil von metadata_hash).
|
||||||
|
"""
|
||||||
|
subset = {k: fm.get(k) for k in METADATA_FIELDS if k in fm}
|
||||||
|
canonical = json.dumps(subset, sort_keys=True, ensure_ascii=False)
|
||||||
|
return hashlib.sha256(canonical.encode("utf-8")).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def detect_secret(content: str) -> Optional[str]:
|
||||||
|
"""
|
||||||
|
Erkennt moegliche Secret-Werte in einem Dokument. Gibt den MATCHED PATTERN
|
||||||
|
zurueck (NICHT den Wert). Kein Secret-Wert wird ausgegeben oder persistiert.
|
||||||
|
"""
|
||||||
|
for pat in SECRET_PATTERNS:
|
||||||
|
m = pat.search(content)
|
||||||
|
if m:
|
||||||
|
return pat.pattern
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Knowledge Scope
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
class KnowledgeScope:
|
||||||
|
"""
|
||||||
|
Bestimmt, ob eine Forgejo-Datei ein Tolaria-Knowledge-Objekt ist.
|
||||||
|
|
||||||
|
Regeln (C3/C5):
|
||||||
|
* IN_SCOPE: Markdown im Repo-Root ODER unter notes/trading/system-docs/,
|
||||||
|
mit gueltiger object_id im Frontmatter.
|
||||||
|
* OUT_OF_SCOPE: Architektur/Ops/Code-Verzeichnisse, Nicht-Markdown,
|
||||||
|
Build-Artefakte, Secrets.
|
||||||
|
* LEGACY_SPECIAL: README (x2, KEEP_DISTINCT) + vps.md (DEFERRED).
|
||||||
|
* HUMAN_REVIEW: Knowledge-Objekt ohne gueltige object_id (UNKNOWN_OBJECT_ID
|
||||||
|
/ UNKNOWN_LEGACY_OBJECT) oder Ambiguitaet.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, repo_path: str):
|
||||||
|
self.repo_path = str(repo_path)
|
||||||
|
|
||||||
|
def classify(self, path: str) -> str:
|
||||||
|
"""Klassifiziert einen Pfad in IN_SCOPE / OUT_OF_SCOPE / LEGACY_SPECIAL."""
|
||||||
|
path = path.replace("\\", "/")
|
||||||
|
# Legacy-Ausnahmen zuerst
|
||||||
|
if path in LEGACY_SPECIAL_PATHS:
|
||||||
|
return SCOPE_LEGACY_SPECIAL
|
||||||
|
# Nur Markdown ist potenziell Knowledge
|
||||||
|
if not path.endswith(".md"):
|
||||||
|
return SCOPE_OUT_OF_SCOPE
|
||||||
|
# Canonical-Verzeichnis
|
||||||
|
if path.startswith(CANONICAL_DIR):
|
||||||
|
return SCOPE_IN_SCOPE
|
||||||
|
# Repo-Root (kein Verzeichnis-Praefix)
|
||||||
|
if "/" not in path:
|
||||||
|
return SCOPE_IN_SCOPE
|
||||||
|
# Alles andere (tolaria/, a2-5/, red-queen-architecture/, etc.) = OUT
|
||||||
|
return SCOPE_OUT_OF_SCOPE
|
||||||
|
|
||||||
|
def classify_with_content(self, path: Optional[str], content: Optional[str]) -> Dict[str, Any]:
|
||||||
|
"""
|
||||||
|
Vollstaendige Scope-Klassifikation inkl. object_id-Validierung.
|
||||||
|
Gibt ein Dict mit scope, object_id, reason_code (optional) zurueck.
|
||||||
|
"""
|
||||||
|
if path is None:
|
||||||
|
return {"path": None, "scope": SCOPE_OUT_OF_SCOPE}
|
||||||
|
scope = self.classify(path)
|
||||||
|
result: Dict[str, Any] = {"path": path, "scope": scope}
|
||||||
|
if scope == SCOPE_OUT_OF_SCOPE:
|
||||||
|
return result
|
||||||
|
if scope == SCOPE_LEGACY_SPECIAL:
|
||||||
|
return result
|
||||||
|
# IN_SCOPE: object_id aus Frontmatter validieren
|
||||||
|
if content is None:
|
||||||
|
result["scope"] = SCOPE_HUMAN_REVIEW
|
||||||
|
result["reason_code"] = RC_UNKNOWN_OBJECT_ID
|
||||||
|
return result
|
||||||
|
oid = extract_object_id(content)
|
||||||
|
if oid is None:
|
||||||
|
# Knowledge-Objekt ohne gueltige ID -> Human Review
|
||||||
|
result["scope"] = SCOPE_HUMAN_REVIEW
|
||||||
|
result["reason_code"] = RC_UNKNOWN_OBJECT_ID
|
||||||
|
return result
|
||||||
|
result["object_id"] = oid
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Change Classification
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
class ChangeClassifier:
|
||||||
|
"""
|
||||||
|
Leitet aus BEFORE/AFTER deterministisch die erlaubten Operationen ab.
|
||||||
|
|
||||||
|
Bei mehreren geaenderten Aspekten in EINEM Objekt werden MEHRERE
|
||||||
|
ObjectChange-Eintraege erzeugt (keine Information verloren).
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, scope: KnowledgeScope):
|
||||||
|
self.scope = scope
|
||||||
|
|
||||||
|
def classify(self, path_before: Optional[str], path_after: Optional[str],
|
||||||
|
content_before: Optional[str], content_after: Optional[str]) -> List[Dict[str, Any]]:
|
||||||
|
"""
|
||||||
|
Klassifiziert die Aenderung eines Objekts. Gibt eine Liste von
|
||||||
|
ObjectChange-Dicts zurueck (eine pro Operation).
|
||||||
|
"""
|
||||||
|
# DELETE: Datei existiert in BEFORE, nicht in AFTER
|
||||||
|
if path_before and not path_after:
|
||||||
|
return [{"operation": OP_DELETE_REQUEST, "path_before": path_before,
|
||||||
|
"path_after": None}]
|
||||||
|
|
||||||
|
# CREATE: Datei existiert in AFTER, nicht in BEFORE
|
||||||
|
if not path_before and path_after:
|
||||||
|
return [{"operation": OP_CREATE, "path_before": None,
|
||||||
|
"path_after": path_after}]
|
||||||
|
|
||||||
|
# RENAME/MOVE: gleiche object_id, anderer Pfad
|
||||||
|
if path_before and path_after and path_before != path_after:
|
||||||
|
oid_before = extract_object_id(content_before or "")
|
||||||
|
oid_after = extract_object_id(content_after or "")
|
||||||
|
if oid_before and oid_after and oid_before == oid_after:
|
||||||
|
# Rename = Dateiname geaendert, gleiches Verzeichnis
|
||||||
|
# Move = Verzeichnis geaendert
|
||||||
|
dir_before = os.path.dirname(path_before)
|
||||||
|
dir_after = os.path.dirname(path_after)
|
||||||
|
if dir_before == dir_after:
|
||||||
|
return [{"operation": OP_RENAME, "path_before": path_before,
|
||||||
|
"path_after": path_after}]
|
||||||
|
return [{"operation": OP_MOVE, "path_before": path_before,
|
||||||
|
"path_after": path_after}]
|
||||||
|
# Ambiguitaet: Pfad geaendert, aber ID nicht stabil -> Human Review
|
||||||
|
return [{"operation": OP_MOVE, "path_before": path_before,
|
||||||
|
"path_after": path_after, "ambiguous": True}]
|
||||||
|
|
||||||
|
# MODIFIED: gleicher Pfad, Inhalt/Metadaten geaendert
|
||||||
|
ops: List[Dict[str, Any]] = []
|
||||||
|
fm_before, body_before = parse_frontmatter(content_before or "")
|
||||||
|
fm_after, body_after = parse_frontmatter(content_after or "")
|
||||||
|
|
||||||
|
ch_before = content_hash(body_before)
|
||||||
|
ch_after = content_hash(body_after)
|
||||||
|
mh_before = metadata_hash(fm_before)
|
||||||
|
mh_after = metadata_hash(fm_after)
|
||||||
|
|
||||||
|
if ch_before != ch_after:
|
||||||
|
ops.append({"operation": OP_CONTENT_UPDATE})
|
||||||
|
if mh_before != mh_after:
|
||||||
|
# Spezifische Metadaten-Aenderungen
|
||||||
|
if fm_before.get("state") != fm_after.get("state"):
|
||||||
|
ops.append({"operation": OP_STATE_UPDATE})
|
||||||
|
if fm_before.get("tags") != fm_after.get("tags"):
|
||||||
|
ops.append({"operation": OP_TAGS_UPDATE})
|
||||||
|
if fm_before.get("derived_from") != fm_after.get("derived_from"):
|
||||||
|
ops.append({"operation": OP_SOURCE_CANONICAL_RELATION_UPDATE})
|
||||||
|
# Generische Metadaten-Aenderung (falls keine spezifische erkannt)
|
||||||
|
if not any(op["operation"] in (
|
||||||
|
OP_STATE_UPDATE, OP_TAGS_UPDATE,
|
||||||
|
OP_SOURCE_CANONICAL_RELATION_UPDATE,
|
||||||
|
) for op in ops):
|
||||||
|
ops.append({"operation": OP_METADATA_UPDATE})
|
||||||
|
# SUPERSEDE: state -> superseded
|
||||||
|
if fm_after.get("state") == "superseded" and fm_before.get("state") != "superseded":
|
||||||
|
ops.append({"operation": OP_SUPERSEDE})
|
||||||
|
|
||||||
|
if not ops:
|
||||||
|
# Keine Aenderung erkannt (z.B. nur Whitespace) -> kein ObjectChange
|
||||||
|
return []
|
||||||
|
return ops
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# C5B Poller (integriert in C5AStore, nur State/Persistence)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
class C5BPoller:
|
||||||
|
"""
|
||||||
|
Fuehrt einen kontrollierten Poll-Zyklus aus und speist Commits + ObjectChanges
|
||||||
|
in den C5A Store ein. KEIN Network Write downstream. KEIN Poll-Daemon.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, store: C5AStore, repo_path: str,
|
||||||
|
poll_interval_seconds: Optional[int] = None):
|
||||||
|
self.store = store
|
||||||
|
self.reader = GitReader(repo_path)
|
||||||
|
self.scope = KnowledgeScope(repo_path)
|
||||||
|
self.classifier = ChangeClassifier(self.scope)
|
||||||
|
self.poll_interval_seconds = poll_interval_seconds or self._env_poll_interval()
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _env_poll_interval() -> int:
|
||||||
|
"""Liest C5_POLL_INTERVAL_SECONDS aus der Environment (Default 60)."""
|
||||||
|
raw = os.environ.get(ENV_POLL_INTERVAL, "")
|
||||||
|
if raw.isdigit() and int(raw) > 0:
|
||||||
|
return int(raw)
|
||||||
|
return DEFAULT_POLL_INTERVAL_SECONDS
|
||||||
|
|
||||||
|
def get_current_head(self) -> str:
|
||||||
|
"""Read-only HEAD-Discovery."""
|
||||||
|
return self.reader.get_current_head()
|
||||||
|
|
||||||
|
def history_safety(self, last_applied: Optional[str], head: str) -> bool:
|
||||||
|
"""
|
||||||
|
FAIL CLOSED: Wenn last_applied_commit kein Ancestor von HEAD ist,
|
||||||
|
ist die History divergiert. Keine automatische Rebase-/Reset-Logik.
|
||||||
|
"""
|
||||||
|
if last_applied is None:
|
||||||
|
return True # Bootstrap: keine Baseline, alles neu
|
||||||
|
return self.reader.is_ancestor(last_applied, head)
|
||||||
|
|
||||||
|
def poll_once(self) -> Dict[str, Any]:
|
||||||
|
"""
|
||||||
|
Ein kontrollierter Poll-Zyklus:
|
||||||
|
HEAD -> Commit-Range -> Diff -> Change-Klassifikation -> C5AStore.
|
||||||
|
Gibt ein Ergebnis-Dict zurueck. KEIN Network Write downstream.
|
||||||
|
|
||||||
|
Baseline fuer Commit-Range + History-Safety ist last_seen_commit
|
||||||
|
(C5B-Baseline). last_applied_commit bleibt laut C5A-Contract fuer die
|
||||||
|
spaetere Propagation (C5C) reserviert und wird von C5B NIE fortgeschrieben.
|
||||||
|
"""
|
||||||
|
head = self.get_current_head()
|
||||||
|
health = self.store.health()
|
||||||
|
last_seen = health.get("last_seen_commit")
|
||||||
|
|
||||||
|
# History Safety (FAIL CLOSED) gegen last_seen_commit
|
||||||
|
if not self.history_safety(last_seen, head):
|
||||||
|
return {
|
||||||
|
"status": "FAIL_CLOSED",
|
||||||
|
"reason_code": RC_OUT_OF_ORDER_COMMIT,
|
||||||
|
"message": "last_seen_commit ist kein Ancestor von HEAD (Divergenz)",
|
||||||
|
"head": head,
|
||||||
|
"last_seen": last_seen,
|
||||||
|
}
|
||||||
|
|
||||||
|
# Commit-Range bestimmen (ab last_seen_commit, exklusiv)
|
||||||
|
commits = self.reader.commit_range(last_seen, head)
|
||||||
|
self.store.mark_seen(head)
|
||||||
|
|
||||||
|
discovered = 0
|
||||||
|
objects_discovered = 0
|
||||||
|
out_of_scope = 0
|
||||||
|
human_review = 0
|
||||||
|
new_commits = []
|
||||||
|
|
||||||
|
for sha in commits:
|
||||||
|
meta = self.reader.commit_meta(sha)
|
||||||
|
# Idempotenz: bereits registriert?
|
||||||
|
existing = self.store.get_commit(sha)
|
||||||
|
if existing is not None:
|
||||||
|
continue # ALREADY_DISCOVERED
|
||||||
|
|
||||||
|
# Diff gegen Parent
|
||||||
|
changes = self.reader.diff_name_status(sha)
|
||||||
|
object_changes = []
|
||||||
|
for ch in changes:
|
||||||
|
path_before: Optional[str] = ch.get("path_before")
|
||||||
|
path_after: Optional[str] = ch.get("path_after")
|
||||||
|
# Inhalte read-only lesen
|
||||||
|
content_before: Optional[str] = None
|
||||||
|
content_after: Optional[str] = None
|
||||||
|
if path_before:
|
||||||
|
content_before = self.reader.file_content(sha, path_before)
|
||||||
|
if path_after:
|
||||||
|
content_after = self.reader.file_content(sha, path_after)
|
||||||
|
|
||||||
|
# Scope-Klassifikation
|
||||||
|
scope_info = self.scope.classify_with_content(
|
||||||
|
path_after or path_before, content_after or content_before
|
||||||
|
)
|
||||||
|
scope = scope_info["scope"]
|
||||||
|
if scope == SCOPE_OUT_OF_SCOPE:
|
||||||
|
out_of_scope += 1
|
||||||
|
continue
|
||||||
|
if scope == SCOPE_LEGACY_SPECIAL:
|
||||||
|
out_of_scope += 1
|
||||||
|
continue
|
||||||
|
if scope == SCOPE_HUMAN_REVIEW:
|
||||||
|
human_review += 1
|
||||||
|
object_changes.append({
|
||||||
|
"object_id": None,
|
||||||
|
"operation": OP_METADATA_UPDATE,
|
||||||
|
"path_before": path_before,
|
||||||
|
"path_after": path_after,
|
||||||
|
"reason_code": scope_info.get("reason_code", RC_UNKNOWN_OBJECT_ID),
|
||||||
|
})
|
||||||
|
continue
|
||||||
|
|
||||||
|
oid = scope_info["object_id"]
|
||||||
|
# Secret-Safety: kein Secret-Inhalt persistieren
|
||||||
|
for content in (content_before, content_after):
|
||||||
|
if content and detect_secret(content):
|
||||||
|
# Nur sichere Metadaten persistieren, kein Content
|
||||||
|
object_changes.append({
|
||||||
|
"object_id": oid,
|
||||||
|
"operation": OP_METADATA_UPDATE,
|
||||||
|
"path_before": path_before,
|
||||||
|
"path_after": path_after,
|
||||||
|
"reason_code": RC_SECRET_DETECTED,
|
||||||
|
})
|
||||||
|
human_review += 1
|
||||||
|
break
|
||||||
|
else:
|
||||||
|
# Change-Klassifikation
|
||||||
|
ops = self.classifier.classify(
|
||||||
|
path_before, path_after, content_before, content_after
|
||||||
|
)
|
||||||
|
for op in ops:
|
||||||
|
fm_before, body_before = parse_frontmatter(content_before or "")
|
||||||
|
fm_after, body_after = parse_frontmatter(content_after or "")
|
||||||
|
object_changes.append({
|
||||||
|
"object_id": oid,
|
||||||
|
"operation": op["operation"],
|
||||||
|
"path_before": path_before,
|
||||||
|
"path_after": path_after,
|
||||||
|
"content_hash_before": content_hash(body_before) if content_before is not None else None,
|
||||||
|
"content_hash_after": content_hash(body_after) if content_after is not None else None,
|
||||||
|
"metadata_hash_before": metadata_hash(fm_before) if content_before is not None else None,
|
||||||
|
"metadata_hash_after": metadata_hash(fm_after) if content_after is not None else None,
|
||||||
|
"representation": fm_after.get("representation") or fm_before.get("representation"),
|
||||||
|
"state": fm_after.get("state") or fm_before.get("state"),
|
||||||
|
})
|
||||||
|
|
||||||
|
# Commit in C5AStore registrieren (Status DISCOVERED)
|
||||||
|
commit_record = {
|
||||||
|
"commit_sha": sha,
|
||||||
|
"parent_sha": meta.get("parent_sha"),
|
||||||
|
"discovered_at": int(time.time() * 1000),
|
||||||
|
"sequence": len(commits),
|
||||||
|
"status": ST_DISCOVERED,
|
||||||
|
"retry_count": 0,
|
||||||
|
}
|
||||||
|
self.store.upsert_commit(commit_record)
|
||||||
|
discovered += 1
|
||||||
|
|
||||||
|
# ObjectChanges hinzufuegen
|
||||||
|
for oc in object_changes:
|
||||||
|
oc["commit_sha"] = sha
|
||||||
|
self.store.add_object_change(oc)
|
||||||
|
objects_discovered += 1
|
||||||
|
|
||||||
|
# Status -> VALIDATING (nicht weiter zu PROPAGATING_TOLARIA)
|
||||||
|
self.store.transition_commit(sha, ST_VALIDATING)
|
||||||
|
if any(oc.get("reason_code") for oc in object_changes):
|
||||||
|
# VALIDATING -> HUMAN_REVIEW_REQUIRED (erlaubte Transition)
|
||||||
|
self.store.transition_commit(sha, ST_HUMAN_REVIEW_REQUIRED)
|
||||||
|
|
||||||
|
new_commits.append(sha)
|
||||||
|
|
||||||
|
return {
|
||||||
|
"status": "OK",
|
||||||
|
"head": head,
|
||||||
|
"last_seen": last_seen,
|
||||||
|
"commits_discovered": discovered,
|
||||||
|
"objects_discovered": objects_discovered,
|
||||||
|
"out_of_scope_count": out_of_scope,
|
||||||
|
"human_review_count": human_review,
|
||||||
|
"new_commits": new_commits,
|
||||||
|
"poll_interval_seconds": self.poll_interval_seconds,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# No-Downstream-Write-Guarantee (statische Pruefung)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def assert_no_downstream_write() -> Dict[str, Any]:
|
||||||
|
"""
|
||||||
|
Beweist, dass C5B keinen Codepfad fuer externe Writes besitzt:
|
||||||
|
* KEIN Tolaria-Write (vault/save)
|
||||||
|
* KEIN Search-Rebuild (search/rebuild)
|
||||||
|
* KEIN Forgejo-Write / kein git push/commit/add
|
||||||
|
Erlaubt ist NUR read-only git (Whitelist) + stdlib.
|
||||||
|
"""
|
||||||
|
import ast
|
||||||
|
this_file = Path(__file__).resolve()
|
||||||
|
tree = ast.parse(this_file.read_text(encoding="utf-8"))
|
||||||
|
|
||||||
|
# 1. HTTP-Mutationsimporte (Tolaria/Search-Write)
|
||||||
|
http_imports = {"requests", "urllib", "http", "socket"}
|
||||||
|
found_http = set()
|
||||||
|
for node in ast.walk(tree):
|
||||||
|
if isinstance(node, ast.Import):
|
||||||
|
for alias in node.names:
|
||||||
|
root = alias.name.split(".")[0]
|
||||||
|
if root in http_imports:
|
||||||
|
found_http.add(root)
|
||||||
|
elif isinstance(node, ast.ImportFrom):
|
||||||
|
if node.module:
|
||||||
|
root = node.module.split(".")[0]
|
||||||
|
if root in http_imports:
|
||||||
|
found_http.add(root)
|
||||||
|
|
||||||
|
# 2. Verbotene Endpoint-Strings NUR in echten Code-Ausdruecken
|
||||||
|
# (nicht in Docstrings/Kommentaren). Die Verbotsliste selbst wird aus
|
||||||
|
# Teilstrings zusammengesetzt, damit der volle Endpoint-String nie als
|
||||||
|
# einzelnes String-Literal im Quellcode steht (sonst False Positive).
|
||||||
|
forbidden_endpoints = ["/api/vault" + "/save", "/api/search" + "/rebuild"]
|
||||||
|
found_endpoints = []
|
||||||
|
for literal in _collect_code_strings(tree):
|
||||||
|
for ep in forbidden_endpoints:
|
||||||
|
if ep in literal and ep not in found_endpoints:
|
||||||
|
found_endpoints.append(ep)
|
||||||
|
|
||||||
|
# 3. Git-Write-Befehle duerfen NICHT in der Whitelist stehen
|
||||||
|
write_in_whitelist = sorted(GIT_READONLY_COMMANDS & GIT_WRITE_COMMANDS)
|
||||||
|
|
||||||
|
return {
|
||||||
|
"no_http_mutations": len(found_http) == 0 and len(found_endpoints) == 0,
|
||||||
|
"http_imports_found": sorted(found_http),
|
||||||
|
"forbidden_endpoints_found": found_endpoints,
|
||||||
|
"no_git_write_in_whitelist": len(write_in_whitelist) == 0,
|
||||||
|
"git_write_in_whitelist": write_in_whitelist,
|
||||||
|
"no_downstream_write": (
|
||||||
|
len(found_http) == 0
|
||||||
|
and len(found_endpoints) == 0
|
||||||
|
and len(write_in_whitelist) == 0
|
||||||
|
),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _collect_code_strings(tree: ast.AST) -> List[str]:
|
||||||
|
"""
|
||||||
|
Sammelt alle String-Literale, die KEINE Docstrings sind (echter Code).
|
||||||
|
Docstrings = erstes Statement eines Modul-/Funktions-/Klassen-Body.
|
||||||
|
"""
|
||||||
|
import ast as _ast
|
||||||
|
|
||||||
|
docstring_nodes = set()
|
||||||
|
for node in _ast.walk(tree):
|
||||||
|
if isinstance(node, (_ast.Module, _ast.FunctionDef, _ast.AsyncFunctionDef,
|
||||||
|
_ast.ClassDef)):
|
||||||
|
body = getattr(node, "body", None)
|
||||||
|
if body and isinstance(body[0], _ast.Expr) and isinstance(
|
||||||
|
body[0].value, _ast.Constant
|
||||||
|
) and isinstance(body[0].value.value, str):
|
||||||
|
docstring_nodes.add(id(body[0].value))
|
||||||
|
|
||||||
|
result = []
|
||||||
|
for node in _ast.walk(tree):
|
||||||
|
if isinstance(node, _ast.Constant) and isinstance(node.value, str):
|
||||||
|
if id(node) in docstring_nodes:
|
||||||
|
continue
|
||||||
|
result.append(node.value)
|
||||||
|
return result
|
||||||
539
tolaria/c5-sync-service/test_c5b.py
Normal file
539
tolaria/c5-sync-service/test_c5b.py
Normal file
|
|
@ -0,0 +1,539 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Red Queen — C5B Testsuite (Forgejo Polling & Change Detection Engine).
|
||||||
|
|
||||||
|
Testet gegen ein kontrolliertes lokales Git-Fixture-Repo (read-only fuer C5B).
|
||||||
|
Jeder Test nutzt eine frische temp-DB (tempfile.mkdtemp), nie die Produkt-DB.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import shutil
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
import unittest
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Dict
|
||||||
|
|
||||||
|
# C5B importieren (aus demselben Verzeichnis)
|
||||||
|
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||||
|
|
||||||
|
from rq_c5a import C5AStore, ST_DISCOVERED, ST_VALIDATING, ST_HUMAN_REVIEW_REQUIRED, ST_APPLIED
|
||||||
|
from rq_c5b import (
|
||||||
|
C5BPoller,
|
||||||
|
GitReader,
|
||||||
|
KnowledgeScope,
|
||||||
|
ChangeClassifier,
|
||||||
|
assert_no_downstream_write,
|
||||||
|
parse_frontmatter,
|
||||||
|
extract_object_id,
|
||||||
|
content_hash,
|
||||||
|
metadata_hash,
|
||||||
|
detect_secret,
|
||||||
|
SCOPE_IN_SCOPE,
|
||||||
|
SCOPE_OUT_OF_SCOPE,
|
||||||
|
SCOPE_LEGACY_SPECIAL,
|
||||||
|
SCOPE_HUMAN_REVIEW,
|
||||||
|
OP_CREATE,
|
||||||
|
OP_CONTENT_UPDATE,
|
||||||
|
OP_METADATA_UPDATE,
|
||||||
|
OP_STATE_UPDATE,
|
||||||
|
OP_TAGS_UPDATE,
|
||||||
|
OP_SOURCE_CANONICAL_RELATION_UPDATE,
|
||||||
|
OP_RENAME,
|
||||||
|
OP_MOVE,
|
||||||
|
OP_DELETE_REQUEST,
|
||||||
|
OP_SUPERSEDE,
|
||||||
|
RC_FORGEJO_UNAVAILABLE,
|
||||||
|
RC_OUT_OF_ORDER_COMMIT,
|
||||||
|
RC_UNKNOWN_OBJECT_ID,
|
||||||
|
RC_SECRET_DETECTED,
|
||||||
|
GitWriteBlockedError,
|
||||||
|
ForgejoUnavailableError,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Fixture: kontrolliertes Git-Repo
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def _git(repo: str, *args: str) -> str:
|
||||||
|
"""Fuehrt einen git-Befehl im Fixture-Repo aus (Test-Helfer, darf schreiben)."""
|
||||||
|
proc = subprocess.run(
|
||||||
|
["git", "-C", repo] + list(args),
|
||||||
|
capture_output=True, text=True,
|
||||||
|
)
|
||||||
|
if proc.returncode != 0:
|
||||||
|
raise RuntimeError(f"git {' '.join(args)} fehlgeschlagen: {proc.stderr}")
|
||||||
|
return proc.stdout
|
||||||
|
|
||||||
|
|
||||||
|
def _write(repo: str, path: str, content: str) -> None:
|
||||||
|
full = os.path.join(repo, path)
|
||||||
|
os.makedirs(os.path.dirname(full), exist_ok=True)
|
||||||
|
with open(full, "w", encoding="utf-8") as f:
|
||||||
|
f.write(content)
|
||||||
|
|
||||||
|
|
||||||
|
def _commit(repo: str, message: str) -> str:
|
||||||
|
_git(repo, "add", "-A")
|
||||||
|
_git(repo, "commit", "-m", message, "--no-verify")
|
||||||
|
return _git(repo, "rev-parse", "HEAD").strip()
|
||||||
|
|
||||||
|
|
||||||
|
def _fm(id_: str, **extra) -> str:
|
||||||
|
"""Baut Frontmatter fuer ein Knowledge-Objekt."""
|
||||||
|
lines = ["---", f"knowledge_schema: 1", f"id: {id_}"]
|
||||||
|
for k, v in extra.items():
|
||||||
|
if isinstance(v, list):
|
||||||
|
lines.append(f"{k}: [{', '.join(v)}]")
|
||||||
|
else:
|
||||||
|
lines.append(f"{k}: {v}")
|
||||||
|
lines.append("---")
|
||||||
|
return "\n".join(lines) + "\n"
|
||||||
|
|
||||||
|
|
||||||
|
class FixtureRepo:
|
||||||
|
"""Erstellt ein kontrolliertes Git-Fixture-Repo fuer C5B-Tests."""
|
||||||
|
|
||||||
|
def __init__(self):
|
||||||
|
self.dir = tempfile.mkdtemp(prefix="c5b_fixture_")
|
||||||
|
_git(self.dir, "init", "-q", "-b", "main")
|
||||||
|
_git(self.dir, "config", "user.email", "test@test")
|
||||||
|
_git(self.dir, "config", "user.name", "Test")
|
||||||
|
self.commits: Dict[str, str] = {}
|
||||||
|
|
||||||
|
def write(self, path: str, content: str) -> None:
|
||||||
|
_write(self.dir, path, content)
|
||||||
|
|
||||||
|
def commit(self, message: str) -> str:
|
||||||
|
sha = _commit(self.dir, message)
|
||||||
|
self.commits[message] = sha
|
||||||
|
return sha
|
||||||
|
|
||||||
|
def cleanup(self) -> None:
|
||||||
|
shutil.rmtree(self.dir, ignore_errors=True)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Tests
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
class TestFrontmatterAndHashes(unittest.TestCase):
|
||||||
|
"""Frontmatter-Parsing, Object-ID-Extraktion, Hash-Model."""
|
||||||
|
|
||||||
|
def test_parse_frontmatter(self):
|
||||||
|
content = _fm("object/48bd264f-607b-15f1-5f73-3e922af9b19d",
|
||||||
|
type="arch", role="module", representation="source",
|
||||||
|
state="current", tags=["a", "b"])
|
||||||
|
fm, body = parse_frontmatter(content)
|
||||||
|
self.assertEqual(fm["id"], "object/48bd264f-607b-15f1-5f73-3e922af9b19d")
|
||||||
|
self.assertEqual(fm["type"], "arch")
|
||||||
|
self.assertEqual(fm["tags"], ["a", "b"])
|
||||||
|
self.assertNotIn("---", body)
|
||||||
|
|
||||||
|
def test_extract_object_id_valid(self):
|
||||||
|
content = _fm("object/48bd264f-607b-15f1-5f73-3e922af9b19d")
|
||||||
|
self.assertEqual(
|
||||||
|
extract_object_id(content),
|
||||||
|
"object/48bd264f-607b-15f1-5f73-3e922af9b19d",
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_extract_object_id_invalid(self):
|
||||||
|
# Kein object/-Praefix, kein FULL_UUID
|
||||||
|
self.assertIsNone(extract_object_id("---\nid: object/abc\n---\n"))
|
||||||
|
self.assertIsNone(extract_object_id("---\nid: 12345\n---\n"))
|
||||||
|
self.assertIsNone(extract_object_id("kein frontmatter"))
|
||||||
|
|
||||||
|
def test_content_hash_deterministic(self):
|
||||||
|
self.assertEqual(content_hash("abc"), content_hash("abc"))
|
||||||
|
self.assertNotEqual(content_hash("abc"), content_hash("abd"))
|
||||||
|
|
||||||
|
def test_metadata_hash_fields(self):
|
||||||
|
fm1 = {"id": "object/77b02661-d67b-4bae-b612-01d1287cea6b", "type": "arch", "role": "module",
|
||||||
|
"representation": "source", "state": "current",
|
||||||
|
"knowledge_schema": "1"}
|
||||||
|
fm2 = dict(fm1)
|
||||||
|
fm2["state"] = "historical"
|
||||||
|
self.assertNotEqual(metadata_hash(fm1), metadata_hash(fm2))
|
||||||
|
# Pfad ist NICHT Teil von metadata_hash
|
||||||
|
self.assertEqual(metadata_hash(fm1), metadata_hash(fm1))
|
||||||
|
|
||||||
|
def test_detect_secret(self):
|
||||||
|
self.assertIsNotNone(detect_secret("key: sk-abcdefghijklmnopqrstuvwxyz123456"))
|
||||||
|
self.assertIsNotNone(detect_secret("-----BEGIN RSA PRIVATE KEY-----\n..."))
|
||||||
|
self.assertIsNone(detect_secret("modul-07-risk-manager.md"))
|
||||||
|
|
||||||
|
|
||||||
|
class TestKnowledgeScope(unittest.TestCase):
|
||||||
|
"""Knowledge-Scope-Klassifikation."""
|
||||||
|
|
||||||
|
def setUp(self):
|
||||||
|
self.dir = tempfile.mkdtemp(prefix="c5b_scope_")
|
||||||
|
self.scope = KnowledgeScope(self.dir)
|
||||||
|
|
||||||
|
def tearDown(self):
|
||||||
|
shutil.rmtree(self.dir, ignore_errors=True)
|
||||||
|
|
||||||
|
def test_root_markdown_in_scope(self):
|
||||||
|
self.assertEqual(self.scope.classify("modul-09-execution-service.md"), SCOPE_IN_SCOPE)
|
||||||
|
|
||||||
|
def test_canonical_in_scope(self):
|
||||||
|
self.assertEqual(
|
||||||
|
self.scope.classify("notes/trading/system-docs/modul-09-execution-service.md"),
|
||||||
|
SCOPE_IN_SCOPE,
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_out_of_scope_dirs(self):
|
||||||
|
for p in ["tolaria/C5_SYNC_ARCHITECTURE_DESIGN.md",
|
||||||
|
"a2/README.md",
|
||||||
|
"red-queen-architecture/AGENT_CONTRACTS.md",
|
||||||
|
"notion-safety-brain/README.md",
|
||||||
|
"backup_patches/x.md",
|
||||||
|
"notes/reference/vps-infrastruktur.md"]:
|
||||||
|
self.assertEqual(self.scope.classify(p), SCOPE_OUT_OF_SCOPE, p)
|
||||||
|
|
||||||
|
def test_non_markdown_out_of_scope(self):
|
||||||
|
self.assertEqual(self.scope.classify("index_source.json"), SCOPE_OUT_OF_SCOPE)
|
||||||
|
|
||||||
|
def test_legacy_special(self):
|
||||||
|
self.assertEqual(self.scope.classify("README.md"), SCOPE_LEGACY_SPECIAL)
|
||||||
|
self.assertEqual(self.scope.classify("notes/trading/system-docs/README.md"), SCOPE_LEGACY_SPECIAL)
|
||||||
|
self.assertEqual(self.scope.classify("vps.md"), SCOPE_LEGACY_SPECIAL)
|
||||||
|
|
||||||
|
def test_in_scope_with_valid_id(self):
|
||||||
|
content = _fm("object/48bd264f-607b-15f1-5f73-3e922af9b19d")
|
||||||
|
r = self.scope.classify_with_content("modul-09.md", content)
|
||||||
|
self.assertEqual(r["scope"], SCOPE_IN_SCOPE)
|
||||||
|
self.assertEqual(r["object_id"], "object/48bd264f-607b-15f1-5f73-3e922af9b19d")
|
||||||
|
|
||||||
|
def test_in_scope_without_id_human_review(self):
|
||||||
|
r = self.scope.classify_with_content("modul-09.md", "kein frontmatter")
|
||||||
|
self.assertEqual(r["scope"], SCOPE_HUMAN_REVIEW)
|
||||||
|
self.assertEqual(r["reason_code"], RC_UNKNOWN_OBJECT_ID)
|
||||||
|
|
||||||
|
|
||||||
|
class TestChangeClassifier(unittest.TestCase):
|
||||||
|
"""Change-Klassifikation (BEFORE/AFTER -> Operationen)."""
|
||||||
|
|
||||||
|
def setUp(self):
|
||||||
|
self.dir = tempfile.mkdtemp(prefix="c5b_cls_")
|
||||||
|
self.cls = ChangeClassifier(KnowledgeScope(self.dir))
|
||||||
|
|
||||||
|
def tearDown(self):
|
||||||
|
shutil.rmtree(self.dir, ignore_errors=True)
|
||||||
|
|
||||||
|
def test_create(self):
|
||||||
|
ops = self.cls.classify(None, "modul-09.md", None, _fm("object/77b02661-d67b-4bae-b612-01d1287cea6b"))
|
||||||
|
self.assertEqual(ops[0]["operation"], OP_CREATE)
|
||||||
|
|
||||||
|
def test_delete_request(self):
|
||||||
|
ops = self.cls.classify("modul-09.md", None, _fm("object/77b02661-d67b-4bae-b612-01d1287cea6b"), None)
|
||||||
|
self.assertEqual(ops[0]["operation"], OP_DELETE_REQUEST)
|
||||||
|
|
||||||
|
def test_content_update(self):
|
||||||
|
before = _fm("object/77b02661-d67b-4bae-b612-01d1287cea6b", state="current") + "body v1"
|
||||||
|
after = _fm("object/77b02661-d67b-4bae-b612-01d1287cea6b", state="current") + "body v2"
|
||||||
|
ops = self.cls.classify("modul-09.md", "modul-09.md", before, after)
|
||||||
|
self.assertIn(OP_CONTENT_UPDATE, [o["operation"] for o in ops])
|
||||||
|
|
||||||
|
def test_state_update(self):
|
||||||
|
before = _fm("object/77b02661-d67b-4bae-b612-01d1287cea6b", state="current") + "body"
|
||||||
|
after = _fm("object/77b02661-d67b-4bae-b612-01d1287cea6b", state="historical") + "body"
|
||||||
|
ops = self.cls.classify("modul-09.md", "modul-09.md", before, after)
|
||||||
|
self.assertIn(OP_STATE_UPDATE, [o["operation"] for o in ops])
|
||||||
|
|
||||||
|
def test_tags_update(self):
|
||||||
|
before = _fm("object/77b02661-d67b-4bae-b612-01d1287cea6b", tags=["a"]) + "body"
|
||||||
|
after = _fm("object/77b02661-d67b-4bae-b612-01d1287cea6b", tags=["a", "b"]) + "body"
|
||||||
|
ops = self.cls.classify("modul-09.md", "modul-09.md", before, after)
|
||||||
|
self.assertIn(OP_TAGS_UPDATE, [o["operation"] for o in ops])
|
||||||
|
|
||||||
|
def test_relation_update(self):
|
||||||
|
before = _fm("object/77b02661-d67b-4bae-b612-01d1287cea6b") + "body"
|
||||||
|
after = _fm("object/77b02661-d67b-4bae-b612-01d1287cea6b", derived_from="object/y") + "body"
|
||||||
|
ops = self.cls.classify("modul-09.md", "modul-09.md", before, after)
|
||||||
|
self.assertIn(OP_SOURCE_CANONICAL_RELATION_UPDATE, [o["operation"] for o in ops])
|
||||||
|
|
||||||
|
def test_rename_same_id(self):
|
||||||
|
before = _fm("object/77b02661-d67b-4bae-b612-01d1287cea6b") + "body"
|
||||||
|
after = _fm("object/77b02661-d67b-4bae-b612-01d1287cea6b") + "body"
|
||||||
|
ops = self.cls.classify("modul-09.md", "modul-09-new.md", before, after)
|
||||||
|
self.assertIn(OP_RENAME, [o["operation"] for o in ops])
|
||||||
|
|
||||||
|
def test_move_same_id(self):
|
||||||
|
before = _fm("object/77b02661-d67b-4bae-b612-01d1287cea6b") + "body"
|
||||||
|
after = _fm("object/77b02661-d67b-4bae-b612-01d1287cea6b") + "body"
|
||||||
|
ops = self.cls.classify("modul-09.md", "sub/modul-09.md", before, after)
|
||||||
|
self.assertIn(OP_MOVE, [o["operation"] for o in ops])
|
||||||
|
|
||||||
|
def test_supersede(self):
|
||||||
|
before = _fm("object/77b02661-d67b-4bae-b612-01d1287cea6b", state="current") + "body"
|
||||||
|
after = _fm("object/77b02661-d67b-4bae-b612-01d1287cea6b", state="superseded") + "body"
|
||||||
|
ops = self.cls.classify("modul-09.md", "modul-09.md", before, after)
|
||||||
|
self.assertIn(OP_SUPERSEDE, [o["operation"] for o in ops])
|
||||||
|
|
||||||
|
def test_no_change(self):
|
||||||
|
before = _fm("object/77b02661-d67b-4bae-b612-01d1287cea6b") + "body"
|
||||||
|
after = _fm("object/77b02661-d67b-4bae-b612-01d1287cea6b") + "body"
|
||||||
|
ops = self.cls.classify("modul-09.md", "modul-09.md", before, after)
|
||||||
|
self.assertEqual(ops, [])
|
||||||
|
|
||||||
|
|
||||||
|
class TestGitReader(unittest.TestCase):
|
||||||
|
"""GitReader read-only + Write-Blockade."""
|
||||||
|
|
||||||
|
def test_write_commands_blocked(self):
|
||||||
|
fx = FixtureRepo()
|
||||||
|
try:
|
||||||
|
reader = GitReader(fx.dir)
|
||||||
|
for cmd in ["push", "commit", "add", "reset", "checkout", "merge", "rebase"]:
|
||||||
|
with self.assertRaises(GitWriteBlockedError):
|
||||||
|
reader._run([cmd, "x"])
|
||||||
|
finally:
|
||||||
|
fx.cleanup()
|
||||||
|
|
||||||
|
def test_head_discovery(self):
|
||||||
|
fx = FixtureRepo()
|
||||||
|
try:
|
||||||
|
fx.write("modul-09.md", _fm("object/77b02661-d67b-4bae-b612-01d1287cea6b") + "body")
|
||||||
|
sha = fx.commit("c1")
|
||||||
|
reader = GitReader(fx.dir)
|
||||||
|
self.assertEqual(reader.get_current_head(), sha)
|
||||||
|
finally:
|
||||||
|
fx.cleanup()
|
||||||
|
|
||||||
|
def test_forgejo_unavailable(self):
|
||||||
|
with self.assertRaises(ForgejoUnavailableError):
|
||||||
|
GitReader("/nonexistent/path")
|
||||||
|
|
||||||
|
|
||||||
|
class TestC5BPoller(unittest.TestCase):
|
||||||
|
"""End-to-End Poll-Zyklen gegen Fixture-Repo."""
|
||||||
|
|
||||||
|
def _new_store(self):
|
||||||
|
db = os.path.join(tempfile.mkdtemp(prefix="c5b_db_"), "c5a.db")
|
||||||
|
return C5AStore(db)
|
||||||
|
|
||||||
|
def test_single_commit_discovery(self):
|
||||||
|
fx = FixtureRepo()
|
||||||
|
try:
|
||||||
|
fx.write("modul-09.md", _fm("object/48bd264f-607b-15f1-5f73-3e922af9b19d") + "body")
|
||||||
|
sha = fx.commit("c1")
|
||||||
|
store = self._new_store()
|
||||||
|
poller = C5BPoller(store, fx.dir)
|
||||||
|
r = poller.poll_once()
|
||||||
|
self.assertEqual(r["status"], "OK")
|
||||||
|
self.assertEqual(r["commits_discovered"], 1)
|
||||||
|
self.assertEqual(r["head"], sha)
|
||||||
|
# Commit registriert, Status VALIDATING (nicht weiter)
|
||||||
|
c = store.get_commit(sha)
|
||||||
|
self.assertEqual(c["status"], ST_VALIDATING)
|
||||||
|
store.close()
|
||||||
|
finally:
|
||||||
|
fx.cleanup()
|
||||||
|
|
||||||
|
def test_multi_commit_ordering(self):
|
||||||
|
fx = FixtureRepo()
|
||||||
|
try:
|
||||||
|
fx.write("modul-09.md", _fm("object/77b02661-d67b-4bae-b612-01d1287cea6b") + "v1")
|
||||||
|
a = fx.commit("A")
|
||||||
|
fx.write("modul-09.md", _fm("object/77b02661-d67b-4bae-b612-01d1287cea6b") + "v2")
|
||||||
|
b = fx.commit("B")
|
||||||
|
fx.write("modul-09.md", _fm("object/77b02661-d67b-4bae-b612-01d1287cea6b") + "v3")
|
||||||
|
c = fx.commit("C")
|
||||||
|
store = self._new_store()
|
||||||
|
poller = C5BPoller(store, fx.dir)
|
||||||
|
r = poller.poll_once()
|
||||||
|
self.assertEqual(r["commits_discovered"], 3)
|
||||||
|
# Reihenfolge A, B, C (chronologisch)
|
||||||
|
commits = store.list_commits()
|
||||||
|
self.assertEqual([c["commit_sha"] for c in commits], [a, b, c])
|
||||||
|
store.close()
|
||||||
|
finally:
|
||||||
|
fx.cleanup()
|
||||||
|
|
||||||
|
def test_duplicate_poll_idempotent(self):
|
||||||
|
fx = FixtureRepo()
|
||||||
|
try:
|
||||||
|
fx.write("modul-09.md", _fm("object/77b02661-d67b-4bae-b612-01d1287cea6b") + "body")
|
||||||
|
fx.commit("c1")
|
||||||
|
store = self._new_store()
|
||||||
|
poller = C5BPoller(store, fx.dir)
|
||||||
|
r1 = poller.poll_once()
|
||||||
|
r2 = poller.poll_once() # gleicher HEAD
|
||||||
|
self.assertEqual(r1["commits_discovered"], 1)
|
||||||
|
self.assertEqual(r2["commits_discovered"], 0) # keine neuen
|
||||||
|
# Keine doppelten ObjectChanges
|
||||||
|
objs = store.list_object_changes(fx.commits["c1"])
|
||||||
|
self.assertEqual(len(objs), 1)
|
||||||
|
store.close()
|
||||||
|
finally:
|
||||||
|
fx.cleanup()
|
||||||
|
|
||||||
|
def test_lost_poll_recovery(self):
|
||||||
|
fx = FixtureRepo()
|
||||||
|
try:
|
||||||
|
fx.write("modul-09.md", _fm("object/77b02661-d67b-4bae-b612-01d1287cea6b") + "v1")
|
||||||
|
a = fx.commit("A")
|
||||||
|
store = self._new_store()
|
||||||
|
poller = C5BPoller(store, fx.dir)
|
||||||
|
poller.poll_once() # Poll bei A
|
||||||
|
# Zwei Poll-Intervalle ausfallen: B, C, D entstehen
|
||||||
|
fx.write("modul-09.md", _fm("object/77b02661-d67b-4bae-b612-01d1287cea6b") + "v2")
|
||||||
|
b = fx.commit("B")
|
||||||
|
fx.write("modul-09.md", _fm("object/77b02661-d67b-4bae-b612-01d1287cea6b") + "v3")
|
||||||
|
c = fx.commit("C")
|
||||||
|
fx.write("modul-09.md", _fm("object/77b02661-d67b-4bae-b612-01d1287cea6b") + "v4")
|
||||||
|
d = fx.commit("D")
|
||||||
|
r = poller.poll_once() # naechster Poll bei D
|
||||||
|
self.assertEqual(r["commits_discovered"], 3)
|
||||||
|
self.assertEqual(r["new_commits"], [b, c, d])
|
||||||
|
store.close()
|
||||||
|
finally:
|
||||||
|
fx.cleanup()
|
||||||
|
|
||||||
|
def test_out_of_scope_not_registered(self):
|
||||||
|
fx = FixtureRepo()
|
||||||
|
try:
|
||||||
|
fx.write("tolaria/C5_SYNC_ARCHITECTURE_DESIGN.md", "# design")
|
||||||
|
fx.write("modul-09.md", _fm("object/77b02661-d67b-4bae-b612-01d1287cea6b") + "body")
|
||||||
|
fx.commit("c1")
|
||||||
|
store = self._new_store()
|
||||||
|
poller = C5BPoller(store, fx.dir)
|
||||||
|
r = poller.poll_once()
|
||||||
|
self.assertEqual(r["commits_discovered"], 1)
|
||||||
|
self.assertEqual(r["out_of_scope_count"], 1) # tolaria/ ist out-of-scope
|
||||||
|
store.close()
|
||||||
|
finally:
|
||||||
|
fx.cleanup()
|
||||||
|
|
||||||
|
def test_legacy_special_not_registered(self):
|
||||||
|
fx = FixtureRepo()
|
||||||
|
try:
|
||||||
|
fx.write("README.md", "# README")
|
||||||
|
fx.write("vps.md", "# vps")
|
||||||
|
fx.write("modul-09.md", _fm("object/77b02661-d67b-4bae-b612-01d1287cea6b") + "body")
|
||||||
|
fx.commit("c1")
|
||||||
|
store = self._new_store()
|
||||||
|
poller = C5BPoller(store, fx.dir)
|
||||||
|
r = poller.poll_once()
|
||||||
|
self.assertEqual(r["out_of_scope_count"], 2) # README + vps
|
||||||
|
store.close()
|
||||||
|
finally:
|
||||||
|
fx.cleanup()
|
||||||
|
|
||||||
|
def test_unknown_object_id_human_review(self):
|
||||||
|
fx = FixtureRepo()
|
||||||
|
try:
|
||||||
|
fx.write("modul-09.md", "kein frontmatter, keine id")
|
||||||
|
fx.commit("c1")
|
||||||
|
store = self._new_store()
|
||||||
|
poller = C5BPoller(store, fx.dir)
|
||||||
|
r = poller.poll_once()
|
||||||
|
self.assertEqual(r["human_review_count"], 1)
|
||||||
|
c = store.get_commit(fx.commits["c1"])
|
||||||
|
self.assertEqual(c["status"], ST_HUMAN_REVIEW_REQUIRED)
|
||||||
|
store.close()
|
||||||
|
finally:
|
||||||
|
fx.cleanup()
|
||||||
|
|
||||||
|
def test_secret_detection_fail_closed(self):
|
||||||
|
fx = FixtureRepo()
|
||||||
|
try:
|
||||||
|
fx.write("modul-09.md", _fm("object/77b02661-d67b-4bae-b612-01d1287cea6b") + "key: sk-abcdefghijklmnopqrstuvwxyz123456")
|
||||||
|
fx.commit("c1")
|
||||||
|
store = self._new_store()
|
||||||
|
poller = C5BPoller(store, fx.dir)
|
||||||
|
r = poller.poll_once()
|
||||||
|
self.assertEqual(r["human_review_count"], 1)
|
||||||
|
c = store.get_commit(fx.commits["c1"])
|
||||||
|
self.assertEqual(c["status"], ST_HUMAN_REVIEW_REQUIRED)
|
||||||
|
# Kein Secret-Inhalt in der DB
|
||||||
|
objs = store.list_object_changes(fx.commits["c1"])
|
||||||
|
for o in objs:
|
||||||
|
self.assertNotIn("sk-", json.dumps(o))
|
||||||
|
store.close()
|
||||||
|
finally:
|
||||||
|
fx.cleanup()
|
||||||
|
|
||||||
|
def test_history_divergence_fail_closed(self):
|
||||||
|
fx = FixtureRepo()
|
||||||
|
try:
|
||||||
|
fx.write("modul-09.md", _fm("object/77b02661-d67b-4bae-b612-01d1287cea6b") + "v1")
|
||||||
|
a = fx.commit("A")
|
||||||
|
store = self._new_store()
|
||||||
|
poller = C5BPoller(store, fx.dir)
|
||||||
|
poller.poll_once()
|
||||||
|
# last_applied = a. Jetzt History-Rewrite: neuer Root ohne a als Ancestor.
|
||||||
|
# Wir wechseln auf einen Orphan-Branch (neuer Root), der a NICHT enthaelt.
|
||||||
|
_git(fx.dir, "checkout", "--orphan", "newroot")
|
||||||
|
_git(fx.dir, "rm", "-rf", ".")
|
||||||
|
fx.write("modul-09.md", _fm("object/77b02661-d67b-4bae-b612-01d1287cea6b") + "v2")
|
||||||
|
b = fx.commit("B")
|
||||||
|
# HEAD ist jetzt b (newroot), a ist NICHT in dessen Ancestry
|
||||||
|
self.assertNotEqual(a, b)
|
||||||
|
r = poller.poll_once()
|
||||||
|
self.assertEqual(r["status"], "FAIL_CLOSED")
|
||||||
|
self.assertEqual(r["reason_code"], RC_OUT_OF_ORDER_COMMIT)
|
||||||
|
store.close()
|
||||||
|
finally:
|
||||||
|
fx.cleanup()
|
||||||
|
|
||||||
|
def test_c5a_persistence_integration(self):
|
||||||
|
fx = FixtureRepo()
|
||||||
|
try:
|
||||||
|
fx.write("modul-09.md", _fm("object/77b02661-d67b-4bae-b612-01d1287cea6b") + "body")
|
||||||
|
sha = fx.commit("c1")
|
||||||
|
db = os.path.join(tempfile.mkdtemp(prefix="c5b_db_"), "c5a.db")
|
||||||
|
store = C5AStore(db)
|
||||||
|
poller = C5BPoller(store, fx.dir)
|
||||||
|
poller.poll_once()
|
||||||
|
store.close()
|
||||||
|
# Neuer Store auf gleicher DB: State restart-fest
|
||||||
|
store2 = C5AStore(db)
|
||||||
|
c = store2.get_commit(sha)
|
||||||
|
self.assertEqual(c["status"], ST_VALIDATING)
|
||||||
|
store2.close()
|
||||||
|
finally:
|
||||||
|
fx.cleanup()
|
||||||
|
|
||||||
|
def test_no_downstream_write(self):
|
||||||
|
r = assert_no_downstream_write()
|
||||||
|
self.assertTrue(r["no_downstream_write"])
|
||||||
|
self.assertEqual(r["http_imports_found"], [])
|
||||||
|
self.assertEqual(r["forbidden_endpoints_found"], [])
|
||||||
|
self.assertEqual(r["git_write_in_whitelist"], [])
|
||||||
|
|
||||||
|
|
||||||
|
class TestPollIntervalConfig(unittest.TestCase):
|
||||||
|
"""Poll-Intervall ist Environment-konfigurierbar."""
|
||||||
|
|
||||||
|
def test_default_60(self):
|
||||||
|
fx = FixtureRepo()
|
||||||
|
try:
|
||||||
|
store = C5AStore(os.path.join(tempfile.mkdtemp(), "c5a.db"))
|
||||||
|
poller = C5BPoller(store, fx.dir)
|
||||||
|
self.assertEqual(poller.poll_interval_seconds, 60)
|
||||||
|
store.close()
|
||||||
|
finally:
|
||||||
|
fx.cleanup()
|
||||||
|
|
||||||
|
def test_env_override(self):
|
||||||
|
fx = FixtureRepo()
|
||||||
|
try:
|
||||||
|
os.environ["C5_POLL_INTERVAL_SECONDS"] = "120"
|
||||||
|
store = C5AStore(os.path.join(tempfile.mkdtemp(), "c5a.db"))
|
||||||
|
poller = C5BPoller(store, fx.dir)
|
||||||
|
self.assertEqual(poller.poll_interval_seconds, 120)
|
||||||
|
store.close()
|
||||||
|
del os.environ["C5_POLL_INTERVAL_SECONDS"]
|
||||||
|
finally:
|
||||||
|
fx.cleanup()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main(verbosity=2)
|
||||||
Loading…
Reference in a new issue