wave1: persist canonical L0 schema and fingerprint tooling
This commit is contained in:
parent
e56567f7cf
commit
41e68aeb21
10 changed files with 5409 additions and 0 deletions
|
|
@ -0,0 +1,105 @@
|
||||||
|
# D1/D2 — Canonical Schema Contract & Fingerprint Tooling (SoT)
|
||||||
|
|
||||||
|
**Status:** FORMALLY PERSISTED — authoritative origin/main
|
||||||
|
**Datum:** 2026-09-08
|
||||||
|
**Verdict:** D1+D2 FORMALLY PERSISTED TO AUTHORITATIVE SOT
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Zweck
|
||||||
|
|
||||||
|
Dieses Verzeichnis materialisiert den **CURRENT VERIFIED PRODUCTION SCHEMA CONTRACT (L0)**
|
||||||
|
als kanonische, deterministisch fingerprintbare Autorität. Es ist das Ergebnis der
|
||||||
|
D1/D2-Remediation (Schema-Contract + deterministischer Fingerprint) und wurde durch den
|
||||||
|
unabhängigen D1+D2-Checker mit **Verdict A** bestätigt.
|
||||||
|
|
||||||
|
## 2. Canonical Fingerprint
|
||||||
|
|
||||||
|
```
|
||||||
|
b49f64337ffafc0e73d46e009aa802cd0172d59421b7df61d3eb198775c1986d
|
||||||
|
```
|
||||||
|
|
||||||
|
Dieser Fingerprint ist deterministisch (H1==H2==H3), sensitiv auf alle semantischen
|
||||||
|
Schema-Änderungen (type, nullable, default, PK, UNIQUE, FK, CHECK, index), und fail-closed.
|
||||||
|
|
||||||
|
## 3. DR-Reproducibility (ehrlich getrennt)
|
||||||
|
|
||||||
|
| Aspekt | Status |
|
||||||
|
|--------|--------|
|
||||||
|
| **CURRENT-STATE SCHEMA REPRODUCIBLE FROM L0** | **YES** — frische DB aus L0-SQL rekonstruiert, REBUILT_HASH == PRODUCTION_HASH == b49f6433…, Semantic-Diff TOTAL_MISMATCH=0 |
|
||||||
|
| **HISTORICAL EVOLUTION REPRODUCIBLE** | **NO** — historische DDL-Chronologie (Wave1→WaveD→Closure) bleibt nicht aus L0 rekonstruierbar |
|
||||||
|
|
||||||
|
Der L0-Contract beansprucht ausschliesslich **CURRENT PRODUCTION TRUTH**, nicht historische
|
||||||
|
Migrationschronologie. Historische Artefakte bleiben immutable.
|
||||||
|
|
||||||
|
## 4. Canonical Contract Semantics
|
||||||
|
|
||||||
|
| Tabelle | Spalten |
|
||||||
|
|---------|---------|
|
||||||
|
| historical_bar | 41 |
|
||||||
|
| derived_bar | 35 |
|
||||||
|
| data_ingestion_run | 29 |
|
||||||
|
| dataset_version | 23 |
|
||||||
|
| ingestion_provenance | 41 |
|
||||||
|
|
||||||
|
**Semantic Overrides (Owner-Canonical):**
|
||||||
|
- `last_successful_chunk` = **timestamp with time zone / timestamptz** (Production + Owner-Entscheidung)
|
||||||
|
- `interrupt_reason` = **text** · PRESERVED · origin unknown · do not remove
|
||||||
|
- `technical_quality` = **text** · nullable · default `'VALID'`
|
||||||
|
|
||||||
|
## 5. Historische Wahrheit (nicht rückwirkend ändern)
|
||||||
|
|
||||||
|
- **WaveD historisches Artefakt:** `last_successful_chunk = text` (immutable Evidence)
|
||||||
|
- **Owner/current canonical contract:** `last_successful_chunk = timestamptz`
|
||||||
|
- Historische Artefakte: Wave1 `6983526da3ca51fd2ececffbef0ccbd66b048a3d11c228000011aaba3e1d6bc5`,
|
||||||
|
WaveD `7aa4e2a10c98fa327e000a49f6a497efb5814c8808dba429dfebfc90c544a02b`,
|
||||||
|
Closure `e56567f7cfae68da70c11ece6f576e974488f343` — **unverändert**.
|
||||||
|
|
||||||
|
## 6. Open Debts (nicht fälschlich geschlossen)
|
||||||
|
|
||||||
|
| Debt | Status |
|
||||||
|
|------|--------|
|
||||||
|
| **Debt A** — Current-state DR/schema reproducibility gap | **CLOSED** by D1/D2 |
|
||||||
|
| **Debt A** — Historical DDL evolution provenance gap | **REMAINS OPEN** |
|
||||||
|
| **Debt B** — interrupt_reason origin unknown | **OPEN** (non-blocking) |
|
||||||
|
| **Debt C** — WaveD text vs current Owner-canonical timestamptz | **OPEN** (non-blocking) |
|
||||||
|
|
||||||
|
## 7. SoT-Layout
|
||||||
|
|
||||||
|
```
|
||||||
|
d1d2-schema-contract/
|
||||||
|
├── SCHEMA_AUTHORITY.md # dieses Dokument
|
||||||
|
├── contract/
|
||||||
|
│ ├── historical_v2_schema.sql # L0-Schema-Contract (schema-only)
|
||||||
|
│ ├── historical_v2_schema.json # maschinenlesbares kanonisches Modell
|
||||||
|
│ ├── historical_v2_schema.sha256 # deterministischer Fingerprint
|
||||||
|
│ └── README.md # Contract-Dokumentation
|
||||||
|
└── tools/
|
||||||
|
├── canonical_schema.py # Kanonisierungs-Implementierung
|
||||||
|
├── capture_schema.py # Capture-Parser (read-only)
|
||||||
|
├── compare_schema.py # Semantic-Diff-CLI
|
||||||
|
├── fingerprint_schema.py # Fingerprint-CLI
|
||||||
|
└── gen_l0_sql.py # L0-SQL-Generator
|
||||||
|
```
|
||||||
|
|
||||||
|
**Nicht persistiert:** Maker-Evidence, Checker-Evidence, temporäre DBs, /tmp-Dateien,
|
||||||
|
raw Scratch Outputs, runtime-spezifische Dateien, Credentials/Secrets, Transport-Manifeste,
|
||||||
|
`gen_evidence.py` (enthält hardcodierte Maker-Pfade, nicht environment-neutral).
|
||||||
|
|
||||||
|
## 8. Tooling-Nutzung
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Fingerprint aus machine contract
|
||||||
|
python3 tools/fingerprint_schema.py contract/historical_v2_schema.json
|
||||||
|
# → b49f64337ffafc0e73d46e009aa802cd0172d59421b7df61d3eb198775c1986d
|
||||||
|
|
||||||
|
# Semantic-Diff (expected vs actual)
|
||||||
|
python3 tools/compare_schema.py contract/historical_v2_schema.json <actual_model.json>
|
||||||
|
# → TOTAL_MISMATCH=0 bei Übereinstimmung; Exit 1 bei Drift (fail-closed)
|
||||||
|
|
||||||
|
# L0-SQL generieren
|
||||||
|
python3 tools/gen_l0_sql.py contract/historical_v2_schema.json --out /tmp/l0.sql
|
||||||
|
```
|
||||||
|
|
||||||
|
Alle Tools sind environment-neutral (keine absoluten Pfade, keine DB-Credentials,
|
||||||
|
keine mutierenden Production-SQL-Pfade). Read-only.
|
||||||
|
|
@ -0,0 +1,20 @@
|
||||||
|
# Historical V2 — Canonical Current Schema Contract (L0)
|
||||||
|
|
||||||
|
Diese Artefakte materialisieren den **CURRENT VERIFIED PRODUCTION SCHEMA CONTRACT**
|
||||||
|
als kanonische L0-Autorität (D1/D2 Remediation, forward-only).
|
||||||
|
|
||||||
|
- `historical_v2_schema.sql` — schema-only PostgreSQL-Contract (kein DML/Secrets/volatile Artefakte)
|
||||||
|
- `historical_v2_schema.json` — maschinenlesbares kanonisches Modell
|
||||||
|
- `historical_v2_schema.sha256` — deterministischer SHA-256-Fingerprint
|
||||||
|
- Kanonischer Fingerprint: `b49f64337ffafc0e73d46e009aa802cd0172d59421b7df61d3eb198775c1986d`
|
||||||
|
|
||||||
|
Dies ist KEINE historische Migrationskette. DO NOT INVENT HISTORY: der L0-Contract
|
||||||
|
beansprucht ausschliesslich CURRENT PRODUCTION TRUTH (§1).
|
||||||
|
Historische Artefakte (Wave1 `6983526da3ca51fd2ececffbef0ccbd66b048a3d11c228000011aaba3e1d6bc5`, WaveD `7aa4e2a10c98fa327e000a49f6a497efb5814c8808dba429dfebfc90c544a02b`) bleiben unverändert.
|
||||||
|
|
||||||
|
**Semantic Overrides (§4):**
|
||||||
|
- last_successful_chunk = timestamp with time zone (Production+Owner; WaveD-historical text bleibt immutable Evidence)
|
||||||
|
- interrupt_reason = text (preserved, origin unknown, do not remove)
|
||||||
|
- technical_quality = text nullable default 'VALID' (historical provenance debt OPEN)
|
||||||
|
|
||||||
|
**SoT:** origin/main = `e56567f7cfae68da70c11ece6f576e974488f343`
|
||||||
File diff suppressed because it is too large
Load diff
|
|
@ -0,0 +1 @@
|
||||||
|
b49f64337ffafc0e73d46e009aa802cd0172d59421b7df61d3eb198775c1986d
|
||||||
|
|
@ -0,0 +1,601 @@
|
||||||
|
-- ============================================================
|
||||||
|
-- CANONICAL CURRENT PRODUCTION SCHEMA CONTRACT (L0)
|
||||||
|
-- db: historical · schema: public
|
||||||
|
-- Auto-generated by gen_l0_sql.py from read-only capture
|
||||||
|
-- Modus: CURRENT VERIFIED PRODUCTION TRUTH — NICHT historische Migration
|
||||||
|
-- ============================================================
|
||||||
|
|
||||||
|
-- SEMANTIC OVERRIDES (Owner-Canonical)
|
||||||
|
-- last_successful_chunk = timestamp with time zone (Production + Owner decision)
|
||||||
|
-- interrupt_reason = text · PRESERVED CURRENT CONTRACT · origin unknown · do not remove
|
||||||
|
-- technical_quality = text · nullable · default 'VALID' · historical provenance debt OPEN
|
||||||
|
|
||||||
|
CREATE SEQUENCE IF NOT EXISTS "calendar_event_event_id_seq";
|
||||||
|
CREATE SEQUENCE IF NOT EXISTS "corporate_action_action_id_seq";
|
||||||
|
CREATE SEQUENCE IF NOT EXISTS "daily_quality_report_report_id_seq";
|
||||||
|
CREATE SEQUENCE IF NOT EXISTS "data_gap_gap_id_seq";
|
||||||
|
CREATE SEQUENCE IF NOT EXISTS "data_ingestion_run_ingestion_id_seq";
|
||||||
|
CREATE SEQUENCE IF NOT EXISTS "data_quality_issue_issue_id_seq";
|
||||||
|
CREATE SEQUENCE IF NOT EXISTS "dataset_dataset_id_seq";
|
||||||
|
CREATE SEQUENCE IF NOT EXISTS "dataset_version_dataset_version_id_seq";
|
||||||
|
CREATE SEQUENCE IF NOT EXISTS "derived_bar_derived_bar_id_seq";
|
||||||
|
CREATE SEQUENCE IF NOT EXISTS "historical_bar_bar_id_seq";
|
||||||
|
CREATE SEQUENCE IF NOT EXISTS "historical_tick_tick_id_seq";
|
||||||
|
CREATE SEQUENCE IF NOT EXISTS "ingestion_provenance_provenance_id_seq";
|
||||||
|
CREATE SEQUENCE IF NOT EXISTS "instrument_instrument_id_seq";
|
||||||
|
CREATE SEQUENCE IF NOT EXISTS "market_calendar_calendar_id_seq";
|
||||||
|
CREATE SEQUENCE IF NOT EXISTS "provider_provider_id_seq";
|
||||||
|
|
||||||
|
-- Table: calendar_event (9 cols)
|
||||||
|
CREATE TABLE IF NOT EXISTS "calendar_event" (
|
||||||
|
"event_id" bigint NOT NULL DEFAULT nextval('calendar_event_event_id_seq'::regclass),
|
||||||
|
"calendar_id" bigint NOT NULL,
|
||||||
|
"event_date" date NOT NULL,
|
||||||
|
"kind" text NOT NULL,
|
||||||
|
"reason" text,
|
||||||
|
"session_phase_override" text,
|
||||||
|
"source" text,
|
||||||
|
"notes" text,
|
||||||
|
"created_at" timestamp with time zone NOT NULL DEFAULT now()
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Table: corporate_action (9 cols)
|
||||||
|
CREATE TABLE IF NOT EXISTS "corporate_action" (
|
||||||
|
"action_id" bigint NOT NULL DEFAULT nextval('corporate_action_action_id_seq'::regclass),
|
||||||
|
"instrument_id" bigint NOT NULL,
|
||||||
|
"action_type" text NOT NULL,
|
||||||
|
"ex_date" date NOT NULL,
|
||||||
|
"ratio_num" double precision,
|
||||||
|
"ratio_denom" double precision,
|
||||||
|
"dividend" double precision,
|
||||||
|
"currency" text,
|
||||||
|
"provider_id" bigint
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Table: daily_quality_report (23 cols)
|
||||||
|
CREATE TABLE IF NOT EXISTS "daily_quality_report" (
|
||||||
|
"report_id" bigint NOT NULL DEFAULT nextval('daily_quality_report_report_id_seq'::regclass),
|
||||||
|
"instrument_id" bigint NOT NULL,
|
||||||
|
"provider_id" bigint NOT NULL,
|
||||||
|
"report_date" date NOT NULL,
|
||||||
|
"feed_type" text DEFAULT 'REFERENCE'::text,
|
||||||
|
"expected_m1" bigint,
|
||||||
|
"received_m1" bigint,
|
||||||
|
"eligible" bigint,
|
||||||
|
"excluded" bigint,
|
||||||
|
"conditional" bigint,
|
||||||
|
"stale_count" bigint,
|
||||||
|
"zero_volume" bigint,
|
||||||
|
"spread_median_price" double precision,
|
||||||
|
"spread_p95_price" double precision,
|
||||||
|
"spread_p99_price" double precision,
|
||||||
|
"gaps_count" bigint,
|
||||||
|
"coverage_ratio" double precision,
|
||||||
|
"session_anomalies" bigint,
|
||||||
|
"status" text NOT NULL,
|
||||||
|
"quality_model_version" text,
|
||||||
|
"session_model_version" text,
|
||||||
|
"calendar_version" text,
|
||||||
|
"generated_at" timestamp with time zone NOT NULL DEFAULT now()
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Table: data_gap (9 cols)
|
||||||
|
CREATE TABLE IF NOT EXISTS "data_gap" (
|
||||||
|
"gap_id" bigint NOT NULL DEFAULT nextval('data_gap_gap_id_seq'::regclass),
|
||||||
|
"provider_id" bigint NOT NULL,
|
||||||
|
"instrument_id" bigint NOT NULL,
|
||||||
|
"timeframe" text NOT NULL,
|
||||||
|
"start_utc" timestamp with time zone NOT NULL,
|
||||||
|
"end_utc" timestamp with time zone NOT NULL,
|
||||||
|
"kind" text NOT NULL,
|
||||||
|
"backfilled" boolean NOT NULL DEFAULT false,
|
||||||
|
"filled_at" timestamp with time zone
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Table: data_ingestion_run (29 cols)
|
||||||
|
CREATE TABLE IF NOT EXISTS "data_ingestion_run" (
|
||||||
|
"ingestion_id" bigint NOT NULL DEFAULT nextval('data_ingestion_run_ingestion_id_seq'::regclass),
|
||||||
|
"provider_id" bigint NOT NULL,
|
||||||
|
"instrument_id" bigint NOT NULL,
|
||||||
|
"timeframe" text NOT NULL,
|
||||||
|
"start_utc" timestamp with time zone,
|
||||||
|
"end_utc" timestamp with time zone,
|
||||||
|
"rows_inserted" bigint DEFAULT 0,
|
||||||
|
"rows_skipped" bigint DEFAULT 0,
|
||||||
|
"status" text NOT NULL DEFAULT 'RUNNING'::text,
|
||||||
|
"started_at" timestamp with time zone NOT NULL DEFAULT now(),
|
||||||
|
"finished_at" timestamp with time zone,
|
||||||
|
"checkpoint" jsonb,
|
||||||
|
"status_fsm" text DEFAULT 'PENDING'::text,
|
||||||
|
"rows_downloaded" bigint DEFAULT 0,
|
||||||
|
"rows_decoded" bigint DEFAULT 0,
|
||||||
|
"rows_valid" bigint DEFAULT 0,
|
||||||
|
"rows_suspect" bigint DEFAULT 0,
|
||||||
|
"rows_excluded" bigint DEFAULT 0,
|
||||||
|
"rows_derived" bigint DEFAULT 0,
|
||||||
|
"last_successful_chunk" timestamp with time zone,
|
||||||
|
"last_timestamp" timestamp with time zone,
|
||||||
|
"updated_at" timestamp with time zone DEFAULT now(),
|
||||||
|
"interrupt_reason" text,
|
||||||
|
"rows_existing" bigint DEFAULT 0,
|
||||||
|
"rows_deduplicated" bigint DEFAULT 0,
|
||||||
|
"rows_404" bigint DEFAULT 0,
|
||||||
|
"rows_503" bigint DEFAULT 0,
|
||||||
|
"rows_provider_unavailable" bigint DEFAULT 0,
|
||||||
|
"rows_failed" bigint DEFAULT 0
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Table: data_quality_issue (9 cols)
|
||||||
|
CREATE TABLE IF NOT EXISTS "data_quality_issue" (
|
||||||
|
"issue_id" bigint NOT NULL DEFAULT nextval('data_quality_issue_issue_id_seq'::regclass),
|
||||||
|
"instrument_id" bigint NOT NULL,
|
||||||
|
"timeframe" text NOT NULL,
|
||||||
|
"ts_utc" timestamp with time zone,
|
||||||
|
"severity" text NOT NULL,
|
||||||
|
"issue_type" text NOT NULL,
|
||||||
|
"detail" jsonb,
|
||||||
|
"resolved" boolean NOT NULL DEFAULT false,
|
||||||
|
"created_at" timestamp with time zone NOT NULL DEFAULT now()
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Table: dataset (6 cols)
|
||||||
|
CREATE TABLE IF NOT EXISTS "dataset" (
|
||||||
|
"dataset_id" bigint NOT NULL DEFAULT nextval('dataset_dataset_id_seq'::regclass),
|
||||||
|
"name" text NOT NULL,
|
||||||
|
"provider_id" bigint NOT NULL,
|
||||||
|
"feed_type" text NOT NULL,
|
||||||
|
"description" text,
|
||||||
|
"created_at" timestamp with time zone NOT NULL DEFAULT now()
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Table: dataset_version (23 cols)
|
||||||
|
CREATE TABLE IF NOT EXISTS "dataset_version" (
|
||||||
|
"dataset_version_id" bigint NOT NULL DEFAULT nextval('dataset_version_dataset_version_id_seq'::regclass),
|
||||||
|
"dataset_id" bigint NOT NULL,
|
||||||
|
"version_label" text NOT NULL,
|
||||||
|
"provider_id" bigint NOT NULL,
|
||||||
|
"feed_type" text NOT NULL,
|
||||||
|
"instrument_id" bigint NOT NULL,
|
||||||
|
"timeframe" text NOT NULL,
|
||||||
|
"start_utc" timestamp with time zone NOT NULL,
|
||||||
|
"end_utc" timestamp with time zone NOT NULL,
|
||||||
|
"raw_source" text NOT NULL,
|
||||||
|
"normalization_version" text NOT NULL,
|
||||||
|
"aggregation_version" text,
|
||||||
|
"quality_version" text NOT NULL,
|
||||||
|
"dataset_version_hash" text NOT NULL,
|
||||||
|
"data_as_of" timestamp with time zone NOT NULL,
|
||||||
|
"notes" text,
|
||||||
|
"created_at" timestamp with time zone NOT NULL DEFAULT now(),
|
||||||
|
"quality_model_version" text,
|
||||||
|
"stale_model_version" text,
|
||||||
|
"session_model_version" text,
|
||||||
|
"calendar_version" text,
|
||||||
|
"eligibility_model_version" text,
|
||||||
|
"is_fixture" boolean NOT NULL DEFAULT false
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Table: derived_bar (35 cols)
|
||||||
|
CREATE TABLE IF NOT EXISTS "derived_bar" (
|
||||||
|
"derived_bar_id" bigint NOT NULL DEFAULT nextval('derived_bar_derived_bar_id_seq'::regclass),
|
||||||
|
"provider_id" bigint NOT NULL,
|
||||||
|
"instrument_id" bigint NOT NULL,
|
||||||
|
"feed_type" text NOT NULL DEFAULT 'REFERENCE'::text,
|
||||||
|
"price_basis" text NOT NULL DEFAULT 'bid_ask'::text,
|
||||||
|
"timeframe" text NOT NULL,
|
||||||
|
"ts_utc" timestamp with time zone NOT NULL,
|
||||||
|
"source_timeframe" text NOT NULL,
|
||||||
|
"aggregation_version" text NOT NULL,
|
||||||
|
"bid_open" double precision,
|
||||||
|
"bid_high" double precision,
|
||||||
|
"bid_low" double precision,
|
||||||
|
"bid_close" double precision,
|
||||||
|
"ask_open" double precision,
|
||||||
|
"ask_high" double precision,
|
||||||
|
"ask_low" double precision,
|
||||||
|
"ask_close" double precision,
|
||||||
|
"mid_open" double precision,
|
||||||
|
"mid_high" double precision,
|
||||||
|
"mid_low" double precision,
|
||||||
|
"mid_close" double precision,
|
||||||
|
"volume" double precision,
|
||||||
|
"volume_type" text DEFAULT 'tick'::text,
|
||||||
|
"quality_status" text NOT NULL DEFAULT 'VALID'::text,
|
||||||
|
"quality_flags" _text,
|
||||||
|
"dataset_version_id" bigint NOT NULL,
|
||||||
|
"expected_bar_count" bigint,
|
||||||
|
"actual_bar_count" bigint,
|
||||||
|
"eligible_bar_count" bigint,
|
||||||
|
"excluded_bar_count" bigint,
|
||||||
|
"coverage_ratio" double precision,
|
||||||
|
"aggregation_quality" text,
|
||||||
|
"eligibility_model_version" text DEFAULT 'eligibility_v1'::text,
|
||||||
|
"session_model_version" text DEFAULT 'fx_session_v1'::text,
|
||||||
|
"calendar_version" text DEFAULT 'fx_cal_v1'::text
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Table: historical_bar (41 cols)
|
||||||
|
CREATE TABLE IF NOT EXISTS "historical_bar" (
|
||||||
|
"bar_id" bigint NOT NULL DEFAULT nextval('historical_bar_bar_id_seq'::regclass),
|
||||||
|
"provider_id" bigint NOT NULL,
|
||||||
|
"instrument_id" bigint NOT NULL,
|
||||||
|
"feed_type" text NOT NULL DEFAULT 'REFERENCE'::text,
|
||||||
|
"price_basis" text NOT NULL DEFAULT 'bid_ask'::text,
|
||||||
|
"timeframe" text NOT NULL,
|
||||||
|
"ts_utc" timestamp with time zone NOT NULL,
|
||||||
|
"raw_or_derived" text NOT NULL DEFAULT 'raw'::text,
|
||||||
|
"source_timeframe" text,
|
||||||
|
"aggregation_version" text,
|
||||||
|
"bid_open" double precision,
|
||||||
|
"bid_high" double precision,
|
||||||
|
"bid_low" double precision,
|
||||||
|
"bid_close" double precision,
|
||||||
|
"ask_open" double precision,
|
||||||
|
"ask_high" double precision,
|
||||||
|
"ask_low" double precision,
|
||||||
|
"ask_close" double precision,
|
||||||
|
"mid_open" double precision,
|
||||||
|
"mid_high" double precision,
|
||||||
|
"mid_low" double precision,
|
||||||
|
"mid_close" double precision,
|
||||||
|
"volume" double precision,
|
||||||
|
"volume_type" text DEFAULT 'tick'::text,
|
||||||
|
"quality_status" text NOT NULL DEFAULT 'VALID'::text,
|
||||||
|
"quality_flags" _text,
|
||||||
|
"source_timestamp" timestamp with time zone,
|
||||||
|
"ingested_at" timestamp with time zone NOT NULL DEFAULT now(),
|
||||||
|
"dataset_version_id" bigint NOT NULL,
|
||||||
|
"technical_quality" text DEFAULT 'VALID'::text,
|
||||||
|
"market_quality" text,
|
||||||
|
"session_state" text,
|
||||||
|
"session_phase" text,
|
||||||
|
"eligibility" text DEFAULT 'ELIGIBLE'::text,
|
||||||
|
"exclusion_reason" text,
|
||||||
|
"quality_model_version" text DEFAULT 'quality_v1'::text,
|
||||||
|
"session_model_version" text,
|
||||||
|
"calendar_version" text,
|
||||||
|
"eligibility_model_version" text DEFAULT 'eligibility_v1'::text,
|
||||||
|
"effective_eligibility" text,
|
||||||
|
"stale_model_version" text
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Table: historical_tick (10 cols)
|
||||||
|
CREATE TABLE IF NOT EXISTS "historical_tick" (
|
||||||
|
"tick_id" bigint NOT NULL DEFAULT nextval('historical_tick_tick_id_seq'::regclass),
|
||||||
|
"provider_id" bigint NOT NULL,
|
||||||
|
"instrument_id" bigint NOT NULL,
|
||||||
|
"feed_type" text NOT NULL DEFAULT 'REFERENCE'::text,
|
||||||
|
"ts_utc" timestamp with time zone NOT NULL,
|
||||||
|
"bid" double precision,
|
||||||
|
"ask" double precision,
|
||||||
|
"last" double precision,
|
||||||
|
"volume" double precision,
|
||||||
|
"dataset_version_id" bigint NOT NULL
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Table: ingestion_provenance (41 cols)
|
||||||
|
CREATE TABLE IF NOT EXISTS "ingestion_provenance" (
|
||||||
|
"provenance_id" bigint NOT NULL,
|
||||||
|
"ingestion_run_id" bigint NOT NULL,
|
||||||
|
"provider_id" bigint NOT NULL,
|
||||||
|
"provider_name_snapshot" text NOT NULL,
|
||||||
|
"provider_environment_class" text NOT NULL,
|
||||||
|
"provider_dataset" text NOT NULL,
|
||||||
|
"provider_feed" text NOT NULL,
|
||||||
|
"provider_instrument_id" text NOT NULL,
|
||||||
|
"canonical_instrument_id" bigint NOT NULL,
|
||||||
|
"asset_class" text NOT NULL,
|
||||||
|
"request_start" timestamp with time zone NOT NULL,
|
||||||
|
"request_end" timestamp with time zone NOT NULL,
|
||||||
|
"actual_start" timestamp with time zone NOT NULL,
|
||||||
|
"actual_end" timestamp with time zone NOT NULL,
|
||||||
|
"request_timestamp" timestamp with time zone NOT NULL,
|
||||||
|
"response_timestamp" timestamp with time zone NOT NULL,
|
||||||
|
"http_status" integer NOT NULL,
|
||||||
|
"provider_metadata" jsonb,
|
||||||
|
"provider_version" text,
|
||||||
|
"raw_payload_sha256" text NOT NULL,
|
||||||
|
"raw_content_length" bigint NOT NULL,
|
||||||
|
"compression" text NOT NULL,
|
||||||
|
"content_type" text,
|
||||||
|
"parser_version" text NOT NULL,
|
||||||
|
"adapter_version" text NOT NULL,
|
||||||
|
"normalization_version" text NOT NULL,
|
||||||
|
"timezone_source" text NOT NULL,
|
||||||
|
"timezone_target" text NOT NULL,
|
||||||
|
"timezone_transform_version" text NOT NULL,
|
||||||
|
"price_type" text NOT NULL,
|
||||||
|
"granularity" text NOT NULL,
|
||||||
|
"correction_status" text NOT NULL DEFAULT 'ORIGINAL'::text,
|
||||||
|
"license_class" text NOT NULL DEFAULT 'UNKNOWN'::text,
|
||||||
|
"software_git_commit" text NOT NULL,
|
||||||
|
"software_image_id" text NOT NULL,
|
||||||
|
"software_build_input_sha" text,
|
||||||
|
"request_identity_hash" text NOT NULL,
|
||||||
|
"supersedes_provenance_id" bigint,
|
||||||
|
"raw_storage_ref" text,
|
||||||
|
"extra" jsonb,
|
||||||
|
"created_at" timestamp with time zone NOT NULL DEFAULT now()
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Table: instrument (13 cols)
|
||||||
|
CREATE TABLE IF NOT EXISTS "instrument" (
|
||||||
|
"instrument_id" bigint NOT NULL DEFAULT nextval('instrument_instrument_id_seq'::regclass),
|
||||||
|
"symbol" text NOT NULL,
|
||||||
|
"asset_class" text NOT NULL,
|
||||||
|
"name" text,
|
||||||
|
"currency" text,
|
||||||
|
"exchange" text,
|
||||||
|
"mic" text,
|
||||||
|
"isin" text,
|
||||||
|
"tick_size" double precision,
|
||||||
|
"contract_size" double precision,
|
||||||
|
"min_deal_size" double precision,
|
||||||
|
"active" boolean NOT NULL DEFAULT true,
|
||||||
|
"created_at" timestamp with time zone NOT NULL DEFAULT now()
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Table: market_calendar (13 cols)
|
||||||
|
CREATE TABLE IF NOT EXISTS "market_calendar" (
|
||||||
|
"calendar_id" bigint NOT NULL DEFAULT nextval('market_calendar_calendar_id_seq'::regclass),
|
||||||
|
"calendar_name" text NOT NULL,
|
||||||
|
"asset_class" text NOT NULL,
|
||||||
|
"venue_exchange" text,
|
||||||
|
"timezone" text NOT NULL,
|
||||||
|
"week_open_utc" text,
|
||||||
|
"week_close_utc" text,
|
||||||
|
"rollover_utc" text,
|
||||||
|
"session_open_utc" text,
|
||||||
|
"session_close_utc" text,
|
||||||
|
"notes" text,
|
||||||
|
"calendar_version" text NOT NULL,
|
||||||
|
"created_at" timestamp with time zone NOT NULL DEFAULT now()
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Table: provider (6 cols)
|
||||||
|
CREATE TABLE IF NOT EXISTS "provider" (
|
||||||
|
"provider_id" bigint NOT NULL DEFAULT nextval('provider_provider_id_seq'::regclass),
|
||||||
|
"provider_key" text NOT NULL,
|
||||||
|
"name" text NOT NULL,
|
||||||
|
"data_type" text NOT NULL,
|
||||||
|
"pricing_plan" text,
|
||||||
|
"license_terms" text
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Table: provider_instrument (6 cols)
|
||||||
|
CREATE TABLE IF NOT EXISTS "provider_instrument" (
|
||||||
|
"provider_id" bigint NOT NULL,
|
||||||
|
"instrument_id" bigint NOT NULL,
|
||||||
|
"provider_symbol" text NOT NULL,
|
||||||
|
"broker_epic" text,
|
||||||
|
"extra_metadata" jsonb,
|
||||||
|
"active" boolean NOT NULL DEFAULT true
|
||||||
|
);
|
||||||
|
|
||||||
|
ALTER TABLE "calendar_event" ADD CONSTRAINT "calendar_event_pkey" PRIMARY KEY ("event_id");
|
||||||
|
ALTER TABLE "corporate_action" ADD CONSTRAINT "corporate_action_pkey" PRIMARY KEY ("action_id");
|
||||||
|
ALTER TABLE "daily_quality_report" ADD CONSTRAINT "daily_quality_report_pkey" PRIMARY KEY ("report_id");
|
||||||
|
ALTER TABLE "data_gap" ADD CONSTRAINT "data_gap_pkey" PRIMARY KEY ("gap_id");
|
||||||
|
ALTER TABLE "data_ingestion_run" ADD CONSTRAINT "data_ingestion_run_pkey" PRIMARY KEY ("ingestion_id");
|
||||||
|
ALTER TABLE "data_quality_issue" ADD CONSTRAINT "data_quality_issue_pkey" PRIMARY KEY ("issue_id");
|
||||||
|
ALTER TABLE "dataset" ADD CONSTRAINT "dataset_pkey" PRIMARY KEY ("dataset_id");
|
||||||
|
ALTER TABLE "dataset_version" ADD CONSTRAINT "dataset_version_pkey" PRIMARY KEY ("dataset_version_id");
|
||||||
|
ALTER TABLE "derived_bar" ADD CONSTRAINT "derived_bar_pkey" PRIMARY KEY ("derived_bar_id");
|
||||||
|
ALTER TABLE "historical_bar" ADD CONSTRAINT "historical_bar_pkey" PRIMARY KEY ("bar_id");
|
||||||
|
ALTER TABLE "historical_tick" ADD CONSTRAINT "historical_tick_pkey" PRIMARY KEY ("tick_id");
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "ingestion_provenance_pkey" PRIMARY KEY ("provenance_id");
|
||||||
|
ALTER TABLE "instrument" ADD CONSTRAINT "instrument_pkey" PRIMARY KEY ("instrument_id");
|
||||||
|
ALTER TABLE "market_calendar" ADD CONSTRAINT "market_calendar_pkey" PRIMARY KEY ("calendar_id");
|
||||||
|
ALTER TABLE "provider" ADD CONSTRAINT "provider_pkey" PRIMARY KEY ("provider_id");
|
||||||
|
ALTER TABLE "provider_instrument" ADD CONSTRAINT "provider_instrument_pkey" PRIMARY KEY ("provider_id", "instrument_id");
|
||||||
|
|
||||||
|
ALTER TABLE "calendar_event" ADD CONSTRAINT "calendar_event_calendar_id_event_date_kind_key" UNIQUE ("calendar_id", "event_date", "kind");
|
||||||
|
ALTER TABLE "corporate_action" ADD CONSTRAINT "corporate_action_instrument_id_action_type_ex_date_provider_key" UNIQUE ("instrument_id", "action_type", "ex_date", "provider_id");
|
||||||
|
ALTER TABLE "daily_quality_report" ADD CONSTRAINT "daily_quality_report_instrument_id_provider_id_report_date__key" UNIQUE ("instrument_id", "provider_id", "report_date", "feed_type");
|
||||||
|
ALTER TABLE "data_gap" ADD CONSTRAINT "data_gap_unique" UNIQUE ("provider_id", "instrument_id", "timeframe", "start_utc", "end_utc", "kind");
|
||||||
|
ALTER TABLE "data_ingestion_run" ADD CONSTRAINT "data_ingestion_run_provider_id_instrument_id_timeframe_star_key" UNIQUE ("provider_id", "instrument_id", "timeframe", "started_at");
|
||||||
|
ALTER TABLE "dataset" ADD CONSTRAINT "dataset_name_key" UNIQUE ("name");
|
||||||
|
ALTER TABLE "dataset_version" ADD CONSTRAINT "dataset_version_dataset_version_hash_key" UNIQUE ("dataset_version_hash");
|
||||||
|
ALTER TABLE "derived_bar" ADD CONSTRAINT "derived_bar_provider_id_instrument_id_feed_type_price_basis_key" UNIQUE ("provider_id", "instrument_id", "feed_type", "price_basis", "timeframe", "ts_utc", "aggregation_version");
|
||||||
|
ALTER TABLE "historical_bar" ADD CONSTRAINT "historical_bar_provider_id_instrument_id_feed_type_price_ba_key" UNIQUE ("provider_id", "instrument_id", "feed_type", "price_basis", "timeframe", "ts_utc", "raw_or_derived");
|
||||||
|
ALTER TABLE "historical_tick" ADD CONSTRAINT "historical_tick_provider_id_instrument_id_feed_type_ts_utc__key" UNIQUE ("provider_id", "instrument_id", "feed_type", "ts_utc", "bid", "ask");
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "ingestion_provenance_ingestion_run_id_key" UNIQUE ("ingestion_run_id");
|
||||||
|
ALTER TABLE "instrument" ADD CONSTRAINT "instrument_symbol_key" UNIQUE ("symbol");
|
||||||
|
ALTER TABLE "market_calendar" ADD CONSTRAINT "market_calendar_calendar_name_key" UNIQUE ("calendar_name");
|
||||||
|
ALTER TABLE "provider" ADD CONSTRAINT "provider_provider_key_key" UNIQUE ("provider_key");
|
||||||
|
ALTER TABLE "provider_instrument" ADD CONSTRAINT "provider_instrument_provider_id_provider_symbol_key" UNIQUE ("provider_id", "provider_symbol");
|
||||||
|
|
||||||
|
ALTER TABLE "calendar_event" ADD CONSTRAINT "calendar_event_calendar_id_fkey" FOREIGN KEY ("calendar_id") REFERENCES "market_calendar" ("calendar_id");
|
||||||
|
ALTER TABLE "corporate_action" ADD CONSTRAINT "corporate_action_instrument_id_fkey" FOREIGN KEY ("instrument_id") REFERENCES "instrument" ("instrument_id");
|
||||||
|
ALTER TABLE "corporate_action" ADD CONSTRAINT "corporate_action_provider_id_fkey" FOREIGN KEY ("provider_id") REFERENCES "provider" ("provider_id");
|
||||||
|
ALTER TABLE "daily_quality_report" ADD CONSTRAINT "daily_quality_report_instrument_id_fkey" FOREIGN KEY ("instrument_id") REFERENCES "instrument" ("instrument_id");
|
||||||
|
ALTER TABLE "daily_quality_report" ADD CONSTRAINT "daily_quality_report_provider_id_fkey" FOREIGN KEY ("provider_id") REFERENCES "provider" ("provider_id");
|
||||||
|
ALTER TABLE "data_gap" ADD CONSTRAINT "data_gap_instrument_id_fkey" FOREIGN KEY ("instrument_id") REFERENCES "instrument" ("instrument_id");
|
||||||
|
ALTER TABLE "data_gap" ADD CONSTRAINT "data_gap_provider_id_fkey" FOREIGN KEY ("provider_id") REFERENCES "provider" ("provider_id");
|
||||||
|
ALTER TABLE "data_ingestion_run" ADD CONSTRAINT "data_ingestion_run_instrument_id_fkey" FOREIGN KEY ("instrument_id") REFERENCES "instrument" ("instrument_id");
|
||||||
|
ALTER TABLE "data_ingestion_run" ADD CONSTRAINT "data_ingestion_run_provider_id_fkey" FOREIGN KEY ("provider_id") REFERENCES "provider" ("provider_id");
|
||||||
|
ALTER TABLE "data_quality_issue" ADD CONSTRAINT "data_quality_issue_instrument_id_fkey" FOREIGN KEY ("instrument_id") REFERENCES "instrument" ("instrument_id");
|
||||||
|
ALTER TABLE "dataset" ADD CONSTRAINT "dataset_provider_id_fkey" FOREIGN KEY ("provider_id") REFERENCES "provider" ("provider_id");
|
||||||
|
ALTER TABLE "dataset_version" ADD CONSTRAINT "dataset_version_dataset_id_fkey" FOREIGN KEY ("dataset_id") REFERENCES "dataset" ("dataset_id");
|
||||||
|
ALTER TABLE "dataset_version" ADD CONSTRAINT "dataset_version_instrument_id_fkey" FOREIGN KEY ("instrument_id") REFERENCES "instrument" ("instrument_id");
|
||||||
|
ALTER TABLE "dataset_version" ADD CONSTRAINT "dataset_version_provider_id_fkey" FOREIGN KEY ("provider_id") REFERENCES "provider" ("provider_id");
|
||||||
|
ALTER TABLE "derived_bar" ADD CONSTRAINT "derived_bar_dataset_version_id_fkey" FOREIGN KEY ("dataset_version_id") REFERENCES "dataset_version" ("dataset_version_id");
|
||||||
|
ALTER TABLE "derived_bar" ADD CONSTRAINT "derived_bar_instrument_id_fkey" FOREIGN KEY ("instrument_id") REFERENCES "instrument" ("instrument_id");
|
||||||
|
ALTER TABLE "derived_bar" ADD CONSTRAINT "derived_bar_provider_id_fkey" FOREIGN KEY ("provider_id") REFERENCES "provider" ("provider_id");
|
||||||
|
ALTER TABLE "historical_bar" ADD CONSTRAINT "historical_bar_dataset_version_id_fkey" FOREIGN KEY ("dataset_version_id") REFERENCES "dataset_version" ("dataset_version_id");
|
||||||
|
ALTER TABLE "historical_bar" ADD CONSTRAINT "historical_bar_instrument_id_fkey" FOREIGN KEY ("instrument_id") REFERENCES "instrument" ("instrument_id");
|
||||||
|
ALTER TABLE "historical_bar" ADD CONSTRAINT "historical_bar_provider_id_fkey" FOREIGN KEY ("provider_id") REFERENCES "provider" ("provider_id");
|
||||||
|
ALTER TABLE "historical_tick" ADD CONSTRAINT "historical_tick_dataset_version_id_fkey" FOREIGN KEY ("dataset_version_id") REFERENCES "dataset_version" ("dataset_version_id");
|
||||||
|
ALTER TABLE "historical_tick" ADD CONSTRAINT "historical_tick_instrument_id_fkey" FOREIGN KEY ("instrument_id") REFERENCES "instrument" ("instrument_id");
|
||||||
|
ALTER TABLE "historical_tick" ADD CONSTRAINT "historical_tick_provider_id_fkey" FOREIGN KEY ("provider_id") REFERENCES "provider" ("provider_id");
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "ingestion_provenance_canonical_instrument_id_fkey" FOREIGN KEY ("canonical_instrument_id") REFERENCES "instrument" ("instrument_id");
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "ingestion_provenance_provider_id_fkey" FOREIGN KEY ("provider_id") REFERENCES "provider" ("provider_id");
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "ingestion_provenance_supersedes_fkey" FOREIGN KEY ("supersedes_provenance_id") REFERENCES "ingestion_provenance" ("provenance_id");
|
||||||
|
ALTER TABLE "provider_instrument" ADD CONSTRAINT "provider_instrument_instrument_id_fkey" FOREIGN KEY ("instrument_id") REFERENCES "instrument" ("instrument_id");
|
||||||
|
ALTER TABLE "provider_instrument" ADD CONSTRAINT "provider_instrument_provider_id_fkey" FOREIGN KEY ("provider_id") REFERENCES "provider" ("provider_id");
|
||||||
|
|
||||||
|
ALTER TABLE "calendar_event" ADD CONSTRAINT "2200_16679_1_not_null" CHECK (event_id IS NOT NULL);
|
||||||
|
ALTER TABLE "calendar_event" ADD CONSTRAINT "2200_16679_2_not_null" CHECK (calendar_id IS NOT NULL);
|
||||||
|
ALTER TABLE "calendar_event" ADD CONSTRAINT "2200_16679_3_not_null" CHECK (event_date IS NOT NULL);
|
||||||
|
ALTER TABLE "calendar_event" ADD CONSTRAINT "2200_16679_4_not_null" CHECK (kind IS NOT NULL);
|
||||||
|
ALTER TABLE "calendar_event" ADD CONSTRAINT "2200_16679_9_not_null" CHECK (created_at IS NOT NULL);
|
||||||
|
ALTER TABLE "corporate_action" ADD CONSTRAINT "2200_16567_1_not_null" CHECK (action_id IS NOT NULL);
|
||||||
|
ALTER TABLE "corporate_action" ADD CONSTRAINT "2200_16567_2_not_null" CHECK (instrument_id IS NOT NULL);
|
||||||
|
ALTER TABLE "corporate_action" ADD CONSTRAINT "2200_16567_3_not_null" CHECK (action_type IS NOT NULL);
|
||||||
|
ALTER TABLE "corporate_action" ADD CONSTRAINT "2200_16567_4_not_null" CHECK (ex_date IS NOT NULL);
|
||||||
|
ALTER TABLE "daily_quality_report" ADD CONSTRAINT "2200_16696_19_not_null" CHECK (status IS NOT NULL);
|
||||||
|
ALTER TABLE "daily_quality_report" ADD CONSTRAINT "2200_16696_1_not_null" CHECK (report_id IS NOT NULL);
|
||||||
|
ALTER TABLE "daily_quality_report" ADD CONSTRAINT "2200_16696_23_not_null" CHECK (generated_at IS NOT NULL);
|
||||||
|
ALTER TABLE "daily_quality_report" ADD CONSTRAINT "2200_16696_2_not_null" CHECK (instrument_id IS NOT NULL);
|
||||||
|
ALTER TABLE "daily_quality_report" ADD CONSTRAINT "2200_16696_3_not_null" CHECK (provider_id IS NOT NULL);
|
||||||
|
ALTER TABLE "daily_quality_report" ADD CONSTRAINT "2200_16696_4_not_null" CHECK (report_date IS NOT NULL);
|
||||||
|
ALTER TABLE "data_gap" ADD CONSTRAINT "2200_16613_1_not_null" CHECK (gap_id IS NOT NULL);
|
||||||
|
ALTER TABLE "data_gap" ADD CONSTRAINT "2200_16613_2_not_null" CHECK (provider_id IS NOT NULL);
|
||||||
|
ALTER TABLE "data_gap" ADD CONSTRAINT "2200_16613_3_not_null" CHECK (instrument_id IS NOT NULL);
|
||||||
|
ALTER TABLE "data_gap" ADD CONSTRAINT "2200_16613_4_not_null" CHECK (timeframe IS NOT NULL);
|
||||||
|
ALTER TABLE "data_gap" ADD CONSTRAINT "2200_16613_5_not_null" CHECK (start_utc IS NOT NULL);
|
||||||
|
ALTER TABLE "data_gap" ADD CONSTRAINT "2200_16613_6_not_null" CHECK (end_utc IS NOT NULL);
|
||||||
|
ALTER TABLE "data_gap" ADD CONSTRAINT "2200_16613_7_not_null" CHECK (kind IS NOT NULL);
|
||||||
|
ALTER TABLE "data_gap" ADD CONSTRAINT "2200_16613_8_not_null" CHECK (backfilled IS NOT NULL);
|
||||||
|
ALTER TABLE "data_ingestion_run" ADD CONSTRAINT "2200_16588_10_not_null" CHECK (started_at IS NOT NULL);
|
||||||
|
ALTER TABLE "data_ingestion_run" ADD CONSTRAINT "2200_16588_1_not_null" CHECK (ingestion_id IS NOT NULL);
|
||||||
|
ALTER TABLE "data_ingestion_run" ADD CONSTRAINT "2200_16588_2_not_null" CHECK (provider_id IS NOT NULL);
|
||||||
|
ALTER TABLE "data_ingestion_run" ADD CONSTRAINT "2200_16588_3_not_null" CHECK (instrument_id IS NOT NULL);
|
||||||
|
ALTER TABLE "data_ingestion_run" ADD CONSTRAINT "2200_16588_4_not_null" CHECK (timeframe IS NOT NULL);
|
||||||
|
ALTER TABLE "data_ingestion_run" ADD CONSTRAINT "2200_16588_9_not_null" CHECK (status IS NOT NULL);
|
||||||
|
ALTER TABLE "data_quality_issue" ADD CONSTRAINT "2200_16633_1_not_null" CHECK (issue_id IS NOT NULL);
|
||||||
|
ALTER TABLE "data_quality_issue" ADD CONSTRAINT "2200_16633_2_not_null" CHECK (instrument_id IS NOT NULL);
|
||||||
|
ALTER TABLE "data_quality_issue" ADD CONSTRAINT "2200_16633_3_not_null" CHECK (timeframe IS NOT NULL);
|
||||||
|
ALTER TABLE "data_quality_issue" ADD CONSTRAINT "2200_16633_5_not_null" CHECK (severity IS NOT NULL);
|
||||||
|
ALTER TABLE "data_quality_issue" ADD CONSTRAINT "2200_16633_6_not_null" CHECK (issue_type IS NOT NULL);
|
||||||
|
ALTER TABLE "data_quality_issue" ADD CONSTRAINT "2200_16633_8_not_null" CHECK (resolved IS NOT NULL);
|
||||||
|
ALTER TABLE "data_quality_issue" ADD CONSTRAINT "2200_16633_9_not_null" CHECK (created_at IS NOT NULL);
|
||||||
|
ALTER TABLE "dataset" ADD CONSTRAINT "2200_16430_1_not_null" CHECK (dataset_id IS NOT NULL);
|
||||||
|
ALTER TABLE "dataset" ADD CONSTRAINT "2200_16430_2_not_null" CHECK (name IS NOT NULL);
|
||||||
|
ALTER TABLE "dataset" ADD CONSTRAINT "2200_16430_3_not_null" CHECK (provider_id IS NOT NULL);
|
||||||
|
ALTER TABLE "dataset" ADD CONSTRAINT "2200_16430_4_not_null" CHECK (feed_type IS NOT NULL);
|
||||||
|
ALTER TABLE "dataset" ADD CONSTRAINT "2200_16430_6_not_null" CHECK (created_at IS NOT NULL);
|
||||||
|
ALTER TABLE "dataset_version" ADD CONSTRAINT "2200_16447_10_not_null" CHECK (raw_source IS NOT NULL);
|
||||||
|
ALTER TABLE "dataset_version" ADD CONSTRAINT "2200_16447_11_not_null" CHECK (normalization_version IS NOT NULL);
|
||||||
|
ALTER TABLE "dataset_version" ADD CONSTRAINT "2200_16447_13_not_null" CHECK (quality_version IS NOT NULL);
|
||||||
|
ALTER TABLE "dataset_version" ADD CONSTRAINT "2200_16447_14_not_null" CHECK (dataset_version_hash IS NOT NULL);
|
||||||
|
ALTER TABLE "dataset_version" ADD CONSTRAINT "2200_16447_15_not_null" CHECK (data_as_of IS NOT NULL);
|
||||||
|
ALTER TABLE "dataset_version" ADD CONSTRAINT "2200_16447_17_not_null" CHECK (created_at IS NOT NULL);
|
||||||
|
ALTER TABLE "dataset_version" ADD CONSTRAINT "2200_16447_1_not_null" CHECK (dataset_version_id IS NOT NULL);
|
||||||
|
ALTER TABLE "dataset_version" ADD CONSTRAINT "2200_16447_23_not_null" CHECK (is_fixture IS NOT NULL);
|
||||||
|
ALTER TABLE "dataset_version" ADD CONSTRAINT "2200_16447_2_not_null" CHECK (dataset_id IS NOT NULL);
|
||||||
|
ALTER TABLE "dataset_version" ADD CONSTRAINT "2200_16447_3_not_null" CHECK (version_label IS NOT NULL);
|
||||||
|
ALTER TABLE "dataset_version" ADD CONSTRAINT "2200_16447_4_not_null" CHECK (provider_id IS NOT NULL);
|
||||||
|
ALTER TABLE "dataset_version" ADD CONSTRAINT "2200_16447_5_not_null" CHECK (feed_type IS NOT NULL);
|
||||||
|
ALTER TABLE "dataset_version" ADD CONSTRAINT "2200_16447_6_not_null" CHECK (instrument_id IS NOT NULL);
|
||||||
|
ALTER TABLE "dataset_version" ADD CONSTRAINT "2200_16447_7_not_null" CHECK (timeframe IS NOT NULL);
|
||||||
|
ALTER TABLE "dataset_version" ADD CONSTRAINT "2200_16447_8_not_null" CHECK (start_utc IS NOT NULL);
|
||||||
|
ALTER TABLE "dataset_version" ADD CONSTRAINT "2200_16447_9_not_null" CHECK (end_utc IS NOT NULL);
|
||||||
|
ALTER TABLE "derived_bar" ADD CONSTRAINT "2200_16508_1_not_null" CHECK (derived_bar_id IS NOT NULL);
|
||||||
|
ALTER TABLE "derived_bar" ADD CONSTRAINT "2200_16508_24_not_null" CHECK (quality_status IS NOT NULL);
|
||||||
|
ALTER TABLE "derived_bar" ADD CONSTRAINT "2200_16508_26_not_null" CHECK (dataset_version_id IS NOT NULL);
|
||||||
|
ALTER TABLE "derived_bar" ADD CONSTRAINT "2200_16508_2_not_null" CHECK (provider_id IS NOT NULL);
|
||||||
|
ALTER TABLE "derived_bar" ADD CONSTRAINT "2200_16508_3_not_null" CHECK (instrument_id IS NOT NULL);
|
||||||
|
ALTER TABLE "derived_bar" ADD CONSTRAINT "2200_16508_4_not_null" CHECK (feed_type IS NOT NULL);
|
||||||
|
ALTER TABLE "derived_bar" ADD CONSTRAINT "2200_16508_5_not_null" CHECK (price_basis IS NOT NULL);
|
||||||
|
ALTER TABLE "derived_bar" ADD CONSTRAINT "2200_16508_6_not_null" CHECK (timeframe IS NOT NULL);
|
||||||
|
ALTER TABLE "derived_bar" ADD CONSTRAINT "2200_16508_7_not_null" CHECK (ts_utc IS NOT NULL);
|
||||||
|
ALTER TABLE "derived_bar" ADD CONSTRAINT "2200_16508_8_not_null" CHECK (source_timeframe IS NOT NULL);
|
||||||
|
ALTER TABLE "derived_bar" ADD CONSTRAINT "2200_16508_9_not_null" CHECK (aggregation_version IS NOT NULL);
|
||||||
|
ALTER TABLE "historical_bar" ADD CONSTRAINT "2200_16474_1_not_null" CHECK (bar_id IS NOT NULL);
|
||||||
|
ALTER TABLE "historical_bar" ADD CONSTRAINT "2200_16474_25_not_null" CHECK (quality_status IS NOT NULL);
|
||||||
|
ALTER TABLE "historical_bar" ADD CONSTRAINT "2200_16474_28_not_null" CHECK (ingested_at IS NOT NULL);
|
||||||
|
ALTER TABLE "historical_bar" ADD CONSTRAINT "2200_16474_29_not_null" CHECK (dataset_version_id IS NOT NULL);
|
||||||
|
ALTER TABLE "historical_bar" ADD CONSTRAINT "2200_16474_2_not_null" CHECK (provider_id IS NOT NULL);
|
||||||
|
ALTER TABLE "historical_bar" ADD CONSTRAINT "2200_16474_3_not_null" CHECK (instrument_id IS NOT NULL);
|
||||||
|
ALTER TABLE "historical_bar" ADD CONSTRAINT "2200_16474_4_not_null" CHECK (feed_type IS NOT NULL);
|
||||||
|
ALTER TABLE "historical_bar" ADD CONSTRAINT "2200_16474_5_not_null" CHECK (price_basis IS NOT NULL);
|
||||||
|
ALTER TABLE "historical_bar" ADD CONSTRAINT "2200_16474_6_not_null" CHECK (timeframe IS NOT NULL);
|
||||||
|
ALTER TABLE "historical_bar" ADD CONSTRAINT "2200_16474_7_not_null" CHECK (ts_utc IS NOT NULL);
|
||||||
|
ALTER TABLE "historical_bar" ADD CONSTRAINT "2200_16474_8_not_null" CHECK (raw_or_derived IS NOT NULL);
|
||||||
|
ALTER TABLE "historical_tick" ADD CONSTRAINT "2200_16539_10_not_null" CHECK (dataset_version_id IS NOT NULL);
|
||||||
|
ALTER TABLE "historical_tick" ADD CONSTRAINT "2200_16539_1_not_null" CHECK (tick_id IS NOT NULL);
|
||||||
|
ALTER TABLE "historical_tick" ADD CONSTRAINT "2200_16539_2_not_null" CHECK (provider_id IS NOT NULL);
|
||||||
|
ALTER TABLE "historical_tick" ADD CONSTRAINT "2200_16539_3_not_null" CHECK (instrument_id IS NOT NULL);
|
||||||
|
ALTER TABLE "historical_tick" ADD CONSTRAINT "2200_16539_4_not_null" CHECK (feed_type IS NOT NULL);
|
||||||
|
ALTER TABLE "historical_tick" ADD CONSTRAINT "2200_16539_5_not_null" CHECK (ts_utc IS NOT NULL);
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "2200_16741_10_not_null" CHECK (asset_class IS NOT NULL);
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "2200_16741_11_not_null" CHECK (request_start IS NOT NULL);
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "2200_16741_12_not_null" CHECK (request_end IS NOT NULL);
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "2200_16741_13_not_null" CHECK (actual_start IS NOT NULL);
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "2200_16741_14_not_null" CHECK (actual_end IS NOT NULL);
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "2200_16741_15_not_null" CHECK (request_timestamp IS NOT NULL);
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "2200_16741_16_not_null" CHECK (response_timestamp IS NOT NULL);
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "2200_16741_17_not_null" CHECK (http_status IS NOT NULL);
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "2200_16741_1_not_null" CHECK (provenance_id IS NOT NULL);
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "2200_16741_20_not_null" CHECK (raw_payload_sha256 IS NOT NULL);
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "2200_16741_21_not_null" CHECK (raw_content_length IS NOT NULL);
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "2200_16741_22_not_null" CHECK (compression IS NOT NULL);
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "2200_16741_24_not_null" CHECK (parser_version IS NOT NULL);
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "2200_16741_25_not_null" CHECK (adapter_version IS NOT NULL);
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "2200_16741_26_not_null" CHECK (normalization_version IS NOT NULL);
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "2200_16741_27_not_null" CHECK (timezone_source IS NOT NULL);
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "2200_16741_28_not_null" CHECK (timezone_target IS NOT NULL);
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "2200_16741_29_not_null" CHECK (timezone_transform_version IS NOT NULL);
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "2200_16741_2_not_null" CHECK (ingestion_run_id IS NOT NULL);
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "2200_16741_30_not_null" CHECK (price_type IS NOT NULL);
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "2200_16741_31_not_null" CHECK (granularity IS NOT NULL);
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "2200_16741_32_not_null" CHECK (correction_status IS NOT NULL);
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "2200_16741_33_not_null" CHECK (license_class IS NOT NULL);
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "2200_16741_34_not_null" CHECK (software_git_commit IS NOT NULL);
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "2200_16741_35_not_null" CHECK (software_image_id IS NOT NULL);
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "2200_16741_37_not_null" CHECK (request_identity_hash IS NOT NULL);
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "2200_16741_3_not_null" CHECK (provider_id IS NOT NULL);
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "2200_16741_41_not_null" CHECK (created_at IS NOT NULL);
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "2200_16741_4_not_null" CHECK (provider_name_snapshot IS NOT NULL);
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "2200_16741_5_not_null" CHECK (provider_environment_class IS NOT NULL);
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "2200_16741_6_not_null" CHECK (provider_dataset IS NOT NULL);
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "2200_16741_7_not_null" CHECK (provider_feed IS NOT NULL);
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "2200_16741_8_not_null" CHECK (provider_instrument_id IS NOT NULL);
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "2200_16741_9_not_null" CHECK (canonical_instrument_id IS NOT NULL);
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "chk_prov_actual_range" CHECK (((actual_end >= actual_start)));
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "chk_prov_compression" CHECK (((compression = ANY (ARRAY['none'::text, 'lzma'::text, 'gzip'::text, 'zstd'::text, 'deflate'::text, 'unknown'::text]))));
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "chk_prov_content_length" CHECK (((raw_content_length >= 0)));
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "chk_prov_correction" CHECK (((correction_status = ANY (ARRAY['ORIGINAL'::text, 'CORRECTED'::text, 'SUPERSEDED'::text, 'UNKNOWN'::text]))));
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "chk_prov_env_class" CHECK (((provider_environment_class = ANY (ARRAY['PROD'::text, 'SANDBOX'::text, 'TEST'::text, 'UNKNOWN'::text]))));
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "chk_prov_http_status" CHECK ((((http_status >= 100) AND (http_status <= 599))));
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "chk_prov_license" CHECK (((license_class = ANY (ARRAY['UNKNOWN'::text, 'PRIVATE_INTERNAL'::text, 'INTERNAL_DERIVED_ALLOWED'::text, 'STORAGE_RESTRICTED'::text, 'REDISTRIBUTION_RESTRICTED'::text, 'EXPIRED'::text, 'REVOKED'::text]))));
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "chk_prov_price_type" CHECK (((price_type = ANY (ARRAY['bid'::text, 'ask'::text, 'mid'::text, 'bid_ask'::text, 'last'::text, 'unknown'::text]))));
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "chk_prov_sha256" CHECK (((raw_payload_sha256 ~ '^[0-9a-f]{64}$'::text)));
|
||||||
|
ALTER TABLE "ingestion_provenance" ADD CONSTRAINT "chk_prov_time_range" CHECK (((request_end >= request_start)));
|
||||||
|
ALTER TABLE "instrument" ADD CONSTRAINT "2200_16386_12_not_null" CHECK (active IS NOT NULL);
|
||||||
|
ALTER TABLE "instrument" ADD CONSTRAINT "2200_16386_13_not_null" CHECK (created_at IS NOT NULL);
|
||||||
|
ALTER TABLE "instrument" ADD CONSTRAINT "2200_16386_1_not_null" CHECK (instrument_id IS NOT NULL);
|
||||||
|
ALTER TABLE "instrument" ADD CONSTRAINT "2200_16386_2_not_null" CHECK (symbol IS NOT NULL);
|
||||||
|
ALTER TABLE "instrument" ADD CONSTRAINT "2200_16386_3_not_null" CHECK (asset_class IS NOT NULL);
|
||||||
|
ALTER TABLE "market_calendar" ADD CONSTRAINT "2200_16667_12_not_null" CHECK (calendar_version IS NOT NULL);
|
||||||
|
ALTER TABLE "market_calendar" ADD CONSTRAINT "2200_16667_13_not_null" CHECK (created_at IS NOT NULL);
|
||||||
|
ALTER TABLE "market_calendar" ADD CONSTRAINT "2200_16667_1_not_null" CHECK (calendar_id IS NOT NULL);
|
||||||
|
ALTER TABLE "market_calendar" ADD CONSTRAINT "2200_16667_2_not_null" CHECK (calendar_name IS NOT NULL);
|
||||||
|
ALTER TABLE "market_calendar" ADD CONSTRAINT "2200_16667_3_not_null" CHECK (asset_class IS NOT NULL);
|
||||||
|
ALTER TABLE "market_calendar" ADD CONSTRAINT "2200_16667_5_not_null" CHECK (timezone IS NOT NULL);
|
||||||
|
ALTER TABLE "provider" ADD CONSTRAINT "2200_16399_1_not_null" CHECK (provider_id IS NOT NULL);
|
||||||
|
ALTER TABLE "provider" ADD CONSTRAINT "2200_16399_2_not_null" CHECK (provider_key IS NOT NULL);
|
||||||
|
ALTER TABLE "provider" ADD CONSTRAINT "2200_16399_3_not_null" CHECK (name IS NOT NULL);
|
||||||
|
ALTER TABLE "provider" ADD CONSTRAINT "2200_16399_4_not_null" CHECK (data_type IS NOT NULL);
|
||||||
|
ALTER TABLE "provider_instrument" ADD CONSTRAINT "2200_16409_1_not_null" CHECK (provider_id IS NOT NULL);
|
||||||
|
ALTER TABLE "provider_instrument" ADD CONSTRAINT "2200_16409_2_not_null" CHECK (instrument_id IS NOT NULL);
|
||||||
|
ALTER TABLE "provider_instrument" ADD CONSTRAINT "2200_16409_3_not_null" CHECK (provider_symbol IS NOT NULL);
|
||||||
|
ALTER TABLE "provider_instrument" ADD CONSTRAINT "2200_16409_6_not_null" CHECK (active IS NOT NULL);
|
||||||
|
|
||||||
|
CREATE INDEX idx_bar_dsver ON public.historical_bar USING btree (dataset_version_id);
|
||||||
|
CREATE INDEX idx_bar_elig ON public.historical_bar USING btree (eligibility);
|
||||||
|
CREATE INDEX idx_bar_inst_tf_ts ON public.historical_bar USING btree (instrument_id, timeframe, ts_utc);
|
||||||
|
CREATE INDEX idx_bar_sess_ts ON public.historical_bar USING btree (session_state, instrument_id, ts_utc);
|
||||||
|
CREATE INDEX idx_bar_session ON public.historical_bar USING btree (session_state, session_phase);
|
||||||
|
CREATE INDEX idx_bar_techq ON public.historical_bar USING btree (technical_quality);
|
||||||
|
CREATE INDEX idx_derived_inst_tf_ts ON public.derived_bar USING btree (instrument_id, timeframe, ts_utc);
|
||||||
|
CREATE INDEX idx_historical_bar_market_quality ON public.historical_bar USING btree (instrument_id, market_quality);
|
||||||
|
CREATE INDEX idx_historical_bar_session_state ON public.historical_bar USING btree (instrument_id, session_state);
|
||||||
|
CREATE INDEX idx_prov_adapter_version ON public.ingestion_provenance USING btree (adapter_version);
|
||||||
|
CREATE INDEX idx_prov_instrument ON public.ingestion_provenance USING btree (canonical_instrument_id);
|
||||||
|
CREATE INDEX idx_prov_provider_feed ON public.ingestion_provenance USING btree (provider_id, provider_feed);
|
||||||
|
CREATE INDEX idx_prov_raw_sha ON public.ingestion_provenance USING btree (raw_payload_sha256);
|
||||||
|
CREATE INDEX idx_prov_request_identity ON public.ingestion_provenance USING btree (request_identity_hash);
|
||||||
|
CREATE INDEX idx_prov_time_range ON public.ingestion_provenance USING btree (request_start, request_end);
|
||||||
|
CREATE INDEX idx_tick_inst_ts ON public.historical_tick USING btree (instrument_id, ts_utc);
|
||||||
|
|
@ -0,0 +1,174 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
D2 — canonical_schema.py
|
||||||
|
Deterministischer Schema-Kanonisierer + Fingerprint für das Historical V2 Subsystem.
|
||||||
|
|
||||||
|
Liefert EINE autoritative Kanonisierungsfunktion, die aus einem Captured-Schema-Modell
|
||||||
|
(JSON, wie von capture_schema.py erzeugt) eine deterministische semantische
|
||||||
|
Serialisierung baut und daraus den SHA-256-Fingerprint berechnet.
|
||||||
|
|
||||||
|
Kanonisierungsregeln (siehe Mission §7):
|
||||||
|
- Schemas/Tabellen alphabetisch
|
||||||
|
- Spalten nach ordinal_position
|
||||||
|
- Constraints deterministisch nach (type, name, content)
|
||||||
|
- Indexe deterministisch
|
||||||
|
- Whitespace/Identifier-Quoting normalisiert, aber keine semantischen Unterschiede weggewaschen
|
||||||
|
|
||||||
|
Fingerprint-Coverage (§8): Tabellenexistenz, Spaltenname, ordinal_position,
|
||||||
|
Datentyp, Nullability, Default, Primary Key, UNIQUE, FK, CHECK, Index-Definitionen.
|
||||||
|
"""
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
|
||||||
|
|
||||||
|
def _norm_whitespace(s):
|
||||||
|
"""Normalisiere Whitespace; reduziere Wiederholungen, aber erhalte Wörter/Zeichen."""
|
||||||
|
if s is None:
|
||||||
|
return None
|
||||||
|
return " ".join(str(s).split())
|
||||||
|
|
||||||
|
|
||||||
|
def sorted_cols(columns):
|
||||||
|
"""Spalten einer Tabelle, sortiert nach ordinal_position."""
|
||||||
|
return sorted(columns, key=lambda c: (c.get("ordinal", 0), c.get("column", "")))
|
||||||
|
|
||||||
|
|
||||||
|
def canonical_columns(columns):
|
||||||
|
"""Spalten → deterministische Liste von Tuplen."""
|
||||||
|
out = []
|
||||||
|
for c in sorted_cols(columns):
|
||||||
|
out.append((
|
||||||
|
c["column"],
|
||||||
|
int(c.get("ordinal", 0)),
|
||||||
|
_norm_whitespace(c.get("type")),
|
||||||
|
_norm_whitespace(c.get("udt")),
|
||||||
|
bool(c.get("nullable")),
|
||||||
|
_norm_whitespace(c.get("default")),
|
||||||
|
))
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def canonical_constraints(constraints, kind):
|
||||||
|
"""Constraints eines Typs (PRIMARY KEY, UNIQUE) → sortierte Liste."""
|
||||||
|
out = []
|
||||||
|
for con in constraints:
|
||||||
|
cols = tuple(c for c in con.get("columns") or [])
|
||||||
|
out.append((
|
||||||
|
con.get("table", ""),
|
||||||
|
con.get("constraint", ""),
|
||||||
|
cols,
|
||||||
|
))
|
||||||
|
return sorted(out, key=lambda x: (x[0], x[1], x[2]))
|
||||||
|
|
||||||
|
|
||||||
|
def canonical_fks(fks):
|
||||||
|
"""FK → sortierte Liste inkl. ref_table und ref_columns."""
|
||||||
|
out = []
|
||||||
|
for fk in fks:
|
||||||
|
out.append((
|
||||||
|
fk.get("table", ""),
|
||||||
|
fk.get("constraint", ""),
|
||||||
|
tuple(fk.get("columns") or []),
|
||||||
|
fk.get("ref_table", ""),
|
||||||
|
tuple(fk.get("ref_columns") or []),
|
||||||
|
))
|
||||||
|
return sorted(out, key=lambda x: (x[0], x[1]))
|
||||||
|
|
||||||
|
|
||||||
|
def canonical_checks(checks):
|
||||||
|
"""CHECK-Constraints → sortierte Liste mit normierter Definition.
|
||||||
|
|
||||||
|
Auto-generierte PostgreSQL 'NOT NULL'-Checks (CONSTRAINT-Name endet auf
|
||||||
|
'_not_null', von PostgreSQL automatisch pro NOT-NULL-Spalte erzeugt) werden
|
||||||
|
AUSGESCHLOSSEN: sie sind redundant zu 'nullable' und ihre Definition
|
||||||
|
unterscheidet sich zwischen DBs (Production 'x IS NOT NULL' vs. Checker
|
||||||
|
'((x IS NOT NULL))') → semantisches Rauschen. Zuverlässige Erkennung NUR
|
||||||
|
über den Namenssuffix '_not_null'. Explizit benannte User-Checks
|
||||||
|
(chk_* etc.) bleiben erhalten.
|
||||||
|
"""
|
||||||
|
out = []
|
||||||
|
for chk in checks:
|
||||||
|
name = chk.get("constraint", "")
|
||||||
|
if name.endswith("_not_null"):
|
||||||
|
continue
|
||||||
|
out.append((
|
||||||
|
chk.get("table", ""),
|
||||||
|
name,
|
||||||
|
_norm_whitespace(chk.get("definition")) or "",
|
||||||
|
))
|
||||||
|
return sorted(out, key=lambda x: (x[0], x[1], x[2]))
|
||||||
|
|
||||||
|
|
||||||
|
def canonical_indexes(indexes):
|
||||||
|
"""Index-Definitionen → sortierte Liste mit normierter Definition."""
|
||||||
|
out = []
|
||||||
|
for idx in indexes:
|
||||||
|
out.append((
|
||||||
|
idx.get("table", ""),
|
||||||
|
idx.get("index", ""),
|
||||||
|
_norm_whitespace(idx.get("definition")),
|
||||||
|
))
|
||||||
|
return sorted(out, key=lambda x: (x[0], x[1], x[2]))
|
||||||
|
|
||||||
|
|
||||||
|
def canonical_sequences(sequences):
|
||||||
|
"""Sequenzen → sortierte Liste (nur Existenz/Name)."""
|
||||||
|
return sorted(sequences or [])
|
||||||
|
|
||||||
|
|
||||||
|
def canonical_document(model):
|
||||||
|
"""Baue die vollständige deterministische semantische Serialisierung."""
|
||||||
|
tables = sorted(model.get("tables") or [])
|
||||||
|
cols_by_table = {}
|
||||||
|
for c in model.get("columns") or []:
|
||||||
|
cols_by_table.setdefault(c["table"], []).append(c)
|
||||||
|
|
||||||
|
pks = canonical_constraints(model.get("primary_keys") or [], "PRIMARY KEY")
|
||||||
|
uniques = canonical_constraints(model.get("unique_constraints") or [], "UNIQUE")
|
||||||
|
fks = canonical_fks(model.get("foreign_keys") or [])
|
||||||
|
checks = canonical_checks(model.get("check_constraints") or [])
|
||||||
|
indexes = canonical_indexes(model.get("indexes") or [])
|
||||||
|
sequences = canonical_sequences(model.get("sequences") or [])
|
||||||
|
|
||||||
|
doc = []
|
||||||
|
doc.append("schema=public")
|
||||||
|
doc.append("sequences=" + "|".join(sequences))
|
||||||
|
doc.append("tables=" + "|".join(tables))
|
||||||
|
for t in tables:
|
||||||
|
doc.append(f"table:{t}:cols=" + ";".join(
|
||||||
|
f"{name}|{ordp}|{typ}|{udt}|{'n' if nul else 'y'}|{dflt}"
|
||||||
|
for (name, ordp, typ, udt, nul, dflt) in canonical_columns(cols_by_table.get(t, []))
|
||||||
|
))
|
||||||
|
doc.append("pk=" + ";".join(
|
||||||
|
f"{t}|{n}|{','.join(c)}"
|
||||||
|
for (t, n, c) in pks
|
||||||
|
))
|
||||||
|
doc.append("unique=" + ";".join(
|
||||||
|
f"{t}|{n}|{','.join(c)}"
|
||||||
|
for (t, n, c) in uniques
|
||||||
|
))
|
||||||
|
doc.append("fk=" + ";".join(
|
||||||
|
f"{t}|{n}|{','.join(c)}->{rt}|{','.join(rc)}"
|
||||||
|
for (t, n, c, rt, rc) in fks
|
||||||
|
))
|
||||||
|
doc.append("check=" + ";".join(
|
||||||
|
f"{t}|{n}|{d}"
|
||||||
|
for (t, n, d) in checks
|
||||||
|
))
|
||||||
|
doc.append("index=" + ";".join(
|
||||||
|
f"{t}|{n}|{d}"
|
||||||
|
for (t, n, d) in indexes
|
||||||
|
))
|
||||||
|
return "\n".join(doc)
|
||||||
|
|
||||||
|
|
||||||
|
def fingerprint(model):
|
||||||
|
"""SHA-256 über die kanonische Serialisierung."""
|
||||||
|
doc = canonical_document(model)
|
||||||
|
return hashlib.sha256(doc.encode("utf-8")).hexdigest(), doc
|
||||||
|
|
||||||
|
|
||||||
|
def load_model(path):
|
||||||
|
"""Lade ein Captured-Schema-Modell (JSON)."""
|
||||||
|
with open(path) as f:
|
||||||
|
return json.load(f)
|
||||||
|
|
@ -0,0 +1,95 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
D2 — capture_schema.py
|
||||||
|
Liest Schema-Metadaten (read-only) von einer DB und erzeugt das kanonische Modell-JSON.
|
||||||
|
|
||||||
|
Quellenmodus: Erwartet eine Datei mit den 7 JSON-Aggregaten aus dem Capture-SQL
|
||||||
|
(der read-only psql-Ausgabe), ODER ein bereits geparstes Modell-JSON.
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
capture_schema.py --from-capture <capture.json> --out <model.json>
|
||||||
|
capture_schema.py --from-model <model.json> --out <model.json> # Normalisierung
|
||||||
|
|
||||||
|
Der DB-Zugriff selbst erfolgt ausschließlich über das read-only Capture-SQL
|
||||||
|
(BEGIN TRANSACTION READ ONLY). Dieses Tool verarbeitet nur die Ergebnisse.
|
||||||
|
"""
|
||||||
|
import json
|
||||||
|
import sys
|
||||||
|
import argparse
|
||||||
|
|
||||||
|
AGG_NAMES = ["columns", "primary_keys", "unique_constraints",
|
||||||
|
"foreign_keys", "check_constraints", "indexes", "tables",
|
||||||
|
"sequences"]
|
||||||
|
|
||||||
|
|
||||||
|
def strip_psql_noise(text):
|
||||||
|
"""Entferne psql-Kontrollzeilen (BEGIN/on/COMMIT/leer/--)."""
|
||||||
|
out = []
|
||||||
|
for line in text.splitlines():
|
||||||
|
s = line.strip()
|
||||||
|
if s and s not in ("BEGIN", "on", "COMMIT") and not s.startswith("--"):
|
||||||
|
out.append(s)
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def parse_raw_capture(text):
|
||||||
|
"""Extrahiere die N JSON-Aggregate aus der psql-Ausgabe.
|
||||||
|
|
||||||
|
psql splittet große JSON-Arrays über mehrere Zeilen; daher wird nicht
|
||||||
|
zeilenzählend, sondern per JSON-Streaming (raw_decode) geparst.
|
||||||
|
"""
|
||||||
|
data = "\n".join(strip_psql_noise(text))
|
||||||
|
decoder = json.JSONDecoder()
|
||||||
|
aggs = []
|
||||||
|
i = 0
|
||||||
|
n = len(data)
|
||||||
|
while i < n and len(aggs) < len(AGG_NAMES):
|
||||||
|
while i < n and data[i] in " \t\r\n":
|
||||||
|
i += 1
|
||||||
|
if i >= n:
|
||||||
|
break
|
||||||
|
if data[i] not in "[{":
|
||||||
|
# non-JSON noise (z.B. Fehlermeldung) -> nächste Position
|
||||||
|
i += 1
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
val, end = decoder.raw_decode(data, i)
|
||||||
|
except json.JSONDecodeError as e:
|
||||||
|
raise SystemExit(f"FEHLER: JSON-Parse bei {i}: {e}")
|
||||||
|
aggs.append(val if val is not None else [])
|
||||||
|
i = end
|
||||||
|
if len(aggs) != len(AGG_NAMES):
|
||||||
|
raise SystemExit(
|
||||||
|
f"FEHLER: erwartete {len(AGG_NAMES)} JSON-Aggregate, fand {len(aggs)}")
|
||||||
|
return dict(zip(AGG_NAMES, aggs))
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
ap.add_argument("--from-capture", help="Rohe read-only psql-Capture-Datei (7 Aggregate)")
|
||||||
|
ap.add_argument("--from-model", help="Bereits geparstes Modell-JSON")
|
||||||
|
ap.add_argument("--out", required=True, help="Ziel-Modell-JSON")
|
||||||
|
args = ap.parse_args()
|
||||||
|
|
||||||
|
if args.from_capture:
|
||||||
|
with open(args.from_capture) as f:
|
||||||
|
model = parse_raw_capture(f.read())
|
||||||
|
elif args.from_model:
|
||||||
|
with open(args.from_model) as f:
|
||||||
|
model = json.load(f)
|
||||||
|
else:
|
||||||
|
raise SystemExit("FEHLER: --from-capture oder --from-model erforderlich")
|
||||||
|
|
||||||
|
# Konsistenz: columns/tables müssen übereinstimmen
|
||||||
|
col_tables = sorted({c["table"] for c in model["columns"]})
|
||||||
|
model["tables"] = sorted(model.get("tables") or col_tables)
|
||||||
|
|
||||||
|
with open(args.out, "w") as f:
|
||||||
|
json.dump(model, f, indent=2, sort_keys=True)
|
||||||
|
print(f"Modell geschrieben: {args.out}")
|
||||||
|
print(f"Tabellen: {len(model['tables'])} | Spalten: {len(model['columns'])}")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
|
|
@ -0,0 +1,108 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
D2 — compare_schema.py
|
||||||
|
Vergleicht zwei Schema-Modelle (Soll = kanonischer L0-Contract, Ist = tatsächlich).
|
||||||
|
|
||||||
|
Semantische Diff-Dimensionen (§12):
|
||||||
|
fehlende/extras Tabellen, Spalten, Typ-Mismatches, Nullable-Mismatches,
|
||||||
|
Default-Mismatches, PK, UNIQUE, FK, CHECK, Index-Mismatches.
|
||||||
|
|
||||||
|
Fail-closed: exit-Nonzero bei beliebigem semantischem Mismatch.
|
||||||
|
Acceptance: alle Mismatch-Counts == 0.
|
||||||
|
"""
|
||||||
|
import sys
|
||||||
|
import json
|
||||||
|
import argparse
|
||||||
|
from canonical_schema import (canonical_columns, canonical_constraints,
|
||||||
|
canonical_fks, canonical_checks, canonical_indexes)
|
||||||
|
|
||||||
|
|
||||||
|
def diff_counts(actual, expected):
|
||||||
|
"""Liefert dict mit Mismatch-Counts. 0 = keine."""
|
||||||
|
counts = {
|
||||||
|
"missing_tables": 0, "extra_tables": 0,
|
||||||
|
"missing_columns": 0, "extra_columns": 0,
|
||||||
|
"type_mismatch": 0, "nullable_mismatch": 0, "default_mismatch": 0,
|
||||||
|
"pk_mismatch": 0, "unique_mismatch": 0, "fk_mismatch": 0,
|
||||||
|
"check_mismatch": 0, "index_mismatch": 0,
|
||||||
|
}
|
||||||
|
# Tabellen
|
||||||
|
a_tbl = set((actual.get("tables") or []))
|
||||||
|
e_tbl = set((expected.get("tables") or []))
|
||||||
|
counts["missing_tables"] = len(e_tbl - a_tbl)
|
||||||
|
counts["extra_tables"] = len(a_tbl - e_tbl)
|
||||||
|
|
||||||
|
# Spalten je Tabelle
|
||||||
|
a_cols = {}
|
||||||
|
for c in actual.get("columns") or []:
|
||||||
|
a_cols.setdefault(c["table"], []).append(c)
|
||||||
|
e_cols = {}
|
||||||
|
for c in expected.get("columns") or []:
|
||||||
|
e_cols.setdefault(c["table"], []).append(c)
|
||||||
|
for tbl in e_tbl:
|
||||||
|
ec = {c["column"]: c for c in e_cols.get(tbl, [])}
|
||||||
|
ac = {c["column"]: c for c in a_cols.get(tbl, [])}
|
||||||
|
for cname, cexp in ec.items():
|
||||||
|
if cname not in ac:
|
||||||
|
counts["missing_columns"] += 1
|
||||||
|
continue
|
||||||
|
cact = ac[cname]
|
||||||
|
if _n(cact.get("type")) != _n(cexp.get("type")):
|
||||||
|
counts["type_mismatch"] += 1
|
||||||
|
if bool(cact.get("nullable")) != bool(cexp.get("nullable")):
|
||||||
|
counts["nullable_mismatch"] += 1
|
||||||
|
if _n(cact.get("default")) != _n(cexp.get("default")):
|
||||||
|
counts["default_mismatch"] += 1
|
||||||
|
for cname in ac:
|
||||||
|
if cname not in ec:
|
||||||
|
counts["extra_columns"] += 1
|
||||||
|
|
||||||
|
# Constraints
|
||||||
|
if canonical_constraints(actual.get("primary_keys") or [], "PK") != \
|
||||||
|
canonical_constraints(expected.get("primary_keys") or [], "PK"):
|
||||||
|
counts["pk_mismatch"] += 1
|
||||||
|
if canonical_constraints(actual.get("unique_constraints") or [], "UNIQUE") != \
|
||||||
|
canonical_constraints(expected.get("unique_constraints") or [], "UNIQUE"):
|
||||||
|
counts["unique_mismatch"] += 1
|
||||||
|
if canonical_fks(actual.get("foreign_keys") or []) != \
|
||||||
|
canonical_fks(expected.get("foreign_keys") or []):
|
||||||
|
counts["fk_mismatch"] += 1
|
||||||
|
if canonical_checks(actual.get("check_constraints") or []) != \
|
||||||
|
canonical_checks(expected.get("check_constraints") or []):
|
||||||
|
counts["check_mismatch"] += 1
|
||||||
|
if canonical_indexes(actual.get("indexes") or []) != \
|
||||||
|
canonical_indexes(expected.get("indexes") or []):
|
||||||
|
counts["index_mismatch"] += 1
|
||||||
|
|
||||||
|
return counts
|
||||||
|
|
||||||
|
|
||||||
|
def _n(x):
|
||||||
|
return None if x is None else " ".join(str(x).split())
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
ap.add_argument("expected", help="Kanonisches L0-Contract-Modell")
|
||||||
|
ap.add_argument("actual", help="Tatsächlich gecaptures Modell")
|
||||||
|
args = ap.parse_args()
|
||||||
|
|
||||||
|
with open(args.expected) as f:
|
||||||
|
expected = json.load(f)
|
||||||
|
with open(args.actual) as f:
|
||||||
|
actual = json.load(f)
|
||||||
|
|
||||||
|
counts = diff_counts(actual, expected)
|
||||||
|
total = sum(counts.values())
|
||||||
|
print(json.dumps(counts, indent=2))
|
||||||
|
print(f"TOTAL_MISMATCH={total}")
|
||||||
|
|
||||||
|
if total == 0:
|
||||||
|
print("PASS: Semantic diff = 0")
|
||||||
|
return 0
|
||||||
|
print("FAIL: semantic drift detected", file=sys.stderr)
|
||||||
|
return 1 # FAIL CLOSED
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
|
|
@ -0,0 +1,47 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
D2 — fingerprint_schema.py
|
||||||
|
Deterministischer SHA-256 über ein Captured-Schema-Modell.
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
fingerprint_schema.py <model.json>
|
||||||
|
→ schreibt nur den Hex-Digest auf stdout (fail-closed: exit 0 bei Erfolg)
|
||||||
|
fingerprint_schema.py <model.json> --expect <digest>
|
||||||
|
→ exit 0 wenn MATCH, sonst exit 1 (FAIL CLOSED)
|
||||||
|
|
||||||
|
Der Fingerprint deckt ab: Tabellenexistenz, Spaltenname, ordinal_position,
|
||||||
|
Datentyp, Nullability, Default, PK, UNIQUE, FK, CHECK, Index-Definitionen (§8).
|
||||||
|
"""
|
||||||
|
import sys
|
||||||
|
import argparse
|
||||||
|
from canonical_schema import fingerprint, load_model
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
ap.add_argument("model")
|
||||||
|
ap.add_argument("--expect", default=None, help="Erwarteter Digest (fail-closed Vergleich)")
|
||||||
|
ap.add_argument("--no-hash", action="store_true", help="Nur kanonische Serialisierung ausgeben")
|
||||||
|
args = ap.parse_args()
|
||||||
|
|
||||||
|
model = load_model(args.model)
|
||||||
|
digest, doc = fingerprint(model)
|
||||||
|
|
||||||
|
if args.no_hash:
|
||||||
|
print(doc)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
if args.expect:
|
||||||
|
if digest == args.expect:
|
||||||
|
print(f"MATCH {digest}")
|
||||||
|
return 0
|
||||||
|
else:
|
||||||
|
print(f"MISMATCH expected={args.expect} actual={digest}", file=sys.stderr)
|
||||||
|
return 1 # FAIL CLOSED
|
||||||
|
|
||||||
|
print(digest)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
|
|
@ -0,0 +1,136 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
D2 — gen_l0_sql.py
|
||||||
|
Erzeugt den kanonischen L0-SQL-Contract (schema-only) aus einem Captured-Schema-Modell.
|
||||||
|
|
||||||
|
Repräsentiert die aktuelle kanonische Production-Truth. Enthält KEINE:
|
||||||
|
data rows, credentials, volatile timestamps, container IDs, host paths, secrets.
|
||||||
|
Kein CREATE DATABASE, keine Grant/GRANT-relevanten Owner, keine Umgebungs-Sequenzen.
|
||||||
|
|
||||||
|
Semantische Overrides (§4) werden als Kommentar-Blöcke dokumentiert:
|
||||||
|
last_successful_chunk = timestamp with time zone (Owner-canonical)
|
||||||
|
interrupt_reason = text (preserved current contract)
|
||||||
|
technical_quality = text nullable default 'VALID'
|
||||||
|
"""
|
||||||
|
import json
|
||||||
|
import sys
|
||||||
|
import argparse
|
||||||
|
|
||||||
|
TYPE_MAP = {
|
||||||
|
'bigint': 'bigint', 'integer': 'integer', 'smallint': 'smallint',
|
||||||
|
'numeric': 'numeric', 'timestamp with time zone': 'timestamp with time zone',
|
||||||
|
'boolean': 'boolean', 'text': 'text', 'date': 'date', 'double precision': 'double precision',
|
||||||
|
'jsonb': 'jsonb', 'real': 'real', 'character varying': 'character varying',
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def ident(s):
|
||||||
|
return '"' + str(s).replace('"', '""') + '"'
|
||||||
|
|
||||||
|
|
||||||
|
def col_def(c):
|
||||||
|
typ = TYPE_MAP.get(c.get('type'), c.get('udt', c.get('type', 'text')))
|
||||||
|
nullable = '' if not c.get('nullable') else ''
|
||||||
|
notnull = ' NOT NULL' if not c.get('nullable') else ''
|
||||||
|
dflt = c.get('default')
|
||||||
|
default_part = ''
|
||||||
|
if dflt:
|
||||||
|
default_part = ' DEFAULT ' + dflt
|
||||||
|
return f" {ident(c['column'])} {typ}{notnull}{default_part}"
|
||||||
|
|
||||||
|
|
||||||
|
def build_sql(model, dbname="historical"):
|
||||||
|
out = []
|
||||||
|
out.append("-- ============================================================")
|
||||||
|
out.append(f"-- CANONICAL CURRENT PRODUCTION SCHEMA CONTRACT (L0)")
|
||||||
|
out.append(f"-- db: {dbname} · schema: public")
|
||||||
|
out.append("-- Auto-generated by gen_l0_sql.py from read-only capture")
|
||||||
|
out.append("-- Modus: CURRENT VERIFIED PRODUCTION TRUTH — NICHT historische Migration")
|
||||||
|
out.append("-- ============================================================")
|
||||||
|
out.append("")
|
||||||
|
out.append("-- SEMANTIC OVERRIDES (Owner-Canonical)")
|
||||||
|
out.append("-- last_successful_chunk = timestamp with time zone (Production + Owner decision)")
|
||||||
|
out.append("-- interrupt_reason = text · PRESERVED CURRENT CONTRACT · origin unknown · do not remove")
|
||||||
|
out.append("-- technical_quality = text · nullable · default 'VALID' · historical provenance debt OPEN")
|
||||||
|
out.append("")
|
||||||
|
# Sequenzen (für nextval-Defaults)
|
||||||
|
for seq in sorted(model.get("sequences") or []):
|
||||||
|
out.append(f"CREATE SEQUENCE IF NOT EXISTS {ident(seq)};")
|
||||||
|
if model.get("sequences"):
|
||||||
|
out.append("")
|
||||||
|
|
||||||
|
tables = sorted(model.get("tables") or [])
|
||||||
|
cols_by_table = {}
|
||||||
|
for c in model.get("columns") or []:
|
||||||
|
cols_by_table.setdefault(c["table"], []).append(c)
|
||||||
|
|
||||||
|
pks = {}
|
||||||
|
for pk in model.get("primary_keys") or []:
|
||||||
|
cols = sorted(pk.get("columns") or [], key=lambda x: 0)
|
||||||
|
pks[pk["table"]] = [pk.get("columns") or []]
|
||||||
|
|
||||||
|
for t in tables:
|
||||||
|
cols = sorted(cols_by_table.get(t, []), key=lambda c: c.get("ordinal", 0))
|
||||||
|
out.append(f"-- Table: {t} ({len(cols)} cols)")
|
||||||
|
out.append(f"CREATE TABLE IF NOT EXISTS {ident(t)} (")
|
||||||
|
body_lines = [col_def(c) for c in cols]
|
||||||
|
out.append(",\n".join(body_lines))
|
||||||
|
out.append(");")
|
||||||
|
out.append("")
|
||||||
|
|
||||||
|
# PKs
|
||||||
|
for t in tables:
|
||||||
|
for pk in model.get("primary_keys") or []:
|
||||||
|
if pk["table"] == t:
|
||||||
|
colstr = ", ".join(ident(c) for c in pk.get("columns") or [])
|
||||||
|
out.append(f"ALTER TABLE {ident(t)} ADD CONSTRAINT {ident(pk['constraint'])} PRIMARY KEY ({colstr});")
|
||||||
|
out.append("")
|
||||||
|
|
||||||
|
# UNIQUE
|
||||||
|
for uq in sorted(model.get("unique_constraints") or [], key=lambda x: x["table"]):
|
||||||
|
colstr = ", ".join(ident(c) for c in uq.get("columns") or [])
|
||||||
|
out.append(f"ALTER TABLE {ident(uq['table'])} ADD CONSTRAINT {ident(uq['constraint'])} UNIQUE ({colstr});")
|
||||||
|
out.append("")
|
||||||
|
|
||||||
|
# FK
|
||||||
|
for fk in sorted(model.get("foreign_keys") or [], key=lambda x: x["table"]):
|
||||||
|
colstr = ", ".join(ident(c) for c in fk.get("columns") or [])
|
||||||
|
rcolstr = ", ".join(ident(c) for c in fk.get("ref_columns") or [])
|
||||||
|
out.append(f"ALTER TABLE {ident(fk['table'])} ADD CONSTRAINT {ident(fk['constraint'])} FOREIGN KEY ({colstr}) REFERENCES {ident(fk['ref_table'])} ({rcolstr});")
|
||||||
|
out.append("")
|
||||||
|
|
||||||
|
# CHECK - semantische Definition
|
||||||
|
for chk in sorted(model.get("check_constraints") or [], key=lambda x: (x["table"], x["constraint"])):
|
||||||
|
out.append(f"ALTER TABLE {ident(chk['table'])} ADD CONSTRAINT {ident(chk['constraint'])} CHECK ({chk.get('definition','')});")
|
||||||
|
out.append("")
|
||||||
|
|
||||||
|
# Indexe (non-PK/UNIQUE)
|
||||||
|
idx_set = set()
|
||||||
|
for pk in model.get("primary_keys") or []:
|
||||||
|
idx_set.add(pk["constraint"])
|
||||||
|
for uq in model.get("unique_constraints") or []:
|
||||||
|
idx_set.add(uq["constraint"])
|
||||||
|
for idx in sorted(model.get("indexes") or [], key=lambda x: x["index"]):
|
||||||
|
if idx["index"] in idx_set:
|
||||||
|
continue
|
||||||
|
out.append(f"{idx['definition']};")
|
||||||
|
out.append("")
|
||||||
|
return "\n".join(out)
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
ap.add_argument("model")
|
||||||
|
ap.add_argument("--out", required=True)
|
||||||
|
args = ap.parse_args()
|
||||||
|
with open(args.model) as f:
|
||||||
|
model = json.load(f)
|
||||||
|
sql = build_sql(model)
|
||||||
|
with open(args.out, "w") as f:
|
||||||
|
f.write(sql)
|
||||||
|
print(f"L0 SQL-Contract geschrieben: {args.out} ({len(sql)} B)")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
Loading…
Reference in a new issue