# oc_diagnostics Architecture Document
**Version:** 1.5
**Status:** Authoritative Reference
**Scope:** Defines the architecture, structure, and design philosophy of the `oc_diagnostics` runtime package.

---

## Table of Contents

- [1. Overview](#1-overview)
- [2. High-Level Architecture](#2-high-level-architecture)
- [3. Directory Structure](#3-directory-structure)
- [4. Core Components](#4-core-components)
  - [4.1 unified\_debug.py](#41-unified_debugpy)
  - [4.2 id\_instrumentation.py](#42-id_instrumentationpy)
  - [4.3 instrumentation\_middleware.py](#43-instrumentation_middlewarepy)
  - [4.4 ui\_event\_logger.py](#44-ui_event_loggerpy)
  - [4.5 Self-Test Suite](#45-self-test-suite)
  - [4.6 fastapi\_middleware.py](#46-fastapi_middlewarepy)
  - [4.7 logging\_handler.py](#47-logging_handlerpy)
- [5. Diagnostics Configuration](#5-diagnostics-configuration)
- [6. Import Surface](#6-import-surface)
- [7. Lifecycle & Integration](#7-lifecycle--integration)
- [8. Future Enhancements](#8-future-enhancements)
- [9. Appendix: Mindmap Overview](#9-appendix-mindmap-overview)

---

# 1. Overview

`oc_diagnostics` is the unified debugging, tracing, and instrumentation system for all Option C‑compliant projects.

It provides:

- `_dbg` — structured debug logging
- domain and level system
- ID instrumentation for Dash applications
- callback middleware
- UI event logging
- a self‑test suite

It is a **runtime** package.
OCT is a **development‑time** package.
Projects consume both.

`oc_diagnostics` is project‑agnostic and contains no assumptions about any specific application.

---

# 2. High-Level Architecture

```mermaid
flowchart TD

    subgraph CONFIG["Project Diagnostics Config"]
        CFG["oc_diagnostics/debug_config.json"]
    end

    subgraph CORE["oc_diagnostics Core"]
        UDEBUG["unified_debug.py\n(_dbg, domains, levels,\nstdout/stderr interception,\nexception hook)"]
    end

    subgraph INSTR["Instrumentation System (Dash)"]
        IDR["id_instrumentation.py\n(ID rewriting)"]
        MIDDLE["instrumentation_middleware.py\n(callback patching)"]
        LOGGER["ui_event_logger.py\n(UI event capture)"]
        SELFTEST["Self-Test Suite\n(selftest_cli / run_tests.py)"]
    end

    subgraph FASTAPI["FastAPI / ASGI Integration"]
        FMIDDLE["fastapi_middleware.py\n(ASGI middleware)"]
    end

    CFG --> UDEBUG
    UDEBUG --> IDR
    UDEBUG --> MIDDLE
    UDEBUG --> LOGGER
    IDR --> MIDDLE
    MIDDLE --> LOGGER
    UDEBUG --> FMIDDLE
```

---

# 3. Directory Structure

```
oc_diagnostics/
    oc_diagnostics/
        __init__.py
        unified_debug.py
        logging_handler.py
        id_instrumentation.py
        instrumentation_middleware.py
        ui_event_logger.py
        instrumentation_selftest.py
        instrumentation_selftest_cli.py
        instrumentation_selftest_app.py
        fastapi_middleware.py
    tests/
        run_tests.py
        Tests_README.md
    tools/
        migrate_to_oc_diagnostics.py
    docs/
        ARCHITECTURE.md
        CHANGELOG.md
        REFERENCE.md
        INTEGRATION_GUIDE.md
        debug_config.example.json
    README.md
    pyproject.toml
```

This structure is intentionally flat to keep imports simple:

```python
from oc_diagnostics import _dbg
from oc_diagnostics import apply_instrumentation
```

---

# 4. Core Components

## 4.1 unified_debug.py

Responsibilities:

- `_dbg(domain, func, message, level)` — synchronous structured debug logging
- `_adbg()` — async variant; captures caller frame synchronously, offloads file I/O via `asyncio.to_thread` (FS-16)
- domain‑level filtering
- colorized output
- timestamps
- filename + line number inclusion
- stdout/stderr interception
- uncaught exception logging
- output routing (`terminal`, `file`, `both`, `json`)
- JSONL output mode — writes structured JSON records to file, colored text to terminal (FS-07)
- loading project‑specific `debug_config.json`; optional validation with `jsonschema` (FS-18)
- config hot-reload — daemon thread polls config mtime every 2 s when `init(watch_config=True)` (FS-09)
- production mode (`PROD_MODE`) — suppresses stream interception and exception hook when env var or config `"mode"` is `"prod"` (FS-15)
- runtime level override API: `set_debug_level()`, `set_global_setting()` (FS-03)
- `dbg_timed()` context manager and `dbg_profile()` decorator for inline performance measurement (FS-12)
- log rotation — count-based pruning (`max_log_files`) and size-based roll-over (`max_log_size_mb`) (FS-06)
- HTTP header redaction — `HTTP_REDACT_HEADERS` frozenset; sensitive headers shown as `[REDACTED]` at L4 (FS-17)
- configurable UI event properties (`UI_EVENT_PROPERTIES`) loaded from config (FS-10)
- `instrumentation_allow_all` flag — instrument non-Dash objects without raising debug level (FS-13)
- thread-safe log file writes (`_log_lock` protects all file I/O; `LOG_FILE` is initialized lazily on first write)
- one-shot `RuntimeWarning` when `_dbg()` is called before `init()` (surfacing misconfiguration)
- export `ADDITIONAL_STRUCTURAL_PREFIXES` and `ADDITIONAL_STRUCTURAL_EXACT` for config-driven structural ID extension

This module is the foundation of the entire diagnostics system.

---

## 4.2 id_instrumentation.py

Responsibilities:

- rewrite Dash component IDs into pattern‑matching dicts
- preserve structural IDs
- recurse through component trees
- respect instrumentation levels
- support custom components at higher debug levels
- enforce `MAX_INSTRUMENT_DEPTH` (default: 500) to prevent `RecursionError` on pathologically deep or circular component trees; a L1 diagnostic is emitted when the limit is reached
- merge project-defined structural ID extensions (`ADDITIONAL_STRUCTURAL_PREFIXES`, `ADDITIONAL_STRUCTURAL_EXACT`) loaded from `debug_config.json` into `is_structural_id()` checks
- respect `INSTRUMENTATION_ALL` flag — instrument non-Dash objects without requiring a higher debug level (FS-13)

Instrumentation is safe, deterministic, and project‑agnostic.

---

## 4.3 instrumentation_middleware.py

Responsibilities:

- patch Dash callback dispatcher
- wrap callback outputs with instrumentation
- apply instrumentation to any component tree — including **list and tuple** callback returns (each element instrumented individually; container type preserved)
- integrate with UI event logger

Middleware is opportunistic; explicit instrumentation remains authoritative.

---

## 4.4 ui_event_logger.py

Responsibilities:

- capture all user interactions
- listen to instrumented IDs (`{"event": "<id>"}`)
- log property changes for the set defined by `UI_EVENT_PROPERTIES` in `global_settings` (default: `n_clicks`, `value`, `data`, `figure`, `selectedData`) (FS-10)
- route all events through `_dbg`
- expose `REQUIRED_STORE_ID` as a public constant (`"debug-settings-data-store"`) for programmatic use
- emit a **L1 diagnostic warning** at registration time when `REQUIRED_STORE_ID` is absent from `app.layout`
- truncate large event values before logging; summarise `figure` and `data` properties as type+length when `DEBUG_LEVELS["UI"] < 9`

This module provides a universal UI event capture mechanism.

---

## 4.5 Self-Test Suite

The self-test suite spans two locations:

**`oc_diagnostics/` (package-level):**
- `instrumentation_selftest_cli.py` — headless, deterministic CLI test runner;
  validates ID rewriting, middleware, structural protection, and debug-level
  transitions; exits 0 on success, 1 on failure; suitable for CI/CD.
- `instrumentation_selftest_app.py` — manual integration harness; launches a
  Dash development server for visual validation; not for automated testing.
- `instrumentation_selftest.py` — lightweight manual smoke-test entry point.

**`tests/` (project-level):**
- `run_tests.py` — primary automated test runner; covers pure-Python core tests
  (no Dash required), FastAPI ASGI contract tests (no optional dependencies
  required), and optional Dash tests that auto-skip when Dash is absent;
  invokes `instrumentation_selftest_cli` as a subprocess; exits 0 on full
  success, 1 on any failure; suitable for CI/CD integration.

The test suite ensures the entire system behaves consistently across environments.

---

## 4.6 fastapi_middleware.py

Responsibilities:

- implement `DiagnosticsMiddleware` as a **pure ASGI callable** — no dependency on
  `starlette.BaseHTTPMiddleware`; works with FastAPI, Starlette, or any ASGI framework
- log request entry (method, path, client IP) at L2
- log User-Agent at L3; log full request and response headers at L4
- capture HTTP status code from the `http.response.start` ASGI event
- measure and report elapsed wall-clock time in milliseconds
- pass non-HTTP scopes (WebSocket, lifespan) through unchanged
- log application exceptions at L2 and re-raise them
- redact sensitive request and response headers at L4 using `HTTP_REDACT_HEADERS` from `unified_debug`; affected headers are displayed as `[REDACTED]` (FS-17)

Install the optional extra to bring in FastAPI itself:

```bash
pip install oc_diagnostics[fastapi]
```

Enable HTTP logging by adding the `HTTP` domain to `oc_diagnostics/debug_config.json`:

```json
"HTTP": { "level": 2, "color": "blue" }
```

---

## 4.7 logging_handler.py

Responsibilities:

- implement `OcDiagnosticsHandler` — a `logging.Handler` subclass that routes Python
  standard-library log records through `_dbg()` (FS-08)
- accept a configurable `domain` (default: `"GENERAL"`) and an optional `level_map`
  mapping `logging` level integers to `_dbg` diagnostic levels
- default level map: `DEBUG → 4`, `INFO → 2`, `WARNING/ERROR/CRITICAL → 1`
- always available — pure Python, no optional dependencies required

This module bridges the standard `logging` ecosystem with the `oc_diagnostics`
domain-and-level system so third-party libraries and legacy code can participate
in the same diagnostic pipeline without modification.

---

# 5. Diagnostics Configuration

Each project must include:

```
oc_diagnostics/debug_config.json
```

Example:

```json
{
  "domains": {
    "DATA": { "level": 3, "color": "cyan" },
    "UI":   { "level": 4, "color": "blue" },
    "INSTRUMENTATION": 3,
    "HTTP": { "level": 2, "color": "blue" }
  },
  "global_settings": {
    "silent_mode": false,
    "enable_timestamps": true,
    "enable_colors": true,
    "include_filename": true,
    "output_mode": "both",
    "instrumentation_enabled": true,
    "max_log_files": 10,
    "max_log_size_mb": 5.0,
    "instrumentation_allow_all": false,
    "mode": "dev",
    "http_redact_headers": ["authorization", "cookie", "set-cookie", "x-api-key"]
  }
}
```

The configuration file is **project‑owned**, not package‑owned.

---

# 6. Import Surface

`oc_diagnostics` exposes only the public API:

```python
from oc_diagnostics import _dbg                    # synchronous structured logging
from oc_diagnostics import _adbg                   # async variant (FS-16)
from oc_diagnostics import DEBUG_LEVELS
from oc_diagnostics import init
from oc_diagnostics import set_debug_level         # runtime domain level override (FS-03)
from oc_diagnostics import set_global_setting      # runtime global setting override (FS-03)
from oc_diagnostics import dbg_timed               # timing context manager (FS-12)
from oc_diagnostics import dbg_profile             # timing decorator (FS-12)
from oc_diagnostics import OcDiagnosticsHandler    # Python logging integration (FS-08)
from oc_diagnostics import apply_instrumentation   # requires: pip install oc_diagnostics[dash]
from oc_diagnostics import DiagnosticsMiddleware   # requires: pip install oc_diagnostics[fastapi]
```

Both optional exports (`apply_instrumentation`, `DiagnosticsMiddleware`) raise a clear
`ImportError` at call time when their respective optional dependency is absent.
All other exports are always available — no optional dependency required.

Instrumentation constants and helpers are imported explicitly when needed:

```python
from oc_diagnostics.id_instrumentation import instrument_ids, MAX_INSTRUMENT_DEPTH
from oc_diagnostics.ui_event_logger import register_callbacks, REQUIRED_STORE_ID
from oc_diagnostics.unified_debug import ADDITIONAL_STRUCTURAL_PREFIXES, ADDITIONAL_STRUCTURAL_EXACT
from oc_diagnostics.unified_debug import HTTP_REDACT_HEADERS, UI_EVENT_PROPERTIES
```

This keeps the public surface clean and predictable.

---

# 7. Lifecycle & Integration

A typical Option C project uses diagnostics as follows:

1. Project defines `oc_diagnostics/debug_config.json`
2. Project imports `_dbg`
3. Project builds static layout (uninstrumented)
4. Project registers UI event logger
5. Project registers callbacks
6. Project enables instrumentation middleware
7. Dynamic content is instrumented explicitly

This lifecycle ensures correctness and avoids ID‑mismatch issues.

---

# 8. Future Enhancements

Approved:

- diagnostics context object (FS-19)
- integration with OCT's `oct diag` command (OI-26, OI-27)
- richer event metadata

Deferred:

- debug session recorder

---

# 9. Appendix: Mindmap Overview

```mermaid
mindmap
  root((oc_diagnostics))
    Debugging
      _dbg
      _adbg (async)
      domains
      levels
      timestamps
      colors
      stdout/stderr interception
      exception hook
      thread-safe file writes
      pre-init RuntimeWarning
      JSONL output mode
    Runtime API
      set_debug_level
      set_global_setting
    Profiling
      dbg_timed context manager
      dbg_profile decorator
    Config
      hot-reload (watch_config)
      production mode (PROD_MODE)
      log rotation (count + size)
      jsonschema validation
    Instrumentation
      id rewriting
      structural ID protection
      project-defined structural IDs (FS-02)
      instrumentation_allow_all (FS-13)
      recursion
      depth limit (MAX_INSTRUMENT_DEPTH)
      custom components
    Middleware
      callback patching
      dynamic instrumentation
      list/tuple output handling
    UI Event Logger
      event capture
      configurable properties (UI_EVENT_PROPERTIES)
      REQUIRED_STORE_ID
      startup store validation
      value truncation
    Self-Test Suite
      CLI tests (selftest_cli)
      Dash app tests (selftest_app)
      automated runner (run_tests.py)
      FastAPI ASGI tests
      debug-level transitions
    FastAPI Integration
      ASGI middleware
      HTTP domain logging
      header redaction (HTTP_REDACT_HEADERS)
      optional extra
    Python logging Integration
      OcDiagnosticsHandler
      domain routing
      level mapping
```