oc_diagnostics — Unified Debugging & Instrumentation System

oc_diagnostics — Unified Debugging & Instrumentation System#

Version: 1.3 Authoritative Reference Guide Option C Runtime Diagnostics

Table of Contents#

  1. Overview

  2. Unified Debugging System

  3. Instrumentation System

  4. Instrumentation Middleware

  5. UI Event Logger

  6. FastAPI / ASGI Middleware

  7. Self‑Test Suite

  8. Migration Tool

  9. Recommended Debug Configurations

  10. Extending the System

  11. Appendix: File Map

Overview#

oc_diagnostics is the unified debugging, tracing, and instrumentation system for all Option C‑compliant projects.

It is designed to be:

  • Authoritative

  • Predictable

  • Project‑agnostic

  • Self‑contained

  • Future‑proof

  • Developer‑friendly

The system consists of:

  1. Unified Debugging System

  2. Instrumentation System

  3. Instrumentation Middleware

  4. UI Event Logger

  5. FastAPI / ASGI Middleware

  6. Self‑Test Suite

It is a runtime package. OCT is a development‑time package. Projects consume both.

Unified Debugging System#

Goals#

Provide a single, consistent, complete debugging pipeline across all Option C projects.

Debug Levels#

0 = Silent
1 = Errors only
2 = Lifecycle events
3 = Important details
4 = Verbose analysis
5–9 = Deep tracing

Domains#

Projects define their own domains in:

diagnostics/debug_config.json

Common domains include:

UI, DATA, ENGINE, SETTINGS, INSTRUMENTATION, HTTP, PYERR, STDOUT, STDERR

Global Settings#

Loaded from debug_config.json (global_settings object):

Key

Type

Default

Description

silent_mode

bool

false

Hard global off switch — suppresses all output.

enable_timestamps

bool

true

Prefix each log line with HH:MM:SS.mmm.

enable_colors

bool

true

ANSI color codes in terminal output.

include_filename

bool

true

Include filename:line::caller in each log line.

output_mode

str

"both"

Routing target: "terminal", "file", "both", "json".

instrumentation_enabled

bool

true

Master toggle for all ID instrumentation.

instrumentation_allow_all

bool

false

Instrument non-Dash objects without requiring a high debug level (FS-13).

max_log_files

int

0

Keep at most N log files; 0 = unlimited (FS-06).

max_log_size_mb

float

0

Roll over to a new file when current file exceeds N MB; 0 = disabled (FS-06).

ui_event_properties

list[str]

["n_clicks","value","data","figure","selectedData"]

Properties the UI event logger listens to (FS-10).

http_redact_headers

list[str]

["authorization","cookie","set-cookie","x-api-key"]

Request/response headers replaced with [REDACTED] at L4 (FS-17).

mode

str

""

Set to "prod" / "production" / "safe" to activate production mode (FS-15).

structural_id_prefixes

list[str]

[]

Additional ID prefixes treated as structural (never instrumented) (FS-02).

structural_id_exact

list[str]

[]

Additional exact IDs treated as structural (never instrumented) (FS-02).

Environment Variables#

Four environment variables allow the config path, log path, and initialization behavior to be overridden without code changes. All are optional; the default behavior is unchanged when they are absent.

Variable

Default

Description

OC_DIAGNOSTICS_CONFIG

<cwd>/diagnostics/debug_config.json

Absolute path to a debug_config.json file. Takes precedence over the project-root convention. Useful for embedded usage, subprocess invocation, or CI environments where the working directory is not the project root.

OC_DIAGNOSTICS_LOG_DIR

<cwd>/logs

Absolute path to the directory where log files are written. Replaces the default CWD-relative logs/ directory.

UNIFIED_DEBUG_DISABLE_EXCEPTION_HOOK

(unset)

Set to 1 to prevent init() from installing the custom sys.excepthook. Useful when the host environment (pytest, pdb, ASGI framework) manages its own exception handler.

UNIFIED_DEBUG_DISABLE_STDIO_INTERCEPT

(unset)

Set to 1 to prevent init() from replacing sys.stdout/sys.stderr with _StreamInterceptor.

Resolution order for config:

  1. OC_DIAGNOSTICS_CONFIG env var (if set and non-empty)

  2. <cwd>/diagnostics/debug_config.json (project root convention)

  3. debug_config.json next to unified_debug.py (bundled fallback / tests)

CONFIG_SOURCE always records which path was used and how it was resolved.

Example:

OC_DIAGNOSTICS_CONFIG=/etc/myapp/debug_config.json \
OC_DIAGNOSTICS_LOG_DIR=/var/log/myapp \
python myapp.py

Output Routing#

Mode

Terminal

Log file

terminal

Colored text

—

file

—

Plain text (ANSI stripped)

both

Colored text

Plain text (ANSI stripped)

json

Colored text

JSONL (one JSON object per line)

See JSONL Output Mode for structured log details.

Color System#

Each domain may define a color. Errors (level=1) are always red.

Timestamps#

Format:

HH:MM:SS.mmm

Silent Mode#

Hard global off switch.

Exception Hook#

Uncaught exceptions are logged as PYERR (L1) and then delegated to the hook that was active when init() was called. This preserves the behavior of debuggers (pdb), test runners (pytest), and framework-level formatters (rich.traceback.install()) that install their own handlers.

Set UNIFIED_DEBUG_DISABLE_EXCEPTION_HOOK=1 to prevent init() from installing the hook at all.

Stdout/Stderr Interception#

All print() calls and server logs are captured and routed through _dbg.

Using _dbg#

_dbg(domain, func, message, level, override=False)

Pre-init() warning: If _dbg() is called before init(), a one-shot RuntimeWarning is emitted via warnings.warn() (unless SILENT_MODE is true). This surfaces forgotten init() calls at import time rather than silently producing no log-file output.

Example:

func = "load_data"
_dbg("DATA", func, "START", 2)
_dbg("DATA", func, f"rows={len(df)}", 3)
_dbg("DATA", func, "df_head=...", 4)

Async _adbg#

_adbg() is the async variant of _dbg(), designed for use inside async def functions and coroutines (FS-16).

from oc_diagnostics import _adbg

async def fetch_data():
    await _adbg("DATA", "fetch_data", "START", level=2)
    result = await some_async_call()
    await _adbg("DATA", "fetch_data", f"rows={len(result)}", level=3)
    return result

Key behaviors:

  • The caller frame (filename, line number, function name) is captured synchronously before any await, so location information is always accurate.

  • Terminal output is written synchronously (no latency).

  • File I/O is offloaded to asyncio.to_thread() so it never blocks the event loop.

  • JSONL records are serialized and written the same way.

  • Signature mirrors _dbg() exactly; drop-in replacement inside async functions.

Runtime API#

Two functions allow debug levels and global settings to be changed at runtime without editing debug_config.json or restarting the process (FS-03):

set_debug_level(domain, level)#

from oc_diagnostics import set_debug_level

set_debug_level("HTTP", 4)   # enable full header dumps for the current session
set_debug_level("UI", 0)     # silence UI events

  • domain: any string (new domains are created if they do not exist in DEBUG_LEVELS)

  • level: integer 0–9; raises UserWarning and returns without modifying anything if invalid

  • Mutates DEBUG_LEVELS in place; change takes effect immediately

set_global_setting(key, value)#

from oc_diagnostics import set_global_setting

set_global_setting("silent_mode", True)
set_global_setting("output_mode", "json")
set_global_setting("enable_timestamps", False)

Valid keys: silent_mode, enable_timestamps, enable_colors, include_filename, output_mode.

Raises UserWarning for unknown keys and returns without modifying anything.

Profiling and Timing#

Three utilities cover lifecycle and timing tracing (FS-12, FS-19):

dbg_scope(domain, func, label="", level=2) — context manager (no timing)#

from oc_diagnostics import dbg_scope

with dbg_scope("DATA", "load_records"):
    result = load()
# Emits: [DATA][L2] ENTER
#        [DATA][L2] EXIT

Logs ENTER and EXIT at the specified level without any timing overhead. Use this for pure lifecycle tracing where elapsed time is not needed. For timed tracing, use dbg_timed() instead.

dbg_timed(domain, func, label="", level=3) — context manager#

from oc_diagnostics import dbg_timed

with dbg_timed("DATA", "load_records", label="CSV parse", level=3):
    df = pd.read_csv(path)

Emits a ENTER log line at the start and an EXIT (N.N ms) line at exit.

dbg_profile(domain, func, label="", level=3) — decorator#

from oc_diagnostics import dbg_profile

@dbg_profile(domain="ENGINE", level=3)
def compute_result(x, y):
    return x + y

Wraps the decorated function with dbg_timed(); the original return value is preserved. The label defaults to the function name when not specified.

Production Mode#

When running in a production environment, stream interception and the exception hook can be suppressed automatically without code changes (FS-15).

Via environment variable (highest priority):

OC_DIAGNOSTICS_MODE=prod python myapp.py

Via debug_config.json:

"global_settings": {
  "mode": "prod"
}

Accepted values: "prod", "production", "safe".

When production mode is active, init() behaves as if called with intercept_streams=False, intercept_exceptions=False, regardless of the arguments passed. All _dbg() / _adbg() calls continue to work normally; only the stream/exception hook side-effects are suppressed.

Config Hot-Reload#

init() accepts an optional watch_config=True argument that starts a background daemon thread polling the config file every 2 seconds (FS-09):

from oc_diagnostics import init

init(watch_config=True)

When the file’s modification time changes, all globals — DEBUG_LEVELS, COLOR_CODES, OUTPUT_MODE, HTTP_REDACT_HEADERS, UI_EVENT_PROPERTIES, etc. — are updated in place. The DEBUG_LEVELS dict object itself is preserved (clear() + update()), so any downstream references to the dict continue to reflect the new values.

Hot-reload is disabled by default to avoid unnecessary file-system polling in short-lived scripts and test runners.

Log Rotation#

Two complementary rotation strategies are available (FS-06):

Count-based rotation (max_log_files):

When the number of unified_debug_logs_*.txt files in the log directory exceeds max_log_files, the oldest files are deleted before a new file is created.

Size-based rotation (max_log_size_mb):

When the current log file exceeds max_log_size_mb megabytes, LOG_FILE is reset and a new timestamped file is created on the next write.

Both strategies can be active simultaneously. Set either to 0 to disable it.

"global_settings": {
  "max_log_files": 10,
  "max_log_size_mb": 5.0
}

JSONL Output Mode#

Setting output_mode to "json" switches file output to newline-delimited JSON (JSONL), while terminal output continues to display colored human-readable text (FS-07).

"global_settings": {
  "output_mode": "json"
}

Each log line in the file becomes a JSON object:

{"ts":"2026-02-27T14:23:01.042","domain":"DATA","level":3,"func":"load_records","file":"data_loader.py","line":42,"caller":"main","msg":"rows=1500","override":false}

Fields:

Field

Type

Description

ts

str

ISO 8601 timestamp with millisecond precision

domain

str

Debug domain

level

int

Effective debug level

func

str

Logical function name passed to _dbg()

file

str

Source filename

line

int

Source line number

caller

str

Python function name at the call site

msg

str

Message string

override

bool

true when _dbg() was called with override=True

JSONL files are suitable for ingestion by log aggregation systems (Loki, Splunk, etc.) and can be post-processed with jq.

Python logging Integration#

OcDiagnosticsHandler bridges the standard-library logging module with oc_diagnostics, routing log records through _dbg() (FS-08):

import logging
from oc_diagnostics import OcDiagnosticsHandler

logger = logging.getLogger("myapp")
logger.addHandler(OcDiagnosticsHandler(domain="GENERAL"))
logger.setLevel(logging.DEBUG)

logger.info("Server started")      # → _dbg("GENERAL", ..., level=2)
logger.debug("Detail here")        # → _dbg("GENERAL", ..., level=4)
logger.warning("Low disk space")   # → _dbg("GENERAL", ..., level=1)

Default level mapping:

logging level

_dbg level

DEBUG (10)

4

INFO (20)

2

WARNING (30)

1

ERROR (40)

1

CRITICAL (50)

1

Provide a custom level_map to override the defaults:

import logging
from oc_diagnostics import OcDiagnosticsHandler

handler = OcDiagnosticsHandler(
    domain="REQUESTS",
    level_map={
        logging.DEBUG:   3,
        logging.INFO:    2,
        logging.WARNING: 2,
        logging.ERROR:   1,
    },
)
logging.getLogger("requests").addHandler(handler)

Instrumentation System#

Purpose#

Rewrite Dash component IDs so the UI event logger can capture all interactions.

Global Toggle#

instrumentation_enabled = true

Instrumentation Levels#

0     = Off (instrumentation_active() returns False)
1–4   = Standard Dash components only
          - 1–2: static layout instrumentation
          - 3:   static + dynamic (callback output) instrumentation
          - 4:   verbose logging (detailed recursion output)
5–9   = Standard Dash + custom components
          - 5–7: instrumentation_allow_custom_components() becomes True;
                 plain Python objects with a .id attribute are also rewritten
          - 8–9: maximum verbosity / full tracing

“Custom component” means any object that does not inherit from dash.development.base_component.Component. This includes plain Python classes, third-party non-Dash objects, and mock objects used in tests. Standard Dash components (all dash.html, dash.dcc, dash_bootstrap_components, etc.) are always instrumented at level ≥ 1.

Structural ID Protection#

Never instrument:

  • IDs starting with: root-group-, tab-group-, pane-group-, tab-content-

  • Exact IDs such as: theme-root, ui-theme-store, system-settings-store, etc.

Projects may extend these lists without modifying package source by adding structural_id_prefixes and structural_id_exact lists to global_settings in debug_config.json (FS-02):

"global_settings": {
  "structural_id_prefixes": ["my-group-", "app-root-"],
  "structural_id_exact": ["my-app-store", "shared-nav-store"]
}

These are loaded into ADDITIONAL_STRUCTURAL_PREFIXES and ADDITIONAL_STRUCTURAL_EXACT in unified_debug and merged into is_structural_id() checks at runtime.

Instrumentation Rules#

Instrument:

  • Components with string IDs

  • IDs in .id, _props, _prop_ids

  • Nested children

Skip:

  • Structural IDs

  • Dict IDs

  • Components without IDs

  • Custom components unless allowed

Instrumentation Helpers#

  • instrumentation_active() — returns True when level ≥ 1

  • instrumentation_verbose() — returns True when level ≥ 4

  • instrumentation_allow_custom_components() — returns True when level ≥ 5

  • instrumentation_allow_all() — returns True when instrumentation_allow_all: true in config (FS-13); instruments non-Dash objects regardless of debug level

Recursion Depth Limit#

instrument_ids() enforces a maximum recursion depth:

from oc_diagnostics.id_instrumentation import MAX_INSTRUMENT_DEPTH
# MAX_INSTRUMENT_DEPTH == 500

When the limit is exceeded, a L1 diagnostic is emitted and the subtree below that point is returned uninstrumented. Real-world Dash applications rarely exceed 30–50 levels of nesting; the 500-level limit exists as a safety guard against pathological or circular component structures.

The ID Contract (Critical)#

Dash callback decorators match string IDs. Instrumentation rewrites IDs into dicts.

Therefore:

Static layout must not be instrumented before callback registration.

If static layout is instrumented too early:

  • Callbacks will not fire

  • Dynamic content will not appear

  • UI event logger will not receive events

Two‑Phase Instrumentation Model#

Phase 1 — Static instrumentation Performed after callback registration.

Phase 2 — Dynamic instrumentation Performed explicitly inside callbacks:

return apply_instrumentation(dynamic_tree)

Middleware supplements but does not replace explicit calls.

Instrumentation Middleware#

Purpose#

Provide a central integration point for dynamic instrumentation.

Static Layout Instrumentation#

Performed in build_layout:

full_layout = apply_instrumentation(full_layout)

Only safe after callbacks are registered.

Dynamic Callback Instrumentation#

Authoritative mechanism:

return apply_instrumentation(dynamic_tree)

List and Tuple Return Handling#

apply_instrumentation() accepts single components, lists, and tuples. When given a list or tuple it instruments each element individually and returns a container of the same type:

# Callback returning multiple components as a list
return apply_instrumentation([header_div, content_div, footer_div])

# Callback returning a tuple
return apply_instrumentation((header, body))

Middleware Limitations#

Dash 3.x callback_map entries:

  • may contain wrappers

  • may not expose original callbacks

  • may not be complete until after first layout request

  • are not guaranteed stable

Middleware is opportunistic. Explicit instrumentation is authoritative.

Middleware Activation Contract#

Must run:

  • after app creation

  • before layout building

  • after UI event logger registration

UI Event Logger#

Purpose#

Capture all user interactions across the entire UI.

ID Requirements#

Listens only to IDs of the form:

{"event": "<original-id>"}

Store Dependency#

Requires a dcc.Store with a specific ID in the Dash layout:

from oc_diagnostics.ui_event_logger import REQUIRED_STORE_ID
# REQUIRED_STORE_ID == "debug-settings-data-store"

dcc.Store(id=REQUIRED_STORE_ID, storage_type="memory")

Use the exported constant rather than the hard-coded string to guarantee consistency across all call sites.

Startup Store Validation#

register_callbacks(app) inspects app.layout at registration time and emits a L1 diagnostic warning when REQUIRED_STORE_ID is not found:

[UI][L1] WARNING: 'debug-settings-data-store' not found in app.layout —
the UI event logger callback will fail at runtime.

The check is skipped when app.layout is None or a callable (resolved at request time).

Event Value Truncation#

Event values are truncated before logging:

  • All properties: repr() is capped at 500 characters; values exceeding this are annotated with the original length.

  • figure and data properties: logged as <type len=N> summary when DEBUG_LEVELS["UI"] < 9; full repr is logged at level 9.

EVENT: id=my-graph, property=figure, value=<dict len=5> (set UI level=9 for full value)

Callback Contract#

The set of properties listened to is configurable via ui_event_properties in global_settings (FS-10):

"global_settings": {
  "ui_event_properties": ["n_clicks", "value", "data", "figure", "selectedData"]
}

Default properties: n_clicks, value, data, figure, selectedData.

Returns no_update — the callback never modifies the store.

Note: Changing ui_event_properties requires restarting the application; the property list is read at Dash callback registration time.

FastAPI / ASGI Middleware#

Purpose#

Log every HTTP request and response through _dbg() as a pure ASGI middleware callable. Works with FastAPI, Starlette, or any ASGI-compatible framework.

Activation#

from fastapi import FastAPI
from oc_diagnostics import init, DiagnosticsMiddleware

app = FastAPI()

# ... register routes ...

app.add_middleware(DiagnosticsMiddleware)
init()  # activate log file and stream interception

With a custom domain name:

app.add_middleware(DiagnosticsMiddleware, domain="API")

Enable HTTP logging by adding the HTTP domain to diagnostics/debug_config.json:

{
  "domains": {
    "HTTP": { "level": 2, "color": "blue" }
  }
}

Log Levels#

Level

Output

2

Request start + response end (method, path, status, elapsed ms)

3

User-Agent header; 4xx/5xx status class commentary

4

Full request and response header dumps

Header Redaction#

At L4, sensitive headers are replaced with [REDACTED] (FS-17). The default redacted headers are authorization, cookie, set-cookie, and x-api-key.

Customize the list in debug_config.json:

"global_settings": {
  "http_redact_headers": ["authorization", "cookie", "set-cookie", "x-api-key", "x-auth-token"]
}

Contracts#

  • Non-HTTP scopes (WebSocket, lifespan) pass through unchanged.

  • Application exceptions are logged at L2 and then re-raised; the middleware never swallows errors.

  • Request and response bodies are never buffered or modified.

  • Logging output only appears when DEBUG_LEVELS["HTTP"] >= 2; setting the level to 0 disables HTTP logging entirely.

  • No dependency on starlette.BaseHTTPMiddleware; pure ASGI callable.

  • Header redaction is always active at L4; it cannot be disabled without removing the headers from http_redact_headers.

Install the optional extra:

pip install oc_diagnostics[fastapi]

Self‑Test Suite#

Purpose#

Validate the entire instrumentation pipeline.

Automated Runner (tests/run_tests.py)#

The primary automated test runner for CI/CD integration.

Covers:

  • Core tests (no Dash or optional dependencies required):

    • config loading and CONFIG_SOURCE

    • _dbg silent mode

    • _StreamInterceptor interface

    • is_structural_id() classification

    • DEBUG_LEVELS mutability

  • FastAPI ASGI contract tests (no optional dependencies required):

    • HTTP scope: status code pass-through

    • WebSocket scope: non-HTTP pass-through

    • Application exceptions: logged and re-raised

  • Dash tests (auto-skip when Dash is not installed):

    • apply_instrumentation no-op when disabled

    • instrument_ids basic ID rewriting

    • ui_event_logger module import

    • list and tuple callback output instrumentation

    • instrumentation_selftest_cli subprocess

Exit codes: 0 on full pass, 1 on any failure.

CLI Tests#

instrumentation_selftest_cli.py validates:

  • static instrumentation

  • nested recursion

  • structural ID protection

  • custom components

  • debug‑level transitions

All assertions use explicit if not condition: raise AssertionError(message) so checks survive python -O optimization.

Modes:

  • default: FORCE VERBOSE

  • optional: --use-global

Dash App Tests#

instrumentation_selftest_app.py validates:

  • dynamic instrumentation

  • UI event logger

  • dispatcher patching

  • ID rewriting

  • logging behavior

Self‑Test Lifecycle#

  1. Build static layout (uninstrumented)

  2. Register UI event logger

  3. Register callbacks

  4. Enable middleware

  5. Run server

  6. Instrument dynamic content explicitly

Verbosity Modes#

  • default: FORCE VERBOSE

  • optional: --use-global

Migration Tool#

tools/migrate_to_oc_diagnostics.py migrates existing projects from legacy per-file debug.py / unified_debug.py to the oc_diagnostics package.

Usage#

# Preview changes without modifying any files
python tools/migrate_to_oc_diagnostics.py /path/to/project --dry-run

# Migrate and rename diagnostics/ → diagnostics.bak/ (default — backup preserved)
python tools/migrate_to_oc_diagnostics.py /path/to/project

# Migrate and delete the original diagnostics/ directory (no backup)
python tools/migrate_to_oc_diagnostics.py /path/to/project --delete-original

Behavior#

  • Rewrites all from debug import _dbg and related import patterns to from oc_diagnostics import _dbg.

  • Excludes common non-project directories (.git, __pycache__, venv, env, .venv, node_modules, etc.).

  • By default, renames the existing diagnostics/ directory to diagnostics.bak/ so the project config is preserved and recoverable.

  • --delete-original deletes diagnostics/ without creating a backup.

  • --dry-run previews all changes without modifying any files.

Extending the System#

  • Add new domains to diagnostics/debug_config.json.

  • Override levels at runtime with set_debug_level("DOMAIN", N) (preferred) or by mutating DEBUG_LEVELS["DOMAIN"] = N directly.

  • Override global settings at runtime with set_global_setting(key, value).

  • Use override=True in _dbg() to bypass level filtering for critical messages (e.g., always-visible lifecycle events).

  • Use dbg_timed() / dbg_profile() to measure and log elapsed time inline.

  • Use _adbg() inside async def functions for coroutine-safe logging.

  • Use OcDiagnosticsHandler to route third-party logging output through the oc_diagnostics domain-and-level system.

  • Use OC_DIAGNOSTICS_CONFIG to point multiple projects at a shared config.

  • Use UNIFIED_DEBUG_DISABLE_EXCEPTION_HOOK=1 or UNIFIED_DEBUG_DISABLE_STDIO_INTERCEPT=1 when the host environment already manages those channels.

  • Use OC_DIAGNOSTICS_MODE=prod or "mode": "prod" in config to suppress stream interception and the exception hook in production environments.

Appendix: File Map#

oc_diagnostics/
    oc_diagnostics/
        __init__.py
        unified_debug.py
        logging_handler.py
        id_instrumentation.py
        instrumentation_middleware.py
        ui_event_logger.py
        fastapi_middleware.py
        instrumentation_selftest.py
        instrumentation_selftest_cli.py
        instrumentation_selftest_app.py
    tests/
        run_tests.py
        Tests_README.md
    tools/
        migrate_to_oc_diagnostics.py
    docs/
        ARCHITECTURE.md
        CHANGELOG.md
        REFERENCE.md
        INTEGRATION_GUIDE.md
        debug_config.example.json
    README.md
    pyproject.toml