Option‑C Diagnostics Integration Guide

Option‑C Diagnostics Integration Guide#

For all Option C modules, helpers, and future files

This guide defines the Option C modernization rules, the diagnostics conventions, and the file/documentation style used across all Option C‑compliant projects.

It is concise but complete enough for any developer—new or experienced—to apply the style consistently.

For the full diagnostics system, including routing, configuration, and the _dbg implementation, see:

oc_diagnostics/docs/REFERENCE.md

Table of Contents#

1. File Header & Module Structure#

Every file must begin with a single comment line containing the full relative path:

# myproject/src/helpers/data_loader.py

Immediately after that, include a module docstring with:

  • Title

  • 1–3 sentence purpose

  • Responsibilities section

  • Diagnostics section (domain + levels)

  • Optional notes

Use section separators:

# ============================================================
# Public API
# ============================================================

This structure is required for Option C compliance.

2. _dbg Usage#

Import _dbg in every diagnostics‑enabled file:

from oc_diagnostics import _dbg

Call signature:

_dbg(domain, func, message, level)

Meaning of parameters:

  • domain — subsystem emitting the log

  • func — the logical function name (not necessarily the Python function)

  • message — concise, structured text

  • level — verbosity level (0–9)

Example:

func = "assemble_dataset"
_dbg("DATA", func, "START: year_range=...", 2)
_dbg("DATA", func, "series_keys=42", 3)
_dbg("DATA", func, "df_head=...", 4)

3. Debug Levels#

Standard levels (0–4):

0: silent  
1: errors, warnings  
2: lifecycle events  
3: semantic details (counts, shapes, selections)  
4: deep tracing (raw payloads, intermediate structures)

Extended levels (5–9):

  • For extreme verbosity or special debugging cases

  • Use sparingly and only when explicitly needed

4. Debug Domains#

Domains are defined in:

diagnostics/debug_config.json

Example:

{
  "domains": {
    "DATA": { "level": 3, "color": "cyan" },
    "UI":   { "level": 4, "color": "blue" },
    "ENGINE": 2,
    "INSTRUMENTATION": 3
  }
}

Domain selection rules:

  • Use the most specific domain available

  • If no domain fits, use "ENGINE"

  • UI helpers use "UI"

  • Data loading/analytics use "DATA"

  • Instrumentation uses "INSTRUMENTATION"

5. Function Style & Diagnostics Pattern#

Every non‑trivial function should:

  1. Define a func variable:

    func = "assemble_dataset"

  2. Emit a level‑2 log at start or end

  3. Emit level‑3 logs for semantic details

  4. Emit level‑4 logs for deep tracing

  5. Use small helpers for complex logic

Example:

def _filter_by_year_range(df, year_range, func):
    _dbg("DATA", func, "Applying year_range=...", 2)
    ...
    _dbg("DATA", func, "rows_before=..., rows_after=...", 3)
    return df

For lifecycle entry/exit tracing, use dbg_scope() (no timing) or dbg_timed() (with elapsed ms):

from oc_diagnostics import dbg_scope, dbg_timed

# Pure lifecycle (no timing):
with dbg_scope("DATA", func):
    process()

# With timing:
with dbg_timed("DATA", func, label="heavy_step"):
    heavy_step()

This pattern is required for Option C compliance.

6. Docstring & Commenting Style#

Docstrings use NumPy‑style sections:

"""
Description...

Parameters
----------
x : int
    Meaning...

Returns
-------
DataFrame
    Meaning...

Notes
-----
Additional info...
"""

Comments should be structural:

# 1. Filter by dimensions
# 2. Filter by year range
# 3. Aggregate

Avoid redundant comments.

7. Imports & Layout#

Group imports:

  1. Standard library

  2. Third‑party

  3. Project‑local

Avoid wildcard imports except for generated modules.

Example:

import json
import pandas as pd
from oc_diagnostics import _dbg
from myproject.helpers.validation import validate_input

8. Applying Option‑C to New Files#

When creating a new module:

  1. Add file path header

  2. Write module docstring

  3. Import _dbg

  4. Choose a domain

  5. Add diagnostics at levels 2–4

  6. Use small helpers

  7. Keep public API explicit (__all__ if useful)

This ensures consistency across all Option C projects.

9. Full Diagnostics System#

For the complete system, including:

  • _dbg implementation

  • log routing

  • configuration

  • environment variables

  • debug level overrides

  • output formatting

  • instrumentation

  • middleware

  • UI event logging

  • self‑test suite

See:

oc_diagnostics/docs/REFERENCE.md

10. FastAPI Integration#

oc_diagnostics ships a pure ASGI middleware that logs every HTTP request and response through _dbg(). It works with FastAPI, Starlette, or any ASGI-compatible framework.

Installation#

pip install oc_diagnostics[fastapi]

Setup#

Add DiagnosticsMiddleware to your application after registering all routes:

from fastapi import FastAPI
from oc_diagnostics import init, DiagnosticsMiddleware

app = FastAPI()

# ... register routes here ...

app.add_middleware(DiagnosticsMiddleware)
init()  # activate log file and stream interception

With a custom domain name:

app.add_middleware(DiagnosticsMiddleware, domain="API")

Enable HTTP logging#

Add the HTTP domain to your project’s diagnostics/debug_config.json:

{
  "domains": {
    "HTTP": { "level": 2, "color": "blue" }
  }
}

Level guide:

Level

Output

2

Request start + response end lines (method, path, status, elapsed ms)

3

User-Agent header; 4xx/5xx status class commentary

4

Full request and response header dumps

Security note — L4 header logging (OI-25 / FS-17): Level 4 logs request and response headers. Sensitive headers (Authorization, Cookie, Set-Cookie, X-API-Key, and any headers listed in http_redact_headers) are automatically replaced with [REDACTED] — the raw values are never written to the terminal or log file. Still, never enable HTTP level 4 in any environment where log files are not fully secured. Use level 2 or 3 for staging and production environments.

Customize the redacted header list in global_settings:

"http_redact_headers": ["authorization", "cookie", "set-cookie", "x-api-key"]

What gets logged#

[HTTP][L2][fastapi_middleware.py:93::...] START GET /api/data  client=127.0.0.1:54321
[HTTP][L2][fastapi_middleware.py:150::...] END   GET /api/data  -> 200  [4.2ms]

Contracts#

  • Non-HTTP scopes (WebSocket, lifespan) pass through unchanged.

  • Application exceptions are logged at L2 and then re-raised; the middleware never swallows errors.

  • Request and response bodies are never buffered or modified.

  • Logging output only appears when DEBUG_LEVELS["HTTP"] >= 2; setting the level to 0 disables HTTP logging entirely.

11. UI Event Logger Integration#

oc_diagnostics ships a universal UI event logger for Dash applications. It captures all user interactions on instrumented components and routes every genuine event through _dbg().

Required prerequisite — dcc.Store#

The UI event logger writes its callback output to a dcc.Store. This store must be present in your Dash layout before calling register_callbacks(). If it is absent, Dash will raise a generic callback registration error at startup with no indication of the cause.

Add the store to your layout (typically alongside other stores at the top level):

import dash_core_components as dcc  # or: from dash import dcc

layout = html.Div([
    dcc.Store(id="debug-settings-data-store", storage_type="memory"),
    # ... rest of your layout ...
])

The required ID is exported as a constant for programmatic use:

from oc_diagnostics.ui_event_logger import REQUIRED_STORE_ID
# REQUIRED_STORE_ID == "debug-settings-data-store"

A L1 diagnostic warning is emitted at register_callbacks() time when this store is not found in app.layout, making the missing dependency visible before runtime failure.

Setup#

from oc_diagnostics.ui_event_logger import register_callbacks

app = Dash(__name__)
app.layout = html.Div([
    dcc.Store(id="debug-settings-data-store", storage_type="memory"),
    # ... your components with instrumented IDs ...
])

register_callbacks(app)

Log output#

[UI][OVERRIDE][ui_event_logger.py:...] EVENT: id=my-button, property=n_clicks, value=3
[UI][OVERRIDE][ui_event_logger.py:...] EVENT: id=my-dropdown, property=value, value='option-a'

Value truncation#

Event values are truncated to 500 characters by default. For figure and data properties, only a type + length summary is logged unless DEBUG_LEVELS["UI"] >= 9:

EVENT: id=my-graph, property=figure, value=<dict len=5> (set UI level=9 for full value)

Enable UI logging#

Add the UI domain to diagnostics/debug_config.json:

{
  "domains": {
    "UI": { "level": 2, "color": "blue" }
  }
}

12. Config Hot-Reload#

oc_diagnostics can watch the debug_config.json file for changes and apply them live, without restarting the application (FS-09).

Activation#

Pass watch_config=True to init():

from oc_diagnostics import init

init(watch_config=True)

A daemon thread starts in the background, polling the config file’s modification time every 2 seconds. When a change is detected, all globals are updated in place — domain levels, colors, output mode, header redaction list, UI event properties, and more.

What gets updated#

All DEBUG_LEVELS entries, COLOR_CODES, OUTPUT_MODE, SILENT_MODE, ENABLE_TIMESTAMPS, ENABLE_COLORS, INCLUDE_FILENAME, MAX_LOG_FILES, MAX_LOG_SIZE_BYTES, HTTP_REDACT_HEADERS, UI_EVENT_PROPERTIES, INSTRUMENTATION_ALL, and INSTRUMENTATION_ENABLED.

Notes#

  • The DEBUG_LEVELS dict object itself is preserved across reloads — downstream references to the dict continue to reflect the new values.

  • Hot-reload is disabled by default; it adds file-system polling overhead that is unnecessary in short-lived scripts and CI runs.

  • Log rotation configuration (max_log_files, max_log_size_mb) is also updated, but takes effect on the next log write.

  • UI_EVENT_PROPERTIES changes take effect for the next _dbg() call but not for Dash callback inputs — Dash requires a restart to pick up property list changes.

13. Production Mode#

In production, it is often undesirable to replace sys.stdout/sys.stderr or install a custom sys.excepthook. Production mode suppresses both side-effects automatically (FS-15).

Via environment variable#

OC_DIAGNOSTICS_MODE=prod python myapp.py

Via debug_config.json#

"global_settings": {
  "mode": "prod"
}

Accepted values: "prod", "production", "safe".

The environment variable takes precedence over the config file.

Effect#

When production mode is active, init() behaves as if called with intercept_streams=False and intercept_exceptions=False regardless of the arguments passed. All _dbg() calls continue to work as normal — only the stream/exception hook side-effects are suppressed.

Combining with explicit arguments#

If you need to re-enable a specific side-effect in production, do it explicitly after calling init():

import sys
from oc_diagnostics import init

init()  # PROD_MODE suppresses interception
# Manually install the exception hook if needed:
import oc_diagnostics.unified_debug as _ud
sys.excepthook = _ud._exception_hook

14. Async Usage#

oc_diagnostics provides a native async variant of _dbg() for use inside coroutines and async frameworks such as FastAPI, asyncio, and Starlette (FS-16).

Import#

from oc_diagnostics import _adbg

Usage#

async def fetch_data(source: str) -> list:
    await _adbg("DATA", "fetch_data", f"START source={source!r}", level=2)
    result = await some_async_operation(source)
    await _adbg("DATA", "fetch_data", f"rows={len(result)}", level=3)
    return result

Usage in FastAPI endpoints#

from fastapi import FastAPI
from oc_diagnostics import _adbg, init

app = FastAPI()
init()

@app.get("/items/{item_id}")
async def read_item(item_id: int):
    await _adbg("API", "read_item", f"START id={item_id}", level=2)
    item = await db.fetch(item_id)
    await _adbg("API", "read_item", f"found={item is not None}", level=3)
    return item

Key behaviors#

  • Caller frame (filename, line number, function name) is captured synchronously before any await, ensuring accurate location information.

  • Terminal writes are synchronous — no added latency on the event loop.

  • File I/O is offloaded to asyncio.to_thread() — the event loop is never blocked.

  • Signature is identical to _dbg(); drop-in replacement inside async def.

  • Domain-level filtering applies the same way as for _dbg().

15. Python logging Integration#

Third-party libraries and legacy code that use Python’s standard logging module can be routed through oc_diagnostics without modification (FS-08).

Setup#

import logging
from oc_diagnostics import OcDiagnosticsHandler

# Route all httpx logs to the HTTP domain
httpx_logger = logging.getLogger("httpx")
httpx_logger.addHandler(OcDiagnosticsHandler(domain="HTTP"))
httpx_logger.setLevel(logging.DEBUG)

# Route your own application logs to GENERAL
app_logger = logging.getLogger("myapp")
app_logger.addHandler(OcDiagnosticsHandler(domain="GENERAL"))
app_logger.setLevel(logging.INFO)

Default level mapping#

logging level

_dbg level

DEBUG

4

INFO

2

WARNING

1

ERROR

1

CRITICAL

1

Custom level mapping#

import logging
from oc_diagnostics import OcDiagnosticsHandler

handler = OcDiagnosticsHandler(
    domain="SQLALCHEMY",
    level_map={
        logging.DEBUG:   4,
        logging.INFO:    3,
        logging.WARNING: 2,
        logging.ERROR:   1,
    },
)
logging.getLogger("sqlalchemy.engine").addHandler(handler)

Notes#

  • OcDiagnosticsHandler is always available — it has no optional dependencies.

  • You can attach it to the root logger to capture all library output:

    logging.getLogger().addHandler(OcDiagnosticsHandler(domain="GENERAL"))

  • _dbg() level filtering still applies; ensure the domain has a non-zero level in debug_config.json for records to appear.

16. Log Rotation#

oc_diagnostics supports two complementary log rotation strategies that can be active simultaneously (FS-06).

Count-based rotation#

Keeps at most N log files. When the count exceeds the limit, the oldest files are deleted before a new file is created.

"global_settings": {
  "max_log_files": 10
}

Size-based rotation#

Rolls over to a new timestamped file when the current file exceeds a size threshold.

"global_settings": {
  "max_log_size_mb": 5.0
}

Combined configuration#

"global_settings": {
  "max_log_files": 10,
  "max_log_size_mb": 5.0
}

Set either value to 0 (or omit it) to disable that strategy.

Notes#

  • Log files are named unified_debug_logs_<timestamp>.txt (or .jsonl in JSON mode).

  • Count-based pruning runs at startup when the log directory is first created or accessed.

  • Size-based rotation is checked on every write; no background thread is required.

  • Both strategies are compatible with init(watch_config=True) — the thresholds can be changed live without restarting.