# Option‑C Diagnostics Integration Guide  
For all Option C modules, helpers, and future files

This guide defines the **Option C modernization rules**, the **diagnostics conventions**, and the **file/documentation style** used across all Option C‑compliant projects.

It is concise but complete enough for any developer—new or experienced—to apply the style consistently.

For the **full diagnostics system**, including routing, configuration, and the `_dbg` implementation, see:

```
oc_diagnostics/docs/REFERENCE.md
```

---

# Table of Contents
- [1. File Header & Module Structure](#1-file-header--module-structure)
- [2. `_dbg` Usage](#2-_dbg-usage)
- [3. Debug Levels](#3-debug-levels)
- [4. Debug Domains](#4-debug-domains)
- [5. Function Style & Diagnostics Pattern](#5-function-style--diagnostics-pattern)
- [6. Docstring & Commenting Style](#6-docstring--commenting-style)
- [7. Imports & Layout](#7-imports--layout)
- [8. Applying Option‑C to New Files](#8-applying-optionc-to-new-files)
- [9. Full Diagnostics System](#9-full-diagnostics-system)
- [10. FastAPI Integration](#10-fastapi-integration)
- [11. UI Event Logger Integration](#11-ui-event-logger-integration)
- [12. Config Hot-Reload](#12-config-hot-reload)
- [13. Production Mode](#13-production-mode)
- [14. Async Usage](#14-async-usage)
- [15. Python logging Integration](#15-python-logging-integration)
- [16. Log Rotation](#16-log-rotation)

---

# 1. File Header & Module Structure

Every file must begin with a single comment line containing the full relative path:

```
# myproject/src/helpers/data_loader.py
```

Immediately after that, include a module docstring with:

- Title  
- 1–3 sentence purpose  
- Responsibilities section  
- Diagnostics section (domain + levels)  
- Optional notes  

Use section separators:

```python
# ============================================================
# Public API
# ============================================================
```

This structure is required for Option C compliance.

---

# 2. `_dbg` Usage

Import `_dbg` in every diagnostics‑enabled file:

```python
from oc_diagnostics import _dbg
```

Call signature:

```python
_dbg(domain, func, message, level)
```

Meaning of parameters:

- **domain** — subsystem emitting the log  
- **func** — the *logical* function name (not necessarily the Python function)  
- **message** — concise, structured text  
- **level** — verbosity level (0–9)  

Example:

```python
func = "assemble_dataset"
_dbg("DATA", func, "START: year_range=...", 2)
_dbg("DATA", func, "series_keys=42", 3)
_dbg("DATA", func, "df_head=...", 4)
```

---

# 3. Debug Levels

Standard levels (0–4):

```
0: silent  
1: errors, warnings  
2: lifecycle events  
3: semantic details (counts, shapes, selections)  
4: deep tracing (raw payloads, intermediate structures)
```

Extended levels (5–9):

- For extreme verbosity or special debugging cases  
- Use sparingly and only when explicitly needed  

---

# 4. Debug Domains

Domains are defined in:

```
diagnostics/debug_config.json
```

Example:

```json
{
  "domains": {
    "DATA": { "level": 3, "color": "cyan" },
    "UI":   { "level": 4, "color": "blue" },
    "ENGINE": 2,
    "INSTRUMENTATION": 3
  }
}
```

Domain selection rules:

- Use the most specific domain available  
- If no domain fits, use `"ENGINE"`  
- UI helpers use `"UI"`  
- Data loading/analytics use `"DATA"`  
- Instrumentation uses `"INSTRUMENTATION"`  

---

# 5. Function Style & Diagnostics Pattern

Every non‑trivial function should:

1. Define a `func` variable:

   ```python
   func = "assemble_dataset"
   ```

2. Emit a level‑2 log at start or end  
3. Emit level‑3 logs for semantic details  
4. Emit level‑4 logs for deep tracing  
5. Use small helpers for complex logic  

Example:

```python
def _filter_by_year_range(df, year_range, func):
    _dbg("DATA", func, "Applying year_range=...", 2)
    ...
    _dbg("DATA", func, "rows_before=..., rows_after=...", 3)
    return df
```

For lifecycle entry/exit tracing, use `dbg_scope()` (no timing) or `dbg_timed()`
(with elapsed ms):

```python
from oc_diagnostics import dbg_scope, dbg_timed

# Pure lifecycle (no timing):
with dbg_scope("DATA", func):
    process()

# With timing:
with dbg_timed("DATA", func, label="heavy_step"):
    heavy_step()
```

This pattern is required for Option C compliance.

---

# 6. Docstring & Commenting Style

Docstrings use NumPy‑style sections:

```python
"""
Description...

Parameters
----------
x : int
    Meaning...

Returns
-------
DataFrame
    Meaning...

Notes
-----
Additional info...
"""
```

Comments should be structural:

```python
# 1. Filter by dimensions
# 2. Filter by year range
# 3. Aggregate
```

Avoid redundant comments.

---

# 7. Imports & Layout

Group imports:

1. Standard library  
2. Third‑party  
3. Project‑local  

Avoid wildcard imports except for generated modules.

Example:

```python
import json
import pandas as pd
from oc_diagnostics import _dbg
from myproject.helpers.validation import validate_input
```

---

# 8. Applying Option‑C to New Files

When creating a new module:

1. Add file path header  
2. Write module docstring  
3. Import `_dbg`  
4. Choose a domain  
5. Add diagnostics at levels 2–4  
6. Use small helpers  
7. Keep public API explicit (`__all__` if useful)  

This ensures consistency across all Option C projects.

---

# 9. Full Diagnostics System

For the complete system, including:

- `_dbg` implementation
- log routing
- configuration
- environment variables
- debug level overrides
- output formatting
- instrumentation
- middleware
- UI event logging
- self‑test suite

See:

```
oc_diagnostics/docs/REFERENCE.md
```

---

# 10. FastAPI Integration

`oc_diagnostics` ships a pure ASGI middleware that logs every HTTP request and
response through `_dbg()`.  It works with **FastAPI, Starlette, or any
ASGI-compatible framework**.

## Installation

```bash
pip install oc_diagnostics[fastapi]
```

## Setup

Add `DiagnosticsMiddleware` to your application **after** registering all routes:

```python
from fastapi import FastAPI
from oc_diagnostics import init, DiagnosticsMiddleware

app = FastAPI()

# ... register routes here ...

app.add_middleware(DiagnosticsMiddleware)
init()  # activate log file and stream interception
```

With a custom domain name:

```python
app.add_middleware(DiagnosticsMiddleware, domain="API")
```

## Enable HTTP logging

Add the `HTTP` domain to your project's `diagnostics/debug_config.json`:

```json
{
  "domains": {
    "HTTP": { "level": 2, "color": "blue" }
  }
}
```

Level guide:

| Level | Output |
|---|---|
| 2 | Request start + response end lines (method, path, status, elapsed ms) |
| 3 | User-Agent header; 4xx/5xx status class commentary |
| 4 | Full request and response header dumps |

> **Security note — L4 header logging (OI-25 / FS-17):** Level 4 logs request
> and response headers.  Sensitive headers (`Authorization`, `Cookie`, `Set-Cookie`,
> `X-API-Key`, and any headers listed in `http_redact_headers`) are automatically
> replaced with `[REDACTED]` — the raw values are never written to the terminal
> or log file.  **Still, never enable `HTTP` level 4 in any environment where
> log files are not fully secured.**  Use level 2 or 3 for staging and
> production environments.
>
> Customize the redacted header list in `global_settings`:
> ```json
> "http_redact_headers": ["authorization", "cookie", "set-cookie", "x-api-key"]
> ```

## What gets logged

```
[HTTP][L2][fastapi_middleware.py:93::...] START GET /api/data  client=127.0.0.1:54321
[HTTP][L2][fastapi_middleware.py:150::...] END   GET /api/data  -> 200  [4.2ms]
```

## Contracts

- Non-HTTP scopes (WebSocket, lifespan) pass through unchanged.
- Application exceptions are logged at L2 and then re-raised; the middleware
  never swallows errors.
- Request and response bodies are never buffered or modified.
- Logging output only appears when `DEBUG_LEVELS["HTTP"] >= 2`; setting the
  level to `0` disables HTTP logging entirely.

---

# 11. UI Event Logger Integration

`oc_diagnostics` ships a universal UI event logger for Dash applications.
It captures all user interactions on instrumented components and routes
every genuine event through `_dbg()`.

## Required prerequisite — `dcc.Store`

> **The UI event logger writes its callback output to a `dcc.Store`.**
> This store **must** be present in your Dash layout before calling
> `register_callbacks()`.  If it is absent, Dash will raise a generic
> callback registration error at startup with no indication of the cause.

Add the store to your layout (typically alongside other stores at the top
level):

```python
import dash_core_components as dcc  # or: from dash import dcc

layout = html.Div([
    dcc.Store(id="debug-settings-data-store", storage_type="memory"),
    # ... rest of your layout ...
])
```

The required ID is exported as a constant for programmatic use:

```python
from oc_diagnostics.ui_event_logger import REQUIRED_STORE_ID
# REQUIRED_STORE_ID == "debug-settings-data-store"
```

A **L1 diagnostic warning** is emitted at `register_callbacks()` time when
this store is not found in `app.layout`, making the missing dependency
visible before runtime failure.

## Setup

```python
from oc_diagnostics.ui_event_logger import register_callbacks

app = Dash(__name__)
app.layout = html.Div([
    dcc.Store(id="debug-settings-data-store", storage_type="memory"),
    # ... your components with instrumented IDs ...
])

register_callbacks(app)
```

## Log output

```
[UI][OVERRIDE][ui_event_logger.py:...] EVENT: id=my-button, property=n_clicks, value=3
[UI][OVERRIDE][ui_event_logger.py:...] EVENT: id=my-dropdown, property=value, value='option-a'
```

## Value truncation

Event values are truncated to 500 characters by default.  For `figure` and
`data` properties, only a type + length summary is logged unless
`DEBUG_LEVELS["UI"] >= 9`:

```
EVENT: id=my-graph, property=figure, value=<dict len=5> (set UI level=9 for full value)
```

## Enable UI logging

Add the `UI` domain to `diagnostics/debug_config.json`:

```json
{
  "domains": {
    "UI": { "level": 2, "color": "blue" }
  }
}
```

---

# 12. Config Hot-Reload

`oc_diagnostics` can watch the `debug_config.json` file for changes and apply them
live, without restarting the application (FS-09).

## Activation

Pass `watch_config=True` to `init()`:

```python
from oc_diagnostics import init

init(watch_config=True)
```

A daemon thread starts in the background, polling the config file's modification time
every 2 seconds.  When a change is detected, all globals are updated in place — domain
levels, colors, output mode, header redaction list, UI event properties, and more.

## What gets updated

All `DEBUG_LEVELS` entries, `COLOR_CODES`, `OUTPUT_MODE`, `SILENT_MODE`,
`ENABLE_TIMESTAMPS`, `ENABLE_COLORS`, `INCLUDE_FILENAME`, `MAX_LOG_FILES`,
`MAX_LOG_SIZE_BYTES`, `HTTP_REDACT_HEADERS`, `UI_EVENT_PROPERTIES`,
`INSTRUMENTATION_ALL`, and `INSTRUMENTATION_ENABLED`.

## Notes

- The `DEBUG_LEVELS` dict object itself is preserved across reloads — downstream
  references to the dict continue to reflect the new values.
- Hot-reload is disabled by default; it adds file-system polling overhead that is
  unnecessary in short-lived scripts and CI runs.
- Log rotation configuration (`max_log_files`, `max_log_size_mb`) is also updated,
  but takes effect on the next log write.
- `UI_EVENT_PROPERTIES` changes take effect for the *next* `_dbg()` call but
  **not** for Dash callback inputs — Dash requires a restart to pick up property list changes.

---

# 13. Production Mode

In production, it is often undesirable to replace `sys.stdout`/`sys.stderr` or
install a custom `sys.excepthook`.  Production mode suppresses both side-effects
automatically (FS-15).

## Via environment variable

```bash
OC_DIAGNOSTICS_MODE=prod python myapp.py
```

## Via `debug_config.json`

```json
"global_settings": {
  "mode": "prod"
}
```

Accepted values: `"prod"`, `"production"`, `"safe"`.

The environment variable takes precedence over the config file.

## Effect

When production mode is active, `init()` behaves as if called with
`intercept_streams=False` and `intercept_exceptions=False` regardless of the
arguments passed.  All `_dbg()` calls continue to work as normal — only the
stream/exception hook side-effects are suppressed.

## Combining with explicit arguments

If you need to re-enable a specific side-effect in production, do it explicitly after
calling `init()`:

```python
import sys
from oc_diagnostics import init

init()  # PROD_MODE suppresses interception
# Manually install the exception hook if needed:
import oc_diagnostics.unified_debug as _ud
sys.excepthook = _ud._exception_hook
```

---

# 14. Async Usage

`oc_diagnostics` provides a native async variant of `_dbg()` for use inside coroutines
and async frameworks such as FastAPI, asyncio, and Starlette (FS-16).

## Import

```python
from oc_diagnostics import _adbg
```

## Usage

```python
async def fetch_data(source: str) -> list:
    await _adbg("DATA", "fetch_data", f"START source={source!r}", level=2)
    result = await some_async_operation(source)
    await _adbg("DATA", "fetch_data", f"rows={len(result)}", level=3)
    return result
```

## Usage in FastAPI endpoints

```python
from fastapi import FastAPI
from oc_diagnostics import _adbg, init

app = FastAPI()
init()

@app.get("/items/{item_id}")
async def read_item(item_id: int):
    await _adbg("API", "read_item", f"START id={item_id}", level=2)
    item = await db.fetch(item_id)
    await _adbg("API", "read_item", f"found={item is not None}", level=3)
    return item
```

## Key behaviors

- Caller frame (filename, line number, function name) is captured **synchronously**
  before any `await`, ensuring accurate location information.
- Terminal writes are synchronous — no added latency on the event loop.
- File I/O is offloaded to `asyncio.to_thread()` — the event loop is never blocked.
- Signature is identical to `_dbg()`; drop-in replacement inside `async def`.
- Domain-level filtering applies the same way as for `_dbg()`.

---

# 15. Python logging Integration

Third-party libraries and legacy code that use Python's standard `logging` module
can be routed through `oc_diagnostics` without modification (FS-08).

## Setup

```python
import logging
from oc_diagnostics import OcDiagnosticsHandler

# Route all httpx logs to the HTTP domain
httpx_logger = logging.getLogger("httpx")
httpx_logger.addHandler(OcDiagnosticsHandler(domain="HTTP"))
httpx_logger.setLevel(logging.DEBUG)

# Route your own application logs to GENERAL
app_logger = logging.getLogger("myapp")
app_logger.addHandler(OcDiagnosticsHandler(domain="GENERAL"))
app_logger.setLevel(logging.INFO)
```

## Default level mapping

| `logging` level | `_dbg` level |
|---|---|
| `DEBUG` | 4 |
| `INFO` | 2 |
| `WARNING` | 1 |
| `ERROR` | 1 |
| `CRITICAL` | 1 |

## Custom level mapping

```python
import logging
from oc_diagnostics import OcDiagnosticsHandler

handler = OcDiagnosticsHandler(
    domain="SQLALCHEMY",
    level_map={
        logging.DEBUG:   4,
        logging.INFO:    3,
        logging.WARNING: 2,
        logging.ERROR:   1,
    },
)
logging.getLogger("sqlalchemy.engine").addHandler(handler)
```

## Notes

- `OcDiagnosticsHandler` is always available — it has no optional dependencies.
- You can attach it to the root logger to capture all library output:
  ```python
  logging.getLogger().addHandler(OcDiagnosticsHandler(domain="GENERAL"))
  ```
- `_dbg()` level filtering still applies; ensure the domain has a non-zero level
  in `debug_config.json` for records to appear.

---

# 16. Log Rotation

`oc_diagnostics` supports two complementary log rotation strategies that can be
active simultaneously (FS-06).

## Count-based rotation

Keeps at most N log files.  When the count exceeds the limit, the oldest files
are deleted before a new file is created.

```json
"global_settings": {
  "max_log_files": 10
}
```

## Size-based rotation

Rolls over to a new timestamped file when the current file exceeds a size threshold.

```json
"global_settings": {
  "max_log_size_mb": 5.0
}
```

## Combined configuration

```json
"global_settings": {
  "max_log_files": 10,
  "max_log_size_mb": 5.0
}
```

Set either value to `0` (or omit it) to disable that strategy.

## Notes

- Log files are named `unified_debug_logs_<timestamp>.txt` (or `.jsonl` in JSON mode).
- Count-based pruning runs at startup when the log directory is first created or accessed.
- Size-based rotation is checked on every write; no background thread is required.
- Both strategies are compatible with `init(watch_config=True)` — the thresholds
  can be changed live without restarting.

---