production-analytics

production-analytics is a small Python service that derives production analytics from ENLYZE data for Grafana. ENLYZE remains the authority for raw process data; this project persists only derived metrics, events, relevant production-order context, and calculation state.

Status

This repository includes an ENLYZE exploration CLI, a production-run/timeseries gateway, and single-cycle live material polling with JSON state persistence. Database migrations, a continuous polling runner, and HTTP endpoints remain pending.

Intended flow

ENLYZE (raw data) -> retrieval boundary -> calculations -> TimescaleDB -> Grafana
                                      \-> calculation provenance/state

Production orders are the primary attribution context for results. Every persisted result will ultimately be traceable to its machine, order/article context, source time range, and calculation implementation version.

Development

Requires Python 3.11 or newer. Install the project and development tools:

python3 -m pip install -e '.[dev]'
pytest
ruff check .

Copy .env.example to .env for public/local configuration documentation. Put real ENLYZE credentials only in secrets/enlyze.env, which is ignored. The CLI reads that file by default without executing it; it accepts only simple KEY=VALUE lines (quoted values are supported). Environment variables may be used instead when appropriate. Never pass keys as command-line arguments.

The generic request command only performs GET requests and requires the operator to provide an API path that is known to be safe and authorized:

production-analytics enlyze raw /verified/path --pretty
production-analytics enlyze raw /verified/path --save-fixture response.json

Public ENLYZE documentation verifies Bearer-token authentication. Put ENLYZE_API_KEY='...' in the local secret file; the exploration client sends it as an Authorization: Bearer … header and never prints that header/value.

The second command saves a sanitized, reviewable fixture under fixtures/enlyze/ by default. Raw captures belong in the ignored data/raw/enlyze/ directory and must not be committed.

The OpenAPI server URL is https://app.enlyze.com/api/; its operation paths begin with /v2/. Set ENLYZE_BASE_URL to that server URL, not to an operation path. The documented read-only time-series operation is exposed for exploration as production-analytics enlyze timeseries.

The current Compose file has no application service, so it deliberately does not pass ENLYZE credentials to TimescaleDB. A future application service should use env_file: ./secrets/enlyze.env rather than copying secrets into Compose.

Material-consumption integration

MaterialConsumptionIntegrator in calculations accepts MaterialSample values (timezone-aware timestamp, material rate in kg/h, numeric gate value). Configure a strict gate_value > gate_threshold condition and an explicit, positive max_sample_gap_seconds. Each interval uses the preceding sample's rate and gate, converting elapsed seconds to hours to accumulate kg. A longer gap contributes neither consumption nor running time and resets the baseline to the newer sample. No time before the first or after the last sample is inferred.

Call process(sample) or process_many(samples) on the same instance for live input, replay, or successive chunks. Both use the same calculation. The immutable state snapshot exposes cumulative kg, integrated running seconds, last timestamp (UTC), last rate, last gate, and whether the latest gate is active. State restoration and JSON persistence are supported. Naive timestamps and non-finite values are rejected; backwards timestamps raise without changing state. Duplicate timestamps add no consumption but replace the baseline in arrival order. Finite negative material rates are currently accepted and decrease cumulative consumption during affected integrated intervals. This is intentional generic behavior for now; machine-specific validation or clamping may be added later at the input/adapter layer if required by process semantics.

Run python scripts/validate_k7_material_consumption.py from the repository root with the ignored k7-00842-throughput.raw.json and k7-00842-speed.raw.json captures in data/raw/enlyze/. The utility joins common non-null timestamps without filling values and rejects unordered or duplicate capture records. K7-specific signal UUIDs are confined to the utility: Stundenleistung Anlage is gated by Geschwindigkeit Gesamtanlage > 0.5 m/min. With the explicit 20-second validation gap limit (--max-sample-gap-seconds to override), 2,672 common samples yield 5.205555556 h and 5,180.127150811 kg for run 00842, matching the previous manual calculation's rounded results. Synthetic tests verify equivalence to manual interval integration without local captures or live ENLYZE access. Grafana and Bento 1 remain pending.

MaterialPollingService.poll_once(now=...) starts at the open run's start or resumes inclusively at its last processed timestamp. Duplicate boundary samples add no time. A new run UUID for the same order retains cumulative kg and running seconds but resets the temporal baseline; different orders and machines have separate state. Empty responses add no consumption. Naive now is rejected; no open run or a window beginning after now returns None without saving.

JsonMaterialStateStore uses deterministic SHA-256 filenames derived from the machine/order pair, preventing path traversal and sanitized-name collisions. It flushes and fsyncs a temporary file in the state directory before atomic replacement; a failed write leaves the previous primary file intact. Missing state returns None; malformed JSON or state raises ValueError with file context. Use an ignored runtime directory and one polling writer per state key; atomic replacement does not coordinate concurrent read-modify-write cycles. Files from the earlier uncommitted sanitized-filename prototype are not loaded under the new names. The gateway rejects ambiguous open runs, naive windows, and missing columns or malformed records instead of silently skipping them.

Peak-cycle detection

PeakCycleDetector is a pure calculation-domain component for roll length, roll weight, and comparable sawtooth/batch signals. It retains the current maximum and emits it only after the value is below drop_ratio * current_peak continuously for hold_seconds. Configuration is min_peak, drop_ratio (strictly between 0 and 1), non-negative hold_seconds, and positive max_sample_gap_seconds. These are per-signal configuration parameters, not globally fixed process constants. The hold uses elapsed timestamps, never a sample count; an interval longer than the configured maximum sample gap restarts a reset candidate, so a timestamp gap alone is not evidence that a signal remained below threshold. max_sample_gap_seconds is required rather than defaulted, so the source's continuity assumption is explicit for each detector configuration. Reset values need not approach zero (and can be negative), and peak magnitude can vary materially between cycles. Equal timestamps are accepted in arrival order but add no elapsed hold time; backwards timestamps are rejected.

Real-world validations use ignored local captures and are not part of the test suite. K7 roll length (m), variable bf81c547-dccf-4709-aee2-0f78366d1dfc, is the clean reference case: its 10-second capture produced exactly four plausible peaks of about 49.98 m with min_peak=10, drop_ratio=0.75, hold_seconds=60, and max_sample_gap_seconds=20. Run python scripts/validate_k7_roll_length.py when that capture is available.

Bento 2 roll weight (kg), machine 6201f3de-b940-45d7-930c-1481c5ae9e1d, variable 059c0015-9aff-4270-a4ce-f21b83f99378, is the more realistic robustness case. Its observed 10-second signal has plateaus, reset values of roughly 300–470 kg, and materially varying peak heights. With min_peak=800, drop_ratio=0.75, hold_seconds=60, and max_sample_gap_seconds=20, the extended capture produces 13 plausible completed peaks, including a legitimate approximately 1195 kg cycle. Run python scripts/validate_bento2_roll_weight.py when the ignored capture is available. Neither capture should be added to version control or automated tests.

docker compose up -d timescaledb is an optional local database design for a future persistence milestone. It is not required for the bootstrap tests.

See PROJECT_KNOWLEDGE.md for durable project context and docs/roadmap.md for the implementation sequence.