Add continuous material polling runner
This commit is contained in:
@@ -29,3 +29,6 @@ secrets/*
|
|||||||
.DS_Store
|
.DS_Store
|
||||||
|
|
||||||
activate.sh
|
activate.sh
|
||||||
|
|
||||||
|
# Live polling checkpoints
|
||||||
|
data/state/
|
||||||
|
|||||||
@@ -8,8 +8,10 @@ production-order context, and calculation state.
|
|||||||
## Status
|
## Status
|
||||||
|
|
||||||
This repository includes an ENLYZE exploration CLI, a production-run/timeseries
|
This repository includes an ENLYZE exploration CLI, a production-run/timeseries
|
||||||
gateway, and single-cycle live material polling with JSON state persistence.
|
gateway, and a configured continuous material polling runner with atomic JSON
|
||||||
Database migrations, a continuous polling runner, and HTTP endpoints remain pending.
|
checkpoints. The live material-consumption MVP remains in progress: Grafana-oriented
|
||||||
|
derived-total persistence/exposure and database migrations are still pending.
|
||||||
|
There are no HTTP endpoints.
|
||||||
|
|
||||||
## Intended flow
|
## Intended flow
|
||||||
|
|
||||||
@@ -89,8 +91,8 @@ Run `python scripts/validate_k7_material_consumption.py` from the repository roo
|
|||||||
with the ignored `k7-00842-throughput.raw.json` and `k7-00842-speed.raw.json`
|
with the ignored `k7-00842-throughput.raw.json` and `k7-00842-speed.raw.json`
|
||||||
captures in `data/raw/enlyze/`. The utility joins common non-null timestamps
|
captures in `data/raw/enlyze/`. The utility joins common non-null timestamps
|
||||||
without filling values and rejects unordered or duplicate capture records.
|
without filling values and rejects unordered or duplicate capture records.
|
||||||
K7-specific signal UUIDs are confined to the utility: `Stundenleistung Anlage`
|
K7-specific signal UUIDs are declared in the runner configuration and validation
|
||||||
is gated by `Geschwindigkeit Gesamtanlage > 0.5 m/min`. With the explicit
|
utility: `Stundenleistung Anlage` is gated by `Geschwindigkeit Gesamtanlage > 0.5 m/min`. With the explicit
|
||||||
20-second validation gap limit (`--max-sample-gap-seconds` to override),
|
20-second validation gap limit (`--max-sample-gap-seconds` to override),
|
||||||
2,672 common samples yield **5.205555556 h** and **5,180.127150811 kg** for run
|
2,672 common samples yield **5.205555556 h** and **5,180.127150811 kg** for run
|
||||||
00842, matching the previous manual calculation's rounded results. Synthetic
|
00842, matching the previous manual calculation's rounded results. Synthetic
|
||||||
@@ -115,6 +117,68 @@ Files from the earlier uncommitted sanitized-filename prototype are not loaded
|
|||||||
under the new names. The gateway rejects ambiguous open runs, naive windows,
|
under the new names. The gateway rejects ambiguous open runs, naive windows,
|
||||||
and missing columns or malformed records instead of silently skipping them.
|
and missing columns or malformed records instead of silently skipping them.
|
||||||
|
|
||||||
|
## Continuous material polling
|
||||||
|
|
||||||
|
The committed [K7 configuration](config/k7-material-consumption.yaml) declares
|
||||||
|
`k7-fiber-consumption`, a `material_consumption` calculation at version `"1"`.
|
||||||
|
It selects `Stundenleistung Anlage` (kg/h) as the rate and `Geschwindigkeit
|
||||||
|
Gesamtanlage` (m/min) as the gate, integrating only when the gate is **strictly
|
||||||
|
above 0.5**, with a maximum sample gap of 20 seconds. `Anlage läuft` is not the
|
||||||
|
primary gate. The output is `material_consumption` in `kg`, grouped by
|
||||||
|
`production_order`; production orders remain opaque strings, including leading
|
||||||
|
zeros and whitespace.
|
||||||
|
|
||||||
|
Install the declared dependencies in the repository venv, then run from the
|
||||||
|
repository root:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
.venv/bin/python -m pip install -e '.[dev]'
|
||||||
|
.venv/bin/python -m production_analytics.cli run material-poll \
|
||||||
|
--config config/k7-material-consumption.yaml \
|
||||||
|
--calculation-id k7-fiber-consumption \
|
||||||
|
--poll-interval-seconds 10 \
|
||||||
|
--state-directory data/state/material \
|
||||||
|
--secrets-file secrets/enlyze.env
|
||||||
|
```
|
||||||
|
|
||||||
|
The installed `production-analytics run material-poll` command is equivalent.
|
||||||
|
The interval, state directory, and secrets path above are defaults. Paths are
|
||||||
|
relative to the current working directory. Existing `ENLYZE_BASE_URL`,
|
||||||
|
`ENLYZE_API_KEY`, and `ENLYZE_HTTP_TIMEOUT_SECONDS` handling is reused; values
|
||||||
|
in the secrets file override environment values. Keys are never CLI arguments.
|
||||||
|
|
||||||
|
Keep calculation fields in YAML and technical runtime settings in CLI options.
|
||||||
|
The loader requires every field shown in the K7 file, rejects unknown fields,
|
||||||
|
duplicate keys/ids, unsupported types/versions/outputs, and invalid or non-finite
|
||||||
|
numbers. Numeric fields must be YAML numbers, and `version` must be quoted.
|
||||||
|
Multiple material calculations may share a file, but then `--calculation-id` is
|
||||||
|
required. All entries are validated before selection. The unrelated
|
||||||
|
`config/calculations.example.yaml` remains illustrative and is not executable
|
||||||
|
by this material-only runner.
|
||||||
|
|
||||||
|
The foreground runner polls once immediately using UTC, then sleeps for the
|
||||||
|
configured positive interval after each completed cycle, including failures.
|
||||||
|
A slow poll delays the next cycle; there is no overlap or catch-up scheduling.
|
||||||
|
Ctrl-C/SIGINT exits cleanly. Successful cycles print machine, opaque order,
|
||||||
|
cumulative kg, integrated running seconds, and cycle-start timestamp. No open
|
||||||
|
run (or no eligible window) is normal and visible. Errors on stderr identify
|
||||||
|
state loading/saving or ENLYZE/gateway/polling and the exception class; arbitrary
|
||||||
|
exception text and response bodies are withheld to avoid leaking credentials.
|
||||||
|
Startup/configuration failures return non-zero; cycle failures retry after the
|
||||||
|
delay without replacing checkpoints with empty state.
|
||||||
|
|
||||||
|
Checkpoints live in ignored `data/state/material/` by default, with one hashed
|
||||||
|
JSON filename per machine/order pair. Run **one polling writer per state key**;
|
||||||
|
there is no multi-process locking. Use a separate state directory when changing
|
||||||
|
calculation parameters or running a different calculation for the same machine
|
||||||
|
and order: checkpoints are not namespaced by calculation id/version. These files
|
||||||
|
are integration checkpoints, not a Grafana metric store.
|
||||||
|
|
||||||
|
This runner exposes no Grafana/TimescaleDB metric yet. Grafana-oriented derived-total
|
||||||
|
persistence/exposure remains the next milestone; the MVP is not complete.
|
||||||
|
Bento 1 bentonite source and gate selection remain intentionally undefined pending
|
||||||
|
process validation; the generic integration core is unchanged and reusable.
|
||||||
|
|
||||||
## Peak-cycle detection
|
## Peak-cycle detection
|
||||||
|
|
||||||
`PeakCycleDetector` is a pure calculation-domain component for roll length,
|
`PeakCycleDetector` is a pure calculation-domain component for roll length,
|
||||||
|
|||||||
@@ -0,0 +1,14 @@
|
|||||||
|
# Validated K7: Stundenleistung Anlage (kg/h), gated by
|
||||||
|
# Geschwindigkeit Gesamtanlage (m/min) > 0.5. Not Anlage läuft.
|
||||||
|
calculations:
|
||||||
|
- id: k7-fiber-consumption
|
||||||
|
type: material_consumption
|
||||||
|
version: "1"
|
||||||
|
machine_ref: c220f95c-a65e-4cb7-99b7-0626d6c7508c
|
||||||
|
rate_signal_ref: c9d06af5-f6d6-4ede-b6c4-5a98bac77129
|
||||||
|
gate_signal_ref: 823867bb-f5d2-40eb-b875-657155addfd0
|
||||||
|
gate_threshold: 0.5
|
||||||
|
max_sample_gap_seconds: 20.0
|
||||||
|
output_metric: material_consumption
|
||||||
|
output_unit: kg
|
||||||
|
group_by: production_order
|
||||||
+13
-4
@@ -15,10 +15,19 @@ Continue to record sanitized fixtures for newly verified semantics.
|
|||||||
|
|
||||||
## 2. Live material-consumption MVP (in progress)
|
## 2. Live material-consumption MVP (in progress)
|
||||||
|
|
||||||
The integration engine, ENLYZE gateway, single-cycle polling service, and atomic
|
The integration engine, ENLYZE gateway, single-cycle polling service, atomic
|
||||||
JSON checkpoint store are implemented and covered by synthetic tests. Next:
|
JSON checkpoint store, and configured continuous foreground polling runner are
|
||||||
wire a configured continuous polling runner and expose/store derived totals
|
implemented and covered by synthetic tests. Configure the validated K7 instance
|
||||||
for Grafana, followed by the TimescaleDB persistence foundation below.
|
in `config/k7-material-consumption.yaml`; start it with
|
||||||
|
`production-analytics run material-poll --config config/k7-material-consumption.yaml`.
|
||||||
|
Runtime options select the polling interval, state directory (default: ignored
|
||||||
|
`data/state/material/`), and existing secrets-file handling. Only one writer may
|
||||||
|
poll each machine/order state key.
|
||||||
|
|
||||||
|
The runner exposes no Grafana/TimescaleDB metric yet and does not complete the
|
||||||
|
MVP. Grafana-oriented derived-total persistence/exposure is the next milestone,
|
||||||
|
with the TimescaleDB persistence foundation below. No HTTP endpoints, application
|
||||||
|
containers, daemonization, or schedulers have been added.
|
||||||
|
|
||||||
Detect the currently active Production Run and production order, continuously
|
Detect the currently active Production Run and production order, continuously
|
||||||
ingest new ENLYZE samples, maintain persistent incremental integration state,
|
ingest new ENLYZE samples, maintain persistent incremental integration state,
|
||||||
|
|||||||
+1
-1
@@ -10,7 +10,7 @@ readme = "README.md"
|
|||||||
requires-python = ">=3.11"
|
requires-python = ">=3.11"
|
||||||
license = { text = "Proprietary" }
|
license = { text = "Proprietary" }
|
||||||
authors = [{ name = "Production Analytics Team" }]
|
authors = [{ name = "Production Analytics Team" }]
|
||||||
dependencies = []
|
dependencies = ["PyYAML>=6.0"]
|
||||||
|
|
||||||
[project.optional-dependencies]
|
[project.optional-dependencies]
|
||||||
dev = [
|
dev = [
|
||||||
|
|||||||
@@ -0,0 +1,110 @@
|
|||||||
|
"""Strict configuration for executable material-consumption instances."""
|
||||||
|
|
||||||
|
from dataclasses import dataclass, fields
|
||||||
|
from math import isfinite
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import yaml
|
||||||
|
|
||||||
|
|
||||||
|
class CalculationConfigError(ValueError):
|
||||||
|
"""An operator-readable configuration failure, without input values."""
|
||||||
|
|
||||||
|
|
||||||
|
class _UniqueLoader(yaml.SafeLoader):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def _mapping(loader, node):
|
||||||
|
result = {}
|
||||||
|
for key_node, value_node in node.value:
|
||||||
|
key = loader.construct_object(key_node)
|
||||||
|
if not isinstance(key, str) or key in result:
|
||||||
|
raise CalculationConfigError("YAML mapping keys must be unique strings")
|
||||||
|
result[key] = loader.construct_object(value_node)
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
_UniqueLoader.add_constructor(yaml.resolver.BaseResolver.DEFAULT_MAPPING_TAG, _mapping)
|
||||||
|
|
||||||
|
|
||||||
|
def finite_number(value: object, name: str, *, positive: bool = False) -> float:
|
||||||
|
if isinstance(value, bool) or not isinstance(value, (int, float)):
|
||||||
|
raise CalculationConfigError(f"{name} must be a finite number")
|
||||||
|
try:
|
||||||
|
number = float(value)
|
||||||
|
except (OverflowError, ValueError) as exc:
|
||||||
|
raise CalculationConfigError(f"{name} must be a finite number") from exc
|
||||||
|
if not isfinite(number) or (positive and number <= 0):
|
||||||
|
requirement = "finite and greater than zero" if positive else "finite"
|
||||||
|
raise CalculationConfigError(f"{name} must be {requirement}")
|
||||||
|
return number
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class MaterialCalculationConfig:
|
||||||
|
id: str
|
||||||
|
type: str
|
||||||
|
version: str
|
||||||
|
machine_ref: str
|
||||||
|
rate_signal_ref: str
|
||||||
|
gate_signal_ref: str
|
||||||
|
gate_threshold: float
|
||||||
|
max_sample_gap_seconds: float
|
||||||
|
output_metric: str
|
||||||
|
output_unit: str
|
||||||
|
group_by: str
|
||||||
|
|
||||||
|
|
||||||
|
def load_material_calculation(
|
||||||
|
path: str | Path, calculation_id: str | None = None,
|
||||||
|
) -> MaterialCalculationConfig:
|
||||||
|
try:
|
||||||
|
with Path(path).open(encoding="utf-8") as stream:
|
||||||
|
document = yaml.load(stream, Loader=_UniqueLoader)
|
||||||
|
except (OSError, UnicodeError) as exc:
|
||||||
|
raise CalculationConfigError("Cannot read calculation configuration file") from exc
|
||||||
|
except yaml.YAMLError as exc:
|
||||||
|
raise CalculationConfigError("Invalid YAML in calculation configuration") from exc
|
||||||
|
if not isinstance(document, dict) or set(document) != {"calculations"}:
|
||||||
|
raise CalculationConfigError("Configuration must contain only 'calculations'")
|
||||||
|
entries = document["calculations"]
|
||||||
|
if not isinstance(entries, list) or not entries:
|
||||||
|
raise CalculationConfigError("calculations must be a non-empty list")
|
||||||
|
required = {field.name for field in fields(MaterialCalculationConfig)}
|
||||||
|
instances = {}
|
||||||
|
for index, entry in enumerate(entries):
|
||||||
|
prefix = f"calculations[{index}]"
|
||||||
|
if not isinstance(entry, dict):
|
||||||
|
raise CalculationConfigError(f"{prefix} must be a mapping")
|
||||||
|
if "type" in entry and entry["type"] != "material_consumption":
|
||||||
|
raise CalculationConfigError(f"{prefix}.type: only 'material_consumption' is supported")
|
||||||
|
missing = required - entry.keys()
|
||||||
|
if missing:
|
||||||
|
raise CalculationConfigError(f"{prefix}: missing fields: {', '.join(sorted(missing))}")
|
||||||
|
if entry.keys() - required:
|
||||||
|
raise CalculationConfigError(f"{prefix}: unknown fields (check spelling)")
|
||||||
|
for name in sorted(required - {"gate_threshold", "max_sample_gap_seconds"}):
|
||||||
|
if not isinstance(entry[name], str) or not entry[name].strip():
|
||||||
|
raise CalculationConfigError(f"{prefix}.{name} must be a non-empty string")
|
||||||
|
for name, expected in (
|
||||||
|
("type", "material_consumption"), ("version", "1"),
|
||||||
|
("output_metric", "material_consumption"), ("output_unit", "kg"),
|
||||||
|
("group_by", "production_order"),
|
||||||
|
):
|
||||||
|
if entry[name] != expected:
|
||||||
|
raise CalculationConfigError(f"{prefix}.{name}: only {expected!r} is supported")
|
||||||
|
entry["gate_threshold"] = finite_number(entry["gate_threshold"], f"{prefix}.gate_threshold")
|
||||||
|
entry["max_sample_gap_seconds"] = finite_number(
|
||||||
|
entry["max_sample_gap_seconds"], f"{prefix}.max_sample_gap_seconds", positive=True,
|
||||||
|
)
|
||||||
|
if entry["id"] in instances:
|
||||||
|
raise CalculationConfigError("Calculation ids must be unique")
|
||||||
|
instances[entry["id"]] = MaterialCalculationConfig(**entry)
|
||||||
|
if calculation_id is not None:
|
||||||
|
if calculation_id not in instances:
|
||||||
|
raise CalculationConfigError("Requested calculation id was not found")
|
||||||
|
return instances[calculation_id]
|
||||||
|
if len(instances) != 1:
|
||||||
|
raise CalculationConfigError("Multiple calculations: specify --calculation-id")
|
||||||
|
return next(iter(instances.values()))
|
||||||
@@ -1,4 +1,4 @@
|
|||||||
"""Command-line entry point for safe, read-only API exploration."""
|
"""Command-line entry point for exploration and production polling."""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
@@ -70,6 +70,14 @@ def _parser() -> argparse.ArgumentParser:
|
|||||||
default=Path("secrets/enlyze.env"),
|
default=Path("secrets/enlyze.env"),
|
||||||
help="Local dotenv-style secret file (default: secrets/enlyze.env)",
|
help="Local dotenv-style secret file (default: secrets/enlyze.env)",
|
||||||
)
|
)
|
||||||
|
run = namespaces.add_parser("run", help="Foreground production runners")
|
||||||
|
runners = run.add_subparsers(dest="command", required=True)
|
||||||
|
material = runners.add_parser("material-poll", help="Continuously poll material consumption")
|
||||||
|
material.add_argument("--config", type=Path, required=True)
|
||||||
|
material.add_argument("--calculation-id")
|
||||||
|
material.add_argument("--poll-interval-seconds", type=float, default=10.0)
|
||||||
|
material.add_argument("--state-directory", type=Path, default=Path("data/state/material"))
|
||||||
|
material.add_argument("--secrets-file", type=Path, default=Path("secrets/enlyze.env"))
|
||||||
return parser
|
return parser
|
||||||
|
|
||||||
|
|
||||||
@@ -83,8 +91,36 @@ def _write_fixture(directory: Path, name: str, payload: object) -> Path:
|
|||||||
return destination
|
return destination
|
||||||
|
|
||||||
|
|
||||||
|
def _run_material(args: argparse.Namespace) -> int:
|
||||||
|
from production_analytics.calculations.config import (
|
||||||
|
CalculationConfigError,
|
||||||
|
load_material_calculation,
|
||||||
|
)
|
||||||
|
from production_analytics.enlyze.exploration import ConfigurationError
|
||||||
|
from production_analytics.service.material_runtime import build_material_runner
|
||||||
|
|
||||||
|
try:
|
||||||
|
calculation = load_material_calculation(args.config, args.calculation_id)
|
||||||
|
runner = build_material_runner(
|
||||||
|
calculation, poll_interval_seconds=args.poll_interval_seconds,
|
||||||
|
state_directory=args.state_directory, secrets_file=args.secrets_file,
|
||||||
|
)
|
||||||
|
runner.run()
|
||||||
|
except (CalculationConfigError, ConfigurationError) as exc:
|
||||||
|
print(f"Config/startup error: {exc}", file=sys.stderr)
|
||||||
|
return 2
|
||||||
|
except Exception as exc:
|
||||||
|
print(f"Startup/runtime error: {type(exc).__name__}", file=sys.stderr)
|
||||||
|
return 1
|
||||||
|
except KeyboardInterrupt:
|
||||||
|
return 0
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
def main(argv: Sequence[str] | None = None) -> int:
|
def main(argv: Sequence[str] | None = None) -> int:
|
||||||
args = _parser().parse_args(argv)
|
args = _parser().parse_args(argv)
|
||||||
|
if args.namespace == "run":
|
||||||
|
return _run_material(args)
|
||||||
if args.namespace != "enlyze" or args.command not in {"raw", "timeseries"}:
|
if args.namespace != "enlyze" or args.command not in {"raw", "timeseries"}:
|
||||||
return 2
|
return 2
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,71 @@
|
|||||||
|
"""Sequential foreground polling with injectable time and output."""
|
||||||
|
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
from collections.abc import Callable
|
||||||
|
from datetime import UTC, datetime
|
||||||
|
from typing import TextIO
|
||||||
|
|
||||||
|
from production_analytics.calculations.config import finite_number
|
||||||
|
from production_analytics.enlyze.exploration import ConfigurationError, ExplorationError
|
||||||
|
from production_analytics.service.material_polling import MaterialPollingService
|
||||||
|
|
||||||
|
|
||||||
|
class MaterialStateError(RuntimeError):
|
||||||
|
"""State operation failed; message contains only the operation and error class."""
|
||||||
|
|
||||||
|
|
||||||
|
def utc_now() -> datetime:
|
||||||
|
return datetime.now(UTC)
|
||||||
|
|
||||||
|
|
||||||
|
class MaterialPollingRunner:
|
||||||
|
def __init__(
|
||||||
|
self, service: MaterialPollingService, *, machine_id: str,
|
||||||
|
poll_interval_seconds: float, clock: Callable[[], datetime] = utc_now,
|
||||||
|
sleep: Callable[[float], None] = time.sleep,
|
||||||
|
stdout: TextIO | None = None, stderr: TextIO | None = None,
|
||||||
|
) -> None:
|
||||||
|
self.service = service
|
||||||
|
self.machine_id = machine_id
|
||||||
|
self.interval = finite_number(poll_interval_seconds, "poll interval", positive=True)
|
||||||
|
self.clock = clock
|
||||||
|
self.sleep = sleep
|
||||||
|
self.stdout = stdout if stdout is not None else sys.stdout
|
||||||
|
self.stderr = stderr if stderr is not None else sys.stderr
|
||||||
|
|
||||||
|
def run(self) -> None:
|
||||||
|
try:
|
||||||
|
while True:
|
||||||
|
now = self.clock()
|
||||||
|
if now.tzinfo is None or now.utcoffset() is None:
|
||||||
|
raise ConfigurationError("Runner clock must return timezone-aware timestamps")
|
||||||
|
now = now.astimezone(UTC)
|
||||||
|
prefix = f"{now.isoformat()} machine={self.machine_id!r}"
|
||||||
|
try:
|
||||||
|
result = self.service.poll_once(now=now)
|
||||||
|
except ConfigurationError:
|
||||||
|
raise
|
||||||
|
except Exception as exc:
|
||||||
|
if isinstance(exc, MaterialStateError):
|
||||||
|
detail = str(exc)
|
||||||
|
elif isinstance(exc, ExplorationError):
|
||||||
|
detail = f"ENLYZE/gateway: {type(exc).__name__}"
|
||||||
|
else:
|
||||||
|
detail = f"gateway/poll cycle: {type(exc).__name__}"
|
||||||
|
# Do not echo arbitrary exception messages or response bodies.
|
||||||
|
print(f"{prefix} error {detail}", file=self.stderr, flush=True)
|
||||||
|
else:
|
||||||
|
if result is None:
|
||||||
|
detail = "no open Production Run / no eligible polling window"
|
||||||
|
else:
|
||||||
|
detail = (
|
||||||
|
f"production_order={result.run.production_order!r} "
|
||||||
|
f"consumption_kg={result.state.cumulative_consumption_kg:.9f} "
|
||||||
|
f"running_seconds={result.state.integrated_running_seconds:.3f}"
|
||||||
|
)
|
||||||
|
print(f"{prefix} {detail}", file=self.stdout, flush=True)
|
||||||
|
# Fixed delay after completion, including errors; never catch up or overlap.
|
||||||
|
self.sleep(self.interval)
|
||||||
|
except KeyboardInterrupt:
|
||||||
|
print("Material polling stopped", file=self.stdout, flush=True)
|
||||||
@@ -0,0 +1,78 @@
|
|||||||
|
"""Compose the configured live material polling application."""
|
||||||
|
|
||||||
|
import math
|
||||||
|
import os
|
||||||
|
import tempfile
|
||||||
|
from pathlib import Path
|
||||||
|
from urllib.parse import urlsplit
|
||||||
|
|
||||||
|
from production_analytics.calculations.config import MaterialCalculationConfig, finite_number
|
||||||
|
from production_analytics.enlyze.exploration import (
|
||||||
|
ConfigurationError,
|
||||||
|
ExplorationClient,
|
||||||
|
ExplorationSettings,
|
||||||
|
load_secret_file,
|
||||||
|
)
|
||||||
|
from production_analytics.enlyze.gateway import EnlyzeApiGateway
|
||||||
|
from production_analytics.service.material_polling import (
|
||||||
|
MaterialPollingService,
|
||||||
|
MaterialPollingState,
|
||||||
|
)
|
||||||
|
from production_analytics.service.material_runner import MaterialPollingRunner, MaterialStateError
|
||||||
|
from production_analytics.service.material_state_store import JsonMaterialStateStore
|
||||||
|
|
||||||
|
|
||||||
|
class _ReportingStateStore:
|
||||||
|
def __init__(self, store: JsonMaterialStateStore) -> None:
|
||||||
|
self.store = store
|
||||||
|
|
||||||
|
def load(self, machine_id: str, production_order: str) -> MaterialPollingState | None:
|
||||||
|
try:
|
||||||
|
return self.store.load(machine_id, production_order)
|
||||||
|
except Exception as exc:
|
||||||
|
raise MaterialStateError(f"state loading: {type(exc).__name__}") from exc
|
||||||
|
|
||||||
|
def save(
|
||||||
|
self, machine_id: str, production_order: str, state: MaterialPollingState,
|
||||||
|
) -> None:
|
||||||
|
try:
|
||||||
|
self.store.save(machine_id, production_order, state)
|
||||||
|
except Exception as exc:
|
||||||
|
raise MaterialStateError(f"state saving: {type(exc).__name__}") from exc
|
||||||
|
|
||||||
|
|
||||||
|
def build_material_runner(
|
||||||
|
calculation: MaterialCalculationConfig, *, poll_interval_seconds: float,
|
||||||
|
state_directory: Path, secrets_file: Path,
|
||||||
|
) -> MaterialPollingRunner:
|
||||||
|
finite_number(poll_interval_seconds, "poll interval", positive=True)
|
||||||
|
environment = dict(os.environ)
|
||||||
|
environment.update(load_secret_file(secrets_file))
|
||||||
|
settings = ExplorationSettings.from_environment(environment)
|
||||||
|
parsed = urlsplit(settings.base_url)
|
||||||
|
if parsed.scheme not in {"http", "https"} or not parsed.hostname or parsed.username:
|
||||||
|
raise ConfigurationError(
|
||||||
|
"ENLYZE_BASE_URL must be an HTTP(S) server URL without credentials"
|
||||||
|
)
|
||||||
|
if not math.isfinite(settings.timeout_seconds):
|
||||||
|
raise ConfigurationError("ENLYZE_HTTP_TIMEOUT_SECONDS must be finite")
|
||||||
|
try:
|
||||||
|
state_directory.mkdir(parents=True, exist_ok=True)
|
||||||
|
with tempfile.TemporaryFile(dir=state_directory):
|
||||||
|
pass
|
||||||
|
except OSError as exc:
|
||||||
|
raise ConfigurationError(
|
||||||
|
f"State directory startup check failed: {type(exc).__name__}"
|
||||||
|
) from exc
|
||||||
|
service = MaterialPollingService(
|
||||||
|
gateway=EnlyzeApiGateway(ExplorationClient(settings)),
|
||||||
|
state_store=_ReportingStateStore(JsonMaterialStateStore(state_directory)),
|
||||||
|
machine_id=calculation.machine_ref,
|
||||||
|
rate_variable_id=calculation.rate_signal_ref,
|
||||||
|
gate_variable_id=calculation.gate_signal_ref,
|
||||||
|
gate_threshold=calculation.gate_threshold,
|
||||||
|
max_sample_gap_seconds=calculation.max_sample_gap_seconds,
|
||||||
|
)
|
||||||
|
return MaterialPollingRunner(
|
||||||
|
service, machine_id=calculation.machine_ref, poll_interval_seconds=poll_interval_seconds,
|
||||||
|
)
|
||||||
@@ -0,0 +1,277 @@
|
|||||||
|
import io
|
||||||
|
from datetime import UTC, datetime, timedelta
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest.mock import Mock, patch
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
import yaml
|
||||||
|
|
||||||
|
from production_analytics.calculations.config import (
|
||||||
|
CalculationConfigError,
|
||||||
|
load_material_calculation,
|
||||||
|
)
|
||||||
|
from production_analytics.calculations.material_consumption import MaterialIntegrationState
|
||||||
|
from production_analytics.cli.__main__ import main
|
||||||
|
from production_analytics.enlyze.exploration import ConfigurationError
|
||||||
|
from production_analytics.enlyze.gateway import EnlyzeProductionRun
|
||||||
|
from production_analytics.service.material_polling import MaterialPollResult
|
||||||
|
from production_analytics.service.material_runner import MaterialPollingRunner, MaterialStateError
|
||||||
|
from production_analytics.service.material_runtime import (
|
||||||
|
_ReportingStateStore,
|
||||||
|
build_material_runner,
|
||||||
|
)
|
||||||
|
|
||||||
|
EXAMPLE = Path(__file__).resolve().parents[1] / 'config/k7-material-consumption.yaml'
|
||||||
|
NOW = datetime(2026, 1, 1, tzinfo=UTC)
|
||||||
|
|
||||||
|
|
||||||
|
def write_config(tmp_path, entries):
|
||||||
|
path = tmp_path / 'calculations.yaml'
|
||||||
|
path.write_text(yaml.safe_dump({'calculations': entries}))
|
||||||
|
return path
|
||||||
|
|
||||||
|
|
||||||
|
def entry():
|
||||||
|
return yaml.safe_load(EXAMPLE.read_text())['calculations'][0]
|
||||||
|
|
||||||
|
|
||||||
|
def test_valid_k7_config():
|
||||||
|
config = load_material_calculation(EXAMPLE)
|
||||||
|
assert config.id == 'k7-fiber-consumption'
|
||||||
|
assert config.machine_ref == 'c220f95c-a65e-4cb7-99b7-0626d6c7508c'
|
||||||
|
assert config.rate_signal_ref == 'c9d06af5-f6d6-4ede-b6c4-5a98bac77129'
|
||||||
|
assert config.gate_signal_ref == '823867bb-f5d2-40eb-b875-657155addfd0'
|
||||||
|
assert config.gate_threshold == 0.5
|
||||||
|
assert config.max_sample_gap_seconds == 20.0
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize('field', list(entry()))
|
||||||
|
def test_missing_fields(tmp_path, field):
|
||||||
|
data = entry()
|
||||||
|
del data[field]
|
||||||
|
with pytest.raises(CalculationConfigError, match=f'missing fields: {field}'):
|
||||||
|
load_material_calculation(write_config(tmp_path, [data]))
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(('field', 'value'), [
|
||||||
|
('type', 'integration'), ('version', 1), ('version', '2'),
|
||||||
|
('machine_ref', ''), ('rate_signal_ref', ' '), ('gate_signal_ref', None), ('id', ''),
|
||||||
|
('gate_threshold', True), ('gate_threshold', '0.5'), ('gate_threshold', float('nan')),
|
||||||
|
('max_sample_gap_seconds', float('inf')), ('max_sample_gap_seconds', 0),
|
||||||
|
('max_sample_gap_seconds', -1), ('output_unit', 'tonnes'), ('group_by', 'run'),
|
||||||
|
('gate_treshold', 1),
|
||||||
|
])
|
||||||
|
def test_invalid_fields(tmp_path, field, value):
|
||||||
|
data = entry()
|
||||||
|
data[field] = value
|
||||||
|
with pytest.raises(CalculationConfigError):
|
||||||
|
load_material_calculation(write_config(tmp_path, [data]))
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize('content', [
|
||||||
|
'calculations: [', 'calculations: []', 'calculations: {}', 'other: []',
|
||||||
|
'calculations: []\ncalculations: []', 'calculations:\n - id: x\n id: y',
|
||||||
|
])
|
||||||
|
def test_malformed_documents(tmp_path, content):
|
||||||
|
path = tmp_path / 'bad.yaml'
|
||||||
|
path.write_text(content)
|
||||||
|
with pytest.raises(CalculationConfigError):
|
||||||
|
load_material_calculation(path)
|
||||||
|
|
||||||
|
|
||||||
|
def test_selection(tmp_path):
|
||||||
|
first, second = entry(), dict(entry(), id='second')
|
||||||
|
path = write_config(tmp_path, [first, second])
|
||||||
|
with pytest.raises(CalculationConfigError, match='specify --calculation-id'):
|
||||||
|
load_material_calculation(path)
|
||||||
|
assert load_material_calculation(path, 'second').id == 'second'
|
||||||
|
with pytest.raises(CalculationConfigError, match='not found'):
|
||||||
|
load_material_calculation(path, 'absent')
|
||||||
|
with pytest.raises(CalculationConfigError, match='unique'):
|
||||||
|
load_material_calculation(write_config(tmp_path, [first, first]), first['id'])
|
||||||
|
|
||||||
|
|
||||||
|
def test_repeated_sequential_polls_and_long_cycle():
|
||||||
|
events = []
|
||||||
|
elapsed = 0
|
||||||
|
|
||||||
|
def poll(*, now):
|
||||||
|
nonlocal elapsed
|
||||||
|
assert now == NOW + timedelta(seconds=elapsed)
|
||||||
|
events.append('poll-start')
|
||||||
|
elapsed += 25 # Longer than the interval; no real sleep or concurrent work.
|
||||||
|
events.append('poll-end')
|
||||||
|
|
||||||
|
def sleep(seconds):
|
||||||
|
nonlocal elapsed
|
||||||
|
events.append(('sleep', seconds))
|
||||||
|
elapsed += seconds
|
||||||
|
if elapsed >= 70:
|
||||||
|
raise KeyboardInterrupt
|
||||||
|
|
||||||
|
output = io.StringIO()
|
||||||
|
service = Mock(poll_once=Mock(side_effect=poll))
|
||||||
|
MaterialPollingRunner(
|
||||||
|
service, machine_id='machine', poll_interval_seconds=10,
|
||||||
|
clock=lambda: NOW + timedelta(seconds=elapsed), sleep=sleep, stdout=output,
|
||||||
|
).run()
|
||||||
|
assert events == ['poll-start', 'poll-end', ('sleep', 10)] * 2
|
||||||
|
assert service.poll_once.call_count == 2
|
||||||
|
assert output.getvalue().count('no open Production Run') == 2
|
||||||
|
assert 'stopped' in output.getvalue()
|
||||||
|
|
||||||
|
|
||||||
|
def test_error_then_success_preserves_opaque_order_and_reports_totals():
|
||||||
|
run = EnlyzeProductionRun('run', 'machine', None, ' 00842/ABC ', NOW, None)
|
||||||
|
result = MaterialPollResult(run, MaterialIntegrationState(12.5, 20))
|
||||||
|
service = Mock(poll_once=Mock(side_effect=[ValueError('SECRET'), result]))
|
||||||
|
output, errors = io.StringIO(), io.StringIO()
|
||||||
|
sleep = Mock(side_effect=[None, KeyboardInterrupt])
|
||||||
|
MaterialPollingRunner(
|
||||||
|
service, machine_id='machine', poll_interval_seconds=5, clock=lambda: NOW,
|
||||||
|
sleep=sleep, stdout=output, stderr=errors,
|
||||||
|
).run()
|
||||||
|
assert service.poll_once.call_count == 2
|
||||||
|
assert sleep.call_count == 2
|
||||||
|
assert 'gateway/poll cycle: ValueError' in errors.getvalue()
|
||||||
|
assert 'SECRET' not in errors.getvalue()
|
||||||
|
assert "production_order=' 00842/ABC '" in output.getvalue()
|
||||||
|
assert 'consumption_kg=12.500000000 running_seconds=20.000' in output.getvalue()
|
||||||
|
assert NOW.isoformat() in output.getvalue()
|
||||||
|
|
||||||
|
|
||||||
|
def test_interrupt_during_poll():
|
||||||
|
sleep, output = Mock(), io.StringIO()
|
||||||
|
MaterialPollingRunner(
|
||||||
|
Mock(poll_once=Mock(side_effect=KeyboardInterrupt)), machine_id='m',
|
||||||
|
poll_interval_seconds=1, sleep=sleep, stdout=output,
|
||||||
|
).run()
|
||||||
|
sleep.assert_not_called()
|
||||||
|
assert 'stopped' in output.getvalue()
|
||||||
|
|
||||||
|
|
||||||
|
def test_cycle_configuration_failure_is_fatal():
|
||||||
|
sleep = Mock()
|
||||||
|
runner = MaterialPollingRunner(
|
||||||
|
Mock(poll_once=Mock(side_effect=ConfigurationError('bad settings'))),
|
||||||
|
machine_id='m', poll_interval_seconds=1, sleep=sleep,
|
||||||
|
)
|
||||||
|
with pytest.raises(ConfigurationError):
|
||||||
|
runner.run()
|
||||||
|
sleep.assert_not_called()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize('interval', [0, -1, float('nan'), float('inf'), True])
|
||||||
|
def test_invalid_interval(interval):
|
||||||
|
with pytest.raises(CalculationConfigError):
|
||||||
|
MaterialPollingRunner(Mock(), machine_id='m', poll_interval_seconds=interval)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize('operation', ['load', 'save'])
|
||||||
|
def test_state_error_classification(operation):
|
||||||
|
store = Mock()
|
||||||
|
getattr(store, operation).side_effect = OSError('SECRET')
|
||||||
|
wrapped = _ReportingStateStore(store)
|
||||||
|
arguments = ('machine', ' 00842 ') if operation == 'load' else ('machine', ' 00842 ', Mock())
|
||||||
|
phase = 'loading' if operation == 'load' else 'saving'
|
||||||
|
with pytest.raises(MaterialStateError, match=f'state {phase}: OSError') as error:
|
||||||
|
getattr(wrapped, operation)(*arguments)
|
||||||
|
assert 'SECRET' not in str(error.value)
|
||||||
|
|
||||||
|
|
||||||
|
def test_runtime_wiring(tmp_path, monkeypatch):
|
||||||
|
monkeypatch.setenv('ENLYZE_BASE_URL', 'https://example.invalid/api/')
|
||||||
|
secrets = tmp_path / 'secret.env'
|
||||||
|
secrets.write_text('ENLYZE_API_KEY=secret-from-file\n')
|
||||||
|
with patch('production_analytics.service.material_runtime.MaterialPollingService') as service:
|
||||||
|
runner = build_material_runner(
|
||||||
|
load_material_calculation(EXAMPLE), poll_interval_seconds=7,
|
||||||
|
state_directory=tmp_path / 'state', secrets_file=secrets,
|
||||||
|
)
|
||||||
|
kwargs = service.call_args.kwargs
|
||||||
|
assert kwargs['machine_id'] == entry()['machine_ref']
|
||||||
|
assert kwargs['rate_variable_id'] == entry()['rate_signal_ref']
|
||||||
|
assert kwargs['gate_variable_id'] == entry()['gate_signal_ref']
|
||||||
|
assert kwargs['gate_threshold'] == 0.5
|
||||||
|
assert kwargs['max_sample_gap_seconds'] == 20.0
|
||||||
|
assert kwargs['gateway']._client._settings.api_key == 'secret-from-file'
|
||||||
|
assert kwargs['state_store'].store._directory == tmp_path / 'state'
|
||||||
|
assert runner.service is service.return_value
|
||||||
|
assert runner.interval == 7
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_bad_config(tmp_path, capsys):
|
||||||
|
assert main(['run', 'material-poll', '--config', str(tmp_path / 'missing')]) == 2
|
||||||
|
assert 'Config/startup' in capsys.readouterr().err
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_startup_and_success(tmp_path, capsys):
|
||||||
|
args = ['run', 'material-poll', '--config', str(EXAMPLE),
|
||||||
|
'--calculation-id', 'k7-fiber-consumption']
|
||||||
|
with patch('production_analytics.service.material_runtime.build_material_runner') as build:
|
||||||
|
assert main(args) == 0
|
||||||
|
build.return_value.run.assert_called_once_with()
|
||||||
|
assert build.call_args.kwargs['state_directory'] == Path('data/state/material')
|
||||||
|
build.side_effect = ConfigurationError('bad settings')
|
||||||
|
assert main(args) == 2
|
||||||
|
assert 'bad settings' in capsys.readouterr().err
|
||||||
|
|
||||||
|
|
||||||
|
def test_failed_cycle_keeps_checkpoint_and_recovers(tmp_path):
|
||||||
|
from production_analytics.calculations.material_consumption import MaterialSample
|
||||||
|
from production_analytics.service.material_polling import (
|
||||||
|
MaterialPollingService,
|
||||||
|
MaterialPollingState,
|
||||||
|
)
|
||||||
|
from production_analytics.service.material_state_store import JsonMaterialStateStore
|
||||||
|
|
||||||
|
order = ' 00842 '
|
||||||
|
store = JsonMaterialStateStore(tmp_path)
|
||||||
|
store.save('machine', order, MaterialPollingState('run', MaterialIntegrationState(
|
||||||
|
cumulative_consumption_kg=1, integrated_running_seconds=10,
|
||||||
|
last_processed_timestamp=NOW, last_material_rate_kg_per_hour=3600,
|
||||||
|
last_gate_value=1, integration_active=True,
|
||||||
|
)))
|
||||||
|
checkpoint = next(tmp_path.glob('*.json'))
|
||||||
|
original = checkpoint.read_bytes()
|
||||||
|
gateway = Mock()
|
||||||
|
gateway.get_open_production_run.return_value = EnlyzeProductionRun(
|
||||||
|
'run', 'machine', None, order, NOW, None,
|
||||||
|
)
|
||||||
|
gateway.get_material_samples.side_effect = [
|
||||||
|
ValueError('bad response'), [MaterialSample(NOW + timedelta(seconds=10), 3600, 1)],
|
||||||
|
]
|
||||||
|
service = MaterialPollingService(
|
||||||
|
gateway=gateway, state_store=_ReportingStateStore(store), machine_id='machine',
|
||||||
|
rate_variable_id='rate', gate_variable_id='gate', gate_threshold=.5,
|
||||||
|
max_sample_gap_seconds=20,
|
||||||
|
)
|
||||||
|
sleeps = 0
|
||||||
|
|
||||||
|
def sleep(seconds):
|
||||||
|
nonlocal sleeps
|
||||||
|
sleeps += 1
|
||||||
|
if sleeps == 1:
|
||||||
|
assert checkpoint.read_bytes() == original
|
||||||
|
else:
|
||||||
|
raise KeyboardInterrupt
|
||||||
|
|
||||||
|
MaterialPollingRunner(
|
||||||
|
service, machine_id='machine', poll_interval_seconds=1,
|
||||||
|
clock=lambda: NOW + timedelta(seconds=10), sleep=sleep,
|
||||||
|
stdout=io.StringIO(), stderr=io.StringIO(),
|
||||||
|
).run()
|
||||||
|
restored = store.load('machine', order)
|
||||||
|
assert restored.integration_state.cumulative_consumption_kg == 11
|
||||||
|
assert restored.integration_state.integrated_running_seconds == 20
|
||||||
|
|
||||||
|
|
||||||
|
def test_unwritable_state_location_is_startup_failure(tmp_path, monkeypatch, capsys):
|
||||||
|
monkeypatch.setenv('ENLYZE_BASE_URL', 'https://example.invalid/api/')
|
||||||
|
blocked = tmp_path / 'file'
|
||||||
|
blocked.write_text('not a directory')
|
||||||
|
assert main([
|
||||||
|
'run', 'material-poll', '--config', str(EXAMPLE),
|
||||||
|
'--state-directory', str(blocked), '--secrets-file', str(tmp_path / 'absent'),
|
||||||
|
]) == 2
|
||||||
|
assert 'State directory startup check failed' in capsys.readouterr().err
|
||||||
Reference in New Issue
Block a user