From 231e18082c96202e7bcd42d78635771a60e71731 Mon Sep 17 00:00:00 2001 From: Martin Tazl Date: Fri, 4 Sep 2026 16:43:24 +0200 Subject: [PATCH] Generalize live material consumption architecture --- PROJECT_KNOWLEDGE.md | 93 ++++++++++++++++++++++++++++++++++++++------ docs/architecture.md | 63 ++++++++++++++++++++++++++++-- docs/enlyze-api.md | 39 +++++++++++++++++-- docs/roadmap.md | 46 +++++++++++++++------- 4 files changed, 209 insertions(+), 32 deletions(-) diff --git a/PROJECT_KNOWLEDGE.md b/PROJECT_KNOWLEDGE.md index c887500..9671182 100644 --- a/PROJECT_KNOWLEDGE.md +++ b/PROJECT_KNOWLEDGE.md @@ -3,8 +3,12 @@ ## Purpose This service is a calculation and persistence layer between ENLYZE and -Grafana. It must not become a second historian: raw process data stays in -ENLYZE and is retrieved again when historical calculations need reproduction. +Grafana. Its primary use case is to continuously integrate material consumption +for the currently active production order and expose its cumulative value in +Grafana while production is running. K7 fiber is the first validated +application, not the scope of the product. It must not become a second +historian: raw process data stays in ENLYZE and is retrieved again when +historical calculations need reproduction. ## Core architectural decisions @@ -16,6 +20,17 @@ ENLYZE and is retrieved again when historical calculations need reproduction. configuration are committed. - Calculations are modular, testable Python implementations. Configuration declares instances; it is not a generic low-code language. +- Historical production-order analysis/backfill is secondary. It uses the same + generic integration engine as live processing: one integration engine with + live incremental and historical replay/backfill operating modes. + +## Material-consumption integration + +The core calculation is generic rate integration: cumulative material +consumption increases by `material_rate * elapsed_time` when the configured +machine/process-specific validity or gating conditions are satisfied. It is +reusable across machines, materials, rate variables, units, and gates. The +project deliberately does not define a broader configuration schema here. ## Peak-cycle detector @@ -60,18 +75,72 @@ fixtures only. ## Central domain context Production orders connect machine, article/material, source interval, and -derived results. Metrics and events must retain calculation type/version and -source-time-range provenance. +derived results. An ENLYZE Production Run is a time segment; one +`production_order` can have one or more Production Runs. Integrate each run +separately, then aggregate derived values on the production-order level. Never +implicitly bridge gaps between runs. Metrics and events must retain calculation +type/version and source-time-range provenance. + +## K7 fiber integration + +The current K7 candidate fiber mass-flow signal is `Stundenleistung Anlage` +in kg/h. Observed behaviour supports treating it as a process-responsive +band-scale signal, but it can remain frozen at a non-zero value during a +machine stop. It must therefore be integrated only while the validated gate +`Geschwindigkeit Gesamtanlage > 0.5 m/min` is active. The ENLYZE boolean +`Anlage läuft` is not the primary integration gate. + +The process-derived result is `fiber_feed_kg` for a Production Run. It is not +automatically a finished-product mass or a per-order material yield. Material +measured at the band scale may still be in the machine at an order transition, +and ERP feedback can arrive asynchronously or periodically. Differences from +reported finished-product quantities must not automatically be labelled scrap; +an exact per-order yield needs WIP/transport-delay accounting. + +## Bento 1 bentonite-powder integration + +Bento 1 bentonite-powder consumption is the next planned application of the +same generic integration mechanism. Its ENLYZE source variable and its +validity/gating logic have not yet been selected or validated; both must be +determined from Bento 1 process data before implementation. Bento 1 does not +need a separate subsystem or calculation formula. + +## Quantity-total interpretation + +For this customer setup, `production_run.quantity_total` is populated from +ERP/BI production feedback received by ENLYZE, rather than necessarily being +calculated from machine signals. The tested source CSV field is +`PCO_FEEDB_QUANTITY`. For inspected machines/articles it represents reported +finished-product area in m² (for example, 5.00 m × 50 m = 250.00 m²; 4.75 m × +50 m = 237.50 m²; 5.80 m × 50 m = 290.00 m²). The unit displayed by ENLYZE is +separately configured and must not be considered authoritative until its +configuration is validated. ## Known versus unknown -Known from the ENLYZE UI: machine identity, operational/downtime state, -current production order, article/material number, and process signals exist. -It is **not yet verified** how, or whether, each is exposed by the ENLYZE API. -Do not infer endpoint paths, authentication mechanisms, identifiers, paging, -timestamp semantics, or signal payloads. The active open-question list is in +Validated K7 historical tests show physically plausible fiber-feed totals when +the band-scale kg/h signal is speed-gated. For production order +`K 7-12026000842`, its run from 2026-08-31T10:57:54Z to +2026-08-31T18:23:11Z had ERP `quantity_total` of 12,420 m², approximately +5.206 active hours, approximately 5,180.1 kg integrated fiber feed, +approximately 2,174.2 m integrated machine length, and approximately +995.1 kg/h average active throughput. Article external ID `213408` has width +6.0 m and static mean basis weight 390.16 g/m²; the corresponding approximate +reported finished-product mass is 4,845.8 kg. This is a plausibility validation +case, not a precise material-yield calculation. + +Known from the ENLYZE UI/API exploration: machine and product identity, +Production Runs, and relevant K7 process signals are available. Some API and +time-series semantics remain unverified; do not infer them beyond documented +evidence. The active evidence and open-question list is in [docs/enlyze-api.md](docs/enlyze-api.md). +## Future refinement + +An optional future historical mass-balance refinement is to obtain actual +QA/QS basis-weight measurements from SQL instead of using static article basis +weight. It is not a blocker for the live material-consumption MVP. + ## Exploration workflow A deliberately generic, read-only CLI is available as @@ -85,5 +154,7 @@ authentication header; its value is never printed. Unmodified captures are local in ignored `data/raw/enlyze/`. Fixtures written to `fixtures/enlyze/` are sanitized, but must still be reviewed before commit. -The deliverable is verified API observations and sanitized fixtures, not -production calculations or database ingestion. +The exploration deliverable is verified API observations and sanitized +fixtures. The next product deliverable is live incremental material +integration, using a generic material-rate integrator rather than a separate +historical-only calculation path. diff --git a/docs/architecture.md b/docs/architecture.md index cc0f9f6..68cc4e9 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -7,6 +7,19 @@ only to calculate derived results and persists the results, their context, and the minimum state needed for incremental calculation. It does not mirror raw time series into PostgreSQL/TimescaleDB. +## Product direction + +The primary product path is live integration of material consumption for the +currently active production order, with a continuously updated cumulative value +for Grafana. The core mechanism is generic rate integration: cumulative +material consumption increases by `material_rate * elapsed_time` subject to +machine/process-specific validity or gating conditions. It is reusable for +different machines, materials, rate variables, units, and gates. Historical +production-order analysis and backfill are secondary. They must reuse this +generic integration engine rather than introduce separate formulas: one +integration engine, two operating modes (live incremental processing and +historical replay/backfill). + ## Components | Component | Responsibility | Must not do | @@ -18,6 +31,48 @@ time series into PostgreSQL/TimescaleDB. | `cli` | Operator commands; first use is API exploration | Contain business calculations | | `service` | Future FastAPI composition/API layer | Require endpoints during bootstrap | +## Production Run boundaries and aggregation + +ENLYZE Production Runs are time segments, not necessarily one-to-one with a +production order. A `production_order` may contain multiple runs. The +integrator processes every Production Run independently; production-order +totals aggregate the resulting derived values. It must not implicitly integrate +across a gap between runs. + +## Material-consumption integrator + +Live mode detects the active Production Run/order, ingests new ENLYZE samples, +incrementally integrates material consumption while its configured validity +conditions are active, and persists calculation state and the cumulative +result. Replay mode fetches a bounded completed run and applies that same +stateful operator from its initial state. Results retain the run boundary and +source interval; aggregation across runs is a distinct production-order-level +step. + +The generic engine does not prescribe a material, source variable, unit, or +gate. Those are specific to the supported machine/process use case. + +### K7 fiber (first validated implementation) + +For K7, the current validated calculation integrates `Stundenleistung Anlage` +(kg/h) only for intervals where `Geschwindigkeit Gesamtanlage > 0.5 m/min`. +The mass-flow signal can remain non-zero while the machine is stopped, so it +must not be integrated without this gate. `Anlage läuft` is not the primary +gate. + +`fiber_feed_kg` is an objective process-derived feed quantity, not by itself a +finished-product mass or yield. Band-scale material and asynchronously reported +ERP production can be shifted by WIP/transport delay, especially around order +transitions. Any exact per-order yield calculation therefore needs additional +WIP accounting. + +### Bento 1 bentonite powder (planned) + +Bento 1 is the next planned use case for the same material-consumption +integrator. Its bentonite-powder source variable and validity/gating conditions +are not yet selected or validated and must be determined from ENLYZE process +data before implementation. It does not require a separate subsystem. + ## Result provenance Persisted metrics/events need machine and production-order context where @@ -36,9 +91,11 @@ avoids a generic expression language. ## Incremental execution Future calculation state is stored per calculation instance and relevant -context partition. It enables safe continuation (for example an open peak -cycle), while the source interval recorded on results keeps a historical run -reproducible by fetching ENLYZE data again. +context partition. For live material-consumption integration it includes the +latest processed source position, any interval/carry state needed for correct +integration, and the cumulative run value. It enables safe continuation (for +example an open peak cycle), while the source interval recorded on results +keeps a historical run reproducible by fetching ENLYZE data again. ## Peak-cycle semantics diff --git a/docs/enlyze-api.md b/docs/enlyze-api.md index d528864..d8ec04d 100644 --- a/docs/enlyze-api.md +++ b/docs/enlyze-api.md @@ -73,18 +73,49 @@ Each requested variable contains required UUID and optional `resampling_method`: `Downtime` requires UUID, machine UUID, type (`THRESHOLD` or `NO_DATA`), timezone-aware ISO 8601 start, nullable end, and nullable comment/reason/update information. A null end denotes an ongoing downtime. It supports historical downtime retrieval but does not establish a complete machine-state timeline. -## VERIFIED WITH LIVE CUSTOMER API +## VERIFIED WITH CUSTOMER EXPLORATION -None. This execution environment has no ENLYZE DNS/network access, so no customer operation has been sent in this milestone. +The following are customer-specific observations, not general ENLYZE API +semantics: + +- K7 candidate fiber mass-flow variable: `Stundenleistung Anlage` (kg/h). + Historical behaviour supports interpreting it as a process-responsive + band-scale signal. It can remain frozen at a non-zero value during stops. +- Validated K7 integration gate: `Geschwindigkeit Gesamtanlage > 0.5 m/min`. + Do not use the boolean `Anlage läuft` as the primary gate for this + calculation. +- `production_run.quantity_total` is populated from ERP/BI production feedback + received by ENLYZE for this setup, not necessarily from machine signals. The + tested CSV source field is `PCO_FEEDB_QUANTITY`. For inspected + machines/articles it is reported finished-product area in m²: 5.00 m × 50 m + = 250.00 m², 4.75 m × 50 m = 237.50 m², and 5.80 m × 50 m = 290.00 m². + `quantity_total.unit` is separately configured and is not authoritative + without configuration validation. + +Historical validation case (plausibility only): production order +`K 7-12026000842`, Production Run 2026-08-31T10:57:54Z to +2026-08-31T18:23:11Z, had ERP `quantity_total` 12,420 m². Speed-gated +integration produced approximately 5.206 active hours, 5,180.1 kg fiber feed, +2,174.2 m machine length, and 995.1 kg/h average active throughput. Article +external ID `213408` (width 6.0 m; static mean basis weight 390.16 g/m²) gives +an approximate reported finished-product mass of 4,845.8 kg. This supports the +physical plausibility of the feed integration; it is not a precise material +yield calculation. ## STILL UNKNOWN -- Customer response values, permissions, authentication success/failure behavior, default page size, and rate/retention limits. +- Full customer response coverage, permissions, authentication + success/failure behavior, default page size, and rate/retention limits. - Whether production-run filters have the desired interval-overlap behavior; completed-run `start`/`end` are candidate integration boundaries but require live confirmation. -- Which customer variable is a consumption signal and whether its unit/scaling factor makes it usable for integration. +- Whether the validated K7 signal/gate generalize to other machines, articles, + production conditions, or future signal configuration. - Whether omitted `resampling_interval` yields native/raw resolution; its unit and exact temporal meaning of resampling methods. - Sample ordering, boundary inclusion, native sampling behavior, gaps, nulls, quality/bad samples, late arrivals, corrections, and backfills. - A complete historical machine operational-state resource beyond downtimes. +- Exact per-order material yield: band-scale material may remain in the machine + at an order transition and ERP feedback can arrive asynchronously or + periodically. WIP/transport-delay accounting is required before interpreting + feed-versus-finished-product differences as yield or scrap. ## Exact bounded manual exploration sequence diff --git a/docs/roadmap.md b/docs/roadmap.md index c254e40..1f544a6 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -6,27 +6,43 @@ Establish package boundaries, architecture documentation, example calculation configuration, basic domain/protocol models, tests, and an optional local TimescaleDB Compose design. -## 1. ENLYZE API exploration client/CLI (recommended exact next milestone) +## 1. ENLYZE API exploration client/CLI (completed baseline) A read-only, GET-only raw inspection CLI and fixture sanitizer are implemented. -The remaining work is live, authorized exploration of machines, signal catalog, -bounded samples, state history, production-order history, and article/material -context. Record sanitized fixtures and update `docs/enlyze-api.md` with -evidence. Do not implement ingestion or a database schema before this is -complete. +Exploration established a K7 candidate mass-flow signal and speed gate; the +evidence and remaining uncertainties are recorded in `docs/enlyze-api.md`. +Continue to record sanitized fixtures for newly verified semantics. -## 2. Persistence foundation +## 2. Live material-consumption MVP (next implementation milestone) + +Detect the currently active Production Run and production order, continuously +ingest new ENLYZE samples, maintain persistent incremental integration state, +and store/expose cumulative material consumption for Grafana. Implement this +as generic rate integration (`material_rate * elapsed_time`) subject to +configured machine/process-specific validity conditions, so it can serve more +than one machine/material pair. The first validated implementation is K7: +integrate `Stundenleistung Anlage` (kg/h) only while `Geschwindigkeit +Gesamtanlage > 0.5 m/min`; do not use `Anlage läuft` as the primary gate. + +Bento 1 bentonite-powder consumption is the next analogous use case. Select +and validate its ENLYZE source variable and gating logic from process data +before implementing it; neither is defined yet. + +## 3. Persistence foundation Design and migrate a TimescaleDB schema for derived metrics/events, calculation -runs, context, and incremental state. Add Grafana-oriented query examples. +runs, Production Run/order context, and incremental state. Add Grafana-oriented +query examples. -## 3. First vertical slice +## 4. Historical replay/backfill -Implement one configured integration calculation over a verified signal and -production-order context. Include backfill, rerun/provenance behavior, and +Enable bounded historical Production Run replay and production-order analysis +using the same generic integration engine as live processing. Integrate each +Production Run separately and aggregate derived values at the production-order +level; never bridge gaps between runs. Include rerun/provenance behavior and tests against sanitized fixtures. -## 4. Peak detection (first detector completed) +## 5. Peak detection (first detector completed) The generic configurable sawtooth detector is implemented: it retains the current maximum, closes a cycle after a configurable below-peak fraction has @@ -38,5 +54,7 @@ non-zero resets, and varying peak heights without detector special cases. ## Later -State/cycle durations, aggregates by order, rolling statistics, threshold -events, derivatives, and specific-consumption calculations. +State/cycle durations, rolling statistics, threshold events, derivatives, and +specific-consumption calculations. An optional historical mass-balance +refinement is actual QA/QS basis-weight data from SQL, replacing static article +basis weight; it is not required for the live material-consumption MVP.