Generalize live material consumption architecture

PROJECT_KNOWLEDGE.md

@@ -3,8 +3,12 @@
 ## Purpose
 
 This service is a calculation and persistence layer between ENLYZE and
-Grafana. It must not become a second historian: raw process data stays in
+Grafana. Its primary use case is to continuously integrate material consumption
-ENLYZE and is retrieved again when historical calculations need reproduction.
+for the currently active production order and expose its cumulative value in
+Grafana while production is running. K7 fiber is the first validated
+application, not the scope of the product. It must not become a second
+historian: raw process data stays in ENLYZE and is retrieved again when
+historical calculations need reproduction.
 
 ## Core architectural decisions
 
@@ -16,6 +20,17 @@ ENLYZE and is retrieved again when historical calculations need reproduction.
   configuration are committed.
 - Calculations are modular, testable Python implementations. Configuration
   declares instances; it is not a generic low-code language.
+- Historical production-order analysis/backfill is secondary. It uses the same
+  generic integration engine as live processing: one integration engine with
+  live incremental and historical replay/backfill operating modes.
+
+## Material-consumption integration
+
+The core calculation is generic rate integration: cumulative material
+consumption increases by `material_rate * elapsed_time` when the configured
+machine/process-specific validity or gating conditions are satisfied. It is
+reusable across machines, materials, rate variables, units, and gates. The
+project deliberately does not define a broader configuration schema here.
 
 ## Peak-cycle detector
 
@@ -60,18 +75,72 @@ fixtures only.
 ## Central domain context
 
 Production orders connect machine, article/material, source interval, and
-derived results. Metrics and events must retain calculation type/version and
+derived results. An ENLYZE Production Run is a time segment; one
-source-time-range provenance.
+`production_order` can have one or more Production Runs. Integrate each run
+separately, then aggregate derived values on the production-order level. Never
+implicitly bridge gaps between runs. Metrics and events must retain calculation
+type/version and source-time-range provenance.
+
+## K7 fiber integration
+
+The current K7 candidate fiber mass-flow signal is `Stundenleistung Anlage`
+in kg/h. Observed behaviour supports treating it as a process-responsive
+band-scale signal, but it can remain frozen at a non-zero value during a
+machine stop. It must therefore be integrated only while the validated gate
+`Geschwindigkeit Gesamtanlage > 0.5 m/min` is active. The ENLYZE boolean
+`Anlage läuft` is not the primary integration gate.
+
+The process-derived result is `fiber_feed_kg` for a Production Run. It is not
+automatically a finished-product mass or a per-order material yield. Material
+measured at the band scale may still be in the machine at an order transition,
+and ERP feedback can arrive asynchronously or periodically. Differences from
+reported finished-product quantities must not automatically be labelled scrap;
+an exact per-order yield needs WIP/transport-delay accounting.
+
+## Bento 1 bentonite-powder integration
+
+Bento 1 bentonite-powder consumption is the next planned application of the
+same generic integration mechanism. Its ENLYZE source variable and its
+validity/gating logic have not yet been selected or validated; both must be
+determined from Bento 1 process data before implementation. Bento 1 does not
+need a separate subsystem or calculation formula.
+
+## Quantity-total interpretation
+
+For this customer setup, `production_run.quantity_total` is populated from
+ERP/BI production feedback received by ENLYZE, rather than necessarily being
+calculated from machine signals. The tested source CSV field is
+`PCO_FEEDB_QUANTITY`. For inspected machines/articles it represents reported
+finished-product area in m² (for example, 5.00 m × 50 m = 250.00 m²; 4.75 m ×
+50 m = 237.50 m²; 5.80 m × 50 m = 290.00 m²). The unit displayed by ENLYZE is
+separately configured and must not be considered authoritative until its
+configuration is validated.
 
 ## Known versus unknown
 
-Known from the ENLYZE UI: machine identity, operational/downtime state,
+Validated K7 historical tests show physically plausible fiber-feed totals when
-current production order, article/material number, and process signals exist.
+the band-scale kg/h signal is speed-gated. For production order
-It is **not yet verified** how, or whether, each is exposed by the ENLYZE API.
+`K 7-12026000842`, its run from 2026-08-31T10:57:54Z to
-Do not infer endpoint paths, authentication mechanisms, identifiers, paging,
+2026-08-31T18:23:11Z had ERP `quantity_total` of 12,420 m², approximately
-timestamp semantics, or signal payloads. The active open-question list is in
+5.206 active hours, approximately 5,180.1 kg integrated fiber feed,
+approximately 2,174.2 m integrated machine length, and approximately
+995.1 kg/h average active throughput. Article external ID `213408` has width
+6.0 m and static mean basis weight 390.16 g/m²; the corresponding approximate
+reported finished-product mass is 4,845.8 kg. This is a plausibility validation
+case, not a precise material-yield calculation.
+
+Known from the ENLYZE UI/API exploration: machine and product identity,
+Production Runs, and relevant K7 process signals are available. Some API and
+time-series semantics remain unverified; do not infer them beyond documented
+evidence. The active evidence and open-question list is in
 [docs/enlyze-api.md](docs/enlyze-api.md).
 
+## Future refinement
+
+An optional future historical mass-balance refinement is to obtain actual
+QA/QS basis-weight measurements from SQL instead of using static article basis
+weight. It is not a blocker for the live material-consumption MVP.
+
 ## Exploration workflow
 
 A deliberately generic, read-only CLI is available as
@@ -85,5 +154,7 @@ authentication header; its value is never printed.
 
 Unmodified captures are local in ignored `data/raw/enlyze/`. Fixtures written
 to `fixtures/enlyze/` are sanitized, but must still be reviewed before commit.
-The deliverable is verified API observations and sanitized fixtures, not
+The exploration deliverable is verified API observations and sanitized
-production calculations or database ingestion.
+fixtures. The next product deliverable is live incremental material
+integration, using a generic material-rate integrator rather than a separate
+historical-only calculation path.

docs/architecture.md

@@ -7,6 +7,19 @@ only to calculate derived results and persists the results, their context, and
 the minimum state needed for incremental calculation. It does not mirror raw
 time series into PostgreSQL/TimescaleDB.
 
+## Product direction
+
+The primary product path is live integration of material consumption for the
+currently active production order, with a continuously updated cumulative value
+for Grafana. The core mechanism is generic rate integration: cumulative
+material consumption increases by `material_rate * elapsed_time` subject to
+machine/process-specific validity or gating conditions. It is reusable for
+different machines, materials, rate variables, units, and gates. Historical
+production-order analysis and backfill are secondary. They must reuse this
+generic integration engine rather than introduce separate formulas: one
+integration engine, two operating modes (live incremental processing and
+historical replay/backfill).
+
 ## Components
 
 | Component | Responsibility | Must not do |
@@ -18,6 +31,48 @@ time series into PostgreSQL/TimescaleDB.
 | `cli` | Operator commands; first use is API exploration | Contain business calculations |
 | `service` | Future FastAPI composition/API layer | Require endpoints during bootstrap |
 
+## Production Run boundaries and aggregation
+
+ENLYZE Production Runs are time segments, not necessarily one-to-one with a
+production order. A `production_order` may contain multiple runs. The
+integrator processes every Production Run independently; production-order
+totals aggregate the resulting derived values. It must not implicitly integrate
+across a gap between runs.
+
+## Material-consumption integrator
+
+Live mode detects the active Production Run/order, ingests new ENLYZE samples,
+incrementally integrates material consumption while its configured validity
+conditions are active, and persists calculation state and the cumulative
+result. Replay mode fetches a bounded completed run and applies that same
+stateful operator from its initial state. Results retain the run boundary and
+source interval; aggregation across runs is a distinct production-order-level
+step.
+
+The generic engine does not prescribe a material, source variable, unit, or
+gate. Those are specific to the supported machine/process use case.
+
+### K7 fiber (first validated implementation)
+
+For K7, the current validated calculation integrates `Stundenleistung Anlage`
+(kg/h) only for intervals where `Geschwindigkeit Gesamtanlage > 0.5 m/min`.
+The mass-flow signal can remain non-zero while the machine is stopped, so it
+must not be integrated without this gate. `Anlage läuft` is not the primary
+gate.
+
+`fiber_feed_kg` is an objective process-derived feed quantity, not by itself a
+finished-product mass or yield. Band-scale material and asynchronously reported
+ERP production can be shifted by WIP/transport delay, especially around order
+transitions. Any exact per-order yield calculation therefore needs additional
+WIP accounting.
+
+### Bento 1 bentonite powder (planned)
+
+Bento 1 is the next planned use case for the same material-consumption
+integrator. Its bentonite-powder source variable and validity/gating conditions
+are not yet selected or validated and must be determined from ENLYZE process
+data before implementation. It does not require a separate subsystem.
+
 ## Result provenance
 
 Persisted metrics/events need machine and production-order context where
@@ -36,9 +91,11 @@ avoids a generic expression language.
 ## Incremental execution
 
 Future calculation state is stored per calculation instance and relevant
-context partition. It enables safe continuation (for example an open peak
+context partition. For live material-consumption integration it includes the
-cycle), while the source interval recorded on results keeps a historical run
+latest processed source position, any interval/carry state needed for correct
-reproducible by fetching ENLYZE data again.
+integration, and the cumulative run value. It enables safe continuation (for
+example an open peak cycle), while the source interval recorded on results
+keeps a historical run reproducible by fetching ENLYZE data again.
 
 ## Peak-cycle semantics
 

docs/enlyze-api.md

@@ -73,18 +73,49 @@ Each requested variable contains required UUID and optional `resampling_method`:
 
 `Downtime` requires UUID, machine UUID, type (`THRESHOLD` or `NO_DATA`), timezone-aware ISO 8601 start, nullable end, and nullable comment/reason/update information. A null end denotes an ongoing downtime. It supports historical downtime retrieval but does not establish a complete machine-state timeline.
 
-## VERIFIED WITH LIVE CUSTOMER API
+## VERIFIED WITH CUSTOMER EXPLORATION
 
-None. This execution environment has no ENLYZE DNS/network access, so no customer operation has been sent in this milestone.
+The following are customer-specific observations, not general ENLYZE API
+semantics:
+
+- K7 candidate fiber mass-flow variable: `Stundenleistung Anlage` (kg/h).
+  Historical behaviour supports interpreting it as a process-responsive
+  band-scale signal. It can remain frozen at a non-zero value during stops.
+- Validated K7 integration gate: `Geschwindigkeit Gesamtanlage > 0.5 m/min`.
+  Do not use the boolean `Anlage läuft` as the primary gate for this
+  calculation.
+- `production_run.quantity_total` is populated from ERP/BI production feedback
+  received by ENLYZE for this setup, not necessarily from machine signals. The
+  tested CSV source field is `PCO_FEEDB_QUANTITY`. For inspected
+  machines/articles it is reported finished-product area in m²: 5.00 m × 50 m
+  = 250.00 m², 4.75 m × 50 m = 237.50 m², and 5.80 m × 50 m = 290.00 m².
+  `quantity_total.unit` is separately configured and is not authoritative
+  without configuration validation.
+
+Historical validation case (plausibility only): production order
+`K 7-12026000842`, Production Run 2026-08-31T10:57:54Z to
+2026-08-31T18:23:11Z, had ERP `quantity_total` 12,420 m². Speed-gated
+integration produced approximately 5.206 active hours, 5,180.1 kg fiber feed,
+2,174.2 m machine length, and 995.1 kg/h average active throughput. Article
+external ID `213408` (width 6.0 m; static mean basis weight 390.16 g/m²) gives
+an approximate reported finished-product mass of 4,845.8 kg. This supports the
+physical plausibility of the feed integration; it is not a precise material
+yield calculation.
 
 ## STILL UNKNOWN
 
-- Customer response values, permissions, authentication success/failure behavior, default page size, and rate/retention limits.
+- Full customer response coverage, permissions, authentication
+  success/failure behavior, default page size, and rate/retention limits.
 - Whether production-run filters have the desired interval-overlap behavior; completed-run `start`/`end` are candidate integration boundaries but require live confirmation.
-- Which customer variable is a consumption signal and whether its unit/scaling factor makes it usable for integration.
+- Whether the validated K7 signal/gate generalize to other machines, articles,
+  production conditions, or future signal configuration.
 - Whether omitted `resampling_interval` yields native/raw resolution; its unit and exact temporal meaning of resampling methods.
 - Sample ordering, boundary inclusion, native sampling behavior, gaps, nulls, quality/bad samples, late arrivals, corrections, and backfills.
 - A complete historical machine operational-state resource beyond downtimes.
+- Exact per-order material yield: band-scale material may remain in the machine
+  at an order transition and ERP feedback can arrive asynchronously or
+  periodically. WIP/transport-delay accounting is required before interpreting
+  feed-versus-finished-product differences as yield or scrap.
 
 ## Exact bounded manual exploration sequence
 

docs/roadmap.md

@@ -6,27 +6,43 @@ Establish package boundaries, architecture documentation, example calculation
 configuration, basic domain/protocol models, tests, and an optional local
 TimescaleDB Compose design.
 
-## 1. ENLYZE API exploration client/CLI (recommended exact next milestone)
+## 1. ENLYZE API exploration client/CLI (completed baseline)
 
 A read-only, GET-only raw inspection CLI and fixture sanitizer are implemented.
-The remaining work is live, authorized exploration of machines, signal catalog,
+Exploration established a K7 candidate mass-flow signal and speed gate; the
-bounded samples, state history, production-order history, and article/material
+evidence and remaining uncertainties are recorded in `docs/enlyze-api.md`.
-context. Record sanitized fixtures and update `docs/enlyze-api.md` with
+Continue to record sanitized fixtures for newly verified semantics.
-evidence. Do not implement ingestion or a database schema before this is
-complete.
 
-## 2. Persistence foundation
+## 2. Live material-consumption MVP (next implementation milestone)
+
+Detect the currently active Production Run and production order, continuously
+ingest new ENLYZE samples, maintain persistent incremental integration state,
+and store/expose cumulative material consumption for Grafana. Implement this
+as generic rate integration (`material_rate * elapsed_time`) subject to
+configured machine/process-specific validity conditions, so it can serve more
+than one machine/material pair. The first validated implementation is K7:
+integrate `Stundenleistung Anlage` (kg/h) only while `Geschwindigkeit
+Gesamtanlage > 0.5 m/min`; do not use `Anlage läuft` as the primary gate.
+
+Bento 1 bentonite-powder consumption is the next analogous use case. Select
+and validate its ENLYZE source variable and gating logic from process data
+before implementing it; neither is defined yet.
+
+## 3. Persistence foundation
 
 Design and migrate a TimescaleDB schema for derived metrics/events, calculation
-runs, context, and incremental state. Add Grafana-oriented query examples.
+runs, Production Run/order context, and incremental state. Add Grafana-oriented
+query examples.
 
-## 3. First vertical slice
+## 4. Historical replay/backfill
 
-Implement one configured integration calculation over a verified signal and
+Enable bounded historical Production Run replay and production-order analysis
-production-order context. Include backfill, rerun/provenance behavior, and
+using the same generic integration engine as live processing. Integrate each
+Production Run separately and aggregate derived values at the production-order
+level; never bridge gaps between runs. Include rerun/provenance behavior and
 tests against sanitized fixtures.
 
-## 4. Peak detection (first detector completed)
+## 5. Peak detection (first detector completed)
 
 The generic configurable sawtooth detector is implemented: it retains the
 current maximum, closes a cycle after a configurable below-peak fraction has
@@ -38,5 +54,7 @@ non-zero resets, and varying peak heights without detector special cases.
 
 ## Later
 
-State/cycle durations, aggregates by order, rolling statistics, threshold
+State/cycle durations, rolling statistics, threshold events, derivatives, and
-events, derivatives, and specific-consumption calculations.
+specific-consumption calculations. An optional historical mass-balance
+refinement is actual QA/QS basis-weight data from SQL, replacing static article
+basis weight; it is not required for the live material-consumption MVP.