martin/machine-vision-poc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Document Carbofol vision PoC scope and proposed architecture

Browse Source
This commit is contained in:
admin
2026-09-09 05:57:18 +02:00
commit b39dc98751
8 changed files with 326 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+43
View File
@@ -0,0 +1,43 @@
# Python and development environments
__pycache__/
*.py[cod]
.venv/
venv/
.pytest_cache/
.mypy_cache/
.ruff_cache/
.coverage
htmlcov/
*.egg-info/
build/
dist/
# Local credentials and configuration
.env
.env.*
!.env.example
# Runtime evidence, databases, models and generated artifacts
/data/
/models/
/outputs/
/work/
/logs/
*.sqlite
*.sqlite3
*.sqlite-*
*.sqlite3-*
*.db
*.db-*
*.pt
*.pth
*.onnx
*.engine
*.gguf
*.safetensors
*.log
# OS and editor local state
.DS_Store
.idea/
.vscode/
README.md
+58
View File
@@ -0,0 +1,58 @@
# Carbofol Machine Vision PoC
Explore whether camera images and reproducible illumination can reliably reveal surface defects on a Carbofol sealing membrane during production. Build a reviewable defect catalogue and a small operator interface as the foundation for later software development.
**Status:** documentation and source placeholder only. No capture service, trained model, benchmark, validated detection accuracy or deployed application exists in this repository. Hardware has not been selected or purchased as part of this task.
## Requirements from the discussion
| Constraint | Current scope |
|---|---|
| Material | Carbofol membrane; exact product and surface variant to confirm |
| Web speed | Maximum 4 m/min |
| Observed width | Approximately 30–50 cm; not necessarily the whole production width |
| Cameras | Two: complementary views of one surface, or one per surface; interpretation and geometry to confirm |
| Processing time | Less than 60 seconds from capture to evaluation; precise acceptance definition to agree |
| Interface | Continuously show the latest captures, highlight suspected defects and raise an alarm |
| Storage | Persist and catalogue defect evidence and human review |
| Description | Small image-capable language model desirable, optional |
| Software | Open-source preferred; dependencies and model licences to evaluate when selected |
## Current architecture proposal
```text
Two cameras + controlled illumination + trigger
|
Capture and preprocessing
|
Vision / anomaly detection worker
|
SQLite metadata + image filesystem
|
FastAPI + small browser interface
| |
Immediate alarm Human review/catalogue
|
Optional asynchronous VLM description
(updates the stored result and interface later)
```
Detection results, persistence and alarms must not wait for language generation. A VLM can interpret crops; a text-only LLM can only describe supplied features. Predictions remain suggestions until reviewed. The working candidate is an Orin Nano Super 8 GB, subject to image-quality, memory and throughput experiments. An existing computer can support the first acquisition experiments.
This is an experimental inspection assistant. Machine-stop control and production acceptance/rejection integration are outside the current scope.
## Documentation
- [Architecture](docs/architecture.md): capture, illumination, inference, interface and failure handling.
- [Hardware options](docs/hardware-options.md): historical budgets and unresolved procurement choices.
- [Data model](docs/data-model.md): SQLite entities, image storage and review history.
- [Roadmap](docs/roadmap.md): experiments, acceptance gates and open decisions.
- [Discussion provenance](docs/provenance.md): source and treatment of earlier claims.
## Software home
`src/carbofol_inspection/` reserves the Python package. Its README describes intended boundaries. No dependencies, runtime commands or API endpoints are claimed to work yet. Introduce packaging and pinned dependencies with the first executable vertical slice, starting with recorded images before camera integration.
Keep runtime images, databases, model weights, secrets and generated outputs out of Git. The future application should accept an explicit data root, preferably on SSD and outside the checkout. Git tracks code, configuration examples and documentation; it is not the defect database.
This repository is local. Remote hosting and a project licence remain to be selected by the owner.
docs/architecture.md
+55
View File
@@ -0,0 +1,55 @@
# Proposed architecture
All components below are a design proposal, not an implemented or measured system.
## Cameras and illumination
Compare two arrangements before fixing the rig:
1. Two views of the same surface: approximately perpendicular with diffuse illumination, plus an oblique view with grazing light. This was the latest discussion preference for complementary evidence.
2. One camera per membrane surface if both surfaces require inspection. Confirm whether “each side” means opposite surfaces or lateral regions. Each arrangement needs its own coverage and calibration plan.
Experiment with diffuse light, grazing light from either direction, polarization and shielding from ambient light. Record the lighting recipe with every capture. Projected lines are a later option if ordinary illumination cannot reveal the target geometry; depth measurement requires calibration and is not implied by seeing a distorted line.
Use representative material, normal texture and known defects. Fix working distance, field of view, focus, exposure and gain. Machine-vision cameras with manual controls, suitable lenses and hardware triggering are candidates; existing webcams/action cameras can support initial visibility experiments. Check Linux/ARM driver support, sustained dual-camera transfer and power requirements before selecting interfaces.
## Coverage and triggering
4 m/min equals about 66.7 mm/s. At 1/500 s exposure, calculated travel is about 0.133 mm; whether that blur is acceptable depends on the smallest relevant defect. These are geometric calculations, not measurements.
The specified 30–50 cm is cross-web width. Along-web field of view is still unknown. For speed `v`, usable along-web view length `L` and overlap fraction `o`, a gap-free nominal schedule needs `interval <= L * (1-o) / v`; include margins for jitter and unusable image edges. The earlier example of one capture per 250 mm gives 3.75 s at maximum speed, but only works if usable coverage and overlap permit it. A proposed 1–2 fps per camera is an experiment, not a coverage guarantee.
Start with timestamped acquisition; later prefer encoder-triggered groups for repeatable production positions. Store actual per-camera timestamps, trigger identifiers and incomplete groups. Apply physical camera offsets when relating images to the same material. Do not assume that simultaneous frames show the same location. Without an encoder, mark estimated positions explicitly or leave them unknown.
## Processing and edge AI
- Capture adapters assign stable image/group IDs and save original evidence with camera settings.
- Preprocessing applies versioned regions of interest, calibration and normalization; retain the mapping back to original pixels.
- A vision worker starts with classical contrast/texture baselines and an anomaly-detection candidate such as Anomalib. Training uses reviewed normal material. Compare later supervised detection/segmentation once sufficient labels exist.
- Thresholded scores produce suspected anomaly regions, boxes and/or masks. An anomaly score is not a calibrated probability or a confirmed defect class.
- Persist the inspection outcome and enqueue its UI event. Review and alarm operate on that result immediately.
- An optional bounded VLM job receives selected crops and verified metadata. A small quantized 2–4B candidate was discussed, but fit, image support, quality and latency must be tested on the actual runtime. Preserve prompt/model versions and distinguish generated suggestions from human labels.
Use a bounded inference queue and expose queue age. Never silently discard required inspection frames: overload, missing cameras or failed inference produce an explicit incomplete/error condition. Preview frames may be replaced by newer ones without implying inspection coverage. Prioritize detection over VLM work; disable or offload descriptions to an available LAN server if resources are insufficient. No server availability is assumed.
Separate training from online inspection. Record the model artifact hash, preprocessing configuration and threshold for reproducible replay. Verify runtime/export compatibility on the selected device instead of assuming every model supports TensorRT.
## Backend and frontend
Proposed stack: Python with FastAPI; server-rendered HTML/HTMX initially, with polling or SSE for updates. Choose the simplest working transport during implementation. Serve two latest images with timestamps, capture IDs, processing state and per-image overlays. Also show the latest completed inspection, pending work, camera health, storage health and alarm state.
Never apply an older result's overlay to a newer image. “No anomaly detected”, “not inspected yet” and “inspection failed” must remain distinct. An alarm should remain visible until acknowledged; acknowledgement is separate from accepting or rejecting the detection. An initial alarm is visual in the UI; audible/physical outputs and escalation rules are open decisions.
The catalogue supports filtering by run, time, camera, review status and class, with crops, originals and correction controls. Generated text must be visibly marked as unreviewed. Persist events before publishing updates; after reconnection, the client reloads authoritative state from the backend.
## SQLite and filesystem
Use SQLite for relationships and metadata, with images and masks stored as files; see [data model](data-model.md). Keep transactions short and serialize writes initially. Evaluate WAL mode on local storage when concurrent reads are introduced. A separate database server is unnecessary for the proposed single-device PoC, but revisit this if multiple hosts write concurrently.
Use atomic file finalization and record failure states because filesystem writes and database commits are not one transaction. Monitor free space, define retention and test coherent backups of both images and metadata. On storage failure, report degraded inspection and preserve the distinction between detection and successful evidence storage.
## Timing and validation
Proposed interpretation of the <60 s requirement: capture timestamp through completed evaluation, durable result and visible alarm when applicable, including queueing. Confirm this definition with the operator. Measure optional description completion separately; its deadline is undecided. Earlier <1 s detection and 10–20 s description figures were aspirations, not acceptance commitments or results.
Measure full-pipeline latency, backlog, dropped/incomplete captures, memory, storage growth and false alarms under sustained two-camera load. Low frame rate alone does not establish that the target hardware is sufficient. At maximum speed the membrane travels 4 m in 60 s; tracking and operator response must account for this displacement.
docs/data-model.md
+61
View File
@@ -0,0 +1,61 @@
# Proposed SQLite and filesystem data model
This is a logical design, not an installed schema or migration. Use stable UUID-style text IDs, UTC timestamps and explicit units. Enable SQLite foreign keys on every connection. Store original images once; reference them from inspections and reviews.
## Entities
| Entity | Key fields and relationships |
|---|---|
| `production_runs` | `id`, product/surface variant, lot, start/end timestamps, notes |
| `capture_configs` | `id`, immutable camera/lighting settings, calibration version/hash, trigger mode, preprocessing configuration |
| `capture_groups` | `id`, `production_run_id`, trigger ID, expected camera IDs, group status, optional `position_m`, position source (`encoder`, `estimated`, `unknown`) |
| `captures` | `id`, `capture_group_id`, camera ID, captured timestamp, `capture_config_id`, original asset ID, width/height in pixels, acquisition status/error |
| `model_versions` | `id`, task, model name, artifact hash, runtime and preprocessing versions, configuration snapshot |
| `inspections` | `id`, `capture_id`, `model_version_id`, start/completion timestamps, threshold, nullable score and anomaly decision, status (`pending`, `complete`, `error`), error text |
| `defects` | `id`, `inspection_id`, bounding box, region score, optional mask asset ID, predicted class and nullable classifier confidence; one row per suspected region |
| `descriptions` | `id`, `defect_id`, model/prompt version, generated text, status, created/completed timestamps, error; machine suggestions only |
| `reviews` | `id`, `defect_id`, reviewer, timestamp, verdict (`confirmed`, `false_positive`, `uncertain`), human class ID, comment, optional superseded review ID |
| `defect_classes` | `id`, stable code, display label, description, active flag |
| `assets` | `id`, relative path, kind (`original`, `crop`, `overlay`, `mask`), checksum, byte size, dimensions, storage status (`staging`, `ready`, `missing`, `deleted`) |
| `defect_assets` | `defect_id`, `asset_id`, role; links crops/overlays/masks without duplicating original images |
| `alarms` | `id`, `inspection_id`, created timestamp, state, acknowledged timestamp and user; acknowledgement does not imply review |
Relationships: one run has many groups; a group has one capture per expected camera, including failed acquisition records. One capture can have many inspection attempts/model versions. One inspection can produce zero or more suspected regions. A defect can have many descriptions and append-only reviews.
A missing camera or failed inspection never becomes a normal result. Keep score and anomaly decision null until successful evaluation. Enforce uniqueness of group/camera pairs and asset paths. Use indexes on group/run, capture/group, inspection/capture, defect/inspection and review/defect/time; add catalogue query indexes when queries exist.
## Coordinates and evidence
Use bounding boxes `(x, y, width, height)` in original-image pixels, top-left origin, positive size and bounds inside the associated image. Map model output back through preprocessing. Store mask dimensions and transforms explicitly. Physical size is nullable and requires recorded calibration; do not infer millimetres or surface height from generated text.
A defect row is a **camera observation**, not automatically a distinct physical defect. Overlapping frames and complementary views may duplicate the same event. If useful later, add a reviewed `physical_events` table plus observation links, recording association method and confidence. Do not fuse the two surfaces or deduplicate by timestamp alone.
## File layout and consistency
```text
<data-root>/
inspection.sqlite3
images/<run-id>/<group-id>/<camera-id>/<capture-id>.png
derived/<inspection-id>/<defect-id>/crop.png
derived/<inspection-id>/<defect-id>/overlay.png
derived/<inspection-id>/<defect-id>/mask.png
exports/<export-id>/manifest.json
```
Paths in SQLite are relative to the configured data root. PNG is an illustrative lossless format; choose format and bit depth against camera data and storage tests. Avoid moving originals when a review changes the verdict. Derived evidence should remain associated with the model that generated it.
Write assets to temporary paths on the same filesystem, finalize with atomic rename, then commit ready asset references and associated results in a short database transaction. Failures between steps may leave orphan files; reconcile these explicitly after restart. Never publish success with a broken asset reference. Use stable job IDs for retries to avoid duplicate alarms/records.
Define retention separately for defect originals, reviewed normal samples, unreviewed data and regenerable previews. Keep enough normal material to evaluate false negatives and drift. Record deletions rather than leaving unexplained broken paths. Back up SQLite consistently along with its referenced files; account for WAL state and test restoration. Do not copy only the live main database file and assume a complete backup.
## Human-in-the-loop catalogue
1. Persist a suspected region and any machine label; new observations are unreviewed.
2. Present original context, crop, overlay, score and generated description separately.
3. The reviewer confirms, rejects as false positive, or marks uncertain; they can correct the class and add comments.
4. Append a review with identity and time. Derive current review status from the latest non-superseded review; preserve previous decisions and machine output.
5. Export a versioned dataset manifest listing evidence hashes, model versions and approved labels. Exclude uncertain observations from ground truth unless explicitly resolved.
Candidate classes from the discussion: material accumulation, foreign body, fold, hole, surface-structure anomaly and other/unknown. These require domain review. “No defect” is a review verdict, not a physical defect category. An unreviewed model label or generated description is not ground truth.
Allow reviewers to inspect normal captures and add missed regions through a future manual annotation workflow. Detection-only review cannot measure missed defects. Split training and evaluation by production run/lot and linked physical event to prevent overlapping images of the same material leaking across splits. Define representative held-out ground truth before reporting accuracy.
docs/hardware-options.md
+49
View File
@@ -0,0 +1,49 @@
# Hardware options and historical budget snapshot
**Snapshot: discussion of 2026-09-09, Germany/EUR.** The numbers below preserve the conversation, including user corrections. They are not newly verified quotes, current offers or purchase recommendations. VAT, shipping, accessories and supplier eligibility were not consistently established. Recheck complete system contents and prices before purchasing.
## Evolution of the budget
| Historical tier | Discussed total | Original concept and limitation |
|---|---:|---|
| Minimal | €500–900 | Webcam/HQ camera, basic LEDs, preferably existing computer; predates the firm two-camera scope |
| Comfortable | €1,500–3,000 | Machine-vision optics, controlled light, Jetson or existing GPU computer; initial scope estimate |
| High end | €5,000–10,000 | Two industrial cameras, specialized illumination/polarization or projected lines, GPU workstation |
An intermediate proposal was €1,000–1,500 for one machine-vision camera, optics, lighting and an existing computer. These earlier envelopes are not equivalent bills of materials for the final two-camera requirement.
## Compute discussion after corrections
| Candidate | Discussed compute cost | Rationale and unresolved limits |
|---|---:|---|
| Existing Linux/GPU computer | No new compute purchase assumed | Start collecting images if suitable hardware is available; actual availability unknown |
| Jetson Orin Nano Super 8 GB | Approximately €600 | Latest edge PoC candidate; limited shared memory must cover OS, capture buffers, vision and optional VLM |
| Jetson Orin NX 16 GB system | Approximately €1,600 | More memory and embedded headroom; cost premium requires a demonstrated need |
| PC with 16 GB NVIDIA GPU | Approximately €2,000 or more | General development/training option; total configuration and performance unverified |
The earlier NX estimate of €1,000 was corrected by the user as education-only pricing and must not be used as the ordinary procurement baseline. A bare module, a developer kit and a complete system with carrier board, cooling, power supply and storage are different purchases. Earlier US list/module prices do not establish a delivered German system cost.
The discussion also corrected overly optimistic desktop estimates: a GPU alone had been discussed around €1,120–1,250, with complete systems around €1,800–2,200 or above. Specific RTX availability statements are historical, unverified market observations, not a claim that only one 16 GB model can be bought.
The device here is the **Jetson Orin Nano Super**, not the older Jetson Nano. Previous TOPS and language-model fit claims are not application benchmarks. Shared Jetson RAM is not directly comparable with discrete GPU VRAM plus separate system RAM. Do not size the device from model weight size alone: image encoders, runtime buffers, context, camera buffers and other services also consume memory.
## Latest two-camera indicative bill of materials
| Item | Discussed allowance |
|---|---:|
| Orin Nano Super 8 GB | €600 |
| 1–2 TB NVMe SSD | €80–140 |
| Two machine-vision cameras | €600–1,200 |
| Two C-mount lenses | €200–400 |
| Two to four LED/line lights | €200–500 |
| Polarizers and small accessories | Approximately €100 |
| Mounts/profiles | €100–200 |
| **Sum of the listed allowances** | **€1,880–3,140** |
The conversation rounded this to approximately €1,900–3,100. Cabling, trigger/encoder hardware, enclosure, any missing power/cooling, shipping and engineering effort need separate confirmation. SSD capacity is a candidate allowance, not a retention guarantee: calculate it from measured stored image size, frame rate and retention duration.
## Working direction, not a purchase decision
First establish whether the defects are visible under reproducible lighting. Reuse cameras, lenses and illumination across compute platforms. The latest discussion favoured the Nano over the NX because the roughly €1,000 difference could support better image acquisition. Validate this hypothesis with recorded-image benchmarks and a dual-camera trial before committing.
Confirm minimum defect size, working distance, actual pixel resolution, usable field of view, shutter/trigger behaviour, lens coverage, raw capture modes, interface bandwidth and software support. Test local VLM descriptions only after detection is stable; an external description worker remains an option. No performance or memory-fit guarantee has been established.
docs/provenance.md
+9
View File
@@ -0,0 +1,9 @@
# Discussion provenance
This documentation was prepared on 2026-09-09 from the referenced ChatGPT conversation [Hardwarevorschläge Bilderkennung](chatgpt-conversation://6aa0caef-86b8-83eb-959f-4f709252e9a1), conversation ID `6aa0caef-86b8-83eb-959f-4f709252e9a1`. The retrieved conversation contained seven exchanges, from the initial three-budget request through the request to preserve the project in Git. The expanded messages were used to recover the earlier speed, width, illumination, catalogue and budget context.
This is a compact technical synthesis, not a verbatim transcript. Requirements explicitly stated by the user take precedence over assistant suggestions. The latest €1,600 NX correction supersedes the earlier €1,000 education-price assumption. Two cameras supersede the initial one-camera starter proposal.
Historical prices remain attributed discussion estimates; no new market research was performed. Prior claims of ample compute capacity, model fit, deterministic AI decisions and subsecond latency were not measurements and are not carried forward as guarantees. Illustrative scores, defect dimensions and UI examples from the chat are not experimental observations.
The queue/error handling, normalized data model, versioned review history, evidence integrity and phase gates elaborate the discussion into a proposed engineering design. They are not existing implementation or user-approved purchasing decisions. No source images, trained weights or actual production datasets were supplied with this task.
docs/roadmap.md
+39
View File
@@ -0,0 +1,39 @@
# PoC roadmap
No phase has produced measured results yet. The repository establishes the proposed scope and design.
| Phase | Work | Exit evidence |
|---|---|---|
| 0 — Define the inspection task | Confirm material variants, target defects, minimum size, observed surfaces, field of view and operator response | Agreed defect examples and acceptance protocol, including tolerated missed defects/false alarms and latency definition |
| 1 — Establish visibility | Compare diffuse/grazing light, polarization, geometry and exposure on static and moving material | Versioned image set with settings; domain expert can identify target defects reproducibly |
| 2 — Capture and catalogue | Implement recorded-image replay, then two-camera capture; SQLite/filesystem persistence and basic review UI | Traceable images, explicit missing-camera states, repeatable capture coverage, successful backup/restore |
| 3 — Detection baseline | Compare classical CV with anomaly detection; calibrate thresholds and map regions to originals | Held-out results per defect class/size and production run; false alarms, missed defects and localization documented |
| 4 — Integrated edge trial | Deploy chosen vision worker and live UI; benchmark sustained two-camera load and failures | Measured end-to-end latency against <60 s, queue stability, coverage, memory and storage; alarms and recovery demonstrated |
| 5 — Optional descriptions | Evaluate a small VLM locally or on a separate available server | Reviewed description quality, measured resource use and proof that failure/overload cannot block detection |
| 6 — Next-stage decision | Compare visibility, detection quality, usability and complete costs | Evidence-based decision to stop, adjust optics/data, upgrade compute or plan production integration |
## First software slice
Implement replay of recorded images through a placeholder inspection interface, persistent metadata and a page showing two timestamped images. Use explicit “not evaluated” states until a detector exists. Add human review, then a measured baseline detector and camera adapters. Add dependency locking and installation instructions with that first executable implementation.
Tests should cover real failure modes: wrong-image overlays, incomplete camera groups, inference/storage failure, restart after interrupted writes, duplicate retry events and review history. Use a small synthetic fixture set for software checks, clearly separated from real detection evaluation.
## Open decisions
- Which Carbofol variants and defects matter, and what is the smallest relevant defect?
- Do the two cameras observe one surface from two angles, opposite surfaces, or lateral areas?
- What are working distance, usable along-web view, required overlap and position accuracy?
- Is encoder/trigger access available, and what offsets separate observation points?
- Does <60 s include durable storage, UI alarm and descriptions? Proposed baseline excludes optional descriptions.
- What false-negative and false-positive rates are acceptable, using which evaluation units (region, frame, metre or physical defect)?
- Which camera/interface/lens/lighting combination actually reveals the target defects?
- Is a suitable existing computer available? Does a Nano meet measured resource needs?
- What should alarms do, how are they acknowledged, and what happens on system failure?
- How long are originals/normal samples retained; who reviews data and owns backups?
- Which model/runtime versions and licences fit the intended use? Which project licence and remote hosting should the owner choose?
## Evidence discipline
For each experiment record date, material/lot, rig configuration, dataset manifest, code/model versions, threshold, hardware/runtime settings and observed results. Report sample counts and limitations alongside metrics. Measure latency from actual timestamps including queueing, with percentiles and worst observed value under a stated workload. Neither a low average nor advertised TOPS proves the <60 s condition.
Keep the evaluation set separate from threshold tuning and training. Retain negative samples and review missed defects, not only alarms. Promotions of human labels into a training dataset must be explicit and versioned.
src/carbofol_inspection/README.md
+12
View File
@@ -0,0 +1,12 @@
# Future Python package
Reserved for the first executable implementation. Suggested module boundaries:
- `capture`: camera adapters, triggers and recorded-image replay.
- `vision`: preprocessing and detector interface.
- `storage`: SQLite migrations, repository access and asset lifecycle.
- `web`: FastAPI routes, templates and update transport.
- `catalogue`: review history and dataset exports.
- `description`: optional asynchronous VLM integration.
Create these modules as functionality is implemented. Start with a single application and a bounded worker rather than distributed services. No runtime or dependencies are installed by this scaffold.