fluid-wtp-ocr

OCR library for Water Treatment Plant (WTP) logbook images. It sends page images to Google Cloud Document AI custom processors and converts the raw entity output into typed, structured Pydantic models.

Features

• Document AI integration — runs configured custom processors and normalizes the response into RawOcrDocument
• Domain parsers — maps OCR entities to WTP logbook fields (pH, turbidity, TDS, times, and more)
• Typed output — every extracted value is wrapped in OcrField[T] with raw text, parsed value, confidence, and entity metadata
• Row alignment — the WTP left-page parser groups entries across treatment stages into validated WTPLeftPageLogRow objects

Requirements

• Python 3.10+
• A Google Cloud project with Document AI enabled
• Application Default Credentials configured for the GCP project (for example via gcloud auth application-default login)

Installation

poetry install

Or install the package directly:

pip install .

Configuration

Set the following environment variables before calling the OCR functions:

Usage

WTP left page

from fluid_wtp_ocr.get_structured_ocr import get_wtp_left_page_ocr

ocr_result = get_wtp_left_page_ocr(
    file_path="/path/to/logbook-page.pdf",
    mime_type="application/pdf",
)

print(ocr_result.time_elapsed)
print(ocr_result.result.date)

for row in ocr_result.result.log_rows or []:
    if row.sand_filter_entry and row.sand_filter_entry.ph:
        print(row.sand_filter_entry.ph.value)

WTP right page

from fluid_wtp_ocr.get_structured_ocr import get_wtp_right_page_ocr

ocr_result = get_wtp_right_page_ocr(
    file_path="/path/to/logbook-page.pdf",
    mime_type="application/pdf",
)

print(ocr_result.time_elapsed)
print(ocr_result.result.date)

WTP coagulation page

from fluid_wtp_ocr.get_structured_ocr import get_wtp_coagulation_log_ocr

ocr_result = get_wtp_coagulation_log_ocr(
    file_path="/path/to/logbook-page.pdf",
    mime_type="application/pdf",
)

print(ocr_result.time_elapsed)
print(ocr_result.result.date)

Supported MIME types depend on the input file (for example application/pdf, image/jpeg, image/png).

Output structure

get_wtp_left_page_ocr returns an OCRResult[WTPLeftPageOCR]:

OCRResult(
    result=WTPLeftPageOCR(...),
    time_elapsed=1.23,
)

WTPLeftPageOCR contains:

• date — parsed logbook date
• Per-stage entry lists: sand_filter_entries, uf_entries, acf_entries, lead_entries, lag_entries, ro_entries
• log_rows — aligned rows across all six stages when entry counts match; [] if they don't (rows can't be aligned at all). A row missing a stage's time entirely is dropped; a row with a stage's time out of chronological order relative to the previous stage is kept but flagged via chronology_flag/chronology_issues.

Each field on an entry is an OcrField[T]:

OcrField(
    raw_text="8.02",
    value=Decimal("8.02"),
    confidence=0.999,
    entity_id="4",
    normalized_text="8.02",
)

Supported left-page entry types

Architecture

Image/PDF
   │
   ▼
Google Document AI (custom processor)
   │
   ▼
RawOcrDocument          ← entities + full OCR text
   │
   ▼
OcrParser (per processor)
   │
   ▼
OCRResult[LogbookOCR]   ← typed domain models

Processors and parsers are registered by name:

• OCR_PROCESSORS in ocr_processors.py
• OCR_PARSERS in parse_ocr/parsers.py

To add a new document type, define a processor config, implement an OcrParser subclass, and register both in those maps.

Ground-truth regression testing

We keep a manually-reviewed set of real logbook pages (RO and coagulation) as a regression benchmark: whenever a prompt, model, or parser changes, re-run OCR against this set and compare the output to the reviewer-confirmed values, instead of eyeballing a handful of pages in the app.

Getting the fixture bundle

The dataset is exported from fluid-backend with the export_wtp_ocr_fixtures management command (see its docstring there for filtering by date/org/facility) and shared as a .zip — currently attached to the ENG-601 Linear issue. Download it somewhere outside this repo (a gitignored fixtures/ folder here works fine) — the zip contains real facility images and must never be committed.

Each record in the bundle has the source images, meta.json, and (when a reviewer has worked it) expected/review.json — the human-corrected values, which is what this tool treats as ground truth. expected/response_payload.json is only the previous OCR run, not a verified answer, so it's never scored against — it's there for your own before/after reference if you want it.

Running it

poetry run python scripts/run_ground_truth_eval.py \
  --fixtures fixtures/wtp_ocr_fixtures_20260728_061651.zip \
  --record-type ro \
  --report-csv ro_eval.csv \
  --save-baseline

Needs the same environment as normal OCR calls: GEMINI_API_KEY, OCR_PROJECT_ID, OCR_LOCATION, WTP_COAGULATION_OCR_PROCESSOR_ID (see Configuration above). Run once per record type (ro or coagulation) against the matching fixture zip.

Only records with a submitted review are scored by default — an in-progress or unreviewed record has no confirmed answer to compare against. Pass --include-unsubmitted-reviews to score them anyway.

To try it on a couple of records instead of the whole bundle (each --record-id is a live Gemini/Document AI call, so this keeps a quick sanity check cheap), pass --record-id — it's the fixture's records/<this> directory name, also in the CSV's record_id column. Two records per type with solid review coverage from the 2026-07-28/29 fixture bundles:

poetry run python scripts/run_ground_truth_eval.py \
  --fixtures fixtures/wtp_ocr_fixtures_20260728_061651.zip --record-type ro \
  --record-id 2025-04-10_ro_5782e5b1 --record-id 2025-10-13_ro_7cdbbe62 \
  --report-csv ro_smoke.csv

poetry run python scripts/run_ground_truth_eval.py \
  --fixtures fixtures/wtp_ocr_fixtures_20260729_121945.zip --record-type coagulation \
  --record-id 2026-06-29_coagulation_8c23bd55 --record-id 2026-06-17_coagulation_749a2b7b \
  --report-csv coag_smoke.csv

Reading the results

The console prints a per-record accuracy and error breakdown, then an aggregate by record type, and (if a prior --save-baseline run exists) how accuracy moved since then. --report-csv writes one row per compared field — open it in Sheets/Excel: each row is record, page, table, field, ground_truth_value, predicted_value, category, so you can filter down to just the rows that aren't match and go through them.

Every non-matching field is put into one of these categories:

date_check is a separate, informational-only row per record (never counted toward accuracy) comparing the extracted page date to the upload's confirmed log_date — a mismatch here is common and doesn't by itself mean the page date was misread; it can just mean the two dates were never expected to agree.

A record's accuracy is matched fields / all comparable fields — blank-on-both-sides fields aren't counted. Row alignment between OCR output and ground truth uses each entry's time field when present (tolerant of small time differences); tables without a time column, or where OCR badly over/under-counts rows, fall back to a positional match and will show elevated missed_field/hallucinated_value even when the OCR mostly got individual values right — read the CSV for those tables rather than trusting the aggregate percentage alone.

Development

Install dev dependencies:

poetry install --with dev

Run tests:

poetry run pytest

Lint:

poetry run ruff check .