# fluid-wtp-ocr

OCR library for Water Treatment Plant (WTP) logbook images. It sends page images to [Google Cloud Document AI](https://cloud.google.com/document-ai) custom processors and converts the raw entity output into typed, structured Pydantic models.

## Features

- **Document AI integration** — runs configured custom processors and normalizes the response into `RawOcrDocument`
- **Domain parsers** — maps OCR entities to WTP logbook fields (pH, turbidity, TDS, times, and more)
- **Typed output** — every extracted value is wrapped in `OcrField[T]` with raw text, parsed value, confidence, and entity metadata
- **Row alignment** — the WTP left-page parser groups entries across treatment stages into validated `WTPLeftPageLogRow` objects

## Requirements

- Python 3.10+
- A Google Cloud project with Document AI enabled
- Application Default Credentials configured for the GCP project (for example via `gcloud auth application-default login`)

## Installation

```bash
poetry install
```

Or install the package directly:

```bash
pip install .
```

## Configuration

Set the following environment variables before calling the OCR functions:

| Variable | Description |
|---|---|
| `OCR_PROJECT_ID` | Google Cloud project ID |
| `OCR_LOCATION` | Document AI processor region (for example `asia-south1`) |
| `WTP_LEFT_PAGE_OCR_PROCESSOR_ID` | Document AI processor ID for the WTP left-page extractor (legacy path; production uses Gemini for this page, see below) |
| `WTP_RIGHT_PAGE_OCR_PROCESSOR_ID` | Document AI processor ID for the WTP right-page extractor (legacy path; production uses Gemini for this page, see below) |
| `WTP_COAGULATION_OCR_PROCESSOR_ID` | Document AI processor ID for the WTP coagulation extractor — this page still runs on Document AI in production |
| `GEMINI_API_KEY` | Gemini API key, required by `get_wtp_left_page_ocr_gemini`/`get_wtp_right_page_ocr_gemini` in `gemini_ocr/` — the RO left/right page path production actually uses |

## Usage

### WTP left page

```python
from fluid_wtp_ocr.get_structured_ocr import get_wtp_left_page_ocr

ocr_result = get_wtp_left_page_ocr(
    file_path="/path/to/logbook-page.pdf",
    mime_type="application/pdf",
)

print(ocr_result.time_elapsed)
print(ocr_result.result.date)

for row in ocr_result.result.log_rows or []:
    if row.sand_filter_entry and row.sand_filter_entry.ph:
        print(row.sand_filter_entry.ph.value)
```

### WTP right page

```python
from fluid_wtp_ocr.get_structured_ocr import get_wtp_right_page_ocr

ocr_result = get_wtp_right_page_ocr(
    file_path="/path/to/logbook-page.pdf",
    mime_type="application/pdf",
)

print(ocr_result.time_elapsed)
print(ocr_result.result.date)
```

### WTP coagulation page

```python
from fluid_wtp_ocr.get_structured_ocr import get_wtp_coagulation_log_ocr

ocr_result = get_wtp_coagulation_log_ocr(
    file_path="/path/to/logbook-page.pdf",
    mime_type="application/pdf",
)

print(ocr_result.time_elapsed)
print(ocr_result.result.date)
```

Supported MIME types depend on the input file (for example `application/pdf`, `image/jpeg`, `image/png`).

## Output structure

`get_wtp_left_page_ocr` returns an `OCRResult[WTPLeftPageOCR]`:

```python
OCRResult(
    result=WTPLeftPageOCR(...),
    time_elapsed=1.23,
)
```

`WTPLeftPageOCR` contains:

- `date` — parsed logbook date
- Per-stage entry lists: `sand_filter_entries`, `uf_entries`, `acf_entries`, `lead_entries`, `lag_entries`, `ro_entries`
- `log_rows` — aligned rows across all six stages when entry counts match; `[]` if they don't
  (rows can't be aligned at all). A row missing a stage's time entirely is dropped; a row with
  a stage's time out of chronological order relative to the previous stage is kept but flagged
  via `chronology_flag`/`chronology_issues`.

Each field on an entry is an `OcrField[T]`:

```python
OcrField(
    raw_text="8.02",
    value=Decimal("8.02"),
    confidence=0.999,
    entity_id="4",
    normalized_text="8.02",
)
```

### Supported left-page entry types

| Stage | Model | Example fields |
|---|---|---|
| Sand filter | `SandFilterEntry` | time, pH, turbidity, free chlorine, m/p alkalinity |
| UF | `UfEntry` | time, turbidity |
| ACF | `AcfEntry` | time, pH, turbidity, m alkalinity, total hardness, total chlorine |
| Lead | `LeadEntry` | time, pH, turbidity, TDS |
| Lag | `LagEntry` | time, pH, TDS |
| RO | `RoEntry` | time, feed/permeate pH and TDS, turbidity |

## Architecture

```
Image/PDF
   │
   ▼
Google Document AI (custom processor)
   │
   ▼
RawOcrDocument          ← entities + full OCR text
   │
   ▼
OcrParser (per processor)
   │
   ▼
OCRResult[LogbookOCR]   ← typed domain models
```

Processors and parsers are registered by name:

- `OCR_PROCESSORS` in `ocr_processors.py`
- `OCR_PARSERS` in `parse_ocr/parsers.py`

To add a new document type, define a processor config, implement an `OcrParser` subclass, and register both in those maps.

## Ground-truth regression testing

We keep a manually-reviewed set of real logbook pages (RO and coagulation) as a
regression benchmark: whenever a prompt, model, or parser changes, re-run OCR against
this set and compare the output to the reviewer-confirmed values, instead of eyeballing
a handful of pages in the app.

### Getting the fixture bundle

The dataset is exported from `fluid-backend` with the `export_wtp_ocr_fixtures`
management command (see its docstring there for filtering by date/org/facility) and
shared as a `.zip` — currently attached to the ENG-601 Linear issue. Download it
somewhere outside this repo (a gitignored `fixtures/` folder here works fine) — the zip
contains real facility images and must never be committed.

Each record in the bundle has the source images, `meta.json`, and (when a reviewer has
worked it) `expected/review.json` — the human-corrected values, which is what this
tool treats as ground truth. `expected/response_payload.json` is only the *previous*
OCR run, not a verified answer, so it's never scored against — it's there for your own
before/after reference if you want it.

### Running it

```bash
poetry run python scripts/run_ground_truth_eval.py \
  --fixtures fixtures/wtp_ocr_fixtures_20260728_061651.zip \
  --record-type ro \
  --report-csv ro_eval.csv \
  --save-baseline
```

Needs the same environment as normal OCR calls: `GEMINI_API_KEY`, `OCR_PROJECT_ID`,
`OCR_LOCATION`, `WTP_COAGULATION_OCR_PROCESSOR_ID` (see Configuration above). Run once
per record type (`ro` or `coagulation`) against the matching fixture zip.

Only records with a **submitted** review are scored by default — an in-progress or
unreviewed record has no confirmed answer to compare against. Pass
`--include-unsubmitted-reviews` to score them anyway.

To try it on a couple of records instead of the whole bundle (each `--record-id` is a
live Gemini/Document AI call, so this keeps a quick sanity check cheap), pass
`--record-id` — it's the fixture's `records/<this>` directory name, also in the CSV's
`record_id` column. Two records per type with solid review coverage from the
2026-07-28/29 fixture bundles:

```bash
poetry run python scripts/run_ground_truth_eval.py \
  --fixtures fixtures/wtp_ocr_fixtures_20260728_061651.zip --record-type ro \
  --record-id 2025-04-10_ro_5782e5b1 --record-id 2025-10-13_ro_7cdbbe62 \
  --report-csv ro_smoke.csv

poetry run python scripts/run_ground_truth_eval.py \
  --fixtures fixtures/wtp_ocr_fixtures_20260729_121945.zip --record-type coagulation \
  --record-id 2026-06-29_coagulation_8c23bd55 --record-id 2026-06-17_coagulation_749a2b7b \
  --report-csv coag_smoke.csv
```

### Reading the results

The console prints a per-record accuracy and error breakdown, then an aggregate by
record type, and (if a prior `--save-baseline` run exists) how accuracy moved since
then. `--report-csv` writes one row per compared field — open it in Sheets/Excel: each
row is `record, page, table, field, ground_truth_value, predicted_value, category`, so
you can filter down to just the rows that aren't `match` and go through them.

Every non-matching field is put into one of these categories:

| Category | Meaning |
|---|---|
| `missed_field` | Ground truth has a value; OCR extracted nothing |
| `hallucinated_value` | OCR extracted a value; ground truth has none |
| `incorrect_transcription` | OCR read the wrong value |
| `incorrect_normalization` | The raw OCR reading was actually correct, but the normalized/final value doesn't match — the bug is in normalization, not extraction |
| `formatting_mismatch` | Same value, different formatting (punctuation/spacing) than ground truth |
| `wrong_row_or_column_mapping` | A ground-truth value shows up unchanged somewhere else in the record — a row or table mix-up, not a reading error |

`date_check` is a separate, informational-only row per record (never counted toward
accuracy) comparing the extracted page date to the upload's confirmed `log_date` — a
mismatch here is common and doesn't by itself mean the page date was misread; it can
just mean the two dates were never expected to agree.

A record's `accuracy` is `matched fields / all comparable fields` — blank-on-both-sides
fields aren't counted. Row alignment between OCR output and ground truth uses each
entry's `time` field when present (tolerant of small time differences); tables without
a `time` column, or where OCR badly over/under-counts rows, fall back to a positional
match and will show elevated `missed_field`/`hallucinated_value` even when the OCR
mostly got individual values right — read the CSV for those tables rather than trusting
the aggregate percentage alone.

## Development

Install dev dependencies:

```bash
poetry install --with dev
```

Run tests:

```bash
poetry run pytest
```

Lint:

```bash
poetry run ruff check .
```

