OCR Benchmark Pipeline
Benchmarking OCR engines on scanned documents, scored against ground truth, not just timed
The problem
Comparing OCR engines usually means running each one by hand and eyeballing the output. This pipeline automates that: it pulls scanned PDFs from object storage, runs a selected engine against every page, and records timing, resource usage, and, whenever a ground-truth label exists, real accuracy against it (character/word error rate, field extraction, and localization), so engine choices can be backed by actual numbers instead of a spot check.
Architecture
How it works
Ingestion and conversion
PDFs are pulled from a MinIO object store, or read from local disk in offline mode, and converted to PNG images, one per page. Each pipeline run is tracked with a run ID, the engine used, and a status, so every result traces back to a specific run.
OCR and benchmarking
The selected engine runs against every page. Each page's result is timed, and its confidence score and extracted character count are recorded alongside the document and page number. When a ground-truth label exists for that page, the same run also scores character/word error rate and field-extraction accuracy against it, automatically. A separate resource monitor records RAM and CPU usage for the run. All of this is written to PostgreSQL as structured metrics, while the raw OCR output goes to MongoDB as JSON.
Storage and API
PostgreSQL holds run metadata, per-document stats (pages, elapsed time, CPU and memory use), and per-page metrics, indexed by engine so results can be filtered. MongoDB holds the raw OCR JSON for each page. A FastAPI backend serves both, and the Streamlit dashboard uses it to list runs and browse the extracted text for any run.
Accuracy and ground truth
Timing and confidence say how fast an engine ran and how sure it claims to be. Neither says whether it read the document correctly. Three separate metrics do.
Character and word error rate
CER/WER, computed with jiwer, measures raw text fidelity: how close the OCR output is to a ground-truth transcript, character by character and word by word.
Field-extraction accuracy
A higher-level, business-relevant check: did the engine get the actual field values right (Report ID, Date, Route...), not just the raw characters? Two engines can post near-identical CER/WER and very different real-world usefulness if the one character error lands inside a date or an ID.
Localization accuracy
Beyond reading the right value, did the engine also find where it is on the page? Measured as IoU (intersection-over-union) against a ground-truth box, the same style of metric real document-AI benchmarks (FUNSD, CORD, DocVQA) use.
Where the ground truth comes from
Ground truth can't come from reading the same document you're trying to OCR, that would be circular. For the shipped sample document, the labels are free: the generator draws known strings onto the page with PIL, so the string "RPT-1000" exists in memory before a single pixel is drawn, and gets written straight to the label file at that moment. The OCR engines then read only the pixels, with no access to that original string. For real documents this shortcut doesn't apply: labels have to come from a human reading the document, or an already-verified system of record, never from OCR-ing it again.
Benchmark results
The numbers below come from the one shipped sample document (2 labeled pages, run through Tesseract and EasyOCR). PaddleOCR is verified separately, explained below.
| Engine | CER | WER | Field accuracy | Avg. IoU | Sec/page |
|---|---|---|---|---|---|
| Tesseract | 0.0025 | 0.0417 | 100% | 0.92 | 1.39 |
| EasyOCR | 0.0076 | 0.0417 | 60% | 0.64 | 18.82 |
On this document, Tesseract is both more accurate and about 13 times faster than EasyOCR: lower character error rate, every field extracted correctly against 3 of 5 fields for EasyOCR, tighter localization boxes, and 1.4 seconds a page against 18.8. EasyOCR's misses were concentrated on two specific fields (Inspector, Report ID), not spread evenly, which the per-field breakdown in the dashboard makes visible.
PaddleOCR isn't in the table above because this run used the Docker-based dashboard stack, where PaddleOCR can't currently execute (see Limitations). Run independently on the native path, it scored CER/WER 0.0 and got all 5 fields and all 5 localizations correct on the same sample document.
A fourth question, whether an engine's self-reported confidence actually tracks its real accuracy, is also measured. With only 2 pages per engine, though, that correlation comes back statistically undefined. It would need a much larger labeled set to mean anything.
Adding an engine
Every engine implements the same interface (BaseOCREngine.predict), registered through a small loader dict so a new engine's dependencies are only imported when it's actually used. Adding one is three steps: a config file, an engine class, and a loader registration. No other part of the pipeline, conversion, benchmarking, storage, API, or dashboard, needs to change.
Testing
72 unit and integration tests cover the runtime helpers, both engines' output parsing, the CER/WER and field-extraction scoring, and the IoU localization calculations, run with pytest. CI runs ruff, black, and mypy, plus the full test suite, on every push and pull request.
Limitations
- ▸The benchmark numbers above come from one shipped synthetic document (2 labeled pages per engine). That proves the metric machinery works end to end, it isn't a statistically meaningful evaluation. Real scanned documents (noise, skew, low contrast) would likely widen the gap between engines.
- ▸PaddleOCR is implemented and verified correct on the native path, but segfaults inside the Docker container on Apple Silicon: an unresolved upstream bug in PaddlePaddle's linux/arm64 wheel (PaddlePaddle/Paddle#76111), not something fixable from this repo.
- ▸Confidence calibration, whether an engine's self-reported confidence tracks its real accuracy, is measured but statistically undefined with only 2 labeled pages per engine. It needs a much larger labeled set to mean anything.