Confidence score

A confidence score is a model’s own estimate of how sure it is about one specific output — one recognised character, one labelled page region, one classification decision — usually expressed as a number between 0 and 1 or a percentage. It is a signal for where to look, not a guarantee that a high-confidence output is correct or that a low-confidence one is wrong.

Where it comes from

For classical OCR engines like Tesseract, confidence is computed per recognised word from the character-recognition step’s internal probabilities — how closely the pixels matched the engine’s model of each candidate character. For a classifier or a vision-language model doing layout assessment, confidence typically comes from the probability the model itself assigned to its chosen label relative to the alternatives it considered. In both cases the number describes the model’s internal certainty, not an independent check against ground truth — a model can be confidently wrong if the input resembles something it was trained to recognise even when it is actually a different, unfamiliar case.

In Archēglyph

Every extracted region carries a confidence score from the engine that produced it. In the review screen, a region whose confidence drops below 65% renders with a warn-orange tint; regions at or above that threshold stay visually muted so attention goes where it is more likely to be needed. The dataset’s metadata panel also shows a confidence histogram across all regions, and a future faceted statistics plugin (see the roadmap) would surface the confidence distribution as a dataset-level, filterable facet rather than something visible only region by region.

A worked example

A scanned page with a coffee-ring stain over part of a paragraph produces a run of words with confidence scores in the 40-55% range where the stain obscures characters, sitting inside a paragraph whose unstained words score 90%+. The low-confidence run is exactly the visual signal a reviewer needs: not “this document is unreliable,” but “these specific eleven words, and only these, are worth a manual check or a VLM re-run.” A confidence score’s value is precisely this — it localises doubt to the smallest span it can, rather than casting doubt over an entire page or document uniformly.

The threshold is a heuristic, not ground truth

As a rule of thumb, if more than roughly a third of a document’s regions fall below the warn-orange threshold, a whole-document re-run with a vision-language model tends to be cheaper than fixing each region by hand; below that ratio, per-region re-runs and inline edits are usually faster. That threshold is a practical trade-off between review effort and correction quality, not a claim that 65% is where correctness begins — a well-calibrated model at 60% confidence and a poorly-calibrated one at 80% confidence can be equally unreliable in practice, which is why the tint is a prompt to look, not a verdict.

Limitations

Confidence scores from different engines are not directly comparable: a VLM’s stated confidence and a classical OCR engine’s stated confidence come from different underlying mechanisms and are not calibrated against each other on the same scale, so “70% confidence from the VLM” and “70% confidence from Tesseract” are not the same claim. Confidence also says nothing about errors the model is systematically prone to making with high confidence — a font style or an unfamiliar script the engine mis-generalises from training can produce consistently wrong output at consistently high stated confidence, because the model isn’t uncertain about a wrong prediction it makes the same way every time.

Not to be confused with

Provenance records which model produced an output; confidence score records how sure that model was. A passage can carry high confidence from a model whose outputs a researcher has learned to distrust for other reasons entirely, which is why Archēglyph shows both the engine id and the confidence score side by side rather than either alone. See Reviewing a noisy scan for the practical workflow around the review screen’s confidence tint, and Vision-language model for the engine most often used to re-run a low-confidence region.

Bottom line

A confidence score tells you where a model was unsure of itself, which is useful for directing review effort — but it is the model grading its own homework, not an independent correctness check, and it should be read alongside the provenance record and, where it matters, a human’s own judgment of the source page.

FAQ

← Back to the glossary