41dots

Fuzzy search in OCR archives: what are you counting?

Run a six-record example to separate exact occurrences, fuzzy candidates and words missed at line breaks before choosing archive search behavior.

A reader searches a digitized archive for canal. The text contains canal, an OCR error canai, and the unrelated word cabal. Allowing one spelling edit retrieves all three. That helps discover a damaged transcription, but a count labeled “occurrences of canal” would now include a different word.

Keep an exact search available for checking the indexed wording. Offer spelling-tolerant results as candidates, with the matched word visible: show canai beside B and cabal beside D so a reader can accept or reject each passage before citing it.

Six records, three different answers

Consider this small English corpus. Each row is one searchable record; in a page-based archive, it could represent a page. B contains a deliberately introduced OCR substitution. E contains a word split by a hyphen and a newline.

RecordIndexed text
AThe canal opened in 1890. The canal carried coal.
BThe canai carried coal.
CThe canals carried coal.
DThe cabal met at dusk.
EThe ca-
nal carried coal.
FCoal moved along the canal.

The downloadable Python example creates an in-memory SQLite FTS5 index and runs the searches. It requires Python 3 with SQLite FTS5 enabled, uses no additional packages, and writes no files. Save it as archive-search.py, then run:

python3 archive-search.py

The index uses unicode61 tokenization with diacritic removal disabled, without stemming or synonyms. “Exact” here means an indexed token match, not byte-for-byte text equality. SQLite documents its tokenizers and phrase queries; a phrase matches consecutive indexed tokens, so punctuation need not behave like literal characters in a text editor.

The script implements spelling tolerance separately: it selects indexed terms within one insertion, deletion, or substitution of canal, then searches for any of those terms. This is a small teaching rule, suitable for inspecting the example vocabulary; scanning an entire vocabulary this way is not a production search design.

The output includes:

Exact canal: documents=A,F occurrences=3
Phrase "canal carried": documents=A
Expanded terms: cabal, canai, canal, canals
Fuzzy <=1: documents=A,B,C,D,F occurrences=6

Record A contributes two exact occurrences but only one matching document. A result count of two documents therefore cannot answer “how many times does this word occur?” The example counts token positions in the index to obtain the occurrence total.

The fuzzy result adds B, C, and D. B is the intended OCR recovery. C is a plural the reader may or may not want included. D is a false positive for a search about waterways. The spelling rule cannot distinguish those intentions. The six occurrences are occurrences of the expanded terms, not six confirmed instances of canal.

A broken word needs a different repair

Neither query recovers E. The index contains separate tokens ca and nal; neither is within one edit of canal. Increasing spelling tolerance to catch this split would also admit more unrelated vocabulary.

If line-end splits are common, test a separate normalization step that joins likely broken words before indexing. Preserve the original transcription and a reference to the page image so readers can check each repair. Joining every hyphenated line ending blindly can change legitimate wording. Add examples of both broken words and genuine hyphenated compounds before choosing the rule.

Turn the example into acceptance checks

For an archive search interface, show the search mode, matched spelling, and document count. When showing occurrence totals, state whether they cover the exact token or the expanded vocabulary. Link each hit to its page image and label whether the excerpt comes from the original transcription or normalized text.

The script checks that B is recovered, D remains visible as a candidate requiring review, and E remains missed. Replace or extend those records with permitted samples from your archive, then retain expected results when changing tokenization or spelling rules. Add names and short words: a tolerance useful for one term may be too broad for another.

Our bibliographic platform case study describes a Solr-based archive with fuzzy search and occurrence counting. This example uses SQLite to make the counting decision reproducible locally. For a Solr implementation, check the actual analyzer and parser: its fuzzy term and phrase proximity operators have distinct behavior. The script’s results do not specify that project’s query configuration.

All engineering guides

Have a project in mind?hello@41dots.com ↗