Sentence segmentation

Splitting a stream of prose into sentences. The rules look simple — full stop, newline — until you meet “Dr. Huxley, pp. 3–5, wrote:”, a 19th-century semicolon-heavy period, or a scan where line breaks carry meaning.

Why it matters for your research

Nearly every downstream analysis — embedding, classification, summarisation — works on the sentence or the chunk. Bad segmentation is invisible but poisons the results: two half-sentences clustering together because the period was missed is not a finding.

In Archēglyph

We use syntok for sentence splitting, chosen for its robustness on historical prose. Chunks are built by grouping segmented sentences until a size threshold.

Not to be confused with

Tokenization breaks each sentence into the sub-word pieces a model sees; sentence segmentation operates a level higher.

Where historical prose breaks the rules

A rule-based splitter tuned on modern news text assumes a period followed by a capital letter starts a new sentence. Historical and archival prose violates that constantly: abbreviated honorifics (“Dr.”, “Mr.”, “St.”), Latin citation shorthand (“op. cit.”, “ibid.”, “et seq.”), section-number lists, and OCR line breaks that fall mid-sentence because the scanner followed the printed column rather than the grammar. A splitter that treats every period as a boundary turns “Dr. Huxley wrote to the Society” into two fragments, one of which is a floating “Dr.”

Why the failure is invisible

A wrongly-split sentence doesn’t throw an error — it silently becomes two shorter, less coherent units that get embedded and chunked as if they were complete thoughts. The downstream symptom is a chunk that reads as a fragment, an embedding that sits in an odd part of the vector space because it lacks the verb that would have anchored its meaning, or two half-sentences from unrelated topics landing in the same cluster because the actual content that would have separated them was cut in half. Because nothing crashes, segmentation quality is one of those pipeline stages worth spot-checking directly rather than trusting by default.

Why syntok over a simpler splitter

A regex splitter on . (period followed by space) fails on virtually every abbreviation and on any sentence containing a decimal number or an initial. syntok instead uses a small trained model that looks at the token before and after a candidate boundary to judge whether it is really a sentence end, which handles “Dr. Huxley” correctly far more often than a fixed rule list of known abbreviations ever could — the fixed-list approach breaks the moment a corpus uses an abbreviation the list didn’t anticipate, and archival material has an effectively unbounded set of period-marked abbreviations across centuries and disciplines.

Where it sits in the pipeline

Sentence segmentation runs after OCR or VLM extraction produces raw text and before chunking groups sentences into embedding-sized units. Getting it right at this stage is cheaper than fixing it later, because every downstream stage — embedding, indexing, clustering — inherits whatever boundaries were drawn here without re-examining them.

A quick way to spot-check it

Sampling a handful of sentences from a chunked document and reading them in isolation is usually enough to catch systematic segmentation problems: a run of sentences that start mid-clause, or one that is suspiciously short and consists only of a name and a number, both suggest the splitter is misreading a pattern common in that particular source rather than a one-off fluke. Catching that early is far cheaper than diagnosing it later from an odd cluster or a search result that quotes half a thought.

It runs once per document version, not once per query

Segmentation happens during the extract stage, before embedding and indexing, so its output is fixed for a given version of a dataset. Improving the splitter later means re-running extraction on existing documents, not just re-querying — which is one reason segmentation quality gets checked early rather than patched after the fact.

← Back to the glossary