Artefact

artefact/ is the library in our codebase that owns all bundle reads and writes. Every pipeline step that changes a dataset goes through artefact; every read is served from a cached hydrated bundle.

Why it matters for your research. As a user you don’t interact with artefact directly, but it is the reason two concurrent analysis runs on the same dataset don’t corrupt the bundle. The writer lock is Redis-backed; readers operate on hydrated snapshots. The guarantees that make versioning meaningful are implemented here.

In Archēglyph. The implementation detail behind every mention of “bundle” or “versioning”. Reproducibility lives here.

Not to be confused with. The generic British-English noun “artefact” meaning a made thing — we use it in both senses deliberately.

How it fails safe. A bundle is only ever visible to a reader once every file inside it has been written and its hashes match the manifest.json the same way an exported snapshot checks its own manifest. If a pipeline step is interrupted partway — a worker crash, an out-of-memory kill during embedding — the version it was writing simply never gets published, and readers keep serving the last complete version. There is no state in which a half-written index becomes visible; the failure mode is “the run has to be retried,” never “the bundle is now inconsistent.”

Why a lock instead of a queue. artefact could have serialised every write through a single job queue, but that would mean an NER run and a re-embed on the same dataset always wait for each other even when they touch disjoint files. Instead the writer lock is scoped to one dataset’s bundle: two datasets’ pipelines run fully in parallel, and within one dataset, whichever run acquires the lock hydrates the current version, applies its change, and publishes the next version before releasing it. The run queued behind it then hydrates from that new version rather than the one it originally queued against, so it never has to be told to retry — it just sees fresher input than it expected.

What “hydrated” means here. A bundle on disk is a Tantivy directory, a zvec directory, and a metadata SQLite file — three different on-disk formats that have to agree with each other about which chunk is which. Hydrating a bundle means opening all three against the same version and holding them open together behind one handle, so a reader never sees the lexical index from one version answering a query against the vector index from another. Cached hydrated bundles are what make repeated reads of the same dataset cheap; the expensive part is the first open after a version changes, not any individual query against it.

Why this is a glossary entry at all. Every other term in this glossary describes a concept a researcher can reason about directly — a corpus, an embedding, a cluster. Artefact is here because “why didn’t my two analysis runs corrupt each other” is a question researchers do ask, even though the answer lives entirely in implementation rather than in the research question itself.

What changes between artefact versions. The library itself has shipped in two forms: an earlier one that kept file, document, and chunk storage rows in Postgres alongside the metadata, and the current one — the “Postgres-metadata-only” model — where file, document, and chunk storage moved fully into the per-dataset bundle and Postgres holds only metadata such as ownership, naming, and dataset-level settings. That move is what made a snapshot export complete in the first place: when the bundle held only part of a dataset’s state, exporting it and reopening it elsewhere could never be fully self-contained.

A note for anyone reading the source. The library is internal — there is no public API surface for artefact, and none is planned. It is documented here because understanding what guarantees a bundle does and doesn’t make (atomic version publishing, yes; automatic conflict resolution across two independently edited copies, no) helps explain the shape of features built on top of it, like why re-imported snapshots always become forks rather than merges.

← Back to the glossary