Decifra
Codebook-driven text coding, validated against human coding
Document uploads, approximate token estimates, saved provider settings, evidence inspection and a single-server frontend are now implemented. Run decifra serve after building the frontend. See the repository’s docs/MVP_TEST_GUIDE.md for setup, testing and limitations. Older roadmap statements below describe the preceding alpha audit.
The problem
Turning messy text into structured data is a routine need: is this article about a protest? does this statement support or oppose a policy? Does this support ticket describe a billing issue or a bug? Today that mostly means someone reading every document by hand and filling in a spreadsheet against rules they wrote themselves — a codebook.
LLMs can automate a lot of that reading. But everyone who tries ends up reinventing the same pipeline from scratch — and, more importantly, most skip the part that actually makes automated coding trustworthy: checking whether the model followed your specific rules, or just fell back on its own generic idea of the concept.
What Decifra is
Decifra is a local-first tool: you author a codebook (a concept, its categories, definitions, examples, boundary notes), point it at a text corpus, and the software calls an LLM to fill in a structured results table — one category, one rationale, one quoted evidence span per document. Results record the prompt, model, provider mode and response (full CLI stdout or serialized parsed API JSON). Historical reporting and manual-edit provenance still need work; see the repository status report.
It is not a manual qualitative-coding tool — Taguette, QualCoder, and QualiLab already do that well. Decifra is closer to the inverse: it automates the mechanical categorization step so a human doesn’t have to read every row, while treating your own judgment — codebook design, model/parameter choices, and the decision of whether automated output is reliable enough to become real data — as non-negotiable and non-automatable.
What “automated” does not mean
Automation applies to one step: turning text into a category without a human reading every row. It does not remove you from anything else. You are still solely responsible for:
- designing the codebook — which categories exist, how each is defined, where the conceptual boundaries sit;
- choosing the pipeline — which corpus, which model, which provider;
- interpreting results; and
- deciding, via validation, whether automated coding is reliable enough to use as data.
The Codebook Editor is the most important screen in the software, not a form to get through before “the real part.”
Why validation is not a footnote
Halterman & Keith (2025) — and the same group’s 2026 follow-up — show that off-the-shelf LLMs frequently ignore a codebook’s specific operationalization and substitute their own generic notion of a concept instead: including a labor strike under “protest” even when the codebook explicitly excludes it, for example. A model that looks fluent is not the same as a model that followed the instructions.
Decifra ran this failure mode for real, not hypothetically. See Why Validation Matters for the numbers: an under-specified codebook produced results with essentially zero discriminating power on a third of a real test batch, and a hypothesis’s own scope condition got silently ignored. Enriching the codebook — not switching models — fixed both, and the before/after is verifiable, not just asserted.
Current status
Decifra is a functional alpha. The five workflow stages exist, but cost estimation and document-file import in the UI are still missing from the original MVP acceptance checklist. As of 2026-09-07:
- Corpus import — CSV/XLSX/pasted text in the UI; TXT/Markdown/DOCX/textual PDF through the backend.
- Codebook editor — structured form with a YAML preview after saving.
- Runs — kick off an extraction against a corpus + codebook, watch it progress, pick API-key mode (schema enforced via
instructor) or CLI mode (any installed CLI you already have a subscription for). - Results — browsable table, inline edits, CSV/XLSX/JSON export, and a backend GUIDE-LLM-shaped report scaffold whose historical provenance and validation text still need work.
- Validation — gold-label CSV import (QualiLab interop is available through the backend), Cohen’s kappa, precision/recall/F1 per category.
See How It Works for the pipeline end to end, and Architecture for the technical design and why it looks the way it does.
Source, issues, and full history: github.com/mancano-tales/decifra-text-as-data.