The investigation file format

One investigation in one file: the graph, the material written from it, and the verification tasks it was run with. A person creates the file with "Export → Workspace", and the same person decides who gets it.

This document describes the format in words. The machine half is schema/counterattack-workspace-1.schema.json, a JSON Schema of draft 2020-12. The schema is written in English: it is read by programs, and by people who do not read Russian. The two descriptions are held together by tests (tests/format-schema.test.mjs), so they cannot drift apart silently.

Why the format is published. So that another newsroom can check our file with their own tools without installing our product. It is also a commitment: a published format cannot be changed arbitrarily, which is why it is published only now — after the revision log and the transfer have settled.

Two forms of one file

.catk is an ordinary ZIP archive: the recipient opens it with any archiver and reads the documents even without CATK. It is written without compression, because PDFs and scans are already compressed.

Either form can be encrypted with a passphrase. The file then begins with the line CATKENC1, followed by a header in plain text (algorithm, iteration count, salt, nonce) and the ciphertext. The header is readable by anything, so whoever finds the file understands what it is rather than deciding it is corrupt. There is no back door: a forgotten passphrase means the file cannot be opened.

A publication is the same file with fewer records

A published material is a file, not a page: the same format field, the same version, the same schema, the same parsing. Inside is one material and only what it stands on: the records it cites and links to, the relations between them, their evidence, documents with hashes, signatures and closed verification tasks. Not one other record of the investigation. A relation travels only when both its ends do: otherwise the file would keep a line leading nowhere.

A publication is recognised by its publication field:

• articleId and title — which material was published;
• standsOn — the records it stands on, each with a quoted flag: cited in the text, or merely linked to the material;
• omitted — what the file does not carry, and why, in words. An omission nobody mentions is read by the reader as a field that never existed.

Why the coordinates are published, even though they are a place on a map. latitude, longitude, areaRadiusKm and the area are part of the signature's fingerprint. A record without them is no longer the record the signature stands under: every signature would become "under a previous version", the fourth verification step would go dark, and the file would show fewer steps than there are — lying in favour of doubt. The decision not to show a place is taken before signing, not at publication.

The bytes of documents do not travel by default (attachments.included: false): names and hashes let a reader check the chain, while handing over the case materials themselves is a separate decision for each publication. Saying yes to it means .catk, the same second form of the file an investigation has.

The register of corrections is in the file, not only on the page

A corrected publication carries publication.corrections — a list of entries, each with three fields: date (yyyy-mm-dd), what was corrected exactly, and who demanded it. The product does not write an entry missing any of the three; it refuses to build the file and names the missing field. "Corrected on 14 September" without what was corrected looks like a report and says nothing.

The list is cumulative and oldest first: every later version carries all the earlier entries. A reader sent only the latest file has to see the whole history of edits, not the last one of them.

The fields are absent until something has been corrected. An empty list would claim that no correction was ever needed, while an absent field claims nothing — and publications built before the register existed stay files of the same shape. In the same way a record has no eventDate when nobody set a date.

The earlier file is superseded, not overwritten

publication.revision is the version number and publication.supersedes is the version this one replaced: {revision, file, sha256}. The earlier file keeps its own address and its own bytes; the new one names it and its hash. A downloaded file cannot be recalled, so replacement at the same address is not provided for at all, and the hash is what lets a reader check that the earlier version was not quietly rewritten.

The chain of versions is read from the files themselves: each names its predecessor. There is no second list of versions anywhere, so there is nothing for it to disagree with.

A correction legitimately lowers verification. Correct the name, the evidence, the sources, the dates, the place or a document, and the record is no longer the one the signature stands under: signatures become "a previous version" and the fourth step goes dark. That is not a fault in the file but the truth about it — the corrected record is signed again by a second person. The full editorial rule is in CORRECTIONS.md.

What the file must have

Three fields without which there is nothing to read:

• format — always "counterattack-workspace". The file names itself; the reader must not guess from the shape.
• version — an integer, currently 1. This is the version of the format, not of the product.
• graph.nodes — the list of records. It may be empty, but it must be a list.

Everything else is optional. A record, a relation, a cluster, a material and a task require only an id, and it cannot be an empty string: a reference to an empty id leads nowhere. A relation requires both ends: a relation with one end cannot be read.

A version newer than yours is a refusal, not a guess

A reader that knows version 1 must refuse a file of version 2 rather than read it optimistically. The file replaces the workspace as a whole, and an unfamiliar field here means loss, not decoration.

The reverse does not hold: a version 2 reader must still open a version 1 file.

The schema says the file is readable, not that the investigation is good

This is the format's central decision, and it explains almost all the others.

A refusal happens only for what cannot be read: not our format, an unknown version, a graph that is not a list, a record without an id, a relation without an end. Everything else is a line in a report, not a refusal.

That is why the schema has almost no enum. A risk level is written as a string, although the product knows six values and draws each in its own colour. The reason is simple: a file may come from a newer version or from another system through a bridge, and the product opens and shows an unfamiliar value in the colour "unknown", while the editor keeps it as a separate marked option. A schema that rejected such a file would diverge from the product, and an outsider would get a refusal where we ourselves open the file.

The known values are named in the description of each field. That is the difference between "we do not expect this" and "this is not allowed".

What the easily misread fields mean

evidence and sourceIds stand side by side. The first field holds a string the analyst wrote: a link, a document number, a note. The second holds references to records of type "Source". Pointing at a record is stronger than writing a string: several records can point at one source. The product will not turn someone's string into a record by guessing: that would be invented evidence.

confidence is the analyst's opinion. In the interface the field is labelled "Analyst's estimate". Verification is counted from the record itself (source named, document attached, verification task closed, countersigned by a second person) and is not stored in the file: it is recomputed, so it cannot go stale.

confidence may be absent, and that means there is no estimate. An opinion nobody stated is as invented as a number computed from nothing. To whoever reads the file: do not put a default in its place. Zero does not count as absence: a person typed it.

place, latitude and longitude are independent. The name is written by the analyst; the coordinates are set by the same analyst. CATK does not turn a name into coordinates or coordinates into a name: either direction is invented evidence. Hence a legitimate state: a place is named, there are no coordinates. Coordinates come as both or neither.

areaRadiusKm tells a point from an area. Empty and zero mean the same thing: a point. A point claims "exactly here", which a source usually does not say: "the port of Ruse" and "within the city" are areas. FollowTheMoney has no property for the radius, so it travels in the counterattack block, as do the source records a record or relation points at (sourceIds) and the analyst's estimate of a relation.

area on a record and route on a relation are the shape of a place the analyst drew or wrote. An area is a GeoJSON Polygon with one closed ring, a route is a LineString, and on the map the relation's line follows it instead of a straight line. GeoJSON puts longitude first, the reverse of the latitude and longitude fields: a tool that reads the pair the other way round puts the Danube in the Indian Ocean. The analyst writes the shape: nothing derives it from the place name, the record type or a gazetteer. A record of type Person has no area, and a relation with such a record has no route: the product does not build a person's trajectory (SCOPE.md). A shape that arrives with them in a file is kept and not drawn. So is a shape that cannot be read: a border crossing itself, an area with fewer than three points, more than 500 points. A second analyst's signature covers the area; the route lies on a relation and no signature covers it. Through FollowTheMoney the shapes travel in the counterattack block. CSV carries no shape, as it carries no attachments.

An area's ring runs counterclockwise, as RFC 7946 asks: the product turns round a ring drawn clockwise and keeps its first vertex first. A shape from a GIS often comes wrapped in a Feature, with a bbox or with height as the third number of a pair. Such a shape is kept as it came, nothing is taken off it, and the map reads its geometry. Limits of the first version. An area is one ring, without holes. A shape across the 180th meridian is drawn the long way round the map. A route is made of straight segments in the map's projection, with no arcs along the curve of the earth.

eventDate is the date of the event. The time a record was created or edited never stands in for it.

evidenceStatus on a relation is a mark set by hand, and it is separate from whether anything is attached. The product names that difference rather than hiding it.

hash on an attachment is the SHA-256 of the content, in lower-case hexadecimal. It answers the question a name and a size cannot: is this the same file. An empty value means "not checked": the document was attached before this check existed, and a hash cannot be added after the fact, because the content could have changed in exactly that interval.

origin exists only on a borrowed record. A record that came from another system (Aleph, OpenAleph, any other) carries system, licence and, when known, dataset and url. The licence is mandatory: borrowing without it is the path by which a dataset "for non-commercial use only" ends up in a paid product.

The same is written in the evidence field too, as an ordinary sentence. An analyst reads the evidence field and does not read service fields, so a borrowed claim would otherwise look like a verification of one's own. A record made here has no origin field at all. That sentence alone is not evidence: a record or relation whose evidence field carries nothing else reads as having no evidence. It does pass the verification step "source named".

wikidataId is the Wikidata item the analyst linked the record to: Q and digits. It is set by the analyst's press in the dossier, saving the item's document or adding relations from it; the product never links a record to an item by a matching name. Only a linked record has the field. A value that is not an item number is kept as it is and means nothing. When present, the link is part of what a countersignature covers. Through FollowTheMoney it travels as that model's property of the same name.

signatures live inside the record. A second analyst's signature confirms one record in one exact form, it does not cover the whole file, and so it is stored with the record. Each carries its public key: the reader must be able to check it without asking anyone.

A signature means exactly one thing: someone holding this key saw this record in this form. It does not mean the record is true, and it does not prove who that person is — the name beside the key is written by the same person who holds the key. There is no certificate authority here, and we do not pretend there is one.

There are two fields of different kinds, and the difference matters. source is the canonical text that was signed: comparing it with the record, the reader learns whether the signature covers the current form, and no cryptography is needed for that. signature and publicKey answer a different question: is the signature genuine at all. Inside the product that question is settled at import, so the card compares only the text. For someone else's file there is no such guarantee: its author could have written anything into those fields, and the reader must check the signature themselves.

position goes into the file but not into the signature fingerprint. A position on the canvas belongs to the investigation, so it is stored with it. It stays out of the fingerprint: otherwise a signature would come off from dragging.

dueDate and due on an operation are not the same thing

dueDate is the day by which the task should be done, as an ISO 8601 date (YYYY-MM-DD). This is the field to compute with: today's queue, the "due today or overdue" tile.

due is the older free-text string: "Due Aug 27", "Legal review", "Due today". It stays in the format and is stored as written. The product neither parses nor rewrites it: "Aug 27" has no year, "Legal review" is not a date at all, and "Due today" means "today as of a day nobody recorded". Turning that into a date is only possible by guessing, and a guess in a date field becomes an invented fact.

Opening an older file, the product shows this string beside an empty date field and marks it as coming from another version. The date is set by a person.

blocks are the material, body is a derived copy

blocks is the material itself: an ordered list in which every block is either prose the author wrote or a reference to something the investigation already holds. A reference stores an identifier, so the material follows the record itself rather than a cast of it taken on the day it was written.

The known kinds are prose, node, relation, evidence, operation and gaps. The list is open. A reader that meets an unknown kind must keep the block as it is rather than drop it: the file may come from a newer version, and a dropped block is somebody else's material quietly damaged.

body is the plain text of the prose blocks. It is rewritten on every save and exists for whoever reads the file with older code or with other people's software. Editing body achieves nothing: the next save overwrites it from the blocks. A material without blocks (a file written before they existed) reads as a single prose block taken from body.

This is not the same as the dueDate and due pair above. There the second field cannot be derived from the first, and both stay real. Here body is derived from blocks in full, and is therefore declared a copy.

gapsHidden is the author's consent to hide, not silence

At the end of an exported document stands a section naming what is not established about the records it cites. The author may switch it off: a draft for a lawyer and a one-page summary are legitimate documents.

A section switched off does not vanish silently. In its place the document itself carries a line saying the author hid it. A setting lives with whoever prints the file and never reaches the reader; a line in the document does. The gapsHidden field is the trace of that decision in the file.

modelAnswers is what a model answered, not what is established

The field sits in the graph, and it is absent until a model has been asked anything. A record keeps its latest answer: when it was asked (askedAt), which model was asked (requested) and which one answered (servedBy), how it ended (outcome: an answer, a refusal by the classifier, an answer cut short or one that could not be read), what was sent (passages: the fields of the record and of its relations, in the text the model received), what passed the check (proposals) and how many were dropped (dropped).

A proposal enters proposals only if each of its quotations stands word for word in the field of passages it names. That checks the quotation, not the claim: whether the proposal is right is the analyst's decision. A proposal the analyst accepts becomes an edit of the record and lives on as one.

The field is not published: a model's answer is a proposal, not evidence. omitted says so.

The inventory of documents

The attachments field says what happened to the files:

• included — whether the bytes are attached.
• files — what travelled, with its path inside the archive.
• missing — what is named but not attached.

A document that was not in the store goes into missing rather than being passed off as delivered. Silence here is the worst outcome: a recipient who believes the documents are with them will cite what they never saw.

What the format leaves out on purpose

• Verification. It is recomputed from the record; storing it would let it drift from the data.
• Users and permissions. The investigation is stored on one machine; there is nowhere here to take an identity from.
• Signatures over the whole file. Record signatures are in the file: a second analyst's signature arrives as its own file and goes into the record. There is no signature over the whole file and there will not be: a signature confirms one record in a specific form, not the file as a whole.
• Keys. Neither for signing nor for encryption: a key that arrives together with the lock protects nothing.

How to check a file

npx -p ajv-cli@5 -p ajv-formats@3 ajv validate -s schema/counterattack-workspace-1.schema.json -d your-file.json --spec=draft2020 -c ajv-formats

The formats plugin is needed only by ajv: in strict mode it refuses to compile the schema on meeting date-time, although draft 2020-12 allows treating format as an annotation and not checking it. Any other JSON Schema 2020-12 implementation will do just as well, and some of them without plugins. That is the point of publishing: the file is checked with someone else's tools, not ours.

Or with the page "Check an investigation file" — /validator in the product itself. It reads the file in your tab, sends nothing anywhere, and answers to the point: what kind of file this is, whether it matches the schema, how many verification steps each record has passed, which documents did not arrive, whether the signatures are genuine, and which gaps are visible in it. The page comes in two languages: ?lang=en in the address opens it in English, ?lang=ru in Russian.

It checks signatures with cryptography, while comparing strings answers a different question. "The signature is genuine" and "the signature stands under the current form of the record" are different claims, and a file where the second holds and the first does not is forged.