The network
Terse has three layers. The network, described here (NETWORK.md is exact), and the article format (ARTICLE.md, see "Writing an article") are the core: a client that implements those two is a complete Terse client, with or without the machine. The machine (SPEC.md and STATE.md) is an optional third layer that reuses them.
Clients exchange immutable blobs and signed feeds through relays: blob stores reachable over WebSocket. A client is anything that speaks the protocol for a person: a native reader, a machine host (which speaks it for the programs it runs), the command line tool. There are no messages between users and no direct links between clients yet.
Blobs
A blob is 1 to 1,048,576 bytes. Its ID is the BLAKE3-256 hash of its bytes: 32 bytes, written as 64 lowercase hex digits. A program image is a blob, so an image ID is a blob ID.
A client keeps a local blob store (a cache) and uses it when it can. Otherwise it fetches the whole blob from its relays, checks its hash, stores it, and only then uses it. A blob that fails the check counts as not found on that relay. A put stores the blob locally and sends it to every connected relay. Clients never trust relays: every blob is verified on receipt.
A blob reached without an entry (by a link) tells what it is by its first four bytes: TERS is a program image, TDOC a document (data for a program), anything else an article.
Relay protocol
All messages are binary WebSocket messages; multi-byte integers are little-endian. req is a 32-bit number chosen by the client and echoed in the reply; replies may come in any order.
| Message | Direction | Layout |
|---|---|---|
| Get | client to relay | 01h, req (4), blob ID (32) |
| Get reply | relay to client | 81h, req, status (1), then the blob if status is 1 |
| Put | client to relay | 02h, req, the blob's bytes |
| Put reply | relay to client | 82h, req, status, then the ID (32) if status is 1 |
| Publish | client to relay | 03h, req, entry |
| Publish reply | relay to client | 83h, req, status |
| Subscribe | client to relay | 04h, req, feed key (32), after (4) |
| Subscribe reply | relay to client | 84h, req, status, then highest (4) if status is 1 |
| Unsubscribe | client to relay | 05h, req, feed key (32) |
| Unsubscribe reply | relay to client | 85h, req, status |
| Find | client to relay | 06h, req, scope (1), n (1), query (n) |
| Find reply | relay to client | 86h, req, status, count (1), then per result: length (2), entry |
| Peer | relay to relay | 07h, no body, no reply |
| Lookup | client to relay | 08h, req, feed key (32), kind (1), n (1), name (n) |
| Lookup reply | relay to client | 88h, req, status, then the entry if status is 1 |
| Entry | relay to client | 90h, entry |
| Status | Meaning |
|---|---|
| 1 | done |
| 2 | not found (get, lookup) |
| 3 | refused (a put: too large, empty, store full or malformed; find, lookup: a bad request or asked too often) |
A find the relay carries out is answered with status 1 and its results, none if nothing matches.
The relay computes a put's ID itself. It must refuse blobs over 1,048,576 bytes and may limit its store. It closes the connection on a text message or an unparsable message of a known type, and ignores unknown types and replies or pushed entries from a non-peer.
Feeds
A feed is a numbered series of small signed entries from one author, each pointing to a blob. A feed is identified by an Ed25519 public key (32 bytes, 64 lowercase hex digits); a valid key is a curve point that is not of small order. Only the author's client holds the secret key.
The entry
| Offset | Size | Field |
|---|---|---|
| 0 | 32 | feed key |
| 32 | 4 | seq: 1 for the first entry, then increasing |
| 36 | 4 | time: Unix seconds, as claimed by the author |
| 40 | 1 | kind: 1 article, 2 profile, 3 program, 4 document; 0 and 255 reserved; 5 to 254 not yet defined |
| 41 | 3 | size of the blob in bytes |
| 44 | 32 | blob ID |
| 76 | 32 | reply-to: the blob ID this entry responds to, or all zero |
| 108 | 1 | n: title length, 0 to 120 |
| 109 | n | title, UTF-8 |
109 + n | 64 | signature |
The signature is Ed25519 over the 13 ASCII bytes terse-entry-0 followed by bytes 0 to 108 + n of the entry. An entry is at most 293 bytes.
An entry is valid if: n is at most 120 and the entry's length matches it; the signature verifies under the feed key with strict verification (small-order keys and non-canonical signatures are rejected); seq is at least 1; the kind is not 0 or 255; the size is 1 to 1,048,576 (1 to 4,096 for a profile); the title is valid UTF-8; and for a profile or a program the title is a valid display name. Clients and relays discard invalid entries. If two different valid entries have the same feed key and seq, the one received first is kept. seq values need not be consecutive, and the time is not checked.
| Kind | Entry | Title | Blob |
|---|---|---|---|
| 1 | article | any title | UTF-8 text (see "Writing an article") |
| 2 | profile | the feed's display name | the profile document |
| 3 | program | the program's name, a display name | a Terse image |
| 4 | document | any title | a document (starts with TDOC) |
Program and document entries are valid whatever their blob holds; that is found out on use. A feed's current version of a program is its valid program entry with that name and the highest seq the client or relay holds. A program's identity is (feed key, name).
An entry of a kind not listed here (5 to 254) is valid if it meets the rules above (the kind-specific ones apply only to their kinds). Relays pass it on like any other, so that kinds a later revision adds reach clients that know them; a client keeps or skips it without error, and neither downloads, lists in the timeline nor opens it (terse feed list shows [kind 9: title]).
Relay feed messages
- The relay verifies an entry before storing it. Publishing an entry the relay already holds succeeds; publishing a different entry with a
seqit already holds is refused. - After a successful subscribe the relay sends the reply with
highest, the highestseqit holds for the feed (or 0); then every entry it holds withseqgreater thanafter, in increasing order; then each new entry as it is published, until the client unsubscribes or disconnects. Each entry is sent once. - The client has caught up with the feed when it has received an entry whose
seqishighestor greater (valid or not), or at once ifhighestis not greater thanafter. - Unsubscribing from a feed that was not subscribed succeeds.
- A relay may keep only the most recent entries of a feed (at least 256), but always keeps the feed's latest profile entry, however old.
Clients and feeds
- The client keeps a list of subscribed feeds and every valid entry received for them. On connecting to a relay it subscribes to each feed with
after= the highestseqit holds. - Subscribing (when the user asks) adds the feed and subscribes at once when connected; for a feed it holds nothing of,
afteris 0, so the relay sends all it holds. Unsubscribing removes the feed and unsubscribes on the connection. Only valid keys are subscribed to. - Automatic download: for each entry of a subscribed feed whose size is at most 16,384 bytes (a user setting), the client gets the blob unasked, profiles and documents included (not programs, below, or kinds not listed above). A blob whose length differs from the entry's size is not associated with it. Larger blobs are fetched on request.
- Program download (a client that runs programs): once a feed is caught up, and when a program entry arrives, the client gets the image of the current version of each of its programs it lacks (any size up to the largest image), never older ones. The user may turn this off.
- Prefetch: when automatic download stores an article (kind 1), the client gets the first 8 distinct article links in it whose blobs it lacks, within the automatic limit; never program links or images, and prefetched blobs are not scanned. The user may turn this off.
- Arrival order: the client records the order in which it first stored each entry, across restarts; it is the order of its timeline (kinds 1 and 4), newest first. Unsubscribing hides a feed's entries from the timeline; subscribing again puts them back where they were.
- Publishing an article or document: the client puts the blob, then signs and publishes an entry with the next
seq(one more than the highest it holds or has used for its own feed). The entry is stored like a received one; if the client subscribes to its own feed, it is in the timeline at once.
Feed addresses
A key says which feed, not where it is: a client that asks only its own relays finds nothing of a feed published elsewhere. A feed address adds up to two hints, relays that should hold it: <key>@relay.example.org or <key>@relay.example:9000,[2001:db8::7]. A hint is a host (a name, an IPv4 address, an IPv6 one in brackets) and a port (7878 if left out), at most 80 bytes, meaning ws://host:port. A client refuses a malformed address; in an article a malformed hint list leaves the link the key alone. NETWORK.md 4.9 has the grammar.
A client keeps the hints with the subscription. If none of its own relays holds an entry of the feed (each answers highest 0), it follows the feed on the hinted relays too, as feed relays (below: same address checks, same switch, same cap), until the feed's profile has relay lines. Its user's own address takes its hints from their profile's relay lines, then the client's relays, public ones only; with none, only people on the same relays find the feed.
Profiles, display names and nicknames
A profile is a kind 2 entry. Its title is the feed's display name; its blob is the profile document. The profile in force is the feed's valid profile entry with the highest seq.
A display name is 1 to 48 bytes of UTF-8 with no control characters (U+0000 to U+001F, U+007F to U+009F) and no character of the Unicode White_Space property (U+0020, U+00A0, U+3000 and others) first or last.
The profile document is UTF-8 text of at most 4,096 bytes, lines ending with 0Ah: header lines field: value, then optionally an empty line and a free-text description to the end. A header line without : makes the document invalid. Unknown fields are ignored.
| Field | Value |
|---|---|
listed | yes: search may find the name; anything else, or no line, means not listed |
relay | a ws:// or wss:// URL of a relay where the author publishes; 0 to 4 lines; readers take the first four that are URLs |
A client writing its user's profile adds a relay line for each of its non-loopback relays. Names are self-declared and not unique, so a feed's name is always shown with its key or a short form of it (first and last six hex digits). A nickname is a name the user gives a feed (same rules), kept in the client's configuration, never sent anywhere, and shown in preference to the declared name.
Search
A find's scope is 1 (feeds by display name), 2 (article and document titles) or 3 (program names); the query is 1 to 48 bytes of UTF-8. Another scope or a bad query is refused (status 3, count 0). The query matches if it occurs as a contiguous run of bytes in the name or title after A to Z are changed to a to z on both sides; nothing else is folded.
| Scope | Relay returns | Order |
|---|---|---|
| 1 | the profile in force of each feed whose document says listed: yes and whose name matches | names starting with the query first, then the rest; each group by increasing feed key |
| 2 | article and document entries of feeds with a listed profile in force | newest first by the order the relay stored them |
| 3 | program entries, current versions only, of feeds with a listed profile | newest first by store order |
A relay returns at most 16 results from what it holds (a profile document it lacks or cannot parse counts as not listed), and may refuse a connection that searches too often.
Clients ask all their relays and merge: they drop invalid entries, ones of the wrong kind and ones that do not match, and keep one per feed and seq. Scope 1 keeps the highest seq of each feed, ordered as above; scopes 2 and 3 go by rank (each relay's first result, then each one's second, and so on), and scope 3 drops a program another result has a higher seq of. The first 16 are kept. NETWORK.md 4.7 has the rest.
Lookup
A lookup's kind must be 3 (program) and the name a valid display name, else it is refused (3). The relay answers with the current version it holds of that feed's program (1), or 2 if none, listed or not; it may refuse a client that asks too often. A client asks all connected relays, keeps the valid matching entry with the highest seq (stored, not subscribed), then gets the image (a machine host, when a shell starts a program it holds no entry for).
Several relays
A client's relays are those it is configured with (at most eight) and, for each subscribed feed whose profile in force has relay lines, up to two of those not already among its own (else up to two hints of its address, above): the feed relays, which may be turned off. A client keeps at most 16 relay connections. It ignores a relay line that names localhost (or a name under it) or a loopback, unspecified, private or link-local address (NETWORK.md 5.1 lists the ranges), does not connect if a name resolves to one, and connects only to the addresses it checked. A failed or dropped connection is retried after 1 s, doubling up to 30 s. Requests go to the connected relays and wait only while none is connected.
Put and publish go to every connected relay and succeed when one accepts (an article's entry once a relay has its blob); subscribe goes to every connected relay that serves the feed (the client's own and the feed's relays); find and lookup go to all. A get asks the feed's relays first, then the rest two at a time, and is not found only when every connected relay said so or failed.
Relays that mirror each other
A relay may be given other relays as peers: it connects to each and sends Peer (07h). It passes every entry, and every blob up to 16,384 bytes, that it newly stores to its other peers, subscribes on them to the feeds its clients follow and asks them for blobs it lacks. NETWORK.md 5.2 has the details.
What a reading client must and may do
A reading client shows people the articles of the feeds they follow: a native reader, or a machine host running the reader program.
It must hold blobs by ID and verify each one, talk to at least one relay, verify entries, subscribe and unsubscribe by key or address, keep the timeline in arrival order, show articles by ARTICLE.md and show every name with its key. It may publish, set a profile and nicknames, search, use several relays and hints, download automatically and prefetch.
Entries of kinds it does not handle (programs and documents, for a client without the machine; kinds not yet defined, for every client) are kept or skipped without error and shown, if at all, as something it cannot open; so are links to programs, and links to blobs that turn out to be images or documents. "Writing a reader without the machine" walks through building one.
For machine hosts
Programs never see this protocol, nor any secret key: they use the content device, and the host speaks the protocol for them. A program may ask to subscribe, unsubscribe, publish, set the profile or a nickname, and the host asks the user first; the timeline above is what the timeline operation reads. A host may start a program by image ID, checking that the blob is a valid image. Completions arrive as events (events) and cost budget (charges).