The network

Terse has three layers. The network, described here (NETWORK.md is exact), and the article format (ARTICLE.md, see "Writing an article") are the core: a client that implements those two is a complete Terse client, with or without the machine. The machine (SPEC.md and STATE.md) is an optional third layer that reuses them.

Clients exchange immutable blobs and signed feeds through relays: blob stores reachable over WebSocket. A client is anything that speaks the protocol for a person: a native reader, a machine host (which speaks it for the programs it runs), the command line tool. There are no messages between users and no direct links between clients yet.

Blobs

A blob is 1 to 1,048,576 bytes. Its ID is the BLAKE3-256 hash of its bytes: 32 bytes, written as 64 lowercase hex digits. A program image is a blob, so an image ID is a blob ID.

A client keeps a local blob store (a cache) and uses it when it can. Otherwise it fetches the whole blob from its relays, checks its hash, stores it, and only then uses it. A blob that fails the check counts as not found on that relay. A put stores the blob locally and sends it to every connected relay. Clients never trust relays: every blob is verified on receipt.

A blob reached without an entry (by a link) tells what it is by its first four bytes: TERS is a program image, TDOC a document (data for a program), anything else an article.

Relay protocol

All messages are binary WebSocket messages; multi-byte integers are little-endian. req is a 32-bit number chosen by the client and echoed in the reply; replies may come in any order.

MessageDirectionLayout
Getclient to relay01h, req (4), blob ID (32)
Get replyrelay to client81h, req, status (1), then the blob if status is 1
Putclient to relay02h, req, the blob's bytes
Put replyrelay to client82h, req, status, then the ID (32) if status is 1
Publishclient to relay03h, req, entry
Publish replyrelay to client83h, req, status
Subscribeclient to relay04h, req, feed key (32), after (4)
Subscribe replyrelay to client84h, req, status, then highest (4) if status is 1
Unsubscribeclient to relay05h, req, feed key (32)
Unsubscribe replyrelay to client85h, req, status
Findclient to relay06h, req, scope (1), n (1), query (n)
Find replyrelay to client86h, req, status, count (1), then per result: length (2), entry
Peerrelay to relay07h, no body, no reply
Lookupclient to relay08h, req, feed key (32), kind (1), n (1), name (n)
Lookup replyrelay to client88h, req, status, then the entry if status is 1
Entryrelay to client90h, entry
StatusMeaning
1done
2not found (get, lookup)
3refused (a put: too large, empty, store full or malformed; find, lookup: a bad request or asked too often)

A find the relay carries out is answered with status 1 and its results, none if nothing matches.

The relay computes a put's ID itself. It must refuse blobs over 1,048,576 bytes and may limit its store. It closes the connection on a text message or an unparsable message of a known type, and ignores unknown types and replies or pushed entries from a non-peer.

Feeds

A feed is a numbered series of small signed entries from one author, each pointing to a blob. A feed is identified by an Ed25519 public key (32 bytes, 64 lowercase hex digits); a valid key is a curve point that is not of small order. Only the author's client holds the secret key.

The entry

OffsetSizeField
032feed key
324seq: 1 for the first entry, then increasing
364time: Unix seconds, as claimed by the author
401kind: 1 article, 2 profile, 3 program, 4 document; 0 and 255 reserved; 5 to 254 not yet defined
413size of the blob in bytes
4432blob ID
7632reply-to: the blob ID this entry responds to, or all zero
1081n: title length, 0 to 120
109ntitle, UTF-8
109 + n64signature

The signature is Ed25519 over the 13 ASCII bytes terse-entry-0 followed by bytes 0 to 108 + n of the entry. An entry is at most 293 bytes.

An entry is valid if: n is at most 120 and the entry's length matches it; the signature verifies under the feed key with strict verification (small-order keys and non-canonical signatures are rejected); seq is at least 1; the kind is not 0 or 255; the size is 1 to 1,048,576 (1 to 4,096 for a profile); the title is valid UTF-8; and for a profile or a program the title is a valid display name. Clients and relays discard invalid entries. If two different valid entries have the same feed key and seq, the one received first is kept. seq values need not be consecutive, and the time is not checked.

KindEntryTitleBlob
1articleany titleUTF-8 text (see "Writing an article")
2profilethe feed's display namethe profile document
3programthe program's name, a display namea Terse image
4documentany titlea document (starts with TDOC)

Program and document entries are valid whatever their blob holds; that is found out on use. A feed's current version of a program is its valid program entry with that name and the highest seq the client or relay holds. A program's identity is (feed key, name).

An entry of a kind not listed here (5 to 254) is valid if it meets the rules above (the kind-specific ones apply only to their kinds). Relays pass it on like any other, so that kinds a later revision adds reach clients that know them; a client keeps or skips it without error, and neither downloads, lists in the timeline nor opens it (terse feed list shows [kind 9: title]).

Relay feed messages

Clients and feeds

Feed addresses

A key says which feed, not where it is: a client that asks only its own relays finds nothing of a feed published elsewhere. A feed address adds up to two hints, relays that should hold it: <key>@relay.example.org or <key>@relay.example:9000,[2001:db8::7]. A hint is a host (a name, an IPv4 address, an IPv6 one in brackets) and a port (7878 if left out), at most 80 bytes, meaning ws://host:port. A client refuses a malformed address; in an article a malformed hint list leaves the link the key alone. NETWORK.md 4.9 has the grammar.

A client keeps the hints with the subscription. If none of its own relays holds an entry of the feed (each answers highest 0), it follows the feed on the hinted relays too, as feed relays (below: same address checks, same switch, same cap), until the feed's profile has relay lines. Its user's own address takes its hints from their profile's relay lines, then the client's relays, public ones only; with none, only people on the same relays find the feed.

Profiles, display names and nicknames

A profile is a kind 2 entry. Its title is the feed's display name; its blob is the profile document. The profile in force is the feed's valid profile entry with the highest seq.

A display name is 1 to 48 bytes of UTF-8 with no control characters (U+0000 to U+001F, U+007F to U+009F) and no character of the Unicode White_Space property (U+0020, U+00A0, U+3000 and others) first or last.

The profile document is UTF-8 text of at most 4,096 bytes, lines ending with 0Ah: header lines field: value, then optionally an empty line and a free-text description to the end. A header line without : makes the document invalid. Unknown fields are ignored.

FieldValue
listedyes: search may find the name; anything else, or no line, means not listed
relaya ws:// or wss:// URL of a relay where the author publishes; 0 to 4 lines; readers take the first four that are URLs

A client writing its user's profile adds a relay line for each of its non-loopback relays. Names are self-declared and not unique, so a feed's name is always shown with its key or a short form of it (first and last six hex digits). A nickname is a name the user gives a feed (same rules), kept in the client's configuration, never sent anywhere, and shown in preference to the declared name.

A find's scope is 1 (feeds by display name), 2 (article and document titles) or 3 (program names); the query is 1 to 48 bytes of UTF-8. Another scope or a bad query is refused (status 3, count 0). The query matches if it occurs as a contiguous run of bytes in the name or title after A to Z are changed to a to z on both sides; nothing else is folded.

ScopeRelay returnsOrder
1the profile in force of each feed whose document says listed: yes and whose name matchesnames starting with the query first, then the rest; each group by increasing feed key
2article and document entries of feeds with a listed profile in forcenewest first by the order the relay stored them
3program entries, current versions only, of feeds with a listed profilenewest first by store order

A relay returns at most 16 results from what it holds (a profile document it lacks or cannot parse counts as not listed), and may refuse a connection that searches too often.

Clients ask all their relays and merge: they drop invalid entries, ones of the wrong kind and ones that do not match, and keep one per feed and seq. Scope 1 keeps the highest seq of each feed, ordered as above; scopes 2 and 3 go by rank (each relay's first result, then each one's second, and so on), and scope 3 drops a program another result has a higher seq of. The first 16 are kept. NETWORK.md 4.7 has the rest.

Lookup

A lookup's kind must be 3 (program) and the name a valid display name, else it is refused (3). The relay answers with the current version it holds of that feed's program (1), or 2 if none, listed or not; it may refuse a client that asks too often. A client asks all connected relays, keeps the valid matching entry with the highest seq (stored, not subscribed), then gets the image (a machine host, when a shell starts a program it holds no entry for).

Several relays

A client's relays are those it is configured with (at most eight) and, for each subscribed feed whose profile in force has relay lines, up to two of those not already among its own (else up to two hints of its address, above): the feed relays, which may be turned off. A client keeps at most 16 relay connections. It ignores a relay line that names localhost (or a name under it) or a loopback, unspecified, private or link-local address (NETWORK.md 5.1 lists the ranges), does not connect if a name resolves to one, and connects only to the addresses it checked. A failed or dropped connection is retried after 1 s, doubling up to 30 s. Requests go to the connected relays and wait only while none is connected.

Put and publish go to every connected relay and succeed when one accepts (an article's entry once a relay has its blob); subscribe goes to every connected relay that serves the feed (the client's own and the feed's relays); find and lookup go to all. A get asks the feed's relays first, then the rest two at a time, and is not found only when every connected relay said so or failed.

Relays that mirror each other

A relay may be given other relays as peers: it connects to each and sends Peer (07h). It passes every entry, and every blob up to 16,384 bytes, that it newly stores to its other peers, subscribes on them to the feeds its clients follow and asks them for blobs it lacks. NETWORK.md 5.2 has the details.

What a reading client must and may do

A reading client shows people the articles of the feeds they follow: a native reader, or a machine host running the reader program.

It must hold blobs by ID and verify each one, talk to at least one relay, verify entries, subscribe and unsubscribe by key or address, keep the timeline in arrival order, show articles by ARTICLE.md and show every name with its key. It may publish, set a profile and nicknames, search, use several relays and hints, download automatically and prefetch.

Entries of kinds it does not handle (programs and documents, for a client without the machine; kinds not yet defined, for every client) are kept or skipped without error and shown, if at all, as something it cannot open; so are links to programs, and links to blobs that turn out to be images or documents. "Writing a reader without the machine" walks through building one.

For machine hosts

Programs never see this protocol, nor any secret key: they use the content device, and the host speaks the protocol for them. A program may ask to subscribe, unsubscribe, publish, set the profile or a nickname, and the host asks the user first; the timeline above is what the timeline operation reads. A host may start a program by image ID, checking that the blob is a valid image. Completions arrive as events (events) and cost budget (charges).