Content: blobs, requests and statuses (device 4)

The content device, slot 4, version 0, is how a program reaches data outside its machine: blobs fetched from or published to the network, the user's feeds, and small named records kept on the host. Everything goes through one call, submit (400h), and one 52-byte request block. The program never hashes, verifies or transports anything; the host does all of that and only ever hands the program bytes that match the hash it asked for. How the host finds blobs is in the network.

An image gets the device by listing slot 4, version 0 in its header's device requirements (terse pack ... --device 4:0). If the image does not list it, submit returns FFFF_FFFFh and does nothing else. At reset no request is pending.

Blobs

A blob is 1 to 1,048,576 bytes, named by its ID: the BLAKE3-256 hash of its bytes (32 bytes). A program image is a blob, so an image ID is a blob ID. A blob is immutable; a different content is a different blob.

The request block

A request is a 52-byte block that must lie wholly in RAM, at any alignment. Multi-byte fields are little-endian. In C it is content_req. The op table below says which fields each op uses; "submit" means the device reads the field when the request is submitted, "completion" that it writes it when the request completes.

OffsetSizeFieldWritten by
01op: the operation, 01h to 11hprogram
11statusdevice
22reserved, ignorednone
432idboth
364offsetprogram
404lengthboth
444addrprogram
484sizeboth

What each field means, per op:

FieldOpMeaning
idget, publishthe blob ID
idputthe blob ID, written on completion
idsubscribe, unsubscribe, set nicknamea feed key
ididentitythe host's feed key, written on completion
idsave, load, deletethe record's name
offsetget, loadbyte offset into the blob or record
offsetsubscriptions, timeline, names, programs, recordsindex of the first record wanted
offsetfindthe scope, 1 to 3
offsetpublishthe entry's kind: 0 or 1 (article), 4 (document)
offsetset profilebit 0: listed; other bits ignored
lengthget, load, the five record lists, findbuffer size, 1 to 16,384; on completion, bytes delivered
lengthpublishthe title's length, 0 to 120, read at submit
lengthset profilethe name's length, 1 to 48, read at submit
lengthset nicknamethe name's length, 0 to 48, read at submit
addrget, load, record lists, findthe buffer, length bytes wholly in RAM
addrput, savethe data, size bytes wholly in ROM or wholly in RAM
addrpublishthe title, UTF-8, wholly in ROM or wholly in RAM
addrfindalso the query, size bytes in RAM
addrset profile, set nicknamethe name, wholly in ROM or wholly in RAM
sizeget, loadthe blob's or record's size, written on completion
sizeput1 to 1,048,576, read at submit
sizesave0 to 65,536, read at submit
sizerecord listsnumber of records that exist, written on completion
sizefindquery length 1 to 48 at submit; number of results on completion
sizepublish, set profilethe new entry's seq, written on completion

Fields an op does not use are ignored.

Operations

opOperationDetails
01hGetbelow
02hPutbelow
03hSubscriptionsfeeds
04hTimelinefeeds
05hSubscribefeeds
06hUnsubscribefeeds
07hPublishfeeds
08hIdentityfeeds
09hNamesfeeds
0AhFindfeeds
0BhSet profilefeeds
0ChSet nicknamefeeds
0DhProgramsfeeds
0EhSavenamed records
0FhLoadnamed records
10hDeletenamed records
11hRecordsnamed records

Get (01h) asks for up to length bytes of blob id starting at offset, delivered to addr. On success length is the number of bytes delivered (fewer if the blob ends first) and size is the blob's size. A get whose offset is at or beyond the end of the blob completes with 01h, length 0 and size the blob's size.

Put (02h) stores the size bytes at addr as a blob, on this host and on its relays. The bytes are copied at submit, so the program may reuse the memory at once. On success id is the blob's ID.

Submit

a7CallArgumentsResult
400hsubmita0 = address of a request blocka handle 1 to 8, or 0 if rejected

The device takes these steps in order:

The device does not look at the other pending requests: a block already pending may be submitted again, and then both requests write that block when they complete (the higher handle last). A data range of length 0 is valid at any address.

So a 0 result with status 04h means a bad request, and a 0 result with the block untouched means the block was not in RAM or all eight handles were in use. Check the block's own status before the submit if you need to tell them apart, for example by setting it to FFh first.

Completion

A request never completes in the frame it was submitted in. At the start of a later frame, before the program runs, the host may complete any pending requests. Completing one:

OpOn success, also writes
get, loadthe bytes to addr; length and size
subscriptions, timeline, names, programs, records, findthe records to addr; length and size
putid
publish, set profilesize (the entry's seq)
identityid
subscribe, unsubscribe, set nickname, save, deletenothing more

In every case, success or not, the device writes status, frees the handle and queues a content-done event (type 05h: a = handle, b = status; see events). If the event queue is full the event is dropped but the block is still updated, so a program that polls status never misses a completion. A failed request writes only status.

The device writes to the block at the address given at submit, whatever the program has done with that memory since; keep the block (and any result buffer) alive and unused until the request completes. Completions in one frame are applied in increasing handle order.

After a restore. A machine restored from a snapshot has the requests that were pending when the snapshot was taken, and no host will carry them out. Each completes with status 03h in the machine's first frame after the restore, in increasing handle order, before any other completion of that frame; so does a request that was waiting for the user's consent. Nothing else of the block is written.

Status values

ValueC nameMeaning
00hSTATUS_PENDINGpending
01hSTATUS_DONEdone
02hSTATUS_NOT_FOUNDnot found
03hSTATUS_FAILEDfailed: network error or timeout (the host decides when to give up), a host that cannot keep records, a restored request
04hSTATUS_BAD_REQUESTbad request
05hSTATUS_REFUSEDrefused: the user said no, the host does not permit it, the host does not provide the operation (reading profile), or the relays refused a find
06hSTATUS_NO_ROOMno room: a save would exceed the owner's limits

Determinism

Completions (which handle, in which frame, with what status, size and bytes) are inputs to the machine, like events and the clock. A host that records them per frame can replay a run exactly, without the network or the records directory. A request waiting for the user's consent is simply pending; the answer reaches the machine only as its completion. The failed completions after a restore are inputs like any other.

Documents

A document is a blob that says which program reads it. It starts with a header:

OffsetSizeField
04magic 54h 44h 4Fh 43h (TDOC)
41version: 0
532the format owner's feed key; all zero for a built-in program's format
371n: the format name's length, 1 to 48
38nthe format name, a valid display name
38 + nrestthe data, in the format's own layout (possibly empty)

The header is valid if all of it is present and every field is as above (TDOC_HEAD is 38). The format (owner key, name) is the identity of the program that defines it: a shell opens a document by starting that program with the document as its argument. A blob that starts with TERS is an image; a blob with neither is an article. A document is published as an entry of kind 4.

Charges

WhatCharge
any submit, also a rejected one4, plus the ecall
an accepted put or save (its size bytes), publish (its title), find (its query), set profile or set nickname (its name)1 per 4 bytes, rounded up, at submit
bytes a get, load, record list or find writes to addr1 per 4 bytes: all of a frame's completions added up, rounded up once

The delivery charge is taken from the budget of the frame the completions arrive in, before the program runs; the reference machine adds up the bytes of all that frame's completions and rounds once. A rejected submit copies nothing, and the fields a completion writes to the block are free. See charges.

State

The machine's state (used for the state hash and snapshots) holds, if the image lists device 4, one 53-byte record per handle 1 to 8:

OffsetSizeField
01op of the pending request, or 00h if the handle is free (then all else is zero)
14address of its request block
532id as read at submit (get, subscribe, unsubscribe, publish, set nickname), the record name (save, load, delete), the put's blob ID; else zero
374offset as read (reads, find, publish kind); set profile's listed bit; else zero
414length as read (reads, find, publish, set profile, set nickname); else zero
454addr as read (get, load, record lists, find); else zero
494size as read (put, save, find's query length); else zero

The copied data itself (a put's bytes, a title) is not part of the state.

Example: a get

Fetch the first 1,024 bytes of a blob and wait for them.

#include "terse.h"

static content_req req;
static uint8_t buf[1024];

void fetch(const uint8_t *id)
{
    terse_event ev;
    uint8_t h = content_get(&req, id, 0, sizeof buf, buf);
    if (h == 0)
        return; /* req.status 04h: bad request */
    for (;;) {
        while (terse_event_pop(&ev))
            if (ev.type == EV_CONTENT_DONE && ev.a == h)
                goto done;
        terse_wait_frame();
    }
done:
    if (req.status == STATUS_DONE)
        term_write(buf, req.length); /* req.size: blob size */
}

content_get fills the block and calls content_submit. A program with other work polls req.status or handles the event in its main loop instead of waiting. Note that content_submit returns uint8_t, so on an image without device 4 the FFFF_FFFFh result reads as 255, not 0.