Emery Hemingway came up with an excellent hack for getting udev to automatically populate a dataspace with assertions describing the devices attached to a Linux system. As devices are plugged in, reconfigured, and unplugged, the system automatically creates, updates and removes files in /run/etc/syndicate/hardware describing each device.

The trick is to install a udev RUN rule to execute a script which uses udevadm info --json=short to write out a Preserves file. Preserves is, syntactically, a superset of JSON, so it’s possible to directly use JSON output from programs like this as Preserves.

A syndicate-server can then be configured to load all files in /run/etc/syndicate/hardware into a dataspace by placing something like

1
2
let ?devices = dataspace
<require-service <config-watcher "/run/etc/syndicate/hardware" { config: $devices }>>

in the server’s configuration.

udev configuration and script

The run rule, placed in a file named something like /usr/local/lib/udev/rules.d/50-syndicate.rules, should be something like this:

1
ACTION=="add|change|move|remove", RUN+="/usr/local/bin/udev-syndicate-rule.sh"

And the script that does the work of maintaining the Preserves files in response to trigger events from udev, placed in /usr/local/bin/udev-syndicate-rule.sh, should look something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
#!/bin/sh
# Script based on https://git.syndicate-lang.org/ehmry/syndev#assertions-from-udev

if [ -z "$DEVNAME" -o -z "$MAJOR" -o -z "$MINOR" ]
then
        exit
fi

HARDWARE_DIR=/run/etc/syndicate/hardware
PR_FILE=$HARDWARE_DIR/$MAJOR-$MINOR.pr

mkdir -p $HARDWARE_DIR

case "$ACTION" in
        remove)
                rm -f $PR_FILE
                ;;
        *)
                echo '<device ' > $PR_FILE
                udevadm info --json=short $DEVNAME >> $PR_FILE
                echo '>' >> $PR_FILE
                ;;
esac

After adding the rule and the script to your system, you will need to either reboot or run

1
udevadm control --reload-rules && udevadm trigger

to get things started.

Example output

After installing the script on my system, I see about 250 files appear in /run/etc/syndicate/hardware.

For example, if I pipe /run/etc/syndicate/hardware/13-78.pr through preserves-tool for pretty-printing, I see:

<device {
  "CURRENT_TAGS": ":power-switch:"
  "DEVNAME": "/dev/input/event14"
  "DEVPATH": "/devices/pci0000:00/0000:00:01.1/0000:01:00.0/0000:02:00.0/0000:03:00.1/sound/card1/input14/event14"
  "ID_INPUT": "1"
  "ID_INPUT_SWITCH": "1"
  "ID_PATH": "pci-0000:03:00.1"
  "ID_PATH_TAG": "pci-0000_03_00_1"
  "ID_PROCESSING": "1"
  "LIBINPUT_DEVICE_GROUP": "0/0/0:ALSA"
  "MAJOR": "13"
  "MINOR": "78"
  "SUBSYSTEM": "input"
  "SYSNAME": "event14"
  "SYSNUM": "14"
  "TAGS": ":power-switch:"
  "USEC_INITIALIZED": "8598957"
}>

A gatekeeper in Syndicate terminology is an object that “upgrades” long-lived cryptographic names into live references to some underlying object. There are analogous objects in CapTP. (In CapTP, gatekeepers might be built-in to the protocol rather than separate objects. I need to find out more.)

The syndicate-server program includes an implementation of the gatekeeper protocol. This post builds up a short example syndicate-server setup involving gatekeepers. (The following is based on this example config file from the syndicate-server git repository.)

The plan

We will create a TCP listener on port 9222, which speaks unencrypted protocol and allows interaction with the default/system gatekeeper, which has a single noise binding for introducing encrypted interaction with a second gatekeeper, which finally allows resolution of references to other objects.

The outer/default/system gatekeeper will not expose any functionality apart from upgrading to an encrypted, authenticated reference to the inner gatekeeper. All the real services we want to expose will be available via the inner gatekeeper.

The stanzas below are written in the syndicate-server scripting language. They can be placed in a .pr file, e.g. example.pr, and run with syndicate-server -c example.pr.

Creating an inner gatekeeper

First, build a space where we place bindings for the inner gatekeeper to expose.

let ?inner-bindings = dataspace

Next, start, secure, and publish the inner gatekeeper.

<require-service <gatekeeper $inner-bindings>>
? <service-object <gatekeeper $inner-bindings> ?inner-gatekeeper> [
  <bind <noise { key: #[z1w/OLy0wi3Veyk8/D+2182YxcrKpgc8y0ZJEBDrmWs],
                 secretKey: #[qLkyuJw/K4yobr4XVKExbinDwEx9QTt9PfDWyx14/kg],
                 service: world }>
   $inner-gatekeeper #f>
]

The <bind ...> assertion exposes the $inner-gatekeeper object via the outer gatekeeper. Clients have to speak the noise protocol and know the public key #[z1w/OLy0wi3Veyk8/D+2182YxcrKpgc8y0ZJEBDrmWs] to resolve a reference to the inner gatekeeper. The noise protocol itself sets things up to rule out man-in-the-middle and prove to the client it really is talking to the correct server.

You can use the syndicate-macaroon tool (e.g. syndicate-macaroon noise --service world --random) to generate fresh keys for use in noise bindings.

Exposing the outer gatekeeper over unencrypted TCP

Now, expose the outer gatekeeper to the world, via TCP. The system gatekeeper is a primordial syndicate-server object bound to $gatekeeper.

<require-service <relay-listener <tcp "0.0.0.0" 9222> $gatekeeper>>

Let’s imagine we are running on a host called syndicate.example; this hostname will be used in URLs later.

Binding a service to the inner gatekeeper

Finally, let’s expose some behaviour accessible via the inner gatekeeper.

We will create a service dataspace called $world, and we will require clients to hold a sturdyref to it. We will configure the inner gatekeeper to resolve the sturdyref to the $world dataspace.

let ?world = dataspace

We need to generate a strong name, a sturdyref, that we will bind to $world. For this, we can use the syndicate-macaroon tool.

Let’s choose the passphrase hello as our secret that no-one but the service itself should know.

Running syndicate-macaroon mint --oid a-service --phrase hello yields the sturdyref <ref {oid: a-service, sig: #[JTTGQeYCgohMXW/2S2XH8g]}>. That will act as a root capability for $world once we bind it at the inner gatekeeper, using the corresponding sturdy.SturdyDescriptionDetail:

$inner-bindings += <bind <ref {oid: a-service, key: #"hello"}> $world #f>

Configuration done!

That’s it! All together, here is the configuration we set up:

let ?inner-bindings = dataspace

<require-service <gatekeeper $inner-bindings>>
? <service-object <gatekeeper $inner-bindings> ?inner-gatekeeper> [
  <bind <noise { key: #[z1w/OLy0wi3Veyk8/D+2182YxcrKpgc8y0ZJEBDrmWs],
                 secretKey: #[qLkyuJw/K4yobr4XVKExbinDwEx9QTt9PfDWyx14/kg],
                 service: world }>
   $inner-gatekeeper #f>
]

<require-service <relay-listener <tcp "0.0.0.0" 9222> $gatekeeper>>

let ?world = dataspace

$inner-bindings += <bind <ref {oid: a-service, key: #"hello"}> $world #f>

Accessing the $world from some remote client

Now, we can hand out paths to $world involving an initial noise step and a subsequent sturdyref/macaroon step. Clients using such paths benefit from an encrypted, authenticated channel (the noise layer), and use of the $world service is authorized by possession of a correctly-signed sturdyref.

The gatekeeper.Route that a client might use would be something like:

<route [<ws "ws://syndicate.example:9222/">]
  <noise { key: #[z1w/OLy0wi3Veyk8/D+2182YxcrKpgc8y0ZJEBDrmWs], service: world }>
  <ref { oid: a-service, sig: #[CXn7+rAoO3Xr6Y6Laap3OA] }>>

There’s more that sturdyrefs can do: since they are based on Macaroons, they can be attenuated offline (!) to limit the kinds of things clients can assert at $world and the kinds of messages they can send to $world. The example config file gets into more detail.

The Syndicate ecosystem includes a few ways of gluing together dataspaces over a network.

To begin with, there was no authentication or authorization framework involved at all: programs just exchanged assertions and messages using an ancestor of the Syndicate protocol over a plain socket. There was no way of referring to entities/objects/actors at all: Syndicate at this point was data-only. (It turns out this is sufficient for a surprisingly large range of programs!)

Then, I figured out how to use ideas from Macaroons to introduce capabilities on the wire. This allows recovery of session-local object references from long-lived cryptographic names for use in a protocol session. Programs “upgrade” a name like

<ref {oid: a-service, sig: #[JTTGQeYCgohMXW/2S2XH8g]}>

to a real pointer they can use to send assertions and messages to.

Finally, I incorporated the Noise Protocol Framework, making it possible to use Syndicate to open encrypted, authenticated tunnels across untrusted links. The idea is to use a cleartext socket or websocket (or anything else!) to bootstrap communication, and then use assertions and messages related to Noise to open a Noise session across that foundation.

A small example

Here’s a small example from a real system I’m running.

I run a syndicate-server Docker container configured with, among other stanzas, the following service definition (these are fresh random keys, not the ones I’m using live):

let ?shared = dataspace
let ?sk = #[0LFU4CAWxrz0VVS9nY2uSc7EWJTusZUcOP0HUpkwSkE]
let ?pk = #[Y3NCy2ZPbmaH62fFI0sl0YOZmM7K7PhVTmRFmgCAgVA]
@<connect-using # <-- this is just a comment, really
  <route [<ws "ws://LOCATION.TO.MY.SERVER.WEBSOCKET/">]
   <noise {service: a-service key: #[Y3NCy2ZPbmaH62fFI0sl0YOZmM7K7PhVTmRFmgCAgVA]}>>
 >
<bind <noise { service: a-service, key: $pk, secretKey: $sk }> $shared #f>

Then I can securely access the $shared dataspace from a remote syndicate-server instance using a stanza like this:

? <resolve-path
   <route [<ws "ws://LOCATION.TO.MY.SERVER.WEBSOCKET/">]
    <noise {service: a-service key: #[Y3NCy2ZPbmaH62fFI0sl0YOZmM7K7PhVTmRFmgCAgVA]}>>
   _
   _
   <accepted ?sharedDs>>
<the-shared-dataspace-is $sharedDs>

A lightly tricky piece of the puzzle is that the resolve-path handler isn’t built-in to syndicate-server yet: for that part, I’m using a tiny node.js service until I have a chance to implement the resolution logic in Rust. Watch this space.

In the #spritely IRC channel, mala mentioned the multiaddr specification.

From the site, “Multiaddr is a format for encoding addresses from various well-established network protocols.” The aim is to avoid “[leaving] much to interpretation and side-band context”, and to allow people to “build applications that will work with network protocols of the future, and do not accidentally ossify the stack.”

Syndicate defines something similar (though rudimentary) in its transportAddress module and in its notion of a Route.

Semantics for Multiaddr

Multiaddr informally defines its semantics, and gives both a human-readable and a machine-readable concrete syntax.

Let’s borrow preserves and propose a better specification for multiaddr semantics:

• A multiaddr value (a Multiaddr) is a Sequence of protocol addresses. The sequence is implicitly understood to describe a layered protocol stack, with leftward addresses (the “bottom” of the stack) acting as substrate for rightward addresses.

• A protocol address (an Address) is a record with its label being a Symbol representing a protocol name (a ProtocolName), and its fields being zero or more protocol-specific Values.

• Each kind of Address must define a canonical form. Use of the canonical form is mandatory. For example, an ip6 address could define a rule that the associated string must always conform to IETF IPv6 canonical format.

For example, the multiaddr examples written

1
2
3
4
5
/ip4/127.0.0.1/udp/9090/quic
/ip6/::1/tcp/3217
/ip4/127.0.0.1/tcp/80/http/baz.jpg
/dns4/foo.com/tcp/80/http/bar/baz.jpg
/dns6/foo.com/tcp/443/https

could denote the Values

1
2
3
4
5
[<ip4 127 0 0 1> <udp 9090> <quic>]
[<ip6 "::1"> <tcp 3217>]
[<ip4 127 0 0 1> <tcp 80> <http "/baz.jpg">]
[<dns4 "foo.com"> <tcp 80> <http "/bar/baz.jpg">]
[<dns6 "foo.com"> <tcp 443> <https "/">]

Schema for Multiaddr

Much of this structure can be captured by a preserves schema definition. Given such a schema, the multiaddr text and binary syntaxes become special-purpose syntax for preserves Values conforming to the schema.

version 1 .

Multiaddr = [Address ...] .
Address = <<rec> @protocolName symbol @detail [any ...]> .

WellKnownProtocol =
/ <ip4 @a int @b int @c int @d int>
/ <ip6 @addr string>
/ <dns4 @dnsName string>
/ <dns6 @dnsName string>
/ <udp @port int>
/ <tcp @port int>
/ <quic>
/ <http @path string>
/ <https @path string>
/ @unknown Address
.

Side-conditions such as bounding the integers in an ip4 address to the range [0..255] are not currently expressible in preserves schema.

Also, these definitions are too simple, particularly in the case of HTTP(S), where much more of the structure of an HTTP url (username, password, path, query, fragment, etc.) can and should be parsed out.

Since last time, I’ve managed to shrink the boilerplate for experimenting with Syndicate in the browser a little further.

Save the following in an HTML file, and open it:

1
2
3
4
5
6
7
8
9
10
<!DOCTYPE html>
<script src="https://cdn.jsdelivr.net/npm/@syndicate-lang/browser-stdenv"></script>
<script type="syndicate">
  spawn named 'clicker' {
    field counter = 0;
    const b = new SyndicateHtml2.Widget(t => t`<button>clicks: ${counter.value}</button>`)
          .setParent('body')
          .on('click', () => counter.value++);
  }
</script>

You should be greeted with a click-counter button.

Here’s the same as a codepen:

See the Pen Untitled by Tony Garnock-Jones (@leastfixedpoint) on CodePen.

To make experimenting with Syndicate/JS a bit easier, I’ve just dusted off an old idea: getting the compiler from JavaScript with Syndicate DSL extensions down to plain JavaScript to run in the browser.

It’s (relatively) simple. Load a few modules, perhaps via a CDN:

1
2
3
4
5
<script src="https://cdn.jsdelivr.net/npm/@preserves/core/dist/preserves.js"></script>
<script src="https://cdn.jsdelivr.net/npm/@syndicate-lang/core/dist/syndicate.js"></script>
<script src="https://cdn.jsdelivr.net/npm/@syndicate-lang/compiler/dist/syndicate-compiler.js"></script>
<script src="https://cdn.jsdelivr.net/npm/@syndicate-lang/html/dist/syndicate-html.js"></script>
<script src="https://cdn.jsdelivr.net/npm/@syndicate-lang/compiler/dist/syndicate-browser-compiler.js"></script>

Then, add <script type="syndicate"> tags with Syndicate/JS code in them:

1
2
3
4
5
<script type="syndicate">
  spawn {
    assert "Hello, world!";
  }
</script>

The syndicate-browser-compiler.js script automatically translates such scripts into plain JavaScript:

1
2
3
4
5
6
<script>
  const __SYNDICATE__ = Syndicate;
  __SYNDICATE__.Turn.active._spawn(() => {
    __SYNDICATE__.Turn.active.assertDataflow(() => ({ target: currentSyndicateTarget, assertion: "Hello, world!" }));
  });
</script>

Here’s a small (live!) example, hosted at codepen:

See the Pen Standalone Syndicate/JS page, December 2023 by Tony Garnock-Jones (@leastfixedpoint) on CodePen.

It’s still too hard to get started with Syndicate, but this is at least a step in the right direction.

(Just in case the codepen goes away, here is the same example as a standalone webpage.)

I’ve just released syndicate-pty-driver, a small Rust program that responds to a Syndicate protocol for allocating PTYs and spawning subprocesses attached to them.

This allows programs collaborating using Syndicate to run interactive subprocesses on the machine that’s running the PTY driver.

As an example, I’m playing around with a “VR”-like environment for use in a browser on a cellphone with a Cardboard headset and a bluetooth keyboard, loosely based on an idea by John Dean and Joan Creus-Costa (“BonelessVR: A lightweight, high-productivity desktop environment for Google Cardboard”):

The phone runs Syndicate/js in the browser, speaks the Syndicate protocol across a websocket, and a syndicate-server instance at the other end supervises a syndicate-pty-driver. The whole thing is secured by Syndicate’s object capabilities and its use of the Noise protocol for secure transport.

Because the TypeScript implementation has been moving on along its own roadmap, the IDE-support (Language Server Protocol, LSP) part of the TypeScript compiler plugin that I wrote (npm, git) has stopped working properly with recent TypeScript releases.

ts-plugin versions older than 0.15.2 don’t work well with TypeScript 4.9; it’s possible they haven’t worked well since TypeScript 4.5.

The good news

The good news is that @syndicate-lang/ts-plugin version 0.15.2 includes a few changes that make it work much better with TypeScript 4.9. So if you update to 0.15.2, you should see good support for the Syndicate DSL overlaid atop the normal TypeScript support in Emacs, VSCode, etc.

The bad news

The bad news is that the monkeypatching I’ve been doing to hook into the necessary parts of the TypeScript tsserver will no longer work for TypeScript versions 5.0 and higher. The problem is that, in tsserver 4.9, simple properties were used for values like ts.createSourceFile and ts.server.ScriptInfo, so the ts-plugin was able to hook them to insert the Syndicate DSL expansion process. In 5.0, by contrast, Object.defineProperty is used to define a non-writable, non-configurable property. It’s essentially game over for the existing ts-plugin approach at this point.

So what now?

Having ts-plugin IDE support is really nice for Syndicate programming. So I feel compelled to find some kind of solution.

Perhaps I need to write a real LSP implementation, backing into a captive tsserver instance?

I’ve opened an issue on the ts-plugin repository about this.

A few days ago, on the #syndicate IRC channel, user observeSynd (maybe observeSyndicate?) asked a few good questions:

1. Can I take ‘endpoints’ as discussed in the dissertation to be the same thing as ‘entities’ as outlined in the Synit SAM (chapter 18) theory section?
2. Are there any examples in any of the implemetations showcasing how (nested) facets work?
3. Have any of the implementations documented their DSL as would be found at https://syndicate-lang.org/doc/syndicate/ ?

Here are my (short) answers:

1. Endpoints aren’t quite entities, but they’re related. Entities are a later development. They’re much closer to traditional actor references or, even closer still, to E-style object references. Endpoints are a kind of intermediate concept: they’re a combination of a pattern, a subscription, and an object reference. I think the entity-based approach is cleaner, and in hindsight, they were not well described in my dissertation. Perhaps this was a clue that there was something simpler lurking underneath, waiting to be discovered and described.

2. Nested facets: yes, throughout the available examples. However, nesting of facets isn’t lexically apparent, so you have to read the program while imagining the dynamics.

3. Sadly, there’s no documentation of the DSLs yet. I’m embarrassed about this. Hopefully I’ll find time to rectify the situation soon! I’m afraid the various example programs are the only references out there right now.

I’ve just finished a demo of the SqueakPhone and Synit running on real phones. You can see it on youtube, or watch it embedded directly below from an mp4 file hosted on this webserver: