Syndicated Actors feed for tag "preserves-schema"2024-02-19T21:31:57+00:00https://syndicate-lang.org/journal/tag/preserves-schema/index.atomMultiaddr2024-01-11T08:05:18+00:00Tony Garnock-Joneshttps://syndicate-lang.org/journal/2024/01/11/multiaddr<p>In the <a href="irc://libera.chat/#spritely">#spritely IRC channel</a>, <code class="language-plaintext highlighter-rouge">mala</code> mentioned the
<a href="https://multiformats.io/multiaddr/">multiaddr</a> specification.</p>
<p>From the site, “Multiaddr is a format for encoding addresses from various well-established
network protocols.” The aim is to avoid “[leaving] much to interpretation and side-band
context”, and to allow people to “build applications that will work with network protocols of
the future, and do not accidentally ossify the stack.”</p>
<p>Syndicate defines something similar (though rudimentary) in its
<a href="https://git.syndicate-lang.org/syndicate-lang/syndicate-protocols/src/commit/97876335ba8d37575e17481be6bdc40a1bd4959d/schemas/transportAddress.prs">transportAddress</a>
module and in its notion of a
<a href="https://git.syndicate-lang.org/syndicate-lang/syndicate-protocols/src/commit/97876335ba8d37575e17481be6bdc40a1bd4959d/schemas/gatekeeper.prs#L51">Route</a>.</p>
<h2 id="semantics-for-multiaddr">Semantics for Multiaddr</h2>
<p>Multiaddr informally defines its semantics, and gives both a human-readable and a
machine-readable concrete syntax.</p>
<p>Let’s borrow <a href="https://preserves.dev/preserves.html#semantics">preserves</a> and propose a better
specification for multiaddr semantics:</p>
<ul>
<li>
<p>A <em>multiaddr value</em> (a <code class="language-plaintext highlighter-rouge">Multiaddr</code>) is a <a href="https://preserves.dev/preserves.html#sequences"><code class="language-plaintext highlighter-rouge">Sequence</code></a> of <em>protocol addresses</em>. The
sequence is implicitly understood to describe a layered protocol stack, with leftward
addresses (the “bottom” of the stack) acting as substrate for rightward addresses.</p>
</li>
<li>
<p>A <em>protocol address</em> (an <code class="language-plaintext highlighter-rouge">Address</code>) is a record with its label being a <a href="https://preserves.dev/preserves.html#symbols"><code class="language-plaintext highlighter-rouge">Symbol</code></a>
representing a <em>protocol name</em> (a <code class="language-plaintext highlighter-rouge">ProtocolName</code>), and its fields being zero or more
protocol-specific <a href="https://preserves.dev/preserves.html#values"><code class="language-plaintext highlighter-rouge">Value</code></a>s.</p>
</li>
<li>
<p>Each kind of <code class="language-plaintext highlighter-rouge">Address</code> must define a canonical form. Use of the canonical form is mandatory.
For example, an <code class="language-plaintext highlighter-rouge">ip6</code> address could define a rule that the associated string must always
conform to <a href="https://datatracker.ietf.org/doc/html/rfc5952#section-4">IETF IPv6 canonical
format</a>.</p>
</li>
</ul>
<p>For example, the multiaddr
<a href="https://multiformats.io/multiaddr/#network-protocol-ossification">examples</a> written</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
</pre></td><td class="rouge-code"><pre>/ip4/127.0.0.1/udp/9090/quic
/ip6/::1/tcp/3217
/ip4/127.0.0.1/tcp/80/http/baz.jpg
/dns4/foo.com/tcp/80/http/bar/baz.jpg
/dns6/foo.com/tcp/443/https
</pre></td></tr></tbody></table></code></pre></div></div>
<p>could denote the <a href="https://preserves.dev/preserves.html#values"><code class="language-plaintext highlighter-rouge">Value</code></a>s</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
</pre></td><td class="rouge-code"><pre>[<ip4 127 0 0 1> <udp 9090> <quic>]
[<ip6 "::1"> <tcp 3217>]
[<ip4 127 0 0 1> <tcp 80> <http "/baz.jpg">]
[<dns4 "foo.com"> <tcp 80> <http "/bar/baz.jpg">]
[<dns6 "foo.com"> <tcp 443> <https "/">]
</pre></td></tr></tbody></table></code></pre></div></div>
<h2 id="schema-for-multiaddr">Schema for Multiaddr</h2>
<p>Much of this structure can be captured by a preserves
<a href="https://preserves.dev/preserves-schema.html">schema</a> definition. Given such a schema, the
multiaddr text and binary syntaxes become special-purpose syntax for preserves <code class="language-plaintext highlighter-rouge">Value</code>s
conforming to the schema.</p>
<pre><code class="language-preserves-schema">version 1 .
Multiaddr = [Address ...] .
Address = <<rec> @protocolName symbol @detail [any ...]> .
WellKnownProtocol =
/ <ip4 @a int @b int @c int @d int>
/ <ip6 @addr string>
/ <dns4 @dnsName string>
/ <dns6 @dnsName string>
/ <udp @port int>
/ <tcp @port int>
/ <quic>
/ <http @path string>
/ <https @path string>
/ @unknown Address
.
</code></pre>
<p>Side-conditions such as bounding the integers in an <code class="language-plaintext highlighter-rouge">ip4</code> address to the range [0..255] are not
currently expressible in preserves schema.</p>
<p>Also, these definitions are too simple, particularly in the case of HTTP(S), where much more of
the structure of an HTTP url (username, password, path, query, fragment, etc.) can and should
be parsed out.</p>
Updates for July and August2021-09-02T14:36:55+00:00Tony Garnock-Joneshttps://syndicate-lang.org/journal/2021/09/02/update<p>Lots and lots has happened since the last update!</p>
<p>The big-ticket items each get a blog post of their own:</p>
<ul>
<li><a href="/journal/2021/09/02/preserves-path">Preserves Path</a>, a new query language for Preserves data</li>
<li><a href="/journal/2021/09/02/rust-preserves-v1-0-0">Rust Preserves v1.0.0 released</a></li>
<li><a href="/journal/2021/09/02/internal-flow-control">Internal flow control within message brokers</a></li>
<li><a href="/journal/2021/09/02/syndicate-protocols">Common Syndicate protocols in a separate repository</a></li>
<li><a href="/journal/2021/09/02/syndicated-actors-for-rust">Syndicated Actors for Rust</a>, including a new implementation of the <a href="/tools/broker/">Syndicate dataspace server daemon</a></li>
<li><a href="/journal/2021/09/02/syndicated-actors-for-python-3">Syndicated Actors for Python 3</a></li>
<li><a href="/journal/2021/09/02/syndicate-for-bash">Syndicated Actors for Bash</a> (Yes, really! 🙂)</li>
<li><a href="/journal/2021/09/02/synit">Progress on Syndicate-at-the-System-Layer</a></li>
</ul>
Common Syndicate protocols repository2021-09-02T12:28:33+00:00Tony Garnock-Joneshttps://syndicate-lang.org/journal/2021/09/02/syndicate-protocols<p>Because there are a number of different Syndicate <a href="/code/">implementations</a>, and they all need to interoperate, I’ve used
Preserves Schema<sup id="fnref:preserves-schema-links" role="doc-noteref"><a href="#fn:preserves-schema-links" class="footnote" rel="footnote">1</a></sup> to define message formats
that all the implementations can share.</p>
<p>You can find the common protocol definitions in a new git repository:</p>
<p class="aligncenter"><code class="language-plaintext highlighter-rouge">git clone </code><a href="https://git.syndicate-lang.org/syndicate-lang/syndicate-protocols"><code class="language-plaintext highlighter-rouge">https://git.syndicate-lang.org/syndicate-lang/syndicate-protocols</code></a></p>
<p>I used
<a href="https://github.com/git/git/blob/master/contrib/subtree/git-subtree.txt"><code class="language-plaintext highlighter-rouge">git-subtree</code></a>
to carve out a
<a href="https://git.syndicate-lang.org/syndicate-lang/novy-syndicate/src/commit/9bb081292b8b9c01961e724d5977aa00e1fd2fc4/schemas">portion of the <code class="language-plaintext highlighter-rouge">novy-syndicate</code> source tree</a>
to form the new repository. It seems to be working well. In the
projects that depend on the protocol definitions, I run the following
every now and then to sync up with the <code class="language-plaintext highlighter-rouge">syndicate-protocols</code>
repository:</p>
<div class="language-plaintext aligncenter highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
</pre></td><td class="rouge-code"><pre>git subtree pull -P protocols \
-m 'Merge latest changes from the syndicate-protocols repository' \
git@git.syndicate-lang.org:syndicate-lang/syndicate-protocols \
main
</pre></td></tr></tbody></table></code></pre></div></div>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:preserves-schema-links" role="doc-endnote">
<p>See
<a href="https://preserves.dev/preserves-schema.html">here</a>
for more about Preserves Schema, or check out
<a href="/journal/tag/preserves-schema/">posts tagged with <code class="language-plaintext highlighter-rouge">#preserves-schema</code></a>. <a href="#fnref:preserves-schema-links" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>
Rust Preserves v1.0.0 released2021-09-02T10:45:22+00:00Tony Garnock-Joneshttps://syndicate-lang.org/journal/2021/09/02/rust-preserves-v1-0-0<p>As part of my other implementation efforts, I made enough improvements
to the Rust Preserves implementation to warrant releasing version
1.0.0.</p>
<p>This release supports the Preserves data model and the binary and text
codecs. It also includes a Preserves Schema compiler for Rust (for use
e.g. in <code class="language-plaintext highlighter-rouge">build.rs</code>) and an implementation of <a href="/journal/2021/09/02/preserves-path">Preserves Path</a>.</p>
<p>There are four crates:</p>
<ul>
<li>
<p><a href="https://crates.io/crates/preserves"><code class="language-plaintext highlighter-rouge">preserves</code></a>, Rust
representations of
<a href="https://preserves.dev/preserves.html#starting-with-semantics"><code class="language-plaintext highlighter-rouge">Value</code>s</a>
plus utilities and binary and text codecs;</p>
</li>
<li>
<p><a href="https://crates.io/crates/preserves-schema"><code class="language-plaintext highlighter-rouge">preserves-schema</code></a>, a
Rust implementation of
<a href="https://preserves.dev/preserves-schema.html">Preserves Schema</a>;</p>
</li>
<li>
<p><a href="https://crates.io/crates/preserves-path"><code class="language-plaintext highlighter-rouge">preserves-path</code></a>, a Rust
implementation of
<a href="https://preserves.dev/preserves-path.html">Preserves Path</a>;
and</p>
</li>
<li>
<p><a href="https://crates.io/crates/preserves-tools"><code class="language-plaintext highlighter-rouge">preserves-tools</code></a>, a
crate of command-line utilities (including a “swiss army knife”,
<a href="https://preserves.dev/doc/preserves-tool.html"><code class="language-plaintext highlighter-rouge">preserves-tool</code></a>)
for working with Preserves documents.</p>
</li>
</ul>
Major progress on capability-based syndicate-rkt implementation2021-06-04T21:22:30+00:00Tony Garnock-Joneshttps://syndicate-lang.org/journal/2021/06/04/progress-on-ocap-syndicate-rkt<p>I’ve been working on the <a href="https://git.syndicate-lang.org/syndicate-lang/syndicate-rkt/src/branch/main"><code class="language-plaintext highlighter-rouge">novy</code> branch of <code class="language-plaintext highlighter-rouge">syndicate-rkt</code></a> (<strong>Update: this is now the main branch</strong>), following the new
design I developed for the <a href="https://git.syndicate-lang.org/syndicate-lang/novy-syndicate"><code class="language-plaintext highlighter-rouge">novy-syndicate</code> TypeScript prototype</a>, driving the design further and
working out new syntax ideas.</p>
<h2 id="syndicaterkt-example">Syndicate/rkt example</h2>
<p>Here’s <a href="https://git.syndicate-lang.org/syndicate-lang/syndicate-rkt/src/branch/main/syndicate-examples/box-and-client.rkt">an example program, <code class="language-plaintext highlighter-rouge">box-and-client.rkt</code></a>
in the new Syndicate/rkt language:</p>
<div class="language-racket aligncenter linenos highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
</pre></td><td class="rouge-code"><pre><span class="o">#</span><span class="nv">lang</span> <span class="nv">syndicate</span>
<span class="p">(</span><span class="nf">message-struct</span> <span class="nv">set-box</span> <span class="p">(</span><span class="nf">new-value</span><span class="p">))</span>
<span class="p">(</span><span class="nf">assertion-struct</span> <span class="nv">box-state</span> <span class="p">(</span><span class="nf">value</span><span class="p">))</span>
<span class="p">(</span><span class="nf">module+</span> <span class="nv">main</span>
<span class="p">(</span><span class="nf">actor-system/dataspace</span> <span class="p">(</span><span class="nf">ds</span><span class="p">)</span>
<span class="p">(</span><span class="nf">spawn</span> <span class="nt">#:name</span> <span class="ss">'box</span>
<span class="p">(</span><span class="nf">define-field</span> <span class="nv">current-value</span> <span class="mi">0</span><span class="p">)</span>
<span class="p">(</span><span class="nf">at</span> <span class="nv">ds</span>
<span class="p">(</span><span class="nf">assert</span> <span class="p">(</span><span class="nf">box-state</span> <span class="p">(</span><span class="nf">current-value</span><span class="p">)))</span>
<span class="p">(</span><span class="nf">on</span> <span class="p">(</span><span class="nf">message</span> <span class="p">(</span><span class="nf">set-box</span> <span class="nv">$new-value</span><span class="p">))</span>
<span class="p">(</span><span class="nf">log-info</span> <span class="s">"box: taking on new-value ~v"</span> <span class="nv">new-value</span><span class="p">)</span>
<span class="p">(</span><span class="nf">current-value</span> <span class="nv">new-value</span><span class="p">)))</span>
<span class="p">(</span><span class="nf">stop-on-true</span> <span class="p">(</span><span class="nb">=</span> <span class="p">(</span><span class="nf">current-value</span><span class="p">)</span> <span class="mi">10</span><span class="p">)</span>
<span class="p">(</span><span class="nf">log-info</span> <span class="s">"box: terminating"</span><span class="p">)))</span>
<span class="p">(</span><span class="nf">spawn</span> <span class="nt">#:name</span> <span class="ss">'client</span>
<span class="p">(</span><span class="nf">at</span> <span class="nv">ds</span>
<span class="p">(</span><span class="nf">stop-on</span> <span class="p">(</span><span class="nf">retracted</span> <span class="p">(</span><span class="nf">Observe</span> <span class="p">(</span><span class="nf">:pattern</span> <span class="p">(</span><span class="nf">set-box</span> <span class="o">,</span><span class="nv">_</span><span class="p">))</span> <span class="nv">_</span><span class="p">))</span>
<span class="p">(</span><span class="nf">log-info</span> <span class="s">"client: box has gone"</span><span class="p">))</span>
<span class="p">(</span><span class="nf">on</span> <span class="p">(</span><span class="nf">asserted</span> <span class="p">(</span><span class="nf">box-state</span> <span class="nv">$v</span><span class="p">))</span>
<span class="p">(</span><span class="nf">log-info</span> <span class="s">"client: learned that box's value is now ~v"</span> <span class="nv">v</span><span class="p">)</span>
<span class="p">(</span><span class="nf">send!</span> <span class="nv">ds</span> <span class="p">(</span><span class="nf">set-box</span> <span class="p">(</span><span class="nb">+</span> <span class="nv">v</span> <span class="mi">1</span><span class="p">))))</span>
<span class="p">(</span><span class="nf">on</span> <span class="p">(</span><span class="nf">retracted</span> <span class="p">(</span><span class="nf">box-state</span> <span class="nv">_</span><span class="p">))</span>
<span class="p">(</span><span class="nf">log-info</span> <span class="s">"client: box state disappeared"</span><span class="p">))))))</span>
</pre></td></tr></tbody></table></code></pre></div></div>
<p>The program consists of two actors, <code class="language-plaintext highlighter-rouge">'box</code> and <code class="language-plaintext highlighter-rouge">'client</code>. The box
actor publishes the value of its <code class="language-plaintext highlighter-rouge">current-value</code> field, wrapped in a
<code class="language-plaintext highlighter-rouge">box-state</code> record constructor, to the dataspace (line 11). It reacts
to <code class="language-plaintext highlighter-rouge">set-box</code> messages sent by peers (lines 12–14); in this case, the
client actor, which sends <code class="language-plaintext highlighter-rouge">set-box</code> to increment the value each time
it learns of an updated value from the box (lines 22–24).</p>
<p>The box actor terminates once <code class="language-plaintext highlighter-rouge">current-value</code> reaches <code class="language-plaintext highlighter-rouge">10</code>. The client
notices the termination of the box actor in two ways (just to show
them off): first, by noticing that the <code class="language-plaintext highlighter-rouge">box-state</code> record was
unpublished from the dataspace (lines 25–26); and second, by noticing
that all subscribers to <code class="language-plaintext highlighter-rouge">set-box</code> messages have vanished (lines
20–21).</p>
<h2 id="whats-different-whats-new">What’s different? What’s new?</h2>
<h3 id="explicit-object-references">Explicit object references</h3>
<p>The most notable change from previous dataspace programs is the
explicit reference to the dataspace, <code class="language-plaintext highlighter-rouge">ds</code>. Assertions and
subscriptions are now <em>located</em> at a specific (possibly remote)
object, usually but not always a dataspace.</p>
<h3 id="capability-based-security">Capability-based security</h3>
<p>Related is support for
<a href="https://research.google/pubs/pub41892/">macaroon</a>-style “sturdy
references” (analogous to the
<a href="http://wiki.erights.org/wiki/SturdyRef">SturdyRef</a> concept from E).
Here’s an example from a secure<sup>*</sup> chat demo app:</p>
<div class="language-plaintext aligncenter linenos highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
</pre></td><td class="rouge-code"><pre><ref "syndicate" [[<or [
<rewrite <bind p <compound <rec Present 1> {0: <lit "tonyg">}>> <ref p>>,
<rewrite <bind p <compound <rec Says 2> {
0: <lit "tonyg">,
1: String
}>> <ref p>>
]>]] #[oHFy7B4NPVqhD6zJmNPbhg==]>
</pre></td></tr></tbody></table></code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">oid</code> (<code class="language-plaintext highlighter-rouge">"syndicate"</code> on line 1) identifies the target object. The
patterns (lines 2–6) attenuate the authority of the capability to
only permit transmission of <code class="language-plaintext highlighter-rouge">Present</code> and <code class="language-plaintext highlighter-rouge">Says</code> records. The
signature (line 7) proves to the target object that the capability is
genuine and untampered-with.</p>
<p>I’ve implemented most of the necessary plumbing for these, but have
yet to complete the client/server portion of the system that actually
makes use of them. For an example of their use,
<a href="https://git.syndicate-lang.org/syndicate-lang/novy-syndicate/src/commit/594fe010672d27182c34c00cc80b29817c9a297d/README.md#take-it-for-a-spin">see <code class="language-plaintext highlighter-rouge">novy-syndicate</code></a>.</p>
<h3 id="schema-support">Schema support</h3>
<p>Another interesting change is support for (the relatively new)
Preserves Schema. You can use <code class="language-plaintext highlighter-rouge">assertion-struct</code> and <code class="language-plaintext highlighter-rouge">message-struct</code>
as in previous dialects, or you can use Schema-defined types to
establish subscriptions and place assertions with a peer.</p>
<h3 id="full-pattern-matching-dataspace-implementation">Full pattern-matching dataspace implementation</h3>
<p>Unlike the <code class="language-plaintext highlighter-rouge">novy-syndicate</code> prototype, this implementation is the
first capability-based design to have a proper “skeleton”-based
dataspace that supports the full range of dataspace patterns. This
allows us to write, for example, subscriptions like</p>
<div class="language-racket aligncenter highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
</pre></td><td class="rouge-code"><pre><span class="p">(</span><span class="nf">on</span> <span class="p">(</span><span class="nf">retracted</span> <span class="p">(</span><span class="nf">box-state</span> <span class="nv">_</span><span class="p">))</span> <span class="o">...</span><span class="p">)</span>
</pre></td></tr></tbody></table></code></pre></div></div>
<p>which only fires when <em>all</em> <code class="language-plaintext highlighter-rouge">box-state</code> assertions are withdrawn.</p>
<h3 id="patterns-over-hash-tables">Patterns over hash-tables</h3>
<p>Previous implementations could only match fields in records (with
constant labels) and elements of arrays/lists. This new implementation
is also able to express and match patterns over named-key elements in
<code class="language-plaintext highlighter-rouge">Dictionary</code> <code class="language-plaintext highlighter-rouge">Value</code>s.</p>
<p>This lets actors express patterns over JSON-like Preserves documents,
for example.</p>
<h3 id="pattern-quasiquotation">Pattern quasiquotation</h3>
<p>One of the issues I hoped the new architecture would shed light on is
<em>pattern quotation</em>. In order to express interest <em>in interest
expressed by some other party</em>, you need to be able to <em>describe</em> the
subscriptions that are of interest to you. That means you must be able
to write patterns <em>over patterns</em>.</p>
<p>Previous implementations didn’t get this right. It was not possible to
precisely express interest in subscriptions that bound (or did not
bind) certain portions of their input; and it was not possible to
precisely express the difference between being interested in a binding
or binding a portion of the pattern to be matched itself.</p>
<p>The new design solves these issues with a
<a href="https://docs.racket-lang.org/guide/qq.html">quasiquote</a>-like
facility. Here’s a pattern that matches “subscriptions to unary
set-box records”:</p>
<div class="language-plaintext aligncenter highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
</pre></td><td class="rouge-code"><pre>(Observe (:pattern (set-box ,_)) _)
</pre></td></tr></tbody></table></code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">:pattern</code> wrapper introduces a quoted pattern, and
unquote-discard (“<code class="language-plaintext highlighter-rouge">,_</code>”) pops back out a level to say that we don’t
care what the subscriber has put in their pattern at that position.
For example, they may have elected to bind the value inside the
<code class="language-plaintext highlighter-rouge">set-box</code>, or they may have elected to ignore it, or they may have
elected to match only certain values of it, and so on. By discarding
that portion of the pattern, we ignore the specific choice the
matching subscriber made.</p>
<p>If instead we use unquote-bind (“<code class="language-plaintext highlighter-rouge">,$</code><em>id</em>”), we extract a portion of
the pattern each subscriber placed in the dataspace:</p>
<div class="language-plaintext aligncenter highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
</pre></td><td class="rouge-code"><pre>(Observe (:pattern (set-box ,$value-pat)) _)
</pre></td></tr></tbody></table></code></pre></div></div>
<p>For example, if some subscriber is binding the value in the <code class="language-plaintext highlighter-rouge">set-box</code>
to an identifier <code class="language-plaintext highlighter-rouge">new-value</code>, but otherwise placing no constraints on
it, we will be given the following value for <code class="language-plaintext highlighter-rouge">value-pat</code>:</p>
<div class="language-plaintext aligncenter highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
</pre></td><td class="rouge-code"><pre><bind new-value <_>>
</pre></td></tr></tbody></table></code></pre></div></div>
<p>If, on the other hand, a subscriber is completely ignoring the value
in the <code class="language-plaintext highlighter-rouge">set-box</code>, caring only about the <code class="language-plaintext highlighter-rouge">set-box</code> wrapper itself, we
will be given <code class="language-plaintext highlighter-rouge"><_></code>, the “discard” pattern.</p>
<p>Crucially, we are now able to distinguish between
binding-a-portion-of-the-matched-pattern and
matching-a-portion-that-is-a-binding. We’ve seen the former already
with unquote-bind; the latter is accomplished by using unquote in the
structured syntax for a binding:</p>
<div class="language-plaintext aligncenter highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
</pre></td><td class="rouge-code"><pre>(Observe (:pattern (set-box ($ ,$their-id ,$further-constraint))) _)
</pre></td></tr></tbody></table></code></pre></div></div>
<p>Here we unquote <em>twice</em>. The <code class="language-plaintext highlighter-rouge">($ ...)</code> constructor itself specifies
that we require matching subscriptions to have a binding at this
position. The first unquote extracts the name in the binding, and the
second extracts the subpattern for the binding. For the example above,
we would end up with <code class="language-plaintext highlighter-rouge">their-id</code> bound to the symbol <code class="language-plaintext highlighter-rouge">new-value</code> and
<code class="language-plaintext highlighter-rouge">further-constraint</code> bound to the subpattern <code class="language-plaintext highlighter-rouge"><_></code>.</p>
<p>Finally, let’s examine a couple of alternatives that don’t work. This
one is missing the <code class="language-plaintext highlighter-rouge">:pattern</code> wrapper, meaning that instead of asking
about <em>patterns</em> over <code class="language-plaintext highlighter-rouge">set-box</code> records, it is asking about observers
that (mistakenly?) specified an <em>actual</em> <code class="language-plaintext highlighter-rouge">set-box</code> record instead of a
pattern!</p>
<div class="language-plaintext aligncenter highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
</pre></td><td class="rouge-code"><pre>(Observe (set-box ,_) _)
</pre></td></tr></tbody></table></code></pre></div></div>
<p>The compiler won’t actually let you use this version, because the
unquote-discard is out of place. There’s no quasiquotation to escape
from, so this is a syntax error.</p>
<p>We might try repairing this by simply removing the unquote:</p>
<div class="language-plaintext aligncenter highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
</pre></td><td class="rouge-code"><pre>(Observe (set-box _) _)
</pre></td></tr></tbody></table></code></pre></div></div>
<p>But this is still asking the wrong question, and will never receive
any interesting matches from other subscribers in the system.</p>
<h2 id="how-does-it-perform">How does it perform?</h2>
<p>Very well! At present it is roughly twice as fast as the previous
Racket implementation. Running
<a href="https://git.syndicate-lang.org/syndicate-lang/syndicate-rkt/src/commit/1290e30c3d872804838665d290a10bf5c3495407/syndicate-examples/speed-tests/box-and-client/with-dataspace.rkt">a benchmark based on the example program above</a>
yields the following on one thread of my Ryzen 3960X:</p>
<div class="language-plaintext aligncenter highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
</pre></td><td class="rouge-code"><pre>syndicate/actor: #<actor:0:dataspace> booting
syndicate/task: #<engine:0> starting
syndicate/actor: #<actor:3:box> booting
syndicate/actor: #<actor:6:client> booting
Box got 100000 (66222.3817422824 Hz)
Box got 200000 (68740.1257393843 Hz)
Box got 300000 (68724.52087884837 Hz)
Box got 400000 (68670.19562623161 Hz)
Box got 500000 (68786.55545277878 Hz)
syndicate/actor: #<actor:3:box> terminated OK
Client detected box termination
syndicate/actor: #<actor:6:client> terminated OK
syndicate/task: #<engine:0> stopping
cpu time: 7330 real time: 7330 gc time: 68
</pre></td></tr></tbody></table></code></pre></div></div>
<p>It means that the program is able to do ~68,000 complete round-trips
per second of update and signalling between the box and client actors.</p>
<h2 id="preserves-and-preserves-schema">Preserves and Preserves Schema</h2>
<p>The new implementation depends heavily on Preserves and Preserves
Schema, so I’ve ended up doing a fair bit of work on those in order to
get things working in Syndicate/rkt. (Among other things, fixing the
<code class="language-plaintext highlighter-rouge">raco pkg install</code> process for the <code class="language-plaintext highlighter-rouge">preserves</code> and <code class="language-plaintext highlighter-rouge">syndicate</code> Racket
packages!)</p>
<p>First, one nice bit of news is a new Preserves implementation,
<a href="https://git.sr.ht/~ehmry/preserves-nim">preserves-nim</a> by Emery
Hemingway, for the <a href="https://nim-lang.org/">Nim</a> programming language.
I’ve linked the various implementations of Preserves and Syrup on the
<a href="https://preserves.dev/">main Preserves webpage</a>.</p>
<p>There have also been changes to the Schema language and tooling. The
main change to the Schema <em>language</em> is a reappraisal of the role of
<code class="language-plaintext highlighter-rouge">Embedded</code> values in schemas. Previously, they were treated as black
boxes - given just enough machinery to parse them out of and serialize
them back into a <code class="language-plaintext highlighter-rouge">Value</code>, but nothing more. Now, they’re given both a
(de)serializer <em>and</em> an “interface type”; the idea is that an
<code class="language-plaintext highlighter-rouge">Embedded</code> represents a capability to some behavioural object - a
closure, an object pointer, an actor reference, a web service, that
kind of thing - and so there may be an associated API that can be
usefully schematized. This makes schematization of <code class="language-plaintext highlighter-rouge">Embedded</code> values
something closely related to the
<a href="https://www2.ccs.neu.edu/racket/pubs/dissertation-dimoulas.pdf">higher-order contracts of Dimoulas</a>;
see the bit on
<a href="https://preserves.dev/preserves-schema.html#appendix-future-work">future work in the spec</a>
for some additional thoughts along these lines, as well as a little
example.</p>
<p>The main change to the Schema <em>tooling</em> is support for <em>plugins</em> in
the Schema compiler, allowing Syndicate/rkt to supply a plugin for
generating dataspace patterns from parsed Schema values. The <code class="language-plaintext highlighter-rouge">#lang
preserves-schema</code> support has been likewise extended so you can supply
plugins in the <code class="language-plaintext highlighter-rouge">#lang</code> line.</p>
<p>Last (and probably least), here’s a fun little schema example:</p>
<div class="language-plaintext aligncenter highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
</pre></td><td class="rouge-code"><pre>version 1 .
JSON =
/ @string string
/ @integer int
/ @double double
/ @boolean JSONBoolean
/ @null =null
/ @array [JSON ...]
/ @object { string: JSON ...:... } .
JSONBoolean = =true / =false .
</pre></td></tr></tbody></table></code></pre></div></div>
<p>It recognises the JSON-interoperable subset of Preserves <code class="language-plaintext highlighter-rouge">Value</code>s!</p>
<h2 id="licensing">Licensing</h2>
<p>A note about licensing: I’ve chosen LGPL 3.0+ as the license for
Syndicate/rkt. Many thanks to
<a href="https://github.com/massimo-zaniboni">Massimo Zaniboni</a> for pointing
out the lack of license, discussing various options with me, and
helping sort out the per-file license headers.</p>
Tools for working with Preserves2021-06-04T15:50:07+00:00Tony Garnock-Joneshttps://syndicate-lang.org/journal/2021/06/04/preserves-tools<p>I’ve added new documentation for a few useful tools for working with
Preserves and Preserves Schema. Find an overview
<a href="https://preserves.dev/#tools">here</a>.</p>
More progress on Preserves Schemas2021-05-25T12:47:11+00:00Tony Garnock-Joneshttps://syndicate-lang.org/journal/2021/05/25/preserves-schema-progress<p>Since my last post, I’ve been working some more on Preserves Schemas.</p>
<h2 id="schema-language-documented">Schema language documented</h2>
<ul>
<li>The most visible result of this round of work is that I have
written a first pass at a
<a href="https://preserves.dev/preserves-schema.html">specification of the Preserves Schema language</a>.
Feedback on the specification and the language very welcome.</li>
</ul>
<h2 id="metaschema-improvements">Metaschema improvements</h2>
<ul>
<li>
<p>I also made
<a href="https://git.syndicate-lang.org/syndicate-lang/preserves/commit/2559a4713f050b23efc492c81d5fa0aed10deee5#diff-12">a major change to the metaschema</a>,
moving <code class="language-plaintext highlighter-rouge">setof</code> and <code class="language-plaintext highlighter-rouge">dictof</code> from <code class="language-plaintext highlighter-rouge">CompoundPattern</code> to
<code class="language-plaintext highlighter-rouge">SimplePattern</code>, and splitting <code class="language-plaintext highlighter-rouge">seqof</code> out from <code class="language-plaintext highlighter-rouge">tuple*</code>.</p>
<p>Placing <code class="language-plaintext highlighter-rouge">seqof</code>, <code class="language-plaintext highlighter-rouge">setof</code> and <code class="language-plaintext highlighter-rouge">dictof</code> in <code class="language-plaintext highlighter-rouge">SimplePattern</code> rather
than <code class="language-plaintext highlighter-rouge">CompoundPattern</code> makes both <code class="language-plaintext highlighter-rouge">SimplePattern</code> and
<code class="language-plaintext highlighter-rouge">CompoundPattern</code> actually <em>meaningful</em>, and this leads to
significant code simplification, which in turn gives me more
confidence in the design of the language.</p>
<p>Patterns in <code class="language-plaintext highlighter-rouge">SimplePattern</code> now denote single host-language values
without interesting substructure; that is, they are the values of
individual <em>fields</em> within a generated record type. Patterns in
<code class="language-plaintext highlighter-rouge">CompoundPattern</code> consistently now denote <em>collections of fields</em>
rather than individual values.</p>
</li>
</ul>
<h2 id="reader-and-checker-improvements">Reader and checker improvements</h2>
<ul>
<li>
<p>I
<a href="https://git.syndicate-lang.org/syndicate-lang/preserves/commit/46d76dfca7470c822f6400257a5868604ac291d0">improved checking of schemas at “read” time</a>.
The checker no longer requires knowledge of generated types for any
particular host language. Schema well-formedness is purely an AST
property.</p>
</li>
<li>
<p>The checker now
<a href="https://git.syndicate-lang.org/syndicate-lang/preserves/commit/0db223ede88f61cd8681160a8783f3b0ae3012fd#diff-1">checks for duplicate variant names</a>
in a union, alongside the existing duplicate-binding and
invertibility checks.</p>
</li>
</ul>
<h2 id="schema-implementation-improvements">Schema implementation improvements</h2>
<ul>
<li>
<p>I
<a href="https://git.syndicate-lang.org/syndicate-lang/preserves/src/commit/badb05944094ade3828822fada2073b282433625/implementations/racket/preserves/preserves-schema/reader.rkt">implemented a second reader, in Racket</a>.
This drove a number of small improvements in the Racket preserves
implementation and the rest of the Racket preserves-schema
codebase.</p>
</li>
<li>
<p>I also added a
<a href="https://git.syndicate-lang.org/syndicate-lang/preserves/src/branch/main/implementations/javascript/packages/schema/test">nascent test suite for the TypeScript schema package</a>.</p>
</li>
</ul>
Preserves Schemas for Racket2021-05-21T20:18:00+00:00Tony Garnock-Joneshttps://syndicate-lang.org/journal/2021/05/21/preserves-schemas-for-racket<p>Today has mostly been spent working on Preserves Schemas.</p>
<p>I made a few changes to the Schema language itself, ancillary changes
to the TypeScript/JavaScript implementation, and built a first pass at
a Racket implementation of a Schema compiler.</p>
<h2 id="schema-language-changes">Schema language changes</h2>
<ul>
<li>
<p><a href="https://gitlab.com/preserves/preserves/-/commit/10380e451aa9bf80f5a9008c544abc0853e9b832">It’s now forbidden to use both “&” (intersection) and “/” (alternation) operators in a single rule.</a> You have
to pick one or the other. If you want to use both, you have to lift
the inner one out to a separate rule.</p>
<pre><code class="language-preserves"> # Not allowed:
BadRule = A / B & C / D .
# Allowed:
GoodRule = A / BC / D .
BC = B & C .
</code></pre>
<p><em>Rationale:</em> It was confusing that there was precedence there at all;
and if both operators are available at the same time, there’s an
ambiguous case with respect to the names chosen for the branches vs
the names chosen for the branches of an intersection. Here’s what the
code used to say about this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
</pre></td><td class="rouge-code"><pre>// TODO: deal with situation where there's an or of ands, where
// the branches of the and are named. The parsing is ambiguous, and
// with the current code I think (?) you end up with the same name
// attached to the or-branch as to the leftmost and-branch.
</pre></td></tr></tbody></table></code></pre></div> </div>
</li>
</ul>
<h2 id="metaschema-changes">Metaschema changes</h2>
<ul>
<li>
<p><a href="https://gitlab.com/preserves/preserves/-/commit/ecdb3143666c3fb40760eabaa02b66f53b09d24c"><code class="language-plaintext highlighter-rouge">Definition</code>s involving <code class="language-plaintext highlighter-rouge">or</code> or <code class="language-plaintext highlighter-rouge">and</code> now have to have at least two branches.</a>
Previously, <code class="language-plaintext highlighter-rouge">or</code> and <code class="language-plaintext highlighter-rouge">and</code> had to have at least <em>one</em> branch, which
was good because it prevented instances from having
sensible-but-unwanted definitions like <code class="language-plaintext highlighter-rouge"><or []></code> or <code class="language-plaintext highlighter-rouge"><and []></code>.</p>
<p>This change improves things further, since for all <em>P</em>, <code class="language-plaintext highlighter-rouge"><or [</code><em>P</em><code class="language-plaintext highlighter-rouge">]>
= </code><em>P</em> and <code class="language-plaintext highlighter-rouge"><and [</code><em>P</em><code class="language-plaintext highlighter-rouge">]> = </code><em>P</em>.</p>
</li>
</ul>
<h2 id="implementation-and-tooling-changes">Implementation and tooling changes</h2>
<ul>
<li>
<p><a href="https://gitlab.com/preserves/preserves/-/commit/20b676df27942097069cae4f81e26902b945f39e">Some “type checking” of Schemas is now performed at “read” time, not just at code-generation time.</a> This
means that all toolchains that use
<a href="https://gitlab.com/preserves/preserves/-/blob/986e7fa30d3f56e34e99f637126b13fb355a8a9c/implementations/javascript/packages/schema/src/reader.ts#L59-63"><code class="language-plaintext highlighter-rouge">readSchema</code> from <code class="language-plaintext highlighter-rouge">reader.ts</code></a>
automatically benefit from <code class="language-plaintext highlighter-rouge">checkSchema</code>.</p>
<p>Concretely, at the moment <code class="language-plaintext highlighter-rouge">checkSchema</code> does duplicate-binding
checks and also ensures that a Schema specifies a bijection between
plain Preserves content and the parsed data structures specified by
the Schema.</p>
</li>
<li>
<p><a href="https://gitlab.com/preserves/preserves/-/commit/49cba14b4fe8154924bc135a4301dfd60b07799d">I built an initial Preserves Schema compiler/code-generator for Racket.</a> So
far, it is able to read and generate code for working with the
<a href="https://gitlab.com/preserves/preserves/-/blob/986e7fa30d3f56e34e99f637126b13fb355a8a9c/schema/schema.prs">metaschema</a>.</p>
<p>Here’s an example.
<a href="https://gitlab.com/preserves/preserves/-/blob/986e7fa30d3f56e34e99f637126b13fb355a8a9c/schema/schema.prs#L11-15">This definition</a>
from the metaschema:</p>
<pre><code class="language-preserves">Schema = <schema {
version: Version
embeddedType: EmbeddedTypeName
definitions: Definitions
}>.
</code></pre>
<p>generates the following code (as part of a complete module for all
the definitions in the metaschema):</p>
<div class="language-racket highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
</pre></td><td class="rouge-code"><pre><span class="p">(</span><span class="nf">struct</span> <span class="nv">Schema</span> <span class="p">(</span><span class="nb">version</span> <span class="nv">embeddedType</span> <span class="nv">definitions</span><span class="p">)</span> <span class="nt">#:prefab</span><span class="p">)</span>
<span class="p">(</span><span class="k">define</span> <span class="p">(</span><span class="nf">parse-Schema</span> <span class="nv">input</span><span class="p">)</span>
<span class="p">(</span><span class="nf">match</span> <span class="nv">input</span>
<span class="p">[(</span><span class="k">and</span> <span class="nv">dest</span>
<span class="p">(</span><span class="nf">record</span> <span class="ss">'schema</span>
<span class="p">(</span><span class="nb">list</span>
<span class="p">(</span><span class="nf">hash-table</span>
<span class="p">(</span><span class="ss">'version</span> <span class="p">(</span><span class="nf">app</span> <span class="nv">parse-Version</span> <span class="p">(</span><span class="k">and</span> <span class="nv">$version</span> <span class="p">(</span><span class="nb">not</span> <span class="p">(</span><span class="nf">==</span> <span class="nv">eof</span><span class="p">)))))</span>
<span class="p">(</span><span class="ss">'embeddedType</span> <span class="p">(</span><span class="nf">app</span> <span class="nv">parse-EmbeddedTypeName</span> <span class="p">(</span><span class="k">and</span> <span class="nv">$embeddedType</span> <span class="p">(</span><span class="nb">not</span> <span class="p">(</span><span class="nf">==</span> <span class="nv">eof</span><span class="p">)))))</span>
<span class="p">(</span><span class="ss">'definitions</span> <span class="p">(</span><span class="nf">app</span> <span class="nv">parse-Definitions</span> <span class="p">(</span><span class="k">and</span> <span class="nv">$definitions</span> <span class="p">(</span><span class="nb">not</span> <span class="p">(</span><span class="nf">==</span> <span class="nv">eof</span><span class="p">)))))</span>
<span class="p">(</span><span class="nf">_</span> <span class="nv">_</span><span class="p">)</span> <span class="o">...</span><span class="p">))))</span>
<span class="p">(</span><span class="nf">Schema</span> <span class="nv">$version</span> <span class="nv">$embeddedType</span> <span class="nv">$definitions</span><span class="p">)]</span>
<span class="p">[</span><span class="nf">_</span> <span class="nv">eof</span><span class="p">]))</span>
<span class="p">(</span><span class="k">define</span> <span class="p">(</span><span class="nf">Schema->preserves</span> <span class="nv">input</span><span class="p">)</span>
<span class="p">(</span><span class="nf">match</span> <span class="nv">input</span>
<span class="p">[(</span><span class="nf">Schema</span> <span class="nv">$version</span> <span class="nv">$embeddedType</span> <span class="nv">$definitions</span><span class="p">)</span>
<span class="p">(</span><span class="nf">record</span> <span class="ss">'schema</span>
<span class="p">(</span><span class="nb">list</span>
<span class="p">(</span><span class="nb">hash</span>
<span class="ss">'version</span> <span class="p">(</span><span class="nf">Version->preserves</span> <span class="nv">$version</span><span class="p">)</span>
<span class="ss">'embeddedType</span> <span class="p">(</span><span class="nf">EmbeddedTypeName->preserves</span> <span class="nv">$embeddedType</span><span class="p">)</span>
<span class="ss">'definitions</span> <span class="p">(</span><span class="nf">Definitions->preserves</span> <span class="nv">$definitions</span><span class="p">))))]))</span>
</pre></td></tr></tbody></table></code></pre></div> </div>
</li>
</ul>