Fixing up protocol mismatches on-the-fly

I’ve been fleshing out the syndicate-rkt Racket implementation based on the novy-syndicate TypeScript sketch. I just reached a milestone of TCP-based interoperability between the two implementations (yay!), but there’s an interesting little side track involved that I thought I’d write about.

The novy-syndicate code had a placeholder “dataspace” implementation that had extremely limited pattern-matching. It was only able to offer subscribers the ability to select (1) record assertions having (2) a user-selected, constant label.

For example, a subscriber could elect to receive all records labelled with Present; or with Says. Subscribers were not able to even specify arity of matched records. It really was a placeholder for a proper implementation to come later (ported across from a previous syndicate/js implementation).

By contrast, the syndicate-rkt code has a full-fledged dataspace able to index assertions according to quite sophisticated patterns:

1
2
3
4
5
6
7
8
9
10
11
12
13
; Dataspace patterns: a sublanguage of attenuation patterns.
Pattern = DDiscard / DBind / DLit / DCompound .

DDiscard = <_>.
DBind = <bind @name symbol @pattern Pattern>.
DLit = <lit @value any>.
DCompound = @rec  <compound @ctor CRec  @members { int: Pattern ...:... }>
          / @arr  <compound @ctor CArr  @members { int: Pattern ...:... }>
          / @dict <compound @ctor CDict @members { any: Pattern ...:... }> .

CRec = <rec @label any @arity int>.
CArr = <arr @arity int>.
CDict = <dict>.

Now, I managed to get the novy-syndicate example programs to talk to the full syndicate/rkt dataspace - without changing the code!

The way I did it was to rewrite assertions travelling between the programs on the fly.

And the way I did that was to include “rewrite” statements in the capability I gave to the novy-syndicate client to allow it to connect to the syndicate/rkt server.

The idea was to rewrite assertions-of-interest (subscriptions) from the simple label-only pattern of novy-syndicate to the equivalent full-dataspace pattern of syndicate/rkt, and to rewrite the responses from the dataspace from the arbitrary-arity responses of syndicate/rkt to the simple unary responses of novy-syndicate.

Here’s the rewrite specification,1 which ultimately appears embedded as a “caveat” inside the Macaroon-style capabilities that Syndicate uses:2

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
[ <or [
    <rewrite

     ; Step 1:
     <compound <rec Observe 2> {0: <bind label Symbol>, 1: <bind observer Embedded>}>
     <compound <rec Observe 2> {

       ; Step 1(a):
       0: <compound <rec bind 2> {
         0: <lit assertion>
         1: <compound <rec compound 2> {
           0: <compound <rec rec 2> {0: <ref label>, 1: <lit 1>}>,
           1: <compound <dict> {}>
         }>
       }>

       ; Step 1(b):
       1: <attenuate <ref observer> [
         <rewrite
          <compound <arr 1> {0: <bind v <_>>}>
          <ref v>>
       ]>

     }>>

    ; Step 2:
    <rewrite <bind n <_>> <ref n>>

  ]> ]

It reads:

  1. try matching <Observe L C>, where L is a symbol and C an embedded capability; if it does not match, skip the remainder of this step; otherwise, rewrite it into <Observe <bind assertion ⌜P⌝> f(C)>, where

    1. P is a pattern matching records of the form <L _>, and the quotation operator ⌜·⌝ quotes a pattern over assertions into a term conforming to the Pattern schema above; and

    2. f(C) “attenuates” C by attaching rewrites to it. Any assertion sent to C is required to be of the form [V], and is rewritten into just V.

  2. if the rewrite in step 1 didn’t apply then match anything; call it n; and rewrite it to itself.

The net effect is that when the simple chat example from novy-syndicate asserts

<Observe Present #!C>

the syndicate-rkt server actually sees

<Observe <bind assertion ⌜<Present _>⌝> #!f(C)>

and when syndicate-rkt replies with an actual concrete presence record, for example3

[<Present "Tony">]

the novy-syndicate client will actually receive just

<Present "Tony">

Cool huh?

Now, this works great for Present, which is unary, but not so well for the client’s subscription to Says, which is binary: <Says who what>. So our interoperability is limited here: the client only sees presence information from its peers, and the actual utterances sent get dropped on the floor for lack of an appropriate pattern at the syndicate-rkt dataspace. To fix this, we could include a more complex rewrite specification that treated Presence and Says subscriptions separately and explicitly, with the correct arity for each. But I’m done for now, and will focus on getting a proper dataspace implementation into novy-syndicate instead.

  1. We need a DSL for these rewrite specifications! I’m working on it. It’ll probably look like the existing Syndicate DSL syntax for patterns. 

  2. Here’s the whole capability, including an “oid” identifying the service to be accessed, the sequence of “caveats” rewriting and attenuating information flowing through the capability, and the signature proving the capability’s validity:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    
    <ref "syndicate" [[<or [
      <rewrite <compound <rec Observe 2> {
        0: <bind label Symbol>,
        1: <bind observer Embedded>
      }> <compound <rec Observe 2> {
        0: <compound <rec bind 2> {
          0: <lit assertion>,
          1: <compound <rec compound 2> {
            0: <compound <rec rec 2> {
              0: <ref label>,
              1: <lit 1>
            }>,
            1: <compound <dict> {}>
          }>
        }>,
        1: <attenuate <ref observer> [<rewrite <compound <arr 1> {0: <bind v <_>>}> <ref v>>]>
      }>>,
      <rewrite <bind n <_>> <ref n>>
    ]>]] #[1oCXyvdXylgpWRhgg0w+iw==]>
    

  3. The single-element list is there because the rewritten pattern included a single binding named assertion, so there’s a single value in the list of potentially-many values sent back to the subscriber. The simplified novy-syndicate patterns included exactly one implicit whole-assertion binding, and so the list wrapper is also implicit in the novy-syndicate variation, which is why it has to be explicitly removed to get interoperability here.