Hi @spy16, I hope you're well and that things are returning to normal on your end.

As mentioned over in #26 and #27, I've been making heavy use of Sabre over the past few weeks in the context of Wetware, so I thought I'd share my thoughts on what works and what can be improved.

I'm well aware that the current runtime design has shown some limitations, and comfortable with the fact that parts of Wetware will have to be rewritten once we iron out the creases. My goal in publishing this is to:

• fill you in on the design requirements for Wetware (as per your comment over at #26 (comment)).
• follow up on #26 and hopefully start working on the Clojure-style runtime. I've made room in my schedule to lend a hand, but am looking to you for general direction. 😃

This report is structured as follows:

1. Context: A detailed description of Wetware, and how Sabre fits into the picture. It's a bit lenghty because I wanted to err on the side of completeness. Please forgive me as I slip into pitch-mode from time to time 😅!
2. The Good Parts of working with Sabre. This section highlights where Sabre provides a net gain in productivity, and was generally a delight to use.
3. Pain Points, Papercuts and Suggestions. This section highlights where Sabre could be improved. I've tried to be as specific as possible, and to link to Wetware source code where appropriate. Possible solutions to these issues are also discussed.
4. Miscellanea that are generally on my mind. These are basically issues that I expect to encounter in the mid-to-near term. They are important, but not urgent.

Context

Wetware is a distributed programming language for the cloud. Think Mesos + Lisp. Or Kubernetes + Lisp, if you prefer.

Wetware abstracts your datacenters and cloud environments into a single virtual cloud, and provides you with a simple yet powerful API for building fault-tolerant and elastic systems at scale.

It achieves its goals by layering three core technologies:

1. A Cloud Management Protocol

At its core, Wetware is powered by a simple peer-to-peer protocol that allows hosts to discover each other over the network, and assembles them into a fully-featured virtual cloud.

This virtual cloud is self-healing (i.e. antifragile), truly distributed (with no single point of failure), and comes with out-of-the box support for essential cloud services, including:

• Elastic process management using Virtual Machines (like Amazon EC2), Containers (like Amazon ECS), bare-metal UNIX processes, and even ultra-lightweight threads (goroutines).
• Persistent storage with support for binary blobs (like Amazon S3) and even structured data (maps, lists, strings, sets, etc.).
• Interprocess communication and orchestration via distributed PubSub (like Amazon SQS) and Channels (i.e. network-aware queues).

Wetware's Cloud Management Protocol works out-of-the-box, requires zero configuration, and features first-class support for hybrid and multicloud architectures.

2. A Distributed Data Plane

Unifying data across applications is a major challenge for current cloud architectures. Developers have to deal with dozens (sometimes even hundreds) of independent applications, each producing, encoding and serializing data in its own way. In traditional clouds, ETL and other data operations are time-consuming, error-prone and often require specialized stacks.

Wetware solves this problem by providing

1. High-performance, immutable datastructures for representing data across applications,
2. High-throughput protocols for efficiently sharing large datastructures across the network, and
3. An intuitive API for working concurrently with data in a cluster-wide, shared-memory space.

With Wetware's dataplane, you can coordinate millions of concurrent processes to work on terabyte-sized maps, sets, lists, etc. These immutable and wire-native datastructures protect you from concurrency bugs while avoiding overhead due to (de)serialization.

Lastly, Wetware's location-aware caching means you're always fetching data from the nearest source, avoiding egress costs in hybrid and multicloud environments.

3. A Dynamic Programming Language

The Wetware REPL is the primary means through which users interact with their virtual cloud, and the applications running on top of it. Unsurprisingly, this REPL is a Lisp dialect built with Sabre.

Let's walk through a few examples.

We can simulate a datacenter from the comfort of our laptop by starting any number of Wetware host processes:

# Start a host daemon.
#
# In production, you would run this command once on each
# datacenter host or cloud instance.
#
# In development, you can simulate a datacenter/cloud of
# n hosts by running this command n times.
$ ww start

Next, we start the Wetware REPL and instruct it to dial into the cluster we created above.

# The -dial flag auto-discovers and connects to an
# arbitrary host process.  By default, the shell runs
# locally, without connecting to a virtual cloud.
$ ww shell -dial

We're greeted with an interactive shell that looks like this:

Wetware v0.0.0
Copyright 2020 The Wetware Project
Compiled with go1.15 for darwin

ww »

From here, we can list the hosts in the cloud. If new hosts appear, or if existing hosts fail, these changes to the cluster will be reflected in subsequent calls to ls.

ww » (ls /) ;; list all hosts in the cluster
[
    /SV4e8BwRMPmShMPRcfTmpfTZQN7JQFaqzwt9g2wrF5bj
    /cie5uM1dAuQcbTEpHi4GKsghNVBk6H4orjVs6fmd16vV
]

The ls command returns a core.Vector, which contains a special, Wetware-specific data type: core.Path. These paths point to special locations called ww.Anchor. Anchors are cluster-wide, shared-memory locations. Any Wetware process can read or write to an Anchor, and the Wetware language provides synchronization primitives to deal with the hazards of concurrency and shared memory.

Anchors are organized hierarchically. The root anchor / represents the whole cluster, and its children represent physical hosts. Children of hosts are created dynamically upon access, and can contain any Wetware datatype.

;; Anchors are created transparently on access.
;; You can retrieve the value stored in an Anchor
;; by invoking its Path without arguments.
;; Anchors are empty by default.
ww » (/SV4e8BwRMPmShMPRcfTmpfTZQN7JQFaqzwt9g2wrF5bj/foo)
nil

;; Invoking a Path with a single argument stores
;; the value in the corresponding Anchor.
ww » (/SV4e8BwRMPmShMPRcfTmpfTZQN7JQFaqzwt9g2wrF5bj/foo
   ›   {:foo "foo value"
   ›    :bar 42
   ›    :baz ["hello" "world"] })
nil

;; The stored value is accessible by _any_ Wetware
;; process in the cluster.
;;
;; Let's fetch it from a goroutine running in the
;; remote host `cie5uM1...`
ww » (go /cie5uM1dAuQcbTEpHi4GKsghNVBk6H4orjVs6fmd16vV)
   ›   (print (/SV4e8BwRMPmShMPRcfTmpfTZQN7JQFaqzwt9g2wrF5bj/foo)))
nil

Why did this print nil? Because the form (print (/SV4e8.../foo)) was executed on the remote host cie5uM...! That is, the following things happened:

1. A network connection to cie5uM... was opened.
2. The list corresponding to the print function call was sent over the wire.
3. On the other side, cie5uM... received the list and evaluated it.
4. During evaluation, cie5uM... fetched the value from the Sv4e8.../foo Anchor and printed it.

If we were to check cie5uM...'s logs, we would see the corresponding output.

Important Technical Note: Wetware's datastructures are implemented using the Cap'n Proto schema language, meaning their in-memory representations do not need to be serialized in order to be sent across the network.

Our heavy reliance on capnp has implications for the design of varous Sabre interfaces, as discussed in part 2.

This concludes general introduction to Wetware.

While Wetware is very much in a pre-alpha stage, the foundational code for features 1 - 3 are in place, and the overall design has been validated. Now that we are leaving the proof-of-concept stage, developing the language (and its standard library) will be the focus of the next few months. For this reason, Sabre will continue play a central role in near-term development and I expect to split my development time roughly equally between Wetware and Sabre. As such, I'm hoping the following feedback can serve as a synchronization point between us, and motivate the next few PRs.

The Good Parts

(N.B.: I am exclusively developing on the reader branch, which is itself a branch of runtime.)

Overall, Sabre succeeds in its mission to be an "80% Lisp". The pieces fit together quite well, and most things are easily configurable. This last bit is particularly true of the runtime branch where I was able to write custom implementations for each atom/collection, as well as create some new, specialized datatypes. I have not encountered any fundamental design flaws, which is great!!

The REPL is a breeze to use, requring little effort to set up and configure. This is in large part thanks to your decision to make REPL (and Reader for that matter) concrete structs that hold interfaces internally, as opposed to declaring them as interface types. Doing so allows us to inject dependencies via functional arguments rather than re-writing a whole new implementation just to make minor changes to behavior. The result is a REPL that took me less time to set up than to write this paragraph, so this is a pattern we should definitely continue to exploit.

Relatedly, I think these few lines of code really showcase the ergonomics of functional options. They compose well, are discoverable & extensible, and visually cue the reader to the fact that the repl.New constructor is holding everything in the package together. I'm disproportionately pleased with the outcome.

Lastly, the built-in datatypes are very useful when developing one's own language because they serve as simple stubs until custom datastructures have been developed. In practice, this means I was able to develop other parts of the language in spite of the fact that e.g. Vectors had not yet been implemented in Wetware. It's hard to overstate not only how incredibly useful this is, and how much of that usefulness stems from the fact that Sabre is using native Go datastructures under the hood. Designing one's own language is quite hard, so every ounce of simplicity and familiarity is a godsend. I am strongly in favor of maintaining the existing implementations and not adding persistent datatypes for this reason. An exception might be made for LinkedList since the current implementation is dead-simple and shoe-horning a linked-list into a []runtime.Value is a bit ... backwards. In any case, Sabre really came through for me, here.

Pain Points, Papercuts & Suggestions

I want to stress that this section is longer than its predecessor not because there are more downsides than upsides in Sabre, but because there's always more to say about problems than non-problems! With that said, I've sorted the pain-points I've encountered into a few broad buckets:

1. Error handling
2. Design of container types
3. Reader design

Error Handling

By far the biggest issue I encountered was the handling of errors inside datastructure methods. Throughout our design discussion in #25, our thinking was (understandably) anchored to the existing implementations for Map, Vector, etc. Specifically, we assumed that certain operations (e.g. Count() int) could not result in errors. This turns out to have been an incorrect assumption.

As mentioned in the Context section above, Wetware's core datastructures are generated from a Cap'n Proto schema. As such, simple things such as calling an accessor function often return errors, including for methods like core.Vector.Count(). The result is that my code is quite panicky: Count, Conj, First and Next all panic.

While there are (quite convoluted) ways of avoiding these panics, I think there's a strong argument for changing the method signatures to return errors. Sabre is intended as a general-purpose build-your-own-lisp toolkit, and predicting what users will do with it is nigh impossible. For example, they may write datastructures implemented by SQL tables, which make RPC calls, or which interact with all manner of exotic code. As such, I think we should take the most general approach, which means returning errors almost everywhere.

Design of Container Types

This issue is pretty straightforward. I'd like to implement an analog to Clojure's conj that works on arbitrary containers. Currently, runtime.Vector.Conj returns a Vector, so I'm wondering how this might work. Do you think it's best to resort to reflection in such cases? Might it not be better to return runtime.Value from all Conj methods?

Reader Design

Despite being generally well-designed, there is room for improvement in reader.Reader.

Firstly, #27 adds the ability to modify the table of predefined symbols, which was essential in my case as I have custom implementations for Nil and Bool.

Secondly, relying on Reader.Container to build containers is not appropriate for all situations. The Container method reads a stream of values into a []runtime.Value, and returns it for further processing. In the case of Wetware's core.Vector, this is quite inefficient since:

1. I need to allocate a []runtime.Value.
2. I might need to grow the []runtime.Value, causing additional allocs, but I can't predict the size of the container ahead of time.
3. Once the []runtime.Value is instantiated, I have to loop through it and call core.VectorBuilder.Conj, which also allocates.

In order to avoid the penalty of double-allocation, I wrote readContainerStream, which applies a function to each value as it is decoded by the reader. The performance improvement is significant for large vectors, so I think we should add it as a public method to reader.Reader.

Thirdly, Wetware's reliance on Cap'n Proto means that I must implement custom numeric types. To make matters more complicated, I would like to add additional numeric types analogous to Go's big.Int, big.Float, and big.Rat. As such, I will need the ability to configure the reader's parsing logic for numerical values.

Currently, numerical parsing is hard-coded into the Reader. I suggest adding a reader option called WithNumReader (or perhaps WithNumMacro?) that allows users to configure this bit of logic. I expect this will also have repercussions on sabre.ValueOf, but it should be noted that this function is already outdated with respect to the new runtime datastructure interfaces.

Miscellanea

Lastly, a few notes/questions that are on my mind, but not particularly urgent:

1. The Position type seems very useful, but I'm not sure how it's meant to be used. Who is responsible for keeping it up-to-date, exactly? Any "use it this way" notes you might have would be helpful.
2. I don't quite understand the distinction between GoFunc, Fn and MultiFn. Best I can figure, GoFunc is used to call a native Go function from Sabre, while (Multi)Fn is meant to be dynamically instantiated by defn? From there, I assume MultiFn is used for multi-arity defn forms? (I think I might have answered my own question 😄)
3. I expect to start thinking about Macros in 6-8 weeks or so. Are there any major changes planned for the macro system, or can I rely on what already exists?
4. I'm going to tackle goroutine invokation within the next 2-3 weeks and will keep you appraised of my progress in #15. If you have any thoughts on the subject, I'm very much interested.

Conclusion

I hope you find it as useful to read this experience report as I have found it useful to write. I'm eager to discuss all of this at your earliest convenience, and standing by to help with implementation! 🙂

I have been working on an implementation of the 'case' construct and the package layout including how tests are maintained is causing a bit of pain. I suggest moving the project to a standard golang layout (placing packages in 'pkg') with tests in the same package as the code. As it is now I have to modify every file to create a PR that adds a single function.

Are there any plans to migrate parens' standard library to sabre? I see that some of the basic symbols are bound in the core package, but it's still missing a number of basics e.g. +.

Is this something sabre should provide?

Hi @spy16 ,

I just tried building a project that imports sabre using Go's module system:

$go run cmd/ww/main.go
build command-line-arguments: cannot load github.com/spy16/sabre/core: module github.com/spy16/sabre@latest found (v0.1.1), but does not contain package github.com/spy16/sabre/core

Do you know what the problem might be? I'm happy to push a fix.

It is currently impractical to build specials outside sabre (eg my case implementation in #22) because functions like 'analyze' are not exported.

• A MacroFn invokable value type (i.e., implements both sabre.Value and sabre.Invokable) which expands the macro definition and evaluates when invoked.
• A way to define the macro. defmacro
• A way to see the macro expansion without evaluating - macroexpand

Now that the repository has become somewhat large, i am thinking, May be it would be better to move slang into its own repository.

@lthibault Thoughts?

I'd like to be able to call a function in a separate goroutine using the following syntax:

(go
  (expensive-function arg1 arg2))

A few design considerations:

1. Separate goroutines might switch to different namespaces (by calling (ns 'some.other.namespace)).
2. I would like to implement something a Clojure-style value/reference distinction in my own project. In particular, I am considering something like Clojure's Var, which provides a thread-local symbol mapping facility.

What are your thoughts? I think goroutine support is a must-have, but any idea on how the above items might work? Thread-local storage is usually achieved with context.Context in Go, so I'm thinking that Sabre's goroutines should probably have some soft of context attached.

To be clear: I don't think Sabre should support item 2. It would force us to provide immutable data structures ( I think ... 🤔), which adds a lot of complexity and fails to leverage native Go structures like map and slice (which is kind of the point of Sabre). Rather, I think Sabre should be flexible enough that I can implement these in my own language.

Hi @spy16 ,

I've been making strides on my lisp project, and I'd like to implement something similar to Clojure's ns special form (minus the Java interop stuff, obviously).

My intuition is that this kind of functionality falls outside of Sabre's scope (similar to #3).

If that's the case, I'm not sure how to go about implementing this. Any suggestions?

I'm wondering how to go about implementing comments, eg how to make the interpreter (eg the RELP loop) ignore lines that start with (say) #

Sabre should be able to access member fields/methods of values bound in the scope.

Clojure uses (.MethodName target) or (.-FieldName target) for method and field accesses repsectively.

I am thinking this differentiation is not required in Go since Go types cannot have method+field of same name. But (.MemberName target) seems like a reasonable choice. In favor of keeping special handling to minimum, i am leaning towards exposing a special . function (. member-name target) which finds the member of the target with the given name and returns it,

Any thoughts on this ? @lthibault

Support for generic metadata on symbols and collections like in Clojure can be useful (for example to add documentation.)

Hi, how do I invoke an Invokable with List as its argument without the List is being evaluated? Here's a simple implementation to create a reduce function which requires the callback function to accept List and Int64 type.

package main

import (
	"fmt"

	"github.com/spy16/sabre"
)

const program = `
(print (reduce (fn [acc x] (acc.Cons x)) '() '(1 2 3)))
`

func main() {
	scope := sabre.New()
	scope.BindGo("sum", sum)
	scope.BindGo("print", fmt.Println)
	scope.BindGo("fn", sabre.Lambda)
	scope.BindGo("reduce", reduce)

	_, err := sabre.ReadEvalStr(scope, program)
	if err != nil {
		panic(err)
	}
}

func reduce(fn sabre.MultiFn, acc sabre.Value, list *sabre.List) (sabre.Value, error) {

	for _, v := range list.Values {
		applied, err := fn.Invoke(nil, acc, v)
		if err != nil {
			return nil, err
		}
		acc = applied
	}

	return acc, nil
}

This will produce this error.

$ go run main.go
panic: eval-error in '' (at line 0:0): cannot invoke value of type 'sabre.Int64'

goroutine 1 [running]:
main.main()
	/home/terra/Documents/system33/go/sabre/examples/simple/main.go:22 +0x194
exit status 2

I have finished migrating Wetware over to Slurp. We should archive this repository ASAP to avoid confusion.

Most of the Value types (except for Symbol, List, Vector, Set), return themselves on Eval().. It seems like a redundant requirement to have all Values implement the Eval() method when in most cases it's not required.. I am considering may be it would be simpler to remove this and make Eval() an optional interface.. We have 3 possibilities:

1. Remove the Value type altogether and deal with interface{} types.

2. Turn Value into a marker interface like below:

    // Remove Eval() from current Value interface and rename String() to Source()
    // to avoid ambiguity with commonly used fmt.Stringer
    type Value interface {
          Source() string 
    }

3. Not do anything and keep it as is.

If we choose 1 or 2, eval requirement can become an optional interface:

type Expr interface {
      Eval(scope Scope) (Value, error) // or (interface{}, error) based on 1 or 2
}

@lthibault Any thoughts on this ?

I've just stumbled upon the Joker language, which is an implementation of Clojure in Go. I figured it might be of interest as inspiration for design.

1. define a macro that returns [] on expansion : (def my-macro (macro* [& rest] []))
2. create a list that would invoke my-macro when evaluated: (def a-list '(my-macro 1 2 "hello"))
3. a-list represents the list (my-macro 1 2 "hello")
4. evaluate the list: (eval a-list)
5. check the value of a-list which is now result of macro expansion (i.e., (do [])) instead of its original value (my-macro 1 2 "hello").