ALPHA: Lile is currently considered "Alpha" in that things may change. Currently I am gathering feedback and will finalise Lile shortly to avoid breaking changes going forward.

Lile is a generator and set of tools/libraries to help you quickly create services that communicate via gRPC (REST via a gateway) and publish subscribe.

The primary focus of Lile is to remove the boilerplate when creating new services by creating a basic structure, test examples, Dockerfile, Makefile etc.

As well a simple service generator Lile extends the basic gRPC server to include pluggable options like metrics (e.g. Prometheus), tracing (e.g. Zipkin) and PubSub (e.g. Google PubSub).

Chat to me on Slack on the Gopher Slack channel #lile

Installing Lile is easy, using go get you can install the cmd line app to generate new services and the required libraries.

$ go get -u github.com/lileio/lile/...

You will also need Google's Protocol Buffers installed.

To generate a new service, run lile new with a short folder path.

Lile is smart enough to evaluate username/service to a full $GOPATH directory and defaults to github.com.

$ lile new lileio/users

• Installation
• Creating a Service
• Service Definition
• Generating RPC Methods
• Running and Writing Tests
• Using the Generated cmds
• Adding your own cmds
• Exposing Prometheus Metrics
• Publish & Subscribe
• Publishing an Event
• Automatically Publishing Events
• Subscribing to Events
• Tracing

First, you need to have a working Go installation, once you have Go installed you can then install Lile.

Installing Lile is easy, using go get you can install the cmd line app to generate new services and the required libraries.

$ go get github.com/lileio/lile/...

You will also need Google's Protocol Buffers installed.

On MacOS you can simply brew install protobuf

Lile comes with a 'generator' to quickly generate new Lile services.

Lile follows Go's conventions around $GOPATH (see How to Write Go) and is smart enough to parse your new service's name to create the service in the right place.

If your Github username was lileio and you wanted to create a new service for posting to Slack you might use the following command.

lile new lileio/slack

This will create a project in $GOPATH/src/github.com/lileio/slack

Lile services mainly speak gRPC and therefor use uses protocol buffers as the Interface Definition Language (IDL) for describing both the service interface and the structure of the payload messages. It is possible to use other alternatives if desired.

I highly recommend reading the Google API Design docs for good advice around general naming of RPC methods and messages and how they might translate to REST/JSON if needed.

An example of a service definition can be found in the Lile account_service

service AccountService {
  rpc List (ListAccountsRequest) returns (ListAccountsResponse) {}
  rpc GetById (GetByIdRequest) returns (Account) {}
  rpc GetByEmail (GetByEmailRequest) returns (Account) {}
  rpc AuthenticateByEmail (AuthenticateByEmailRequest) returns (Account) {}
  rpc GeneratePasswordToken (GeneratePasswordTokenRequest) returns (GeneratePasswordTokenResponse) {}
  rpc ResetPassword (ResetPasswordRequest) returns (Account) {}
  rpc ConfirmAccount (ConfirmAccountRequest) returns (Account) {}
  rpc Create (CreateAccountRequest) returns (Account) {}
  rpc Update (UpdateAccountRequest) returns (Account) {}
  rpc Delete (DeleteAccountRequest) returns (google.protobuf.Empty) {}
}

By default Lile will create a example RPC method and a simple messages for request and response.

syntax = "proto3";
option go_package = "github.com/lileio/slack";
package slack;

message Request {
  string id = 1;
}

message Response {
  string id = 1;
}

service Slack {
  rpc Read (Request) returns (Response) {}
}

Let's modify this to be a real service and add our own method.

We're going to create an Announce method that will announce a message to a Slack room.

We're assuming that the Slack team and authentication is already handled by the services configuration, so a user of our service only needs to provide a room and their message. The service is going to send the special Empty response, since we only need to know if an error occurred and don't need to know anything else.

Our proto file now looks like this..

syntax = "proto3";
option go_package = "github.com/lileio/slack";
import "google/protobuf/empty.proto";
package slack;

message AnnounceRequest {
  string channel = 1;
  string msg = 2;
}

service Slack {
  rpc Announce (AnnounceRequest) returns (google.protobuf.Empty) {}
}

We now run the protoc tool with our file and the Lile method generator plugin.

protoc -I . slack.proto --lile-server_out=. --go_out=plugins=grpc:$GOPATH/src

Handily, Lile provides a Makefile with each project that has a proto build step already configured. So we can just run that.

make proto

We can see that Lile will create two files for us in the server directory.

$ make proto
protoc -I . slack.proto --lile-server_out=. --go_out=plugins=grpc:$GOPATH/src
2017/07/12 15:44:01 [Creating] server/announce.go
2017/07/12 15:44:01 [Creating test] server/announce_test.go

Let's take a look at the announce.go file that's created for us.

package server

import (
    "errors"

    "github.com/golang/protobuf/ptypes/empty"
    "github.com/lileio/slack"
    context "golang.org/x/net/context"
)

func (s SlackServer) Announce(ctx context.Context, r *slack.AnnounceRequest) (*empty.Empty, error) {
  return nil, errors.New("not yet implemented")
}

We can now fill in this generated method with the correct implementation. But let's start with a test!

When you generate an RPC method with Lile a counterpart test file is also created. For example, given our announce.go file, Lile will create announce_test.go in the same directory.

This should look something like the following..

package server

import (
	"testing"

	"github.com/lileio/slack"
	"github.com/stretchr/testify/assert"
	context "golang.org/x/net/context"
)

func TestAnnounce(t *testing.T) {
	ctx := context.Background()
	req := &slack.AnnounceRequest{}

	res, err := cli.Announce(ctx, req)
	assert.Nil(t, err)
	assert.NotNil(t, res)
}

You can now run the tests using the Makefile and running make test...

$ make test
=== RUN   TestAnnounce
--- FAIL: TestAnnounce (0.00s)
        Error Trace:    announce_test.go:16
        Error:          Expected nil, but got: &status.statusError{Code:2, Message:"not yet implemented", Details:[]*any.Any(nil)}
        Error Trace:    announce_test.go:17
        Error:          Expected value not to be nil.
FAIL
coverage: 100.0% of statements
FAIL    github.com/lileio/slack/server  0.011s
make: *** [test] Error 2

Our test failed because we haven't implemented our method, at the moment we're returning an error of "unimplemented" in our method.

Let's implement the Announce method in announce.go, here's an example using nlopes' slack library.

package server

import (
	"google.golang.org/grpc"
	"google.golang.org/grpc/codes"

	"github.com/golang/protobuf/ptypes/empty"
	"github.com/lileio/slack"
	sl "github.com/nlopes/slack"
	context "golang.org/x/net/context"
)

var api = sl.New(os.Getenv("SLACK_TOKEN"))

func (s SlackServer) Announce(ctx context.Context, r *slack.AnnounceRequest) (*empty.Empty, error) {
	_, _, err := api.PostMessage(r.Channel, r.Msg, sl.PostMessageParameters{})
	if err != nil {
		return nil, grpc.Errorf(codes.Internal, err.Error())
	}

	return &empty.Empty{}, nil
}

Let's fill out our testing request and then run our tests again..

package server

import (
	"testing"

	"github.com/lileio/slack"
	"github.com/stretchr/testify/assert"
	context "golang.org/x/net/context"
)

func TestAnnounce(t *testing.T) {
	ctx := context.Background()
	req := &slack.AnnounceRequest{
		Channel: "@alex",
		Msg:     "hellooo",
	}

	res, err := cli.Announce(ctx, req)
	assert.Nil(t, err)
	assert.NotNil(t, res)
}

Now if I run the tests with my Slack token as an ENV variable, I should see a passing test!

$ alex@slack: SLACK_TOKEN=zbxkkausdkasugdk make test
go test -v ./... -cover
?       github.com/lileio/slack [no test files]
=== RUN   TestAnnounce
--- PASS: TestAnnounce (0.32s)
PASS
coverage: 75.0% of statements
ok      github.com/lileio/slack/server  0.331s  coverage: 75.0% of statements
?       github.com/lileio/slack/slack   [no test files]
?       github.com/lileio/slack/slack/cmd       [no test files]
?       github.com/lileio/slack/subscribers     [no test files]

Lile generates a cmd line application when you generate your service. You can extend the app with your own cmds or use the build in cmds to run the service.

Runing the cmd line app without any arguments will print the generated help.

For example go run orders/main.go

Running up will run the RPC server.

Running subscribe will listen to pubsub events with your subscribers.

Running up will run both the RPC server and the pubsub subscribers.

To add your own cmd, you can use the built in generator from cobra which powers Lile's cmds

$ cd orders
$ cobra add import

You can not edit the file generated to create your cmd, cobra will automatically add the cmd to the help.

By default Lile collects Prometheus metrics and exposes them at :9000/metrics.

If your service is running, you can use cURL to preview the Prometheus metrics

$ curl :9000/metrics

You should see something along the lines of..

# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
go_gc_duration_seconds{quantile="0.5"} 0
go_gc_duration_seconds{quantile="0.75"} 0
go_gc_duration_seconds{quantile="1"} 0
go_gc_duration_seconds_sum 0
go_gc_duration_seconds_count 0
...
...

The Lile Prometheus metrics implementation plugs itself into gPRC using an interceptor using go-grpc-prometheus providing metrics such as;

grpc_server_started_total
grpc_server_msg_received_total
grpc_server_msg_sent_total
grpc_server_handling_seconds_bucket

For more on using Prometheus, collecting and graphing these metrics, see Getting started at Prometheus.io

And see useful query examples for examples of useful gRPC Prometheus queries.

Whilst most services will communicate predominantly via RPC, Lile provides a library for doing Publish & Subscribe (or Pub Sub) communication.

This is particularly helpful when developing a service that needs to be updated or do some work when another service has performed an action, but you don't want to hold up the request.

Publishers are loosely coupled to subscribers, and need not even know of their existence. Most Lile services will already provide events you can hook into, but you can easily add events to your own service. You can either manually publish events or use the gRPC middleware to automatically publish an event when a RPC method is called.

Lile's Pub Sub is based on "at least once" delivery of message per subscriber. In other words, given an account_service (publisher) that publishes the event account_created, if multiple instances of an email_service (subscriber) and fraud_detection_service (subscriber) are running, only one instance of each email_service and fraud_detection_service will each receive a message.

If Lile pubsub is configured (which happens via env vars automatically or manually) then only a simple call to Publish is required.

Here's an example Get method on an "orders" service.

func (s OrdersServer) Get(ctx context.Context, r *orders.GetRequest) (*orders.GetResponse, error) {
	o, err := getOrder(r.Id)
	if err != nil {
		return nil, err
	}

	res := &orders.GetResponse{
		Id:   o.Id,
		Name: o.Name,
	}
	pubsub.Publish(ctx, "orders_service.Get", res)
	return res, nil
}

Publish takes a context.Context for tracing and metrics, a topic name and a proto.Msg, which is any object that can be serialised to protobuf.

You can automatically publish events when a gRPC method is called using lile's gRPC middleware.

In our main.go we can add the pubsub interceptor, mapping our gRPC methods to our pubsub topics.

The interceptor will automatically publish the gRPC response to that topic, or nothing on error.

lile.AddPubSubInterceptor(map[string]string{
	"Create": "account_service.created",
	"Update": "account_service.updated",
})

Lile generates projects by default with a subscribers.go file with some basic setup to subscribe to events.

Subscribers conform to the lile.Subscriber interface which has a special Setup event for subscribing to events from any topic.

type OrdersServiceSubscriber struct{}

func (s *OrdersServiceSubscriber) Setup(c *pubsub.Client) {
	c.On("shipments.updated", s.ShipmentUpdate, 30*time.Second, true)
}

func (s *OrdersServiceSubscriber) ShipmentUpdate(sh *shipments.Shipment) {
	// do something with sh
}

Functions that listen to topics can take anything that conforms to pubsub's Handler interface

Protobuf messages are automatically decoded.

Lile has built in tracing that reports to opentracing compatible tracers set to the GlobalTracer and by default, Lile with report all gRPC methods and pubsub publish/subscribing actions.

To have Lile send all tracing events to Zipkin via HTTP, set the ZIPKIN_SERVICE_HOST ENV variable to the DNS name of your Zipkin service. Kubernetes will expose the ZIPKIN_SERVICE_HOST automatically to a container if there is service already running named zipkin.

Stackdriver provide a zipkin-collector image that will listen for Zipkin traces, convert and send them to Stackdriver. It's quite awesome if you're looking for tracing but don't want to maintain Zipkin! See the Google Cloud Tracing docs for more