Git Product home page Git Product logo

core's Introduction

English | 中文

CORE

Package core is a service container that elegantly bootstrap and coordinate twelve-factor apps in Go.

Build Go Reference codecov Go Report Card Release

Background

The twelve-factor methodology has proven its worth over the years. Since its invention many fields in technology have changed, many among them are shining and exciting. In the age of Kubernetes, service mesh and serverless architectures, the twelve-factor methodology has not faded away, but rather has happened to be a good fit for nearly all of those powerful platforms.

Scaffolding a twelve-factor go app may not be a difficult task for experienced engineers, but certainly presents some challenges to juniors. For those who are capable of setting things up, there are still many decisions left to make, and choices to be agreed upon within the team.

Package core was created to bootstrap and coordinate such services.

Feature

Package core shares the common concerns of your application:

  • Configuration management: env, flags, files, etc.
  • Pluggable transports: HTTP, gRPC, etc.
  • Dependency injection
  • Job management: Cron, long-running, one-off commandline, etc.
  • Events and Queues
  • Metrics
  • Distributed Tracing
  • Database migrations and seedings
  • Distributed transactions
  • Leader election

Overview

Whatever the app is, the bootstrapping phase is roughly composed by:

  • Read the configuration from out of the binary. Namely, flags, environment variables, and/or configuration files.

  • Initialize dependencies. Databases, message queues, service discoveries, etc.

  • Define how to run the app. HTTP, RPC, command-lines, cronjobs, or more often mixed.

Package core abstracts those repeated steps, keeping them concise, portable yet explicit. Let's see the following snippet:

package main

import (
  "context"
  "net/http"

  "github.com/DoNewsCode/core"
  "github.com/DoNewsCode/core/observability"
  "github.com/DoNewsCode/core/otgorm"
  "github.com/gorilla/mux"
)

func main() {
  // Phase One: create a core from a configuration file
  c := core.New(core.WithYamlFile("config.yaml"))

  // Phase two: bind dependencies
  c.Provide(otgorm.Providers())

  // Phase three: define service
  c.AddModule(core.HttpFunc(func(router *mux.Router) {
    router.HandleFunc("/", func(writer http.ResponseWriter, request *http.Request) {
      writer.Write([]byte("hello world"))
    })
  }))

  // Phase Four: run!
  c.Serve(context.Background())
}

In a few lines, an HTTP service is bootstrapped in the style outlined above. It is simple, explicit, and to some extent, declarative.

The service demonstrated above uses an inline handler function to highlight the point. Normally, for real projects, we will use modules instead. The "module" in package Core's glossary is not necessarily a go module (though it can be). It is simply a group of services.

You may note that the HTTP service doesn't really consume the dependency. That's true.

Let's rewrite the HTTP service to consume the above dependencies.

package main

import (
  "context"
  "net/http"

  "github.com/DoNewsCode/core"
  "github.com/DoNewsCode/core/otgorm"
  "github.com/DoNewsCode/core/srvhttp"
  "github.com/gorilla/mux"
  "gorm.io/gorm"
)

type User struct {
  Id   string
  Name string
}

type Repository struct {
  DB *gorm.DB
}

func (r Repository) Find(id string) (*User, error) {
  var user User
  if err := r.DB.First(&user, id).Error; err != nil {
    return nil, err
  }
  return &user, nil
}

type Handler struct {
  R Repository
}

func (h Handler) ServeHTTP(writer http.ResponseWriter, request *http.Request) {
  encoder := srvhttp.NewResponseEncoder(writer)
  encoder.Encode(h.R.Find(request.URL.Query().Get("id")))
}

type Module struct {
  H Handler
}

func New(db *gorm.DB) Module {
  return Module{Handler{Repository{db}}}
}

func (m Module) ProvideHTTP(router *mux.Router) {
  router.Handle("/", m.H)
}

func main() {
  // Phase One: create a core from a configuration file
  c := core.New(core.WithYamlFile("config.yaml"))

  // Phase two: bind dependencies
  c.Provide(otgorm.Providers())

  // Phase three: define service
  c.AddModuleFunc(New)

  // Phase four: run!
  c.Serve(context.Background())
}

Phase three has been replaced by the c.AddModuleFunc(New). AddModuleFunc populates the arguments to New from dependency containers and add the returned module instance to the internal module registry.

When c.Serve() is called, all registered modules will be scanned for implemented interfaces. The module in the example implements interface:

type HTTPProvider interface {
	ProvideHTTP(router *mux.Router)
}

Therefore, the core knows this module wants to expose HTTP service and subsequently invokes the ProvideHTTP with a router. You can register multiple modules, and each module can implement one or more services.

Now we have a fully workable project, with layers of handler, repository, and entity. Had this been a DDD workshop, we would be expanding the example even further.

That being said, let's redirect our attention to other goodies package core has offered:

  • Package core natively supports multiplexing modules. You could start you project as a monolith with multiple modules, and gradually migrate them into microservices.

  • Package core doesn't lock in transport or framework. For instance, you can use go kit to construct your services, and bring in transports like gRPC, AMQP, thrift, etc. Non-network services like CLI and Cron are also supported. See all available interfaces a module can expose. You can also invent your own.

  • Package core also babysits the services after initialization. The duty includes but not limited to distributed tracing, metrics exporting, error handling, event-dispatching, and leader election.

Be sure to checkout the documentation section to learn more.

Documentation

Design Principles

  • No package global state.
  • Promote dependency injection.
  • Testable code.
  • Minimalist interface design. Easy to decorate and replace.
  • Work with the Go ecosystem rather than reinventing the wheel.
  • End to end Context passing.

Non-Goals

  • Tries to be a Spring, Laravel, or Ruby on Rails.
  • Tries to care about service details.
  • Tries to reimplement the functionality provided by modern platforms.

Works well with

Thanks for JetBrains OS licenses

core's People

Contributors

dependabot[bot] avatar ggxxll avatar lastking avatar lingwei0604 avatar nfangxu avatar reasno avatar renovate-bot avatar renovate[bot] avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

core's Issues

Lint issue for go-kit log

Should we replace github.com/go-kit/kit/log with github.com/go-kit/log again? Now there is the issue:

run golangci-lint
  Running [/home/runner/golangci-lint-1.41.1-linux-amd64/golangci-lint run --out-format=github-actions --disable errcheck --timeout 5m0s] in [] ...
  Error: SA1019: package github.com/go-kit/kit/log is deprecated: Use github.com/go-kit/log instead. (staticcheck)
  Error: SA1019: package github.com/go-kit/kit/log is deprecated: Use github.com/go-kit/log instead. (staticcheck)
  Error: SA1019: package github.com/go-kit/kit/log is deprecated: Use github.com/go-kit/log instead. (staticcheck)

cluster address in env

now we have REDIS_ADDR etc. supported, we can extend the syntax to support clusters addresses.

for example:

REDIS_ADDR=redis-1:7000,redis-2:7000,redis-3:7000

  • redis
  • es
  • kafka
  • etcd

test failures

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x10f45a8]

goroutine 131 [running]:
github.com/DoNewsCode/core/otkafka/processor.(*testHandlerA).Handle(0xc0000112a0, 0x14298e0, 0xc000228c80, 0x0, 0xc00020de50, 0xc000228c90, 0x437c95, 0xc00022a4e0)
	/home/runner/work/core/core/otkafka/processor/processor_test.go:35 +0xa8
github.com/DoNewsCode/core/otkafka/processor.(*handler).handle(0xc000119630, 0x14298e0, 0xc000228c80, 0x0, 0x0)
	/home/runner/work/core/core/otkafka/processor/processor.go:229 +0x205
github.com/DoNewsCode/core/otkafka/processor.(*Processor).ProvideRunGroup.func3(0x4a4d81, 0x0)
	/home/runner/work/core/core/otkafka/processor/processor.go:161 +0x8e
github.com/oklog/run.(*Group).Run.func1(0xc00022a720, 0xc0000afa00, 0xc0000afa20)
	/home/runner/go/pkg/mod/github.com/oklog/[email protected]/group.go:38 +0x35
created by github.com/oklog/run.(*Group).Run
	/home/runner/go/pkg/mod/github.com/oklog/[email protected]/group.go:37 +0x125
FAIL	github.com/DoNewsCode/core/otkafka/processor	3.225s

https://github.com/DoNewsCode/core/runs/2619277443?check_suite_focus=true

@GGXXLL

Drop support for wire

The support for wire-style injector never worked. Currently, we should focus on dig style injector and in the future revisit wire style injectors by creating separate constructors.

0.13 plan

  • metainfo support
  • goroutine pool rewrite

关于添加 RunnerProvider 的提议

RunnerProvider 提供注册 Server 以及 Crontab 的能力, 所有的 Runner 收集在 run.Group 中, 统一进行启动

  • 优势
    • 避免了框架中固定的单 http server 以及单 grpc server
    • 统一了 run.Group, 由使用者控制具体启动的 runner
    • 提供了多变 Server 能力, 可由使用者提供多个 http server 或 grpc server

Auto inject module struct?

type Module struct {
    di.In
    RedisClient redis.UniversalClient optional:"true" 
    DB          gorm.DB

}

c := core.Default()
c.AddModule(&Module{}) //  Auto inject fields via DI container

replace mongo logger with kit log

type Logger interface {
  Trace(message string, args ...Field)
  Debug(message string, args ...Field)
  Info(message string, args ...Field)
  Notice(message string, args ...Field)
  Warning(message string, args ...Field)
  Error(message string, args ...Field)
}


Distinct closers from various sources

Currently, there are 3 types of closers.

  • Closers from DI Provider.
  • Closers from Module's CloserProvider.
  • Cleanup function call in run group.

These three closer phases should be merged if we can not find a clear distinction.

config.Duration Unmarshal is error with koanf

Code:

func TestDuration_Unmarshal(t *testing.T) {
  type d struct {
    D Duration `json:"d" yaml:"d"`
  }

  val := "d: 1m5s\n"
  expected := d{Duration{5*time.Second + time.Minute}}

  k := koanf.New(".")
  k.Load(rawbytes.Provider([]byte(val)), yaml.Parser())

  r := d{}
  err := k.Unmarshal("", &r)
  assert.NoError(t, err)
}

Received error:

* 'D' expected a map, got 'string'

Refactor pool

  • factory mode
  • sync function for test
  • work stealing mode, keep one goroutine pool
  • recycle goroutine with time or jobCount

Add otelastic package

Support opentracing with elasticsearch and include a provider function. Take otredis and otmongo for example.

unierr: Is it necessary to allow err is nil?

Now, unierr.InvalidArgumentErr(nil) will panic. Hope allow err is nil. Is it necessary? such as:

func (h *Handler) Find(writer http.ResponseWriter, request *http.Request) {
	encoder := srvhttp.NewResponseEncoder(writer)
	id := request.URL.Query().Get("id")
	if id == "" {
		// return {"code":3,"message":"InvalidArgumentErr"}
		// unierr.InvalidArgumentErr(nil)

		// return {"code":3,"message":"id is required"}
		// unierr.InvalidArgumentErr(nil, "id is required")

		// return {"code":3,"message":"id is required"}
		encoder.EncodeError(unierr.InvalidArgumentErr(nil,"id is required"))
		return
	}
	m := map[string]interface{}{}
	h.gorm.Table("track").Where("id=?", id).Order("id desc").Find(&m)

	encoder.EncodeResponse(m)
}

run migrations in all databases

Currently, database migrate command will run the migration in default database. It is more friendly if database migrate runs ALL migrations in ALL databases.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.