We're starting to see a fair amount of drift between http.Server's used in Moov. This is intentional as overtime we've adapted servers, but we should consolidate somewhat. There's a lot of re-used features that should be in a common library.

• X-Request-Id / X-Idempotency-Key / X-User-Id extraction
• GET /ping route
• 4xx / 5xx error encoding, logging, etc
  • We can use runtime.Caller(n) here too.
• Metrics and Log timing (via http.ResponseWriter)
• CORS headers

It's not clear how base/log package should be used without looking at the implementation.

There should be a README to guide how users should use the logger.

We should add a timestamp to all logs using

logger = log.With(logger, "ts", log.DefaultTimestampUTC)

logger.Log("msg", "hello")

// Output:
// ts=2016-01-01T12:34:56Z msg=hello

The database package currently supports running a different set of migrations based on the database specified in the config (sqlite or mysql currently). We should support something similar, but have one set of migrations for prod and another set for staging. You can see the sqlite vs mysql migrations at work here:

PayGate Version: ``

What were you trying to do?

use database.InMemorySqliteConfig with database.NewAndMigrate

What did you expect to see?

connection to database created and database migrated

What did you see?

err:"no such table: items"

From the X-Idempotency-Key issue (moov-io/paygate#15 (comment)) we want to implement a redis implementation of idempotent.Recorder.

There's https://github.com/rafaeljusto/redigomock which looks like it works with https://github.com/gomodule/redigo. It's GPL'd though.

@wadearnold That project is GPL v2, can we use that? Probably not?
https://github.com/rafaeljusto/redigomock/blob/master/LICENSE

See: moov-io/paygate#21

For development, it's really helpful to inspect the database tables after a test run.

The existing implementation of ID() string only creates random hex strings. There's nothing to ensure uniqueness. We should migrate to something which does.

Currently our stack cares these strings are 40 characters long and of hex strings to strip them from Prometheus paths. (e.g. GET /users/fffffffffffffff/accounts to get-users-accounts)

Candidates

• https://github.com/gofrs/uuid

We've got a fairly good sqlite setup working for our apps and consolidating that would help keep apps uniform. I'm proposing a few new packages.

github.com/moov-io/base/database

// migrate runs our database migrations (defined at the top of this file)
// over a sqlite database it creates first.
// To configure where on disk the sqlite db is set SQLITE_DB_PATH.
//
// You use db like any other database/sql driver.
//
// https://github.com/mattn/go-sqlite3/blob/master/_example/simple/simple.go
// https://astaxie.gitbooks.io/build-web-application-with-golang/en/05.3.html
func Migrate(db *sql.DB, logger log.Logger) error {
	logger.Log("sqlite", "starting sqlite migrations...")
	for i := range migrations {
		row := migrations[i]
		res, err := db.Exec(row)
		if err != nil {
			return fmt.Errorf("migration #%d [%s...] had problem: %v", i, row[:40], err)
		}
		n, err := res.RowsAffected()
		if err == nil {
			logger.Log("sqlite", fmt.Sprintf("migration #%d [%s...] changed %d rows", i, row[:40], n))
		}
	}
	logger.Log("sqlite", "finished migrations")
	return nil
}

Perhaps one day we'll adopt flyway or a version table. For now this works.

github.com/moov-io/base/database/sqlite

func DBPath() string {
	path := os.Getenv("SQLITE_DB_PATH")
	if path == "" || strings.Contains(path, "..") {
		// set default if empty or trying to escape
		// don't filepath.ABS to avoid full-fs reads
		path = "paygate.db"
	}
	return path
}

func Connection(path string) (*sql.DB, error) {
	db, err := sql.Open("sqlite3", path)
	if err != nil {
		err = fmt.Errorf("problem opening sqlite3 file: %v", err)
		logger.Log("sqlite", err)
		return nil, err
	}
	return db, nil
}

func Version() string {
	sqliteVersion, _, _ := sqlite3.Version()
	return sqliteVersion	
}

github.com/moov-io/base/database/sqlite/testsqlite

type TestSqliteDB struct {
	db *sql.DB
	dir string // temp dir created for sqlite files
}

func (r *TestSqliteDB) Close() error {
	if err := r.db.Close(); err != nil {
		return err
	}
	return os.RemoveAll(r.dir)
}

// Connection returns a TestSqliteDB which can be used in tests
// as a clean sqlite database. All migrations are ran on the db before.
//
// Callers should call close on the returned *TestSqliteDB.
func Connection(migrator func (db *sql.DB, logger log.Logger) error) (*TestSqliteDB, error) {
	dir, err := ioutil.TempDir("", "test-sqlite")
	if err != nil {
		return nil, err
	}

	db, err := createSqliteConnection(filepath.Join(dir, "database.db"))
	if err != nil {
		return nil, err
	}

	logger := log.NewLogfmtLogger(ioutil.Discard)
	if err := migrator(db, logger); err != nil {
		return nil, err
	}
	return &TestSqliteDB{db, dir}, nil
}

Our server should be able to handle unexpected panics by:

• logging the error info and stack trace
• set the response code to 500

The Context interface and related methods should accept an interface{} over a string.

type Context interface {
	Context() map[string]string // -> map[string]interface{}
}

since writeValue will be called by go-kit/log to properly encode the string.

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• fix(deps): update module github.com/prometheus/client_golang to v1.20.4
• fix(deps): update module github.com/rickar/cal/v2 to v2.1.19
• fix(deps): update module google.golang.org/grpc to v1.66.2
• fix(deps): update module github.com/golang-migrate/migrate/v4 to v4.18.1
• fix(deps): update opentelemetry-go monorepo to v1.30.0 (go.opentelemetry.io/otel, go.opentelemetry.io/otel/exporters/otlp/otlptrace, go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc, go.opentelemetry.io/otel/exporters/stdout/stdouttrace, go.opentelemetry.io/otel/sdk, go.opentelemetry.io/otel/trace)
• Click on this checkbox to rebase all open PRs at once

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

Refactor CheckRoutingNumber validator from ach to base for reuse in wire, and icl.

calculateCheckDigit (checksum) and roundUp10 would need to be refactored in as well.

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Error type: undefined. Note: this is a nested preset so please contact the preset author if you are unable to fix it yourself.

There will be a need to re-use Parse Error / ErrorList for moov-io api's, packages, etc.

Copy ParseError / ErrorList from moov-io / ach into base library for re-use.

Put in root of github.com/moov-io/base and the it can be imported from other packages: base.ErrorList

Replace megacheck with staticcheck

Over in moov-io/ach#298 (and slack) we decided to have timestamp values in our API be ISO 8601. This is a common format usable by lots of languages, but Go doesn't seem to have support for it.

There are a few stale implementations laying around Github. Let's fine one, dust it off and incorporate it into our base repository.

Here's the structure: moov-io/ach#298 (comment)

See: https://github.com/moov-io/api/issues/33

cc: @alovak

We can do

logger.Set("orgIDs", log.String(fmt.Sprintf("[%s]",strings.Join(organizationIDs, ","))))

but a string array method would be helpful:

logger.Set("orgIDs", log.Strings(organizationIDs))

Both would output orgIDs=["id-1", "id-2", "id-3"]

Type signature for the valuer would be:

func Strings(vals []string) Valuer

Expose the bank time functionality as a utility as part of our rest endpoint.