Git Product home page Git Product logo

embedded-postgres's Introduction

Godoc Coverage Status Build Status Build Status Go Report Card

embedded-postgres

Run a real Postgres database locally on Linux, OSX or Windows as part of another Go application or test.

When testing this provides a higher level of confidence than using any in memory alternative. It also requires no other external dependencies outside of the Go build ecosystem.

Heavily inspired by Java projects zonkyio/embedded-postgres and opentable/otj-pg-embedded and reliant on the great work being done by zonkyio/embedded-postgres-binaries in order to fetch precompiled binaries from Maven.

Installation

embedded-postgres uses Go modules and as such can be referenced by release version for use as a library. Use the following to add the latest release to your project.

go get -u github.com/fergusstrange/embedded-postgres

How to use

This library aims to require as little configuration as possible, favouring overridable defaults

Configuration Default Value
Username postgres
Password postgres
Database postgres
Version 15.3.0
Encoding UTF8
Locale C
Version 15.3.0
CachePath $USER_HOME/.embedded-postgres-go/
RuntimePath $USER_HOME/.embedded-postgres-go/extracted
DataPath $USER_HOME/.embedded-postgres-go/extracted/data
BinariesPath $USER_HOME/.embedded-postgres-go/extracted
BinaryRepositoryURL https://repo1.maven.org/maven2
Port 5432
StartTimeout 15 Seconds
StartParameters map[string]string{"max_connections": "101"}

The RuntimePath directory is erased and recreated at each Start() and therefore not suitable for persistent data.

If a persistent data location is required, set DataPath to a directory outside RuntimePath.

If the RuntimePath directory is empty or already initialized but with an incompatible postgres version, it will be removed and Postgres reinitialized.

Postgres binaries will be downloaded and placed in BinaryPath if BinaryPath/bin doesn't exist. BinaryRepositoryURL parameter allow overriding maven repository url for Postgres binaries. If the directory does exist, whatever binary version is placed there will be used (no version check is done).
If your test need to run multiple different versions of Postgres for different tests, make sure BinaryPath is a subdirectory of RuntimePath.

A single Postgres instance can be created, started and stopped as follows

postgres := embeddedpostgres.NewDatabase()
err := postgres.Start()

// Do test logic

err := postgres.Stop()

or created with custom configuration

logger := &bytes.Buffer{}
postgres := NewDatabase(DefaultConfig().
Username("beer").
Password("wine").
Database("gin").
Version(V12).
RuntimePath("/tmp").
BinaryRepositoryURL("https://repo.local/central.proxy").
Port(9876).
StartTimeout(45 * time.Second).
StartParameters(map[string]string{"max_connections": "200"}).
Logger(logger))
err := postgres.Start()

// Do test logic

err := postgres.Stop()

It should be noted that if postgres.Stop() is not called then the child Postgres process will not be released and the caller will block.

Examples

There are a number of realistic representations of how to use this library in examples.

Credits

Contributing

View the contributing guide.

embedded-postgres's People

Contributors

alecsammon avatar bdoherty-tovala avatar bearsh avatar bulanovk avatar chris-rock avatar dependabot[bot] avatar der-eismann avatar djcass44 avatar dor-utila avatar eldarkurbanov avatar fergusstrange avatar goguson avatar mackinleysmith avatar mboulton-fathom avatar michaelbrewer avatar mishas avatar movit avatar nathanfranke avatar naveensrinivasan avatar onrik avatar samuelmarks avatar sharmapankaj2512 avatar steinfletcher avatar tachiniererin avatar zzzfelix avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

embedded-postgres's Issues

Get error "unable to extract postgres archive: xz: data is truncated or corrupt" in GitLab pipeline.

When starting the postgres daemon, I get the following error:

unable to extract postgres archive: xz: data is truncated or corrupt

This never happens locally, but only in a GitLab pipeline. Furthermore, it only seems to happen on parallel runs of the postgres daemon -- I start one postgres daemon on a different port for each test function.

We start the postgres daemon in the following way

runtimePath := filepath.Join(projectDir, ".dbtest", t.Name())
pg := embeddedpostgres.NewDatabase(embeddedpostgres.DefaultConfig().
    Username(user).
    Password(pw).
    Database(dbname).
    Version(embeddedpostgres.V14).
    RuntimePath(runtimePath).
    DataPath(fmt.Sprintf("%s/data", runtimePath)).
    Logger(os.Stdout).
    Port(port).
    StartTimeout(45 * time.Second))

if err := pg.Start(); err != nil {
    t.Fatalf("Failed to start embedded postgres daemon: %s\n", err)
}

I have also experimented with the BinariesPath option, but without any luck.

Any ideas of what the real issue is and how one might go about resolving it?

Run as a different user (non-root)

Hi,

First off, thanks for an awesome project! We're using embedded-postgres over at coder/coder as a tool to quickly get up-and-running and it's been great.

We'd love to be able to launch embedded-postgres as a different (non-root) user to further improve the experience for users running Docker on macOS.

I'll spare you the full details as to why we'd need this functionality (for the curious coder/coder#5263), but TL;DR, we'd like to avoid extra steps where users have to change Docker socket permissions which has side-effects outside the container. The easiest way to achieve this is to run as root. Being able to run embedded-postgres as a different user would allow us to keep control within the main process running in the container.

I saw that there was already a PR for this functionality, #81, but I thought it would be worthwhile to open an issue for it to show our support for the feature.

Ours is just one use-case though, and I understand if you still feel it's not a great fit for your project. Cheers.

Expose an interface to retrieve binaries from non-network sources

I'd like to run some tests using embedded-postgres in an environment which runs tests under network isolation. It should be possible to pre-fetch the requisite binary distributions and then provide them to embedded postgres on disk rather than only via the network.

Looking at the current interfaces, perhaps a remote download interface could be exposed to allow for a custom downloader implementation to be provided which only fetches from a local disk cache rather than from the internet.

Platform Tests

Create a separate test suite module to test on Windows, OSX and Linux using Actions running a quick smoke test of one of each of the major available Postgres versions.

Allow specifying a custom RemoteFetchStrategy/CacheLocator ?

Not sure if I'm missing something from reading the source code, but it seems that although these two types are exported it's impossible to actually use a custom implementation because these fields are private on the EmbeddedPostgres struct.

My use case is that I want to be able to use a pre-downloaded postgres binary (or at least download from a different URL that I control).

CVE-2020-16845 still occurs on dependant products

Hello @naveensrinivasan and @fergusstrange ,

I just updated embedded-postgres to v1.11.0, and I still get the github.com/ulikunitz/[email protected] dependency in my go.sum (after go mod tidy).

I think the reason for this is, that in #42 , you've used a replace directive, that only does something local, instead of a require (// indirect) directive that would affect projects depending on yours.
From https://golang.org/ref/mod#go-mod-file-replace:

replace directives only apply in the main module’s go.mod file and are ignored in other modules. See Minimal version selection for details.

I think the following fix will do the trick:

- replace github.com/ulikunitz/xz => github.com/ulikunitz/xz v0.5.8
+ require github.com/ulikunitz/xz v0.5.10 // indirect

(Please also note, that there's no reason to use v0.5.8 when v0.5.10 is available)

Postgres extensions

Is it possible to install and use Postgres extensions? if yes, how? I need AGE specifically.

Unable to test with github actions

I have set up in my local some test and been able to run them. Works fine. Now I am trying to run them on github actions but I get an error. I have check the code it seems an issue when trying to get the .txz file inside the jar.

I get this error panic: error fetching postgres: cannot find binary in archive retrieved from https://repo1.maven.org/maven2/io/zonky/test/postgres/embedded-postgres-binaries-linux-amd64/12.8.0/embedded-postgres-binaries-linux-amd64-12.8.0.jar [recovered] panic: error fetching postgres: cannot find binary in archive retrieved from https://repo1.maven.org/maven2/io/zonky/test/postgres/embedded-postgres-binaries-linux-amd64/12.8.0/embedded-postgres-binaries-linux-amd64-12.8.0.jar

This is my github actions yaml.

on: [pull_request,push]
jobs:
  Explore-GitHub-Actions:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-go@v2
        with:
          go-version: '^1.13.1' # The Go version to download (if necessary) and use.}
      - go get -u github.com/fergusstrange/embedded-postgres
      - run: go test -v
        env:
          DB_HOST: localhost
          DB_PORT: 5432
          DB_USER: postgres
          DB_NAME: postgres
          DB_PASSWORD: postgres
          LC_CTYPE: UTF-8

I have no idea on what can be wrong. I have tested that the url can be downloaded and that seems to be ok. I have checked the actions you define in this proyect to test it and it doesn´t look very different from mine.

Error when DataBase is Specified in Config

timed out waiting for database to become available the following error consistently gets returned when trying to start the DB. This only happens when passing a custom database name in the config.

example

	dbName := "test_" + RandStringRunes(5)
	port, err := freeport.GetFreePort()
	if err != nil {
		t.Fatal(err)
	}
	user, password := "postgres", "postgres"
	database := embeddedpostgres.NewDatabase(embeddedpostgres.DefaultConfig().
		Username(user).
		Password(password).
		Port(uint32(port)).
		Version(embeddedpostgres.V14).
		Database(dbName).
		Logger(nil))

commenting out the custom database allows the db to be started up without time-ing out.

Logger parametr does'twork correctly

Hey... I figured out that when I do something like
.......
logger, _ := zap.NewProduction()
defer logger.Sync()
logger.Info("failed to fetch URL",
// Structured context as strongly typed Field values.
zap.String("url", url),
zap.Int("attempt", 3),
zap.Duration("backoff", time.Second),
)
....
w := &zapio.Writer{Log: logger}

......
database := embeddedpostgres.NewDatabase(embeddedpostgres.DefaultConfig().
Version(embeddedpostgres.V12).
Port(Port).
RuntimePath(runtimePath).
StartTimeout(CACHE_DB_STARTUP_DELLAY_SECONDS).
Database(DbName).
Password(DbUsername).
Username(DbPasswd).
Logger(w))

the new io.writer for the instance is used (I see a new zap formatting applied for all messages) but by the end of the database startup it hangs and I need to kill the app.

i tried the lorgus and standard go logger and all of them were hanging after the database started up. The reason to use this embedded Postgre database is to get all logging coming from one place otherwise we can easily run the postgres instance at the pod and make it "embedded" so I think logging not working correctly is the ISSUE.

List of all supported versions? - From Maven?

Supported versions are available here: https://mvnrepository.com/artifact/io.zonky.test.postgres/embedded-postgres-binaries-bom

Considering we're in a AOT compiled language, I want to write a CLI that:

  • Lists all installable PostgreSQL versions [that work on current OS];
  • List all installed PostgreSQL versions;
  • Installs specific version (e.g., 15 to install latest 15 version or 15.2.0 for specific version or latest for latest)

Now it would be nice if the CLI worked either without any HTTP calls required for it to fail with "version not available" error, which could be done by generate a large list | # of vars like: https://github.com/fergusstrange/embedded-postgres/blob/baa0dbf/config.go#L125-L133

OR if you had an initial large list | # of vars like above, and if the version specified by the user erred then an HTTP request to Maven would be made to see if that version is now available; and otherwise raise the "version not available" error.

EDIT: Here's an initial large list

var POSTGRES_VERSIONS [117]embeddedpostgres.PostgresVersion = [117]embeddedpostgres.PostgresVersion{
	"15.2.0",
	"15.1.0",
	"15.0.0",
	"14.7.0",
	"14.6.0",
	"14.5.0",
	"14.4.0",
	"14.3.0",
	"14.2.0",
	"14.1.0",
	"14.0.0",
	"13.10.0",
	"13.9.0",
	"13.8.0",
	"13.7.0",
	"13.6.0",
	"13.5.0",
	"13.4.0",
	"13.3.0",
	"13.2.0",
	"13.1.0-1",
	"13.1.0",
	"13.0.0",
	"12.14.0",
	"12.13.0",
	"12.12.0",
	"12.11.0",
	"12.10.0",
	"12.9.0",
	"12.8.0",
	"12.7.0",
	"12.6.0",
	"12.5.0-1",
	"12.5.0",
	"12.4.0",
	"12.3.0",
	"12.2.0",
	"12.1.0-1",
	"12.1.0",
	"12.0.0",
	"11.19.0",
	"11.18.0",
	"11.17.0",
	"11.16.0",
	"11.15.0",
	"11.14.0",
	"11.13.0",
	"11.12.0",
	"11.11.0",
	"11.10.0-1",
	"11.10.0",
	"11.9.0",
	"11.8.0",
	"11.7.0",
	"11.6.0-1",
	"11.6.0",
	"11.5.0",
	"11.4.0",
	"11.3.0",
	"11.2.0",
	"11.1.0",
	"11.0.0",
	"10.23.0",
	"10.22.0",
	"10.21.0",
	"10.20.0",
	"10.19.0",
	"10.18.0",
	"10.17.0",
	"10.16.0",
	"10.15.0-1",
	"10.15.0",
	"10.14.0",
	"10.13.0",
	"10.12.0",
	"10.11.0-1",
	"10.11.0",
	"10.10.0",
	"10.9.0",
	"10.8.0",
	"10.7.0",
	"10.6.0",
	"10.5.0-1",
	"10.5.0",
	"10.4.0",
	"9.6.24",
	"9.5.25",
	"9.5.24-1",
	"9.5.24",
	"9.5.23",
	"9.5.22",
	"9.5.21",
	"9.5.20-1",
	"9.5.20",
	"9.5.19",
	"9.5.18",
	"9.5.17",
	"9.5.16",
	"9.5.15",
	"9.5.14-1",
	"9.5.14",
	"9.5.13",
	"9.4.26",
	"9.4.25-1",
	"9.4.25",
	"9.4.24",
	"9.4.23",
	"9.4.22",
	"9.4.21",
	"9.4.20",
	"9.4.19-1",
	"9.4.19",
	"9.4.18",
	"9.3.25",
	"9.3.24-1",
	"9.3.24",
	"9.3.23"}

EDIT: CLI is now available in working order @ https://github.com/offscale/postgres-version-manager-go

Return an error instead of deleting data on upgrade

When upgrading the version of PostgreSQL, all data was deleted. This seems like undesired user behavior because there's not a clear warning message that everything will be reset.

reuseData := dataDirIsValid(ep.config.dataPath, ep.config.version)
if !reuseData {
if err := ep.cleanDataDirectoryAndInit(); err != nil {
return err
}
}

I think returning an error due to a data version mismatch PostgreSQL won't start is more expected, and causes less risk of users just losing all of their data.

Panic leads to stuck process/process not being closed.

Hey, awesome work so far, I tried the library, the issue I had is, if the test code is panicking the process will never be killed, even using recover did not work for me. The test were just stuck. And I had a postgres process blocking the configured port which needs to be killed manually.

Any advice?

What I tried from a setup point of view is the following, but the test still got stuck on panic and I had a process still running:

func TestMain(m *testing.M) {
	testPostgresInstance = embeddedpostgres.NewDatabase()
	if err := testPostgresInstance.Start(); err != nil {
		panic(err)
	}
	defer func() {
		if err := testPostgresInstance.Stop(); err != nil {
			panic(err)
		}
	}()
	m.Run()
	if r := recover(); r != nil {
		if err := testPostgresInstance.Stop(); err != nil {
			panic(err)
		}
	}
}

Custom logger prevent .Start() function from returning

Hi, I'm trying to pass a custom logger to the embedded postgres,
for some reason doing so causes the process to not return after .Start() function of embedded postgres.
I'm posting the code and it's output.

package main

import (
	"fmt"
	"log"

	embeddedpostgres "github.com/fergusstrange/embedded-postgres"
)

func main() {
	postgres := embeddedpostgres.NewDatabase(embeddedpostgres.DefaultConfig().Logger(customLogger{}))
	err := postgres.Start()
	if err != nil {
		log.Fatal(err)
	}
	postgres.Stop()
}

type customLogger struct{}

func (cl customLogger) Write(p []byte) (n int, err error) {
	fmt.Print(string(p))
	return len(p), nil
}
The files belonging to this database system will be owned by user "xxx".
This user must also own the server process.

The database cluster will be initialized with locales
  COLLATE:  C
  CTYPE:    UTF-8
  MESSAGES: C
  MONETARY: C
  NUMERIC:  C
  TIME:     C
The default database encoding has accordingly been set to "UTF8".
initdb: could not find suitable text search configuration for locale "UTF-8"
The default text search configuration will be set to "simple".

Data page checksums are disabled.

creating directory /Users/xxx/.embedded-postgres-go/extracted/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Asia/yyy
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    /Users/xxx/.embedded-postgres-go/extracted/bin/pg_ctl -D /Users/xxx/.embedded-postgres-go/extracted/data -l logfile start

2021/06/20 13:29:26 /Users/xxx/.embedded-postgres-go/extracted/bin/pg_ctl start -w -D /Users/xxx/.embedded-postgres-go/extracted/data -o "-p 5432"
waiting for server to start....2021-06-20 13:29:27.060 IDT [81800] LOG:  starting PostgreSQL 12.6 on x86_64-apple-darwin, compiled by Apple LLVM version 6.0 (clang-600.0.54) (based on LLVM 3.5svn), 64-bit
2021-06-20 13:29:27.063 IDT [81800] LOG:  listening on IPv6 address "::1", port 5432
2021-06-20 13:29:27.063 IDT [81800] LOG:  listening on IPv4 address "127.0.0.1", port 5432
2021-06-20 13:29:27.064 IDT [81800] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2021-06-20 13:29:27.074 IDT [81801] LOG:  database system was shut down at 2021-06-20 13:29:26 IDT
2021-06-20 13:29:27.077 IDT [81800] LOG:  database system is ready to accept connections
 done
server started

thanks!

Issues changing defaults via DefaultConfig()

I'm trying to change the defaults via DefaultConfig and is not working as per README.
Maybe you can point me in the right direction.

go version
go version go1.17.1 linux/amd64
package main

import (
  "bytes"
  "fmt"
  "time"

  "github.com/fergusstrange/embedded-postgres"
)

func main() {
  V12 := "12.10.0"
  logger := &bytes.Buffer{}
  postgres := embeddedpostgres.NewDatabase(DefaultConfig().
        Username("beer").
        Password("wine").
        Database("gin").
        Version(V12).
        RuntimePath("/tmp").
        BinaryRepositoryURL("https://repo1.maven.org/maven2").
        Port(9876).
        StartTimeout(45 * time.Second).
        Logger(logger))
  err := postgres.Start()
  fmt.Println(err)
}
go build a.go
# command-line-arguments
./a.go:14:44: undefined: DefaultConfig

Any idea what I'm doing wrong? Thanks! Great project BTW!

Extensive README

  • Document basic use

  • Link to examples

  • Link to API Docs

  • Contribution guidelines

  • License

  • Inspiration links

  • Running tests

  • Command line link

Issues with M1 ARM Macs

Using version 19.0 which downloads embedded-postgres 14.5.0.

Currently the defaultVersionStrategy overrides detected ARM architectures to use the amd64 architecture download for macs instead.

However this version seems to have some issues when running certain commands: in my case, a create function would always hang. This seems to be a problem specific to the amd binary running on the arm architecture.

For postgres versions 10.20, 11.15, 12.10, 13.6 and then >= 14.2, however, native ARM builds DO exist: https://repo1.maven.org/maven2/io/zonky/test/postgres/embedded-postgres-binaries-darwin-arm64v8/

I found that builds native to the ARM/M1 architecture don't exhibit the same problem: the function is created as expected and the process does not hang.

If the exception in the defaultVersionStrategy for versions >= 14.2 were removed, allowing the darwin/arm64v8 binaries to be used on these architectures, it would unblock users that run M1 macs that are experiencing this problem.

Can't change `postgres.conf` values

I want to override in the MAX_CONNECTIONS variable which is located in RuntimePath/postgresql.conf. The problem is, this value needs to be overridden prior to the DB process starting in order to take effect. postgres.Start() clears RuntimePath directory, which doesn't allow a custom postgresql.conf to be injected. There's also no restart mechanism that would utilize the same postgresql.conf file without wiping RuntimePath.

Any logger example to only process stderr?

Hi,

At first use I saw a lot of logs in my tests due to the DB setup so I decided to discard them with:

embeddedpostgres.DefaultConfig().Logger(ioutil.Discard)

Quick questions:

  • is there any reason to maintain them since the client errors should give some insights in case of failures?
  • if so, do you have a logger example where it would just catch errors from the server? Like catching only stderr and not stdout? But I'm not sure it's possible since the logger is a io.Writer without more information on the original output channel

Thank you,

defer not working, and so DB not getting shutdown.

Can someone help me with why this code does not call the embeddedPostgres.Stop() ?

There is a sleep forever to simulate work being done.

This also means that on the 2nd run, the execution fails with:

2021/11/20 12:35:01 process already listening on port 5432
package main

import (
	"log"

	embeddedpostgres "github.com/fergusstrange/embedded-postgres"
)

func main() {

	embeddedPostgres := embeddedpostgres.NewDatabase()
	if err := embeddedPostgres.Start(); err != nil {
		log.Fatal(err)
	}
	defer func() {
		if err := embeddedPostgres.Stop(); err != nil {
			log.Fatal(err)
		}
	}()

	log.Println("ready ...")

	// sleep forever
	blockForever()

}

func blockForever() {
	select {}
}

Race in remote fetcher can cause parallel tests to fail

This code in remote_fetch.go has a race that can be triggered when the cache hasn't been populated yet. If you have multiple tests running in parallel, multiple processes will try to download the remote archive and write it to the cache location.

This can lead to errors like this:

--- FAIL: TestSuite (0.59s)
    database.go:49: could not start database: &{%!e(string=unable to extract postgres archive: xz: data is truncated or corrupt)}
FAIL

That can happen when the test has successfully downloaded the archive into the cache location, and opens the file; at the same time, another test starts writing its own cache file, and the first one ends up reading partially-written file, and failing to uncompress it.

The usual way to fix this is to write the data to a temporary file, and move it into the final location, which is an atomic operation (except on Windows, if that's a worry). If Windows support is desired, you can try the move, and if it fails, check if the error is because the target file exists (which means some other process "won" the race).

Running multiple tests in parallel causes an error

$ go test ./...
ok      github.com/nathanfranke/my-project/dummy1 2.171s
--- FAIL: TestDummy (0.13s)
    dummy_test.go:81: creating embedded database...
    dummy_test.go:84: unable to extract postgres archive /home/nathan/.embedded-postgres-go/embedded-postgres-binaries-linux-amd64-12.8.0.txz to /home/nathan/.embedded-postgres-go/extracted
FAIL
FAIL    github.com/nathanfranke/my-project/dummy2 0.130s
FAIL

Support Intel and Arm MAC

Note: this issue is created to isolate from the other issues that per affected on Linux and windows.

Look into support for better logging on new Mac ARM, with rosetta emulation support by default.

Longer term support for Mac ARM when the packages are available on Maven.

Description is misleading

This is not a an embedded, in-memory postgres database in Go. It's a Java database with a Go wrapper.

unable to extract postgres archive: xz: data is truncated or corrupt

When using embedded-postgres in a docker container we sometimes see the following error:

Failed to start embedded postgres: unable to extract postgres archive: xz: data is truncated or corrupt
Perhaps there's a timing error in the code that fetches the jar file from maven and extracts the embedded .tgz file containing the postgres binaries.

Cannot find binary in archive

Hey all, stumbled upon this repository and decided to give it a try.

Unfortunately the following snippet does not seem to work me.

pg := embeddedpostgres.NewDatabase()

if err := pg.Start(); err != nil {
  t.Error(err)
  return
}

defer pg.Stop()

This gives me the error error fetching postgres: cannot find binary in archive retrieved from https://repo1.maven.org/maven2/io/zonky/test/postgres/embedded-postgres-binaries-linux-amd64/12.8.0/embedded-postgres-binaries-linux-amd64-12.8.0.jar

go version gives me: go version go1.17.1 linux/amd64 and I'm running Ubuntu 20.04.01 with kernel 5.8.0-23-generic.

Am I doing something wrong?

cleanup deleted my project

The cleanup just deleted my whole project because I set RuntimePath to ./ and apparently the Stop method deletes the entire directory? Luckily version control, but I lost all my certs and an hour or so of work. Cool project, but this was a pretty big surprise and a major foot-gun.

Add error info when 'unable to init db'

Currently, I get this error, without any useful information:

panic: localpg: failed to start PgSQL: unable to init database using: /tmp/go_embedded_pgsql_binaries_3773798719/bin/initdb -A password -U username -D /tmp/go_embedded_pgsql_runtime_2105005878/data --pwfile=/tmp/go_embedded_pgsql_runtime_2105005878/pwfile --locale=en_US.UTF-8

Unable to run in official docker golang:1.15

unable to init database using: /root/.embedded-postgres-go/extracted/bin/initdb -A password -U postgres -D /root/.embedded-postgres-go/extracted/data --pwfile=/root/.embedded-postgres-go/extracted/pwfile

Unexported field 'version' usage

Trying to write a simple command-line interface to embedded-postgres-binaries:

package main

import (
	"io"
	"log"
	"os"
	"time"

	embeddedpostgres "github.com/fergusstrange/embedded-postgres"
)

func main() {
	const version embeddedpostgres.PostgresVersion = embeddedpostgres.V15
	const port uint32 = 5432
	const database string = "postgres"
	const username string = "postgres"
	const password string = "postgres"
	runtimePath, err := os.MkdirTemp("", "prepare_database_test")
	if err != nil {
		panic(err)
	}
	var dataPath string = runtimePath
	var binariesPath string = runtimePath
	const locale string = "en_US.UTF-8"
	const binaryRepositoryURL string = "https://repo1.maven.org/maven2"
	const startTimeout time.Duration = 15 * time.Second // nanosecond
	var logger io.Writer = os.Stdout

	const config = embeddedpostgres.Config{
		version:             version,
		port:                port,
		database:            database,
		username:            username,
		password:            password,
		runtimePath:         runtimePath,
		dataPath:            dataPath,
		binariesPath:        binariesPath,
		locale:              locale,
		binaryRepositoryURL: binaryRepositoryURL,
		startTimeout:        startTimeout,
		logger:              logger,
	}

	embeddedPostgres := embeddedpostgres.NewDatabase(config)
	if err := embeddedPostgres.Start(); err != nil {
		log.Fatal(err)
	}

	defer func() {
		if err := embeddedPostgres.Stop(); err != nil {
			log.Fatal(err)
		}
	}()
}

Errors:

# command-line-arguments
./hello.go:30:3: unknown field version in struct literal of type embeddedpostgres.Config
./hello.go:31:3: unknown field port in struct literal of type embeddedpostgres.Config
./hello.go:32:3: unknown field database in struct literal of type embeddedpostgres.Config
./hello.go:33:3: unknown field username in struct literal of type embeddedpostgres.Config
./hello.go:34:3: unknown field password in struct literal of type embeddedpostgres.Config
./hello.go:35:3: unknown field runtimePath in struct literal of type embeddedpostgres.Config
./hello.go:36:3: unknown field dataPath in struct literal of type embeddedpostgres.Config
./hello.go:37:3: unknown field binariesPath in struct literal of type embeddedpostgres.Config
./hello.go:38:3: unknown field locale in struct literal of type embeddedpostgres.Config
./hello.go:39:3: unknown field binaryRepositoryURL in struct literal of type embeddedpostgres.Config
./hello.go:39:3: too many errors

Compilation finished with exit code 1

I'll probably expose subcommands: version (will install if nonexistent); start_server; create_db ; delete_db; and stop_server. Use-case is primarily CI/CD workflows.

Can you export all the fields that are required for this functionality, or tell me what I'm doing wrong?

Unable to init database

unable to init database using '/var/folders/70/rld0rwh92bg781zkxt1x6szh0000gn/T/TestProcessEvmEvent_IRIsExist1738323574/001/bin/initdb -A password -U postgres -D /var/folders/70/rld0rwh92bg781zkxt1x6szh0000gn/T/TestProcessEvmEvent_IRIsExist1738323574/001/data --pwfile=/var/folders/70/rld0rwh92bg781zkxt1x6szh0000gn/T/TestProcessEvmEvent_IRIsExist1738323574/001/pwfile': exit status 1

System Specs:

Apple M1 Pro
MacOS Ventura 13.0

using github.com/fergusstrange/embedded-postgres v1.19.0

Only way i've been able to reconcile this is by restarting my computer.

More examples

Create examples for the following

  • Basic use
  • Use with migrations
  • Use with a few SQL libraries
  • Use as part of a wider webapp flow

Better support for using as application database

I just stumbled on this project which is kind of what I've been looking for a long to easily use postgres without caring about the installation of the server.

But there are some points which are rather unfortunate to use the lib as-is in a project other than for tests.

  1. Start method is a monolith
    • db data directory is erased on each start
    • db is initialized each time
    • db is created each time
  2. db data is in RuntimePath
  3. not infrastructure to test if a db data directory is already available

@fergusstrange would you accept PRs which change the above mentioned behavior? I would like to see them integrated in this project rather then maintain a fork of this lib...

Cgo?

Does this package require cgo enabled?
Is it equivalent to an actual postgres server or a substitute with most of the same features and behaviours?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.