Git Product home page Git Product logo

testfixtures's Introduction

testfixtures

PkgGoDev

Warning: this package will wipe the database data before loading the fixtures! It is supposed to be used on a test database. Please, double check if you are running it against the correct database.

TIP: There are options not described in this README page. It's recommended that you also check the documentation.

Writing tests is hard, even more when you have to deal with an SQL database. This package aims to make writing functional tests for web apps written in Go easier.

Basically this package mimics the "Ruby on Rails' way" of writing tests for database applications, where sample data is kept in fixtures files. Before the execution of every test, the test database is cleaned and the fixture data is loaded into the database.

The idea is running tests against a real database, instead of relying in mocks, which is boring to setup and may lead to production bugs not being caught in the tests.

Installation

First, import it like this:

import (
        "github.com/go-testfixtures/testfixtures/v3"
)

Usage

Create a folder for the fixture files. Each file should contain data for a single table and have the name <table_name>.yml:

myapp/
  myapp.go
  myapp_test.go
  ...
  fixtures/
    posts.yml
    comments.yml
    tags.yml
    posts_tags.yml
    ...

The file would look like this (it can have as many record you want):

# comments.yml
- id: 1
  post_id: 1
  content: A comment...
  author_name: John Doe
  author_email: [email protected]
  created_at: 2020-12-31 23:59:59
  updated_at: 2020-12-31 23:59:59

- id: 2
  post_id: 2
  content: Another comment...
  author_name: John Doe
  author_email: [email protected]
  created_at: 2020-12-31 23:59:59
  updated_at: 2020-12-31 23:59:59

# ...

An YAML object or array will be converted to JSON. It will be stored on a native JSON type like JSONB on PostgreSQL & CockroachDB or as a TEXT or VARCHAR column on other databases.

- id: 1
  post_attributes:
    author: John Due
    author_email: [email protected]
    title: "..."
    tags:
      - programming
      - go
      - testing
    post: "..."

Binary columns can be represented as hexadecimal strings (should start with 0x):

- id: 1
  binary_column: 0x1234567890abcdef

If you need to write raw SQL, probably to call a function, prefix the value of the column with RAW=:

- id: 1
  uuid_column: RAW=uuid_generate_v4()
  postgis_type_column: RAW=ST_GeomFromText('params...')
  created_at: RAW=NOW()
  updated_at: RAW=NOW()

Your tests would look like this:

package myapp

import (
        "database/sql"

        _ "github.com/lib/pq"
        "github.com/go-testfixtures/testfixtures/v3"
)

var (
        db *sql.DB
        fixtures *testfixtures.Loader
)

func TestMain(m *testing.M) {
        var err error

        // Open connection to the test database.
        // Do NOT import fixtures in a production database!
        // Existing data would be deleted.
        db, err = sql.Open("postgres", "dbname=myapp_test")
        if err != nil {
                ...
        }

        fixtures, err = testfixtures.New(
                testfixtures.Database(db), // You database connection
                testfixtures.Dialect("postgres"), // Available: "postgresql", "timescaledb", "mysql", "mariadb", "sqlite" and "sqlserver"
                testfixtures.Directory("testdata/fixtures"), // The directory containing the YAML files
        )
        if err != nil {
                ...
        }

        os.Exit(m.Run())
}

func prepareTestDatabase() {
        if err := fixtures.Load(); err != nil {
                ...
        }
}

func TestX(t *testing.T) {
        prepareTestDatabase()

        // Your test here ...
}

func TestY(t *testing.T) {
        prepareTestDatabase()

        // Your test here ...
}

func TestZ(t *testing.T) {
        prepareTestDatabase()

        // Your test here ...
}

Alternatively, you can use the Files option, to specify which files you want to load into the database:

fixtures, err := testfixtures.New(
        testfixtures.Database(db),
        testfixtures.Dialect("postgres"),
        testfixtures.Files(
                "fixtures/orders.yml",
                "fixtures/customers.yml",
        ),
)
if err != nil {
        ...
}

With Paths option, you can specify the paths that fixtures will load from. Path can be directory or file. If directory, we will search YAML files in it.

fixtures, err := testfixtures.New(
        testfixtures.Database(db),
        testfixtures.Dialect("postgres"),
        testfixtures.Paths(
                "fixtures/orders.yml",
                "fixtures/customers.yml",
                "common_fixtures/users"
        ),
)
if err != nil {
        ...
}

Single file on multiple tables

You can use the FilesMultiTables option, to specify which files you want to load into the database with support multiple tables (file name does not affect table names):

fixtures, err := testfixtures.New(
        testfixtures.Database(db),
        testfixtures.Dialect("postgres"),
        testfixtures.FilesMultiTables(
                "fixtures/test_case1.yml",
                "fixtures/test_case2.yml",
                "fixtures/posts_comments.yml",
        ),
)
if err != nil {
        ...
}

The file would look like this (it can have as many tables and records you want):

# test_case1.yml
posts:
  - id: 1
    post_id: 1
    content: A comment...
    author_name: John Doe
    author_email: [email protected]
    created_at: 2020-12-31 23:59:59
    updated_at: 2020-12-31 23:59:59

  - id: 2
    post_id: 2
    content: Another comment...
    author_name: John Doe
    author_email: [email protected]
    created_at: 2020-12-31 23:59:59
    updated_at: 2020-12-31 23:59:59

comments:
  - id: 1
    post_id: 1
    content: Post 1 comment 1
    author_name: John Doe
    author_email: [email protected]
    created_at: 2016-01-01 12:30:12
    updated_at: 2016-01-01 12:30:12
# ...

Security check

In order to prevent you from accidentally wiping the wrong database, this package will refuse to load fixtures if the database name (or database filename for SQLite) doesn't contains "test". If you want to disable this check, use:

testfixtures.New(
        ...
        testfixtures.DangerousSkipTestDatabaseCheck(),
)

Disable cleanup

If you want to disable cleanup, you can also do like below. This is usually not recommended, and should be used mostly for debugging.

testfixtures.New(
        ...
        testfixtures.DangerousSkipCleanupFixtureTables(),
)

Sequences

For PostgreSQL and MySQL/MariaDB, this package also resets all sequences to a high number to prevent duplicated primary keys while running the tests. The default is 10000, but you can change that with:

testfixtures.New(
        ...
        testfixtures.ResetSequencesTo(10000),
)

Or, if you want to skip the reset of sequences entirely:

testfixtures.New(
        ...
        testfixtures.SkipResetSequences(),
)

Force DELETE FROM ... on ClickHouse

By default, when using ClickHouse, this library will use TRUNCATE ... to clean the database. If you want to force the use of DELETE FROM ... you can do it by doing:

testfixtures.New(
        ...
        testfixtures.ClickhouseUseDeleteFrom(),
)

Compatible databases

PostgreSQL / TimescaleDB / CockroachDB

This package has three approaches to disable foreign keys while importing fixtures for PostgreSQL databases:

With DISABLE TRIGGER

This is the default approach. For that use:

testfixtures.New(
        ...
        testfixtures.Dialect("postgres"), // or "timescaledb"
)

With the above snippet this package will use DISABLE TRIGGER to temporarily disabling foreign key constraints while loading fixtures. This work with any version of PostgreSQL, but it is required to be connected in the database as a SUPERUSER. You can make a PostgreSQL user a SUPERUSER with:

ALTER USER your_user SUPERUSER;

With ALTER CONSTRAINT

This approach don't require to be connected as a SUPERUSER, but only work with PostgreSQL versions >= 9.4. Try this if you are getting foreign key violation errors with the previous approach. It is as simple as using:

testfixtures.New(
        ...
        testfixtures.Dialect("postgres"),
        testfixtures.UseAlterConstraint(),
)

With DROP CONSTRAINT

This approach is implemented to support databases that do not support above methods (namely CockroachDB).

testfixtures.New(
        ...
        testfixtures.Dialect("postgres"),
        testfixtures.UseDropConstraint(),
)

Tested using the github.com/lib/pq and github.com/jackc/pgx drivers.

MySQL / MariaDB

Just make sure the connection string have the multistatement parameter set to true, and use:

testfixtures.New(
        ...
        testfixtures.Dialect("mysql"), // or "mariadb"
)

Tested using the github.com/go-sql-driver/mysql driver.

SQLite

SQLite is also supported. It is recommended to create foreign keys as DEFERRABLE (the default) to prevent problems. See more on the SQLite documentation. (Foreign key constraints are no-op by default on SQLite, but enabling it is recommended).

testfixtures.New(
        ...
        testfixtures.Dialect("sqlite"),
)

Tested using the github.com/mattn/go-sqlite3 driver.

Microsoft SQL Server

SQL Server support requires SQL Server >= 2008. Inserting on IDENTITY columns are handled as well. Just make sure you are logged in with a user with ALTER TABLE permission.

testfixtures.New(
        ...
        testfixtures.Dialect("sqlserver"),
)

Tested using the mssql and sqlserver drivers from the github.com/denisenkom/go-mssqldb lib.

ClickHouse

testfixtures.New(
        ...
        testfixtures.Dialect("clickhouse"),
)

Templating

Testfixtures supports templating, but it's disabled by default. Most people won't need it, but it may be useful to dynamically generate data.

Enable it by doing:

testfixtures.New(
        ...
        testfixtures.Template(),

        // the above options are optional
        TemplateFuncs(...),
        TemplateDelims("{{", "}}"),
        TemplateOptions("missingkey=zero"),
        TemplateData(...),
)

The YAML file could look like this:

# It's possible generate values...
- id: {{sha256 "my-awesome-post}}
  title: My Awesome Post
  text: {{randomText}}

# ... or records
{{range $post := $.Posts}}
- id: {{$post.Id}}
  title: {{$post.Title}}
  text: {{$post.Text}}
{{end}}

Generating fixtures for a existing database

The following code will generate a YAML file for each table of the database into a given folder. It may be useful to boostrap a test scenario from a sample database of your app.

dumper, err := testfixtures.NewDumper(
        testfixtures.DumpDatabase(db),
        testfixtures.DumpDialect("postgres"), // or your database of choice
        testfixtures.DumpDirectory("tmp/fixtures"),
        testfixtures.DumpTables( // optional, will dump all table if not given
          "posts",
          "comments",
          "tags",
        ),
)
if err != nil {
        ...
}
if err := dumper.Dump(); err != nil {
        ...
}

This was intended to run in small sample databases. It will likely break if run in a production/big database.

Gotchas

Parallel testing

This library doesn't yet support running tests in parallel! Running tests in parallel can result in random data being present in the database, which will likely cause tests to randomly/intermittently fail.

This is specially tricky since it's not immediately clear that go test ./... run tests for each package in parallel. If more than one package use this library, you can face this issue. Please, use go test -p 1 ./... or run tests for each package in separated commands to fix this issue.

If you're looking into being able to run tests in parallel you can try using testfixtures together with the txdb package, which allows wrapping each test run in a transaction.

CLI

We also have a CLI to load fixtures in a given database.

Grab it from the releases page or install with Homebrew:

brew install go-testfixtures/tap/testfixtures

Usage is like this:

# load
testfixtures -d postgres -c "postgres://user:password@localhost/database" -D testdata/fixtures
# dump
testfixtures --dump -d postgres -c "postgres://user:password@localhost/database" -D testdata/fixtures

The connection string changes for each database driver.

Use testfixtures --help for all flags.

Contributing

We recommend you to install Task and Docker before contributing to this package, since some stuff is automated using these tools.

It's recommended to use Docker Compose to run tests, since it runs tests for all supported databases once. To do that you just need to run:

task docker

But if you want to run tests locally, copy the .sample.env file as .env and edit it according to your database setup. You'll need to create a database (likely names testfixtures_test) before continuing. Then run the command for the database you want to run tests against:

task test:pg # PostgreSQL
task test:crdb # CockroachDB
task test:mysql # MySQL
task test:sqlite # SQLite
task test:sqlserver # Microsoft SQL Server

GitHub Actions (CI) runs the same Docker setup available locally.

Alternatives

If you don't think using fixtures is a good idea, you can try one of these packages instead:

  • factory-go: Factory for Go. Inspired by Python's Factory Boy and Ruby's Factory Girl
  • fixtory: go generate based type-safe, DRY, flexible test fixture factory
  • go-txdb (Single transaction SQL driver for Go): Use a single database transaction for each functional test, so you can rollback to previous state between tests to have the same database state in all tests
  • go-sqlmock: A mock for the sql.DB interface. This allow you to unit test database code without having to connect to a real database
  • dbcleaner - Clean database for testing, inspired by database_cleaner for Ruby

testfixtures's People

Contributors

almalki avatar andreynering avatar bluele avatar chclaus avatar corvinusy avatar dependabot[bot] avatar dgsb avatar ekhabarov avatar ethantkoenig avatar fifsky avatar fufuceng avatar georlav avatar gitter-badger avatar gmassman avatar jeffmeyers avatar jerome1337 avatar johejo avatar k-yomo avatar kolaente avatar mkozjak avatar momer avatar mthuberty avatar nt0xa avatar oz-dev-v0 avatar phelrine avatar richard-to avatar tbutts avatar techknowlogick avatar usk81 avatar wxiaoguang avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

testfixtures's Issues

Reset sequences before loading fixtures (maybe)

Currently the only time sequences are modified is after fixtures are loaded. I propose that, in addition to the current behavior, to set sequences to 1 before loading fixtures.

When omitting the serial columns from the fixture files, subsequent test runs result in fixture records with serial values greater than ResetSequencesTo. Thus when the sequences are reset, it is reset to a value before the fixture records.

That was confusing to type. Example, run 1:

  id   |       name        
-------+-------------------
     1 | 1) fixture
 10000 | 2) from test
 10001 | 3) also from test

Run 2:

  id   |       name        
-------+-------------------
 10002 | 1) fixture
 10000 | 2) from test
 10001 | 3) also from test

As soon as you increase the number of rows generated by a given run, conflict.

I've worked around this by specifying ids in the fixture files. This is probably a good practice anyway, and the current behavior acts as a reasonable forcing function. An alternative fix for this issue might be simply documenting a best practice of specifying values for serial columns.

A robust fix would be to take the max of the serial columns, though that's a bit more involved. The query in this question would be a good starting point though: https://dba.stackexchange.com/questions/47098/how-do-i-determine-if-a-column-is-defined-as-a-serial-data-type-instead-of-an-in

TLDR, please do one of:

  1. Reset sequences to 1 before loading fixtures then to ResetSequencesTo afterwards.
  2. Update docs to recommend that all serial columns have values and that they be less than ResetSequencesTo
  3. At least in the case of postgres, set ResetSequencesTo automatically using the result of select max(<serial column>)+1 from <schema>.<table>.

Load fixtures from http.FileSystem to support files bundled with packr

For some testing scenarios it can be handy to generate a binary that is able to populate it's own test database. A prerequisite for this is to bundle the fixture files with the binary. One possible way to do so is packr. But there is no straightforward way to make the bundled files available for the testfixtures package.

Luckily packr implements http.FileSystem in order to represent the bundled files in the binary. I therefore suggest to add the following functions to testfixtures:

func NewFileSystem(db *sql.DB, helper Helper, fs http.FileSystem) (*Context, error) 
func LoadFixtureFileSystem(db *sql.DB, helper Helper, fs http.FileSystem) error

This would allow testfixtures to work together with packr or other sources for fixture files that implement the http.FileSystem interface.

Tests sometimes run before Load() method finishes

Go version: go1.11.2 linux/amd64
DB: MariaDB 5.5.60
testfixtures: v2.5.1

I'm running into some weird issues where it seems that func (c *Context) Load() either isn't blocking the tests from running until it finishes or silently errors out. In a couple of my tests I have a yaml file with 7 rows defined, sometimes running my tests everything works as expected the table is correctly cleaned of old data and the new data is populated and my tests passes. Other times only 1-2 rows may get loaded which causes my tests to be wrong due to the missing data.

I'm not too sure of an easy way to trace down what exactly is going on though that is causing this error.

Conflict on primary key when reseting sequence on postgres

Somehow, primary key conflict occurs when sequence are used as primary key default value.
It happens when tests are run several time.
As workaround an option has been added to the postgres helper to no reset sequences #38
This issue is opened to track the root cause of the problem which have not been completely identified.

Support Single Fixture File Loading

It doesn't appear this functionality is available. Can I suggest it as a feature?

Example (just add a singularized version of the existing LoadFixtures method)

testfixtures.LoadFixture("../fixtures/<my-table>.yml", db, &testfixtures.MySQLHelper{})

Upcoming changes in v3

This is an issue to track changes to be made in v3.

Work in progress: #57

Breaking changes

  • Move from gopkg.in/testfixtures.v2 to github.com/go-testfixtures/testfixtures/v3
  • Remove Oracle support
  • Rewrite the public API to use functional options

New features

  • Add option to choose a specific time zone to parse date/time objects. On the absence of this option, testfixtures should use time.Location as it does today
  • Have and CLI and publish it using GoReleaser (#54)
  • [Maybe?] Implement transaction fixtures (#24)
  • [Maybe?] Allow templating in YAML files (#25)

How do I seed multiple tables from a single file?

Ideally I want to be able to do

# TestUserService_OnePost.yaml
users:
    - id: 2
      name: Joe
posts:
    - id: 1
      user_id: 2
      content: "hi"

The example in the README shows you have to make one file per table, what I want is to have individual files for each test case which contain all the necessary data for each table as shown in the example. Currently I've been using https://github.com/romanyx/polluter which does the job but isn't maintained. I'd like to migrate to this eventually.

Provide method to teardown fixture data

I've run into a situation where my build fails depending on the order of the tests and the nature of their foreign key relationships. I am using the UseAlterConstraint method with Postgres, and not running tests in parallel.

For example:

Test 1 creates fixtures for Table A & Table B. Table A has a record which references some record in Table B.
Test 2 only creates fixtures in Table B. Because there is leftover data in table A from test 1, now test 2 fails when testfixtures tries to clear Table B.

What I had to do to solve this is to copy the logic to make the constraints deferrable (I am using Postgres). Then I can do a teardown at the end of each test where I delete all the fixture data.

So what I am looking for is a method something like:
fixtures.Unload()

This would remove everything created by fixtures.Load(), to avoid potential conflicts with subsequent tests.

JSONB types in Postgres

Whenever I add to a fixture a map or array, I get an error like:

"sql: converting Exec argument #3's type: unsupported type map[interface {}]interface {}, a map"

Any chance we can support JSONB?

Error "pq: unsupported jsonb version number 123"

Hi! In YAML i configured an object

- order_code: 123456
  created_at: 2020-11-19 12:29:10
  updated_at: 2020-11-19 12:38:36
  data:
      AfterUpdateData: 
        PickupTimestamp: 2020-11-24T08:34:25.46251+03:00
        AtClientTimestamp: 2020-11-19T15:38:35.806412+03:00
        AtVendorTimestamp: 2020-11-24T08:34:25.46251+03:00
      BeforeUpdateData: 
        PickupTimestamp: 2020-11-24T08:34:25.46251+03:00
        AtClientTimestamp: 2020-11-24T08:34:25.46251+03:00
        AtVendorTimestamp: 2020-11-24T08:34:25.46251+03:00

Appliyng of this fixture leads to an error

testfixtures: error inserting record: pq: unsupported jsonb version number 123, on file: delivery.yml, index: 0, sql: INSERT INTO "delivery" ("order_code", "created_at", "updated_at", "data") VALUES ($1, $2, $3, $4), params: [123456 2020-11-19 12:29:10 +0300 MSK 2020-11-19 12:38:36 +0300 MSK map[AfterUpdateData:map[AtClientTimestamp:2020-11-19T15:38:35.806412+03:00 AtVendorTimestamp:2020-11-24T08:34:25.46251+03:00 PickupTimestamp:2020-11-24T08:34:25.46251+03:00] BeforeUpdateData:map[AtClientTimestamp:2020-11-24T08:34:25.46251+03:00 AtVendorTimestamp:2020-11-24T08:34:25.46251+03:00 PickupTimestamp:2020-11-24T08:34:25.46251+03:00]]]

Same problem described here lib/pq#528 . The solution - to cast []byte or json.RawMessage to string before inserting. Is it possible?

fixtures.Load is not working

I generated the fixtures using GenerateFixtures API with my postgres development database. Got the fixtures under the testdata/fixtures folder.

After that I have been trying to load those fixtures into a test database and run my tests but fixtures.Load() call is failing as I don't see any relations in the database and thus all my tests are failing.

Do suggest if I am missing anything.

PostgreSQL - Failure if sequences exist in other namespaces

@andreynering and contributors, this is a dope project, thanks for working on it!

I have a test database with two schemas:

  1. public
  • table_a
  1. exampleschema
  • table_b

Currently, postgresql.go searches for sequences to reset via

"SELECT relname FROM pg_class WHERE relkind = 'S'"

(https://github.com/go-testfixtures/testfixtures/blob/master/postgresql.go#L86-L107)

Unfortunately, when I try to load fixtures via fixtures.Load(), this error crops up

momer@ubuntu:~/.golang/src/my-domain/my-company/myproject$ go test -v
{"level":"info","msg":"Successfully opened DB Connection","time":"2017-07-19T14:06:01-05:00"}
=== RUN   TestInsertIntoTableA
{"level":"fatal","msg":"pq: relation \"schema_namespaced_table_b_id_seq\" does not exist","time":"2017-07-19T14:06:01-05:00"}
exit status 1
FAIL    my-domain/my-company/myproject   0.039s

with this output on PostgreSQL's stdout

postgres    | ERROR:  relation "schema_namespaced_table_b_id_seq" does not exist at character 15
postgres    | STATEMENT:  SELECT SETVAL('schema_namespaced_table_b_id_seq', 10000)

I've put together a pull request which resolves these issues, and will comment it below! Test suite for PostgreSQL still passes via go test -tags postgresql

How do I install this package?

I get the following error when using "go get ":

can't load package: package github.com/go-testfixtures/testfixtures: code in directory /home/im
ad/.go/src/github.com/go-testfixtures/testfixtures expects import "github.com/go-testfixtures/t
estfixtures/v3"

Trigger error

Hi,

running the following command results in this weird error:

err = testfixtures.LoadFixtures(FixturesPath, db.DB(), &testfixtures.PostgreSQLHelper{})
if err != nil {
    log.Fatal(err)
}

2016/07/22 22:20:43 pq: permission denied: "RI_ConstraintTrigger_a_19013" is a system trigger

Any ideas?

Add support for "pgx" postgres driver/dialect.

I'm currently using pgx (https://github.com/jackc/pgx) instead of pq I would like to use this driver/dialect in testfixtures, but it's not currently supported. Seems like I can just add pgx to the helperForDialect function. I tested it out and it seemed to work. It also passed the unit tests.

There was one slight difference that I noticed. In one of my fixtures I have a zip code string field, which I populate like this 11111. In postgres/pq it gets converted to a string automatically. But I noticed in pgx I need to explicitly make the zip code a string (e.g. '11111'). There may be more differences, but haven't tested it too extensively so far.

func helperForDialect(dialect string) (helper, error) {
	switch dialect {
	case "postgres", "postgresql", "timescaledb", "pgx":
		return &postgreSQL{}, nil
	case "mysql", "mariadb":
		return &mySQL{}, nil
	case "sqlite", "sqlite3":
		return &sqlite{}, nil
	case "mssql", "sqlserver":
		return &sqlserver{}, nil
	default:
		return nil, fmt.Errorf(`testfixtures: unrecognized dialect "%s"`, dialect)
	}
}

Question about popularity, and transactions (separate questions)

Coming from the rails world I'm just baffled as to why this, the most popular golang database fixture package, only has 151 stars. Do gophers not test with databases? Is everybody just mocking database queries (which is insane)? Do they just not test?

Does it make sense to have the option of passing a sql.Tx so tests can be run in parallel? (and disable the automatic database cleaning - which, by the way, is greatly appreciated)

testfixtures.UseAlterConstraint() -> Does not work with `ON DELETE CASCADE`

Database: Postgres 9.6

I didn't want to make my test user role a superuser, so I tried the UseAlterConstraint approach.

When I was testing, I found that one of my tables was always empty, even though it should have been populated with my fixtures.

My guess is that ON DELETE CASCADE on my foreign keys is a causing this problem.

Example:

parent
 - id

child
  - id
  - parent_id (fkey with on delete cascade)

Now this is largely dependent on the order that the data is deleted and inserted. So if child is processed before parent, then the child table will be empty due to the fkey cascade.

DELETE FROM "child"
INSERT FIXTURES into "child"
DELETE FROM "parent" (A casade appears to delete from child)
INSERT FIXTURES into "parent"

Looking at the codebase, I wonder why the deletes aren't done together. Then after that perform all the necessary inserts? Is there a reason why we need to delete/insert for each table like this?

can the entire table data be deleted and modified to delete the corresponding data

Now when importing data, the data in the relevant table will be deleted. This approach is a bit rude (for example, some data I don’t want to delete or others want to use the data), can it be modified to delete the data that conflicts with the imported data (For example, specify a single primary key?). I know this is a bit more complicated than deleting the entire table, but generally speaking, we expect to delete this part of the data that we imported.

Can't correctly load binary data from generated yml file

I use SQLite3 to store small binary data (data type blob). Call GenerateFixtures() and obtain a .yml file like following:

- type: 0
  value:
  - 8
  - 128
  - 8
  - 16
  - 13
...

When I use Load() to read this *.yml file and overwrite the whole DB entries, the values are not the same as they were when I generated the fixtures. Except for blob, other data types remain intact.

Before GenerateFixtures():

INSERT INTO logs VALUES(2,1024,1,X'088008100d18a4f3f488a8c3aad015','2019-05-21 17:55:01.882239396+08:00',NULL,0,0);

After Load():

INSERT INTO logs VALUES(2,1024,1,X'5b382c3132382c382c31362c31332c32342c3136342c3234332c3234342c3133362c3136382c3139352c3137302c3230382c32315d','2019-05-21 17:55:01.882239396+08:00',NULL,0,0);

I believe the inconsistency is caused by json.Marshal in jsonArray's receiver function Value(). It will automatically convert []int to marshaled []byte, but the result is unexpected.

An easy workaround is like this:

func (a jsonArray) Value() (driver.Value, error) {
    var arr []byte
    for _, v := range a {
        arr = append(arr, byte(v.(int)))
    }
    return arr, nil
}

However, it just considers int array case.

Any suggestions?

Implement "transactional fixtures" (or "transactional tests")

Originally asked in #23, but I've being thinking about it for some time.

Today, this lib cleans the database, load everything and then commit the changes for every test run. Rails has an option called "transactional fixtures" (or "transactional tests"), where fixtures are loaded once before all tests, and then each test runs inside a transaction. This would allow concurrent running of tests that touch the database.

Needs investigation on implementation and the better public interface to expose this.

Ideas and help would be very much appreciated.

Provide CLI to allow database seeding outside of tests

It would be great to have a command line interface with basic fixtures load/unload commands.

This would allow users to seed database using same fixtures and use them in development environment. It would be useful for manual testing or as demonstration data.

Error when using templates with incorrect order

I like test fixtures but encountered a minor (documentation) problem:

When I use this:

...
testfixtures.Database(db.DB()),
testfixtures.Dialect("sqlite"),
testfixtures.Directory("testdata/fixtures"),
testfixtures.Template(),
testfixtures.TemplateFuncs(template.FuncMap{
	"now": now,
}),

I get an error:

testfixtures: could not unmarshal YAML: yaml: invalid map key: map[interface {}]interface {}{"now":interface {}(nil)}

When I move the line with Directory() down a bit, everything works fine:

testfixtures.Database(db.DB()),
testfixtures.Dialect("sqlite"),
testfixtures.Template(),
testfixtures.TemplateFuncs(template.FuncMap{
	"now": now,
}),
testfixtures.Directory("testdata/fixtures"),

I could not see this in the documentation. It might be worth mentioning?

UseDropConstraint() not applied and tests failed because of relational constrains

Hi there,
my postgres is version 10.10 and my golang is 1.15.
I used the following config to initial fixtures,

f, err := testfixtures.New(
	testfixtures.Database(db),
	testfixtures.Dialect("postgresql"),
	testfixtures.Directory("../fixtures"),
	testfixtures.DangerousSkipTestDatabaseCheck(),
	testfixtures.UseDropConstraint(),
)

with UseDropConstraint still, I'm getting foreign key constraint

17:37:27 XXXXXXXXX.go:412: testfixtures: error inserting record: ERROR: insert or update on table "XXXXXXXXX" violates foreign key constraint "XXXXXXXXX_fk" (SQLSTATE 23503)
( X are masked character)

any thought and idea why it's not applied and I see the error.

Thanks

Cannot change time.Location of parsing date

I would like to change time.Location of date, but cannot change because of this code.

testfixtures/time.go

Lines 25 to 34 in fa3fb89

func tryStrToDate(s string) (time.Time, error) {
for _, f := range timeFormats {
t, err := time.ParseInLocation(f, s, time.Local)
if err != nil {
continue
}
return t, nil
}
return time.Time{}, ErrCouldNotConvertToTime
}

So, I fixed my yaml file the following temporary.

-  datetime: 2018-01-01 18:00:00
+  datetime: RAW="2018-01-01 18:00:00"

Is there other solution?

JSON supports

It would be better if you can support the following format.

-
  id: 1
  slice_of_struct:
     -
       name: myname1
       value: myvalue1
     -
       name: myname2
       value: myvalue2

Can't unload or load empty table

Some of my tests require that my queries return zero results because the DB is empty.

If I don't call load on my first test, I can get this behaviour, however on subsequent runs, due to having loaded in a later test, this test now fails (more than zero results) as the sqlite DB still exists.

It'd be helpful to have an unload/reset or a way to load in an empty table using the NewFiles for this purpose.

[PostgreSQL]ALTER CONSTRAINT problem

Environment

  • PostgreSQL 10
  • testfixtures 2.4.3
  • Go 1.8.3

Error message

2017/12/20 19:43:41 pq: update or delete on table "users" violates foreign key constraint "environments_created_by_users_id_foreign" on table "environments"

SUPERUSER + &testfixtures.PostgreSQL{} => Success
SUPERUSER + &testfixtures.PostgreSQL{UseAlterConstraint: true} => Failure
USER + &testfixtures.PostgreSQL{} => Failure
USER + &testfixtures.PostgreSQL{UseAlterConstraint: true} => Failure

It seems ALTER CONSTRAINT is ineffective.
Do you have any information?

Support for PostGIS fields?

Hi
I'm struggling to find out how can I use a PostGIS field in my fixtures. For example, for updating my model I need to run this command:

update my_table set polygon = ST_GeomFromText("SRID=4326; POLYGON ((11 22, 33 44, 55 66))") where id = 1;

and in my fixtures I have:

- id: 1
  polygon: ST_GeomFromText("SRID=4326; POLYGON ((11 22, 33 44, 55 66))")

But in the SQL command it executes, it wraps ST_GeomFromText... in quotations, which results in error.
How can I tell it not to wrap some value in quotations?

Thanks.

Reset MySQL autoincrement?

Can this product truncate table before insert yaml?
Now table's auto increments is not cleared.
I want reset auto increment value when insert yaml.

I'm using mysql.

thanks.

Errors are sometimes swallowed incorrectly in databaseName

I ran into an issue recently where my database had run out of connections but I received an error from testfixtures that my database name didn't include "test." This is a bit misleading and took me some time to track down.

How do you feel about modifying the helper interface to return (string, error)? This matches other methods on the interface and while it would be a breaking change it could help folks track down errors more easily when they occur.

Happy to PR this if you think it's a sane move. Thanks for the library!

I want to create array type test data in postgres, but I’m struggling

I created the following yml file and ran it, but I am getting an error

users.yml

- id: 1
  warehouse_ids:
    - 1
    - 2

error

&testfixtures.InsertError{
  Err: &pq.Error{
    Severity:         "ERROR",
    Code:             "22P02",
    Message:          "malformed array literal: \"[1,2]\"",
    Detail:           "Missing \"]\" after array dimensions.",
    Hint:             "",
    Position:         "",
    InternalPosition: "",
    InternalQuery:    "",
    Where:            "",
    Schema:           "",
    Table:            "",
    Column:           "",
    DataTypeName:     "",
    Constraint:       "",
    File:             "arrayfuncs.c",
    Line:             "292",
    Routine:          "array_in",
  },
   SQL:    "INSERT INTO \"users\" (\"id\", \"warehouse_ids\") VALUES ($1, $2)",
 Params: []interface {}{
    1,
    "[1,2]",
  },
}

How can I solve this?

DB Connection Refused

Using Postgres I can connect to the database fine using sql.Open, but passing the variable to testfixtures produces this error:

dial tcp [::1]:5432: connect: connection refused

Here is my code:

dbConnectionString := os.Getenv("DATABASE_TEST_CONNECTION_STRING")
db, err := sql.Open("postgres", dbConnectionString)
if err != nil {
	log.Println("DEBUG: ERROR ", err)
}

_, err = testfixtures.NewFolder(db, &testfixtures.PostgreSQLHelper{}, "testdata")
 if err != nil {
     log.Println("DEBUG: ", err)
 }

More detailed template documentation

HI,
I'm looking to use templates fixtures but can't figure out how it works.
The given example is not extremely explicit.

I am currently using two fixture files

{{range $u := $.Users}}
  - username: {{$u.username}}
{{end}}
{{range $a := $.Articles}}
- title: {{$a.title}}
  content: {{$a.content}}
{{end}}

And trying to load it like

testfixtures.New(
		testfixtures.Template(),
                (...)
		testfixtures.TemplateDelims("{{", "}}"),
		testfixtures.TemplateData(map[string][]map[string]string {
			"Users": {{"username": "test"}, {"username": "test2"}},
			"Articles": {{"title": "test", "content": "test"}, {"title": "test2", "content": "test2"}},
		}),
		)

I get the error testfixtures: fixture is not a slice or map. I'm not too sure what data structure I should put in to allow to load slices of maps of data.
Thanks

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.