Tests sometimes run before Load() method finishes about testfixtures HOT 8 CLOSED

Hi @vendion,

This bug is very odd. Are you running tests in parallel? Because this lib currently doesn't support that.

Keep me updated if you have more info about this bug.

from testfixtures.

My tests should not be running in parallel, I'm using a table-driven test design to define my tests and then ranging over it similar to the following:

type args struct {
		ctx   context.Context
		db    *database.DB
		reads database.MeterReads
	}

	tests := []struct {
		name   string
		args   args
		input  string
		golden string
		want   Data
	}{
		{
			name: "test read import",
			args: args{
				db: nil,
			},
			input:  "valid-5-reads-no-new-meters.input",
			golden: "valid-5-reads-no-new-meters.golden",
		},
	}

        for _, tt := range tests {
		_, tt.args.ctx, _ = testhelper.GetLogger(httpContext)
		tt.args.db = testhelper.OpenTestDB(t) // calls `sql.Open()`, and `testfixtures.NewFolder` to open the connection pool and sets the directory to load the YAML files from.
		testhelper.PrepareTestDataBase(t) // calls `.Load()` method on the `testfixtures.Context` struct
		err := json.Unmarshal(testhelper.HelperLoadBytes(t, tt.input), &tt.args.reads)
		testhelper.Ok(t, err)

		t.Run(tt.name, func(t *testing.T) {
			got, err := InsertMeterReads(tt.args.ctx, tt.args.db, tt.args.reads)
			testhelper.Ok(t, err)

			if *update {
				testhelper.UpdateGoldenFile(t, tt.golden, true, got)
			}

			err = json.Unmarshal(testhelper.HelperLoadBytes(t, tt.golden), &tt.want)
			testhelper.Ok(t, err)

			testhelper.Equals(t, got, tt.want)
		})
	}

In this case I only have one test defined so ranging over the tests struct should only happen once, with the database connection and table(s) being preformed at the start of each test.

I do have other functions which more tests defined which I have yet to have any issues with. The only way I can think of that would case the tests to run in parallel is running all my tests with go test ./... runs them in parallel. That is unless you mean I should limit go test command like go test -p 1 ./....

from testfixtures.

@vendion Hmm... I don't think go test ./... runs tests in parallel. But if it does, that could be the cause of this problem.

You could just try to run tests for each package separately, a couple of times, to see if the issue happens.

from testfixtures.

So far in my testing it does seem that it has to do with how go test ./... runs the tests, or at least might give some indication on the cause.

Lately I have been running my tests with -p 1 argument (full command now being go test -p 1 ./...) and my tests have been consistent about running without hitting this issue. I can also run the test for each package separately using go test -count=1 ./internal/(package name) without issue (the -count argument to bypass Go's cache for tests).

That said if I run all tests without the -p 1 argument I hit this issue.

from testfixtures.

Note: Once we're 100% sure about what is causing this problem we should update the documentation.

from testfixtures.

I wanted to just chime in and say that I also got bit by this issue. I'm not sure what's triggering go to run the tests in parallel though. All the documentation mentions that only tests marked with t.Parallel() should trigger a parallel run.

These were my test options -race -covermode=atomic -coverprofile=coverage.txt -failfast -p 1 to get it working, removing the -p causes the tests to fail again.

from testfixtures.

I found this commit that might have something to do with it: golang/go@0a0f8ba#diff-9a7673701332f3ded83deb4c307eefc9R1424
Basically it looks like go will run tests in parallel across packages by default?

from testfixtures.

Hi @redorb,

Thanks for the information! Now, it's clear for me that go test ./... will really run tests for each package in parallel.

I updated the documentation to mention this issue.

from testfixtures.