kuredoro / scold Goto Github PK

Input/Output black box test automation tool.

License: BSD 3-Clause "New" or "Revised" License

Go 98.92% C++ 0.97% JavaScript 0.12%

competitive-programming tool productivity testing test-automation testing-tools go code-runner code-execution command-line

scold's People

Stargazers

Watchers

scold's Issues

`ignore_last_newline` test suite option

Proposal to add ignore_last_newline test suite option to allow the program's output to have no trailing new line.

Restore backwards-compatibility of `tl` test suite option

Old tl expects a float and assumes that they are seconds, but #28 introduces a new syntax. These two forms, however, do not contradict each other, but switching from one to another begs new major version, which is unacceptable.

On Runtime Error verdict, also print the exit code.

This could be used for debugging by the users.

Add option to choose the number of worker threads

Right now, the application under testing will be run N times in parallel, where N = number of tests. On some real-time systems (like mine with linux-zen that hangs if something spawns armies of itself) instantiating a lot of processes can hang the machine.

Suggestion: add the option to set the max number of processes running in parallel.
CLI flag: -w --workers N

Refactor printer using a decorator pattern

As much of functionality should be in the main package and tested, but the artistic representation of scold cli is untestable, because it is data.

The current "printer" (which is a function with lots of global state) does more than just printing: it pushes the events into a channel and then consumes them in another thread. The code looks messy to say the least.

We can separate the logic into small testable pieces concerned with only one thing: accepting and consuming events in another thread? Should be one printer. Accepting events and consuming them, but only when some condition is met? Another printer. Finally, the pretty printer should contain as little logic as possible.

Course of action:

Replace callbacks in TestingBatch with an interface
Implement AsyncPrinter wrapper
Move this big printing function to the PrettyPrinter struct
Modify main to work with the new printers.

Add `--fail-fast` flag to stop testing after the first non `OK` verdict

Not to wait the whole test suite to finish and waste time.

Cannot forward arguments to the executable that start with hyphen

I just found out a flaw in go-arg that prevents it from reading positional string arguments that start with hyphen. That basically means that negative numbers and flags cannot be forwarded to the executable.

Executable search fails on Windows

The executable is deemed executable currently if it has at least one executable permission. But on Windows, these are always zero. A more sophisticated building and inclusion of sources for different OSes is probably needed for cross-platform solution.

Add `--ordered` or make tests ordered by default

I'm really 50/50 here. On one hand, ordered output of tests is good, because it makes the testing results reproducible. When you do competitive programming, then you might want to expect that failing test to appear always in the same part of the screen. On the other hand, the rate of the output will be "jagged", so it will not be... pretty?

Add option to specify arguments to the executable

Use case: not all languages compile. Dynamic languages like JS need to specify the interpreter and the path to the code. So, with this feature we could do something like this:

$ cptest -i inputs.txt node --args code

cptest should return non-zero error code on any errors before test execution and on IE verdicts.

Pretty straightforward, but because go's native way is to just return from main, this was not implemented.

Detect negative exit codes and print negative versions instead.

Example main.cpp

int main()
{
    return -1;
}

Current output:

$ scold a.exe
--- RE: Test 1 (0.014s)
Input:
-1

Answer:
-1\n

Exit code: 4294967295

Output:
Stderr:

Proposal: make the output be

$ scold a.exe
--- RE: Test 3 (0.014s)
Input:
-1

Answer:
-1\n

Exit code: -1 (unsigned: 4294967295)

Output:
Stderr:

On TL correct answer is not shown

main.cpp

#include <iostream>

using namespace std;

int main()
{
    int a;
    cin >> a;

    while (true)
    {
        a++;
    }

    return 0;
}

inputs.txt

1
---
1
===
2
---
2
===
3
---
3
===
4
---
4
===
5
---
5

If the endless loop is removed, and replaced with cout << a + 1 << '\n' to cause WA, the answer IS shown.

Configuration is not validated

Test suite configuration can be defined as the first test case in the inputs file. The problem is that the values are not checked in any way. For instance,

tl = -1.5
prec = -1
===
1.234
---
1.234

Will result in panic:

fatal error: concurrent map read and map write

Additionally, command line arguments should be checker also. Specifically, -j flag that can be invoked with negative numbers:

$ cptest -j -5 a.out
panic: makechan: size out of range

goroutine 1 [running]:
github.com/shettyh/threadpool.NewThreadPool(0xfffffffffffffffb, 0xfffffffffffffffb, 0x400)
	/home/nekonyaaa/go/pkg/mod/github.com/shettyh/[email protected]/threadpool.go:22 +0x65
github.com/kuredoro/cptest.NewThreadPool(...)
	/cptest/thread_pool.go:39
main.main()
	cptest/cmd/cptest/main.go:127 +0x2f0

Update docs and code to replace "key" terminology with "field"

At the time of writing ScanConfig the terminology for the "key" part of key-value pairs was... "key", but with the introduction of StringMapUnmarshal a synonym for "key" was introduced: a "field".

It seems, "field" sounds and describes the concept better... But in any case, without this discrepancy between terminology, the code will be easier to read.

Executbles in the PATH should be acceptable as executables

This feature is needed for dynamic languages support: node, python, ruby, etc. all call an interpreter with an argument as the source file. But right now, since these interpreters are not in the current directory, they are not found.

Add linters and go fmt to the CI workflow

In particular golang-ci

Move to a printing library that allows embedding colors inside format strings

I want "error" to be highlighted red. This is already done for inputs.txt errors, but it's not done universally. The main problem is that this colored "error" should be constructed outside of Printf and then embedded into it as Printf("%v", errorStr). The reason why it needs to be constructed outside is that aurora.Au colorizer is either enabled or disable depending on cli flags, and so the "error" label should be constructed after init().

We could extract these colorful strings into the global scope and continue embedding them into Printf, but it would be much more readable if we didn't embed them, and all the arguments to Printf were actually meaningful.

Add flag that allows to choose what tests to run.

Something in the spirit of

$ cptest -t 1 3 4 app
=== RUN Test 1
=== RUN Test 3
=== RUN Test 4
...

This could be used together with #4 to debug a specific test.

Pretty progress bar

It was kinda beautiful when cptest simulated output of go test, but with worker thread in place this is not the case anymore

$ cptest a.out -j 4 --no-colors
time limit: 1.5s
floating point precision: 6 digit(s)
job count: 4
=== RUN	Test 1
=== RUN	Test 2
=== RUN	Test 3
=== RUN	Test 4
=== RUN	Test 5
--- OK:	Test 2 (0.198s)
=== RUN	Test 6
--- OK:	Test 4 (0.527s)
=== RUN	Test 7
--- OK:	Test 3 (0.704s)
=== RUN	Test 8
--- OK:	Test 6 (0.265s)
=== RUN	Test 9
--- OK:	Test 1 (0.797s)

The RUN and then verdict part is very repetitive, so it's better to get rid of it and replace with a progress bar.

`--output=json` flag to allow external tooling to be developed on top of cptest

cptest produces go test -v-like reports, but if somebody wants to process the reports themselves (to generate html pages, etc.), they have no pain-less way to do it. This proposal aims to increase interoperability with other tools.

The format needs to be decided. It should interoperate with #6 too. This is minimal, I think

[
    {
        "input": "foo\n",
        "expected": "foo\n",
        "stdout": "bar\n",
        "stderr": "whaaa?\n",
        "exit_code": 0,
        "run_time": "1.04s",
        "verdict": "TL"
    },
    null,
    {
        // ...
    }
]

Add `--postpone-report` flag

The --postpone-report flag should print all verdict diagnostics after all the test cases were run. It should interplay with #31 and #57 features.

Processes that exceed the time limit are not killed

The TestingBatch does not kill goroutines that time limited and they still can push the completion result back to the complete channel.

`raw` test suite option

Proposal to add raw test suite option to prevent scraping of the whitespaces, tabs, and all other non-printable characters while parsing answers and program's outputs.

cptest cli is not tested

My initial idea was to separate testable code into a library and untestable code into a cli application. That's, in fact, why I'm obsessed with 100% coverage on it. But now it's clear that even though cli has specifics like fancy printers or progress bars, it doesn't mean they shouldn't be tested! They should, and mainly, because they can be tested, unlike Executable that implements Processer. I now have end up with _examples folder on which I check that cptest works as expected. Are we back to manual labor? I want to make sure that cli works as expected via tests, so that I don't need to check it manually. When #33 gets merged, the labor will be tripled (or even novemplied!)

If stderr is not empty, but test is WA, print the stderr anyway.

This could be used for debugging purposes by the users.

Print output and stderr on TL if any

Right now, on TL no useful information is printed.

Make all "error:" colored

Add escape sequences to `inputs.txt`

So that non-printable characters could be written down in printable ASCII

Refactor core library API to consider backwards compatibility

The API is very rigid and nearly every issue in issues requires API break.

Ah... If only I figured out versioning before starting this project and marking it with v0....

Unknown configuration fields should emit a warning, not an error.

The fact that these are errors, essentially breaks the future backwards-compatibility! * facepalm *

Unfortunately, we didn't have an architecture to support warnings yet at the time :(

If executable is a directory, cptest won't prevent itself from running.

Reproduce as:

$ cptest .
time limit: 1.5s
floating point precision: 6 digit(s)
job count: 12
--- IE:	Test 12 (0.001s)
Input:
275

Answer:
275\n

Error:
executable: fork/exec /home/nekonyaaa/code/skyarrow/cptest/eg/04_1000_tests: permission denied

--- IE:	Test 5 (0.001s)
Input:
675

Answer:
675\n

Error:
executable: fork/exec /home/nekonyaaa/code/skyarrow/cptest/eg/04_1000_tests: permission denied

--- IE:	Test 13 (0.000s)
Input:
455

Answer:
455\n

Error:
executable: fork/exec /home/nekonyaaa/code/skyarrow/cptest/eg/04_1000_tests: permission denied
< and so on >

Don't print colors and progress bar in non-tty mode.

This is easy to do via is-tty package in init(). However, we need to provide a possibility to force color output, even if stdout is not a tty.

Proposition for new cli:

$ cptest --color always --progress-bar always a.out
$ cptest --color never --progress-bar never a.out

New lines mismatch

If an output has one more or one less new line lexeme, then all lexemes after it will be mismatched. Consider ignoring excessive new lines when comparing outputs.

Add README.md for each _example

Describe what _example shows and how it is intended to be run

Deprecate time limits without a unit suffix and add warnings for `inputs.txt` parser.

Proposed diagnostics:

$ scold a.out
inputs.txt:1: warning: a value of PositiveDuration type without a unit suffix is deprecated. Specify seconds explicitly with 's' suffix.
     1| tl = 10.0
<rest of the scold output as if everything was okay>

Rare occasional panic in tests

The runtime and internal error test for TestingBatch can occasionally panic or fail.

Error reported on occasional fail:

TestingBatch_test.go:163: test was called 4 times, want 5

Backtrace for panic:

fatal error: concurrent map read and map write

goroutine 45 [running]:
runtime.throw(0x5a7e3a, 0x21)
	/usr/lib/go/src/runtime/panic.go:1116 +0x72 fp=0xc000111cf8 sp=0xc000111cc8 pc=0x436d52
runtime.mapaccess1_fast64(0x578260, 0xc000108e70, 0x2, 0xc000178af8)
	/usr/lib/go/src/runtime/map_fast64.go:21 +0x198 fp=0xc000111d20 sp=0xc000111cf8 pc=0x4120b8
github.com/kureduro/cptest.(*TestingBatch).Run(0xc00014e280)
	/home/nekonyaaa/code/skyarrow/cptest/TestingBatch.go:165 +0x445 fp=0xc000111e48 sp=0xc000111d20 pc=0x54f825
github.com/kureduro/cptest_test.TestTestingBatch.func3(0xc000170c00)
	/home/nekonyaaa/code/skyarrow/cptest/TestingBatch_test.go:152 +0x2bb fp=0xc000111f80 sp=0xc000111e48 pc=0x55629b
testing.tRunner(0xc000170c00, 0x5ac9a8)
	/usr/lib/go/src/testing/testing.go:1123 +0xef fp=0xc000111fd0 sp=0xc000111f80 pc=0x50228f
runtime.goexit()
	/usr/lib/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc000111fd8 sp=0xc000111fd0 pc=0x46db21
created by testing.(*T).Run
	/usr/lib/go/src/testing/testing.go:1168 +0x2b3

goroutine 1 [chan receive]:
testing.(*T).Run(0xc000170480, 0x5a3438, 0x10, 0x5ac9c0, 0x48c901)
	/usr/lib/go/src/testing/testing.go:1169 +0x2da
testing.runTests.func1(0xc00011e180)
	/usr/lib/go/src/testing/testing.go:1439 +0x78
testing.tRunner(0xc00011e180, 0xc00008dde0)
	/usr/lib/go/src/testing/testing.go:1123 +0xef
testing.runTests(0xc00012c120, 0x6b1ec0, 0x6, 0x6, 0xbfde77c668101ca4, 0x8bb2cc8dae, 0x6b5b00, 0x40d430)
	/usr/lib/go/src/testing/testing.go:1437 +0x2fe
testing.(*M).Run(0xc00014e000, 0x0)
	/usr/lib/go/src/testing/testing.go:1345 +0x1eb
main.main()
	_testmain.go:55 +0x138

goroutine 38 [chan receive]:
testing.(*T).Run(0xc000170c00, 0x5a7ab0, 0x20, 0x5ac9a8, 0x2e50d58e501)
	/usr/lib/go/src/testing/testing.go:1169 +0x2da
github.com/kureduro/cptest_test.TestTestingBatch(0xc000170480)
	/home/nekonyaaa/code/skyarrow/cptest/TestingBatch_test.go:100 +0xae
testing.tRunner(0xc000170480, 0x5ac9c0)
	/usr/lib/go/src/testing/testing.go:1123 +0xef
created by testing.(*T).Run
	/usr/lib/go/src/testing/testing.go:1168 +0x2b3

goroutine 46 [chan send]:
github.com/kureduro/cptest.(*TestingBatch).launchTest.func1.1(0xc00014e280, 0x1)
	/home/nekonyaaa/code/skyarrow/cptest/TestingBatch.go:109 +0x1f9
github.com/kureduro/cptest.(*TestingBatch).launchTest.func1(0xc00014e280, 0x1, 0x5a0c3d, 0x1)
	/home/nekonyaaa/code/skyarrow/cptest/TestingBatch.go:120 +0x2bf
created by github.com/kureduro/cptest.(*TestingBatch).launchTest
	/home/nekonyaaa/code/skyarrow/cptest/TestingBatch.go:99 +0x5d

goroutine 48 [chan send]:
github.com/kureduro/cptest.(*TestingBatch).launchTest.func1.1(0xc00014e280, 0x3)
	/home/nekonyaaa/code/skyarrow/cptest/TestingBatch.go:109 +0x1f9
github.com/kureduro/cptest.(*TestingBatch).launchTest.func1(0xc00014e280, 0x3, 0x5a0c3f, 0x1)
	/home/nekonyaaa/code/skyarrow/cptest/TestingBatch.go:120 +0x2bf
created by github.com/kureduro/cptest.(*TestingBatch).launchTest
	/home/nekonyaaa/code/skyarrow/cptest/TestingBatch.go:99 +0x5d

Possibly this is issue with the defer statement in launchTest function. However, further investigation is needed.

Printer callback promotes accessing `TestingBatch` without synchronization.

Reproducible, but not always:

$ cptest .
< aronud 100 tests >

Error:
executable: fork/exec /home/nekonyaaa/code/skyarrow/cptest/eg/04_1000_tests: permission denied

fatal error: concurrent map read and map write

goroutine 68 [running]:
runtime.throw({0x526637, 0x0})
	/usr/local/go/src/runtime/panic.go:1198 +0x71 fp=0xc00019fea0 sp=0xc00019fe70 pc=0x432831
runtime.mapaccess1_fast64(0x54b280, 0xc0000b2060, 0x54)
	/usr/local/go/src/runtime/map_fast64.go:21 +0x172 fp=0xc00019fec0 sp=0xc00019fea0 pc=0x411332
main.printVerboseResult(0xc0004b4000)
	/home/nekonyaaa/code/skyarrow/cptest/cmd/cptest/printer.go:64 +0x2fa fp=0xc00019ff88 sp=0xc00019fec0 pc=0x4f1cda
main.verboseResultPrinterWorker()
	/home/nekonyaaa/code/skyarrow/cptest/cmd/cptest/printer.go:34 +0x3e fp=0xc00019ffb8 sp=0xc00019ff88 pc=0x4f199e
main.main.func1()
	/home/nekonyaaa/code/skyarrow/cptest/cmd/cptest/main.go:163 +0x25 fp=0xc00019ffe0 sp=0xc00019ffb8 pc=0x4f17e5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00019ffe8 sp=0xc00019ffe0 pc=0x460ba1
created by main.main
	/home/nekonyaaa/code/skyarrow/cptest/cmd/cptest/main.go:162 +0x665

goroutine 1 [select]:
github.com/kuredoro/cptest.(*TestingBatch).Run(0xc00018c0c0)
	/home/nekonyaaa/code/skyarrow/cptest/testing_batch.go:208 +0x314
main.main()
	/home/nekonyaaa/code/skyarrow/cptest/cmd/cptest/main.go:167 +0x672

goroutine 6 [runnable]:
os.newFile(0xa, {0x51fa7d, 0x2}, 0x2)
	/usr/local/go/src/os/file_unix.go:123 +0x1a9
os.Pipe()
	/usr/local/go/src/os/pipe_linux.go:32 +0x1a5
os/exec.(*Cmd).StdoutPipe(0xc00049e000)
	/usr/local/go/src/os/exec/exec.go:626 +0x3e
main.(*Executable).Run(0x504820, {0x54d298, 0xc000518040}, {0x54b2a0, 0xc000204000})
	/home/nekonyaaa/code/skyarrow/cptest/cmd/cptest/executable.go:21 +0xe5
github.com/kuredoro/cptest.(*TestingBatch).launchTest(0xc00018c0c0, 0x61, {0xc0001347b0, 0x4})
	/home/nekonyaaa/code/skyarrow/cptest/testing_batch.go:155 +0x2b9
github.com/kuredoro/cptest.(*TestingBatch).Run.func3()
	/home/nekonyaaa/code/skyarrow/cptest/testing_batch.go:230 +0x3e
github.com/kuredoro/cptest.RunnableFunc.Run(0xc000264480)
	/home/nekonyaaa/code/skyarrow/cptest/thread_pool.go:22 +0x1a
github.com/shettyh/threadpool.Worker.executeJob({0xc000057fb0, 0xc000057f90, 0x0}, {0x504520, 0xc0004a2000})
	/home/nekonyaaa/go/pkg/mod/github.com/shettyh/[email protected]/worker.go:40 +0xc2
github.com/shettyh/threadpool.Worker.Start.func1()
	/home/nekonyaaa/go/pkg/mod/github.com/shettyh/[email protected]/worker.go:26 +0x3f
created by github.com/shettyh/threadpool.Worker.Start
	/home/nekonyaaa/go/pkg/mod/github.com/shettyh/[email protected]/worker.go:17 +0xaf

goroutine 7 [runnable]:
github.com/shettyh/threadpool.Worker.Start.func1()
	/home/nekonyaaa/go/pkg/mod/github.com/shettyh/[email protected]/worker.go:22 +0xa8
created by github.com/shettyh/threadpool.Worker.Start
	/home/nekonyaaa/go/pkg/mod/github.com/shettyh/[email protected]/worker.go:17 +0xaf

goroutine 8 [semacquire]:
sync.runtime_SemacquireMutex(0xc000022000, 0x44, 0xc0003a7400)
	/usr/local/go/src/runtime/sema.go:71 +0x25
sync.(*Mutex).lockSlow(0x637ae0)
	/usr/local/go/src/sync/mutex.go:138 +0x165
sync.(*Mutex).Lock(...)
	/usr/local/go/src/sync/mutex.go:81
sync.(*RWMutex).Lock(0xc0003f7b00)
	/usr/local/go/src/sync/rwmutex.go:111 +0x36
syscall.forkExec({0xc00001a0c0, 0xc000022000}, {0xc000395550, 0x1, 0x1}, 0x619426f900000300)
	/usr/local/go/src/syscall/exec_unix.go:204 +0x312
syscall.StartProcess(...)
	/usr/local/go/src/syscall/exec_unix.go:264
os.startProcess({0xc00001a0c0, 0xc0003f78c0}, {0xc000395550, 0x1, 0x1}, 0xc0004bbbf0)
	/usr/local/go/src/os/exec_posix.go:55 +0x332
os.StartProcess({0xc00001a0c0, 0x35}, {0xc000395550, 0x1, 0x1}, 0x7fcb68255448)
	/usr/local/go/src/os/exec.go:109 +0x5a
os/exec.(*Cmd).Start(0xc0003fe420)
	/usr/local/go/src/os/exec/exec.go:422 +0x60a
main.(*Executable).Run(0x504820, {0x54d298, 0xc0003b73c0}, {0x54b2a0, 0xc00039ba20})
	/home/nekonyaaa/code/skyarrow/cptest/cmd/cptest/executable.go:31 +0x2ae
github.com/kuredoro/cptest.(*TestingBatch).launchTest(0xc00018c0c0, 0x5a, {0xc000134740, 0x4})
	/home/nekonyaaa/code/skyarrow/cptest/testing_batch.go:155 +0x2b9
github.com/kuredoro/cptest.(*TestingBatch).Run.func3()
	/home/nekonyaaa/code/skyarrow/cptest/testing_batch.go:230 +0x3e
github.com/kuredoro/cptest.RunnableFunc.Run(0xc00036d8c0)
	/home/nekonyaaa/code/skyarrow/cptest/thread_pool.go:22 +0x1a
github.com/shettyh/threadpool.Worker.executeJob({0xc0004bbfb0, 0xc0004bbf90, 0x0}, {0x504520, 0xc0003167b0})
	/home/nekonyaaa/go/pkg/mod/github.com/shettyh/[email protected]/worker.go:40 +0xc2
github.com/shettyh/threadpool.Worker.Start.func1()
	/home/nekonyaaa/go/pkg/mod/github.com/shettyh/[email protected]/worker.go:26 +0x3f
created by github.com/shettyh/threadpool.Worker.Start
	/home/nekonyaaa/go/pkg/mod/github.com/shettyh/[email protected]/worker.go:17 +0xaf

goroutine 9 [semacquire]:
sync.runtime_SemacquireMutex(0xc000022000, 0x44, 0xc0004aca00)
	/usr/local/go/src/runtime/sema.go:71 +0x25
sync.(*Mutex).lockSlow(0x637ae0)
	/usr/local/go/src/sync/mutex.go:138 +0x165
sync.(*Mutex).Lock(...)
	/usr/local/go/src/sync/mutex.go:81
sync.(*RWMutex).Lock(0xc000788240)
	/usr/local/go/src/sync/rwmutex.go:111 +0x36
syscall.forkExec({0xc00001a0c0, 0xc000022000}, {0xc00049aa20, 0x1, 0x1}, 0xeafdb8a800000300)
	/usr/local/go/src/syscall/exec_unix.go:204 +0x312
syscall.StartProcess(...)
	/usr/local/go/src/syscall/exec_unix.go:264
os.startProcess({0xc00001a0c0, 0xc000788000}, {0xc00049aa20, 0x1, 0x1}, 0xc000058bf0)
	/usr/local/go/src/os/exec_posix.go:55 +0x332
os.StartProcess({0xc00001a0c0, 0x35}, {0xc00049aa20, 0x1, 0x1}, 0x7fcb682d6878)
	/usr/local/go/src/os/exec.go:109 +0x5a
os/exec.(*Cmd).Start(0xc00049f340)
	/usr/local/go/src/os/exec/exec.go:422 +0x60a
main.(*Executable).Run(0x504820, {0x54d298, 0xc000498ac0}, {0x54b2a0, 0xc00049cc80})
	/home/nekonyaaa/code/skyarrow/cptest/cmd/cptest/executable.go:31 +0x2ae
github.com/kuredoro/cptest.(*TestingBatch).launchTest(0xc00018c0c0, 0x59, {0xc000134730, 0x4})
	/home/nekonyaaa/code/skyarrow/cptest/testing_batch.go:155 +0x2b9
github.com/kuredoro/cptest.(*TestingBatch).Run.func3()
	/home/nekonyaaa/code/skyarrow/cptest/testing_batch.go:230 +0x3e
github.com/kuredoro/cptest.RunnableFunc.Run(0xc000264360)
	/home/nekonyaaa/code/skyarrow/cptest/thread_pool.go:22 +0x1a
github.com/shettyh/threadpool.Worker.executeJob({0xc000058fb0, 0xc000058f90, 0x0}, {0x504520, 0xc0004a2540})
	/home/nekonyaaa/go/pkg/mod/github.com/shettyh/[email protected]/worker.go:40 +0xc2
github.com/shettyh/threadpool.Worker.Start.func1()
	/home/nekonyaaa/go/pkg/mod/github.com/shettyh/[email protected]/worker.go:26 +0x3f
created by github.com/shettyh/threadpool.Worker.Start
	/home/nekonyaaa/go/pkg/mod/github.com/shettyh/[email protected]/worker.go:17 +0xaf

goroutine 10 [runnable]:
sync.runtime_Semrelease(0x0, 0x0, 0x2336)
	/usr/local/go/src/runtime/sema.go:66 +0x19
sync.(*Mutex).unlockSlow(0x0, 0x0)
	/usr/local/go/src/sync/mutex.go:224 +0x65
sync.(*Mutex).Unlock(...)
	/usr/local/go/src/sync/mutex.go:190
sync.(*RWMutex).Unlock(0x637ae0)
	/usr/local/go/src/sync/rwmutex.go:149 +0x9b
syscall.forkExec({0xc00001a0c0, 0xc000022000}, {0xc00049a850, 0x1, 0x1}, 0x199303200000300)
	/usr/local/go/src/syscall/exec_unix.go:217 +0x3c6
syscall.StartProcess(...)
	/usr/local/go/src/syscall/exec_unix.go:264
os.startProcess({0xc00001a0c0, 0xc0004fdb00}, {0xc00049a850, 0x1, 0x1}, 0xc000514bf0)
	/usr/local/go/src/os/exec_posix.go:55 +0x332
os.StartProcess({0xc00001a0c0, 0x35}, {0xc00049a850, 0x1, 0x1}, 0x7fcb682d6878)
	/usr/local/go/src/os/exec.go:109 +0x5a
os/exec.(*Cmd).Start(0xc00049f1e0)
	/usr/local/go/src/os/exec/exec.go:422 +0x60a
main.(*Executable).Run(0x504820, {0x54d298, 0xc000498a00}, {0x54b2a0, 0xc00049ca40})
	/home/nekonyaaa/code/skyarrow/cptest/cmd/cptest/executable.go:31 +0x2ae
github.com/kuredoro/cptest.(*TestingBatch).launchTest(0xc00018c0c0, 0x56, {0xc000134700, 0x4})
	/home/nekonyaaa/code/skyarrow/cptest/testing_batch.go:155 +0x2b9
github.com/kuredoro/cptest.(*TestingBatch).Run.func3()
	/home/nekonyaaa/code/skyarrow/cptest/testing_batch.go:230 +0x3e
github.com/kuredoro/cptest.RunnableFunc.Run(0xc00036d7a0)
	/home/nekonyaaa/code/skyarrow/cptest/thread_pool.go:22 +0x1a
github.com/shettyh/threadpool.Worker.executeJob({0xc000514fb0, 0xc000514f90, 0x0}, {0x504520, 0xc000316780})
	/home/nekonyaaa/go/pkg/mod/github.com/shettyh/[email protected]/worker.go:40 +0xc2
github.com/shettyh/threadpool.Worker.Start.func1()
	/home/nekonyaaa/go/pkg/mod/github.com/shettyh/[email protected]/worker.go:26 +0x3f
created by github.com/shettyh/threadpool.Worker.Start
	/home/nekonyaaa/go/pkg/mod/github.com/shettyh/[email protected]/worker.go:17 +0xaf

goroutine 11 [semacquire]:
sync.runtime_SemacquireMutex(0xc000022000, 0x44, 0xc00022aa00)
	/usr/local/go/src/runtime/sema.go:71 +0x25
sync.(*Mutex).lockSlow(0x637ae0)
	/usr/local/go/src/sync/mutex.go:138 +0x165
sync.(*Mutex).Lock(...)
	/usr/local/go/src/sync/mutex.go:81
sync.(*RWMutex).Lock(0xc000239d40)
	/usr/local/go/src/sync/rwmutex.go:111 +0x36
syscall.forkExec({0xc00001a0c0, 0xc000022000}, {0xc000202b90, 0x1, 0x1}, 0x84d3fd9e00000300)
	/usr/local/go/src/syscall/exec_unix.go:204 +0x312
syscall.StartProcess(...)
	/usr/local/go/src/syscall/exec_unix.go:264
os.startProcess({0xc00001a0c0, 0xc000239b00}, {0xc000202b90, 0x1, 0x1}, 0xc000515bf0)
	/usr/local/go/src/os/exec_posix.go:55 +0x332
os.StartProcess({0xc00001a0c0, 0x35}, {0xc000202b90, 0x1, 0x1}, 0x7fcb682f1be0)
	/usr/local/go/src/os/exec.go:109 +0x5a
os/exec.(*Cmd).Start(0xc0002111e0)
	/usr/local/go/src/os/exec/exec.go:422 +0x60a
main.(*Executable).Run(0x504820, {0x54d298, 0xc000200a00}, {0x54b2a0, 0xc000204e60})
	/home/nekonyaaa/code/skyarrow/cptest/cmd/cptest/executable.go:31 +0x2ae
github.com/kuredoro/cptest.(*TestingBatch).launchTest(0xc00018c0c0, 0x58, {0xc000134720, 0x4})
	/home/nekonyaaa/code/skyarrow/cptest/testing_batch.go:155 +0x2b9
github.com/kuredoro/cptest.(*TestingBatch).Run.func3()
	/home/nekonyaaa/code/skyarrow/cptest/testing_batch.go:230 +0x3e
github.com/kuredoro/cptest.RunnableFunc.Run(0xc0002645a0)
	/home/nekonyaaa/code/skyarrow/cptest/thread_pool.go:22 +0x1a
github.com/shettyh/threadpool.Worker.executeJob({0xc000515fb0, 0xc000515f90, 0x0}, {0x504520, 0xc0004a2510})
	/home/nekonyaaa/go/pkg/mod/github.com/shettyh/[email protected]/worker.go:40 +0xc2
github.com/shettyh/threadpool.Worker.Start.func1()
	/home/nekonyaaa/go/pkg/mod/github.com/shettyh/[email protected]/worker.go:26 +0x3f
created by github.com/shettyh/threadpool.Worker.Start
	/home/nekonyaaa/go/pkg/mod/github.com/shettyh/[email protected]/worker.go:17 +0xaf

goroutine 12 [semacquire]:
sync.runtime_SemacquireMutex(0xc000022000, 0x44, 0xc00058e100)
	/usr/local/go/src/runtime/sema.go:71 +0x25
sync.(*Mutex).lockSlow(0x637ae0)
	/usr/local/go/src/sync/mutex.go:138 +0x165
sync.(*Mutex).Lock(...)
	/usr/local/go/src/sync/mutex.go:81
sync.(*RWMutex).Lock(0x637ae0)
	/usr/local/go/src/sync/rwmutex.go:111 +0x36
syscall.forkExec({0xc00001a0c0, 0xc000022000}, {0xc0005080b0, 0x1, 0x1}, 0xd35dc13800000300)
	/usr/local/go/src/syscall/exec_unix.go:204 +0x312
syscall.StartProcess(...)
	/usr/local/go/src/syscall/exec_unix.go:264
os.startProcess({0xc00001a0c0, 0xc00050ab40}, {0xc0005080b0, 0x1, 0x1}, 0xc00043fbf0)
	/usr/local/go/src/os/exec_posix.go:55 +0x332
os.StartProcess({0xc00001a0c0, 0x35}, {0xc0005080b0, 0x1, 0x1}, 0x7fcb681c5c40)
	/usr/local/go/src/os/exec.go:109 +0x5a
os/exec.(*Cmd).Start(0xc0005802c0)
	/usr/local/go/src/os/exec/exec.go:422 +0x60a
main.(*Executable).Run(0x504820, {0x54d298, 0xc000506180}, {0x54b2a0, 0xc00050c0e0})
	/home/nekonyaaa/code/skyarrow/cptest/cmd/cptest/executable.go:31 +0x2ae
github.com/kuredoro/cptest.(*TestingBatch).launchTest(0xc00018c0c0, 0x5f, {0xc000134790, 0x3})
	/home/nekonyaaa/code/skyarrow/cptest/testing_batch.go:155 +0x2b9
github.com/kuredoro/cptest.(*TestingBatch).Run.func3()
	/home/nekonyaaa/code/skyarrow/cptest/testing_batch.go:230 +0x3e
github.com/kuredoro/cptest.RunnableFunc.Run(0xc00036daa0)
	/home/nekonyaaa/code/skyarrow/cptest/thread_pool.go:22 +0x1a
github.com/shettyh/threadpool.Worker.executeJob({0xc00043ffb0, 0xc00043ff90, 0x0}, {0x504520, 0xc00028a000})
	/home/nekonyaaa/go/pkg/mod/github.com/shettyh/[email protected]/worker.go:40 +0xc2
github.com/shettyh/threadpool.Worker.Start.func1()
	/home/nekonyaaa/go/pkg/mod/github.com/shettyh/[email protected]/worker.go:26 +0x3f
created by github.com/shettyh/threadpool.Worker.Start
	/home/nekonyaaa/go/pkg/mod/github.com/shettyh/[email protected]/worker.go:17 +0xaf

goroutine 13 [semacquire]:
sync.runtime_SemacquireMutex(0xc000022000, 0x44, 0xc0004acb00)
	/usr/local/go/src/runtime/sema.go:71 +0x25
sync.(*Mutex).lockSlow(0x637ae0)
	/usr/local/go/src/sync/mutex.go:138 +0x165
sync.(*Mutex).Lock(...)
	/usr/local/go/src/sync/mutex.go:81
sync.(*RWMutex).Lock(0xc000788b40)
	/usr/local/go/src/sync/rwmutex.go:111 +0x36
syscall.forkExec({0xc00001a0c0, 0xc000022000}, {0xc00049aac0, 0x1, 0x1}, 0x1a17021300010300)
	/usr/local/go/src/syscall/exec_unix.go:204 +0x312
syscall.StartProcess(...)
	/usr/local/go/src/syscall/exec_unix.go:264
os.startProcess({0xc00001a0c0, 0xc000788900}, {0xc00049aac0, 0x1, 0x1}, 0xc0000c5bf0)
	/usr/local/go/src/os/exec_posix.go:55 +0x332
os.StartProcess({0xc00001a0c0, 0x35}, {0xc00049aac0, 0x1, 0x1}, 0x7fcb682d6878)
	/usr/local/go/src/os/exec.go:109 +0x5a
os/exec.(*Cmd).Start(0xc00049f600)
	/usr/local/go/src/os/exec/exec.go:422 +0x60a
main.(*Executable).Run(0x504820, {0x54d298, 0xc000498c40}, {0x54b2a0, 0xc00049cd40})
	/home/nekonyaaa/code/skyarrow/cptest/cmd/cptest/executable.go:31 +0x2ae
github.com/kuredoro/cptest.(*TestingBatch).launchTest(0xc00018c0c0, 0x5c, {0xc000134760, 0x4})
	/home/nekonyaaa/code/skyarrow/cptest/testing_batch.go:155 +0x2b9
github.com/kuredoro/cptest.(*TestingBatch).Run.func3()
	/home/nekonyaaa/code/skyarrow/cptest/testing_batch.go:230 +0x3e
github.com/kuredoro/cptest.RunnableFunc.Run(0xc00036db00)
	/home/nekonyaaa/code/skyarrow/cptest/thread_pool.go:22 +0x1a
github.com/shettyh/threadpool.Worker.executeJob({0xc0000c5fb0, 0xc0000c5f90, 0x0}, {0x504520, 0xc000316810})
	/home/nekonyaaa/go/pkg/mod/github.com/shettyh/[email protected]/worker.go:40 +0xc2
github.com/shettyh/threadpool.Worker.Start.func1()
	/home/nekonyaaa/go/pkg/mod/github.com/shettyh/[email protected]/worker.go:26 +0x3f
created by github.com/shettyh/threadpool.Worker.Start
	/home/nekonyaaa/go/pkg/mod/github.com/shettyh/[email protected]/worker.go:17 +0xaf

goroutine 14 [semacquire]:
sync.runtime_SemacquireMutex(0xc000022000, 0x44, 0xc0004acb00)
	/usr/local/go/src/runtime/sema.go:71 +0x25
sync.(*Mutex).lockSlow(0x637ae0)
	/usr/local/go/src/sync/mutex.go:138 +0x165
sync.(*Mutex).Lock(...)
	/usr/local/go/src/sync/mutex.go:81
sync.(*RWMutex).Lock(0xc000788fc0)
	/usr/local/go/src/sync/rwmutex.go:111 +0x36
syscall.forkExec({0xc00001a0c0, 0xc000022000}, {0xc00049ab10, 0x1, 0x1}, 0xf101348200000300)
	/usr/local/go/src/syscall/exec_unix.go:204 +0x312
syscall.StartProcess(...)
	/usr/local/go/src/syscall/exec_unix.go:264
os.startProcess({0xc00001a0c0, 0xc000788d80}, {0xc00049ab10, 0x1, 0x1}, 0xc000510bf0)
	/usr/local/go/src/os/exec_posix.go:55 +0x332
os.StartProcess({0xc00001a0c0, 0x35}, {0xc00049ab10, 0x1, 0x1}, 0x7fcb682d6878)
	/usr/local/go/src/os/exec.go:109 +0x5a
os/exec.(*Cmd).Start(0xc00049f760)
	/usr/local/go/src/os/exec/exec.go:422 +0x60a
main.(*Executable).Run(0x504820, {0x54d298, 0xc000498d00}, {0x54b2a0, 0xc00049cda0})
	/home/nekonyaaa/code/skyarrow/cptest/cmd/cptest/executable.go:31 +0x2ae
github.com/kuredoro/cptest.(*TestingBatch).launchTest(0xc00018c0c0, 0x5d, {0xc000134770, 0x4})
	/home/nekonyaaa/code/skyarrow/cptest/testing_batch.go:155 +0x2b9
github.com/kuredoro/cptest.(*TestingBatch).Run.func3()
	/home/nekonyaaa/code/skyarrow/cptest/testing_batch.go:230 +0x3e
github.com/kuredoro/cptest.RunnableFunc.Run(0xc000215f20)
	/home/nekonyaaa/code/skyarrow/cptest/thread_pool.go:22 +0x1a
github.com/shettyh/threadpool.Worker.executeJob({0xc000510fb0, 0xc000510f90, 0x0}, {0x504520, 0xc0003989f0})
	/home/nekonyaaa/go/pkg/mod/github.com/shettyh/[email protected]/worker.go:40 +0xc2
github.com/shettyh/threadpool.Worker.Start.func1()
	/home/nekonyaaa/go/pkg/mod/github.com/shettyh/[email protected]/worker.go:26 +0x3f
created by github.com/shettyh/threadpool.Worker.Start
	/home/nekonyaaa/go/pkg/mod/github.com/shettyh/[email protected]/worker.go:17 +0xaf

goroutine 15 [semacquire]:
sync.runtime_SemacquireMutex(0xc000022000, 0x44, 0xc0003a7500)
	/usr/local/go/src/runtime/sema.go:71 +0x25
sync.(*Mutex).lockSlow(0x637ae0)
	/usr/local/go/src/sync/mutex.go:138 +0x165
sync.(*Mutex).Lock(...)
	/usr/local/go/src/sync/mutex.go:81
sync.(*RWMutex).Lock(0xc000706000)
	/usr/local/go/src/sync/rwmutex.go:111 +0x36
syscall.forkExec({0xc00001a0c0, 0xc000022000}, {0xc000395620, 0x1, 0x1}, 0x9ae60efd00000300)
	/usr/local/go/src/syscall/exec_unix.go:204 +0x312
syscall.StartProcess(...)
	/usr/local/go/src/syscall/exec_unix.go:264
os.startProcess({0xc00001a0c0, 0xc0003f7d40}, {0xc000395620, 0x1, 0x1}, 0xc0004bfbf0)
	/usr/local/go/src/os/exec_posix.go:55 +0x332
os.StartProcess({0xc00001a0c0, 0x35}, {0xc000395620, 0x1, 0x1}, 0x7fcb68255448)
	/usr/local/go/src/os/exec.go:109 +0x5a
os/exec.(*Cmd).Start(0xc0003fe580)
	/usr/local/go/src/os/exec/exec.go:422 +0x60a
main.(*Executable).Run(0x504820, {0x54d298, 0xc0003b7480}, {0x54b2a0, 0xc00039bb20})
	/home/nekonyaaa/code/skyarrow/cptest/cmd/cptest/executable.go:31 +0x2ae
github.com/kuredoro/cptest.(*TestingBatch).launchTest(0xc00018c0c0, 0x5e, {0xc000134780, 0x4})
	/home/nekonyaaa/code/skyarrow/cptest/testing_batch.go:155 +0x2b9
github.com/kuredoro/cptest.(*TestingBatch).Run.func3()
	/home/nekonyaaa/code/skyarrow/cptest/testing_batch.go:230 +0x3e
github.com/kuredoro/cptest.RunnableFunc.Run(0xc00036dc20)
	/home/nekonyaaa/code/skyarrow/cptest/thread_pool.go:22 +0x1a
github.com/shettyh/threadpool.Worker.executeJob({0xc0004bffb0, 0xc0004bff90, 0x0}, {0x504520, 0xc000316840})
	/home/nekonyaaa/go/pkg/mod/github.com/shettyh/[email protected]/worker.go:40 +0xc2
github.com/shettyh/threadpool.Worker.Start.func1()
	/home/nekonyaaa/go/pkg/mod/github.com/shettyh/[email protected]/worker.go:26 +0x3f
created by github.com/shettyh/threadpool.Worker.Start
	/home/nekonyaaa/go/pkg/mod/github.com/shettyh/[email protected]/worker.go:17 +0xaf

goroutine 16 [semacquire]:
sync.runtime_SemacquireMutex(0xc000022000, 0x44, 0xc00022aa00)
	/usr/local/go/src/runtime/sema.go:71 +0x25
sync.(*Mutex).lockSlow(0x637ae0)
	/usr/local/go/src/sync/mutex.go:138 +0x165
sync.(*Mutex).Lock(...)
	/usr/local/go/src/sync/mutex.go:81
sync.(*RWMutex).Lock(0xc0002398c0)
	/usr/local/go/src/sync/rwmutex.go:111 +0x36
syscall.forkExec({0xc00001a0c0, 0xc000022000}, {0xc000202b40, 0x1, 0x1}, 0xa50aaa1f00000300)
	/usr/local/go/src/syscall/exec_unix.go:204 +0x312
syscall.StartProcess(...)
	/usr/local/go/src/syscall/exec_unix.go:264
os.startProcess({0xc00001a0c0, 0xc000239680}, {0xc000202b40, 0x1, 0x1}, 0xc00043bbf0)
	/usr/local/go/src/os/exec_posix.go:55 +0x332
os.StartProcess({0xc00001a0c0, 0x35}, {0xc000202b40, 0x1, 0x1}, 0x7fcb682f1be0)
	/usr/local/go/src/os/exec.go:109 +0x5a
os/exec.(*Cmd).Start(0xc000211080)
	/usr/local/go/src/os/exec/exec.go:422 +0x60a
main.(*Executable).Run(0x504820, {0x54d298, 0xc000200940}, {0x54b2a0, 0xc000204e00})
	/home/nekonyaaa/code/skyarrow/cptest/cmd/cptest/executable.go:31 +0x2ae
github.com/kuredoro/cptest.(*TestingBatch).launchTest(0xc00018c0c0, 0x57, {0xc000134710, 0x4})
	/home/nekonyaaa/code/skyarrow/cptest/testing_batch.go:155 +0x2b9
github.com/kuredoro/cptest.(*TestingBatch).Run.func3()
	/home/nekonyaaa/code/skyarrow/cptest/testing_batch.go:230 +0x3e
github.com/kuredoro/cptest.RunnableFunc.Run(0xc00036d620)
	/home/nekonyaaa/code/skyarrow/cptest/thread_pool.go:22 +0x1a
github.com/shettyh/threadpool.Worker.executeJob({0xc00043bfb0, 0xc00043bf90, 0x0}, {0x504520, 0xc0004a24e0})
	/home/nekonyaaa/go/pkg/mod/github.com/shettyh/[email protected]/worker.go:40 +0xc2
github.com/shettyh/threadpool.Worker.Start.func1()
	/home/nekonyaaa/go/pkg/mod/github.com/shettyh/[email protected]/worker.go:26 +0x3f
created by github.com/shettyh/threadpool.Worker.Start
	/home/nekonyaaa/go/pkg/mod/github.com/shettyh/[email protected]/worker.go:17 +0xaf

goroutine 66 [semacquire]:
sync.runtime_SemacquireMutex(0xc000022000, 0x44, 0xc0004aca00)
	/usr/local/go/src/runtime/sema.go:71 +0x25
sync.(*Mutex).lockSlow(0x637ae0)
	/usr/local/go/src/sync/mutex.go:138 +0x165
sync.(*Mutex).Lock(...)
	/usr/local/go/src/sync/mutex.go:81
sync.(*RWMutex).Lock(0xc0007886c0)
	/usr/local/go/src/sync/rwmutex.go:111 +0x36
syscall.forkExec({0xc00001a0c0, 0xc000022000}, {0xc00049aa70, 0x1, 0x1}, 0xa6f0a1800000300)
	/usr/local/go/src/syscall/exec_unix.go:204 +0x312
syscall.StartProcess(...)
	/usr/local/go/src/syscall/exec_unix.go:264
os.startProcess({0xc00001a0c0, 0xc000788480}, {0xc00049aa70, 0x1, 0x1}, 0xc00021bbf0)
	/usr/local/go/src/os/exec_posix.go:55 +0x332
os.StartProcess({0xc00001a0c0, 0x35}, {0xc00049aa70, 0x1, 0x1}, 0x7fcb682d6878)
	/usr/local/go/src/os/exec.go:109 +0x5a
os/exec.(*Cmd).Start(0xc00049f4a0)
	/usr/local/go/src/os/exec/exec.go:422 +0x60a
main.(*Executable).Run(0x504820, {0x54d298, 0xc000498b80}, {0x54b2a0, 0xc00049cce0})
	/home/nekonyaaa/code/skyarrow/cptest/cmd/cptest/executable.go:31 +0x2ae
github.com/kuredoro/cptest.(*TestingBatch).launchTest(0xc00018c0c0, 0x5b, {0xc000134750, 0x4})
	/home/nekonyaaa/code/skyarrow/cptest/testing_batch.go:155 +0x2b9
github.com/kuredoro/cptest.(*TestingBatch).Run.func3()
	/home/nekonyaaa/code/skyarrow/cptest/testing_batch.go:230 +0x3e
github.com/kuredoro/cptest.RunnableFunc.Run(0xc00036d9e0)
	/home/nekonyaaa/code/skyarrow/cptest/thread_pool.go:22 +0x1a
github.com/shettyh/threadpool.Worker.executeJob({0xc00021bfb0, 0xc00021bf90, 0x0}, {0x504520, 0xc0003167e0})
	/home/nekonyaaa/go/pkg/mod/github.com/shettyh/[email protected]/worker.go:40 +0xc2
github.com/shettyh/threadpool.Worker.Start.func1()
	/home/nekonyaaa/go/pkg/mod/github.com/shettyh/[email protected]/worker.go:26 +0x3f
created by github.com/shettyh/threadpool.Worker.Start
	/home/nekonyaaa/go/pkg/mod/github.com/shettyh/[email protected]/worker.go:17 +0xaf

goroutine 67 [select]:
github.com/shettyh/threadpool.(*ThreadPool).dispatch(0xc00009a270)
	/home/nekonyaaa/go/pkg/mod/github.com/shettyh/[email protected]/threadpool.go:78 +0xeb
created by github.com/shettyh/threadpool.(*ThreadPool).createPool
	/home/nekonyaaa/go/pkg/mod/github.com/shettyh/[email protected]/threadpool.go:71 +0xaf

Then how should the progress bar look like?

// Section 1
  Section 1 |**********     ||| 78/100

// Section 2
  Section 2 ||*****          || 7/15

// Section 3
  Section 3 |||********       | 45/50

idea 2

// Section 1
  Testing  [=========>     ] <<

// Section 2
  Testing O [====>          ] <

// Section 3
  Testing OF [=======>       ] 

// Section 4
  Testing OFO

idea 3

// Section 1
  Section 1  [=========>     ] 78/100
  Suite      [               ] 0/3


// Section 2
  Section 2  [====>          ] 7/15
  Suite      [======>        ] 1/3


// Section 3
  Section 3  [============>  ] 45/50
  Suite      [===============] 2/3

Rare panic during runtime

using default time limit: 6s
=== RUN	Test 1
=== RUN	Test 2
=== RUN	Test 3
fatal error: concurrent map read and map write

goroutine 1 [running]:
runtime.throw(0x52b1d9, 0x21)
	/usr/lib/go/src/runtime/panic.go:1116 +0x72 fp=0xc000051bc0 sp=0xc000051b90 pc=0x433ad2
runtime.mapaccess1_fast64(0x506a40, 0xc00006e3c0, 0x3, 0xc00012c0d8)
	/usr/lib/go/src/runtime/map_fast64.go:21 +0x198 fp=0xc000051be8 sp=0xc000051bc0 pc=0x411358
github.com/kuredoro/cptest.(*TestingBatch).Run(0xc000126000)
	/home/nekonyaaa/code/skyarrow/cptest/TestingBatch.go:164 +0x37d fp=0xc000051d70 sp=0xc000051be8 pc=0x4e99bd
main.main()
	/home/nekonyaaa/code/skyarrow/cptest/cmd/cptest/main.go:137 +0x499 fp=0xc000051f88 sp=0xc000051d70 pc=0x4f0b79
runtime.main()
	/usr/lib/go/src/runtime/proc.go:204 +0x209 fp=0xc000051fe0 sp=0xc000051f88 pc=0x4362c9
runtime.goexit()
	/usr/lib/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc000051fe8 sp=0xc000051fe0 pc=0x466961

goroutine 6 [chan receive]:
github.com/kuredoro/cptest.NewConfigurableStopwatcher.func1(0x165a0bc00, 0xc00006c0c0)
	/home/nekonyaaa/code/skyarrow/cptest/stopwatcher.go:61 +0x97
created by github.com/kuredoro/cptest.NewConfigurableStopwatcher
	/home/nekonyaaa/code/skyarrow/cptest/stopwatcher.go:55 +0x6b

goroutine 7 [chan send]:
github.com/kuredoro/cptest.(*TestingBatch).launchTest.func1.1(0xc000126000, 0x1)
	/home/nekonyaaa/code/skyarrow/cptest/TestingBatch.go:114 +0x1f9
github.com/kuredoro/cptest.(*TestingBatch).launchTest.func1(0xc000126000, 0x1, 0xc0000142a0, 0x3)
	/home/nekonyaaa/code/skyarrow/cptest/TestingBatch.go:125 +0x2a7
created by github.com/kuredoro/cptest.(*TestingBatch).launchTest
	/home/nekonyaaa/code/skyarrow/cptest/TestingBatch.go:104 +0x5d

goroutine 8 [chan send]:
github.com/kuredoro/cptest.(*TestingBatch).launchTest.func1.1(0xc000126000, 0x2)
	/home/nekonyaaa/code/skyarrow/cptest/TestingBatch.go:114 +0x1f9
github.com/kuredoro/cptest.(*TestingBatch).launchTest.func1(0xc000126000, 0x2, 0xc0000142b0, 0x3)
	/home/nekonyaaa/code/skyarrow/cptest/TestingBatch.go:125 +0x2a7
created by github.com/kuredoro/cptest.(*TestingBatch).launchTest
	/home/nekonyaaa/code/skyarrow/cptest/TestingBatch.go:104 +0x5d

Prettier error messages

Consider this bad inputs.txt:

foo = bar
Tl = 10.0
= two
===

what the?

===
chu
---
chu

Right now, the error messages are like this:

$ cptest -i inputs.txt .
error: load tests: scan config: line 3: key cannot be empty
error: load tests: line 1: field "foo" doesn't exist
error: load tests: line 2: value "10.0" doesn't match Duration type
error: load tests: test 1: IO separator missing

Proposal for new output:

$ cptest -i inputs.txt .
inputs.txt:1: error: unknown configuration key "foo"
     1| foo = bar
inputs.txt:2: error: value "10.0" doesn't match Duration type
     2| Tl = 10.0
inputs.txt:3: error: configuration key cannot be empty
     3| = two
inputs.txt:5: error: test 1: IO separator missing
     5|
     6| what the?
     7|

Note that the errors are sorted by line numbers now.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.