grandma run testname

.command('run <testname> [options]', 'Run a test by name.', function(yargs) {
    return yargs
        .demand(2, ['duration', 'rate'], 'Example: --duration=10m --rate=50')
        .example('grandma run testname --duration=2m --rate=200');
})

Even thought there are aliases for duration and rate, those aliases to not satisfy the demands and the command does not run.

Use the context object to allow things like this.metric.myName = 1234.

• Numeric metrics only
• Consider what happens when the metric is left undefined by some test runs
• Report on custom metrics

... to help with large files

https://github.com/bluesmoon/node-faststats

Detect outliers and do one/all of these:

• remove them from the data set entirely
• calculate summaries with and without them
• display plots without outliers

I get the following error:

> grandma "run"

env: node\r: No such file or directory

I switched to unix line endings in ./bin/__cli and that fixed the problem for me.

• use silent forks by default
• do not allow logging in beforeAll and afterAll
• add --debug flag to allow logging

Seems like yargs does not like what I am doing here:

.command('report <glob> [options]', 'Summarize the content of report files.', function(yargs) {
    return yargs
        .demand('glob', 'The files to report. Example: *.txt');
})

Blocked by #9

I did a -d 5ms -r 5 and the tests never stopped. Weird.

Need to define a format for this output.

I am thinking "perf".

Also, consider a .grandmarc file to use for the default options, instead of following the mocha model.

Something that uses the standard callback(err, optionalData), but I would perhaps want to define some error data that can be reported on. Assume that the lack of an error means the test succeeded.

Right now, grandma will wait for any existing test to finish after reaching the test duration before exiting. If the tests get stuck, grandma will never exit. It needs a max timeout to wait before exiting, because sanity.

• add configurable timeout
• use Duration-js for CLI timeout duration
• unit tests
• update README
• update API doc
• add to CLI help

grandma list

• consider allowing custom name tests instead of using the file name
• consider allowing custom test description to be used in the list command

See text-table for making tables.

This means that when one test is done, it triggers n new tests to start, n being the concurrent number set to run, causing test runs to snowball.

The API definitely suggests this, but it does not quite work right now.

... for when you want to run grandma for a really long time

https://github.com/catdad/grandma/blob/master/lib/stats.js#L49

Sometimes, tests might take seconds or minutes to complete, and always measuring the y-axis in milliseconds makes those results difficult to read.

Reading gzipped report functionality was removed in 5094e14. The util to combine the streams will not need to unzip them before combining.

Allow a maximum concurrency mode instead of rate.

"Run x number of tests at the same time for y period of time."

Current stats are super slow on large data sets... probably due to the 3 percentile calls.

Since execution is done in threads, it is hard to have a shared beforeAll and afterAll.

1. One option would be to execute those in the parent before and after the test runs, but that would not allow the scope to be shared. Ideally, we would want the scope to be shared between the separate threads as well.
2. Another option would be to ignore scope for those methods.

Do note that if the test module itself tries to set a global variable in beforeAll, that variable won't be available to all threads running the test, since each one requires the module individually.

Text files get really big really quickly. Think about something that would make these smaller... maybe gzipping?

grandma report {glob}

• Consider API that allows combining multiple test output files in order to generate one report.
• Keep in mind that these test outputs can be super long.