mbland / go-script-bash Goto Github PK

Per http://mywiki.wooledge.org/Arguments, it may be better practice to replace IFS-based value splitting with something more like:

IFS=, read -ra value_array <<< "$value"

Joining items using ${value_array[*]} might still require the save-and-restore pattern in some form. I'm also considering encapsulating those operations in @go.split and @go.join functions, like with other languages.

Right now @go.log_command just logs the command itself, not its output. It should be updated to something like this:

while read -r line; do
  line="${line%$'\r'}"  # Windows
  line="${line//%/%%}" # printf
  line="${line//\\/\\\\}" # Backslashes
  for fd in "$([email protected]_level_file_desciptors "$level_index"); do
    printf "${line}\n"
  done
done < $("${args[@]}" 2>&1)
exit_status="$?"

Not sure how close the above is to what will actually work, especially with regards to exit_status. (Maybe $("$args[@]" 2>&1; exit_status="$?") would be the trick?)

Should be easy to run ./go new-command foo to generate the following and make it executable, and to open EDITOR if defined:

#! /usr/bin/env bash
#
# Description of foo

_foo() {
  :
}

_foo "$@"

There should probably be something similar for modules as well.

Right now it only strips escape codes of the form \\e\[[0-9]{1,3}m. It won't handle codes like \e[30;47m at all.

I'm thinking something like this (which might live in the format module):

  local format_pattern='\\e\[[0-9]+(;[0-9])*m'
  while [[ "$value" =~ $format_pattern ]]; do
    value="${value//${BASH_REMATCH[0]}}"
  done

Currently @go.log_timestamp returns its result by printing it to standard output. While this is the way one would typically return single values from shell functions, it's not ideal in this framework, since practically every other function uses a __go_* variable defined by the parent to return the value. This is to make the framework as fast as possible by creating as few subshells as possible, as those subshells add up quick in this context—especially on Windows, where fork() isn't directly supported, and requires an expensive workaround. (This last note needs to go in the go-script-bash coding/testing guide I need to write per #29.)

There could be a generic mechanism for an application to delegate to a specific plugin, such that it adopts the help text of a plugin script or module automatically.

Basically, a script can be written right now as:

#! /bin/bash
#
# Delegates to plugins/foo/bin/bar

. "$_GO_SCRIPTS_DIR/plugins/foo/bin/bar" "$@"

And we can eventually parse # Delegates to plugins/foo/bin/bar to include that help text instead. Eventually we could even remove the need to include any implementation, possibly making it optional.

Along with #82, implement @go.test_filter thus (already tried it and it works!):

@go.test_filter() {
  if [[ -n "$TEST_FILTER" && ! "$BATS_TEST_DESCRIPTION" =~ $TEST_FILTER ]]; then 
    skip 
  fi
}

Call this from the setup() of every test file, and then use it thus:

$ TEST_FILTER='strip' ./go test format
 - format: does nothing for empty argv (skipped)
 - format: pads argv items (skipped)
 - format: zip empty items (skipped)
 - format: zip matching items (skipped)
 ✓ format: strip formatting codes from empty string
 ✓ format: strip formatting codes from string with no codes
 ✓ format: strip formatting codes from string with one code
 ✓ format: strip formatting codes from string with multiple codes

8 tests, 0 failures, 4 skipped

real    0m0.691s
user    0m0.329s
sys     0m0.343s

Between this and @go.test_printf, could make iterating over specific test cases much easier, with minimal temporary hand-hacks to isolate specific conditions and inspect program state.

After thinking more about return_from_bats_assertion per #48 and #50, I'm wondering if calling set -o functrace can be called in the unset 'BATS_{CURRENT,PREVIOUS}_STACK_TRACE conditions (at least the CURRENT condition), obviating the need for an assertion function to call set +o functrace after every other assertion it calls.

Now that the plugin protocol is complete per the merging of #136 and closing of #120, there should be a new ./go plugins install subcommand to automatically and recursively install plugins similarly to npm install. It will likely read from .config/go-script-bash/plugins per #134.

Per @JohnOmernik, it may prove convenient to print log messages both the console and a file.

A proof-of-concept of the approach is already implemented in the file module as @go.fds_printf. That function won't be directly applicable; the console will need to include format codes, and the file output should have them stripped by default. Also per #35, it may be desirable to convert the output to JSON for some output files.

As it's often helpful to inspect the internal state of various variables while running Bats tests, it'd be helpful to have a function like the following:

@go.test_printf() {
  if [[ -n "$TEST_DEBUG" ]]; then
    @go.printf "$@" >&2
  fi
}

Then the command line may be prefixed with TEST_DEBUG='true', or inside individual test cases you can specify TEST_DEBUG='true' run ....

Theoretically @go.log DEBUG may cover this, but I'm expecting usage of @go.test_printf to entail temporary injections that shouldn't get checked into the master branch of a project.

This is almost a theoretical concern, as @go.log RUN emits messages to standard output by default. Still, it may prove wise to ensure that even when the user has replaced the default @go.log RUN file descriptor with something other than standard output or error, that commands running under @go.log_command always print RUN messages to standard output. Otherwise they would not, as [email protected]_command_should_skip_file_descriptor would return true for all RUN file descriptors.

In fact, this should be nearly trivial by adding this to @go.log just before the for loop:

  if [[ "$__GO_LOG_COMMAND_DEPTH" != '0' && "$log_level" == 'RUN' ]]; then
    __go_log_level_file_descriptors=('1')
  fi

If you use tab completion, your current directory is changed to the rootdir for the go script.

For example:

cd /tmp
git clone [email protected]:mbland/go-script-bash.git
cd go-script-bash
eval "$(./go env -)"
cd /tmp
go he<tab>
pwd

The result is that the current working directory is /tmp/go-script-bash.

I would expect that running the go script would not change my current working directory.

There are tests for the directory being changed when completion is in use, so I imagine there is a good reason for it. Will you please explain?

Thinking about #35 and researching options, I'm thinking it might not be too difficult to write a pure Bash JSON parser/emitter module. It would probably borrow much of the interface from JSON.sh, but avoid using grep, pipes, etc. (Unlike JSON.sh, it'll be Bash 3.2+ specific.)

Once the module is in place, emitting JSON logs (and testing the behavior of such) should prove cleaner and easier.

This would preserve the blank lines from output, which are eliminated from lines by default. The implementation would be:

split_bats_output_into_lines() {
  local line
  lines=()

  while IFS= read -r line; do
    lines+=("${line%$'\r'}")
  done <<<"$output"
}

I am liking the new changes. One thing I noticed was now when I use @go.log FATAL it produces my error message as well as a stack trace.

From a user point of view, if I am trapping an error (thus producing the @go.log FATAL) what I provide to them should be sufficient... perhaps there are other cases where I wish to provide them a stack trace, but I've found most of the place I've used @go.log FATAL my message is sufficient.

For example:

If I run a script and the script expects a certain argument -c=/path/to/conf

I try to get the conf file, and if it doesn't exist

if [ ! -f "CONF_FILE" ]; then
@go.log FATAL "Please provide a valid path to configuration file with -c="
fi

The stack trace is not needed here, and if anything confuses the user. (Think "Java Errors")

Thus, I am guessing you have a way to enable or disable stack traces globally, however, maybe we should have two "FATAL"s, one for trapped errors like mine, and another when we need more debugging?

DEBUGFATAL? (lots of characters?) DBFTL? DBFATAL? FEMFATAL? (Ok that last one is pushing it).

Am I making sense or rambling?

I am trying to log a json output from an API with @go.log

The value of the variable I am printing is

{"timestamp":1482524700778,"timeofday":"2016-12-23 08:25:00.778 GMT+0000","status":"OK","total":0,"data":[],"messages":["Successfully created volume: 'zeta.shared.dockerregv2'"]}

The error I am getting is:

/home/zetaadm/zetago/scripts/go-script-bash/lib/log: line 172: printf: `m': invalid format character

Though I'm hoping that the files in lib/bats and the existing tests are as clear as possible, it probably wouldn't hurt to develop and introductory tutorial.

Per @JohnOmernik, once #38 is decided and implemented, we may wish to create an interactive command to help produce the log module config file. This will happen after v1.3.0.

This one is very small, basically moving some of the logic in the new file module to another dedicated module.

I'd considered adding it to go-core.bash, but decided it might be best to give it its own module, since go-core.bash should be as lean as possible and not all programs or modules are likely to require the behavior. (That's why I added the module capability, and probably why modules as a concept were invented in the first place!)

This will be important for plugin compatibility. Already have a sketch implementation in-hand.

Per #131, I realized there'd been a bug from #130 whereby [email protected]_plugin_command_script assumed the _GO_PLUGINS_DIR of the command script should be the immediate parent. #130 fixed this such that _GO_PLUGINS_DIR is set to the nearest /bin parent dir, but I still need to add a few test cases for subcommand scripts to make sure this condition holds.

Per @JohnOmernik, it would be nice to provide the user with a more comprehensive configuration API. We should knock some ideas around, but as a first draft, I'm thinking of a sourced bash file (logging.conf per John's suggestion) something like:

_GO_LOG_DATEFMT='%Y-%m-%d %H:%M:%S'
_GO_LOG_OPTIONS_INFO=(
  "fmt:default /dev/stdout"
  "date:true /var/log/app/info.log"
  "/var/log/app/info.json"
)
_GO_LOG_OPTIONS_FATAL=(
  "fmt:\e[1m\e[32m /dev/stderr"
  "fmt:same /var/log/app/fatal.log"
  "/var/log/app/fatal.json"
)
_GO_LOG_OPTIONS_CUSTOM=(
  "fmt:json /var/log/app/custom.log"
)

And so forth. I'm gravitating towards Bash vs. JSON or other formats because it minimizes the need for external tooling and I have an idea of how to implement this. For example:

• The option:value parameters are first because they won't contain spaces; I can split the string, loop over the values to parse the options, then join the remaining strings back into a file path (to handle paths with spaces).
• Options like fmt:same will inherit the format of the preceding entry.
• File names ending in .json will automatically get converted to JSON format, but other file names can specify fmt:json as well.
• One timestamp format to rule them all. If it's defined, all log messages will receive them, unless date:false is specified.
• JSON format always gets a timestamp.
• Builtin levels will remain as they are unless overridden by a _GO_LOG_OPTIONS_* setting.

This is completely up for discussion. I'm open to being convinced to use things like YAML and JSON, too, but tend to favor Bash-only solutions to minimize dependencies.

The plugin protocol mimics that of npm's node_modules. This needs to be documented in the README at least, possibly in the plugins builtin, and possibly as a proper website manual.

Right now, the only way to add file descriptors to a log level is to call @go.log_add_output_file. Should add this behavior to @go.log_add_or_update_log_level as well.

It will basically be identical to the updated @go.join, with a stripped-down variable name check:

test_join() {
  if [[ ! "$2" =~ ^[[:alpha:]_][[:alnum:]_]*$ ]]; then
    printf '"%s" is not a valid variable identifier.\n" "$2" >&2
    return 1
  fi

  local IFS="${1:-$'\n'}"
  printf -v "$2" "${*:3}"
}

Note that this ensures the delimiter won't get added to the end, and then need to get trimmed off.

Also, for writing files where it's needed:

  printf '%s\n' "${lines[@]}" >path/to/file

As mentioned in #76, fork() isn't native to Windows and requires a expensive hack to emulate. While the go-script-bash framework itself avoids subshells and pipelines pretty thoroughly and itself runs fairly quickly everywhere, Bats is written in more traditional Bash that doesn't shy away from either. This makes the current suite of tests on Windows take on the order of 30min, when they take on the order of five or less on Linux and macOS systems. This is consistent across Git for Windows, Cygwin, MSYS2 (which is the basis for Git for Windows), and even the Windows Subsystem for Linux now included in Windows 10.

Also, knowing from experience that folks often won't write tests if they're slow to run—especially if it's too slow to experiment effectively and achieve the flow state necessary for deep learning—the inherent slowness of the Bats framework may turn off devs on Windows in particular. Since I'm hoping go-script-bash will prove not only useful for writing ./go scripts and other Bash apps, but also for encouraging thorough and effective Bats testing as well (with the lib/bats library helping out), the slowness may become a significant obstacle to this goal, especially for Windows users.

And there's a huge opportunity here, given the prevalence of Git for Windows and the Windows Subsystem for Linux! See also:

• Windows Subsystem for Linux installation guide
• The "WSL syscall examples" section of WSL System Calls for notes on emulating fork()
• Windows Subsystem for Linux Frequently Asked Questions
• Windows Subsystem for Linux Wikipedia page
• Fun with the Windows Subsystem for Linux

Consequently, I'm considering forking sstephenson/bats and seeing if there are any significant performance wins to be had without substantially complicating the code. On top of being a potentially huge positive factor for Windows, it will likely reap benefits for the other platforms as well.

To make it easier to compose new commands and modules from existing ones, it needs to be easier to stub out implementations. I'm thinking that introducing a more flexible lookup path mechanism for both may be the ticket.

Thinking about the upcoming JSON support and how to handle @go.log_command output, I'm thinking of emitting each line as something like:

{"d":_GO_LOG_COMMAND_DEPTH,"o":"line of output"}

It may also be useful to add the depth to the other @go.log calls as well.

When modules paths are checked in go-script-bash/lib/internal/use basically the script tries to source them and if it fails, it just ends up on unknown module

if ! . "$_GO_CORE_DIR/lib/$__go_module_name" 2>/dev/null &&
! . "$_GO_SCRIPTS_DIR/plugins/${__go_module_name////lib/}" 2>/dev/null &&
! . "$_GO_SCRIPTS_DIR/lib/$__go_module_name" 2>/dev/null; then
@go.printf "ERROR: Unknown module: $__go_module_name" >&2
exit 1
fi

However, this doesn't differentiate between the module not existing, and the module having an error. I.e. if the module file exists, but has a bash syntax error, it will fail with the same "Unknown module" error that it fails with if the module file doesn't exist.

Ideally, we'd have two messages, one for issue with the module file not existing, another stating the file doesn't exist, however there was an error in loading it.

Inspired by the need for @go.join and @go.split per #62, #81, and commit 99ab780. All these deep technical reasons should not get lost to the sands of commit log history, or excessively clutter the code comments in multiple places, if there's an effective practice focused on maintenance of a centralized artifact.

The first implementation of demo-core (#57) will provide a list of available subcommands whenever it's actually executed, rather than one of its subcommands. When the argument list is empty, it's not an error; when it isn't empty, a subcommand wasn't found, hence it is an error.

The next iteration will include a select statement-based interface that will prompt the user to select a subcommand whenever the standard input (file descriptor zero) is a terminal. Since this behavior seems like it could be generally useful, I'll implement it as a module function.

The following causes @go.log to emit mbland/go-script-bash/lib/log: line 194: printf: Y': invalid format characterand stop printing variables after_GO_LOG_TIMESTAMP_FORMAT`:

_GO_LOG_TIMESTAMP_FORMAT='%Y-%m-%d %H:%M:%S'
. "$_GO_USE_MODULES" 'log'
@go.log INFO "$(@go vars)"

The following works as expected:

_GO_LOG_TIMESTAMP_FORMAT='%Y-%m-%d %H:%M:%S'
. "$_GO_USE_MODULES" 'log'
vars="$(@go vars)"
@go.log INFO "${vars//%/%%}"

However, @go.log should escape % characters on behalf of the caller so that the first case works as expected.

Some of the functions used to test the framework itself may be useful to others writing programs based on the framework. lib/bats already contains reusable modules that depend on Bats, but not go-script-bash; I'm thinking maybe of moving tests/environment.bash to lib/test-environment.bash for these helpers.

2017 happened some time ago.

Thinking of patterning a .config/go-script-bash directory convention along the lines of the XDG_CONFIG_HOME spec. This way there could be a mechanism to import .config/go-script-bash/* to get the top-level config for the project (some possibly standard, but can pull in user-defined config like this, too).

This could also support a recursive mechanism to import and install plugins by following every .config/go-script-bash/plugins file in the _GO_PLUGINS_DIR tree. Only plugins defined in the top-level .config/go-script-bash/plugins file would then be available to the top-level command script, to support an npm-like node_modules plugin directory structure where by common plugins are shared, but without cluttering up the main command namespace.

More generally, there could be a mechanism to determine which commands to expose as part of the public vs. private interface, but that's for a future issue.

As I'm starting to focus on emitting JSON from the log module, I'm starting to realize I need more flexible assertions to examine file output, particularly if those files contain timestamps.

Specifically, in the log/timestamp test cases I figured out that setting _GO_LOG_TIMESTAMP_FORMAT='%M:%S' and validating output against [0-5][0-9]:[0-5][0-9] should be enough to validate timestamp behavior without breaking at any particular time or any particular locale. With the new logging tests, I potentially need to validate that and other patterns against multiple lines in a file, and spelling each one out manually with assert_line_matches may prove tedious.

Also, I realized some of the behavior in assert_log_file_equals from tests/log/helpers.bash is already ripe for extraction.

Hence I've bitten the bullet and begun to build out these assertions, and even added a fail_if assertion negator in the process. I'll likely address both #48 and #51 in the upcoming pull request that adds these new assertions.

After working a bit with traps as part of @go.log_command and in tests/assertions, it occurs to me it might be helpful to produce a trap module that helps command scripts set traps that invoke any previously defined traps.

For example, any command scripts written in Bash executed via @go.log_command are sourced into the [email protected]_command_invoke environment. If that command script needs to set an EXIT trap, it also needs to take care to invoke the EXIT trap defined by [email protected]_command_invoke as well.

I'm thinking something along the lines of:

@go.parse_existing_trap_command {
  local signal="$1"
  local trap_command="$(trap -p $signal)"
  trap_command="${trap_command#*\'}"
  __go_existing_trap="${trap_command%\'*}"
}

@go.add_trap_command() {
  local commands="$1"
  local signal="$2"
  local __go_existing_trap
  @go.parse_existing_trap_command "$signal"
  trap "$commands; $__go_existing_trap" "$signal"
}

@go.remove_trap_command() {
  local command_to_remove="$1"
  local signal="$2"
  @go.replace_trap_command "$command_to_remove" '' "$signal"
}

@go.replace_trap_command() {
  local existing_command="$1; "
  local new_command="$2; "
  local signal="$3"

  local __go_existing_trap
  @go.parse_existing_trap "$signal"

  if [[ ! "$__go_existing_trap" =~ $existing_command ]]; then
    @go.printf 'Existing trap commands for %s not found: %s\n' \
      "$signal" "${existing_command%; }" >&2
    @go.print_stack_trace >&2
    exit 1
  fi
  trap "${__go_existing_trap/$existing_command/$new_command}" "$signal"
}

Alternatively, it may be desirable to update [email protected]_command_script to invoke the command using a new $BASH process if __GO_LOG_COMMAND_DEPTH -ne '0'. Or perhaps both?

Either way, the module may be generally useful all the same.

Per @JohnOmernik, this will enable more robust log processing by powerful tools.

I'm pretty this option should be at least optional per log level. I'm also considering making it optional per output file, so that a regular log message can be written to the console or a file, and a JSON representation of the same message can get sent to another file.

Might be worth adding jq to the process:

• http://blog.librato.com/posts/jq-json
• https://stackoverflow.com/questions/38860529/create-json-using-jq-from-pipe-separated-keys-and-values-in-bash

Silly me, I just realized that you can include an entire string in $'...', not just the \n or other control characters.

After integrating #44, some readonly variables that may be initialized with user-defined values must be set before calling . "$_GO_USE_MODULES". There needs to be better documentation of this pattern and of the setup of specific modules (e.g. log).

Got the idea to start writing a demo program for @go.log, which got me thinking how to generalize it. Already most of the way through the implementation.

It just occurred to me that some eval statements that assign to variables can be replaced with printf -v, and that some generic library functions (in lib/*) can return their results by a similar mechanism rather than relying upon required variable declarations. Should be relatively quick and painless (in fact, I've already eliminated eval instances locally), but want to track it nonetheless.

The helper functions in tests/assertions may be useful enough to extract into a new lib/bats/assertion-test-helpers module, to make it easier to write robust new Bats assertions based on lib/bats/assertions. This should follow the completion of #48.

For consistency's sake, since command -v output is usually discarded anyway.

Should be relatively straightforward to add by replacing the default TIMEFORMAT from:

$'\nreal\t%3lR\nuser\t%3lU\nsys%3lS'

with something like this in @go.log_command_invoke:

TIMEFORMAT='real %3lR user %3lU sys %3lS'

to produce output something like this:

$time { sleep 1; echo 'Hello, World!'; }
Hello, World!
real 0m1.005s user 0m0.001s sys 0m0.002s

Then the timing info can be parsed in a similar fashion to the exit status.

It may be worth looking throughout the code for places it would make sense to add a @go.print_stack_trace call, to make it easier for the user to diagnose issues detected by the framework.

Also, a lot of the calls now are in a context of the form:

if [[ "$BAD_STUFF_HAPPENED" ]]; then
  echo "Some helpful context" >&2
  @go.print_stack_trace 1 >&2
  exit 1
fi

I'd like to update the interface with more optional parameters, maybe even keywords. For example, the following would produce the same effect:

if [[ "$BAD_STUFF_HAPPENED" ]]; then
  @go.print_stack_trace "Some helpful context" exit:1 >&2
fi

There would also be an optional skip: parameter, replacing the current first positional parameter and defaulting to skip:1 (to place the caller of the function calling @go.print_stack_trace at the top of the stack). This would be a breaking API change, but again, since the current adoption of this function is so contained, I'm looking for only a minor version bump.

More ideas welcome!

Per the instructions in lib/bats/assertions, it's imperative that every Bats assertion function using that module call return_from_bats_assertion immediately before returning. However, while the need to call return_from_bats_assertion in the failure case is documented thoroughly in the comments for that function, the need to call it in the success case isn't, and the ramifications of not calling it in the success case aren't tested, either.

Specifically, if an assertion doesn't call return_from_bats_assertion in the success case, the assertion's entry isn't cleaned from the Bats stack traces and set -o functrace doesn't get called. When that assertion is followed by another assertion that fails, the failure output may contain a reference to the earlier, passing assertion. When assertions are composed from existing assertions that also fail to call return_from_bats_assertion in the success case, the problem compounds.

Right now, assert_lines_equal is missing this success-case call to return_from_bats_assertion. It's the only assertion in the file missing the call, but an additional two checks need to be added to the expect_success test helper to ensure all assertions satisfy this condition.

Now that I'm trying to write a separate plugin, I'm hitting some rough edges. Basically, I'd like plugins to potentially be standalone programs, but that collides with some presumptions the core framework makes regarding command and module lookup paths. The solution here may overlap with #118.

Per @JohnOmernik, there should be the option to precede log messages with strftime-style timestamps.

Starting with some 4.x version of Bash, the %(datefmt)T format specifier provides exactly this behavior; Bash 3.2, as ships with macOS, does not. So there'd need to be logic to use the builtin format if available, resort to the date program if not, and produce a warning if neither happens to be available.