kvz / bash3boilerplate Goto Github PK
View Code? Open in Web Editor NEWTemplates to write better Bash scripts
Home Page: http://bash3boilerplate.sh
License: MIT License
Templates to write better Bash scripts
Home Page: http://bash3boilerplate.sh
License: MIT License
https://github.com/sstephenson/bats
We should consider recommending this in best practices, as well as using it for our own tests
I did not realize until after some trial and error that the best place to insert the code within the script is within the function that you want to export. I was having issues with the exporting the function. I think a recommendation within the read me on where to insert the boiler plate code would help others would reduce quick start friction:
#!/usr/bin/env bash
function foo(){
// Boilerplate code
}
#Recommended export code already outlined within the Function Packaging section of the read me
if [ "${BASH_SOURCE[0]}" != "${0}" ]; then
export -f foo
else
foo "${@}"
exit $?
fi
See, for example, https://travis-ci.org/kvz/bash3boilerplate/jobs/110544645
I see examples for setting defaults for arguments with options, what about arguments without options?
I'd like to add arguments for some features that have the same starting letter:
--walltime-template
--walltime-subject
Right now, short options are limited to a single letter, so it's not clear how I can expose these.
Possible implementation here:
https://coderwall.com/p/wil8da/debug-bash-scripts-with-line-numbers-more-info
e.g.
IFS=$'\n'
for line in $(cat many-lines.txt); do
echo "${line}"
done
unset IFS
This is a follow-up to Issue #35
We use lots of bashisms, for good reason.
Perhaps we should check and bail out if a non-bash shell is running the script?
One way to check: https://askubuntu.com/questions/97522/how-can-my-script-determine-whether-its-being-run-by-bash-or-dash
We can trap CTRL-C and call a function in bash:
https://rimuhosting.com/knowledgebase/linux/misc/trapping-ctrl-c-in-bash
# trap ctrl-c and call ctrl_c()
trap ctrl_c INT
function ctrl_c() {
echo "** Trapped CTRL-C"
}
I don't have a use case right now, but it's definitely helpful to allow graceful cleanup of an interrupted script.
No sure if this behaviour is intended. I'm using bash4. This is what I had to do to work around it: QwertyZW@0cdc704
qwertyzw% ./example.sh
(18-06-25 6:20:41) <1> [~/bash3boilerplate]
qwertyzw% PS4='Line ${LINENO}: ' bash -x ./example.sh
Line 31: read -r -d '' __usage
Line 39: true
Line 42: read -r -d '' __helptext
Line 46: true
LLLine 49: dirname ./example.sh
LLine 49: cd .
LLine 49: pwd
Line 49: source /home/alkhishm/bash3boilerplate/main.sh
LLine 19: set -o errexit
LLine 21: set -o errtrace
LLine 23: set -o nounset
LLine 25: set -o pipefail
LLine 29: [[ /home/alkhishm/bash3boilerplate/main.sh != \.\/\e\x\a\m\p\l\e\.\s\h ]]
LLine 30: __i_am_main_script=0
LLine 32: [[ -n x ]]
LLine 33: [[ ./example.sh = \.\/\e\x\a\m\p\l\e\.\s\h ]]
LLine 34: __i_am_main_script=1
LLine 37: __b3bp_external_usage=true
LLine 38: __b3bp_tmp_source_idx=1
LLLLine 47: dirname ./example.sh
LLLine 47: cd .
LLLine 47: pwd
LLine 47: __dir=/home/alkhishm/bash3boilerplate
LLLine 48: basename ./example.sh
LLine 48: __file=/home/alkhishm/bash3boilerplate/example.sh
LLLine 49: basename /home/alkhishm/bash3boilerplate/example.sh .sh
LLine 49: __base=example
LLLine 50: printf %q /home/alkhishm/bash3boilerplate/example.sh
LLLine 50: (( 0 ))
LLine 50: __invocation=/home/alkhishm/bash3boilerplate/example.sh
(18-06-25 6:20:44) <1> [~/bash3boilerplate]
qwertyzw% git checkout master
Previous HEAD position was 327cba7... Add magic variable that stores the full command line invocation (#99)
Switched to branch 'master'
Your branch is up-to-date with 'origin/master'.
(18-06-25 6:20:48) <0> [~/bash3boilerplate]
qwertyzw% ./example.sh
Option -f (--file) requires an argument
-f --file [arg] Filename to process. Required.
-t --temp [arg] Location of tempfile. Default="/tmp/bar"
-v Enable verbose mode, print script as it is executed
-d --debug Enables debug mode
-h --help This page
-n --no-color Disable color output
-1 --one Do just one thing
This is Bash3 Boilerplate's help text. Feel free to add any description of your
program or elaborate more on command-line arguments. This section is not
parsed and will be added as-is to the help.
Example:
read -r -d '' __usage <<-'EOF' || true # exits non-zero when EOF encountered
-s --subject [arg] Specific subject files to process.
-v --verbose Enable verbose mode for all scripts.
-d --debug Enables debug mode.
-h --help This help page.
-n --dry-run Don't submit any jobs.
-r --reg-command [arg] Provide an alternative registration command. Default="mb_register.sh"
-f --factor [arg] Scaling factor for time and memory estimates. Default="1.15"
EOF
read -r -d '' __helptext <<-'EOF' || true # exits non-zero when EOF encountered
MAGeTBrain implementation using ANTs
Supports MINC and NIFTI input files (ANTs must be built with MINC support)
Invocation: mb.sh [options] -- [stage 1] [stage 2] ... [stage N]
Standard stages: template, subject, resample, vote, run (template, subject, resample, vote)
Multiatlas stages: multiatlas-resample, multiatlas-vote, multiatlas (template, multiatlas-resample, multiatlas-vote)
Other stages: init, status, cleanup
Multiple commands will run multiple stages. Order is not checked.
EOF
User tried to do:
$ mb.sh --template
#Instead of
$ mb.sh -- template
There was no error thrown for the invalid command-line option "--template"
I noticed that the logging colors are defined in the __b3bp_log function and that they are set every time the function is called. Highly unlikely it have a noticeable impact on performance, but debugging stuff is terrible with the overhead. Is it necessary to redeclare them on each call? Of course I can move them outside myself, but if there is a specific reasoning for this structure I would like to hear it.
As Vaphell suggests:
on top of what parent said the [[ ]] / (( )) dichotomy is really useful because it allows for 2 visually distinct modes of work which reduces the cognitive cost:
text [[ ]] with patterns regexen and shit
ingeger math (( )) using C syntax lookalike for int math and comparisons.
(( c++ ))
(( x = y + 3 )) # equivalent to x=$(( y + 3 ))
if (( RANDOM%17 == 4 ))
text is assumed to be a variable, so $ can be dropped for additional readability ($ is required only if the > expression is supposed to make use of positional params to disambiguate 1 and $1
[ ] is everything crammed together, using less than stellar switch based syntax.
Perhaps we have use cases already where we want to apply this, if not a deciding on one form and adding that to the README/website would be nice
The __os variable is set incorrectly on Windows (using git bash). Recommend adding this to the boilerplate.
if [[ "${OSTYPE:-}" == "msys"* ]]; then
__os="Windows"
fi
A nice optional feature would be the ability to log debug/info/notice/warn/error/critical/alert to file.
if [[ "${BASH_SOURCE[0]}" != "${0}" ]] ; then
__being_sourced=true
else
__being_sourced=false
fi
$__being_sourced && set -o errexit # exit on any errors, so long as we're not sourcing the script, since that would quit
See 9350e05#commitcomment-30019068
This change breaks the following call:
script.sh --long value --switch
The value of arg_l will be "value--switch" instead of "value", and arg_s will not be defined.Cause is in line 242ff, where OPTIND is advanced by the number in has_arg, which before was either 0 or 1, but now 2 is possible and has a special meaning but also affects this operation: ${@:OPTIND:${!__b3bp_tmp_varname}}
Following calls are unaffected, which is why the tests don't break (insufficient test cases ;)) and may explain why this hasn't been reported yet:
script.sh --switch --long value (only last option has value)
script.sh --long=value --switch (using "long-option=value" syntax)
script.sh -l value --switch (single-character options are unaffected for some reason, though I'm not sure why...)My personal solution is to use a separate (bool) variable instead of extending the has_arg by another value: _b3bp_tmp_requires_arg${__b3bp_tmp_opt:0:1}
Folks,
It looks like the latest version of shellcheck complains about a few things in main.sh. Maybe SC#### numbers have changed for some of these things that might need to be skipped?
`
$ shellcheck main.sh
In main.sh line 65:
local color_debug="\x1b[35m"
^-- SC1117: Backslash is literal in "\x". Prefer explicit escaping: "\x".
In main.sh line 67:
local color_info="\x1b[32m"
^-- SC1117: Backslash is literal in "\x". Prefer explicit escaping: "\x".
In main.sh line 69:
local color_notice="\x1b[34m"
^-- SC1117: Backslash is literal in "\x". Prefer explicit escaping: "\x".
In main.sh line 71:
local color_warning="\x1b[33m"
^-- SC1117: Backslash is literal in "\x". Prefer explicit escaping: "\x".
In main.sh line 73:
local color_error="\x1b[31m"
^-- SC1117: Backslash is literal in "\x". Prefer explicit escaping: "\x".
In main.sh line 75:
local color_critical="\x1b[1;31m"
^-- SC1117: Backslash is literal in "\x". Prefer explicit escaping: "\x".
In main.sh line 77:
local color_alert="\x1b[1;33;41m"
^-- SC1117: Backslash is literal in "\x". Prefer explicit escaping: "\x".
In main.sh line 79:
local color_emergency="\x1b[1;4;5;33;41m"
^-- SC1117: Backslash is literal in "\x". Prefer explicit escaping: "\x".
In main.sh line 84:
local color_reset="\x1b[0m"
^-- SC1117: Backslash is literal in "\x". Prefer explicit escaping: "\x".
In main.sh line 86:
if [[ "${NO_COLOR:-}" = "true" ]] || ( [[ "${TERM:-}" != "xterm"* ]] && [[ "${TERM:-}" != "screen"* ]] ) || [[ ! -t 2 ]]; then
^-- SC2235: Use { ..; } instead of (..) to avoid subshell overhead.
In main.sh line 252:
((OPTIND+=_b3bp_tmp_has_arg${__b3bp_tmp_opt}))
^-- SC1105: Shells disambiguate (( differently or not at all. For subshell, add spaces around ( . For ((, fix parsing errors.
^-- SC2030: Modification of OPTIND is local (to subshell caused by (..) group).
In main.sh line 269:
shift $((OPTIND-1))
^-- SC2031: OPTIND was modified in a subshell. That change might be lost.
In main.sh line 383:
info "$(echo -e "multiple lines example - line #1\nmultiple lines example - line #2\nimagine logging the output of 'ls -al /path/'")"
^-- SC1117: Backslash is literal in "\n". Prefer explicit escaping: "\n".
^-- SC1117: Backslash is literal in "\n". Prefer explicit escaping: "\n".
`
$ LOG_LEVEL=7 /bin/bash -o pipefail -o nounset -o errexit ./main.sh -f -t
2016-02-16 16:27:12 UTC [ debug] cli arg arg_f = () -> -t
2016-02-16 16:27:12 UTC [ info] You are on OSX
2016-02-16 16:27:12 UTC [ debug] Info useful to developers for debugging the application, not useful during operations.
2016-02-16 16:27:12 UTC [ info] Normal operational messages - may be harvested for reporting, measuring throughput, etc. - no action required.
2016-02-16 16:27:12 UTC [ notice] Events that are unusual but not error conditions - might be summarized in an email to developers or admins to spot potential problems - no immediate action required.
2016-02-16 16:27:12 UTC [ warning] Warning messages, not an error, but indication that an error will occur if action is not taken, e.g. file system 85% full - each item must be resolved within a given time. This is a debug message
2016-02-16 16:27:12 UTC [ error] Non-urgent failures, these should be relayed to developers or admins; each item must be resolved within a given time.
2016-02-16 16:27:12 UTC [ critical] Should be corrected immediately, but indicates failure in a primary system, an example is a loss of a backup ISP connection.
2016-02-16 16:27:12 UTC [ alert] Should be corrected immediately, therefore notify staff who can fix the problem. An example would be the loss of a primary ISP connection.
2016-02-16 16:27:12 UTC [emergency] A "panic" condition usually affecting multiple apps/servers/sites. At this level it would usually notify all tech staff on call.
2016-02-16 16:27:12 UTC [ info] Cleaning up. Done
It would be cool to have basic tests ran on Travis CI.
Bonus points for running against multiple bash versions and GNU/BSD tools. Unsure yet how to do that tho.
I'm testing your script and it is really useful. However, I found one problem, not sure if I am using it wrong or it's actually a bug.
When you use a tilde (~
) in paths, it seems that is converted to something else (although when you print it, it looks OK).
For example, take your main and place a file in home (~
) and try to access it and all the commands will fail.
[ -f ${arg_t} ] && echo "temp ${arg_t} exist" || echo "temp ${arg_t} does not exist"
stat ${arg_t}
And I set the read
block as
-t --temp [arg] Location of tempfile. Default="~/foo"
But if you set it to something else, without the tilde it works.
Line 78 in 9b739b4
The line
if [ "${NO_COLOR}" = "true" ] || [[ "${TERM:-}" != "xterm"* ]] || [ -t 1 ]; then
seems to be wrong really in at least 2 different ways.
I'm guessing here but a quick check seems to suggest that nobody noticed until now because __b3bp_log
always got called in a subshell which actually was not connected to a terminal. If the function would be just called - as suggested by shellcheck - the code would not work as expected.
Now I can fix that, but before that I wanted to ask if there is something about this code, or the decisions leading to this code, which I overlooked or I don't understand.
As Browsing_From_Work mentioned on https://www.reddit.com/r/programming/comments/4parog/bash3_boilerplate_template_for_writing_better/d4ji2yh there are some good arguments for it. For convenience, the comment also points to this page.
I see a lot of pros, and only a possible con that we might have more explaining to do. But I think we can figure out that part fine. It might also save us some explaining of pitfalls with [
We are already using it in order to do pattern matching to figure out the OS family
Thoughts?
See discussion in PR #81
This may be the same bug as #5 and I think I have found a fix.... but it's possible they are distinct.
./main.sh -f foo -t
should populate arg_t
with the value parsed from the usage string default. However, opt
is set to ?
, the help
function is called and getopts
aborts.
As detailed on this SO post you can actually put getopts
into silent mode an explicitly test for missing option arguments. Here is some more info from bash hackers wiki: http://wiki.bash-hackers.org/howto/getopts_tutorial#error_reporting
This technique will allow b3bp to not barf when optional option arguments are missing.
It's there in the examples but the existing color code doesn't check/use it.
It would be nice to be able to have better control over the usage statement. In particular:
It's possible that the last point is a non-issue, I need to examine the usage text parsing more carefully.
The first point will need some different usage statement parsing logic. It's possible that some sort of special line break character or string could be used, and then when the usage statement is parsed for options, consecutive lines containing the special character/string would be concatenated, but when the usage statement is printed a line break is inserted at that character. An easy candidate could be \n
which when echo
ed with echo -e
would insert a line break. Thoughts?
Not sure what to do about the second point yet...
templater.sh does its magic trying to use sed.
this works out fine as long as there is no surprise in any env var.
this cannot even really be fixed as such because really sed is the wrong tool for this job.
possibly use perl where quoting (\Q
and \E
) can be used or do as most others suggest and use eval+cat.
or at least try to replace only vars which actually get referenced in the template and live with the fact that if this variable has unexpected content stuff breaks - right now a sed command is issued for each existing env var. this is bound to break no matter what.
opinions? remarks? ideas?
I propose the text below as a new FAQ question. If you agree, please feel free to copy, paste, and edit it or let me know if you want me to branch, commit, and submit a pull request.
How do I access a potentially unset environment variable?
The set -o nounset line in main.sh causes error termination when an unset environment variables is detected as unbound. To avoid this, declare all environment variables before use, e.g.,
#!/usr/bin/env bash
# Include main.sh (NOTE: comment final line before executing this script):
. main.sh
declare FOO
# Testing FOO below would cause termination with a non-zero exit code without the above declaration
if [[ -z ${FOO} ]]; then
info "FOO is empty."
else
info "FOO has a value"
fi
Hi @kvz,
First of all I want to say, your project is awesome, and you deserve credit, recognition and attribution for it! 💯 However, I am wondering if adopting a CC0 or, IMO less desirable, the unlicense might help promote adoption. Since this is intended to be a boiler plate, it is not clear to me, exactly how and under what conditions the MIT license needs to be preserved and distributed with projects that bootstrap shell scripts with bash3boilerplate. Part of this is my admitted inexperience with software licenses, but I also feel like this is a "ship of Theseus" problem at what point does borrowing components from the script require that the MIT license be included, referenced etc.
I certainly defer to your judgement here, this issue is only intended to prompt discussion and reconsideration of licensing concerns.
We received feedback by galaktos let's put it to good use!
if [ "${BASH_SOURCE[0]}" != ${0} ]; then
Use a single equal sign when checking if [ "${NAME}" = "Kevin" ], double or triple signs are not needed in Bash
Use {} to enclose your variables in. Otherwise Bash will try to access the $ENVIRONMENT_app variable in /srv/$ENVIRONMENT_app, whereas you probably intended /srv/${ENVIRONMENT}_app.
Use :- if you want to test variables that could be undeclared. For instance: if [ "${NAME:-}" = "Kevin" ] will set
$NAME to be empty if it’s not declared instead of throwing an error. You can also set it to noname like so if [ "$ {NAME:-noname}" = "Kevin" ]
easy fix, just need an ||true
I had the idea to create a simple website for this under e.g. the bash3boiletplate.sh domain name using GitHub pages. As a first iteration it would render the README.md to HTML so it could serve as a simple single front page. But I feel it would give the project some more substance, what do you think?
I'd like to provide some info for the options after "--" in my script.
Ideally, I would do this by writing it in the usage doc, but it gets parsed.
I'd like a way for lines to be ignored by the parser.
Is it intended that only an "emergency" causes an exit?
I would expect anything above critical should immediately exit.
From the docs:
"Safe by default (break on error, pipefail, etc.)"
As mhurron mentions, it might be a good idea to adopt / advise on linting
Not sure how hard this would be to implement, but parsing the usage block to find the "Required" arguments would eliminate the extra code at line:
https://github.com/kvz/bash3boilerplate/blob/master/main.sh#L234
It would be great to have a FAQ or User Guide describing some common use cases. I'd like to know how to add a new command-line flag and have spent at least of couple of hours trying so far. Is there an example somewhere?
Hi, and thanks for your work, this is really helpful :)
I have a little question, how would you do for short and long argument name.
Let's say I have an argument filename
, and you can either use it with -f
or --filename
. It is also required.
Would you have a quick way to adapt you boiler plate to fit that?
Thanks a lot again :)
Pierre
Right now, if I want to provide multiple inputs to an argument I have to do this:
-s "arg1 arg2"
It would be nice if it instead could be:
-s arg1 -s arg2
Or possibly,
-s arg1 arg1
My sense is that bash3boilerplate has at least two distinct but related goals: (1) provide some useful code that others might copy, paste, and modify/include and (2) demonstrate best practices. Toward the latter end, it would be great to have a style guide that explains come of the choices (and any related tradeoffs) employed in bash3boilerplate codes. Alternatively, style questions could be added to the FAQ, but I think of the FAQ as being more about how to use bash3boilerplate and a Style Guide being about how a user can write their own better bash 3 code. Either way, here are a couple of topics:
P.S. With a Style Guide, you guys probably would have the beginnings of a great book. :) It probably wouldn't have to be very long (e.g., 25-50 pages) to be useful. In fact, it's probably more useful if it's short and can be devoured quickly. The content could be driven completely by going through bash3boilerplate and detailing the decisions you made at each point in its development.
Having the script automatically parse some sort of configuration file (json,xml,yaml,bash) that dictates default values for its arguements would nice.
Assume a case where the scripts are written for a multitude of runtimes (say for e.g; servers) and the default parameters are meant to be different for each runtime.
maybe we should add PR templates to remind people...
Allow debug info with line number and function name when ERR signal is raised:
The second URL listed under "More info" in main.sh appears to be broken.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.