Friday, August 3, 2012

Better Bash Scripting in 15 Minutes

The tips and tricks below originally appeared as one of Google's "Testing on the Toilet" (TOTT) episodes. 
This is a revised and augmented version.

Safer Scripting

I start every bash script with the following prolog:
set -o nounset
set -o errexit
This will take care of two very common errors:
  1. Referencing undefined variables (which default to "") 
  2. Ignoring failing commands
The two settings also have shorthands (“-u” and “-e”) but the longer versions are more readable.

If a failing command is to be tolerated use this idiom:
if ! <possible failing command> ; then
    echo "failure ignored"
Note that some Linux commands have options which as a side-effect suppress some failures, e.g.
mkdir -p” and “rm -f”.

Also note, that the “errexit” mode, while a valuable first line of defense, does not catch all failures, i.e. under certain circumstances failing commands will go undetected.
(For more info, have a look at this thread.)

A reader suggested the additional use of "set -o pipefail"


Bash lets you define functions which behave like other commands -- use them liberally; it will give your bash scripts a much needed boost in readability:
ExtractBashComments() {
    egrep "^#"
cat | ExtractBashComments | wc 
comments=$(ExtractBashComments <

Some more instructive examples:
SumLines() {  # iterating over stdin - similar to awk      
    local sum=0
    local line=””
    while read line ; do
        sum=$((${sum} + ${line}))
    echo ${sum}
SumLines < data_one_number_per_line.txt 
log() {  # classic logger
   local prefix="[$(date +%Y/%m/%d\ %H:%M:%S)]: "
   echo "${prefix} $@" >&2
log "INFO" "a message"

Try moving all bash code into functions leaving only global variable/constant definitions and a call to “main” at the top-level.

Variable Annotations 

Bash allows for a limited form of variable annotations. The most important ones are:
  • local (for local variables inside a function)
  • readonly (for read-only variables)
# a useful idiom: DEFAULT_VAL can be overwritten
#       with an environment variable of the same name
myfunc() {
   # initialize a local variable with the global default
   local some_var=${DEFAULT_VAL}
Note that it is possible to make a variable read-only that wasn't before:
readonly x
x=7   # failure

Strive to annotate almost all variables in a bash script with either local or readonly.

Favor  $() over backticks (`)

Backticks are hard to read and in some fonts easily confused with single quotes.
$()also permits nesting without the quoting headaches.

# both commands below print out: A-B-C-D
echo "A-`echo B-\`echo C-\\\`echo D\\\`\``"
echo "A-$(echo B-$(echo C-$(echo D)))"

Favor [[]] (double brackets) over [] 

[[]] avoids problems like unexpected pathname expansion, offers some syntactical improvements,
and adds new functionality:

Operator        Meaning
||             logical or (double brackets only)
&&           logical and (double brackets only)
<            string comparison (no escaping necessary within double brackets)
-lt          numerical comparison
=             string matching with globbing
==         string matching with globbing (double brackets only, see below)
=~            string matching with regular expressions (double brackets only , see below)
-n            string is non-empty        
-z            string is empty
-eq           numerical equality

-ne           numerical inequality

single bracket
[ "${name}" \> "a" -o ${name} \< "m" ]

double brackets
 [[ "${name}" > "a" && "${name}" < "m"  ]]

Regular Expressions/Globbing

These new capabilities within double brackets are best illustrated via examples:
[[ "$t" == abc* ]]         # true (globbing)
[[ "$t" == "abc*" ]]       # false (literal matching)
[[ "$t" =~ [abc]+[123]+ ]] # true (regular expression)
[[ "$t" =~ "abc*" ]]       # false (literal matching)
Note, that starting with bash version 3.2 the regular or globbing expression
must not be quoted. If your expression contains whitespace you can store it in a variable:
r="a b+"
[[ "a bbb" =~ $r ]]        # true
Globbing based string matching  is also available via the case statement:
case $t in
abc*)  <action> ;;

String Manipulation

Bash has a number of (underappreciated) ways to manipulate strings.

len="${#f}" # = 20 (string length) 
# slicing: ${<var>:<start>} or ${<var>:<start>:<length>}
slice1="${f:6}" # = "path2/file.ext"
slice2="${f:6:5}" # = "path2"
slice3="${f: -8}" # = "file.ext"(Note: space before "-")
slice4="${f:${pos}:${len}}" # = "path2"

Substitution (with globbing)
single_subst="${f/path?/x}"   # = "x/path2/file.ext"
global_subst="${f//path?/x}"  # = "x/x/file.ext"
# string splitting
readonly DIR_SEP="/"
array=(${f//${DIR_SEP}/ })
second_dir="${array[1]}"     # = path2

Deletion at beginning/end (with globbing)
# deletion at string beginning extension="${f#*.}"  # = "ext" 
# greedy deletion at string beginning
filename="${f##*/}"  # = "file.ext"
# deletion at string end
dirname="${f%/*}"    # = "path1/path2"
# greedy deletion at end
root="${f%%/*}"      # = "path1"

Avoiding Temporary Files

Some commands expect filenames as parameters  so straightforward pipelining does not work.
This is where <() operator comes in handy as it takes a command and transforms it into something
which can be used as a filename:

# download and diff two webpages
diff <(wget -O - url1) <(wget -O - url2)
Also useful are "here documents" which allow arbitrary multi-line string to be passed
in on stdin. The two occurrences  of 'MARKER' brackets the document.
'MARKER' can be any text.

# DELIMITER is an arbitrary string
command  << MARKER

If parameter substitution is undesirable simply put quotes around the first occurrence of MARKER:
command << 'MARKER'
no substitution is happening here.
$ (dollar sign) is passed through verbatim.

Built-In Variables

For reference

$0   name of the script
$n   positional parameters to script/function
$$  PID of the script
$! PID of the last command executed (and run in the background)
$? exit status of the last command  (${PIPESTATUS} for pipelined commands)
$# number of parameters to script/function
$@  all parameters to script/function (sees arguments as separate word)
$*    all parameters to script/function (sees arguments as single word)


$*   is rarely the right choice.
$@ handles empty parameter list and white-space within parameters correctly
$@ should usually be quoted like so "$@"


To perform a syntax check/dry run of your bash script run:

bash -n

To produce a trace of every command executed run:

bash -v

To produce a trace of the expanded command use:

bash -x

-v and -x can also be made permanent by adding
set -o verbose and set -o xtrace to the script prolog.
This might be useful if the script is run on a remote machine, e.g.
a build-bot and you are logging the output for remote inspection.

Signs you should not be using a bash script

  • your script is longer than a few hundred lines of code
  • you need data structures beyond simple arrays
  • you have a hard time working around quoting issues
  • you do a lot of string manipulation
  • you do not have much need for invoking other programs or pipe-lining them
  • you worry about performance
Instead consider scripting languages like Python or Ruby.


Thanks to Peter Brinkmann and Kim Hazelwood for their feedback on drafts of this post.