Debugging Techniques for Bash Scripts

Debuggers are very useful tools while working with programming languages. It helps programmers find and fix bugs more efficiently by allowing them to step through the code one line at a time, examine variables and the program’s behavior as it executes, and check the program's state at various points in its execution.

This tutorial will look at the various techniques to debug Bash shell scripts. The Bash shell doesn’t provide any built-in debugger. However, some specific commands and tools can be used for this purpose.

Using shellcheck tool for static analysis

shellcheck is an open-source static analysis tool that can be used to detect errors in the script and also suggest fixes.

It is a Linux utility and is available for installation in a wide number of distributions through standard package managers.

On Redhat/Yum based distributions:

sudo yum install shellcheck

On Fedora:

sudo apt install shellcheck

On Debian-based systems:

sudo apt install shellcheck

Let’s take the following example to understand the utility of shellcheck:

cat shellcheckUtility.sh
#!/bin/bash

## 1. Conditional operators in wrong braces
num=5
denom=0
(( $num -gt $denom ))

## 2. Multi-word string without double quotes
multiWordString=Hello world
echo $multiWordString

## 3. Variable referenced in single quotes
animal='giraffe'
singleQuoteStrings='Animal name is $animal'

## 4. Variable assignment separated by a space
a = 2

## 5. Code suggestions
cat 'shellcheckUtility.sh' | grep num

We see several syntactical, logical, and coding style issues in the script.

Let’s invoke shellcheck and see what it suggests for each of these cases:

In the first example, we are using () instead of []. shellcheck detects that and additionally suggests to use [].
In the second example, we have a multi-word string. But, we are trying to echo this without double quotes(“). It suggests us to enclose this in double quotes to prevent word splitting.
In the third example, we define a variable singleQuoteStrings in single quotes referencing another variable. It detects this and suggests to use double quotes.
In the fourth example, we are defining a variable with space between =, variable and the value. This is a frequent mistake beginners do when starting with shell scripting.
In the fifth example, we use cat <file> in conjunction with grep. This is a bad coding style and cat is not required here. Instead, grep <var> <file> can be used which is suggested by shellcheck.

Using Bash options

Bash provides several options that can be used for debugging the script. These options can be set or unset within the script or/and can also be set in the command line while invoking the script. Let us explore these options in the next section.

Using noexec mode (-n)

No execution mode (-n) helps us to dry run the script and check the syntax before its execution. On executing the script in noexec mode, Bash will read the commands and report syntax errors, if any, but will not execute the script.

Let’s see the working of noexec mode with an example:

cat noExecMode.sh
#!/bin/bash

function func1() {
       echo $1
}

str="hello"
func1(str)

On executing the script with -n flag:

In this example, we were trying to call the function with a wrong syntax func1(str) instead of func1 str. Dry running the script with the -n flag detects that the statement is syntactically incorrect.

Using verbose option (-v)

While running the script in noexec mode helps us to validate and fix the syntax errors, but it does not help in identifying any run-time errors that may come up during the execution.

The verbose option (-v) can be used in such cases.

On setting this option, each command in the script is printed to the standard out (stdout) before its execution. This allows us to step through the script flow line by line.

To understand the working of verbose option, let us go through the following script:

cat divisonByZero.sh
#!/bin/bash

set -v

num=5
echo $num

denom=0
echo $denom

## Division
echo "Evaluate $num/$denom"

We enable the verbose option by using set -v command.

Now, let’s invoke the script:

On executing the script, we see each line is printed to the stdout before the execution.

Using xtrace option (-x)

Verbose option can be useful to understand the flow of control of the script by stepping through one line at a time. But, it does not show us the exact values of variables and the commands in the script.

This is where the execution trace option (-x) comes in.

It can help us to debug logical errors in the code by printing the expanded result of commands on each line to the stdout before they are executed.

Let’s see what happens when we invoke the same script using -x option instead of -v.

We see several lines starting with + in the stdout. These are the debug outputs generated by the xtrace mode.
All the debug outputs now show the variable substituted values of the commands.
Suppose, in this example, denom was derived by another series of calculations instead of being statically assigned. In that case, running the script in xtrace mode would help to easily uncover logical issues leading to the undefined division by zero scenario.

Using unset option (-u)

Often, when working with scripts, we can mistakenly reference an undefined variable or make a typo in the variable name.

Bash doesn’t identify this as an error and the script executes as expected.

Let’s see what happens when we have an undefined variable in the script:

cat unsetOpt.sh
#!/bin/bash

firstname="harry"
echo "Fullname is $firstname $lastname"

On executing the script:

Although lastname wasn’t defined, the script executed successfully but the result is logically incorrect.

Let’s add set -u in the script and invoke it again:

This time it identifies that we are trying to reference an undefined variable, throws out an error, and immediately terminates the script.

Debugging a subset of the script

In the previous examples, we were setting the debug options within the script, and the option would be active for all the lines following it in the script.

But, in real use cases, we need to enable debugging only for specific portions of the script to reduce the trace information printed to stdout.

This can be done by enabling the debug option before the code block and disabling it afterward using the set +<option> command.

Let’s illustrate this with an example:

cat debugSubset.sh
#!/bin/bash

function doDivide() {
       set -x
       let "result = $1/$2"
       echo "Result is $result"
       set +x
}

num=6
denom=2

doDivide $num $denom

On invoking the script:

We see that the debug outputs are only generated for the commands in doDivide function. This way we can focus on debugging complex parts of the program while generating minimum noise to the stdout.

Enabling Options at Runtime

In the previous sections, we saw how to enable the debug options within the script.

Bash also allows to set these options from the command line while invoking the script.

Let’s see how we can execute divisionByZero script at runtime:

From the stdout, we see that the debug statements are the same as using the option within the script but this way, we have a flexibility to run the scripts on debug mode only when needed.

Using the trap command

The trap command allows us to specify a command or a set of commands to be executed repeatedly before each statement in the script.

Let’s take the following example:

cat trapCmd.sh
1 #!/bin/bash
2
3 trap 'echo "[DEBUG ${LINENO}] num: $num, denom: $denom, division result: $result"' DEBUG
4 
5 num=6
6 denom=2
7
8 ## Perform division
9 result=$((num/denom))
10 echo "Result is $result"

Here, we define an echo statement with the DEBUG signal and pass it to the trap command to print the values of num, denom, and the result variables to the stdout.

Let’s see what happens on executing the script:

Till line 5, all the variables are undefined, and hence it outputs empty values.
On line 5, num gets defined - trap command generates [DEBUG 6] num: 6, denom: , division result: before executing line 6
Similarly, on line 6 and line 9, denom and result get defined thus displaying the values in the respective debug outputs.

Debugging running scripts using strace

strace is a useful command-line tool that can be used to trace the system calls between a program and the kernel. It allows you to see what system calls are being made by a process, the arguments passed to those calls, as well as the time spent in each of those calls.

Let’s illustrate the working of strace with an example:

cat straceCmd.sh
#!/bin/bash

num=5
denom=0

## Trace syscalls
while :; do
       sleep 5 &
       echo "Evaluate $num/$denom"
done

Now, let’s invoke the script:

Looking at the ps --ef --forest output, we see several processes forked out from the main process.

Now, let’s attach to the main process using strace -p <pid> (pid = 2312). Using -f in the command enables strace to attach to all the forked processes and using -c flag prints the summary of all the sys calls at the end.

Looking at the strace output, we see several Process <pid> attached outputs. This is due to the sleep command which creates forks off the main process.

Additionally, we can also see the statistics and time spent in each sys call at the end.

Conclusion

Although Bash does not have a built-in debugger, there are a wide number of shell options, commands, and Liinux utilities to help in debugging.
shellcheck is an open-source tool that can be used for syntax checking and linting of shell scripts.
Bash noexec mode (-n) can be used to quickly check the script for any syntax errors.
If we need to dive deeper and debug issues at runtime, options such as verbose(-v), xtrace(-x), unset(-u) can be used.
Bash allows us to debug a subset of the script by using the set +<option> command to disable the option.
Bash also allows setting these options at the script invocation time. This is great when we don’t want to run the script in debug mode at all times.
trap command is another bash built-in that can be used to repeatedly print debug outputs to the stdout.
If we need to attach to a running script and trace the system calls made by the script or understand the behavior between the script and the kernel, strace utility can be used.

About The Author

Madhur Batra

Madhur Batra is a highly skilled software engineer with 8 years of personal programming experience and 4 years of professional industry experience. He holds a B.E. degree in Information Technology from NSUT, Delhi. Madhur’s expertise lies in C/C++, Python, Golang, Shell scripting, Azure, systems programming, and computer networking. With a strong background in these areas, he is well-equipped to tackle complex software development projects and contribute effectively to any team.