When it comes to text processing or filtering logs, it becomes important to read the contents line by line. Using the bash while loop you can read the contents one line at a time and use it as per our needs. We can read from a file, a command and from a variable. We can even read lines into multiple variables.
Prerequisites
- A text editor
- Access to a terminal
Syntax - bash while read line
We can generalize how to read line by line using the while loop as follows:
while IFS='some field separator' read -r var1 var2 var3 …
do
….
….
done < YWhere,
- var1, var2, var3 … are the variables to store the data of every line. For each line these variables are separated by the specified IFS. We don’t require the IFS and the multiple variables for reading the entire line in one variable.
- Y represents the method to feed the data to the loop
Examples
Let's go through use cases with some while read line examples.
Assume we have 2 sample files
1. sample1.txt with the following contents:
Roger Federer SUI
Rafael Nadal ESP
Novak Djokovic SER
Martina Hingis SUI
Steffi Graf GER2. sample2.txt with the following contents:
USA;Washington D.C.;Dollar
UAE; Abu Dhabi;Dirham
South Africa;Pretoria;Rand
Japan;Tokyo;Yen
1. Read line from file
In Bash, reading lines from a file is pretty easy. We can do it using the while loop. We just have to make use of the built-in read command to read one line at a time of the specified file.
Example:
#!/bin/bash
while read -r line
do
echo "$line" #printing the line; perform any other operation on line variable
count=`echo $line | wc -c` #counting the number of letters on each line
echo "Characters: $count”
done < sample1.txt
For each iteration of the while loop, the read command reads that particular line of the file and assigns it to the bash shell variable $line. The loop runs till the number of lines in the file. After reading the last line of the file, the while loop stops. The -r flag of the read command prevents the interpretation of any backslash-escaped characters while reading.
2. Read line from command
Just like we read each line from a file, we can read the entire multiline output of the command one line at a time. We can achieve this using process substitution, a very useful Bash extension. Process substitution delivers the output of a process into the stdin of another process.
Example
#!/bin/bash
while read -r line
do
echo line is "$line" #printing the line; perform any other operation on line variable
done < <(ls -l)
Process substitution allows us to turn the output of the command (ls-l) into a temporary file. In every iteration, the line variable is getting assigned that particular line of the output. This temporary file vanishes after exiting the loop.
3. Read line from variable
We can also read line by line from a variable storing output in multiple lines. We will use the same logic we have used earlier.
Example:
#!/bin/bash
var=`ls -l` #var holds the output of ls -l
while read -r line
do
echo line is "$line"
done <<< $var #command substitution
while read -r line
do
echo line from here is "$line"
done < <(echo "$var") #process substitution
In this example we are storing the output of ls -l command in a variable using the backticks (`) technique.
In the first case we are doing the command substitution $() and making use of the here-string <<<. For the nth run of the while loop, the line number n will be stored in the variable line.
Another way to read line by line from a variable is to make use of the process substitution which we have seen in the previous example. For that, we are using the variable along with the echo command and feeding it to the loop.
4. Reading from pipe
Another interesting way to read line by line is to make use of the shell pipe. This technique can work both with multiline files, variables as well as commands. Pipe instructs the shell to use the output of the command on the left as the input.
Example:
#!/bin/bash
x=`ls -l`
echo "$x" | #pipe on variable
while read -r line
do
echo line is "$line"
done
ls -l | #pipe on command
while read -r line
do
echo line is "$line"
done
cat sample1.txt | #pipe on file
while read -r line
do
echo "$line"
done
From this example, we can see how we can make use of the shell pipe to feed the file, variable and the command to read the contents line by line. The read command will read from standard input by default, so we just pipe into it.
5. Reading lines into multiple variables
Sometimes we have to read comma separated value (CSV) files where the number of columns are fixed. We can read each entry of the csv in a variable and process it accordingly.
Example
#!/bin/bash
while read -r a b c
do
echo "First entry is $a"
echo "Second entry is $b"
echo "Third entry is $c"
done < sample1.txt
while IFS=';' read -r a b c
do
echo "First entry is $a"
echo "Second entry is $b"
echo "Third entry is $c"
done < sample2.txt
In this first case, each line of the entry is split into 3 variables. For each line $a will store the first entry, $b will hold the second and $c the third.
However, the entries may not always be separated by a space just like sample2.txt. To overcome that, we have added the internal field separator (IFS). This variable indicates how the entries are separated, which in our case is a semicolon (;).
Conclusion
- We can use the Bash while loop to read from a file, a command and from a variable.
- Process and command substitution come in handy while reading line by line.
- While reading lines into multiple variables, IFS can be used to split the entries and store them in the corresponding variables.
Comments