Redirections in Linux with Examples

1. Introduction

Many times, when we execute a command that generates a long output, we wish to capture the results into a file. Other times, we feel lacking when we use commands that don't have an option for reading files. Here is where file redirection comes into play.

File redirection in Linux comes in handy in a variety of situations. Need to save the output of a command to a file? No problem. Want to use the contents of a file as input for a command such as tr? Easy peasy. And sometimes, you just need to discard output that would otherwise clutter up your terminal.

Redirection operators let you manipulate input and output streams. These will enable you to send the output of a command to a file, or use the contents of a file as input for a command. Although piping is a form of redirection between two or more commands, but redirections typically refers to the process of redirecting output to a file or using a file's content as input to a command.

In this tutorial, you will be acquainted with the concept of I/O redirections, and the different ways to control the flow of standard output, standard input and standard error.

2. Getting The Full Picture

Command-line tools usually read input from the keyboard and write output to the screen — but that's not always the case. With redirections, we can change the flow of data, making commands and files read data from a different source or write it to another target.

2.1. Understanding Linux Data Streams

In Linux, data streams refer to three standard I/O channels known as standard input (stdin), standard output (stdout), and standard error (stderr). These channels represent the way data moves into or out of a program:

• Stdin (Standard input) is the default input source for a command and has a file descriptor of 0. It represents the data that a command receives as input, and by default, stdin is linked with the keyboard.
• Stdout (Standard output) is the default output destination for a command and has a file descriptor of 1. It represents the output generated by a command that appears on-screen.
• Stderr (Standard error) is the channel for error messages produced by a command or the shell and has a file descriptor of 2. It also writes error messages on-screen.

2.2. Using Redirection

The shell controls the input and output of the commands it runs from or to files using special symbols. You can use the less than symbol < and the greater than symbol > to move data between commands and files. The > symbol sends the output of a command to a file as input, while < sends the content of a file to a program as input.

Each of these symbols is a block of data that has two file descriptors, one for reading and the other for writing. For instance:

• > : The left side reads > the right side writes
• < : The left side writes < the right side reads

Try to imagine the < or > symbols as a pointer showing how data is directed to the command from a file, or vice versa.

When you deal with redirections interactively with the shell, always keep the subsequent synopsis in mind are:

• command < input-file : reads from input-file
• command > output-file : create/overwrites output-file
• command >> output-file :  Append to output-file

3. Redirecting Output

In this section, you'll see various examples of redirecting the standard output of a process into a file.

3.1. Using the Output Redirection Operator

First thing first, let's start by firing up the Terminal. Once you're in, let's try out this command:

$ echo "Why so serious?"
Why so serious?

Naturally, the output of the echo command goes to the screen. Implementing what we've learned, let's use the greater than symbol to make standard output goes to a new file instead of the screen:

$ echo "Why so serious?" > bouhannana

You may ask where is the output? The command uses the > symbol to make the shell writes the output directly into the bouhannana file without filling up your screen. Let's be sure and check the content of this new file:

$ cat bouhannana
Why so serious?

See? The expected output is stored in the file. Another thing to keep in mind is when using the output redirection operator, it'll create a new file with the filename you specified. However, if the file already exists, the shell will overwrite your file:

$ echo "Where is everyone?" > bouhannana

Now let's check its content:

$ cat bouhannana

Did you notice what just happened? The shell wiped out the previous content to write the new output. This is something you should always be careful about so you'll not accidentally lose important data.

One great trick for creating files is to use the output redirection with no command in the left operand:

$ > bouhannana
$ ls

The command creates an empty file simply because it has no data to redirect.

Keep in mind that the > operator is synonymous with 1>, which explicitly specifies the stdout file descriptor.

3.2. Using the Append Redirection Operator

Linux also offers an operator to avoid truncating an existing file while redirecting output to it. Just by using a double greater than symbol >> instead of one, the shell will append output to the existing file without overwriting it.

Let's check the following file:

$ cat quotes.txt
A man who suffers before it is necessary, suffers more than is necessary.

Next, let's redirect output to the file without losing the previous data:

$ echo "Fear is pain arising from the anticipation of evil." >> quotes.txt 
$ cat quotes.txt 
A man who suffers before it is necessary, suffers more than is necessary.
Fear is pain arising from the anticipation of evil.

The shell kept the previous text and appended the new one. This operator will also create a new file if none matched the given filename.

You can use the append redirection (or the output redirection) to write directly into a file without using a text editor. To achieve this trick you can use the cat command and the >> operator just as so:

$ cat >> quotes.txt
The arrow shot by the archer may or may not kill a single person. 
But stratagems devised by wise men can kill even babes in the womb.

When you've finished typing, just hit ctrl+d to exit the mode and save the file. 

$ cat quotes.txt

This trick redirects the standard output of the keyboard directly into the file rather than the screen.

4. Redirecting Input

In this part of the article, we'll take a look at some examples of commands that receive input from a file. It's important to remember that only commands that generally receive input from the keyboard can get their input from a file. This form of redirection is also useful in case you're dealing with a command that can't handle files.

4.1. Using the Input Redirection Operator

Let's search for a specific word using grep, but having it "receive" the content of a file using the input redirection operator:

$ grep -w necessary < quotes.txt

The grep command above receives its input from the right-hand operand (quotes.txt) and then returns the desired result.

One could ask what's the difference between "grep -w necessary < quotes.txt" and "grep -w necessary quotes.txt"?  In the latter command, grep opens and processes the file's content. However, in our command, grep simply receives a stream of data (file content) and is not aware if the file quotes.txt exists or not.

Let's put this claim under the scoop for better understanding:

$ grep -w necessary < idontexist
bash: quote: No such file or directory
$ grep -w necessary idontexist
grep: quote: No such file or directory

Notice that in the first command, it's the shell that complains about the error in opening the file and not grep, since the grep command is not aware of the file. In contrast, the second command, gerp has the file as a parameter, so it's the one that catches errors related to processing the file.

4.2. When to use?

There are plenty of cases where you can opt for the < operator, but only a handful of cases where it's really necessary. One such case is when using the tr command to manipulate characters from standard input. Unlike grep, this command doesn't have a built-in option for opening and processing data in a file. However, you can circumvent this limitation using file - or pipe - redirections.

Suppose you have a text file with private phone numbers in it, and you want to delete all these digits before sending it to someone:

$ cat < employees.txt

Simply construct a valid tr command for such purpose and redirect the file's content to its stdin:

$ tr -d [:digit:]  < employees.txt

The command reads the input streams from the employees.txt file, and the "[:digit:]" character class matches all digits so that the -d option deletes them.

5. Redirecting Error Output

In addition to stdin and stdout, there's a third data stream; Standard error or stderr. And just like its cousin stdout, the stderr also writes to the screen by default. As though this stream appears on-screen, it's reserved for error messages only.

What does all this mean? Simply put, our typical > operator won't redirect error messages, as it implicitly means 1> (1 is the file descriptor for stdout). Now, let's check the following example where the command produces an error:

$ cat quotes.txt idontexist > outfile

The shell didn't pass the error message to the output file, instead kept the default behavior. To change this, you should use the output operator and explicitly specify the number of the stderr file descriptor, using the syntax "2>".

Let's check how this works:

$ cat quotes.txt idontexist > outfile 2> std_error

Now the command will redirect the successful results of cat to outfile. However, if there is an error, it will redirect it to the std_error file

Let's check the outcome:

$ cat std_error

You can also specify the stderr file descriptor number when dealing with append redirection:

$ cat idontexist stdout_file.txt > outfile 2>> std_error

5.1. Discarding Error Messages

If you have a command that could produce a lot of errors and you don't want them to clutter your screen or fill up files, you have a solution. You may point the standard error to the /dev/null special file (null device):

$ cat quotes.txt idontexist > outfile 2> /dev/null

This file is always empty, so it won't store any of the error messages and will save you from having to look at them.

5.2. Redirecting Output and Errors into a Single File

To redirect both standard output and standard error to the same file, we can use the & symbol along with the output > or append >> operators. This is useful when we want to capture all on-screen output, including error messages, into a single file:

$ cat quotes.txt idontexist &> output_and_error
$ cat output_and_error

The &> operator combines both streams into one file. It's basically a shorthand for 2>&1, which means that we're telling the shell to redirect the standard error (file descriptor 2) to the same place as the standard output (file descriptor 1).

6. Conclusion

In conclusion, redirection is an ancient and potent feature deeply rooted in the Linux command line, granting users the ability to manipulate input and output streams effortlessly. By redirecting output to a file or input from a file, you gain the power to save and process data seamlessly in diverse ways. Moreover, embracing the ins and outs of redirections opens up a world of possibilities letting you do some cool tricks. For instance, you can create new files on the fly without using specific commands, or write directly into a file without relying on a text editor.