This is the tertiary part of the Bash Ane-Liners Explained commodity series. In this part I'll teach y'all all about input/output redirection. I'll use just the best fustigate practices, diverse bash idioms and tricks. I desire to illustrate how to get various tasks done with just bash built-in commands and bash programming language constructs.

See the outset part of the series for introduction. After I'm washed with the series I'll release an ebook (similar to my ebooks on awk, sed, and perl), and also bash1line.txt (like to my perl1line.txt).

Too see my other manufactures about working fast in bash from 2007 and 2008:

  • Working Productively in Fustigate's Emacs Command Line Editing Mode (comes with a crook sheet)
  • Working Productively in Bash's Vi Command Line Editing Manner (comes with a cheat sheet)
  • The Definitive Guide to Bash Control Line History (comes with a crook canvass)

Let's start.

Role Three: Redirections

Working with redirections in fustigate is really piece of cake once you realize that it'south all nigh manipulating file descriptors. When bash starts it opens the three standard file descriptors: stdin (file descriptor 0), stdout (file descriptor 1), and stderr (file descriptor 2). Y'all tin open more file descriptors (such as 3, iv, v, ...), and you lot can close them. You can also copy file descriptors. And you can write to them and read from them.

File descriptors always bespeak to some file (unless they're airtight). Usually when bash starts all 3 file descriptors, stdin, stdout, and stderr, point to your terminal. The input is read from what you blazon in the terminal and both outputs are sent to the terminal.

Bold your terminal is /dev/tty0, here is how the file descriptor table looks similar when bash starts:

When bash runs a control it forks a child procedure (meet man 2 fork) that inherits all the file descriptors from the parent process, then it sets up the redirections that you specified, and execs the command (see human being 3 exec).

To be a pro at bash redirections all yous need to do is visualize how the file descriptors get inverse when redirections happen. The graphics illustrations will help you.

1. Redirect the standard output of a command to a file 

$ command >file

Operator > is the output redirection operator. Bash outset tries to open the file for writing and if it succeeds it sends the stdout of command to the newly opened file. If it fails opening the file, the whole command fails.

Writing command >file is the same every bit writing command 1>file. The number 1 stands for stdout, which is the file descriptor number for standard output.

Here is how the file descriptor table changes. Bash opens file and replaces file descriptor one with the file descriptor that points to file. And so all the output that gets written to file descriptor i from now on ends up being written to file:

In general you can write control n>file, which will redirect the file descriptor n to file.

For instance,

$ ls > file_list

Redirects the output of the ls control to the file_list file.

2. Redirect the standard error of a command to a file 

$ control 2> file

Here bash redirects the stderr to file. The number 2 stands for stderr.

Hither is how the file descriptor table changes:

Bash opens file for writing, gets the file descriptor for this file, and information technology replaces file descriptor ii with the file descriptor of this file. Then now anything written to stderr gets written to file.

3. Redirect both standard output and standard error to a file 

$ command &>file

This ane-liner uses the &> operator to redirect both output streams - stdout and stderr - from command to file. This is fustigate's shortcut for apace redirecting both streams to the same destination.

Here is how the file descriptor table looks like later bash has redirected both streams:

As you lot tin can encounter both stdout and stderr at present betoken to file. So anything written to stdout and stderr gets written to file.

There are several ways to redirect both streams to the aforementioned destination. You can redirect each stream one after another:

$ command >file 2>&1

This is a much more common way to redirect both streams to a file. First stdout is redirected to file, and then stderr is duplicated to exist the same as stdout. So both streams end up pointing to file.

When bash sees several redirections it processes them from left to right. Permit'due south go through the steps and run into how that happens. Before running any commands fustigate'south file descriptor table looks like this:

At present bash processes the first redirection >file. We've seen this before and it makes stdout point to file:

Side by side bash sees the second redirection ii>&1. We haven't seen this redirection before. This ane duplicates file descriptor ii to be a copy of file descriptor 1 and we become:

Both streams have been redirected to file.

However be careful here! Writing:

command >file 2>&1

Is not the aforementioned as writing:

$ control 2>&1 >file

The guild of redirects matters in bash! This control redirects only the standard output to the file. The stderr will all the same print to the terminal. To empathize why that happens, let's get through the steps again. So before running the command the file descriptor tabular array looks like this:

Now bash processes redirections left to right. Information technology get-go sees 2>&1 so it duplicates stderr to stdout. The file descriptor table becomes:

Now bash sees the 2d redirect >file and it redirects stdout to file:

Do you see what happens here? Stdout now points to file but the stderr all the same points to the terminal! Everything that gets written to stderr still gets printed out to the screen! So be very, very careful with the order of redirects!

Besides note that in fustigate, writing this:

$ command &>file

Is exactly the aforementioned as:

$ command >&file

The beginning form is preferred however.

4. Discard the standard output of a command 

$ command > /dev/null

The special file /dev/zero discards all information written to it. So what we're doing hither is redirecting stdout to this special file and it gets discarded. Here is how information technology looks from the file descriptor tabular array's perspective:

Similarly, past combining the previous one-liners, we can discard both stdout and stderr by doing:

$ control >/dev/null two>&i

Or just simply:

$ command &>/dev/goose egg

File descriptor tabular array for this feat looks like this:

5. Redirect the contents of a file to the stdin of a command 

$ command <file

Here bash tries to open the file for reading before running any commands. If opening the file fails, bash quits with error and doesn't run the command. If opening the file succeeds, bash uses the file descriptor of the opened file as the stdin file descriptor for the command.

After doing that the file descriptor tabular array looks like this:

Hither is an example. Suppose you want to read the first line of the file in a variable. You tin can simply do this:

$ read -r line < file

Fustigate's born read command reads a single line from standard input. By using the input redirection operator < we set up it up to read the line from the file.

6. Redirect a bunch of text to the stdin of a command 

$ command &lt;&lt;EOL your multi-line text goes here EOL

Here we use the here-certificate redirection operator <<Marker. This operator instructs bash to read the input from stdin until a line containing only MARKER is establish. At this point bash passes the all the input read so far to the stdin of the command.

Here is a common instance. Suppose you've copied a bunch of URLs to the clipboard and you want to remove http:// part of them. A quick 1-liner to do this would be:

$ sed 's|http://||' <<EOL http://url1.com http://url2.com http://url3.com EOL

Here the input of a listing of URLs is redirected to the sed control that strips http:// from the input.

This case produces this output:

url1.com url2.com url3.com

vii. Redirect a single line of text to the stdin of a control 

$ command &lt;&lt;&lt; "foo bar baz"

For example, let'south say you quickly desire to laissez passer the text in your clipboard every bit the stdin to a control. Instead of doing something like:

$ echo "clipboard contents" | command

Yous can at present just write:

$ command &lt;&lt;&lt; "clipboard contents"

This trick inverse my life when I learned it!

8. Redirect stderr of all commands to a file forever 

$ exec 2>file $ command1 $ command2 $ ...

This one-liner uses the born exec bash command. If you specify redirects after information technology, and so they will final forever, meaning until y'all change them or exit the script/shell.

In this case the 2>file redirect is setup that redirects the stderr of the current crush to the file. Running commands afterwards setting up this redirect will accept the stderr of all of them redirected to file. It's really useful in situations when you want to have a complete log of all errors that happened in the script, but you don't want to specify 2>file after every single command!

In full general exec tin can take an optional argument of a command. If it's specified, bash replaces itself with the command. So what you lot get is just that command running, and there is no more beat.

9. Open up a file for reading using a custom file descriptor 

$ exec 3&lt;file

Here we utilize the exec command once again and specify the iii<file redirect to information technology. What this does is opens the file for reading and assigns the opened file-descriptor to the trounce'south file descriptor number three. The file descriptor tabular array now looks like this:

Now y'all tin can read from the file descriptor iii, like this:

$ read -u 3 line

This reads a line from the file that nosotros just opened as fd 3.

Or yous tin can use regular shell commands such as grep and operate on file descriptor 3:

$ grep "foo" <&3

What happens here is file descriptor iii gets duplicated to file descriptor i - stdin of grep. Just retrieve that once you read the file descriptor information technology's been exhausted and you need to shut it and open information technology again to use it. (You can't rewind an fd in fustigate.)

After you're washed using fd 3, you can shut it this way:

$ exec iii>&-

Here the file descriptor three is duped to -, which is bash's special way to say "shut this fd".

10. Open a file for writing using a custom file descriptor 

$ exec 4>file

Here we simply tell bash to open file for writing and assign it number 4. The file descriptor table looks like this:

As you tin encounter file descriptors don't take to be used in gild, you tin can open any file descriptor number you similar from 0 to 255.

Now we tin can just write to the file descriptor 4:

$ echo "foo" >&4

And nosotros tin close the file descriptor four:

$ exec 4>&-

Information technology's so simple now in one case we learned how to piece of work with custom file descriptors!

xi. Open up a file both for writing and reading 

$ exec 3<>file

Here nosotros use bash's diamond operator <>. The diamond operator opens a file descriptor for both reading and writing.

And then for example, if you do this:

$ repeat "foo bar" > file   # write string "foo bar" to file "file". $ exec five<> file           # open "file" for rw and assign information technology fd 5. $ read -n 3 var <&five       # read the first three characters from fd v. $ echo $var

This will output foo every bit we only read the first 3 chars from the file.

At present nosotros can write some stuff to the file:

$ echo -n + >&v           # write "+" at 4th position. $ exec 5>&-               # shut fd 5. $ cat file

This will output foo+bar every bit nosotros wrote the + char at quaternary position in the file.

12. Ship the output from multiple commands to a file 

$ (command1; command2) >file

This one-liner uses the (commands) construct that runs the commands a sub-beat out. A sub-trounce is a child procedure launched by the current shell.

So what happens here is the commands command1 and command2 go executed in the sub-crush, and bash redirects their output to file.

thirteen. Execute commands in a shell through a file

Open 2 shells. In beat out 1 do this:

mkfifo fifo exec < fifo

In trounce 2 do this:

exec iii> fifo; echo 'echo examination' >&three

Now take a wait in beat out 1. It will execute echo test. You can keep writing commands to fifo and vanquish 1 volition go on executing them.

Here is how it works.

In beat out ane nosotros use the mkfifo command to create a named pipe called fifo. A named pipe (also called a FIFO) is similar to a regular pipe, except that information technology's accessed equally part of the file system. It can exist opened past multiple processes for reading or writing. When processes are exchanging data via the FIFO, the kernel passes all data internally without writing it to the file organization. Thus, the FIFO special file has no contents on the file organization; the file organisation entry but serves as a reference point so that processes can access the pipe using a name in the file system.

Next nosotros use exec < fifo to replace current shell's stdin with fifo.

Now in shell 2 we open the named pipe for writing and assign it a custom file descriptor 3. Adjacent we simply write echo examination to the file descriptor 3, which goes to fifo.

Since trounce 1'southward stdin is continued to this pipe it executes information technology! Really simple!

14. Access a website through fustigate 

$ exec iii<>/dev/tcp/world wide web.google.com/80 $ echo -eastward "GET / HTTP/1.1\n\n" >&3 $ true cat <&3

Bash treats the /dev/tcp/host/port as a special file. It doesn't need to exist on your system. This special file is for opening tcp connections through bash.

In this example we first open file descriptor three for reading and writing and betoken information technology to /dev/tcp/www.google.com/80 special file, which is a connection to www.google.com on port lxxx.

Next nosotros write Become / HTTP/one.1\due north\n to file descriptor 3. And then nosotros simply read the response dorsum from the same file descriptor by using cat.

Similarly yous tin can create a UDP connection through /dev/udp/host/port special file.

With /dev/tcp/host/port you can even write things like port scanners in fustigate!

fifteen. Prevent overwriting the contents of a file when redirecting output 

$ set -o noclobber

This turns on the noclobber pick for the current trounce. The noclobber selection prevents yous from overwriting existing files with the > operator.

If y'all try redirecting output to a file that exists, you lot'll get an error:

$ programme > file fustigate: file: cannot overwrite existing file

If you're 100% sure that you desire to overwrite the file, apply the >| redirection operator:

$ programme >| file

This succeeds every bit it overrides the noclobber option.

16. Redirect standard input to a file and print it to standard output 

$ command | tee file

The tee command is super handy. It's not part of bash merely you'll utilise information technology ofttimes. Information technology takes an input stream and prints it both to standard output and to a file.

In this example it takes the stdout of command, puts it in file, and prints it to stdout.

Here is a graphical illustration of how it works:

17. Send stdout of one procedure to stdin of another process 

$ command1 | command2

This is elementary piping. I'm sure everyone is familiar with this. I'm just including it here for completeness. Only to remind you lot, a pipage connects stdout of command1 with the stdin of command2.

Information technology can be illustrated with a graphic:

As you can see, everything sent to file descriptor 1 (stdout) of command1 gets redirected through a pipe to file descriptor 0 (stdin) of command2.

You can read more about pipes in human two piping.

18. Send stdout and stderr of one process to stdin of some other process 

$ command1 |& command2

This works on bash versions starting 4.0. The |& redirection operator sends both stdout and stderr of command1 over a pipage to stdin of command2.

Every bit the new features of fustigate 4.0 aren't widely used, the onetime, and more than portable manner to do the same is:

$ command1 2>&ane | command2

Hither is an illustration that shows what happens with file descriptors:

Kickoff command1'south stderr is redirected to stdout, and and so a pipe is setup betwixt command1's stdout and command2's stdin.

19. Requite file descriptors names 

$ exec {filew}>output_file

Named file descriptors is a characteristic of bash 4.ane. Named file descriptors look similar {varname}. You can utilise them just similar regular, numeric, file descriptors. Bash internally chooses a complimentary file descriptor and assigns information technology a name.

20. Order of redirections

You lot tin put the redirections anywhere in the command you want. Cheque out these iii examples, they all do the same:

$ echo hullo >/tmp/example  $ repeat >/tmp/example howdy  $ >/tmp/example echo hello

Got to love bash!

21. Swap stdout and stderr 

$ control 3>&one 1>&2 2>&3

Here we first duplicate file descriptor 3 to be a copy of stdout. And so we duplicate stdout to be a copy of stderr, and finally we duplicate stderr to be a re-create of file descriptor 3, which is stdout. As a result we've swapped stdout and stderr.

Allow'due south go through each redirection with illustrations. Before running the control, we've file descriptors pointing to the final:

Side by side fustigate setups three>&1 redirection. This creates file descriptor three to be a re-create of file descriptor 1:

Adjacent bash setups 1>&ii redirection. This makes file descriptor 1 to be a copy of file descriptor 2:

Side by side fustigate setups two>&3 redirection. This makes file descriptor 2 to be a re-create of file descriptor 3:

If we desire to be squeamish citizens nosotros tin can also close file descriptor three as it'due south no longer needed:

$ command 3>&1 ane>&2 2>&three iii>&-

The file descriptor table then looks similar this:

As y'all tin see, file descriptors 1 and 2 have been swapped.

22. Transport stdout to one procedure and stderr to another process 

$ control > >(stdout_cmd) 2> >(stderr_cmd)

This ane-liner uses procedure substitution. The >(...) operator runs the commands in ... with stdin connected to the read part of an anonymous named pipage. Bash replaces the operator with the filename of the anonymous pipage.

And so for example, the first substitution >(stdout_cmd) might return /dev/fd/threescore, and the second commutation might return /dev/fd/61. Both of these files are named pipes that bash created on the fly. Both named pipes have the commands as readers. The commands await for someone to write to the pipes so they can read the information.

The control then looks like this:

$ command > /dev/fd/60 2> /dev/fd/61

At present these are just simple redirections. Stdout gets redirected to /dev/fd/60, and stderr gets redirected to /dev/fd/61.

When the control writes to stdout, the process backside /dev/fd/60 (process stdout_cmd) reads the information. And when the command writes to stderr, the procedure backside /dev/fd/61 (process stderr_cmd) reads the data.

23. Observe the exit codes of all piped commands

Let's say yous run several commands all piped together:

$ cmd1 | cmd2 | cmd3 | cmd4

And you want to discover out the get out status codes of all these commands. How practise you exercise information technology? In that location is no easy manner to go the exit codes of all commands as bash returns only the leave lawmaking of the final control.

Fustigate developers thought about this and they added a special PIPESTATUS array that saves the get out codes of all the commands in the pipage stream.

The elements of the PIPESTATUS array correspond to the leave codes of the commands. Here'south an case:

$ echo 'pants are cool' | grep 'moo' | sed 's/o/x/' | awk '{ print $1 }' $ repeat ${PIPESTATUS[@]} 0 1 0 0

In this case grep 'moo' fails, and the 2nd element of the PIPESTATUS assortment indicates failure.

Shoutouts

Shoutouts to bash hackers wiki for their illustrated redirection tutorial and bash version changes.

Enjoy!

Bask the commodity and let me know in the comments what yous think about it! If you call back that I forgot some interesting bash ane-liners related to redirections, let me know in the comments below!