Module 4.2 – Piping and I/O Redirection In Linux

Input and output (I/O) Redirection In Linux With Examples

Redirection In Linux is simply pointing/(redirecting) the output of a command to a different location.

In Linux, the input, output, or error output of a command can be redirected to a different file for various reasons. I remember redirecting the output of a few important commands many times in one of the projects I did sometime last 2 years.

I was to upgrade several RHEL 6 servers to RHEL 7, but before I started the upgrade process, I redirected the output of so many important files to another file for comparison after the upgrade. More so, If there are issues, I can easily check the files I redirected the output of the commands to for analysis.

There are three streams of I/O redirection in Linux. They are standard output, standard input, and standard error.

1. Standard output (stdout)

The > character is used to redirect the output of a command.

For the purpose of this activity, to view the contents of the fstab file, simply use the command,

[root@HQEBPRD ~]# cat /etc/fstab

# units generated from this file.
#
/dev/mapper/rhel-root   /                       xfs     defaults        0 0
UUID=e2529557-6f6c-457a-8e9b-f9fecb9af482 /boot                   xfs     defaults        0 0
/dev/mapper/rhel-swap   swap                    swap    defaults        0 0
#/dev/sdb1 /tekneed  ntfs-3g  defaults  0 0
#/dev/sdb2 /tekneed2  ntfs-3g  defaults  0 0
[root@HQEBPRD ~]#

To redirect the output of the command, “cat /etc/fstab” to a different location, use the command,

[root@HQEBPRD ~]# cat /etc/fstab > /tmp/test5

To verify the redirection, use the command,

[root@HQEBPRD ~]# cat /tmp/test5

# units generated from this file.
#
/dev/mapper/rhel-root   /                       xfs     defaults        0 0
UUID=e2529557-6f6c-457a-8e9b-f9fecb9af482 /boot                   xfs     defaults        0 0
/dev/mapper/rhel-swap   swap                    swap    defaults        0 0
#/dev/sdb1 /tekneed  ntfs-3g  defaults  0 0
#/dev/sdb2 /tekneed2  ntfs-3g  defaults  0 0
[root@HQEBPRD ~]#

To redirect the output of a command to a file that already contains contents, use the character >> . if you use > again, it will overwrite the former contents in that file.

To redirect the output of the command, “ls /boot” to “/tmp/test5”, use the command,

[root@HQEBPRD ~]# ls /boot >> /tmp/test5

To verify the redirection, use the command,

[root@HQEBPRD ~]# cat /tmp/test5

# units generated from this file.
#
/dev/mapper/rhel-root   /                       xfs     defaults        0 0
UUID=e2529557-6f6c-457a-8e9b-f9fecb9af482 /boot                   xfs     defaults        0 0
/dev/mapper/rhel-swap   swap                    swap    defaults        0 0
#/dev/sdb1 /tekneed  ntfs-3g  defaults  0 0
#/dev/sdb2 /tekneed2  ntfs-3g  defaults  0 0
config-4.18.0-147.el8.x86_64
efi
grub2
initramfs-0-rescue-a42eb5b5bacc41df93f034f80848713a.img
initramfs-4.18.0-147.el8.x86_64.img
initramfs-4.18.0-147.el8.x86_64kdump.img
loader
System.map-4.18.0-147.el8.x86_64
vmlinuz-0-rescue-a42eb5b5bacc41df93f034f80848713a
vmlinuz-4.18.0-147.el8.x86_64
[root@HQEBPRD ~]#

you can see that the output of the command, “ls boot” is added to “test5” file.

2. Standard Input (stdin)

The character, < is used to redirect the input of a file. Standard input means the input of a file can as well be redirected from a file. This simply means the contents in a file can be read and be redirected to the shell. An example is when you use the command,

[root@HQEBPRD ~]# wc  < /var/log/messages

  4410  70100 605824
[root@HQEBPRD ~]#

To print the new line count, use the command,

[root@HQEBPRD ~]# wc -l < /var/log/messages

4416

3. Standard error (stderr)

The character, 2> is used to redirect the output of an error to a different location. For example, if you run the command below, you will see some error messages.

[root@HQEBPRD ~]# du -sch /*

0       /bin
140M    /boot
0       /dev
30M     /etc
48K     /home
0       /lib
0       /lib64
0       /media
4.0K    /medis
0       /mnt
0       /opt
du: cannot access '/proc/24515': No such file or directory
du: cannot access '/proc/24517/task/24517/fd/3': No such file or directory
du: cannot access '/proc/24517/task/24517/fdinfo/3': No such file or directory
du: cannot access '/proc/24517/fd/3': No such file or directory
du: cannot access '/proc/24517/fdinfo/3': No such file or directory
0       /proc
12M     /root
7.4G    /run
0       /sbin
0       /srv
0       /sys
0       /teknee
0       /tekneed
0       /tekneed2
0       /test
204K    /tmp
3.8G    /usr
265M    /var
12G     total
[root@HQEBPRD ~]#

To redirect these error messages to a different location or another file, either for further analysis or any reason, use the command,

[root@HQEBPRD ~]# du -sch /* 2> /tmp/test6

0       /bin
140M    /boot
0       /dev
30M     /etc
48K     /home
0       /lib
0       /lib64
0       /media
4.0K    /medis
0       /mnt
0       /opt
0       /proc
12M     /root
7.4G    /run
0       /sbin
0       /srv
0       /sys
0       /teknee
0       /tekneed
0       /tekneed2
0       /test
208K    /tmp
3.8G    /usr
265M    /var
12G     total
[root@HQEBPRD ~]#

you can see that the error messages are no longer displayed. you can redirect error messages to another file for further analysis.

To verify the error redirection, use the command,

[root@HQEBPRD ~]# cat /tmp/test6

du: cannot access '/proc/24611/task/24611/fd/3': No such file or directory
du: cannot access '/proc/24611/task/24611/fdinfo/3': No such file or directory
du: cannot access '/proc/24611/fd/3': No such file or directory
du: cannot access '/proc/24611/fdinfo/3': No such file or directory
[root@HQEBPRD ~]#

To redirect more error to the same file, use the character, 2>> so not to overwrite the contents in the file.

---

Piping In Linux With Examples

piping is a way of sending the output of a command as the input of another command with the pipe (|) operator for easier filtering/match of an output.

just as water flows in a pipe from the inlet to the outlet and to the inlet of another pipe, the outlet(output) of a command in Linux can also be the inlet(input) of another command in Linux.

With the pipe utility, more than one Linux commands can be used at a time.

for example,

command 1 | command2 | command 3 |command n……

A practical example is when you run the command,

ls -l /sys/module | more

NB: tap the space bar to continue viewing

pipe takes the output of the “ls” command and gives it to the “more” command, then more displays the content the way more always displays a content as we saw in our previous tutorial when we looked at how more works .

Naturally, you can’t even use the command, “more” on listing a directory directly but pipe can help in this case

One of the differences between “|” and I/O redirection is that “|” redirects the output of a command to another command while > or 2> redirects to a file.

Let’s see more piping examples for clarity.

run the command below

[root@HQEBPRD module]# ls -l /sys/module

total 0
drwxr-xr-x. 3 root root 0 Apr 15 15:50 8250
drwxr-xr-x. 5 root root 0 Apr 15 15:50 ac97_bus
drwxr-xr-x. 3 root root 0 Apr 15 15:50 acpi
drwxr-xr-x. 3 root root 0 Apr 15 15:50 acpiphp
drwxr-xr-x. 7 root root 0 Apr 15 15:50 ahci
drwxr-xr-x. 6 root root 0 Apr 15 15:50 ata_generic
drwxr-xr-x. 6 root root 0 Apr 15 15:50 ata_piix
drwxr-xr-x. 3 root root 0 Apr 15 15:50 battery
drwxr-xr-x. 3 root root 0 Apr 15 15:50 blk_cgroup
drwxr-xr-x. 3 root root 0 Apr 15 15:50 block
drwxr-xr-x. 5 root root 0 Apr 15 15:50 bridge
drwxr-xr-x. 3 root root 0 Apr 15 15:50 button
drwxr-xr-x. 5 root root 0 Apr 15 15:50 cdrom
drwxr-xr-x. 3 root root 0 Apr 15 15:50 cec
drwxr-xr-x. 2 root root 0 Apr 15 15:50 configfs
drwxr-xr-x. 3 root root 0 Apr 15 15:50 cpufreq
drwxr-xr-x. 3 root root 0 Apr 15 15:50 cpuidle
drwxr-xr-x. 5 root root 0 Apr 15 15:50 crc32c_intel
drwxr-xr-x. 5 root root 0 Apr 15 15:50 crc32_pclmul
drwxr-xr-x. 5 root root 0 Apr 15 15:50 crct10dif_pclmul
drwxr-xr-x. 3 root root 0 Apr 15 15:50 cryptomgr
drwxr-xr-x. 3 root root 0 Apr 15 15:50 debug_core
drwxr-xr-x. 5 root root 0 Apr 15 15:50 dm_log
drwxr-xr-x. 6 root root 0 Apr 15 15:50 dm_mirror
drwxr-xr-x. 6 root root 0 Apr 15 15:50 dm_mod
drwxr-xr-x. 5 root root 0 Apr 15 15:50 dm_region_hash
drwxr-xr-x. 6 root root 0 Apr 15 15:50 drm
drwxr-xr-x. 6 root root 0 Apr 15 15:50 drm_kms_helper
drwxr-xr-x. 3 root root 0 Apr 15 15:50 dynamic_debug
drwxr-xr-x. 7 root root 0 Apr 15 15:50 e1000
drwxr-xr-x. 3 root root 0 Apr 15 15:50 edac_core

Now try to run the same command above, pipe it with the “more” command to see the differences.

[root@HQEBPRD module]# ls -l /sys/module |more

total 0
drwxr-xr-x. 3 root root 0 Apr 15 15:50 8250
drwxr-xr-x. 5 root root 0 Apr 15 15:50 ac97_bus
drwxr-xr-x. 3 root root 0 Apr 15 15:50 acpi
drwxr-xr-x. 3 root root 0 Apr 15 15:50 acpiphp
drwxr-xr-x. 7 root root 0 Apr 15 15:50 ahci
drwxr-xr-x. 6 root root 0 Apr 15 15:50 ata_generic

Again, run the command below,

[root@HQEBPRD module]# sort --help

Usage: sort [OPTION]... [FILE]...
  or:  sort [OPTION]... --files0-from=F
Write sorted concatenation of all FILE(s) to standard output.

With no FILE, or when FILE is -, read standard input.

Mandatory arguments to long options are mandatory for short options too.
Ordering options:

  -b, --ignore-leading-blanks  ignore leading blanks
  -d, --dictionary-order      consider only blanks and alphanumeric characters
  -f, --ignore-case           fold lower case to upper case characters
  -g, --general-numeric-sort  compare according to general numerical value
  -i, --ignore-nonprinting    consider only printable characters
  -M, --month-sort            compare (unknown) < 'JAN' < ... < 'DEC'
  -h, --human-numeric-sort    compare human readable numbers (e.g., 2K 1G)
  -n, --numeric-sort          compare according to string numerical value
  -R, --random-sort           shuffle, but group identical keys.  See shuf(1)
      --random-source=FILE    get random bytes from FILE
  -r, --reverse               reverse the result of comparisons
      --sort=WORD             sort according to WORD:
                                general-numeric -g, human-numeric -h, month -M,
                                numeric -n, random -R, version -V
  -V, --version-sort          natural sort of (version) numbers within text

Other options:

Pipe the command to more and see the difference.

N:B: tap space bar to continue viewing

[root@HQEBPRD module]# sort --help |more

Usage: sort [OPTION]... [FILE]...
  or:  sort [OPTION]... --files0-from=F
Write sorted concatenation of all FILE(s) to standard output.

With no FILE, or when FILE is -, read standard input.

Mandatory arguments to long options are mandatory for short options too.
Ordering options:

  -b, --ignore-leading-blanks  ignore leading blanks
  -d, --dictionary-order      consider only blanks and alphanumeric characters
  -f, --ignore-case           fold lower case to upper case characters
  -g, --general-numeric-sort  compare according to general numerical value
  -i, --ignore-nonprinting    consider only printable characters
  -M, --month-sort            compare (unknown) < 'JAN' < ... < 'DEC'
  -h, --human-numeric-sort    compare human readable numbers (e.g., 2K 1G)
  -n, --numeric-sort          compare according to string numerical value
  -R, --random-sort           shuffle, but group identical keys.  See shuf(1)
      --random-source=FILE    get random bytes from FILE
  -r, --reverse               reverse the result of comparisons
      --sort=WORD             sort according to WORD:
                                general-numeric -g, human-numeric -h, month -M,

Again, you can use the command below to view the first 10 lines of the “/sys/module” directory by piping the “ls -l” command to the “head” command

[root@HQEBPRD module]# ls -l /sys/module |head -10

total 0
drwxr-xr-x. 3 root root 0 Apr 15 15:50 8250
drwxr-xr-x. 5 root root 0 Apr 15 15:50 ac97_bus
drwxr-xr-x. 3 root root 0 Apr 15 15:50 acpi
drwxr-xr-x. 3 root root 0 Apr 15 15:50 acpiphp
drwxr-xr-x. 7 root root 0 Apr 15 15:50 ahci
drwxr-xr-x. 6 root root 0 Apr 15 15:50 ata_generic
drwxr-xr-x. 6 root root 0 Apr 15 15:50 ata_piix
drwxr-xr-x. 3 root root 0 Apr 15 15:50 battery
drwxr-xr-x. 3 root root 0 Apr 15 15:50 blk_cgroup

Again, let’s pipe “ls -l” to “head” and to “tail” with example on the /sys/module directory.

( ls -l |head |tail )

[root@HQEBPRD module]# ls -l |head -10 |tail -5

drwxr-xr-x. 7 root root 0 Apr 15 15:50 ahci
drwxr-xr-x. 6 root root 0 Apr 15 15:50 ata_generic
drwxr-xr-x. 6 root root 0 Apr 15 15:50 ata_piix
drwxr-xr-x. 3 root root 0 Apr 15 15:50 battery
drwxr-xr-x. 3 root root 0 Apr 15 15:50 blk_cgroup
[root@HQEBPRD module]#

Redirections and pipes can also be combined. Let’s see this with example on “/sys/module” directory

[root@HQEBPRD module]# ls -l |head -10 > /tmp/now

[root@HQEBPRD module]# cat /tmp/now

total 0
drwxr-xr-x. 3 root root 0 Apr 15 15:50 8250
drwxr-xr-x. 5 root root 0 Apr 15 15:50 ac97_bus
drwxr-xr-x. 3 root root 0 Apr 15 15:50 acpi
drwxr-xr-x. 3 root root 0 Apr 15 15:50 acpiphp
drwxr-xr-x. 7 root root 0 Apr 15 15:50 ahci
drwxr-xr-x. 6 root root 0 Apr 15 15:50 ata_generic
drwxr-xr-x. 6 root root 0 Apr 15 15:50 ata_piix
drwxr-xr-x. 3 root root 0 Apr 15 15:50 battery
drwxr-xr-x. 3 root root 0 Apr 15 15:50 blk_cgroup
[root@HQEBPRD module]#

more example

[root@HQEBPRD module]# who -a |more

           system boot  2020-04-14 22:26
           run-level 5  2020-04-14 22:27
root     + tty2         2020-04-14 22:28  old         2459 (tty2)
root     + pts/1        2020-04-15 22:16   .         13252 (192.168.170.1)
           pts/2        2020-04-15 12:01              4429 id=ts/2  term=0 exit=0
[root@HQEBPRD module]#

N:B: to be able to effectively use pipe, especially for scripting without errors, run the commands one after the other on the shell, if it successfully runs, you can introduce the pipe command.

Your feedback is welcomed. If you love others, you will share with others