CSC231 Bash Tutorial 3

--D. Thiebaut (talk) 10:23, 16 February 2017 (EST)

---

In this lab you will learn about the manual pages, and pipes.

<showafterdate after="20170217 11:00" before "20170601 00:00">

Manual Pages

• Linux contains its own documentation. It's called the Linux Manual Pages, and you access them using the man command.
• Let's get the manual pages for some of the commands you know:

man ls
 

• the line ls [OPTION]... [FILE]... indicates that the command takes optional arguments, or switches, and optionally the name of a file. The line below indicates that -a ls will not ignore entries starting with a dot.
• Let's see how to apply this information (do not type the text in parentheses):

cd ls ls *.asm          ls -a ls -l

(go back to your home directory) (this will list all the files) (this will list only the files ending with .asm) (this will list all the files in your home directory, including the hidden files) (the minus-ell will ask for additional information about each file, including when it was created)

• Other man pages you should look at:

man rm
man cp
man mkdir
man rmdir
man tail

• If for some reason you want to use a command but you don't know its name, try searching the man pages for a keyword. For example, you forgot the command name hexdump, but remember that it displays information in hexadecimal...

man -k hexadecimal

If you go up the list, you should find hexdump.

The man pages were created at a time when search engines did not exist, and they were the quick way for Linux users to have access to the documentation of Linux commands. Nowadays you can probably get less cryptic and more informative examples by using Google to search for various commands, either by name, or by keywords. None the less, what you will get on Google will only be a variation on the original contents of the man pages.

Pipes

• Linux applications/programs/commands work with streams of information. They get their input from different kinds of streams. The input stream is usually the keyboard, and we referred to it as stdin for standard input. The output usually goes to an output stream, which most often is the video display, and we refer to it as stdout for standard output. (Image taken from RyansTutorials.net)
• In some cases, the information we need requires using many different commands, and using the output of one as the input of the other. This is where the idea of a pipe taking the output of one command and feeding it to the input of an other command comes in.

Example 1

• Let's study the files contained in a large directory:

ls /etc

this will list all the files in the /etc directory. These are the preferences or options files for many of the apps on aurora. Unfortunately, we get too much information to look at.

• We know that less can display the contents of a file one page at a time. So we will take the output of ls /etc and pipe it to less, as follows:

ls  /etc   |  less

Observe that the output of the ls command is now viewable one page at a time. You can go up and down with b and space keys. Quit with q.

The vertical bar | is the symbol use to take the output of the command on its left and feed it as an input to the command on its right.

The wc Command

• Here's a new command that is often useful: wc, which stands for word-count. It will count the number of words, or lines, or characters in a file.
• Let's see how it works. If you have gotten rid of Ulysses, get a fresh copy of it:

getcopy Ulysses.txt

• Let's get its stats:

 wc Ulysses.txt

• the output:

 32577  267949 1580927 Ulysses.txt

which means that Ulysses contains 32577 lines, 267j,949 words, andn 1,580,927 characters.

• We can also specify that we want only one of these quantities, by adding a switch to wc:

  wc -l Ulysses.txt
  wc -w Ulysses.txt
  wc -c Ulysses.txt

Bring up the man page for wc and see what other switches it supports.

Example 2: Counting stuff

• How many lines are in the /etc directory? Well, we can do an ls and pipe the output to wc and get what we want:

 ls /etc   | wc

• We see that the number of lines (file name) is the same as the number of words. That's because file names in Linux normally do not contain spaces.
• If we wanted to be accurate and get only the number of files, and not three numbers:

 ls /etc  | wc -l

Challenge #1:

How many users are registered users on aurora? Hints: all users have their home directory in /Users. The exception is student accounts, which are located somewhere else (because they are erased regularly, between semesters).

Challenge #2:

How many Molecular-Biology accounts are registered users on aurora? Hints: they all start with "molebio" followed by 2 numbers.
(This one is tricky, because if you type ls /Users/molebio*, ls will return not only the directories, but also the files these directories contain... Look at the man page for ls to figure out what switch you can use to display only the directory name, and not what it contains).

Filtering with grep

• Grep is a powerful and fast way of searching text on Linux. We used it in the last tutorial to find all the Mulligans in Ulysses.txt:

grep -i mulligan Ulysses.txt

The command grep -i mulligan Ulysses.txt can be written a different way, one that highlights how grep works in a pipe:
     cat Ulysses.txt | grep -i Mulligan
The two commands generate identical outputs. The only difference is that in the first one, the file is provided to grep as an argument. In the second case, grep is provided with the stream generated by cat.

• How many lines do we have in Ulysses? Simple, we just feed the output of grep to the input of wc:

grep -i mulligan Ulysses.txt  | wc -l

• did you get 164?

• Grep also has a nice feature. You can ask it to find the lines not containing a particular strings. For example, if I wanted the list of all the users in /Users on aurora, excluding the Molecular-Biology accounts, we could do this:

 ls /Users | grep -v molebio

• and if I wanted to count how many non-molebio users we have:

 ls /Users | grep -v molebio  | wc -l 

Challenge #3:

How many user accounts (in /Users) were created on December 24?

Challenge #4

How many user accounts (in /Users) have a name that starts with the letter 'y'?
(if you find a number close to 130, your search is too wide. The actual number is closer to 15...)

Working on Logs

Logs are files that Linux maintains of all the activities on a particular machine. Typically these files are accessible only to the admin person who manages the computer, so we, as users, cannot access them.
But I copied a log from a server that I manage in my lab, and made it available to the class. You can get it as follows:

getcopy auth.log

• Look at the (very cryptic) information it contains with less

• I am suspecting that people have been trying to access my server and to login to it. I know that the log will record an unsuccessful attempt to log into a computer with the keyword "Failed password" in the log line.
• List all the lines of the log containing "Failed password" (you will need to use quotes around the two words to make them a string for grep).

Challenge #5

On February 16, how many recorded failed login attempts (lines) were logged by people other than User dthieb?

Challenge #6

When was the first failed login that is not from User dthieb in the auth log. Just printing the line corresponding to this attempt is sufficient. Your series of commands should display only 1 line.

Moodle Quiz

You are done with this lab, and ready for the Moodle Quiz!
The quiz will ask you to answer several questions relating to the file auth.log.1, which you need to get a copy of:

getcopy auth.log.1

</showafterdate>











































<showafterdate after="20170217 11:30" before="20170601 00:00"> Solution to the Challenges

Challenge 1

ls /Users | wc -l
928

Challenge 2

ls -d /Users/molebio* | wc -l
64

Challenge 3

ls -l /Users/ | grep "Dec 24" | wc -l
3

Challenge 4

ls -l /Users/ | grep -v 2016 | wc -l
720

Challenge 5

grep "Failed password" auth.log | grep -v dthieb | wc -l
67

Challenge 6

grep "Failed password" auth.log | grep -v dthieb | head -n 1
Feb 13 12:30:48 dftServer sshd[19111]: Failed password for invalid user vivek from 61.172.252.70 port 34660 ssh2

The first such attempt was on Feb 13, at 12:30 p.m.

</showafterdate>