CSC231 Bash Tutorial 4
--D. Thiebaut (talk) 14:13, 22 February 2017 (EST)
<showafterdate after="20170104 11:00" before="20171231 00:00">
Contents
Using the Vi Editor
Vi is the default editor with most distributions of Linux. If you get a Raspberry Pi, for example, and install Linux on it, chances are that the only editor you will have access to is vi, or vim (Vi IMproved), which is a friendlier version of vi.
This means that every so often, when you deal with Linux, you are faced with having to do some editing and emacs is not around, and you may not have privileges to install it, so knowing the basic commands of vi can be a life saver.
- Go through the first 3 sections of this interactive tutorial on Vim:
- Introduction
- Two Modes
- Basic Movement
- Once you are done, get a copy of a short text file, edit it (enter your first and last name, as well as your account number in the right place), and submit it. Do not user emacs to edit the file! The steps are illustrated below:
cp ~cs231a/bin/getcopy . ./getcopy viFile.txt vi viFile.txt ~cs231a/bin/submitVi viFile.txt
- Good short reference on vi
- Colorado State's page on vi
Redirection
- We have seen in the previous lab how to use pipes to take the output of a program and feed it to the input of another.
- Redirection is similar, except that the input or the output is fed from, or fed to a file, respectively.
Example 1: storing the output of a command into a file
- Try this:
last
- this will list the last login times of all the users on Aurora. If you want to target a particular user, say 231a, your instructor, you can try this:
last | grep "231a " (you need a space at the end to make sure you don't match with student accounts)
- Unfortunately, our class accounts have changed recently. They used to be of the form 231a-xx, but now they are cs231a-xx, and the extra two letters at the front (cs) prevents us from see the suffix -xx in the list of users generated by Linux. So we can't really figure out who logged when regarding holders of class accounts.
- Assume you want to store these login times of User cs231a into a file, so that you can share it with somebody else, or send as an attachment. You simply use the > sign to store the output of the last command into a file:
last | grep "231a " > logins231a.txt
- You won't see any output, as it is being redirected. Look at the files in your directory and see that you have a new file called logins231a.txt.
- Display the contents of this file with the cat or with the less command.
You now have a new way of storing the output of a program into a file.
Challenge #1: |
Create a text file called "231Students.txt" that contains their last login information, but make sure the information from User 231a (your instructor) is not stored in the file. (Hints: check the man page for grep and locate the -v switch).
Side-step: Sorting text
Linux sports a command to sort text. It's called "sort." To see how it works, try this:
last | sort
- If you had wanted the output in reverse order, you could have typed:
last | sort -r
- Sometimes the information returned by a linux command contains numbers at the beginning of the first line. du, which returns the "disk usage" of a directory, i.e. the amount of disk space taken by each file, is such a command. Let's try it:
du /etc
- You get a list of all the files in the /etc directory, with their size at the beginning of each line.
- Try to sort this list:
du /etc | sort
- See the problem? Sorting numbers alphabetically does not yield what we expect. Instead we can tell sort to sort numerically:
du /etc | sort -n
Challenge #2: |
Create a text file called "etc_decreasing.txt" using redirection, and make it contain the list of files in /etc, sorted numerically by file size, largest first.
Challenge #3: |
Create a text file using redirection that contains only the 10 largest files found in the /etc/ directory. The file should be called etc_top10.txt. Remove the last line of the output of sort which lists the total for the /etc directory. We want the 10 largest excluding /etc. In other words, /etc/ shouldn't be included in the etc_top10.txt file.
Redirecting Stderr
- Run the du command another time, and pay attention to the lines that contain a "Permission denied" message:
du /etc
- When I run it, I get some "Permission denied" messages interspersed here and there:
28 /etc/pm 4 /etc/ndiswrapper 8 /etc/rc4.d 4 /etc/mate-settings-daemon/xrandr 8 /etc/mate-settings-daemon du: cannot read directory ‘/etc/chatscripts’: Permission denied 4 /etc/chatscripts 28088 /etc
- These messages are error messages reported by du. They do not create du to crash. du is just reporting failure to look at a particular directory or file.
- Let's capture the output of du to a file, and then we can search the file for the "Permission denied" messages:
du /etc > duErrors.txt
- What just happened? It looks like what you just typed printed only the error lines to the screen.
- and let's look in the file just captured:
cat duErrors.txt
- Do you see any of the Permission error lines in duErrors.txt?
- Why?
- What's happening?
- The explanation is that when du outputs one of the "Permission denied" lines, it does not output them to sdtout, the standard output. It outputs the to a different output stream, called stderr, for standard error. This is the stream one normally uses to output error messages.
- By default, outputs to stderr and stdout are sent to the screen. If you use > to capture the output of a program, you capture only the part sent to stdout and not the part sent to stderr. To capture the output sent to stderr to a file, we need to use 2> as a symbol:
du /etc 2> duErrors.txt
- and now try:
cat duErrors.txt
- You have captured the stderr output to a file.
Getting Rid of Stderr
Sometimes we know that there will be message sent to stderr by the command/program, but we don't want to see them. Possibly because there are too many.
For example, assume we want to apply du to all the student accounts, which live in /Users/classes/ and sort the resulting list to see who's hogging up the most disk space:
du /Users/classes | sort -n | tail -n 10
- Try the previous command. See how the error messages are polluting the output?
- What we could do, instead, is sent the stderr lines to a file, and sort only the stdout lines coming out of du:
du /Users/classes 2> duErrors.txt | sort -n | tail -n 10
- Try it.
- The output is much cleaner, isn't it?
Challenge #4: |
Get a copy of the java program GenErrors.class
getcopy GenErrors.class
and run it:
java GenErrors
- How many lines does this program send to stderr?
- How many lines does it send to stdout?
Redirecting stdin
Stdin normally represents the keyboard. To see how you can redirect it, create this very simple Python program called demo.py:
#! /usr/bin/env python3 print( "Enter 3 integers, one per line: " ) x = int( input( "> " ) ) y = int( input( "> " ) ) z = int( input( "> " ) ) print( "sum = ", x+y+z )
- and make the program executable:
chmod +x demo.py
- and run it:
./demo.py
- Assume that this program computes something much more involved than the simple summation shown here, and that you have to run and test your program many times with the same values, say 10, 20, and 30. One way to spend less time running your program is to force it to get its inputs from a file rather than from the keyboard, and force you to enter the same numbers over and over:
- Create a text file your favorite editor called myinputs.txt and add 3 lines containing the 3 numbers:
10 20 30
- 10 should be one the first line. 30 on the third. Make sure you type ENTER at least twice after the last line.
- Now run the program and force it to get its input from the file myinputs.txt:
./demo.py < myinputs.txt
- You will notice that the output is confusing. None of the numbers obtained from the input file appear on the screen, and all the input prompts are on the same line. This is because when you type an input at the keyboard, it automatically is echoed on on the screen, including when you press the ENTER key: it moves the cursor on the screen to the next line. When you feed the input to the program from a file, then there's no echo to the display. This is why the 3 prompts are on the same line, and why we don't see the 3 numbers that are fed to the program.
- Get used to it. That's how redirecting the input works.
Challenge #5: |
Run the demo.py program again, so that it gets its input from the myinputs.txt file, and such that you capture its output, and only the line that says "sum =" into a file called myoutput.txt.
Moodle Quiz
You are ready for the Moodle quiz.
- Get a copy of the file GenErrorsQuiz as follows:
getcopy GenErrorsQuiz.class chmod +x GenErrorsQuiz.class
- Run the program:
java GenErrorsQuiz line 1 line 2 line 3 line 4 line 5 line 6 line 7 line 8 line 9 line 10 ... Error line 237 ...
- The quiz on Moodle will ask you precise questions about the output generated by this java program.
</showafterdate>
<showafterdate after="20171004 11:30" before="20171231 00:00">
Challenges Solutions
- Challenge 1
last | grep "231a" | grep -v "231a " > 231students.txt
- Challenge 2
du /etc/ | sort -n -r > etc_decreasing.txt
- Challenge 3
du /etc | sort -n | tail -n 11 | head -n 10 > etc_top10.txt
- Challenge 4
java GenErrors 2> GenErrors.txt | wc -l 9915
- 9915 lines are sent to stdout
wc -l GenErrors.txt 85 GenErrors.txt
- 85 lines are sent to stderr
- Challenge 5
./demo.py < myinputs.txt | grep sum > myoutputs.txt cat myoutputs.txt > > > sum = 60
</showafterdate>
<showafterdate after="20171008 12:00" before="20171231 00:00">
Solution to Quiz
Source
class GenErrorsQuiz { public static void main( String args[] ) { int count=0; String notUsed1 = "Summer will soon be here!"; String notUsed2 = "Pizza is the staple of a computer scientist's diet"; String notUsed3 = "Lady Gaga is a good singer"; for ( int j=0; j<2000000; j++ ) { int i = j/2; if (i==500000) System.out.println( "ligne numero " + j + ": The 3 Bs of good life: Brie, Baguette, and Beaujolais" ); else if ((i+1)%119==0 ) System.err.println( "Error line " + (j+1) ); else if (i==600000) System.err.println( "Bad line:" + j + ": Fall Break is approaching!" ); else System.out.println( "line " + (j+1) ); } } }
Answers
cs231a@aurora ~/handout $ java GenErrorsQuiz > GenErrorsQuiz.stdout 2> GenErrorsQuiz.stderr cs231a@aurora ~/handout $ ls -ltr | tail -n 5 -rw------- 1 cs231a cs231a 122332 Oct 4 13:27 dummy -rw------- 1 cs231a cs231a 24679852 Oct 4 13:28 GenErrors.stdout -rw------- 1 cs231a cs231a 310070 Oct 4 13:28 GenErrors.stderr -rw------- 1 cs231a cs231a 24679852 Oct 4 13:37 GenErrorsQuiz.stdout -rw------- 1 cs231a cs231a 310070 Oct 4 13:37 GenErrorsQuiz.stderr cs231a@aurora ~/handout $ wc -l GenErrorsQuiz.stdout 1983192 GenErrorsQuiz.stdout cs231a@aurora ~/handout $ wc -l GenErrorsQuiz.stderr 16808 GenErrorsQuiz.stderr cs231a@aurora ~/handout $ head GenErrorsQuiz.stdout line 1 line 2 line 3 line 4 line 5 line 6 line 7 line 8 line 9 line 10 cs231a@aurora ~/handout $ grep -v line GenErrorsQuiz.stdout ligne numero 1000000: The 3 Bs of good life: Brie, Baguette, and Beaujolais ligne numero 1000001: The 3 Bs of good life: Brie, Baguette, and Beaujolais cs231a@aurora ~/handout $ head -n 3 GenErrorsQuiz.stderr Error line 237 Error line 238 Error line 475 cs231a@aurora ~/handout $ grep -v Error GenErrorsQuiz.stderr Bad line:1200000: Fall Break is approaching! Bad line:1200001: Fall Break is approaching!
</showafterdate>
