Difference between revisions of "CSC231 Bash Tutorial 4"

From dftwiki3
Jump to: navigation, search
 
(3 intermediate revisions by the same user not shown)
Line 19: Line 19:
 
* Once you are done, get a copy of a short text file, edit it (enter your first and last name, as well as your account number in the right place), and submit it.  <font color="magenta">Do not user emacs to edit the file!</font>  The steps are illustrated below:
 
* Once you are done, get a copy of a short text file, edit it (enter your first and last name, as well as your account number in the right place), and submit it.  <font color="magenta">Do not user emacs to edit the file!</font>  The steps are illustrated below:
 
   
 
   
  getcopy viFile.txt
+
  cp ~cs231a/bin/getcopy .
 +
./getcopy viFile.txt
 
  vi viFile.txt
 
  vi viFile.txt
  submitVi viFile.txt
+
  ~cs231a/bin/submitVi viFile.txt
 
   
 
   
 
<br />
 
<br />
Line 147: Line 148:
 
   
 
   
 
:These messages are error messages reported by '''du'''.  They do not create '''du''' to crash.  '''du''' is just reporting failure to look at a particular directory or file.
 
:These messages are error messages reported by '''du'''.  They do not create '''du''' to crash.  '''du''' is just reporting failure to look at a particular directory or file.
* Let's capture all these "Permission denied" message to a file:
+
* Let's capture the output of du to a file, and then we can search the file for the "Permission denied" messages:
 
   
 
   
 
  du /etc  > duErrors.txt
 
  du /etc  > duErrors.txt
 
   
 
   
 +
* What just happened?  It looks like what you just typed printed '''only''' the error lines to the screen.
 
* and let's look in the file just captured:
 
* and let's look in the file just captured:
 
   
 
   
 
  cat duErrors.txt
 
  cat duErrors.txt
 
   
 
   
:?
+
 
* Do you see anything in duErrors.txt?
+
* Do you see any of the Permission error lines in duErrors.txt?
 
* Why?
 
* Why?
 
* What's happening?
 
* What's happening?
Line 197: Line 199:
 
Get a copy of the java program '''GenErrors.class'''
 
Get a copy of the java program '''GenErrors.class'''
 
   
 
   
  getcopy GenErrors.class
+
  ./getcopy GenErrors.class
 
   
 
   
 
and run it:
 
and run it:
Line 271: Line 273:
 
* Get a copy of the file GenErrorsQuiz as follows:
 
* Get a copy of the file GenErrorsQuiz as follows:
 
   
 
   
  getcopy GenErrorsQuiz.class
+
  ./getcopy GenErrorsQuiz.class
 
  chmod +x GenErrorsQuiz.class
 
  chmod +x GenErrorsQuiz.class
 
   
 
   
Line 337: Line 339:
 
<br />
 
<br />
 
<br />
 
<br />
 +
<showafterdate after="20171008 12:00" before="20171231 00:00">
 +
=Solution to Quiz=
 +
<br />
 +
==Source==
 +
<br />
 +
::<source lang="java">
 +
class GenErrorsQuiz {
 +
 +
    public static void main( String args[] ) {
 +
int count=0;
 +
String notUsed1 = "Summer will soon be here!";
 +
String notUsed2 = "Pizza is the staple of a computer scientist's diet";
 +
String notUsed3 = "Lady Gaga is a good singer";
 +
 +
for ( int j=0; j<2000000; j++ ) {
 +
    int i = j/2;
 +
if (i==500000)
 +
    System.out.println( "ligne numero " + j +
 +
": The 3 Bs of good life: Brie, Baguette, and Beaujolais" );  
 +
else if ((i+1)%119==0 )
 +
    System.err.println( "Error line " + (j+1) );
 +
   
 +
else if (i==600000)
 +
    System.err.println( "Bad line:" + j +
 +
": Fall Break is approaching!" );
 +
 +
else
 +
    System.out.println( "line " + (j+1) );
 +
}
 +
    }
 +
}
 +
 +
</source>
 +
==Answers==
 +
<br />
 +
::<source lang="text">
 +
cs231a@aurora ~/handout $ java GenErrorsQuiz > GenErrorsQuiz.stdout 2> GenErrorsQuiz.stderr
 +
cs231a@aurora ~/handout $ ls -ltr | tail -n 5
 +
-rw------- 1 cs231a cs231a  122332 Oct  4 13:27 dummy
 +
-rw------- 1 cs231a cs231a 24679852 Oct  4 13:28 GenErrors.stdout
 +
-rw------- 1 cs231a cs231a  310070 Oct  4 13:28 GenErrors.stderr
 +
-rw------- 1 cs231a cs231a 24679852 Oct  4 13:37 GenErrorsQuiz.stdout
 +
-rw------- 1 cs231a cs231a  310070 Oct  4 13:37 GenErrorsQuiz.stderr
 +
cs231a@aurora ~/handout $ wc -l GenErrorsQuiz.stdout
 +
1983192 GenErrorsQuiz.stdout
 +
cs231a@aurora ~/handout $ wc -l GenErrorsQuiz.stderr
 +
16808 GenErrorsQuiz.stderr
 +
cs231a@aurora ~/handout $ head GenErrorsQuiz.stdout
 +
line 1
 +
line 2
 +
line 3
 +
line 4
 +
line 5
 +
line 6
 +
line 7
 +
line 8
 +
line 9
 +
line 10
 +
cs231a@aurora ~/handout $ grep -v line GenErrorsQuiz.stdout
 +
ligne numero 1000000: The 3 Bs of good life: Brie, Baguette, and Beaujolais
 +
ligne numero 1000001: The 3 Bs of good life: Brie, Baguette, and Beaujolais
 +
cs231a@aurora ~/handout $ head -n 3 GenErrorsQuiz.stderr
 +
Error line 237
 +
Error line 238
 +
Error line 475
 +
cs231a@aurora ~/handout $ grep -v Error GenErrorsQuiz.stderr
 +
Bad line:1200000: Fall Break is approaching!
 +
Bad line:1200001: Fall Break is approaching!
 +
</source>
 +
 +
</showafterdate>
 
<br />
 
<br />
 
<br />
 
<br />

Latest revision as of 11:41, 6 October 2017

--D. Thiebaut (talk) 14:13, 22 February 2017 (EST)


<showafterdate after="20170104 11:00" before="20171231 00:00">

Using the Vi Editor

InteractiveVIM.jpg


Vi is the default editor with most distributions of Linux. If you get a Raspberry Pi, for example, and install Linux on it, chances are that the only editor you will have access to is vi, or vim (Vi IMproved), which is a friendlier version of vi.
This means that every so often, when you deal with Linux, you are faced with having to do some editing and emacs is not around, and you may not have privileges to install it, so knowing the basic commands of vi can be a life saver.

  • Introduction
  • Two Modes
  • Basic Movement


  • Once you are done, get a copy of a short text file, edit it (enter your first and last name, as well as your account number in the right place), and submit it. Do not user emacs to edit the file! The steps are illustrated below:
cp ~cs231a/bin/getcopy .
./getcopy viFile.txt
vi viFile.txt
~cs231a/bin/submitVi viFile.txt


Good short reference on vi
Colorado State's page on vi


CodeLiverOil.jpg


Redirection


  • We have seen in the previous lab how to use pipes to take the output of a program and feed it to the input of another.
  • Redirection is similar, except that the input or the output is fed from, or fed to a file, respectively.


Example 1: storing the output of a command into a file


  • Try this:
last

this will list the last login times of all the users on Aurora. If you want to target a particular user, say 231a, your instructor, you can try this:
last | grep "231a "                    (you need a space at the end to make sure you don't match with student accounts)               
  • Unfortunately, our class accounts have changed recently. They used to be of the form 231a-xx, but now they are cs231a-xx, and the extra two letters at the front (cs) prevents us from see the suffix -xx in the list of users generated by Linux. So we can't really figure out who logged when regarding holders of class accounts.
  • Assume you want to store these login times of User cs231a into a file, so that you can share it with somebody else, or send as an attachment. You simply use the > sign to store the output of the last command into a file:
last | grep "231a "  >  logins231a.txt

  • You won't see any output, as it is being redirected. Look at the files in your directory and see that you have a new file called logins231a.txt.
  • Display the contents of this file with the cat or with the less command.


You now have a new way of storing the output of a program into a file.


Challenge #1:

QuestionMark1.jpg


Create a text file called "231Students.txt" that contains their last login information, but make sure the information from User 231a (your instructor) is not stored in the file. (Hints: check the man page for grep and locate the -v switch).





Side-step: Sorting text


Linux sports a command to sort text. It's called "sort." To see how it works, try this:

last | sort

  • If you had wanted the output in reverse order, you could have typed:
last | sort -r

  • Sometimes the information returned by a linux command contains numbers at the beginning of the first line. du, which returns the "disk usage" of a directory, i.e. the amount of disk space taken by each file, is such a command. Let's try it:
du /etc

You get a list of all the files in the /etc directory, with their size at the beginning of each line.
  • Try to sort this list:
du /etc  | sort

  • See the problem? Sorting numbers alphabetically does not yield what we expect. Instead we can tell sort to sort numerically:
du /etc  | sort -n


Challenge #2:

QuestionMark2.jpg


Create a text file called "etc_decreasing.txt" using redirection, and make it contain the list of files in /etc, sorted numerically by file size, largest first.







Challenge #3:

QuestionMark3.jpg


Create a text file using redirection that contains only the 10 largest files found in the /etc/ directory. The file should be called etc_top10.txt. Remove the last line of the output of sort which lists the total for the /etc directory. We want the 10 largest excluding /etc. In other words, /etc/ shouldn't be included in the etc_top10.txt file.







Redirecting Stderr


  • Run the du command another time, and pay attention to the lines that contain a "Permission denied" message:
 du /etc

When I run it, I get some "Permission denied" messages interspersed here and there:
28	/etc/pm
4	/etc/ndiswrapper
8	/etc/rc4.d
4	/etc/mate-settings-daemon/xrandr
8	/etc/mate-settings-daemon
du: cannot read directory ‘/etc/chatscripts’: Permission denied
4	/etc/chatscripts
28088	/etc

These messages are error messages reported by du. They do not create du to crash. du is just reporting failure to look at a particular directory or file.
  • Let's capture the output of du to a file, and then we can search the file for the "Permission denied" messages:
du /etc  > duErrors.txt

  • What just happened? It looks like what you just typed printed only the error lines to the screen.
  • and let's look in the file just captured:
cat duErrors.txt

  • Do you see any of the Permission error lines in duErrors.txt?
  • Why?
  • What's happening?
The explanation is that when du outputs one of the "Permission denied" lines, it does not output them to sdtout, the standard output. It outputs the to a different output stream, called stderr, for standard error. This is the stream one normally uses to output error messages.


By default, outputs to stderr and stdout are sent to the screen. If you use > to capture the output of a program, you capture only the part sent to stdout and not the part sent to stderr. To capture the output sent to stderr to a file, we need to use 2> as a symbol:
du /etc 2>  duErrors.txt
and now try:
cat duErrors.txt

You have captured the stderr output to a file.


Getting Rid of Stderr


Sometimes we know that there will be message sent to stderr by the command/program, but we don't want to see them. Possibly because there are too many.

For example, assume we want to apply du to all the student accounts, which live in /Users/classes/ and sort the resulting list to see who's hogging up the most disk space:

du /Users/classes  | sort -n | tail -n 10

  • Try the previous command. See how the error messages are polluting the output?
  • What we could do, instead, is sent the stderr lines to a file, and sort only the stdout lines coming out of du:
du /Users/classes 2> duErrors.txt  | sort -n | tail -n 10

  • Try it.
  • The output is much cleaner, isn't it?


Challenge #4:

QuestionMark4.jpg


Get a copy of the java program GenErrors.class

./getcopy GenErrors.class

and run it:

java GenErrors

How many lines does this program send to stderr?
How many lines does it send to stdout?



Redirecting stdin


Stdin normally represents the keyboard. To see how you can redirect it, create this very simple Python program called demo.py:

#! /usr/bin/env python3

print( "Enter 3 integers, one per line: " )
x = int( input( "> " ) )
y = int( input( "> " ) )
z = int( input( "> " ) )
print( "sum = ", x+y+z )


and make the program executable:
chmod +x demo.py

and run it:
./demo.py

  • Assume that this program computes something much more involved than the simple summation shown here, and that you have to run and test your program many times with the same values, say 10, 20, and 30. One way to spend less time running your program is to force it to get its inputs from a file rather than from the keyboard, and force you to enter the same numbers over and over:
  • Create a text file your favorite editor called myinputs.txt and add 3 lines containing the 3 numbers:
10
20
30

10 should be one the first line. 30 on the third. Make sure you type ENTER at least twice after the last line.
  • Now run the program and force it to get its input from the file myinputs.txt:
./demo.py < myinputs.txt


  • You will notice that the output is confusing. None of the numbers obtained from the input file appear on the screen, and all the input prompts are on the same line. This is because when you type an input at the keyboard, it automatically is echoed on on the screen, including when you press the ENTER key: it moves the cursor on the screen to the next line. When you feed the input to the program from a file, then there's no echo to the display. This is why the 3 prompts are on the same line, and why we don't see the 3 numbers that are fed to the program.
  • Get used to it. That's how redirecting the input works.



Challenge #5:

QuestionMark5.jpg


Run the demo.py program again, so that it gets its input from the myinputs.txt file, and such that you capture its output, and only the line that says "sum =" into a file called myoutput.txt.









Moodle Quiz


You are ready for the Moodle quiz.

  • Get a copy of the file GenErrorsQuiz as follows:
./getcopy GenErrorsQuiz.class
chmod +x GenErrorsQuiz.class

  • Run the program:
java GenErrorsQuiz
line 1
line 2
line 3
line 4
line 5
line 6
line 7
line 8
line 9
line 10
...
Error line 237
...

  • The quiz on Moodle will ask you precise questions about the output generated by this java program.


</showafterdate>






<showafterdate after="20171004 11:30" before="20171231 00:00">

Challenges Solutions


Challenge 1
last | grep "231a" | grep -v "231a "  > 231students.txt

Challenge 2
du /etc/ | sort -n -r > etc_decreasing.txt
Challenge 3
du /etc | sort -n | tail -n 11 | head -n 10 > etc_top10.txt
Challenge 4
java GenErrors 2> GenErrors.txt | wc -l
9915

9915 lines are sent to stdout
wc -l GenErrors.txt
85 GenErrors.txt

85 lines are sent to stderr
Challenge 5
./demo.py < myinputs.txt  | grep sum > myoutputs.txt
cat myoutputs.txt 
> > > sum =  60

</showafterdate>

<showafterdate after="20171008 12:00" before="20171231 00:00">

Solution to Quiz


Source


class GenErrorsQuiz {

    public static void main( String args[] ) {
	int count=0;
	String notUsed1 = "Summer will soon be here!";
	String notUsed2 = "Pizza is the staple of a computer scientist's diet";
	String notUsed3 = "Lady Gaga is a good singer";

	for ( int j=0; j<2000000; j++ ) {
	    int i = j/2;
		if (i==500000)
		    System.out.println( "ligne numero " + j + 
					": The 3 Bs of good life: Brie, Baguette, and Beaujolais" );	  
		else if ((i+1)%119==0 ) 
		    System.err.println( "Error line " + (j+1) );
		    
		else if (i==600000)
		    System.err.println( "Bad line:" + j + 
					": Fall Break is approaching!" );

		else 
		    System.out.println( "line " + (j+1) );
	}
    }
}

Answers


cs231a@aurora ~/handout $ java GenErrorsQuiz > GenErrorsQuiz.stdout 2> GenErrorsQuiz.stderr
cs231a@aurora ~/handout $ ls -ltr | tail -n 5
-rw------- 1 cs231a cs231a   122332 Oct  4 13:27 dummy
-rw------- 1 cs231a cs231a 24679852 Oct  4 13:28 GenErrors.stdout
-rw------- 1 cs231a cs231a   310070 Oct  4 13:28 GenErrors.stderr
-rw------- 1 cs231a cs231a 24679852 Oct  4 13:37 GenErrorsQuiz.stdout
-rw------- 1 cs231a cs231a   310070 Oct  4 13:37 GenErrorsQuiz.stderr
cs231a@aurora ~/handout $ wc -l GenErrorsQuiz.stdout
1983192 GenErrorsQuiz.stdout
cs231a@aurora ~/handout $ wc -l GenErrorsQuiz.stderr
16808 GenErrorsQuiz.stderr
cs231a@aurora ~/handout $ head GenErrorsQuiz.stdout
line 1
line 2
line 3
line 4
line 5
line 6
line 7
line 8
line 9
line 10
cs231a@aurora ~/handout $ grep -v line GenErrorsQuiz.stdout
ligne numero 1000000: The 3 Bs of good life: Brie, Baguette, and Beaujolais
ligne numero 1000001: The 3 Bs of good life: Brie, Baguette, and Beaujolais
cs231a@aurora ~/handout $ head -n 3 GenErrorsQuiz.stderr
Error line 237
Error line 238
Error line 475
cs231a@aurora ~/handout $ grep -v Error GenErrorsQuiz.stderr
Bad line:1200000: Fall Break is approaching!
Bad line:1200001: Fall Break is approaching!

</showafterdate>