CSC352 Class Page 2010

Main Page | Syllabus | Schedule | Links & Resources

Hadoop-Related

• Hadoop FAQ Page
• Hadoop IPs

Projects

• Project 1: Started Feb 16, due March 2nd. A good example of a solution.
• Project 2: Deals with the Xgrid. A collection of good proposed solutions.
• Project 3 Deals with processing Wikipedia pages on Hadoop/MapReduce.

• Project 4 has officially started on 1/26/10! and is now Over. Find its shared wiki page here! Congratulations to the class for setting up the individual nodes of the first Smith Cloud Cluster!

Python Threads

Week	Topics	Reading
Week 1 1/25	Tuesday Introduce Syllabus Interrupts Thursday The PogoPlug... The iPad... Python Review NQueens.py: a python program to find the first solution for N queens on an NxN board. Lecture Notes	Read What is a Thread? [1] What is a Processes? [2] Reference material for when we start programming Python.org's page on Thread Objects. It's the reference, but is not always crystal clear... Linux Gazette Article on Python Threads. Good overall introduction. Norm Matloff's introduction to Python Threads. More details, less easy to follow. For Discussion next Tuesday read the paper by Asanovic K. et al, The Landscape of Parallel Computing Research: A View from Berkeley.
Week 2 2/1	Tuesday discussion (prepare a 1-page summary) of the paper by Asanovic K. et al, The Landscape of Parallel Computing Research: A View from Berkeley, Dec. 2006. (cached copy) Thursday Python classExample1.py classExample2.py classExample3.py serialPing.py Python Threads threadedPing.py : a threaded solution to the serial ping program ThreadedNQueens.py : a threaded solution to the N-Queens program Homework #1 Lecture Notes	Make sure to read Norman Matloff's tutorial on threads in the section on Python on the Resource Page. Also, don't hesitate to check Python.org for information on threads.
Week 3 2/8	Tuesday An experiment with Control-C... Sharing in shared memory the problem atomicity of operations semaphores fair safe live Python locks examples: threadedPingWithLocks.py threadedPingWithSemaphores.py Thursday Discussion: "Why Von Neumann?" Continuation of Shared Memory and Sharing Python Queues threadedPingWithQueues.py Some notes on Python and treads An alternative to Threading: Multiprocessing Available with Python 2.6 and up only multiprocessingNQueens.py Comparison of the two methods Performance measures Speedup Execution time Processor utilization Processor efficiency Processor efficacy Guidlines for presenting research papers Deadlocks Homework #2, Solution 1 and Solution 2 Lecture Notes	Measuring Performance Video from Sun Microsystems on parallelizing applications and improving their performance (19 minutes) Deadlocks PEP 371 (cached copy)
Week 4 2/15	Tuesday Discusion of Xen and the Art of Virtualization (Presentation by Le) One person presents Everybody else submits a 1-page summary in 3 parts. Review of HW #1 A visit of the XGrid system (Photos) Thursday Python Q&A Amdahl's Law Deadlocks: the main rule How to Read Technical Papers Summarize this! Good background information: MPI Lecture Notes	Additional information on Xen can be found in Mauer, R., Xen Virtualization and Linux Clustering, Linux Journal January 12th, 2006. A video on the Python GIL discovered by Diana. A video by Oblong Industries (in reference to the NYT article to summarize)

XGrid Programming

Week	Topics	Reading
Week 5	Tuesday Introduction to the next projects A Video presentation of the XGrid (watch first 10 minutes) Tutorial/Lab #1: Monte Carlo on the XGrid Thursday: Presentation and discussion of Amdahl's Law in the Multicore Era, Mark Hill and Michael Marty, IEEE Computer, July 2008, and accompanying dynamic graph. (cached copy) Continuation of the Xgrid Monte-Carlo tutorial Lecture Notes	Apple's XGrid Introduction
Week 6	Tuesday Presentation of Projects #2 and #3 Videos Pres by DT John Udell's capture of the page Heavy Metal Umlaut Wikipedia Research Projects Intro to Projects #2 and #3 (presentation) Thursday Canceled by DT Lecture Notes	Read Hughes, B., Building Computational Grids with Apple's XGrid Middleware, ACM International Conference Proceeding Series, Vol. 167, Hobart, Tasmania, Australia, 2006. For next Tuesday.
Week 7	Tuesday Presentation and discussion of Building Computational Grids with Apple's XGrid Middleware, by Hughes, B. (ACM International Conference Proceeding Series, Vol. 167, Hobart, Tasmania, Australia, 2006.) (cached copy) Define Homework #3 Guidelines for projects. (Presentation) (We stopped on the bell-curve). Thursday Guidelines for projects. (Presentation) XGrid Lab 2 Scheduling Processor Scheduling (OS) Multiprocessor Scheduling XGrid Scheduling Homework #3 and its solution programs. Lecture Notes
Week 8	Tuesday Sign-up for paper presentations here! Class participation on the decomposition of Homework 3/Project 2 (the serial part) Thursday Continuation of decomposition of Homework 3/Project 2 (the parallel part) XGrid Lab 3: running jobs on the Science Center XGrid. Lecture Notes

Cloud Computing

Week	Topics	Reading
Week 9	Tuesday Presentation of MapReduce: Simplified Data Processing on Large Clusters (Yang) Introduction to MapReduce/Hadoop (We stopped at Section 4). Thursday Continuation of Introduction to MapReduce/Hadoop and Tutorial #1, Tutorial #2 Lecture Notes Hadoop Howtos and FAQs	Map-Reduce tutorial from apache.org: a must-read! Map-Reduce Basics from Yahoo.com: another must-read! Section 6 in Tom White's Hadoop, the Definitive Guide, available on Google Books.
Week 10	Tuesday Presentation of MapReduce: Distributed Computing for Machine Learning Continuation of Introduction to MapReduce/Hadoop Tutorial #1, Tutorial #1: output logs Tutorial #1.1: Task Timelines Thursday Question of the day: What's wrong with this picture? Continuation of Introduction to MapReduce/Hadoop Lab for today: Compare WordCount on 1 vs WordCount on 6 Section 5 of Tutorial 1 Create your own version of the Java WordCount program Section 4 of Tutorial 1 Create your own Counters Section 6 of Tutorial 1: count Buck! Generate Timelines Tutorial 1.1 Counting words in Python Tutorial 2 Lecture Notes Homework #4 and a solution.
Week 11	Tuesday Presentation of MapReduce: A Flexible Data Processing Tool ( cached copy) Compare with Paulson, Rasin, Abadi, DeWitt, Madden, and Stonebraker's paper A Comparison of Approaches to Large Scale Data-Analysis, SIGMOD-09, June 2009. Thursday Hypertable is an open-source project parallel to Google's BigTable... Art vs Science... Some preliminary thinking about the final project... Streaming whole files WordCount in C++ Visualizations of Hadoop Data Transfers, from the U. of Nebraska (more videos) Monitoring a Cluster of Computers as a school of fish (U. Nebraska) The evolution of Hadoop (Code-Swarm) Lecture Notes A solution for Project 2 has been posted!	Hadoop Tutorial from Yahoo Developer Network (YDN)
Week 12	Tuesday Presentation of Above the Clouds: A Berkeley View of Cloud Computing, Part I Signing on to Amazon AWS Uploading data to AWS and counting words Word-counting using Streaming Python on AWS Costs of maintaining a Hadoop cluster on AWS Thursday Continuation of the AWS labs Starting an EC2 instance on AWS Lecture Notes
Week 13	Tuesday Presentation of Above the Clouds: A Berkeley View of Cloud Computing, Part II Problem of the day: discussion Work on projects Thursday Presentation of Inside the Mind of Google wrap up Lecture Notes

Selected Solutions for papers, homework, or projects

• Amdahl's Law in the Multicore Era summary
• Selected Solutions for Project 1