CSC352 2017 DT's Notes
--D. Thiebaut (talk) 13:25, 14 November 2016 (EST)
<onlydft>
Contents
2013
Page with Public & Private Class Notes
- Go to this page for the 2013 Class Notes
TOC:
- 1 Resources 2013
- 1.1 Rocco's Presentation 10/10/13
- 1.2 Hadoop
- 1.3 On-Line
- 1.4 Papers
- 1.5 Art
- 1.6 Some good references
- 2 Misc. Topics
- 3 XSEDE.ORG
- 4 Update 2015: Downloading images to Hadoop0
- 5 Downloading All Wikipedia Images
- 6 Download the page statistics
- 6.1 Links of Interest
- 7 Resources 2010
- 8 Map-Reduce/Hadoop
- 8.1 Options for Setup
- 8.1.1 Xen Live CD
- 8.1.2 Setting up Hadoop using VmWare
- 8.2 Setting Up Hadoop and Eclipse on the Mac
- 8.2.1 Install Hadoop
- 8.2.2 Verify configuration of Hadoop
- 8.3 Setting up Eclipse for Hadoop
- 8.3.1 Map-Reduce Locations
- 8.3.2 DFS Locations
- 8.4 Create a new project with Eclipse
- 8.4.1 Project
- 8.5 Map/Reduce driver class
- 8.5.1 Running the Project
- 8.1 Options for Setup
- 9 WordCount Example on Eclipse on Mac
- 9.1 Mapper
- 9.2 Reducer
- 9.3 Driver
- 9.4 Run WordCount Project
- 10 Notes on doing example in Yahoo Tutorial, Module 2
2013 Private Notes Page
Threads
Programs
- NQueens.py
- threadedNQueens.py
- classExample1.py
- classExample2.py
- classExample3.py
- serialPing.py
- threadedPing.py
- searchKeywordsRetrieveEtexts.py
- threadedSearchKeywordRetrieveEtexts.py
Setting up documents and swish-e
(Note: there are 2 other alternatives: sphinx and zend-lucene. Sphinx requires data in xml form or in mysql database)
- all operations on xgridmac
- downloaded & installed swish-e. Install dir is ~thiebaut/research/swish-e/
- downloaded & installed www.etext.org in http://xgridmac.dyndns.org/~thiebaut/www_etext_org/
- swish-e index stored in ~thiebaut/Site/swish-e
- added swishe.php in ~thiebaut/Site/swish-e/
- test:
cd
cd Site/swish-e
php swishe.php search=love
...
<br>
<br>rank: 20
<br>score: 809
<br>url: http://xgridmac.dyndns.org/~thiebaut/www_etext_org/Religious_357/Polyamory/Keys2LovingUnity.html
<br>link: <a href="http://xgridmac.dyndns.org/~thiebaut/www_etext_org/Religious_357/Polyamory/Keys2LovingUnity.html">link</a>
<br>file: Keys2LovingUnity.html
<br>offset: 47813
<br>
- test on Web at url http://xgridmac.dyndns.org/~thiebaut/swish-e/swishe.php?search=love
- test with delay: http://xgridmac.dyndns.org/~thiebaut/swish-e/swishe.php?delay=20&search='local%20government'
Where delay is number of 1/10s of a second to wait. This is a bound as the true delay is random between 0.1 sec and the integer specified times 1/10 seconds.)
Project
- Project 1
- Threading in Python: given two lists of keywords, List1 and List2, retrieve docs from a site (xgridmac.dyndns.org, yahoo, google) that respond/match List1. Filter the docs received and keep only those that contain most of the words in List2.
- Project 2
- XGrid: process a gzip xml dump of wikipedia and break it up into individual pages (9 million or so of them)!
- Project 3
- Map-Reduce: process wikipedia pages and create an index of words and their associated categories
Papers
Notes on a View from Berkeley paper
2017
Ideas
- Latex still important
- last 2 weeks presentations: 10 minutes each, on a subject that we didn't cover. Need 10-min presentation plus 2 page prospectus.
- GPU
- Deeplearning
- Top 500. Why, what, what do we learn, lessons?
- CUDA
- OpenMP
- Debugging parallel programming
- Tensorflow
- Vampir: Trace Analyzer tool
- TotalView: Debugger
- C/C++ tutorial. Still good
- Optimization options in C -O, -O2, -O3
Resources
Spark
- Apache Spark: A Unified Engine for Big Data Processing, Matei Zaharia et al (pdf)
Papers
- Microsoft bets future chips reprogram fly, Wired.com. (pdf)
- Comments on Microsoft bets future chips reprogram fly, Bob Burger, FPGA
<onlydft>
