CSC352 Project 1

This project is due TBA. You can work individually or in pairs. If you work in pairs, you have to spend the majority of the time working together on the programming and analysis.

Under Construction! Not released officially!

Introduction

This project is an extension of Problem 2 of Homework #2, but with a few differences.

1. You need to implement both the multiprocessing and threaded versions of the program (you can use the solution program(s) that will be made available as soon as Homework 2 is graded).
2. The program will analyze each document received and will remove all the stop words, then compute the frequency of occurrence of each word and rank the words by frequency, from most frequent to least frequent.
3. The program will assign a score to the document that is the inverse of the ranking of the search term in the list of ordered words. For example, if you search for "match" and the list of most frequent words is [ "document", "perform", "english", "text", "match", "score",...], then the score of the document will be 1/5, since "match" is in fifth position in the list.
4. The program will sort the documents by decreasing scores.
5. Your program will output the documents in a way similar to that of Homework 2 (with a context around the search term), and will indicate the score of each document.
6. You are free to decide how to format the output. It should show at least
   • the search term
   • the url of each document
   • the score of each document
   • the context for the search term in each document

Performance Analysis

• Run several experiments where you are going to measure the average and max number of searches performed per unit of time for your two implementations.

• Make sure that whatever you ask the multiprocessing version to do, you will need the threaded version to do as well.

• Use scripts to help you run the experiments and gather the data.

• Report the results, and make sure you specify the conditions of your experiments (the type of computer you were using, the number of cores, the speed of the processor, the time of day, etc.).

• If possible (not required, but recommended), run your experiments on different architectures (a single core and a dual-core, for example)

• Comment on the results, and explain, as best you can, why one out-performs the other, or why both methods perform similarly.

Submission

Store your programs, comments, and analysis in a pdf, and submit it as follows:

  submit project1  project1.pdf

Additional Information

• Word Frequency using Python
• Use Python to Detect the Most Frequent Words in a File
• List of English Stop Words
• What is a Stop Word?
• How to get CPU information on a Linux Machine
• CPU Id, a utility to find the CPU info on Windows machines