Difference between revisions of "CSC352 Project 1"
(→Introduction) |
(→Introduction) |
||
| Line 12: | Line 12: | ||
This project is an extension of Problem 2 of [[CSC352 Homework 2 | Homework #2]], but with a few differences. | This project is an extension of Problem 2 of [[CSC352 Homework 2 | Homework #2]], but with a few differences. | ||
| − | # You need to implement both the multiprocessing and threaded versions of the program (you can use the solution | + | # You need to implement both the multiprocessing and threaded versions of the program (you can use the solution program(s) that will be made available as soon as Homework 2 is graded). |
# The program will analyze each document received and will remove all the [http://en.wikipedia.org/wiki/Stop_words stop words], then compute the frequency of occurrence of each word and rank the words by frequency, from most frequent to least frequent. | # The program will analyze each document received and will remove all the [http://en.wikipedia.org/wiki/Stop_words stop words], then compute the frequency of occurrence of each word and rank the words by frequency, from most frequent to least frequent. | ||
# The program will assign a score to the document that is the inverse of the ranking of the search term in the list of ordered words. For example, if you search for "match" and the list of most frequent words is [ "document", "perform", "english", "text", "match", "score",...], then the score of the document will be 1/5, since "match" is in fifth position in the list. | # The program will assign a score to the document that is the inverse of the ranking of the search term in the list of ordered words. For example, if you search for "match" and the list of most frequent words is [ "document", "perform", "english", "text", "match", "score",...], then the score of the document will be 1/5, since "match" is in fifth position in the list. | ||
Revision as of 09:25, 13 February 2010
This project is due TBA. You can work individually or in pairs. If you work in pairs, you have to spend the majority of the time working together on the programming and analysis.
Under Construction! Not released officially!
Introduction
This project is an extension of Problem 2 of Homework #2, but with a few differences.
- You need to implement both the multiprocessing and threaded versions of the program (you can use the solution program(s) that will be made available as soon as Homework 2 is graded).
- The program will analyze each document received and will remove all the stop words, then compute the frequency of occurrence of each word and rank the words by frequency, from most frequent to least frequent.
- The program will assign a score to the document that is the inverse of the ranking of the search term in the list of ordered words. For example, if you search for "match" and the list of most frequent words is [ "document", "perform", "english", "text", "match", "score",...], then the score of the document will be 1/5, since "match" is in fifth position in the list.
- The program will sort the documents by decreasing scores.
- Your program will output the documents in way similar to that of Homework 2 (with a context around the search term), and will indicate the score of each document.
- You are free to decide how to format the output. It should show at least
- the search term
- the url of each document
- the score of each document
- the context for the search term in each document
Performance Analysis
- Run several experiments where you are going to measure the average and max number of searches performed per unit of time for your two implementations.
- Make sure that whatever you ask the multiprocessing version to do, you will need the threaded version to do as well.
- Use scripts to help you run the experiment and gather the data.
- Report the results, and make sure you specify the conditions of your experiments (the type of computer you were using, the time of day, etc.).
- Comment on the results and, if possible, explain your understanding for why you may discover that one performs better than the other, or why both methods perform similarly.
Additional Information
- Word Frequency using Python
- Use Python to Detect the Most Frequent Words in a File
- List of English Stop Words
- What is a Stop Word?
