A high-level description of the implementation, with particular emphasis on the design decisions related to data structures and algorithms
## Description of Implementation
### How it Works
### Crawler
Our crawler works as follows: URLs to crawl, URLs that have been crawl, and dissallowed domains are saved in HashSets. We used HashSets because they enforce uniqueness (we do not want to crawl the same URL multiple times) and operate in O(1) time for adding, removing, and contains operations. Seed URLs are added either by command line arguments or from a specified file. The seeds are loaded into the HashMap. From there, until we run out of URLs to crawl or hit our page crawl limit, we crawl each page one at a time. The crawl algorithm works as follows: the next URL is removed from the HashSet. We then check if its domain is known to allow crawling. If the domain has not been crawled before, we check the website's robots.txt file to see what pages are allowed to be crawled and save those preferences so we can respect them later. If the has been crawled before, then then we simply check if the URL is in the allowed Domains HashSet. If it is not allowed, the URL is skipped. If it is allowed, we then update the wait delay and write to a file as much data as allowed per website (default 1KB). After a page is finished being crawled (and we are below the crawled page limit), the crawler fetches the next page and the process repeats.
#### Checker
Our checker is using two different methods to assign confidence points. The first method uses a State Machine and the second is using an n-Grams inspired implementation. The final score will be a weighted sum of these scores provided by two different methods.
##### State Machine Checker
In order to have a good working State machine we need to first update the grammar and roles of each word manually. This step will be automatized to some extent in the next milestones. Tokens are provided in `CheckerCorrector/SQLite/mydatabase.db`, and also the basic graph provided by `CheckerCorrector/DirectedGraph/BasicGraph.java`. A sentence will first go through a typo checker and get updated if needed (it will also affect the confidence score). Then the sentence will be tokenized, and using the provided graph it will check whether the sentence is following the correct format or not, for each miss on any edge of the graph a penalty will be added to the confidence score.
The typo corrector is calculating the distance of a word to dictionary(supposly provided by crawler). Here we used the small dictionary of homewrok2, and replace the misspelled word with closest distant word with some condition.
##### n-Grams checker
This checker used the crawled data and gave a score by summing up all the n_grams probabilities of phrases in a sentence. In order to store the data of the crawled data we used SHA-256 hash and store the result in CheckerCorrector/SQLite/hash_database.db.
#### Proof of Effictiveness
In order to show the effictiveness of our tool we used ChatGBT to write a script that is using a third-party Python library to make the exact same score for each sentence. These can be found in CheckerCorrector/samples/ directory. Both Json generated by our tool and the third party are available.
#### Corrector
The current corrector uses the typo corrector which was used in the checker and also the state machine to suggest possible corrections. Corrections of the typo corrector are based on the most similar path through the state machine. In order to find a similar path we are first doing a DFS on the graph starting from the first token and storing all the possible paths. Then, based on the most similar path we decide whether we should change/delete/add a token and suggest a token with a similar role.
The correction at this point is limited to the state machine complexity, but it will be improved for the next milestone.
### Implemented Features
- Provide real-time status and statistics ( e.g., what URL is being processed, rate of processing, size of storage, etc.) feedback for the crawler. [10%]
@@ -55,26 +73,6 @@ Data Parser, Android App by Michael Harkess
Language checker and corrector by Reza Sajjadi
## Advanced Technical Information
### How it Works
#### Checker
Our checker is using two different methods to assign confidence points. The first method uses a State Machine and the second is using an n-Grams inspired implementation. The final score will be a weighted sum of these scores provided by two different methods.
##### State Machine Checker
In order to have a good working State machine we need to first update the grammar and roles of each word manually. This step will be automatized to some extent in the next milestones. Tokens are provided in `CheckerCorrector/SQLite/mydatabase.db`, and also the basic graph provided by `CheckerCorrector/DirectedGraph/BasicGraph.java`. A sentence will first go through a typo checker and get updated if needed (it will also affect the confidence score). Then the sentence will be tokenized, and using the provided graph it will check whether the sentence is following the correct format or not, for each miss on any edge of the graph a penalty will be added to the confidence score.
The typo corrector is calculating the distance of a word to dictionary(supposly provided by crawler). Here we used the small dictionary of homewrok2, and replace the misspelled word with closest distant word with some condition.
##### n-Grams checker
This checker used the crawled data and gave a score by summing up all the n_grams probabilities of phrases in a sentence. In order to store the data of the crawled data we used SHA-256 hash and store the result in CheckerCorrector/SQLite/hash_database.db.
#### Proof of Effictiveness
In order to show the effictiveness of our tool we used ChatGBT to write a script that is using a third-party Python library to make the exact same score for each sentence. These can be found in CheckerCorrector/samples/ directory. Both Json generated by our tool and the third party are available.
#### Corrector
The current corrector uses the typo corrector which was used in the checker and also the state machine to suggest possible corrections. Corrections of the typo corrector are based on the most similar path through the state machine. In order to find a similar path we are first doing a DFS on the graph starting from the first token and storing all the possible paths. Then, based on the most similar path we decide whether we should change/delete/add a token and suggest a token with a similar role.
The correction at this point is limited to the state machine complexity, but it will be improved for the next milestone.
### Known Issues/Bugs
#### Android App
- If a dialog box pops up and the device/screen is rotated, then the app will unexpectedly crash
