@@ -34,45 +34,35 @@ The correction at this point is limited to the state machine complexity, but it
### Implemented Features
- Provide real-time status and statistics ( e.g., what URL is being processed, rate of processing, size of storage, etc.) feedback for the crawler. [10%]
- Implemented by calculating statistics as the crawler crawls each page. We report the length of the page processed in bytes, the total amount of data crawled thus far, the number of links extracted from the page, the number of pages crawled, the number of URLs left available to crawl, the current crawling rate in pages per second, the current crawling rate in links extracted per second, and the current crawling rate in bytes per second. Usage: use "--stats" argument for the crawler.
- Implemented by calculating statistics as the crawler crawls each page. We report the length of the page processed in bytes, the total amount of data crawled thus far, the number of links extracted from the page, the number of pages crawled, the number of URLs left available to crawl, the current crawling rate in pages per second, the current crawling rate in links extracted per second, and the current crawling rate in bytes per second. Usage: use `--stats` argument for the crawler.
- Provide a list of reasonable corrections to a suspicious text, ranked in order of how different they are from the original text. [15%]
- There are four possible corrections. First correction to typos will come, then correction that include remove/insertion or changing a word will come. Since corrections are fine-tuned each individual is improtant. Also, since a typo correction will lead to a big change it comes first.
- There are four possible corrections. First correction to typos will come, then correction that include remove/insertion or changing a word will come. Since corrections are fine-tuned each individual is important. Also, since a typo correction will lead to a big change it comes first.
- The implementation is just by dumping the changes into `correction_details.txt`.
- Graphical User Interface that highlights suspicious and non-suspicious textual elements in a given text. [15%]
- This interface is interactive and choose each phrases with 3 words and check it the confidence for it and color it based on it on the specterum.
- This interface is interactive and choose each phrases with 3 words and check it the confidence for it and color it based on it on the spectrum.
- The implementation of this part was with JTextArea of java.awt.
- Extend your crawler to crawling social media posts of some large network ( e.g., Twitter, Facebook, LinkedIn, Truth, MeWe…). [15%]
- Implemented by adding a social media platform as a seed to our crawler. We are using Tumblr as our social media platform. It is considered a major platform with more than 500 million monthly users. Choosing a social media platform was difficult since most platforms no longer allow crawling as a safeguard against their data being used to train LLMs. After careful examination of possible sites, we decided that Tumblr was the best choice that still allowed crawlers. Usage: use "--social" argument for the crawler.
- Implemented by adding a social media platform as a seed to our crawler. We are using Tumblr as our social media platform. It is considered a major platform with more than 500 million monthly users. Choosing a social media platform was difficult since most platforms no longer allow crawling as a safeguard against their data being used to train LLMs. After careful examination of possible sites, we decided that Tumblr was the best choice that still allowed crawlers. Usage: use `--social` argument for the crawler.
- Provide a graphical human feedback system for deciding among possible phrase corrections, with feedback into the suspicion levels reported by the system. [15%]
- This feature will run a GUI that represent the possible correction as a button where user can accept or reject them. By appling the changes a new set of suggestions will show. User can also go to the next sentence by another button.
- For the implementation we use a GUI interface and implement it in the corrector so that we can interact with GUI and get and pass the information.
- Develop and Android client for your checker. [15%]
- For the most part, the Android client of the checker is interfacing with the Checker CLI tool to produce its output in a pop-up window on the app. To do this, the Checker tool was modified so that it did not write the checked result to a JSON file (and instead kept it as a JSON string) as well as not require CLI arguments for the script to function. In addition, helper class functions were used to display the output to the user in the app.
- Extend your system to a language in which none of the team members have fluency. [15% per language, up to 3] - 1x
- We added the ability to crawl Dutch websites by adding a Dutch webpage seed to our crawler. Usage: use "--dutchSeed" argument for the crawler. Additional Dutch websites to crawl can be specified by adding their URLs to a file and using the "--file [filename.txt]" argument for the crawler. Also, for the checker and corrector things are similar to english but you need to usee --dutch option.
- We added the ability to crawl Dutch websites by adding a Dutch webpage seed to our crawler. Usage: use `--dutchSeed` argument for the crawler. Additional Dutch websites to crawl can be specified by adding their URLs to a file and using the `--file [filename.txt]` argument for the crawler. Also, for the checker and corrector things are similar to english but you need to usee `--dutch` option.
- Provide a reasonable translation from English to another language based on common language structures. [30%]
- We added the ability to translation from Dutch to English and from English to Dutch. We accomplished this by crawling a Dutch to English online dictionary and translating the words literally. Usage: use ""--dutchDict" argument with crawler to re-crawl the dictionary. For translation we used crawled dictionary and translate the data on both side."--translateToEnglish" and "--translateToDutch" in corrector will do this.
#### Changed Features from Initial Defense Report
##### Added
- Feature 1
##### Removed
- Feature 1
- We added the ability to translation from Dutch to English and from English to Dutch. We accomplished this by crawling a Dutch to English online dictionary and translating the words literally. Usage: use `--dutchDict` argument with crawler to re-crawl the dictionary. For translation we used crawled dictionary and translate the data on both side.`--translateToEnglish` and `--translateToDutch` in corrector will do this.
Webcrawler, Data Parser, Webcrawler stats, Webcrawler social media, Extend system to other language (partial), Provide a reasonable translation (partial) by Alex Melnick
Android App by Michael Harkess
Data Parser, Android App by Michael Harkess
All of the package and module in CheckerCorrector including: Language checker and corrector with all of the GUI features for them, including feature 3 and 5 and other features like giving user feadback on tokens, and the list of possible corrections, The extention to dutch and translation for dutch on checker/corrector the by Reza Sajjadi
@@ -80,6 +70,7 @@ This work breakdown is signed by Everyone:
Seyed Reza Sajjadinasab
Alexander Ross Melnick
Michael Harkess
### Known Issues/Bugs
#### Android App
- If a dialog box pops up and the device/screen is rotated, then the app will unexpectedly crash
