A high-level description of the implementation, with particular emphasis on the design decisions related to data structures and algorithms
Our crawler works as follows: URLs to crawl, URLs that have been crawl, and dissallowed domains are saved in HashSets. We used HashSets because they enforce uniqueness (we do not want to crawl the same URL multiple times) and operate in O(1) time for adding, removing, and contains operations. Seed URLs are added either by command line arguments or from a specified file. The seeds are loaded into the HashMap. From there, until we run out of URLs to crawl or hit our page crawl limit, we crawl each page one at a time. The crawl algorithm works as follows: the next URL is removed from the HashSet. We then check if its domain is known to allow crawling. If the domain has not been crawled before, we check the website's robots.txt file to see what pages are allowed to be crawled and save those preferences so we can respect them later. If the has been crawled before, then then we simply check if the URL is in the allowed Domains HashSet. If it is not allowed, the URL is skipped. If it is allowed, we then update the wait delay and write to a file as much data as allowed per website (default 1KB). After a page is finished being crawled (and we are below the crawled page limit), the crawler fetches the next page and the process repeats.
### Implemented Features
- Provide real-time status and statistics ( e.g., what URL is being processed, rate of processing, size of storage, etc.) feedback for the crawler. [10%]
