Loading UpdateEdits.md +12 −7 Original line number Diff line number Diff line # Alex - Accepted Feedback: - Removed stack traces from error messages for the scrawler - Added displaying number of pages crawled thus far regardless of --stats argument - Blacklisted assets.tumblr.com Loading @@ -6,7 +7,11 @@ - Added to INSTALL.md: - Added some common websites to try crawling - Added some instructions for seed URLs - Rejected Feedback: - n/a - Other Changes: - Emphasized the reduced amount of URLs available when running without --xl - Crawler now fetches a new URL without incrementing the crawl count if the URL contains a file extension for a image, video, audio, or PDF # Reza Loading Loading
UpdateEdits.md +12 −7 Original line number Diff line number Diff line # Alex - Accepted Feedback: - Removed stack traces from error messages for the scrawler - Added displaying number of pages crawled thus far regardless of --stats argument - Blacklisted assets.tumblr.com Loading @@ -6,7 +7,11 @@ - Added to INSTALL.md: - Added some common websites to try crawling - Added some instructions for seed URLs - Rejected Feedback: - n/a - Other Changes: - Emphasized the reduced amount of URLs available when running without --xl - Crawler now fetches a new URL without incrementing the crawl count if the URL contains a file extension for a image, video, audio, or PDF # Reza Loading