Added some additional options. Biggest addition is the option to crawl 1MB... See merge request ec504/ec504_projects/group6!15