Website Crawler lets you crawl an entire website or pages until reaching a specific level and provides helpful information about each crawled page.

The results gathered following the crawl can be exported as Google Sitemap(XML) or Yahoo Sitemap(TXT). Additionally, you can add selected URLs to the "Selected URL" list and from this list you can add them as project URLs, as referrers URLs or export them to a file.

For each crawled page, Website Crawler provides it's page rank, meta information (title, description, robots, keywords, author), the number of outbound and onsite links, HTTP response and the size of the respective page.

Website Crawler can help you not only crawl your website and export the results as a sitemap for the main search engines, but also analyze each crawled URL and see if the page meets your optimization requirements. In this way your optimization and marketing efforts will have a positive impact for your website rankings.

NoteThis feature is only available in the Enterprise version.