How to use Website Crawler?
1. Go to Dashboard
After entering to your account, go to Websites Dashboard.
2. Enter your domain
Enter your domain to the form with http/https protocol. Use www only if it is preferred version.
3. Audit has started
Right after clicking Start, you can see how the website crawling goes.
4. Wait some time
The time of crawling depends on the amount of URLs your website has.
5. Review non-200 URLs
After the audit is finished, you can begin to work with collected results. It's important to understand that we rank parameters in the left sidebar by their importance for SEO audit. Fix the errors step-by-step.
For example, we have three 301 redirects. With Anchors feature we can see which pages link to URL with 301 http status code. It helps to replace such links to the links with 200 status code. The same algorithm works for pages with 4xx and 5xx status codes.
6. Use detailed reports for specific URLs
Just click on URL you want to research deeper and you'll find detailed report that includes data from both tools: On Page Audit and Website Audit.
7. Generate sitemap.xml file
Download XML Sitemap file in 1 click for each website you have crawled. Generator is developed strictly following to Google guidelines. So, after downloading the file you can add it to Google Search Console immediately.
8. Analyze Page Weight
After fixing the most critical errors, you can improve your link structure. Page Weight parameter is calculated due to Google PageRank formula.
For example, we see that 3 pages with the biggest Page Weight are account pages. It's not normal. It's better to change link structure in such a way, that your most important landing pages will have the biggest Page Weight.
9. Visualize website structure
There are different ways to visualize website structure. You may already use some tools and most of them focused on the structure itself. We created a visualizer to improve SEO & traffic performance first. The most performed site structures went from website siloing. Bruce Clay explains the importance of silo structure very clear:
"Siloing your site improves the relevancy of your web pages for your desired search terms. Links are strategically used within theme-focused sections of the site — and PageRank (i.e., linking value) gets passed between closely related pages and landing pages, reinforcing the themes."
Our visualizer builds the tree of your website depends on internal linking structure (it's called virtual siloing). Read this Clay's guide, visualize website structure with our tool and explore what changes you should make in your internal linking to improve rankings.
10. Delegate work on the website
If you want to fix errors with the help of freelancer or your subordinate, just use Export in CSV or Share features.
11. Make sure issues were eliminated
After fixing the errors, recrawl domain and make sure that issues were eliminated.