How to use Website Crawler?

 

1. Go to Dashboard

checklist-1.png

After entering to your account, go to Websites Dashboard

 

2. Enter your domain

checklist-2.png

Enter your domain to the form with http/https protocol. Use www only if it is preferred version.

 

3. Audit has started

checklist-3.png

Right after clicking Start, you can see how the website crawling goes. 

 

4. Wait some time

checklist-4.png

The time of crawling depends on the amount of URLs your website has.

 

5. Review non-200 URLs

checklist-6.png

After the audit is finished, you can begin to work with collected results. It's important to understand that we rank parameters in the left sidebar by their importance for SEO audit. Fix the errors step-by-step.

For example, we have three 301 redirects. With Anchors feature we can see which pages link to URL with 301 http status code. It helps to replace such links to the links with 200 status code. The same algorithm works for pages with 4xx and 5xx status codes.

 

6. Use detailed reports for specific URLs

checklist-page-info.png

Just click on URL you want to research deeper and you'll find detailed report that includes data from both tools: On Page Audit and Website Audit.

 

7. Generate sitemap.xml file

checklist-export-sitemap.png

Download XML Sitemap file in 1 click for each website you have crawled. Generator is developed strictly following to Google guidelines. So, after downloading the file you can add it to Google Search Console immediately.

 

8. Analyze Page Weight

checklist-7.png

After fixing the most critical errors, you can improve your link structure. Page Weight parameter is calculated due to Google PageRank formula.

For example, we see that 3 pages with the biggest Page Weight are account pages. It's not normal. It's better to change link structure in such a way, that your most important landing pages will have the biggest Page Weight.

 

9. Visualize website structure

checklist-website-visualizer.png

There are different ways to visualize website structure. You may already use some tools and most of them focused on the structure itself. We created a visualizer to improve SEO & traffic performance first. The most performed site structures went from website siloing. Bruce Clay explains the importance of silo structure very clear:

"Siloing your site improves the relevancy of your web pages for your desired search terms. Links are strategically used within theme-focused sections of the site — and PageRank (i.e., linking value) gets passed between closely related pages and landing pages, reinforcing the themes."

Our visualizer builds the tree of your website depends on internal linking structure (it's called virtual siloing). Read this Clay's guide, visualize website structure with our tool and explore what changes you should make in your internal linking to improve rankings.

 

10. Delegate work on the website

checklist-share-feature.png

If you want to fix errors with the help of freelancer or your subordinate, just use Export in CSV or Share features.

 

11. Make sure issues were eliminated

checklist-5.png

After fixing the errors, recrawl domain and make sure that issues were eliminated. 

Did this answer your question? Thanks for the feedback There was a problem submitting your feedback. Please try again later.

Still need help? Contact Us Contact Us