Indexing Audit

- Perform the site:yoursite.com command in the Google search box or the url:yoursite.com command in the Bing search box.
If pages are returned in the results, it means Bing or Google bots have managed to crawl and to index your site recently (or in the past).
- Have you implemented a robots.txt file?

This is the first file web crawlers visit on a website. If they cannot find it, they (most often) stop crawling the website. Implement a robots.txt file if you have not done so.

- Have you implemented a sitemap.xml?

If not, implement one and make sure it is registered in your robots.txt. Try to make it as exhaustive as possible and wait until web crawlers process it. You may submit you sitemap to search engines to accelerate the process too.

- Have you registered your site into Google Search Console and Bing Search Console?

Check for crawling and robots.txt issues in each Search Console.

- Are all your sitemaps registered in your robots.txt file?

Make sure all the sitemaps created for your website are registered in your robots.txt and wait for cralers to reprocess them. You submit your sitemaps to search engines to accelerate this process too.

- Are your sitemaps not located at the root of your website?

Officially, sitemaps which are not located at the root can only contain urls of pages located in the corresponding directory (or sub-directories). However, many search engines are tolerant and don't enforce this restriction.

Putting all sitemaps at the root is a safe practice.

- Are your sitemaps validated?

An issue in a sitemap's structure can prevent web crawlers from processing it properly. Check for error message in Google Search Console or Bing Webmaster Tools if you have registered your site or use an online validator.

- Does any of your sitemaps contain more than 50 000 entries or is greater than 10 MB (unzipped)?

If so, split your sitemaps until each has at most 50 000 entries and is lesser than 10 MB in size.

- Are your site pages or Javascript files blocked by your robots.txt file?

Restore access to your pages and Javascript files in your robots.txt file and wait for web crawlers to revisit your site.

- Did you disable URLs in Bing Webmaster tools?

When you disable URLs in Bing webmaster tools, these are not indexed. Enable them again.

- Have you posted several thousands (or more) page to your site recently?

For new websites, Google crawlers often start by indexing a couple of pages to see how they rank and how users react to them. Later, they can index more pages.

For established websites, if crawlers don't index pages, it is often an indication they don't find them interesting enough for their users.

- Is your site slow or very slow?

Web crawlers can give up on websites which are too slow to respond to their queries. Search engines don't want to expose their users to such a bad experience. Solve those performance issue, for example with a bigger server and/or by using a CDN.

- Have your configured URL parameters in Google Search Console?

Make sure URL parameters have been configured properly. Otherwise, there is a risk of losing pages from the index.

- Have you set proper DNS servers for your domain name and have you set proper CNAME records to redirect www to non-www domain names (or vice-versa)?

Implement a redirect from www to non-www (or vice-versa).

- Are you suffering from a Google manual action?

You can verify this in Google Search Console. Some manual penalties result in the removal of websites from the index.

- Did you acquire the domain name from a previous owner?

Register the domain on Google Search Console and check whether there is an existing manual action attached to it.