Yandex.Metrica

Google’s Webmaster Tools have undergone quite a lot of change over the past 6 years. So much change that they’re now known as ‘Google Search Console’. With its growth, Google has released a lot of features, such as the ‘Search Analytics’ and ‘Links to your site’, which weren’t present in the old Webmaster Tools.

All these changes are very important for the development of SEO and Google devotes quite a few resources to upgrading their systems. This has also resulted in better management of errors. Many a times, a lot of webmaster/SEO errors, or the ‘crawl’ errors prop up.

SEO errors, or crawl errors are something that normally can’t be avoided and it isn’t necessary that they’ll negatively affect your SEO performance. That being said, you should still tackle them. If you develop practice preventive maintenance and have good habits, doing check on crawl errors every week, you will be able to keep them under control. Ignore them though, and things will get pretty bad – pretty quick. You should remember that less SEO errors encountered by the google bot, the less website and server errors your users will get.

So, without any further ado – let us dive in as to how to tackle these pesky little blighters.

The layout of crawl error :

  1. If you notice, you’ll see that your Search Console is divided into two parts, namely, Site Errors and URL errors. It is very helpful that errors are categorized as such. This provides you with a clear view of errors at the site level, or at the page level. If I was to give them severity rating, site-level errors are very severe as they may damage your website’s usability.

    You can access the crawl errors from your dashboard. Your dashboard have quick links and also provides you with a preview to your website. Here you will see your most important tools –

    1. Crawl Errors

    2. Search Analytics

    3. Sitemaps

    check website for seo errors

Pro Tip: Set all your SEO errors as fixed. Right now, this may seem a bit bizarre – but believe me, it will actually help you tackle any errors you get in a very structured manner. I’ve said this before and I’ll say this again – go through your crawl errors weekly. You will be able to manage them better and if you see some errors that weren’t present the previous week, it’ll narrow your results as to the source.

do 404 error hurt seo

Dealing with Site Errors

The Site Errors are the errors that affect your website as a whole. These are pretty high-level and will affect your entire website. Thus, I won’t recommend skipping/ignoring these.

404 errors seo

  1. DNS Errors

DNS (Domain Name System) errors are simply a result of GoogleBot’s inability to connect with your domain. This usually occurs because of a DNS timeout or a DNS lookup issue.

Since this is the first step in accessing your website, a DNS issue is extremely important to solve. No connection means no traffic, which is not something you’d want!

Fixing DNS Errors

  1. You should use the Fetch as Google tool and fix problems related to your site’s content as well as discoverability in search results.

  2. Check with your DNS provider.

  3. Ensure that your pages return a response code ‘500’ if there’s a server error or a code ‘404’ if the error is caused by a faulty URL. (More on this later)

Server Errors (500): 

If your server takes too long to respond, resulting in the time out of a request, a server error (500) will be triggered. GoogleBot will only wait for a while before giving up.
Server errors are different from DNS errors because of the fact that Google is actually able to connect to your site, but your server takes too long to respond to any requests.

Fixing Server Errors:

  1. Ensure that your server can deal with a burst of traffic!

  2. Understand what type of server error you’re getting. This can again be done with the help of the Fetch as Google tool.

Dealing with URL Errors

Before we move onto fixing these, let us understand what URL errors actually are. Essentially, URL errors are errors that affect only certain pages of your site, not your site as a whole.

In layman terms, it means that upon crawling the URL, the Google-Bot successfully resolved your DNS and connect to your server. It was then able to fetch and read your robots.txt file as well. After this though, something went wrong.

This ‘something-went-wrong’ part of the process has been categorized into different types of URL errors. This can be seen as a type of SEO errors and it’ll hurt your site’s ranking.

check website for seo error

Over view of URL Errors 

  1. A soft 404 error

Usually, when the page requested by a visitor on your site doesn’t exist, a 404 (not found) error is returned by the web server. 404 is simply an HTTP response code and it tell the search engine and both browsers that a page doesn’t exist. This would mean that the content of that page will not be indexed or crawled by search engines.

If your server returns a page, using an URL that does not exist on your website, a soft 404 SEO error is encountered. Usually, this occurs because your server handles these faulty, non-existent URLs as ‘OK’. They then redirect the visitor to a valid page, such as your home page or your custom 404 page.

seo error

Soft 404 errors can also show up if you are 301 redirecting your pages to non-related pages, say – your home page. From experience, I’ve seen that Google can interpret large redirects to a single page, even if it’s truly a 301 redirect as soft 404 errors.

Fixing Soft 404:

  1. The page doesn’t exist anymore

Should your page truly be gone, you should configure your server to always return either a 404 (Not found) response code, or a 401(Gone) one. Many websites, even the one in the above picture tackle this by creating a custom 404 page and it returns links of pages on their website.

Furthermore, you should ensure that the server head doesn’t respond as ‘200’, but rather as 404 or 410. You can even 301 redirect each of your old pages to other pages on your site that are relevant.That being said, I recommend not redirecting a lot of ‘dead-pages’ to your home page. You should simply 404 them.

  1. Live page, but not a soft 404 error

You must ensure the presence of an appropriate amount of content on your pages. Only having thin content, more than often not – will trigger a soft 404. You should also ensure that your 200 response code serving page doesn’t appear to represent a 404 page.

I really find soft 404 errors to be strange. You should really ensure that your website pages aren’t triggering a soft 404 as they often cause a lot of confusion.

2. 404 errors

The classic 404 is most probably the most common and the most misunderstood crawl error. This SEO error will be triggered if the page GoogleBot tried to crawl from the URL doesn’t really exist on your website.
Google says that most of the 404 errors will not affect your website’s ranking – and that you can ignore them safely. However, I recommend that you do not ignore them if your website’s crucial pages return a 404 response code.

Usually, 404 errors are a result of typos, misconfigurations in site or even due to Google’s increased efforts for recognizing and crawling any links that are present in embedded content.

Fixing 404 errors

  1. Classic 404

You will get to know the place from where every 404 errors encountered by the GoogleBot is linked from. All you need to do is to open the crawl report and click on the URL you wish to edit.


seo 404 error
B. 404 errors from an external link

If an external website links to your website using a false URL, simply use the 301 redirect.

C. 404 errors from internal links or sitemap entries

If one of your own pages returns a 404 error, you should fix the entry on the sitemap. You can 301 redirect the 404 URL to its correct destination and make it vanish from your crawl reports. 🙂

3. Access Denied

Google discovers content on your website. It does this by following links from one page to another. Naturally, GoogleBot needs access to your pages to crawl them. Access Denied errors are usually triggered because of the fact that GoogleBot could not access a URL on your website.

This happens because your website may require users to log in or authenticate their identity. You host could be blocking Google from accessing your site as well.

 

Fix Access Denied errors

  • You should test whether your robots.txt is working and isn’t blocking Google.

  • Remove the login from pages that you want Google to crawl. This might be a popup prompt or any in-page.

  • You should use the Fetch as Google tool and fix problems related to your site’s content as well as discoverability in search results. 

For ease, here is a simple table with quick fixes to all the crawl errors I’ve ever faced.

 

URL ERROR

MEANING

FIX

Soft 404

A page that is 404 returned 200

301 redirect or 404 status

404

Requested page doesn’t exist

Fix URL or redirect

403

Access Denied

Remove the blocking program

301

Not followed. Your redirects don’t work

Fix your redirects!

400/405/406

Grouped errors, treated as 404

Fix the URL or redirect

Flash Content (Smartphone)

Lots of pages, having flash that won’t work on smartphones.

Get rid of flash

DNS Error

GoogleBot cannot connect to the DNS server

Fix your DNS issues

So, there you have it, guys. The ultimate guide to fix and counter any webmaster errors you may ever encounter.

Sudheer Varma

About Sudheer Varma

Building relationships by solving reporting challenges of ad agencies around the globe. Digital marketing is the way forward, and our team is using this wave for efficient selling. I believe sales = experience+solving a challenge.