Google’s Webmaster Tools have undergone quite a lot of change over the past 6 years. So much change that they’re now known as ‘Google Search Console’. With its growth, Google has released a lot of features, such as the ‘Search Analytics’ and ‘Links to your site’, which weren’t present in the old Webmaster Tools.
All these changes are very important for the development of SEO and Google devotes quite a few resources to upgrading their systems. This has also resulted in better management of errors. Many a times, a lot of webmaster/SEO errors, or the ‘crawl’ errors prop up.
SEO errors, or crawl errors are something that normally can’t be avoided and it isn’t necessary that they’ll negatively affect your SEO performance. That being said, you should still tackle them. If you develop practice preventive maintenance and have good habits, doing check on crawl errors every week, you will be able to keep them under control. Ignore them though, and things will get pretty bad – pretty quick. You should remember that less SEO errors encountered by the google bot, the less website and server errors your users will get.
So, without any further ado – let us dive in as to how to tackle these pesky little blighters.
The layout of crawl error :
- If you notice, you’ll see that your Search Console is divided into two parts, namely, Site Errors and URL errors. It is very helpful that errors are categorized as such. This provides you with a clear view of errors at the site level, or at the page level. If I was to give them severity rating, site-level errors are very severe as they may damage your website’s usability.
- You can access the crawl errors from your dashboard. Your dashboard have quick links and also provides you with a preview to your website. Here you will see your most important tools –
- Crawl Errors
- Search Analytics
- Sitemaps
Pro Tip: Set all your SEO errors as fixed. Right now, this may seem a bit bizarre – but believe me, it will actually help you tackle any errors you get in a very structured manner. I’ve said this before and I’ll say this again – go through your crawl errors weekly. You will be able to manage them better and if you see some errors that weren’t present the previous week, it’ll narrow your results as to the source.
Dealing with Site Errors
The Site Errors are the errors that affect your website as a whole. These are pretty high-level and will affect your entire website. Thus, I won’t recommend skipping/ignoring these.
- DNS Errors
DNS (Domain Name System) errors are simply a result of GoogleBot’s inability to connect with your domain. This usually occurs because of a DNS timeout or a DNS lookup issue.
Since this is the first step in accessing your website, a DNS issue is extremely important to solve. No connection means no traffic, which is not something you’d want!
Fixing DNS Errors
- You should use the Fetch as Google tool and fix problems related to your site’s content as well as discoverability in search results.
- Check with your DNS provider.
- Ensure that your pages return a response code ‘500’ if there’s a server error or a code ‘404’ if the error is caused by a faulty URL. (More on this later)
Server Errors (500):
If your server takes too long to respond, resulting in the time out of a request, a server error (500) will be triggered. GoogleBot will only wait for a while before giving up.
Server errors are different from DNS errors because of the fact that Google is actually able to connect to your site, but your server takes too long to respond to any requests.
Fixing Server Errors:
- Ensure that your server can deal with a burst of traffic!
- Understand what type of server error you’re getting. This can again be done with the help of the Fetch as Google tool.
Dealing with URL Errors
Before we move onto fixing these, let us understand what URL errors actually are. Essentially, URL errors are errors that affect only certain pages of your site, not your site as a whole.
In layman terms, it means that upon crawling the URL, the Google-Bot successfully resolved your DNS and connect to your server. It was then able to fetch and read your robots.txt file as well. After this though, something went wrong.
This ‘something-went-wrong’ part of the process has been categorized into different types of URL errors. This can be seen as a type of SEO errors and it’ll hurt your site’s ranking.
Over view of URL Errors
- A soft 404 error
Usually, when the page requested by a visitor on your site doesn’t exist, a 404 (not found) error is returned by the web server. 404 is simply an HTTP response code and it tell the search engine and both browsers that a page doesn’t exist. This would mean that the content of that page will not be indexed or crawled by search engines.
If your server returns a page, using an URL that does not exist on your website, a soft 404 SEO error is encountered. Usually, this occurs because your server handles these faulty, non-existent URLs as ‘OK’. They then redirect the visitor to a valid page, such as your home page or your custom 404 page.
Soft 404 errors can also show up if you are 301 redirecting your pages to non-related pages, say – your home page. From experience, I’ve seen that Google can interpret large redirects to a single page, even if it’s truly a 301 redirect as soft 404 errors.
Fixing Soft 404:
- The page doesn’t exist anymore
Should your page truly be gone, you should configure your server to always return either a 404 (Not found) response code, or a 401(Gone) one. Many websites, even the one in the above picture tackle this by creating a custom 404 page and it returns links of pages on their website.
Furthermore, you should ensure that the server head doesn’t respond as ‘200’, but rather as 404 or 410. You can even 301 redirect each of your old pages to other pages on your site that are relevant.That being said, I recommend not redirecting a lot of ‘dead-pages’ to your home page. You should simply 404 them.
- Live page, but not a soft 404 error
You must ensure the presence of an appropriate amount of content on your pages. Only having thin content, more than often not – will trigger a soft 404. You should also ensure that your 200 response code serving page doesn’t appear to represent a 404 page.
I really find soft 404 errors to be strange. You should really ensure that your website pages aren’t triggering a soft 404 as they often cause a lot of confusion.
2. 404 errors
The classic 404 is most probably the most common and the most misunderstood crawl error. This SEO error will be triggered if the page GoogleBot tried to crawl from the URL doesn’t really exist on your website.
Google says that most of the 404 errors will not affect your website’s ranking – and that you can ignore them safely. However, I recommend that you do not ignore them if your website’s crucial pages return a 404 response code.
Usually, 404 errors are a result of typos, misconfigurations in site or even due to Google’s increased efforts for recognizing and crawling any links that are present in embedded content.
Fixing 404 errors
- Classic 404
You will get to know the place from where every 404 errors encountered by the GoogleBot is linked from. All you need to do is to open the crawl report and click on the URL you wish to edit.
B. 404 errors from an external link
If an external website links to your website using a false URL, simply use the 301 redirect.
C. 404 errors from internal links or sitemap entries
If one of your own pages returns a 404 error, you should fix the entry on the sitemap. You can 301 redirect the 404 URL to its correct destination and make it vanish from your crawl reports. 🙂
3. Access Denied
Google discovers content on your website. It does this by following links from one page to another. Naturally, GoogleBot needs access to your pages to crawl them. Access Denied errors are usually triggered because of the fact that GoogleBot could not access a URL on your website.
This happens because your website may require users to log in or authenticate their identity. You host could be blocking Google from accessing your site as well.
Fix Access Denied errors
- You should test whether your robots.txt is working and isn’t blocking Google.
- Remove the login from pages that you want Google to crawl. This might be a popup prompt or any in-page.
- You should use the Fetch as Google tool and fix problems related to your site’s content as well as discoverability in search results.
For ease, here is a simple table with quick fixes to all the crawl errors I’ve ever faced.
URL ERROR
MEANING
FIX
Soft 404
A page that is 404 returned 200
301 redirect or 404 status
404
Requested page doesn’t exist
Fix URL or redirect
403
Access Denied
Remove the blocking program
301
Not followed. Your redirects don’t work
Fix your redirects!
400/405/406
Grouped errors, treated as 404
Fix the URL or redirect
Flash Content (Smartphone)
Lots of pages, having flash that won’t work on smartphones.
Get rid of flash
DNS Error
GoogleBot cannot connect to the DNS server
Fix your DNS issues
So, there you have it, guys. The ultimate guide to fix and counter any webmaster errors you may ever encounter.
