fbpx

How Google search works? The basics of crawling and indexing.

With over 90% of searches done on Google, it's imperative to know how google operates.

Imagine publishing a novel without having penned as much as a diary entry ever before. It might certainly work if you are lucky, but it’s a lot easier to know the core elements of writing beforehand.

Even before a user hits the search option on Google, the search engine has organized information about the webpages in the search index. It is essential to understand how Google organizes information to deliver the most relevant, useful results in a fraction of a second to your users online. While the Google Search Console Help has detailed information on crawling and indexing, we have tried to capture the most essential elements of how Google search works to help you understand how your website stands a better chance of showing up on search results.

Crawling, indexing and ranking - Understanding a website from Google’s point of view

To be at the top of your game on Google search, you need to understand and have in-depth information about these three terms. The process undertaken by Google’s spider crawler to scan the website for content, images, links and other credible information is known as crawling. This helps Google understand what your website is all about, how it should be categorized and allocated search results rankings. You can use the Crawl Stats report for information on Googlebot's activity on your site for the last 3 months. These stats take into account all downloaded content types such as CSS, JavaScript, Flash, and PDF files, and images. You can find a sample report below:


google crawling and indexing

Have you recently added additional pages or changes to your site? It is recommended that you ask Google to (re)index it using any of the methods listed.

The process of indexing refers to Google storing information (all the content it has discovered and deems good enough to serve to searchers), in an index. Just because your site can be discovered and crawled by Google doesn’t necessarily mean that it will be stored in its index and will be ranked soon after. Not just that, pages can be removed from the index too and the most common reasons for the same are: URL is returning a "not found" or server error or violates search engine Webmaster guidelines. Once you have ensured that the above are taken care of, you can use the page- and text-level settings to adjust how Google presents your content in search results.

Did you know that there are over 200 factors that Google considers when delivering search results. Some of them are:

  • High ranking content. It is essential to focus on value of content and not the word count. Apart from that, it is essential to incorporate engaging visual content to complement the written word.
  • Structured data. Google Search can enable a rich set of features for your page in search results. However, it can achieve this if it understands the content on the page or if you have provided additional information in the page code using structured data. Learn more about how content appears in Google search and how to enable rich results with structured data.
  • Mobile-friendliness is another major SEO ranking factor. With an ever-increasing number of people using mobile devices than desktops, there have been changes in how Google ranks search results. Read more about the Google mobile-first index and ensure that your site is mobile optimized.
  • Other factors to consider include - how well your content matches user queries, how quickly your website loads, how many people have shared your content online and more.

If you want to find out more about your site’s crawling or indexing status, please get in touch with us.

Can 404 errors lead to negative impact on your site’s ranking? Yes, it can. But there’s more to 404 than that. If you run a website or blog, it is important that you create a custom 404 page. This can help direct visitors who encounter this error on your website.

What is a 404 error message?
A 404 or Not found error message is an HTTP response code. As per HTTP status code blocks, 400 block shows client errors or it means that a page was not found. 404 error is a result of the requested page not being found. It is usually accompanied by a message as “page not found” or “The page you are trying to access does not exist”.

404 error

They are not all bad as they might be a result of an error on the user’s side too. For instance, if a user mistypes the URL or requesting for a page that does not actually exist. In this case, Google does not penalize you if you served a 404 error as the cause is a ‘bad URL’.

Watch this video to see what John Mueller says is the best way to deal with 404 errors.

How to find the 404 errors on your website?
You need to use the Google Search Console to identify these errors on your website. They are listed under Crawl Errors as soft 404s and “Not Found”. If you think that your page is incorrectly flagged as soft 404, you need to use the URL inspection tool to find the most recently indexed version of that page.

Clever error pages that we found!
With some effort in crafting the right 404 page design, these brands are able to re-engage their users so they continue exploring the site. Here are a few of them:

404 error2

404 error3

404 error4

Why you should create custom 404 pages?
As you would have observed, adding an interactive element even on your error page gives your site’s visitors an action item in order to find the page they were looking for or find an option to help. This increases time on page and reduces bounce rates and both are important from the perspective of Google. Cracking a smile in an otherwise frustrating situation, your user’s are sure to do that with creative error pages. More importantly, it helps them find the information they're looking for. Here are some suggestions for creating an effective 404 page (as per Google):

  • Use friendly and inviting language to let your user know that the page they’re looking for cannot be found.
  • Add links to your most popular pages as well as a link to your home page.
  • Your webserver needs to return an actual 404 HTTP status code when a missing page is requested, to prevent 404 pages from being indexed by the search engines.
  • Ensure that you have an option for users to report broken links.

We hope that this post has helped you understand what a 404 Error Page is and how important is to manage it.

You’re probably thinking leads and sales when you’re asked about the digital marketing metrics that are tracked for brand growth. As websites continue to become an inseparable part of the consumer’s journey, the need to measure digital marketing success also increases. However, with the large amount of data available, choosing the right type of metric to monitor can be a daunting task. The good news is that we have shortlisted few key metrics that are applicable to pretty much every type of digital marketing campaign.

Here are the top 5 essential metrics that you should have your finger on the pulse of.

digital marketing success

Traffic (sessions): Simply put, website traffic refers to the visitors who visit a website. These “visits” also known as sessions measure online business effectiveness at attracting your target audience. This metric helps to gauge how well your marketing efforts are working and it also helps you retrieve an overall snapshot of your site’s performance. While there isn’t a maximum number of a set range for this metric, it is recommended that you find comparisons in the benchmarking report in Google Analytics. The important channels to watch are Direct (when people type the URL to visit the site), Referral (when people visit the site from another website), Organic (when people searched for a keyword on a search engine and landed on the site) or Social media. For best results, you should aim to achieve a balance between the various traffic sources. In the event of traffic from a single source, any change to the strategy or source is likely to impact your website traffic and bottom line.

Conversion rate: Conversion rate indicates the visitors who have completed any goal on your website. The success of your marketing campaigns depends on the conversion rate. Sure the number of leads that you have generated online matters but do you know what matters more? Calculating cost per lead per channel helps you identify the most important lead generation channels and invest in them accordingly. If you have an Ecommerce store, you should consider the impact the online traffic has on your online transactions. With access to a tool such as Google Analytics, you can find your Ecommerce conversion rate by setting up Ecommerce tracking using a few simple steps as listed here. You can look at the various sources of traffic and the conversion rates from these sources. Are they interacting with your content and are they providing value, when they visit - are the pertinent questions to ask when you are trying to ascertain your digital marketing success.

Bounce rate: As defined by Google, a bounce is calculated specifically as a session that triggers only a single request to the Analytics server, such as when a user opens a single page on your site and then exits without triggering any other requests to the Analytics server during that session. It is common knowledge that a high bounce rate is bad and a low bounce rate is good. If you have observed the bounce rate in the overview report of Google Analytics, you should know that this is a site-wide bounce rate and not for a particular section or page. It is calculated by dividing the average number of bounces across all pages by total number of visits across all pages within a time period. If you are wondering, what bounce rate is good, you should note that there are various variables that determine a “good” bounce rate for your site. You can set up benchmarking in Google Analytics or read the bounce rate benchmarks as available here to gain a better understanding of your website’s bounce rate performance.

Social media metrics: The return on social media investment still exists as an elusive part of the marketing puzzle for a number of businesses that seek to capitalize on the eyeballs that spend billions of minutes on these platforms each day. Is social media ROI attainable? Of course, it is when you have set up your social media initiatives from the outset in a manner such that you are tracking the right metrics during the buyer’s journey. Defining and measuring specific customer behaviors, as mentioned below helps you climb out of the vortex of vanity metrics with dubious ties to revenue.

digital marketing success

Finally! The fifth metric that is absolutely essential to determine your digital marketing success is the SEO metric. You want your website to rank for keywords relevant to your niche. But, how do you determine if people are visiting your site as a result of your SEO efforts? The Google Search Console and Google Analytics are invaluable tools in this regard, they can help track traffic by landing pages, top-ranking keywords for your website, mobile traffic (this is super important, considering 60% of all online searches come from mobile phones!)

To sum up, you’ll need to constantly evaluate the position and success of your marketing strategy and tracking the aforementioned metrics are the first step in this direction.

What people are saying...