How Search Engines Work: Site Crawls, Algorithms, and Keywords
A search engine is a digital library, and search engine bots are the internet’s librarians. Everyday, they scan millions upon millions of websites and web pages, organizing information and prioritizing it based on how relevant it is to what users are looking for.
Crawling, Indexing, and Ranking
We already know from our last section that search engine bots are your “secondary audience;” they explore websites and look for a wide variety of information about each individual web page. This exploration can be broken up into 3 stages:
Crawling - Finding your sitemap, scanning the code on your website, and gathering information about keywords, links, site structure, page speed, etc.
Indexing - Adding your website’s data to the internet library, making it discoverable through a wide variety of search terms.
Ranking - Comparing your site to others based on the information gathered during the crawl, and using an algorithm to prioritize it in results, based on any given search request.
Just like a librarian, search engine bots categorize and prioritize information, to provide a user with the best possible results each time they search.
So how do you end up on the first page of results? There are two ways; either by bidding on keyword searches, or optimizing your site for a search engine algorithm.
Searchers and Keywords: Organic Results vs Paid Results
The last time you did a Google search, you might have noticed that the first few results were ads, but these results were still relevant to what you were searching for.
Businesses can decide to bid on specific keywords in order to show up in search results above any organic results. Any time a user clicks on a paid result, the business is billed for that engagement. This is also known as Pay Per Click (PPC) advertising. It is one of the primary ways that search engines make money.
NOTE: Search engines don’t take just any bidder for any keyword. You still have to make sure your site is relevant to the user’s search. If an ad is not relevant or engaging, you will be shown less often, and won’t get the traffic you’re looking for.
Organic results are what you’ll see underneath paid results. These companies did not bid on the keywords they are ranking for - rather, they made sure their site was attractive to search engine bots, and contained the ranking factors needed to appear first.
Ranking Factors: How Search Engines Select Sites
All of the information that search engine bots look for on a web page factors into where you will rank in search results. Ultimately, the decision is made based upon a search engine's current version of their algorithm.
Search Algorithms
In general, an algorithm calculates a specific rank for your site based on a wide variety of factors. Check enough boxes, and your chances for ranking higher increase dramatically.
While it’s impossible to know the exact specifics of each algorithm’s requirements, there are several areas we know that are important to search engines like google. Let's start with the obvious:
Relevance and Keywords
It's highly unlikely that your website will show up for a specific search term if that word is not located anywhere on your page.
That's why it's important to do keyword research, so that you know the specific phrases or words people use to get links to pages like yours. Then, make sure those keywords are introduced early on your page, and your content is organized and tagged, through your title, description, and page headers.
The more relevant keywords are in your content, the more a search engine bot understands that this page contains information that your ideal visitor is looking for.
Backlinks and Domain Authority
If you've ever written a research paper, you know that the sources you use are important.
The books or websites you reference help give your own writing more authority, and convince readers that you are providing information that is accurate, and truthful.
Search engines are no different. They want to know who you are referencing, and more importantly, who is referencing you. These outside references to your site are known as backlinks.
A website with a high number of backlinks is likely to receive a higher domain authority. This means that search engines consider you to be a trustworthy source of information.
It’s not just about quantity, though. Bots look at the domain authority of your backlinks, too. If high authority sites link back to you, that gives your own authority a boost as well.
Site Speed and Mobile Friendliness
Have you visited a site recently that loaded slowly for you? What about one that didn't respond well on your phone? How long did you stay on that site?
Search engine bots also take into account the experience your visitors will have when clicking onto your website.
In particular, they weigh mobile performance first over all other devices.
This means that if your site is slow, or isn't mobile friendly, you will rank lower in search results, even if your content is high quality, packed with keywords, and contains exceptional backlinks.
Location and Language
You might have encountered location-based search results if you've ever searched for a business “near me.”
If your location data is turned on, a search engine will take into account where you are located in the world, and provide you with search results that are most relevant to your area.
Local businesses will also be ranked based on:
Relevance - is your business what a user is looking for?
Distance - how far are you from this user right now?
Current Popularity - is your business already well-known in the local community?
Ratings and Reviews - does your business have positive public reviews, particularly on your Google My Business listing? (If you haven't signed up for one of these, it's free!)
The same logic applies to language; you won't find many search results for pages written in Swahili or Chinese when searching in the United States. If your website is multilingual, your SEO strategy will need to take language results into account.
Web Analytics: Visits and Engagement
This particular ranking factor has caused a lot of debate over correlation versus causation. Does a website that ranks higher in search results get more visits and engagement? Or, do more visits and engagement cause a search engine to rank a site higher and search results?
The answer is, a little bit of both. Higher views and engagement can indicate a site that is authoritative, contains relevant information, and is popular with people. However, traffic alone cannot determine whether or not a site should be ranked more highly in search results.
What we can definitively say is, pages that get a lot of traffic and engagement tend to rank higher in search results overall.
Freshness
This is not a universal ranking factor, but it is important in specific circumstances, for specific searches.
For example, if you're searching for the news headlines today, you won't want Google to return you news from last week, last month, or even last year.
That's why websites that are updated frequently are crawled more often, and may rank higher based on the freshness of their updates.
Search History
Search engines may also take into account the search history of a user, to personalize their results further. Some people may not have these preferences turned on, however, most will still opt in.
