Tech moves fast! Stay ahead of the curve with Techopedia!
Join nearly 200,000 subscribers who receive actionable tech insights from Techopedia.
A search engine is a service that allows Internet users to search for content via the World Wide Web (WWW). A user enters keywords or key phrases into a search engine and receives a list of Web content results in the form of websites, images, videos or other online data that semantically match with the search query.
The list of content returned via a search engine to a user is known as a search engine results page (SERP).
A search engine performs a number of steps to do its job. First a spider/web crawler trawls the web for content that is added to the search engine's index. These small bots can scan all sections and subpages of a website, including content such as video and images.
Hyperlinks are parsed to find internal pages or new sources to crawl when they point to external websites. To help bots do their crawling work in a more efficient way, larger websites usually submit a special XML sitemap to the search engine that acts as a roadmap of the site itself.
Once all data has been fetched by the bots, the crawler adds it to a massive online library of all discovered URLs. This constant and recursive process is known as indexing, and is necessary for a website to be displayed in the SERP. Then, when a user queries a search engine, relevant results are returned based on the search engine's algorithm.
The higher a website is ranked in the SERP, the more relevant it should be to the searcher’s query. Since most users only browse the top results, it is particularly important for a website to rank high enough for certain queries to ensure its success in terms of traffic.
A whole science developed in the last few decades to make sure that a website, or at least some of its pages, “scale” the ranking to reach the first positions. This discipline is known as Search Engine Optimization (SEO).
Early search engines results were based largely on page content, but as websites learned to game the system through advanced SEO practices, algorithms have become much more complex and search results returned can be based on literally hundreds of variables.
Each search engine now uses its proprietary algorithm that weighs many complex factors such as relevancy, accessibility, usability, page speed, content quality, and user intent in order to sort the pages in a certain order.
Those employed as SEOs often expend huge energy trying to unravel the algorithm as the companies are not transparent with how they run, due to the proprietary nature of their business and their desire to prevent manipulation of search engine results.
There used to be a number of search engines with significant market share. As of 2020, Google controls the vast majority of the western market; Microsoft Bing has a small presence in second place. While Yahoo generates many queries, their back-end search technology is outsourced to Microsoft.
In other regions of the world, other search engines hold the majority of the market. In China, for example, the most widely used search engine is Baidu, which was originally launched in 2000, while in Russia more than 50% of users use Yandex.