Search engines are actually
software programs that search the web for keywords, maintain an index of the
data located and the website address where that data resides, and then provides
a user friendly interface for the search process to be completed by the end user
(you and me).
How
do they work?
The searching is done by
humans and/or automatic site searchers called “robots”, “bots”, “spiders” or
“crawlers”.These are simply
programs that are sent out by the search engine to locate and index some or all
of the words it finds on each web page it finds on the internet.Robots for each search engine work differently; some only “crawl” out
and search monthly, some search weekly and the more powerful search engines
perform this search on a daily basis.No
one actually knows how many web pages are actually in existence on the web,
although many have tried to guess, but is in the millions!Imagine how much disk space and processor power it would take to catalog
(or index) every word on every web page on the internet!Since only the largest, most
powerful computers can manage this vast amount of data, only the very large
companies (like Yahoo or Google, etc.) can maintain a catalog that is anywhere near
complete.The smaller companies will
search less often and old data is replaced more frequently by incoming new data
than is done on the larger companies’ servers.That’s why you get more results when you do a search using a large
search engine like Yahoo or Google than you do when you use some of the smaller
ones.
You may also have noticed over the past
couple of years that some of the less popular engines (usually maintained by
smaller companies) have all but disappeared from the internet.Those larger companies are faced with a continual challenge to keep their
hardware storage and indexing capabilities up with the demand placed on them by
the ever expanding internet.As with
most other types of today’s businesses, many of these companies are forming
partnerships or merging into one company.
With
so much data, how are the results prioritized?
The indexing and cataloging programs
that manage the vast amounts of data are really quite remarkable programs.Not only do they remember the web address for the words they find on each
web page, they actually keep track of the number of times that word appears on
each page and how many times that page is accessed by browsers!Relevance is determined by each of these factors but it is done
differently by each search engine.When
webpages are created, the developer can add a “metatag” to the web page
which contains the most relevant keywords for that website.The metatag for a hardware store, for example, would contain keywords
like “nail”, “hammer”, “paint” and so on.The search engines usually search the metatags first when attempting to
determine relevance for your search.If
the engine is able to find enough “hits” using only metatag keywords to
display 40 or so pages of results, it will not search the other indexes unless
you are persistent and browse on to force the more thorough search.
Of course, you can have your website moved to the top or near the top of the
relevance list if you are willing to spend the money!Without any extra effort or expense to you, it will take approximately 4
months for your website to be indexed on all eight of the major webservers
located worldwide.However, it
won’t do you much good to be in an index with 2 or 3 million other websites
that also contain the word or words the user is looking for!You can shorten the process from 4 months to several days for not too
much money (see
example), but you will still be buried among millions of other sites unless
you have some very unusual words on your website or your website is very popular
for some other reason, so it’s not really worth the expense.
Another alternative is a “pay per click” contract such as Overture.com,
Findwhat.com, etc.To do this, you bid on the amount of money (usually a few cents) you will
pay for each person who visits your website.These companies maintain a huge index and browsers are simply directed to
the highest bidder.You could bid 3
cents for each click and someone else with the same words or metatags could bid
4 cents and you would never get a hit (at least not from that server).These types of contracts are the underlying reason you see so many
pop-ups and banners on the web pages you visit.The owners of the website sell advertising to companies like X-10 who pay
the website owner for each hit, offsetting the “pay per click” contract.So remember, every pop-up ad you see while browsing probably cost someone
2 or 3 cents!Doesn’t sound like
much money, but it adds up quickly if your website is being visited 500,000
times a day!At 2 cents per hit,
that’s $10,000 a day or $300,000 per month!
Here are links to the top five
search engines in use today.....
