Major search engines for people who don’t care about getting the right answer: Netscape, GoTo and LookSmart
 

  by Eric Blair

There are, at last count, fifty zillion search engines and indexes available on the internet.  Twirlix, Best of the Web, Google, Yahoo, Ask Jeeves, Netscape, Lycos, Hotbot, GoTo,  LookSmart, Netscape – big ones and small ones that come and go, the list is endless.

The question is, how can you tell the good ones from the bad ones?

The answer to that is as simple as can be.  But before Oregon Magazine gives it to you, we’re going to explain why it is so simple.  First things first, so here’s question number one for you: Why do you use a search engine?

Answer: because you are looking for information.

Now, what kind of information are you looking for?  Answer: the best quality and the broadest selection possible.

If you take a look at the mission statements of the biggest information providers out there (usually accessible from “about us”), you will in each one find a sentence something like this -–"It is our goal to provide the finest, most complete information on the net.”

So, these people understand what it is you want.

The dirty little secret is that a large number of them don’t really do that.  They say they do, but they don’t deliver.  They talk the talk but don’t walk the walk.  Here’s why.

There are two basic types of information providers on the internet – search engines and indexes.  The latter, the index, is a giant library.  A resource.  Some of them don’t even offer public access.  Their massive database of websites is sold to search engines just like a book distributor sells books to retail stores.  It’s the wholesale information business.  The other type is the actual search engine.  Using either a database of their own, or buying information from an index warehouse, they are the equivalent of the bookstore where you shop.

An example of the index would be “mamma.”  An example of the search engine would be the names at the top of this article.  (Some of them, like Google, are their own index warehouse, as well.)

Now that you understand a bit about the infrastructure and operational approaches, we’ll drop the following bomb into the mix.  Some of  these people charge for listing a website.  And, that is the key you need to determine the value of such a service.  Do they charge to list a website, or do they not?

“But,” the charging types say, “we need to make a living, too.  What’s wrong with that?”

 Well, I’ll tell you what’s wrong with that.  Nothing, if you make an honest living.  Everything, if you make it by deception.  You cannot claim to be the best source of information on the internet if your future database will include only those who pay. 

As a brief aside, there’s a really strange business aspect to what is going on.  Indexes and search engines that charge for listings are comparable to the butcher charging the farmer for his cow.  It’s like General Mills charging farmers for their wheat. 

It is a patently ridiculous concept.

But, more to the point of this article, there’s something about this practice that directly impacts you, the information seeker.

Because some of these indexes and search engines (“mamma” and “Ask Jeeves” [see note at bottom] to name one of each) are now charging for listing a site, their databases are quickly becoming incomplete.  How can I make that claim?  Easy.  Imagine that the shelves of public libraries contained only books whose authors paid to stock them there.  (Some of you are old enough to recall the term “payola.”) Imagine that authors even had to bid for space on those shelves – which is the next dirty little secret you’re going to discover here.  Some of these indexes and engines have ranked charge schedules.  It is usually associated with a per-hit scheme of some sort, and often tied to a bidding process.  That’s right, some big search engines auction website placement!  They are auctioning ranking!  The sites you find when you search for information with them are the sites that bid the most to be there!

Let’s set aside the ridiculous practice of top page rankings based on popularity (how many people go to a site), which of course means that if you type in “what’s a good book to read,” comic books will dominate the first hundred links.  (The works of Swift, Dickens, Homer, Marcus Aurelius, Gibbon, Dostoevsky, Proust, Machivelli, Steinbeck and Hemingway will turn up a thousand links later.)  Let’s set aside the fact that a music list based on popularity would place stupid gangster rap at the top and Motzart at the bottom.

Let’s just consider this business of charging not only for appearance, but ranking.

If Joe Smutz, an Art Bell Area 51 devotee who lives in a trailer in Nevada bids higher than an Einstein type for ranking, the physics you are going to find in a search on that engine will be a joke.  You’re going to have to click through fifty pages to find the good stuff.  And, if the Einstein type is working at a patent office, and has a wife and a new kid to support, he may not even be able to afford the non-bid, flat charge type of search engine listing.  His site won’t even be on the list!

A database that takes all comers for free is a pure, chaotic democracy.  You get junk and jewels from one like that. It's up to the user to separate the two.  A database based on the selective judgement of the owners is an ideologically biased construct.  One has to consider the source.  Either, in my opinion, is preferable to one based solely on who can pay, or pay the most,  and who can't.  That idea is the worst one so far in the information age.  It's as silly a concept as an Anti-Steam Society for railroad employees in the late Nineteenth Century -- and potentially just as self-destructive.

For what it's worth, I'm not suggesting we need government regulations against this practice.  We already have ten times as much government as we need.  What I am suggesting is that we let the market decide which information companies survive and which go down the old porcelain pooper.  My vote should be very clear to all, by now.  But, I'm going to do even more.  Read on.

Now, you know the dirty little secrets.  The question at this point is: how can you defend yourself from this sort of thing?  How can you bypass the bad databases and utilize the good ones?  Way up at the top, we told you it was simple, and it is 

Here we go.

The Long Simple Method.

Every search engine/index that is accepting listings will have a tiny little link titled “Add URL.”  Click on it and follow the instructions.  If you don’t have a website, invent one.  You’re not going to list one, anyway.  All you’re doing is to work through the process to see if somewhere along the way, there’s a charge.  If there isn’t, dump out and add this search engine to your list of basic information sources.  If there is, dump out and forget this search engine.

The Short Simple Method.

Go to any major search engine (like those featured by a click on the Netscape “search” icon) and type in the words (in quotes) “Oregon Magazine” Every one of them has been informed a number of time about the existence of this publication -- the most important general-interest online magazine in the state.  If a link to Oregon Magazine comes up on the first page, it’s either a sign that the listing is free (thus it is a good engine) or that the management recently read this article.  If a link to this magazine doesn’t come up on the first page, it’s either a pay engine or one that ranks by popularity.  (Some of the latter offer “Oregon Bridal Magazine,” instead of Oregon Magazine to the specific “Oregon Magazine” request. 

Think of it this way.  Can you imagine asking for information about New York media and not getting the New York Times?  Can you imagine some dumb search engine substituting the New York Bridal Magazine for the New York Times?

If you know of a site that should obviously be on a search engine, but isn’t – particularly if it is on several other similar engines - you know one of two things.  The site owner didn’t submit to that engine (very unlikely if it is a big one) or chose not to pay for the privilege of turning his beef cattle over to that butcher.

Go with the free ones, friends.  And with the ones that don't substitute wrong answers for the right one.  And, when you find one that lists your specific request below those who have bought top position, put them in your secondary list.  That’s the system you’ll find at the bottom of this magazine’s cover page.  Yahoo and Google do a good job.  Lycos responds to the “Oregon Magazine” test by (as of this writing) showing three wrong, probably paid for, links, but below in the general web search area does the right thing. Win4Win.com (see note below) has bid-for-click URLs and free listings -- and makes an effort to weed out misleading and usless sites.

Go with this method and you’ll get the information you want.  Fail to discriminate in your choice of search engines and you’ll get second class answers.

It’s as simple as that.

UPDATE: Ask Jeeves editors are either slow in responding to URL listing requests, or have modified their listing policies from those obtaining at the time this piece was written.  Oregon Magazine, although not at the top of an "Oregon Magazine" quoted request, does appear in their list, now.