Along Came a Spider (Part One)

So, your Web site is up and running. It looks great and on its first day you're excited about getting your first order. But your excitement soon turns to weariness as that one order is the only one that comes in for a whole month. And worse, your Web site statistics show a disappointing hit rate. So much for the perception that “if you build it, they'll come.”

Every Web site owner wants people to visit their site, but very few understand the role search engines play in getting those people there. And fewer still, understand how relevant content can not only attract the search engines, but convert your visitors into paying customers.

Have no fear. This guide will help you understand the relatively simple steps you can take to make sure the search engines send targeted traffic to your site, and increase your sales.

But before we begin it’s important to understand how search engines work, and make the distinction between crawler-based search engines like Google and a directory like DMOZ.

Part One - Understanding the difference between search engines and directories

Crawler-based search engines.

Crawler-based search engines, or spiders, literally “crawl” the Web looking for content. They’re able to do this because of the way pages on the Internet link to other pages by way of hyperlinks. Anyone who’s sat down at the computer “for five minutes” to find information has experienced this linking system—hours later you’re still there, completely off track, clicking away from one page to another to another.

The search engines use this linking system in much the same way as human users. For example, when Google sends its “spider” (fondly known as GoogleBot) to “crawl” the Web it follows links from page to page indexing the content it finds along the way. The information is then stored in a huge database somewhere at Google. Later, when someone enters a particular word or phrase into the search box, Google scans its database for possible matches. It then displays pages that contain, or relate to the word or phrase in an order it considers most relevant.

There are really only two major crawler-based search engines, Google and Yahoo. The others, with the exception of several smaller engines such as, Ask Jeeves/Teoma and engines based outside the United States, get their results from these two. See below to find out which engine supplies and which engines receive.

While Google and Yahoo crawl the Web in much the same way, the results you receive from each can vary greatly. You can see an example of this by searching for “direct mail packages” on both Google and Yahoo. As of today, (and this is certain to fluctuate on a daily basis) a test page from my site (www.juliahyde.com) with the title “Sales Letters and Direct Mail Packages” hovers around number 12 on Google’s results. Perform the same search on Yahoo and the page ranks number one. It also ranks number one on MSN, but that’s because, until MSN officially launches its own search engine, Yahoo supplies its results.

Contrary to popular belief there’s no need to submit your Web site pages to the crawler-based search engines—if your site is built with the search engine’s and your visitor’s best interests in mind, the crawler-based engines will find it on their own. I’ll talk about this in more detail in a future chapter.

The two major crawler-based search engines supply results for:

Google - AOL, Netscape and iWon

Yahoo (synonymous with Inktomi) - MSN, Alta Vista and AllTheWeb

Online Directories

Directories are like giant yellow pages that compile, rank and organize listings into different categories and sub-categories. They do not crawl the Web looking for content but rely on submissions from web site owners. Professional, human editors generally edit directories. Most of them work something like this:

 You want to buy a pair of jeans so you go to a directory like DMOZ and click on the main shopping category.
 Then you click through the sub-categories, “apparel” “retail” “jeans” and so on, until you find exactly what you’re looking for.

Top Directories include:

 The Open Directory (supplies directory services to Google)
 Yahoo
 Looksmart
 Gimpsy
 Zeal
 JoeAnt

Although submitting your site for inclusion in directories will drive some visitors to your site, you should not necessarily base your decision to submit on how much traffic you think you’ll receive from the directory. But, rather, view the submission as an opportunity to obtain a link to your Web site. Why? Because a directory listing will allow the crawler-based engines to follow a link to your site and help get your site indexed in their database quickly. It will also give your site a good quality incoming link (more about this in later chapters).

Next month: Words. Words. Words.