Supplementing Spiders

What we are trying to do is encourage these 'spiders' to come feed on the keywords and 'content' of our websites, in hopes of getting a higher positioning in the search page ranking. It is like trying to put your head above the crowd and be noticed.

The experiment began with a new site that has been online a little more than a month [from the time of this writing]. What was decided was to create the best atmosphere for attracting Googlebot. A plan would be needed to attract Google and save the time, effort and outlay of cash for a Yahoo! listing. Google was also needed to build a page rank. Starting from zero we had to have a game plan.

The conclusion was reached that the time spent now would pay off ten-fold down the road. The new site is a free cross content article directory for websites, ezines and a learning portal. Keyword search was preformed on every search term imaginable. Every site that could be found that remotely related to articles and article directories was reviewed. Some of these sites were exclusive to articles while some only had a directory of articles on site in a subdirectory. What they had in common was that all had a front page or top 10 ranking in Google and/or Yahoo! for 'articles'. Note: there were also 417,000,000 other sites listed for "articles" that were indexed that have never seen the fringes of the front page much less the top 100. With that goal set, our task lay before us. The plan follows:

1.] A decision made on the look and feel of the site. This is important not for the spiders [Google & Yahoo!] but for the site’s future visitors. Reason? What good is it to have a site that no one will come back to because it is unpleasant or because of visual or navigation problems. The robots.txt file was set up so that it would invite the spiders to crawl among the selected pages and directories of the site.

2.] The index page was designed making sure it had an accessible navigation plan based on the suggestions of www.lighthouse.org and www.accessify.com/default.asp. These improvements continue to be added throughout the site. Every link led to content using keywords in the links and text anchors where possible. Optimized the site using a selection of keywords that were relevant to the site’s theme using Good Keywords and Overture. We were fortunate as we were building an article and content directory and we all know articles make excellent spider supplements. Like vitamins for the body...

3.] A Site Map was created and linked from the front page that provided the visitor and spiders with a road map of the site. Using www.netroglycerine.com/sitemap.html, as we did you can easily generate a full sitemap for your website. Enter your URL in the space provided, hit enter on your keyboard and wait for the links to be displayed. Save the page from the File menu and edit the links to suit your requirements.

4.] Next we brought together a collection of articles and content that would provide a rich array of information for both spider and visitor. This was accomplished by adding articles personally written and some hand selected content to fit the site’s topics and categories. These included a wide variety of subjects like short stories, tutorials on uses of herbs to R.S.S and how to use email.

5.] A thorough check of all links was done checking for "title" tags. Links not working were replaced or repaired. Images were checked for appropriate "alt" description tags and made sure all were described in a manner that best suited them using keywords where possible.

6.] Knowing the site was relatively new, articles were submitted to other directories with links back to the site. We also blogged the site and pinged the post and trackback URL and left links where possible to lead back to the site.

7.] Once everything had received one last thorough check, we took a break and then waited for Googlebot to call. We started to watch the server logs for our visitor.

8.] It only took a few days before the first index. We began to check the search terms that had been targeted and a few days later began to see the site in the search results. Checked with Yahoo! and there was no mention. Back to work we went. Polished and reworked the keywords that were targeted and tweaked the description. Made another trip around various article directories and submitted another article with links back to the site and again blogged the site and pinged the post and trackback URL's from it. We were re-indexed within a few days. This time as we check for search terms we began to see more of the site’s content and URL's appearing in the search results. Only this time it was not just one or two, but several different articles that had been placed as content in the categories and topics of the sites.

9.] Googlebot has been fairly regular for the past two weeks with re-index approximately every three days. After each visit the keyword and search terms are checked in the search results to see where we stand in the listing. Also checked are the server logs for specific keywords searched and evaluation of the traffic. We then go back to the site, tweak it, and get ready for another visit.

10.] This is only the beginning. This little exercise will continue even after reaching the goal. A routine has been established that includes periodically checking the server logs for the always, welcomed visitor. Making the adjustments where they are needed and preparing thewebbpage.com for another index.

The site is not listed No.1 with all of the keywords or search terms but thewebbpage.com is appearing on the first page for many. It is rewarding to know that without a single search engine submission, a brand new website can be indexed within a few weeks of launch, raise it's head above the crowd and be noticed. A site that now has the potential to make it.

Supplementing the spiders was, is, and will be well worth the effort. It has been an interesting exercise requiring, in the grand scheme, very little research and time. The best news is the same search terms that are showing up with Google search results are now appearing in Yahoo! results. Now that is what I call 'Supplementing Spiders'. The spider even picked up the tab for the Yahoo!’s inclusion.