Optimizing Dynamic Pages - Part I

The Widget Queen
You are the Widget Queen. You eat, breathe, and live widgets. You sell more
widgets than anyone. You want to reach more widget customers, so you have
decided to sell widgets on the web. You have spared no expense in designing and
building the ultimate widget website. You have widget descriptions; you have
widget specifications; you even have widget movies. The only thing your widget
website does not have is visitors.

Off to the search engines you go. You type in the phrase "left-handed blue
widgets" and look at the results. All of your major competitors are listed.
There are even competitors you have never heard of. But you, the Widget Queen,
do not have a listing there.

What's up with that? What follows is some very basic introductory material
followed by some advanced technical details on dynamic sites and SEO.

What is a search engine?
First of all, you need to understand what a search engine actually searches.
When a potential visitor does a search in a search engine, such as Google or
AllTheWeb/FAST, she is not really searching the web; rather, she is looking at a
database compiled by that search engine. This database consists of the text and
links from the web pages that have been visited by the search engine's robot.

How is a search engine database compiled?
Search engines compile these databases automatically using software programs
called "robots" or "spiders". These automatic programs visit pages on the World
Wide Web, much as humans visit web pages using browsers, by starting at some
arbitrary location and following links. When a website owner "submits" a page to
a search engine, in most cases she is supplying the search engine's robot with a
starting point for their automatic journey. Starting in that location, the robot
then follows links and thus "discovers" other pages in your website or visits
other sites to which your site is linked. (This, by the way, is how search
engines can find individual pages or whole sites that have never been submitted
to them--if there is a link to one site from another site, chances are good that
eventually a search engine robot is going to find that link and follow it.)

Even though robots visit pages like human visitors do, what they can do with
what they "see" is quite different. When a human visitor uses a browser to view
a web page, that visitor can read the text on the page, look at images, play
movies, listen to sounds, submit information in forms, follow hyperlinks, and
any number of other tasks. The human visitor really interacts with the site. The
search engine robot, on the other hand, can only do a few of these things. It is
this difference that can keep your dynamic page from being included in the
search engine database.

What does a robot do?
Search engine robots are very simple creatures. They can "read" text, and they
can follow links. That's it. Robots cannot view a Flash movie, they cannot fill
in a form, and they cannot click a "submit" button. What that means is that no
matter how much great information your web page may contain, if a visitor has to
select it from a list, or type a password, or submit a form full of information
to get there, no robot will ever visit that page.

The origins of dynamic pages
Most dynamic web pages are generated in response to queries run against
databases. Behind your widget website there is a large database of widgets. When
a visitor comes to your site and looks for left-handed blue widgets, it is this
database that supplies the response. The database provides that information to
the visitor. Typically the visitor checks a box or selects from a list or even
types text onto the page and presses a "submit" button. Once she jumps through
those hoops, your visitor gets her page full of left-handed blue widgets.

I can't see you
Unfortunately, when a search engine robot visits this page, it cannot check that
box, it cannot select from that list, and it cannot click the "submit" button.
Put simply, the robot cannot get to page of widgets. If the robot can't get
there, the page will not be included in the search engine database. If it's not
in the database, searchers cannot find it.

So how do you get there?
So how do we attract other visitors to our dynamic page of left-handed blue
widgets? There must be some way to get there without having to click on that
"submit" button.

Next month we will look at several ways to get search engine robots to visit
dynamic web pages. Stay tuned.