Avoid Bumps and Walls For Getting Indexed Better
It is important to know and understand them so that one can modify he website accordingly for giving better access to spiders. Search engine spiders crawl pages following hyperlinks on other pages. But certain things may serve as hindrance to crawling of search engine spiders.
Search engine spiders crawl pages following hyperlinks on other pages. But certain things may serve as hindrance to crawling of search engine spiders.It is important to know and understand them so that one can modify he website accordingly for giving better access to spiders.A link that is too complex to follow or is situated in quite depth from the hyperlink is called a bump.This serves as speed breaker for the spider in a similar fashion to traffic on the road.Website may contain sections of data that cannot be accessed at all by the spiders is called wall.Both of these would block indexing of certain sections of your website.There may be some parts of your site that you do not want the search engines to know.But if you are in business,
you would want to expose maximum of your content and indexed.These are few bumps and walls that you must avoid to do that.Bumps * Dynamic URLs with >2 dynamic parameters would be a bump. These kind of URLs are reluctantly crawled as complex URLs often result in errors. An example of complex URL would be http://www.yoursite.com/page.php?id=3&PF=567&user=premium&name=%Arun% * Hundreds of outgoing links from a page. In such cases spiders may not follow each link. * Pages buried in depth of linking structure. Generally, the spiders will not visit a website link that requires more than 3 clicks from home page until that page has high number of incoming links. * Frames are a bump to crawling.Walls * Pages requiring a login. This is generally case untill you put special request for robots to visit. * Documents blocked by robot.txt file * Links having nofollow tag. * Pages that require drop down menu to access them. * Documents that can only be accessed by a search box. * Pages that can be accessed only after filling and submitting the form. For example a sign up form.Fewer bumps and walls your site has, smoother would be its indexing.A rule of thumb to avoid unwanted bumps and walls is to ensure that the content of your site is easily accessible. Most of the spiders would start from your home page. To get indexed, the page must be accessible from home page.
