At last after 2 weeks, today I can see the indexing status of my website inside Google Webmaster Tools as "Googlebot last successfully accessed your home page on Dec 17, 2006."And the results are really attention-grabbing yet disappointing.
Google shows HTTP error for a page inside a directory which has already been banned using robots.txt:
User-agent: *
Disallow : /folder-main/folder-sub
Which clearly states any Search Engine Crawler or spider; including Google; not to crawl over anything inside the folders mentioned, which includes /folder-main/folder-subBut for some reason, Google or Googlebot still crawled that folder. Although it doesn’t show pages inside it in the search results but Damn! Why Google / Googlebot is not following the rules??? Do they really follow robots.txt as they say???
Conclusion:Google / Googlebot wants to grab everything within the websites either public or private. I don’t know what the reason is behind but as a result we cannot keep things in private anymore. Extremely Disgusting!!!
