Search Engine History - How Google came to dominate

So how did Google get from a market share of <1% in 1999 to almost 80% today? What is so special about Google and how has it come to dominate the scene? In this article, the third in my search history series, I look at the role of luck, timing and the mistakes of competitors in the rise of Google. I also evaluate the unique technology and vision at the heart of the Google algorithm.

The Dot-com Shake-out

You might be surprised to hear me moan about the dot-com mania of 1999-2000. After all, for many of us it was an exciting time. However, for the early search engines, it was essentially a death-knell! The new money entering the market was huge but investors wanted to see returns. Search Engines were clearly part of the future, but where was their revenue stream? How could one make money from free results – especially free results powered by very expensive computer infrastructures? This impatience grew following the dot-com crash of March 2000.

Yahoo! seemed to be the only company with any kind of sustainable “pure internet play” (from banner advertising which was at that time well funded by the dot-com dollar). At the same time, AOL were pursuing a moderately successful “walled garden” approach to traffic (i.e. where 70% of searches on the site led to content within AOL properties). The argument at the time was that making it easy for people to “exit” to other sites (the essential purpose of the modern search engine) meant losing the opportunity to sell them anything from your own site and content partners.

If you stop and think about it, you can see how very intelligent city analysts, investors and board members would be clamouring for (a) conversion to portal and (b) rapid acquisition of content and product / service offerings. Initiatives to expand search capability (and drive even more of your visitors off your site without payment for doing so) were bottom of the strategic shopping list; the battle cry instead was “content is king”.

Altavista, acquired with DEC by Compaq in 1999, was relaunched at huge cost as a portal competitor to Yahoo! Similarly, Excite (now owned by broadband provider @home) tried to become an AOL clone. Lycos, already the largest “pure portal” in the Spring of 1999, merged with the largest Spanish language ISP, Terra Networks (and then ran itself into the ground).

Disney/ABC and NBC had already invested heavily in Infoseek and Snap respectively but quickly began to realize that the revenue they could derive from the increasingly portal-like services was never going to deliver the short term payback which shareholders typically seek. Steve Bornstein, chairman of Walt Disney Internet Group, said in his press release "the Internet environment has continued to shift and change, and therefore our strategies must also change". 400 employees at Go Network were layed off and a buyer sought for the Infoseek engine.

In December of 2000 a surprising report from StatMarket.com found that only 6.86% of all the referring traffic to websites originated from search engines (compared to 47.01% from direct navigation or bookmarks and 46.13% from internet links). The quality of this research has since been questioned, as it was based not on server logs but on the global Hitwise stat counter data (the code for which was only placed by many webmasters on the homepage of their site). However, this data came at a time when there was a collective loss of confidence in search and portal technologies. The dot-com crash of March 2000 had left investors with a major hangover.

Is it any surprise, then, that so many search properties disappeared altogether (or closed their internal crawl capability down) in 2001? With the closure of Northern Light (to the public at least) in January 2002, the Dot-Com shakeout had essentially closed or fatally wounded WebCrawler, Magellan, Excite, Infoseek, Go Network, Snap and Lycos. It also critically wounded Altavista, who were to struggle on without any real investment in their search capability until their acquisition by Yahoo! in 2003.

The Emergence of Paid Search

If the rush to convert all search engines to portals was (at the time at least) flawed strategic thinking, bad timing also played a part in the demise of so many early contenders.

Quietly and almost unnoticed during the mania was a relatively minor search player called go-to.com, renamed Overture in 2001. Whilst paid listings had been tried before, Overture (with a 2.76% market share in January 1999) were the first company was make a success of it, with some advertisers paying up to a dollar a click by 1998/9. Go-to.com, in fact, only carried paid results! Personally, I never really liked or used the site myself for that exact reason. However, they persevered with their strategy, determined to find a sustainable revenue model and confident that the research-oriented search engines of the time had little to offer a growing army of web users looking to shop ‘til they dropped!

History has proved that Overture were right (to persevere with paid ads) although initially there was limited evidence of this. Perhaps if the dot-com investors had only waited a little longer (or started a little later) the value of sponsored results would have become clear. Either way, the big boys hared off into portal oblivion, whilst the Overture tortoise (and the model it spawned) ultimately won the race.

All the best businesses start in Garages

It really is true. The 12x18 foot garage, in which David Packard and William Hewlett launched Hewlett-Packard in 1939, is considered by many to be the birthplace of Silicon Valley. A short drive away is the Los Altos garage of Steve Jobs’ parents, where Apple Computer Inc. was born. Google Inc, similarly traced its early days to a garage – this one in Menlo Park, California – where the newly formed Google Inc spent its first five months as a proper business in late 1998.

However, the real story of Google began more than two years earlier, as the research project (nicknamed “BackRub”) of two Stanford Ph.D. students: Michigan-born Larry Page and Russian Sergey Brin. The simple obsession which preoccupied both was the links between sites on the exponentially growing web. In his original thesis research, Page hypothesised that any link from one site to another was akin to an academic citation and the description (or anchor text) attached to that link like an annotation. If one could identify the sites most cited for any given annotated subject, one could establish a pecking order of significance for any given subject area.

What particularly fascinated Page was what he saw as a fundamental weakness in Tim Berners-Lee’s original design for the web; namely that it was possible to see easily what any site was linking to but very hard (if not impossible) to see the sites linking back. Backrub sought to catalogue all the links on the web and, through doing so, establish which sites had the most “authority” overall and in any given topic. Page insists that it was never his intention at this point to create a search engine. However, intellectual curiosity drove him forward into the creation of a new kind of crawler, based on a link ranking system (now called PageRank after it’s inventor) which was the first of it’s kind.

It was at this point that Sergey Brin was drawn into the project. At this time. Page estimated the size of the web to be approximately 10 million documents and the number of links to be perhaps 100 million. Altavista had impressed everyone by managing to index all those documents on a single large computer. However, indexing both the documents and the links was, to say the least, a big hairy problem; precisely the sort of challenge that appealed to Brin.

In fact, the problem was much tougher than either probably realized at the time. At this point, the web was growing at an exponential rate and the computing resources required for the task were in fact well beyond the humble resources of a student project. This brings me to the part played by lady luck in the formation of Google.

In late 1996, the BackRub crawler was consuming nearly half of all Stanford’s network bandwidth and, on occasions bringing down Stanford’s internet connection. Page and Brin had to beg and borrow storage capacity and were continually running out of space and lines of credit. Whilst both were hardly mucking about, they were not completely sure about a career in business. As Page puts it now, he was either going to be a professor or run a company that made a difference. Nothing in between would do.

So, believe it or not, in the period from 1996-7, all three of the then market leaders were approached by two students with an idea for licensing a new kind of search engine. In August 2004, at the Search Memories session at SES San Jose, the panel included Doug Cutting (formerly Senior Architect, Excite), Steve Kirsch (formerly founder of Infoseek) and Louis Monier, (formerly CTO at AltaVista). Together they reminisced about the early years of search engines and all admitted, one by one, that they had sent the founders of Google packing. Steve Kirsh was the most colorful; "Go pound sand I told them”. Doug Cutting said “I wasn't impressed with their demo at all" and Louis Mornier shrugged "I didn't have the authority to sign a check anyway".

What if Larry Page and Sergey Brin had been just a little bit better at PowerPoint? What if Louis had had his check book with him? This book might have been called “How to get to the top of Altavista” and Digital might still have been in business, continuing the make the machines that power the internet. Lady luck certainly played her part in the proceedings!

However, despite their efforts to dodge destiny, the Stanford graduates did not find a suitable established partner. Some are born great, others achieve greatness and some have greatness thrust upon them. Perhaps Larry and Sergey fall into that rare latter category. By the end of 1998, the Backrub service (renamed Google in September 1997) was growing fast and serving more than 10,000 daily queries. The guys were introduced through a shared contact to Andy Bechtolsheim, an active early-stage investor. After a short demo, Andy literally thrust a check for $100,000 into their hands. Google inc. was born and Brin and Page famously celebrated with a trip to Burger King.

The guys literally lacked a bank account or a legal entity called Google to deposit the check into. So, once the burgers write digested, the two set up Google Inc and rented a garage from a friend. Five months later, the initial investment roll totaled just under $1 million and, by June 1999, a further round (of venture capital funding) exceeded $25 million. If it were fiction, you would say it was a fairy tale… but sometimes truth is stranger than fiction (particularly in the strange world of technology).

From backroom to boardroom in just 5 years

So what does Google mean anyway? Brin and Page close the name, from a brainstorm with fellow students, as a play on the word Googol, meaning the number one followed by 100 zeros. Google's use of the term reflects the company's mission; “to organize the immense, seemingly infinite amount of information available on the web”. As an interesting footnote, the word Googol itself is traceable to the work of American mathematician Edward Kasner in the 1940s. His descendants briefly considered suing Google in 2004 for using the name without royalty. Only in America!

Unlike their competitors, Google had a clear vision, perfect timing and all the luck going. Whilst the big boys fiddled with their content-rich portals, Google ruthlessly focused on building the biggest index in the world, ranking sites organically by their authority and serving paid listings alongside, in emulation of Overture’s successful model. As the web grew into a giant haystack, Google helped you find the needle. These factors, above all, laid the foundation for Google’s later dominance.

The company was also pretty thrifty and managed to make it’s initial venture funding go a very long way (at a time when other dotcom millionaires were spending money like it was going out of fashion). In those early days, licensing the search results to other internet sites and portals was the key source of revenue. Key moments included the use of Google by Yahoo! in 2000 to power their search results and the later decision by AOL to do the same. Suddenly, there was only one serious player in town!

Following the important business wins of 2000 (and the launch in that same year of Google Adwords and the Google Toolbar) the search engine was handling more than 100 million searches a year. By the fourth quarter of 2001, the company was turning a profit – an incredible achievement when one considers that Amazon (founded five years before Google in 1994) only turned its first quarterly profit at the same time.

Between July 2002 and June 2003, global referrals from search engines almost doubled (from 7.18% to 13.46%) and have continued to increase quickly since. Most small business sites now obtain more than 85% of their new visitors from search engines, with Google typically sending about 70% of them. In the UK, search engines have overtaken adult sites (in October 2006) as the most heavily trafficked category on the web.

It was perhaps inevitable that Google would eventually come to market and it's initial public offering took place in August 2004, creating a company worth $23 billion. More importantly, the flotation raised some $1.67 billion, which Google could use to develop new products and make acquisitions. It also made instant paper millionaires of many of Google’s employees.

Perhaps the biggest irony of search history is that AltaVista and the rest were not necessarily wrong in their obsession with portals. They were simply right at the wrong time! The preoccupation of today is not dissimilar to that of 1999, with Google and others all seeking to built “stickability” into their proposition. Suffice to say for now that those IPO billions in the bank were helpful in the acquisitions of YouTube and DoubleClick in 2006-7.

Comment

The history of search engines is really a history of university PHD students, fascinated by the growth of the internet and the desire to capture and bring into good order the vast information resources of humanity. However, search has now outgrown the computing resources and funding capacity of the typical university and the market is increasingly controlled by a small number of well-funded public companies. Google dominate, with a market share of nearly 80% on most pundit's measures. This is the final article in my series on search history. On my blog, I ponder the future of search and ask the question; could Google's market share be at the start of an inevitable decline?