Yesterday’s Thoughts

October 9, 2005

Which Robert?

Robert Scoble wants search engines to pretend that the web is different than it is. I think this would be a bad idea.

Scoble’s objection is that search results don’t reflect his notions of what is important in the world.

“As long as I’m the #1 Robert and Robert Redford isn’t as high up as me then you know that this is actually a search problem.”

This could be done by Google, Yahoo, or MSN pretty easily, but it would be a bad idea. The problem is that on the web we have today, Robert Scoble is more important than Robert Redford.

If you look at the Google results for Scoble and Redford, Scoble has 3.9M listings, while Redford only has 3.1M.

As near as I can tell, and with apologies to Robert’s brother, almost all 3.6M of those hits on Scoble refer to the Microsoft Geek Blogger, while there are hits on Redford that don’t refer to Sundance. All of the first 30 hits on Scoble referred to Robert Scoble, only 11 of the first 30 of the Redford results refer to Robert Redford. I also counted the 970-1000 hits on the two terms at Google. These are the highest results that Google allows the user to see. One hundred percent (28/28) of these hits referred to Robert Scoble while 92% of the sample (24/26) refer to Robert Redford.

Your mileage on these results may vary because of where and when you are pulling your results, but they make the point that Scoble is vastly more represented on the web than Redford. (If it makes Scoble feel any better, all of the links to Robert Redford are complimentary or neutral, while lots of the Scoble links are, umhh, challenging, to put it politely.)

Could Google, MSN or Yahoo make an editorial decision to boost Redford over Scoble because more people are interested in the movie star than in the geek blogger? Sure they could. When I was doing Commerce Search at Inktomi we did this all the time. There were all sorts of custom tweaks for various domains. If someone searched on “mouse,” we didn’t return mouse pads, even though mouse pads are formally as relevant as mice. If someone wants mouse pads, they will search for “mouse pad.” This seemed to make sense at the time. We were only indexing products and we were trying to build a database that made it easy for consumers to identify, locate and buy those products. We were trying to get inside the consumer’s head and send them the results that they expected.

Based on the history, consumers didn’t like it. As we were losing traffic, Google’s neutral results (and simple interface) were attracting consumers and growing like crazy.

There are a number of reasons for this, but I think the most important was that when the results are tweaked, the user experience is not as good. The user develops a mental model of how the search engine works. Even though the first time they search, they may be annoyed not to get what they want, they learn when and how to modify their search terms to get the results that they seek. Tweaking the search results makes the responses to modifications non-linear, and harder for the consumer to predict.

Consumers realize the the search results apply to the web that exists and that this web is limited in a number of ways, but they want the results to accurately reflect that reality, not some editorial reality.

I think that these problems will settle out as more people, and more types of people get on the web and start blogging, or otherwise creating content that links out to the world. If Robert Redford had a blog, and every Robert Redford fan blogged their responses to his posts and, I think most importantly, the various film and fan sites were really part of the web – linking out, taking part in the conversation – then Redford would quickly pass Scoble.

Until that time I hope the search engines resist the call to tweak for expectations based on non-web popularity. Who searches on “Robert” anyway?

Sorry, comments for this entry are closed at this time.