Search Engines

The inimitable Fred Wilson concluded yesterday in "Why Social Beats Search", that "[m]achines can help us find what is good. But with the help of machines, our friends and trusted sources can and will do that even better."  This statement capped off a fresh debate over the weekend about automated, keyword-driven "McContent" creation that started when Michael Arrington posted about "the end of hand crafted content". Richard MacManus also explored the same issues in "Content Farms: Why Media, Blogs, and Google Should Be Worried".

I find this discussion very intriguing because it's nearly a mirror-image to the still-unfolding story of the last big change in this space: How the volume and timeliness of social media has disrupted traditional media.  I explored this subject in-depth recently on ZDNet about how this same transformation is now happening more broadly to other industries as well.  Now we're full circle already: What went around with social media is coming around again rather quickly with McContent. The machines are in the upstart role this time and have the potential to displace social media "moms and pops" who might not be able to match the volume and speed at which automated content can be created.  That of course, depends on if you believe that machines can match the quality of handmade content.  And indeed, if quality ultimately matters as much as volume and timeliness. There's a balance here that I'm not sure we fully understand yet but I'm betting there's probably room for the full spectrum.  This will only be true, however, if we are prepared to accept that the online landscape and current ways of doing business are going to continue to evolve rapidly.

The premise of today's information abundance reaching an unsustainable place isn't a new one. Information overload is a  rapidly growing subject that a lot of smart folks are talking about these days.  One bright area however, and this is the point that Fred Wilson touches on, is that social systems might actually provide an effective filter that will separate the wheat from the chaff by decentralizing the expertise and work of content curation into a sort of crowdsourced collaborative process (an increasingly widespread approach.)  This could make it both scalable and sustainable and I do believe that social content curation is an important trend.  But it's only one step in the right direction as we head into the future dominated by truly vast information abundance. 

One holy grail of search is "search that finds you" just as and when you actually need it.  Encouragingly, I'm now starting to see this happen with social environments like Twitter where I've received more and more tweets lately in the vein of "@dhinchcliffe Thanks for the link, was just looking for this 10 mins ago!" This is the seed of a trend that could be exploited by a very smart company that created the right product design that systematically optimized social recommendations and content referrals into something so much more than it is today: ad hoc serendipity.  Will the company that does this be the ones with the largest active social networks, such as Twitter or Facebook?  Or perhaps Google will figure it out as a component of real-time search?  Or will it just as likely be someone that no one has heard of yet? If the history of the Web is any guide, it will come from a place we won't anticipate.

I also suspect that other forces are in the running and may end up limiting the impact of distributed social curation, or more likely co-opting it.  Emerging trends like Web Squared and its autonomic filters and recommendation systems powered by data shadows as well as advanced forms of Enterprise 2.0 BI are just as likely to provide the solution in the medium to long-term.  Either way, search is only going to get better and social will certainly improve it.  That's not to say social won't disrupt search, but it may only complement the changes happening more broadly.  One big question is whether social can be made to scale enough to be routinely effective for most users.  In the end, that's the big question in my mind: The output of machines can always exceed that of people and that's not necessarily a bad thing as long as we still get access to both results when we need them.