A surprising source of collective intelligence

I’m a bit late in writing about this, but it’s too cool to pass up.  Last week, Google launched a new service, Google Flu Trends, that really demonstrates the power of new sources of information in the digital age.  Flu Trends attempts to warn users of regional outbreaks of the flu.  This enables hospitals, medical practitioners, and individuals to prepare.  It’s not exactly a new idea.  The C.D.C. publishes reports on outbreaks of influenza, based on data compiled from heath care providers.  Another web service, whoissick.org, combines user-reported illnesses with Google maps to show you the various bugs circulating in your area.  But Google Flu Trends may identify outbreaks more quickly because of the unique data source that it uses.

A Google team noticed that certain search terms, like “flu symptoms”, are much more common during flu season.  Only logical, right?  Google employees created a list of these types of searches, and compared the date/location of past searches with C.D.C. data on influenza trends.  It turns out that the number of people with the flu and the number of these types of searches are closely related.  This means that analyzing the numbers of influenza-related searches on Google should provide an estimate of the number of flu cases.  By looking at IP addresses, specific regions can be isolated.  This is a very nifty sort of collective intelligence based on data that is simply a by-product of Google’s primary function.

The sheer number of Google searches makes them an excellent source of collective intelligence.  Nielson Online estimated that 4.8 billion searches were made using Google in September 2008 in the US alone.  That’s roughly 160 million U.S. searches per day.  In addition to the volume of data, search engine results are fascinating sources of information because of their timeliness.  As the New York Times wrote:

[...] the data collected by search engines is particularly powerful, because the keywords and phrases that people type into them represent their most immediate intentions. People may search for “Kauai hotel” when they are planning a vacation and for “foreclosure” when they have trouble with their mortgage. Those queries express the world’s collective desires and needs, its wants and likes.

To me, this is a cool usage of data that most people don’t even realize they are generating.  However, the implications for privacy are a little bit frightening.  If you don’t believe me (and you have a Google account), check your web history.  Google saves every search you make, along with any web pages you visit from the search results.  A year’s worth of search data can create a surprisingly complete picture of a person’s life.  Imagine having access to that data for every single user.  I don’t want to be pessimistic about this.  For the most part, I think the possibility of exciting and useful projects like Flu Trends greatly outweighs the potential hazards of this data.  What do you think?

You can read the New York Times report on Google Flu Trends here.  The official Google Blogs also covered it here.  Statistics on the number of U.S. Google searches were pulled from the Nielson Online news release available here.

Comments

5 Responses to “A surprising source of collective intelligence”

  1. mattsadd on November 19th, 2008 11:16 pm

    I think that given the incredible pace of technological development and the accompanying hacks, it is impossible for anyone to know what the potential hazards are going to be. Then again, this could just be a “fear of the future” reaction. All in all, I’m in favor, as long as Big Brother learns to keep his mouth shut.

    Reply

  2. Sasha on November 23rd, 2008 10:52 am

    You’re definitely a dude… with like 71% certainty…
    http://www.genderanalyzer.com/?url=http%3A%2F%2Fmike.sapak.info%2F

    Google is both scary and impressive.. I’m still formulating my thoughts on this one..

    Reply

  3. Tim Marco on November 24th, 2008 11:10 pm

    This is crazy.

    I tend to disagree that the volume of data google is collecting is actually going to be a bad thing in the long run. I have the sense that the only threat (at least in the long-run) of all the data collection is privacy. And while it may seem a little weird to say that a loss of privacy is the ‘only’ threat we face, I honestly believe that our concerns with privacy are going to be viewed as quaint and tied to our point in history.

    I only say this because the net is more or less non-discriminatory, and the sheer breadth of information about absolutely anyone will mean that we’re all in this boat together. And to me, the real threat from a loss of privacy (as we’ve known it) is that a loss of privacy has always been asymmetrical.

    Big Brother knowing what you do is really scary. But to me, it’s a lot less scary if I’m just as capable of knowing what He has been up to, as well. And if everyone can have basically equal access to everybody else’s information, knowing about one individual person has no real power.

    There is, of course, a huge danger of states or companies or whatever gaining a monopoly on this information, which is scary. And I’m also sure that there’s going to be a pretty rough transition period in the coming years, but I’m not too worried in the long run.

    Then again, that is what Neville Chamberlain said.

    Also, I’m interested to hear your thoughts/experiences with the new generation of mobile phones. I’ve been thinking about them pretty much 24/7 since I got an iPhone, and I’ve been working on something to post about them. But I’m sure that you’ve got some pretty sweet ideas about the topic.

    Reply

    mike.sapak Reply:

    I hope I didn’t come off too negative. I don’t think it’s going to be a bad thing, either. I have concerns, but that’s about it. Hopefully if everyone has concerns, technological possibilities will be tempered by reason. In my opinion, that’s largely been the case with past technological revolutions, and I expect the same of the future. Neither the Utopian nor Dystopian visions will come to fruition. In the end, we’ll have the same old imperfect society.

    A good point about watching Big Brother. Steve Mann calls this sousveillance. Technology has makes it easy to watch from above, but it makes it equally easy to watch from below. A bad wikipedia article on the term: http://en.wikipedia.org/wiki/Sousveillance

    I have virtually no experience with smartphones. As much as I like geeky toys, I hate cell phones and the cost of a decent data plan has always turned me off. I sound like a curmudgeon, I know, but one of my favorite parts of life in Korea is not carrying a cell phone. I think I’m the only one in the country who doesn’t.

    Reply

  4. Tim Marco on November 24th, 2008 11:36 pm

    That is surprising. I thought that, by law, you had to have a phone with 10 gb/s data plans on you at all times in Korea.

    I see where you’re coming from, though. Until I got this iPhone I never had a cell phone with even a color display. But I’ve been very surprised at how much I use it-and 90% of the time it isn’t to make phone calls.

    I didn’t think that you came off too negative about it. The link was interesting, and a pretty good approximation of how I was thinking about it.

    By the way, at my work we have a program for employees to dress down on Fridays in exchange for a $5 donation to a charity suggested by somebody. I suggested One Laptop Per Child this month, so that’ll hopefully raise enough for five or six laptops, which would’ve never been paid for if you hadn’t told me about the program. So there’s some good Karma for ya.

    Reply

Leave a Reply