Kaggle is going to take your job

Ok, maybe not, but I was just introduced to kaggle, which is sort of like oDesk and a lot like topcoder: people post a problem and other people compete to win a prize by solving it most effectively. Kaggle is devoted to data analysis problems. For example, there is currently a contest to win 3 million if you can "Identify patients who will be admitted to a hospital within the next year, using historical claims data." Another contest is to "Identify people who have a high degree of Psychopathy based on Twitter usage," for $1000. Even at those low stakes, there 113 teams competing.

If it's like topcoder, it will be a remarkably cheap way to get great solutions to data problems. I'm not sure why (I guess competing analysts overestimate the probability of winning?), but this is what Karim Lakhani tells me. I'm already working on a paper using topcoder, but I'm now wondering -- half seriously -- if I could outsource my data analysis as well. I probably couldn't, but only because I enjoy this kind of work myself and I'd miss it.

If you work with data, will Kaggle eventually take your job? Well, it makes it easier for people with data to find the best analysts out there. I'm not directly in competition with these folks, since none of these companies would come to me with their consulting gigs anyway. But it does suggest that the consulting market for quantitative social scientists is on the verge of being restructured. I'm excited about that -- I'm willing to bet that we'll learn more this way.

Posted by Richard Nielsen at July 2, 2012 8:55 PM