What is the BIG data?

Tyler told me that he is going to present on a panel about "Big Data." "What is big data?" I asked him and he didn't give me a satisfactory definition. For the past two years, every time I heard about the phrase "big data", it reminded me of the "Big Salad" episode of Seinfeld.

ELAINE: Um, hum, I don't know.. . . A big salad?
GEORGE: What big salad? I'm going to the coffee shop.
ELAINE: They have big salads.
GEORGE: I've never seen a big salad.
ELAINE: They have a big salad.
GEORGE: Is that what I ask for? The BIG salad?
ELAINE: It's okay, you don't…
GEORGE: No, no, Hey I'll get it. What's in the BIG salad?
JERRY: Big lettuce, big carrots, tomatoes like volleyballs.
GEORGE: (???), we'll see you in a little while.

I felt that I sort of know what they were referring to but I cannot really take in the idea of generalizing all the special needs of complex, messy, multi-disciplinary, multi-source, incomplete, biased-designed, ... data into one word "BIG." Yes, they are big. So big that getting them into a form that can be handled by traditional computational mechanisms becomes hard. Innovations are needed to either allow nearly loss-less reduction of the BIG data to a manageable size, or lead to new computational mechanisms for BIG data. This is the foremost step of any big data project. This part of the battle is more computer science than Statistics.

To a statistician, the whole new era of "BIG data" feels like a call for more dynamic models that can capture trends in space and time, better model-based tools for integrating multiple, individually incomplete, data sources, systematic data analysis tools that can mitigate design and sampling biases in the huge collection of existing data. It is like jigsaw puzzles. A small data is like a small puzzle and a BIG data is like a gigantic, bigger-than-a-football-field-kind of a puzzle. Exciting, amazing, fun (?), and intimidating. All our old puzzle-solving tricks won't work well but some fundamental principles still prevail, as long as it is indeed, as we understand, a jigsaw puzzle but not something else.  

America's Safest Cities -- weather wise

In this age of extreme weather, where will you be the safest?

The Hype Cycle

Just heard about this phrase at a workshop. Found it pretty interesting. [Read more on Wikipedia]

The really scary phone

We all sort of know it is hard to keep an entirely private life in today's world. But we have no idea how much of our personal life has been infiltrated until a recent "discovery" of iphone location data.

According to an WSJ article, Apple and Google didn't realize the values in this gigantic pool of data until recently. There is also the recent discovery that social networking and behavioral data can be automatically logged by our cell phones. The use of the data? Better marketing. In a recent article by WSJ, it is pointed out that our phone can actually log our body language during phone calls, the same technology that has made our iphone games more fun. Therefore, statistical methods can (potentially) be applied to study the association between body language and the content of phone calls. A always walks around while calling B, and sits at a desk while calling C. Different social ties, possibly different social influence, and thus different marketing values. Existing research has already achieved a level where  a trained learning method can predict whether two people are discussing politics without actually logging the conversation. This prediction was based on some call history, location, timing of the call, etc.

This seems to suggest a new wave of interesting data, gigantic in size and complex in nature, attractive to both machine/statistical learning researchers and social/behavioral researchers.

PS: Michael Malecki posted the R codes he used to analyze his own iphone data. I can't wait, but yet to find time, to try this on my own iphone data.

Happy 100th Birthday, my dear Tsinghua!

Tsinghua University is celebrating 100 years since it's foundation on 4/24. The beautiful campus of Tsinghua is my dearest hometown.