Do you need that much data?

Recently I read an article about a government agency archiving tweets that are relevant to the country. It is however unable to meet the demands of researchers worldwide who hope to access the archive as they estimated that a search among the first four years of tweets, from 2006 to 2010, would already take about 24 hours.

This brings back the problem of the overloading of data in the Internet, and if there is even a need to store data such as tweets.

Let me explain. I have always been a strong proponent of putting value to whatever information that needs to be stored. Tell me. How important is the tweet that basically tells you that person has woken up and is in the washroom? Obviously that tweet is not important but some will say that there might be important historical information of value within those billions of tweets.

So what do some people do? They take the easy way out. They keep everything. However when you try to ask them what is so important about that data set, I doubt they can answer you directly. They will insist on storing it though, just in case there is one super important tweet within the billions of tweet. Nevermind that you may never find it within the billions of tweets that you have stored.

The problem with the data trove in the social media is that it is almost impossible to tell whether the comment is serious, sarcastic, or a rant just by looking at the words on face value. If you cannot discern the value, why are you storing it?

My personal thinking is that if the human mind sometimes even have problems discerning fact from fiction, what makes you think the search engine can do better just by analyzing those words? The search engine can't.

Unless the writer put some meaningful keywords or tags to the writing, they are just data, not information. Since no useful information is tagged to the writing, it is useless to all because no one will understand the context of the writing.

Yet they want to store it. I can't understand the logic behind it. More business to the storage vendor most probably.