The Mystery of Big Data

Sometimes I feel as The Effective Detective’s virtual (literally) assistant, I am led by The Detective from one mystery straight into another.

“Watson, have you been reading the technical papers lately – in particular regarding the issue of ‘Big Data’?”  The Detective asked.

“Ah yes, there seems to be quite the fascination with it now.” I answered.

“As always Watson, most writers have missed the point, and while larger businesses might be able to afford to dally with all of the mumbo jumbo discussed, many small business people and entrepreneurs are being left out in the cold and are missing an amazing opportunity.” The Detective started, jabbing his pipe in my direction for emphasis.

“How so?” I asked, encouraging him on.

“Elementary, my dear Watson. As always, our friends in the technical press have over-complicated the issue. Perhaps to impress everyone with their knowledge of jargon, or perhaps to help their advertisers sell their extremely expensive analysis software and database systems.

“Data is data my dear Watson, whether it is the ocean that pours into the largest of businesses or the stream that flows quietly into a smaller enterprise. The primary issue is not discovering new and fancy ways to organize, store, and access it, although that can occasionally be useful,” he paused briefly, building my anticipation for the main point.

“The primary issue is understanding what it is you are trying to accomplish with your data be it ‘big’ or otherwise, of course! Collecting customer and sales data has one purpose: to help you make more sales. You need to choose the data that is most important for that goal, not what some consultant or writer has decided is a best practice.” The Detective shot out the last two words and made a face like he had eaten a lemon.

“If you know what you are trying to accomplish, a bloody spreadsheet may turn out to be a perfectly acceptable way to store and retrieve your data! The smaller enterprises need to forget about ‘Big Data’ and start thinking about ‘The Right Data’” he concluded.

“And how do they go about identifying this ‘Right Data’?” I queried.

“Ah, Watson, that is a mystery for another day. Brandy?” replied The Detective with a slight smile.