Big data can fail, but still worth it

Yes, Big data can fail

refer to     story when big data goes bad

it provides two interesting anecdotes which show the real world implications of letting computers decide how to run business (which don’t end up the way expected).

One with an Amazon bid pricing and second with a T-shirt designing company,

But having worked with computers, programming and on business side, the remarkable story of Joshua looks like an everyday reality, nothing really to be surprised about..

but the core message remains relevant – Models on which we are building decisions may be flawed and can fail.

So it brings the question of “Data” and “model

we have GIGO (Garbage in garbage out ), no matter how good your model is, if your input data is junk, incorrect or incomplete the result/output would be the same.

Models take the second point, Now we have real-time very good quality data (by what ever metrics,) and if we have good models ideally it should give us good results,

In some places where we have scientific /mathematical discovery – say in supply chain, inventory levels, based on certain assumptions, we get good results.

But, what if those assumptions are not true,


Irrational behavior (of players) isn’t accounted for..

Simply put, what if the model is not capable enough.  we will have an sub-optimal result,

Till now these things (most of automation) happened inside an organization

(e.g. say HRMS / HR department deciding ratio of team members to a manager say based on incorrect model of revenue by department or so say team size or even some arbitrary number) – The impact would be felt by business, but given the myriad factors it’s almost not possible to blame a singular cause.

Similar case could be true say while distributing marketing budget/ targets/ territories, when done by use of models,

or in Finance – using capital allocation models, valuation models

and model of models especially at enterprise or CEO level while forecasting sales , costs and earnings per share, which are derived from many other models.

This problem is not new, in fact very fabulously tackled by engineering folks, who assemble thousands of components to make cars to mobile phones,

More like Sig-sigma methods,

Big data modelers and data gatherers, following similar principles (tweaked for Big data) can avoid (hopefully) big failures. but remember we didn’t make high quality culture in a day or processes don’t meet six sigma level on day 1.. and despite this high quality, there still are failures (even though it is at six sigma levels).

So at a society and as a big data proponents we need to understand that there will be some failures (and don’t promise it as a fool-proof technology), but a technology worth using..


One thought on “Big data can fail, but still worth it

  1. Pingback: Big Data’s Little Secret | Grandiose Data Delusions

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s