Explosion of Data

To succeed in today's data rich environment, Business Intelligence (BI) experts are looking to un-archive data and put it back into the analytics domain.  "Life Events" are now being analyzed using super-computing resources. Data about having children, buying a new car, or even simple TV purchases do not happen every 6 months by the same consumer. To understand your customer, their family, their patterns over the full life of your relationship with them, data warehousing experts want to include up to 10 or 20 years of data into their analytic environments. All they need is good performance, and a beneficial return on investment (ROI) to receive budget for their programs.

Unfortunately, today’s analytic environments are struggling to maintain a ROI to keep a meager 6 to 13 months of consumer data in play.  Many Fortune 500 companies have data warehouse (DW) systems that archive this time-series customer data for two reasons: First is price; today’s warehouse solutions cost between $100K and $200K per terabyte of user data, making long time series environments completely cost prohibitive.  Companies are looking to cut costs, approaching the CIO with a project asking for a 10x budget increase to put 10 years of data into play versus the 6 to 13 months they currently have at their disposal, would take a brave soul indeed!  Second is “performance at scale”; all existing solutions that allow scale up to and beyond the petabyte range are good at operational business intelligence (BI) but perform poorly for ad hoc query access. These systems are tuned for canned queries and reporting, but the PhDs and statisticians that perform “wide and deep” analytics don’t want to be told what queries to ask or have restricted access patterns. They want to iterate on the “ask, learn, ask more, learn more” cycle without restrictions to the data. Further, they cannot wait hours or days to get the results they are looking for because they will lose their train of thought or just waste hours of time, restricting progress. If the analyst can stay focused, and get to the “aha” moment faster, their ROI targets can be met with one really good query. “Competing on Analytics” is the key to success today and existing solutions are putting up two roadblocks for larger, faster, cheaper data environments.

There are many other examples of where large data should be leveraged, but is not. Medical research and ongoing drug trials may or may not show progress or side effects for years. Wouldn’t it be nice to record your vital measurements with more regularity and keep them available to researchers for longer periods of time? (aka blood pressure, heart rate, side effects, positive results, etc), Add to that being able to combine this time-series with other clinical research data and combine that result with the new wave of personal genomic data to find links between research, what is actually happening, and what genomic codes might be linked to success or failure. Obviously, this would result in massive “joins” of petabytes of data and would be prohibitive to many DW/BI solutions either because of size, performance of results, cost, or more likely– all of the above.