Monday, October 17, 2011

Just Enough Business Intelligence!

This post is a culmination of several aspects that I have been either reading/observing such as:

1. The book titled "The Two second advantage" - a Book by Vivek Ranadive and Kevin Maney
2. Another reputed book "Flow" - by Psychologist Mihaly. Interestingly, the Flow concepts are referred in the first book - The Two Second Advantage as well.
3. An interesting, but offensive post titled "Kill Your Datawarehouse" by Splunk. I evaluated Splunk in my previous organization and I should say its a radical departure from traditional data storage Architectures.
4. Big Data and its implications on the Enterprise.

Traditionally, DW/BI systems have been expensive, untimely and far from realizing business benefits!. In the traditional RDBMS world of DW/BI architectures, Predictive Analytics is being touted as the next big thing!. But, the reality in Enterprise IT has been the reverse. Have blogged about this state of BI in Enterprise IT few months ago.

Coupled with Big Data, Information Overload is becoming the biggest challenge being faced by Enterprises today. No amount of computational advances in the form of Cloud/Grid is going to help solve the information overload issue. It is not the question of computation, but the ability to "qualify" and "condense" the big data to formulate high-level abstract models that can be processed quickly is the key. The Two Second Advantage book calls this process as "Chunking". It further adds that chunking is the method that Human Brain uses to absorb information, learns and make the knowledge as instrinsic part of the consciousness so that it becomes a natural activity for repeated applications. (e.g. Car Driving). Learned in this way, Human brain becomes specialized, learns the ability to be reflexive/adaptive and gains the ability to predict the next few steps. And that's the key!. Only few steps not too far!

The book tries to create a link between computational science and Neuro science. It claims the next generation of Business Intelligence does not lie in long predictive analytics, but predicting just-enough, next set of actions.

For example:

1. Who is the next best talent that we can hire from market?
2. What is the in-houst best talent that is subject to attrition risks?
3. Who are the customers/partners that could default in payments?
4. Who are the promising prospects that would convert to customers?
5. What would the customer buy next?

These new business requirements call for new capabilities such as ability to to scan data from multiple sources - structured/unstructured, Internal/External, ability to condense large amount of data into abstract models, drill-down to details on demand, ability to query and navigate the data networks in quick cycles, and finally the ability to predict just enough next few actions.

This doesn't mean we will dump our traditional DW/BI systems. But, We need to be certainly prepared for hybrid new architectures that would blend the conventional data stores with new age data formats!

Would like to conclude this post with another correlation that I recently read - a poem by former Prime Minster of India VP Singh. He is a versatile personality and one of his skills is in poetry. In his poem on Astrology, the Astrologer talks about Singh's fortunes and challenges in the future, VP singh politely asks the astrologer "What is the dream that I would get in tonight's sleep?". Needless to say Astrologer goes speechless! :-)

Any predictions - What is the next topic that I would blog about?

I know it! :-). If you can predict, I promise to compliment you in my next post.