Innovation and Big Data: A Roadmap
With the influx of money, attention and entrepreneurial energy, there is a massive amount of innovation taking place to solve data centric problems in new ways.
The Oracle BI stack illustrates the landscape changes taking place from hardware to mobile BI apps.
As I see it, there are two clusters of “parallel” innovation: (1) technology/infrastructure centric and (2) business/problem centric.
The interesting thing in the technology/infrastructure centric side is the multiple paths of innovation that are taking along different technology stacks shown below. The disruptive innovation is happening in parallel along 4 different fronts:
- Enterprise Data Stack;
- The Hadoop Ecosystem (Big Data) Stack;
- Enterprise Information Management;
- BI platforms, Analytics Tools and Visualization.
The big enterprise data trend is that enterprise are moving beyond the traditional data warehouse. Relational database management systems (RDBMS) have been deployed as enterprise data warehouses (EDW) for analytic applications when most of the questions were known up front. Their care and feeding required a sophisticated, multistep process and a lot of time. This process supported the need for strong information governance, verifiability, quality, traceability, and security. The sweet spot for traditional EDW includes both running the same reports and queries and tracking the same set of metrics over time. But if the questions changed, things would break and big parts of the end-to-end process would require redeveloping — often starting with the collection of new source data.
While billions have been invested in traditional EDW architectures they are limited in addressing the new class of analytic applications. So a new wave of investment in next generation data architecture has just begun. Different firms are attacking different feature/functionality gaps or problems. All relevant but inter-dependent if you are a CIO or CTO.
The fact that the tools and technologies to tame data are improving and readily available today is a terrific thing. While the Big Data infrastructure is evolving rapidly, there is a tremendous amount of prototyping and piloting taking place on the business side.
The business innovation is happening along three fronts: (1) point solutions; (2) process solutions; (3) vertical specific solutions. We’re talking about innovations in business models and practices—everything from how a company generates insights to innovative ideas for starting new ones and creating new markets.
Sometimes new business models and data are converging in interesting ways around certain point solutions. Take for instance Splunk which solves the machine data problem. Under the covers Splunk is designed to manage log data. Splunk has built-in functions to understand time stamps and time series data as well as other fields related to log data. Given its focus, Splunk has a predefined a set of dashboards to track certain well-known metrics.
Technology and business innovation has to be supported by organizational innovation — how to get your organization to actually acquire, implement, and put the tools and technologies to use. The people, cultural, and political issues surrounding data are very big and very real.
Technology centric innovation invariably creates decision dilemmas between the “status quo” (e.g., Oracle, SAP or IBM stack) and emerging (e.g., Hadoop). Dilemmas emerge when there is tensions between two apparent opposite ideas or concepts. In business we face these dilemmas all the time: cost vs. quality, centralization vs. decentralization, stability vs. change, short term results vs. long term competitiveness. Dilemmas are dynamic but inevitable. They don’t go away. They must be managed by making choices and creating a comprehensive multi-year roadmap.
The challenge for management is how to integrate these various streams of innovation into a cohesive multi-phase roadmap that is relevant for their corporation. How to articulate and put their version of the innovation pattern together. Also how to version this pattern so the firm can execute without getting all tangled up in complexity.
What every corporation needs to do is create a relevant and unified roadmap that brings the technology and business elements together under one umbrella. This is essential for communicating and setting a directional strategy.
The emerging DATA opportunity is clear to most firms. Which path you take to innovation depend on two critical philosophical questions (or perspective and context):
- is Big Data disruptive or incremental for your applications and infrastructure?
- Is your approach a Platform-to-Solution (common services driven) or a Solution-to-Platform (business problem driven) approach?
We are in the early days of this virtuous cycle. Technology innovation and business model innovation are complementary. Technology standards and maturity will also encourage business-model innovation. Business model innovation fuels another round of technology investments.
Right Org Structure for “Data-as-a-Service”, “Information-as-a-service” or Analytics-as-a-service Innovation
Every organization requires custom-made data-driven solutions to a variety of business applications. This means an understanding of business challenges and delivering action oriented solutions. What world-class and best-practice firms do well is a powerful combination of statistical data mining, big data management capability and a consultative approach.
Success depends on having the right leadership and talent in place. Some ingredients include
- Strong talent pool enabled by well-developed hiring, training and performance management processes
- Analytical tools & techniques & methodology specifically applicable to your industry and context
- Ability to ramp up and down at short notice
- Integrated delivery resources across onshore and offshore offering economical cost structure
Ability to understand the data science value chain, handoffs, leverage points and talent needed is what makes the difference between good firms and great firms.
Execution of Innovation
All these have to be aligned with an effective and efficient solutions lifecycle – Design, Develop, Discover and Deliver.
- In the ‘design’ stage, the target is to understand the core business challenge and lay out an analytics solution. In this stage the sources of data, volume of data and resources available are examined
- The ‘develop’ stage consists of gathering and managing available data. In many cases, data from external sources are incorporated to make more meaningful inferences. This is followed by advanced data analysis and modeling with the use of a wide range of analytic software (SAS, CART, MARS, JMP, R, SPSS, Excel, etc.).
- The ‘discover’ stage is about leverging modeling approaches such as classification and regression trees, multivariate linear regression and other advanced techniques are used in building each model. With some amount of manual intervention the models are fine-tuned and revalidated to improve performance. Improved performance leads to more reliable business decisions based on relationships within the data.
- In the ‘deliver’ stage, the learned models are employed and the results are interpreted to form coherent business rules that are implemented in a client environment.
Every stage has to supported by a systematic quality assurance process to ensure the quality throughout the project design, development and delivery.
Summary
Big Data Analytics is all about technology & business model innovation & execution.
Almost all companies talk about the need to mine their customer and transaction data. Using a test-and-learn approach, leading firms tend to design, execute, manage, and measure market-facing DATA experiments. Every new idea has risk and the potential to be partly wrong and partly right, the goal is to quickly iterative-and-learn before committing to an expensive scaling of the idea.
DATA centric implies a new way of thinking about what a business does, and how it does it. However, what is not clear is the roadmap to becoming a DATA centric business. What is the complicating matters is the complexity and challenge of making phased investment decisions in a shifting technology and services landscape as the gap between the technology investment and the capture of value usually lags by several years.
Related articles
- Top 10 categories for Big Data sources and mining technologies (zdnet.com)
- What Do CIOs Need to Know About Big Data? (cochituatemedia.com)
- Big Data Development Challenges: Talent, Cost, Time (informationweek.com)
- How Big Data Startups Could Kill A $30 Billion Industry (businessinsider.com)
