Enterprise Data Hubs, Architecture and Big Data

“Through 2015, more than 85 percent of
Fortune 500 organizations will fail to effectively exploit big data for competitive advantage” – Gartner BI Summit.

It doesn’t take genius to recognize that there is an increasing demand for information to improve shareholder value and gain competitive advantage by leveraging information, data and analytics as a strategic enterprise asset. The question is no longer about the importance of data but when, how, and where to leverage the asset. 

Big Data Architecture is essential for any necessarily complex structure, be it departmental or an enterprise. Every CIO that I have met has the pretty much the same strategy…. develop Enterprise Data Architecture capabilities to provide the following:

• Provide one version of the truth
• Business driven reporting solutions enabled by IT
• Assist continued growth and development of the business over the next 10+ years
• Provide timely and relevant information and reporting !
• Empower users with proactive decision making capabilities
• More time analyzing information and less time collecting and manipulating data

These requirements typically look easy on a Powerpoint slide but are daunting to execute.  Getting to the target state will be a multi-year transformation exercise especially in a world where the consumption models for data are shifting, the buyers/decision-makers (and users) are shifting and the apps+infrastructure is also shifting.

Multi-State Transformation

Data is an unlikely raw material but it’s a valuable asset if used correctly.  Data rationalization, standardization, and consolidation process forces organizations to understand their current data maturity level and move forward on the appropriate roadmap.

As firms undertake the Big Data journey, the enterprise architecture matures through several stages.

• Maturity Level 1: Data Silos (data trapped in applications – No EA Principle)
• Maturity Level 2: Standardized Technology (Applied EA Principle: Rationalization)
• Maturity Level 3: Optimized Data Core (Applied EA Principle: Virtualization, Consolidation, and Automation)
• Maturity Level 4: Data as a Service (Applied EA Principle: Shared Services / Cloud Computing)

To help you calibrate and self-diagnose… I would consider you at Maturity Level 1 if you exhibit the following symptoms:

• Multiple transactional systems for transactional and operational reporting
• Limited use of higher level BI applications
• Disjointed process and pain points around consolidating information
• Access to information limited to one silo at a time
• No single source of truth

Most organizations are at Maturity Level 1.

The first step in the transformation journey involves helping a company develop its Data architecture vision and operating model, with attention to cost, globalization, investiture, or whatever is driving the company strategically. Once that vision is in place, enterprise data architects can guide the organization through an iterative process of rationalization, consolidation, and eventually shared-services and even cloud computing.

Data Rationalization Yields Standardization

The data rationalization exercise helps an organization identify what structures and standards to move towards as they eliminate the complexities and silos they have built up over the years, along with the specific technologies that will help them get there.

Depending on the company, data rationalization could start with a technical discussion and be IT-driven; or it could start at a business level. For example, a company might have distributed operations across the globe and desire to consolidate and standardize its business processes. That could drive change in the application portfolio. Or a company that has gone through M&A might have redundant business processes to rationalize.

Rationalizing involves understanding the current state of an organization’s data asset portfolio and business processes, and then mapping business capabilities to IT capabilities. This is done by developing scoring criteria to analyze the current portfolio, and ultimately by deciding on the standards that will propel the organization forward. A more cohesive data architecture is the outcome of a rationalization exercise.

Standardized technology represents the second level of EA maturity. Organizations at this level have evolved beyond isolated independent silos. They have well-defined corporate data governance and policies, which yields measurable cost savings and the elimination of redundant systems and skill sets.

Data Consolidation Initiatives

Consolidation entails reducing the footprint of your data asset portfolio. That could involve consolidating the number of database servers, application servers and storage devices, consolidating redundant security platforms, or adopting virtualization, grid computing and related consolidation initiatives.

Consolidation may be a by-product of another technology transformation, or it may be the driver of these transformations. But whatever motivates the change, the key is to be in alignment with the overall business strategy. Enterprise data architects understand where the business is going so they can pick the appropriate consolidation strategy.

Execution Roadmap to a Enterprise Data Strategy

Companies that don’t manage their data wisely risk becoming overwhelmed, a poor position from which to extract value.  The 3 V’s – volume, velocity and variety  – can become a problem in a hurry if not managed.

One of the key outcomes of a data architecture exercise is the creation of a strategic data portfolio that continually keeps IT in line with where the business is going.

Having a roadmap is especially important when you move down the path to data-as-a-service strategy. For a company that has a very complex IT infrastructure and application portfolio, having a strategic roadmap helps the organization to move forward incrementally, minimizing risk and giving the IT department every opportunity to deliver value to the business.

Key Questions to Address in a Data Architecture effort

An exploding demand for data and connectivity brings both challenges and opportunities to IT organizations of every size. Survival requires answering basic questions like:

 What are the operational realities of data? Which Vector (volume, velocity, variety) is your focus?
 What are best practices in developing a multi-structured data asset catalog?
 How big data initiatives should change how you value and price IT across the business.
 Benchmark – how are competitors overcoming the architectural challenges associated with big data?
 What are some of the common pitfalls to avoid along the big data journey?

Do you have a Data Architecture Development Process? 

• Business Architecture
• Architecture Vision
• Current State
• Future State
• Roadmap
• Governance

Economic recovery and big data are driving a wave of massive technology refresh underpinned by:

• Desire for IT utility model
• Appetite for new applications
• The advancement of highly-scalable, low-cost compute

With this backdrop, have you rethought your data architecture?  If the elevators don’t work to transport people…the building no matter how beautiful is a failure.

Notes and References

1. According to TOGAF… the purpose of enterprise architecture is to optimize across the enterprise the often fragmented legacy of processes (both manual and automated) into an integrated environment that is responsive to change and supportive of the delivery of the business strategy.
2. Here’s a look at the impact of big data on social, political, and economic intelligence as well as digital exhaust and privacy implications of big data (source: OnlineBusinessDegree.org)

Related articles

• Enterprise Data Architecture and Big Data (architects.dzone.com)
• Gartner: IT to spend $232B on Big Data over 5 years (revolutionanalytics.com)
• Building your big data reference architecture (blogs.splunk.com)