NSA PRISM – The Mother of all Big Data Projects

As a data engineer and scientist, I have been following the NSA PRISM raw intelligence mining program with great interest.  The engineering complexity, breadth and scale is simply amazing compared to say credit card analytics (Fair Issac) or marketing analytics firms like Acxiom.

Some background… PRISM is a top-secret data-mining “connect-the-dots” program aimed at terrorism detection and other pattern extraction authorized by federal judges working under the Foreign Intelligence Surveillance Act (FISA).  PRISM allows the U.S. intelligence community to look for patterns across multiple gateways across a wide range of digital data sources.

PRISM is unstructured big data aggregation framework — audio and video chats, phone call records, photographs, e-mails, documents,  financial transactions and transfers, internet searches, Facebook Posts, smartphone logs and connection logs – and relevant analytics that enable analysts to extract patterns. Save and analyze all of the digital breadcrumbs people don’t even know they are creating.

The whole NSA program raises an interesting debate about “Sed quis custodiet ipsos custodes.” (“But who will watch the watchers.”)

Data Monetization is the End Goal

The billion dollar question facing executives everywhere: How do I monetize my data? What small data or big data monetization strategies should I adopt?  Which analytical investments and strategies really increase revenue? What pilots should I run to test data monetization ideas out?

Data Monetization is the process of converting data (raw data or aggregate data) into something useful and valuable – help make decisions (such as predictive maintenance) based on multiple sources of insight.  Data monetization creates opportunities for organizations with significant data volume to leverage untapped or under-tapped information and create new sources of revenue (e.g., cross-sell and upsell lift;   or prevention of equipment breakdowns).

But, data monetization requires a new IT clock-speed that most firms are struggling with. Aberdeen Research found that the average time it takes for IT to complete BI support requests, with traditional BI software, is 8 days to add a column to a report and 30 days to build a new dashboard.  For an individual information worker trying to find an answer, make a decision, or solve a problem, this is simply untenable. For an organization that is trying to differentiate itself on information innovation or data-driven decision making, it is a major barrier to strategy execution.

To speed up insight generation and decision making (all elements of data monetization) business users are bypassing IT and investing in data visualization (Tableau) or data discovery platforms (Qlikview). These platforms help users ask and answer their own stream of questions and follow their own path to insight. Unlike traditional BI that provides dashboards, heatmaps and canned reports, these tools provide a discovery platform rather than a pre-determined path.

Also companies like Marketo which create marketing automation software are getting into the customer engagement and data monetization game. Their focus is to enable marketing professionals  find more future customers; to build, sustain, and grow relationships with those buyers over time; and to cope with the sheer pace and complexity of engaging with customers in real time across the web, email, social media, online and offline events, video, e-commerce storefronts, mobile devices and a variety of other channels. And in many companies, marketing knits these digital interactions together across multiple disconnected systems. The ability to interact seamlessly with customers across multiple fast-moving digital channels requires an engagement strategy enabled by data and analytic insights. 

Data-as-a-Service (DaaS)

Global CIO request — “I want to build a data-as-a-service offering for my data” to the rest of the organization.   The more advanced CIOs are asking – “should I build data science capabilities as a shared service?”

The CIO challenge is not trivial. Successful organizations today operate within application and data eco-systems which extend across front-to-back functions (sales & marketing all the way to fulfillment and service) and well beyond their own boundaries. They must connect digitally to their suppliers, partners, distributors, resellers, regulators and customers. Each of these have their “data fabrics” and applications which were never designed to connect, so with all the data-as-a-service and big data rhetoric, the application development community being asked to “work magic” in bringing them together.

Underutilization and the complexity of managing growing data sprawl is not new. But the urgency to address this is increasing dramatically during the last several years. Data-as-a-Service (DaaS) is seen as a big opportunity in  improving IT efficiency and performance through centralization of resources. DaaS strategies have increased dramatically  in the last few years with the maturation of technologies such as data virtualization, data integration, MDM,  SOA, BPM  and Platform-as-a-service.

The questions which are accelerating the Data-as-a-Service (DaaS) trend:  How to deliver the right data to the right place at the right time? How to “virtualize” the data often trapped inside applications?  How to support changing business requirements (analytics, reporting, and performance management) in spite of ever changing data volumes and complexity.

Data Management, AML, and KYC Analytics

To roadmap Wall Street priorities for 2013, we have been having an interesting set of meetings recently with MDs and leading architects in various banks and investment services firms.

Got the scoop on analytics projects they are investing in — Anti-Money Laundering (AML) monitoring, trade surveillance and Know Your Customer (KYC) analytics.  To enable AML and KYC initiatives…the big foundational investments in 2013 are around:

1)     Strengthening the Golden Sources – Security Master, Account Master and Customer Master.

2)     Various enterprise data management initiatives – Data Quality, Data Lineage, Data Lifecycle Management, Data Maturity and Enterprise Architecture procedures.

3) Reporting improvements via next generation Enterprise Datawarehouses (EDW) — Reporting on top of EDW addresses the core problems faced by Finance, Risk and Compliance when these functions extract their own feeds of data from the product systems through which the business is conducted and use differing platforms of associated reference data in support of their reporting processes. Lot of current investments are in the areas of Finance EDW which delivers common pool of contracts, positions and balances, organized on an enterprise wide basis and completed by anointed “gold” sources of reference data which ensure consistency and integration of information.

Crawl, walk, Run seems to be the execution game-plan as the data complexity is pretty horrendous. Take for instance, Citi alone….has approximately 200 million accounts and business in 160+ countries and jurisdictions.

The type of data challenges banks like Citi are wrestling with include:

Big Data Company Shakeout in 2013?

Big Data is the latest “next big thing” transforming all areas of business, but amid the hype, there remains confusion about what it all means and how to create business value.

Usually when there is so much hype…there is an inevitable boom-bust-boom cycle. Hence my question:  Is the inevitable Big Data shakeout coming?

Are we in a big data tech bubble? If you are an enterprise customer, how do you prepare for this? What strategies do you adopt to take advantage of the situation? Can you move from lab experiments to production deployments with confidence?

The Case of Drawn to Scale

Drawn to Scale, the four year-old startup behind Spire, shut down recently. Co-founder and CEO Bradford Stephens announced the news in a blog post. Drawn to Scale raised .93M in seed funding.

Spire is a real-time database solution for HBase that lets data scientists query Hadoop clusters using SQL. According to Stephens, the system has been by deployed by American Express, Orange Flurry, and four other companies.

Drawn to Scale showed that its technology was viable in enterprise environments and established a “presence against  competitors who raised 10-100x more cash,” but even that wasn’t enough to save the startup from its financial woes.

As Hadoop evolves and different layers of the data analytics stack get commoditized, specialized vendors like Drawn to Scale will have problems surviving.   SQL-on-Hadoop was a unique feature set…but over time it has become a must-have feature, that is becoming embedded in the stack – e.g., Impala in Cloudera CDH stack.  As a result, firms like Drawn to Scale once unique functionality becomes difficult to monetize.

Startup to Viable Ventures

The Big Data ecosystem is exploding with exciting start-ups, new divisions and new initiatives from established vendors.  Everyone wants to be the vendor/platform of choice in assisting firms deal with the data deluge (Data growth curve: Terabytes -> Petabytes -> Exabytes -> Zettabytes -> Yottabytes -> Brontobytes -> Geopbytes), translate data to information to insight, etc.

In both U.S and Europe, several billion dollars of venture money has been invested in the past three years alone in over 300+ firms.  Firms like Splunk had spectacular IPOs. Others like Cloudera and MapR have raised gobs of money. In the MongoDB space alone – a small market of less than 100M total revenue right now, over $2 Billion is said to have been invested in the past few years.

20 must read Infograhics on Big Data

Big Data emphasizes the exponential growth of data volumes worldwide (collectively, >2.5 Exabytes/ day).

Big Data incorporate the following key tenets: diversification, low latency, and ubiquity. In parallel, the emerging field of data science introduces new terms including, predictive modeling, machine learning, parallelized and in-database algorithms, Map Reduce, and data monetization.

A variety of infographics have been published around Big Data, Data Scientists.  Here is a compendium of some very interesting ones.

Big Data Big Opportunity Original Source

A Data Scientist Study Original Source

Sizing “Mobile + Social” Big Data Stats

Simply astounding:   Did you know that every 60 SECONDS, a tidal wave of unstructured data is being produced, consumed and archived.  As you read this ask yourself: what does this mean?

Social technology adoption by consumers is no longer an early adopter market — it’s a mainstream activity. Mobile is accelerating this trend. All this means a “new customer interaction” model powered by big data is emerging.

Why is big data analytics a good lens for creating value around social:

• New data is coming across multiple dimensions – demographic, geographic, psychographic, behavioral, socialgraphics
• Business decisions approach real-time. Time available to capture data is decreasing.  Analysis of increasing data volumes have to become faster. Operational excellence requires immediate action.  Real-time capture and action is where the state of the art is.
• Coupled with mobile and cloud, it means the emergence of a new Customer Interaction Model for corporations

All this data growth and value creation trends imply that data management, Big Data and real-time analytics is  a big focus in social and mobile data going forward.  Clearly a new style of IT is emerging (see this figure from HP Analyst Briefing which conveys the computing transformation message quite well).

Enterprise Data Architecture and Big Data

“Through 2015, more than 85 percent of
Fortune 500 organizations will fail to effectively exploit big data for competitive advantage” - Gartner BI Summit.

It doesn’t take genius to recognize that there is an increasing demand for information to improve shareholder value and gain competitive advantage by leveraging information, data and analytics as a strategic enterprise asset. The question is no longer about the importance of data but when, how, and where to leverage the asset. 

Innovation and Big Data: A Roadmap

With the influx of money, attention and entrepreneurial energy, there is a massive amount of innovation taking place to solve data centric problems in new ways.

The Oracle BI stack illustrates the landscape changes taking place from hardware to mobile BI apps.

As I see it, there are two clusters of “parallel” innovation: (1) technology/infrastructure centric and (2) business/problem  centric.

The interesting thing in the technology/infrastructure centric side is the multiple paths of innovation that are taking along different technology stacks shown below. The disruptive innovation is happening in parallel along 4 different fronts:

Apple IOS 6 Passbook: Enabling SoLoMoMe + Omni-channel Analytics

At the Analytics Executive Forum, I facilitated a session on Omni-channel analytics. It struck me how every leading consumer facing firm seems convinced that mobile is becoming the dominant B2C interaction channel.  Mobile is the gateway to insight based marketing and the “always addressable customer”….

Insight-based interactions –  The company knows who you are, what you prefer, and communicates with relevant, timely messages, using the power of analytical intelligence to detect patterns, decode strands of information and create meaningful offers and value.

The “always addressable customer.” This is a consumer who fits the bill on three fronts simultaneously: (1)

• Owns and personally uses at least three connected devices; (2)

Goes online multiple times throughout the day;  (3) 

• Goes online from at least three different physical locations

The opposite of insight-based is “spray-and-pray” marketing - The company has very limited knowledge about who you are, forgets what you prefer, and tries to reach you with off-target communications that alienate you – based on fragmented data, poor data quality and  inadequate integration, resulting in confusing, chaotic interactions.  A good example: “I have 2 million frequent flyer miles with your airline and still do not get any recognition, respect or value from this loyalty.”

As companies architect new insight based mobile use cases I suggest that they look at what is coming next. With IOS 6, Apple is delivering several new features. The most interesting is a new integrated time-saver feature called Passbook that I think will be a game changer in terms of user experience and value. Also Apple ditched Google Maps for their own in IOS 6 allows them to integrate user latitude/longitude coordinates combined with maps into a valuable location-aware platform for app development.

Retailers, banks and other customer facing firms/brands better pay attention. 100+ million iPhones are automatically getting this feature with the new OS upgrade making this a mega-disruptor in the coveted target segment everyone is chasing.