Predictive Analytics – A Project or a Program?

Our AMEX credit card was recently compromised.  Someone got hold of the card information and Petro Canada charges started to rack up.   Amex spotted this suspicious pattern and immediately initiated a fraud alert thru multiple touch points.

What does your credit card company know about you?  A lot…maybe more than your spouse. A study of how customers of Canadian Tire were using the company’s credit cards found that 2200 of 100,000 cardholders who used their card at drinking places missed four payments within the next 12 months. By contrast, only 530 of the cardholders who used their card at the dentist missed four payments within the next 12 months. So drinking is a predictor of credit risk.

Predictive analytics is not a fad. It’s not a trend.  In a real-time world, Analytics is a  core business requirement/capability.  However, many organizations flounder in their efforts not because they lack analytics capability but because they lack clear objectives. So the first question is, What do you want to achieve?

Analytics so far has largely been a departmental ad hoc activity.   Even at the most sophisticated corporations, data analytics is  a cumbersome affair. Information accumulates in “data warehouses,” and if a user had a question about some trend, they request “data priests/analysts” to tease the answers out of their costly, fragile systems.  This resulted in a situation where the analytics are done looking in the rearview mirror, hypothesis testing to find out what happened six months ago.

Today it’s possible to gather huge volumes of data and analyze it in near real-time speed. A retailer such as Macy’s  that once pored over last season’s sales information could shift to looking instantly at how an e-mail coupon impacts sales in different regions.  Moving to a realtime model and also building an enterprise level “shared services” model is going to be the next big wave of activity.

Proactive Risk Management – New KPIs for a Dodd-Frank World

The financial crisis of 2007–2011 is driving widespread changes in the U.S regulatory system. Dodd-Frank Act addresses “too big to fail” problem by tightening capital requirements and supervision of large financial firms and hedge funds. It also creates an “orderly liquidation authority” so the government can wind down a failing institution without market chaos.

Financial institutions will be spending billions to strengthen, streamline and automate their recordkeeping, risk management KPIs and dashboard systems. The implications on Data Retention and Archiving, Disaster Recovery and Continuity Planning have been well covered. But leveraging Business Analytics to proactively and reactively manage/monitor risk and compliance is an emerging frontier.

We believe that Business Analytics and real-time data management are poised to play a huge role in regulating the next generation of risk and compliance management in Financial Services industry (FSI).  in this posting, we are going to examine the strategic and structural challenges, the dashboards and KPIs of interest that provide feedback, and what an effective execution roadmap needs to be for every organization.

Are you one of these — Data Scientist, Analytics Guru, Math Geek or Quant Jock?

“The sexy job in the next ten years will be statisticians…”
‐ Hal Varian, Google

Analytics Challenge — California physicians group Heritage Provider Network Inc. is offering $3 million to any person or firm who develops the best model to predict how many days a patient is likely to spend in the hospital in a year’s time. Contestants will receive “anonymized” insurance-claims data to create their models. The goal is to reduce the number of hospital visits, by identifying patients who could benefit from services such as home nurse visits.

The need for analytics talent is growing everywhere. Analytics touches everyone in the modern world. It’s no longer on the sidelines in a support role, but instead is driving business performance and  insights like never before.

Job posting analysis indicate that market demand for data scientists and analytics gurus capable of working with large real-time data sets or “big data” took a huge leap recently.  The most common definition of “big data” is real-time insights drawn from large pools of data. These datasets tend to be so large that they become awkward to work with using on-hand relational database tools, or Excel.

It’s super trendy to be labeled “big data” right now – but that doesn’t mean the business trend’s not real.  Take for the instance the following scenario in B2B supply chains. Coca-Cola Company is leveraging retailers’ POS data (e.g., Walmart) to build customer analytical snapshots, including mobile iPad reporting, and enable the CPFR (Collaborative Planning, Forecasting, and Replenishment) process in Supply Chain. Walmart alone accounts for $4 bln of Coca-Cola company sales.

Airlines, hotels, retail, financial services and e-commerce are industries that deal with big data. The trend is nothing new in financial services (low latency trading, complex event processing, straight thru processing) but radical in traditional industries.  In trading, the value of insights depends on speed of analytics.  Old data or slow analytics translate into losing money.

As data growth in business processes outpaces our ability to absorb, visualize or even process, new talent around Business Analytics will have to emerge. New roles such as Data Scientists, Analytics Savants, Quant Modelers are required in almost every corporation for converting the growing volumes of data into actionable insights.

Look at these data stats.

Harry Potter, The Elephant, The FBI and The Data Warehouse

In the ancient Indian parable of the elephant, six blind men touch an elephant and report  six very different views of the same animal. Compare this scenario to a data warehouse that is getting data from six different sources. “Harry Potter and the Sorcerer’s Stone” as a field in a database can be written as “HP and the Sorcerer’s Stone” or as “Harry Potter I” or simply – “Sorcerer’s Stone”.  In the data warehouse these are four separate movie titles.  For a Harry Potter fan, they are the same movie.  Now increase the number of movies to cover the entire Harry Potter series and further include fifty  languages.  You now have a set of titles which may perplex even a real Harry Potter aficionado.

What does this have to do with data analytics?

New Tools for New Times – Primer on Big Data, Hadoop and “In-memory” Data Clouds

Data growth curve:  Terabytes -> Petabytes -> Exabytes -> Zettabytes -> Yottabytes -> Brontobytes -> Geopbytes.  It is getting more interesting.

Analytical Infrastructure curve: Databases -> Datamarts -> Operational Data Stores (ODS) -> Enterprise Data Warehouses -> Data Appliances -> In-Memory Appliances -> NoSQL Databases -> Hadoop Clusters

———————

In most enterprises, whether it’s a public or private enterprise, there is typically a mountain of data, structured and unstructured data, that contains potential insights about how to serve their customers better, how to engage with customers better and make the processes run more efficiently.  Consider this:

• Online firms–including Facebook, Visa, Zynga–use Big Data technologies like Hadoop to analyze massive amounts of business transactions, machine generated and application data.
• Wall street investment banks, hedge funds, algorithmic and low latency traders are leveraging data appliances such as EMC Greenplum hardware with Hadoop software to do advanced analytics in a “massively scalable” architecture
• Retailers use HP Vertica  or Cloudera analyze massive amounts of data simply, quickly and reliably, resulting in “just-in-time” business intelligence.
• New public and private “data cloud” software startups capable of handling petascale problems are emerging to create a new category – Cloudera, Hortonworks, Northscale, Splunk, Palantir, Factual, Datameer, Aster Data, TellApart.

Data is seen as a resource that can be extracted and refined and turned into something powerful. It takes a certain amount of computing power to analyze the data and pull out and use those insights. That where the new tools like Hadoop, NoSQL, In-memory analytics and other enablers come in.

What business problems are being targeted?

Why are some companies in retail, insurance, financial services and healthcare racing to position themselves in Big Data, in-memory data clouds while others don’t seem to care?

World-class companies are targeting a new set of business problems that were hard to solve before – Modeling true risk, customer churn analysis,  flexible supply chains, loyalty pricing, recommendation engines, ad targeting, precision targeting, PoS transaction analysis, threat analysis, trade surveillance, search quality fine tuning,  and mashups  such as location + ad targeting.

To address these petascale problems an elastic/adaptive infrastructure for data warehousing and analytics capable of three things is converging:

• ability to analyze transactional,  structured and unstructured data on a single platform
• low-latency in-memory or Solid State Devices (SSD) for super high volume web and real-time apps
• Scale out with low cost commodity hardware; distribute processing  and workloads

As a result,  a new BI and Analytics framework is emerging to support public and private cloud deployments.

The Vendor Landscape of BI and Analytics

“In God we trust, all others bring data”

—————————-

The “Raw Data -> Aggregated Data -> Intelligence -> Insights -> Decisions” is a differentiating causal chain in business today.  To service this “data->decision” chain a very large industry is emerging.

The Business Intelligence, Performance Management and Data Analytics is a large confusing software category with multiple sub-categories — mega-vendors (full stack, niche vendors, data discovery, visualization, data appliances, Open Source, Cloud – SaaS, Data Integration, Data Quality, Mobile BI, Services and Custom Analytics).

But the interest in BI and analytics is surging. Arnab Gupta, CEO of Opera states why analytics are taking center stage, “We live in a world where computers, not people, are in the driver’s seat. In banking, virtually 100% of the credit decisions are made by machines. In marketing, advanced algorithms determine messages, sales channels, and products for each consumer. Online, more and more volume is spurred by sophisticated recommender engines. At Amazon.com, 40% of business comes from its “other people like you bought…” program.”  (Businessweek, September 29, 2009).

Here is a list of vendors who participate in this marketspace:

Gartner says – Business Analytics a $14.1 Bln market

The term “business intelligence” (BI) dates back to 1958, when IBM researcher Hans Peter Luhn coined the term in an IBM Journal article.

However, it took until 1980s when decision support systems (DSS) became popular and mid 1990s for BI started to emerge as an umbrella term to cover software-enabled innovations in performance management, planning, reporting, querying, analytics, online analytical processing, integration with operational systems, predictive analytics and related areas.

Gartner 2014 magic quadrant shows the key players in the BI market.  The different players are differentiated based on five abilities— ability to handle large volumes of data, ability to deal with data velocity, variety (structured and unstructured), visualization capabilities and domain/vertical specific accelerators.

Market Evolution

Analytics is becoming three different markets.  First of all, there is the BI market which is actually going through quite a bit of change itself. This is a more consolidated market than we have seen in the past and there is a tremendous amount of work being done by Oracle, SAP, IBM and others to kind of retool it for the next generation of BI. So it is a growing market, lots of upgrade, replatform, modernization demand, lots of clients who are finally realizing that the tools (visualization etc.) are ready to give them some of the capability that they have historically cared about.

The second part of the market is what is called Advanced Analytics. Here you need PhD level data scientists who have backgrounds in machine learning, industry specific domain modeling, and different types of data science who can apply that in a very specific way to specific industry problems. This is a rapidly growing part of IT Services. Also, there are just not enough data scientists to go around.

The third part of the market is Analytics as a Service.  This is about leveraging software-as-a-service platforms as opposed to on-premise. This is about a business model that is more like Business Process Outsourcing (BPO). Clients buy business outcomes; they don’t buy transactions and FTEs.

The analytics market has thousands of boutique consultants who are specialists in particular industries or specific technologies. It includes all the major technology providers, who are all trying to advance their business and capabilities that they are bringing to the market. And then there are vendors  who are just bringing sheer capacity of data science skills to the market and they are coming in from a completely different angle of basically just renting the expertise of their data scientists into the market.

The market is incredibly fragmented. We are in the early stages of growth in the market. Every single one of our clients is building this capability internally and they are looking for more services from vendors, because the opportunity to apply analytics is in every single one function whether it is a customer analytics, industrial Internet, e-commerce platform, is growing. Analytics is embedded into literally every single business interaction.

BI, Analytics [and Big Data] Market Sizing

More recently to support a new generation of cost cutting and growth initiatives, corporations are investing heavily to gain near real-time actionable insights (historical and predictive), and from a mix of disparate spreadsheets  and myriad of systems (legacy, internal silos, customer facing, suppliers, partners, etc.).