Big Data, Analytics and KPIs in E-commerce and Retail Industry
The change in consumer behavior and expectations that e-commerce, mobile and social media are causing is hugely significant – big data and predictive analytics will separate brand/retail winners from losers. This won’t happen overnight but the transformation is for real.
Retail Industry makes up a sizable part of the world economy (6-7%) and covers a large ecosystem - E-commerce, Apparel, Department Stores, Discount Drugstores, Discount Retailers, Electronics, Home Improvement, Specialty Grocery, Specialty Retailers and Consumer Product Goods suppliers.
Retail is increasingly is looking like a barbell – a brand oriented cluster at the high-end, a very thin middle, and a price sensitive cluster at the low end. The consumerization of technology is putting more downward pricing pressure in an already competitive “middle” retail environment. The squeeze is coming from e-commerce and new “point, scan and analyze” technologies that give shoppers decision making tools — powerful pricing, promotion and product information, often in real-time. Applications in iPhones and Droid, like Red Laser can scan barcodes and provide immediate price, product and cross-retailer comparisons. They can even point you to the nearest retailer who can give you free shipping (total cost of purchase optimization). This will lead to further margin erosion for retailers that compete based on price (a sizable chunk of the market in the U.S, Europe and Asia).
Data analytics is not new for retailers. Point of sale transactional data obtained from bar-codes first appeared in 1970s. A pack of Wrigley’s chewing gum was the first item scanned using Universal Product Code (UPC) in a Marsh Supermarket in Troy, Ohio in 1974. Since then, retailers have been applying analytics to get even smarter.
More recent use cases of retail analytics include:
- Corporate Express is using its market basket application to track the baskets of all of its online customers, and can identify which SKUs are most profitable or poor performers when purchased in combinations.
- Liz Claiborne analyzes financial, sales, and inventory data across 30,000 points of sale worldwide — department stores, specialty stores, and catalogs/Internet — to uncover revenue optimization opportunities.
- Limited Brands currently uses MicroStrategy for customer analysis, financial analysis, inventory management, sales analysis, and category management reporting across all of its brands (including Victoria’s Secret, Express, Bath & Body Works, and Henri Bendel).
- Netflix is a heavy user of big data… The company issued a $1M analytics challenge in October 2006: come up with recommendation software that could do a better job predicting the movies customers would like than Netflix’s in-house software, Cinematch. To qualify for the prize, entries had to be at least 10 percent better than Cinematch. (for more detail see notes at end).
- BestBuy had a clickstream analysis problem — 3.5 billion records, 71 million unique cookies, 1.7 million targeted ads required per day. How to make sense of this data? They hired Razorfish to implement a analytic solution on Amazon Web Services and Elastic MapReduce. Solution was a 100 node cluster on demand; processing time was reduced from 2+ days to 8 hours.
In a business with razor-thin margins, small, mid and large retailers are constantly looking for the performance improvement edge. Retailers are racing to become better at slicing and dicing reams of data they collect from multiple channels – online, stores, call center, catalogs and smartphones. They are using business analytics to get insight into retail KPIs (outcome measures) or Metrics (operational activities that serve as leading or lagging indicators). The goal is to drive significant topline and bottomline improvements through improved merchandising decisions, more timely responses to information requests, cost reduction initiatives, enhanced employee productivity, and better service to customers.
Types of Big Data Analytics in Retail
We analyzed the typical retailers and found some interesting patterns of how raw data is being converted into signals and actions. Most of the Retail Analytics and Big Data retail tend to cluster around the following areas:
- Customer Analytics and KPIs
- Merchandizing KPIs — Eliminate Stockouts and Overstocks
- Store Operations Analytics and KPIs
- Vendor and SKU Management Scorecards and KPIs
- Marketing Analytics and KPIs
- Returns, Fraud and Loss Prevention Analytics and KPIs
It’s important to distinguish between retail KPIs and metrics. We view KPIs as clusters of metrics that in aggregate provide a report card. Metrics on the other hand are simply snapshots of an activity. Think of KPIs as overall health vs. metrics as taking temperature or measuring shortness of breadth.
KPIs are like latent variables statistics (as opposed to observable variables). KPIs are variables that are not directly observed but are rather inferred (through a mathematical model) from other variables that are observed (directly measured).
Customer Analytics and KPIs
Objective: Understand Your Most Valuable Customers. Target Them to Maximize Profits and Loyalty
Every retailer is focused on understanding and leveraging all customer touch points, from granular in-store transactional data to online store and web search information to mobile data, in order to most effectively target customers, improve repeat purchase, drive cross-sell. All to maximize profits and customer loyalty.
Providing state-of-the-art customer analytics, retail business analytics enables retailers to integrate and analyze the ever-expanding universe of data to obtain valuable insights about customer behavior, buying patterns, and consumer preferences.
|Area||Type of Analytical Insights|
|Customer and Market Basket Analytics and KPIs –Understanding and leveraging all customer touch points||
What Can You Do With Customer Analytics?
The most important assets for every retailer are customers. The challenge is to look for ways optimize the value from each relationship.
- Discover who your customers are. Identify their unique characteristics, and know what products they purchase and why.
- Expectation and sentiment tracking.
- Track the impact of promotions on customer baskets as they move through your stores and provide a holistic view of all of their behaviors.
- Tap into their transactional data to connect the dots between customers, stores, products, and promotions.
- Move beyond basic segmentation, personas, and list pulls to create targeted microsegments – highly-refined groups of customers based on user-defined characteristics.
Once you understand your customer at this level, you can align effective tactics to tailored customer segments to surround them with a holistic shopping experience. Enable business users in marketing and merchandising to seamlessly integrate insight with promotion planning, weekly flyers, planograms, store assortments, and more. Boost offer performance. Drive larger baskets. End goal – increase sales, customer profitability, and customer loyalty.
Objective: Significantly reduce costs, eliminate the expense of stock-outs and overstocks, and make powerful, rapid decisions
Retailers face immense pressure to balance inventory levels with in-stock positions while maintaining assortments that are relevant and fresh. Buyers and merchandise managers need to be constantly on-the-go either in team meetings, traveling overseas for a buying trip, or working hand-in-hand with vendors.
Whether at their desk or on-the-go, this job requires mobility and constant connectivity to respond to consumer demand, seasonal trends, and fashion styles, all to ensure that customers get the right products.
|Merchandise Analytics and KPIs — Eliminate Stockouts and Overstocks||
The Merchandise Analysis scenarios are designed to improve inventory decisions. Users can quickly accelerate shipments by evaluating top-selling products, make markdown decisions based on seasonal sell-through, cancel shipments for bottom-selling products, and communicate more effectively with vendors.
Store Operations Analytics and KPIs
Objective: Keep Store Managers on the Selling Floor, Not Behind a Desk. Give Store Operations the Right Information at the Right Time to Make the Right Decisions.
Store managers are more effective when they are out on the selling floor, not in a back office. The old paradigm of paper reporting or even web/desktop is now becoming obsolete. Mobility empowers field managers in stores to make near real-time decisions regarding inventory, promotions, pricing, staffing, and store design.
Store managers can now run daily analyses of store performance where they can aggregate by Store sales, SKU sales and sales per employee. In some advanced retailers, the data is refreshed in real-time so they can deal with just-in-time replenishment more cleverly.
Mobile store operations reporting keeps the store managers where they should be: in the store, on the selling floor. Store managers can see, at regular intervals throughout the day, how their sales are performing compared to last year, giving them the information and tools to improve performance.
|Area||Type of Analytical Insights|
|Store Operations Analytics and KPIs||
Store traffic information delivers detail on a store’s performance: Units per Transaction, Average Transactions (Units and $), Conversion Rates, and Sales per Payroll Hour are all provided in the mobile BI app.
Relevant, filtered information delivered when and where it is needed helps store personnel identify anomalies in their operations and drive immediate action in the areas of labor scheduling, customer service, merchandise display, promotional signage, potential stock shortfalls caused by local market conditions, and more.
Vendor and SKU Management Scorecards and KPIs
Objective: Analyze Vendor Performance, Drive Improvement, and Strengthen Negotiations. Improve Performance Across the Supply Chain
Managing vendors in today’s environment requires you to know what is happening in the supply chain real-time, enabling you to make rapid decisions to optimize performance and to strengthen your negotiations on price and quality.
|Area||Type of Analytical Insights|
|Category Management KPIs||
A comprehensive Vendor Scorecard provides the right information to take the right action at the right time. A scorecard helps jointly improve performance by providing a common view of the following:
- How are my products / categories performing?
- Where do I have issues with product sales, returns, or excess inventory?
- How am I performing operationally in delivering product?
- What is the true profit contribution of each product?
- What are the results of my promotion efforts?
- How can I be more effective with the use of promotional funds?
- How are my products forecasted to sell? How many products to produce?
- What is the profit contribution of private label products?
A Vendor Portal provides a platform for collaborating with vendors, enabling timely, valuable business insight, driving efficiencies in cost containment and improving sales performance.
Marketing Analytics and KPIs
Marketing is going through tremendous changes. As more customers move online and social becomes a big factor in how people are influenced to make a purchase, new techniques are emerging around cross-selling, location-based marketing, in-store behavior analysis, customer micro-segmentation, sentiment analysis and enhanced multichannel consumer experience.
Retailers are using big data to integrate marketing promotions, location data and differential pricing. Promotions via large-scale e-mail targeting (many retailers have 25 to 50 million household lists) is still a big challenge for many retailers. Open rates on targeted e-mails with custom catalogs with tailored personalized storefronts are 10 times more effective than those that are not targeted.
Another interesting emerging area of retail big data is sentiment analytics. Combination analytics from integrating profiles, peer sentiment, recommendations and comments on Twitter, Yelp and Facebook are playing a bigger role in influence and purchase decisions. As a result, there is a race among market departments to leverage the voluminous streams of data in social media to drive effective promotions or messages at various touch-points. Social discussion of brands, products and services, and endorsement of them, is very influential. Consumers have always been influenced by the opinions of others, and ignoring a channel that can influence so many consumers so quickly would be unwise.
Competition for the 16-35 age group customers is intense. Emerging social marketing services like Foursquare are changing the dynamics of real-time customer interaction in that age group. Foursquare is a web and mobile application that allows registered users to connect with friends and publish their location. Users ”check-in” at venues using a mobile website, text messaging or a device-specific application by clicking from a list of venues. Each check-in awards the user points and sometimes “badges”. As of June 2011, the company reported it had 10 million registered users. Foursquare is trying to become a sticky portal by moving from social connection to promotions/coupons to even payments.
A list of classic marketing KPIs is provided below.
|Area||Type of Analytical Insights|
|Marketing Analysis KPIs||
Returns, Fraud and Loss Prevention Analytics and KPIs
Ease of returns has always correlated well to customer satisfaction. Thanks to Amazon Prime or Zappos’s 365 day return policy the dynamics of the industry are shifting. Customers love this policy and are eager to migrate to vendors that provide flexibility. But the challenge is predicting return trends by product and sku; and keeping necessary reserves.
In a low margin industry, loss prevention is often a big focus. Retail theft jumped by $3.6 billion to $37.1 billion in 2010, according to National Retail Federation. More goods (44 percent) were swiped by employees, often working with organized groups of thieves, than shoplifters, which accounted for $12 billion, or 33 percent of lost goods. Paperwork errors or vendor fraud accounted for $6.8 billion, or 18 percent.
A list of classic Loss Prevention KPIs is provided below.
|Area||Type of Analytical Insights|
|Loss Prevention KPIs||
The use of IT and digital data in the retail industry has been instrumental in boosting the profitability and productivity of players like Walmart, Tesco, Amazon.com for years. In the coming years, the increased adoption of big data has the potential to increase sector-wide productivity.
In an industry with notoriously tight margins (typically 2-6% range), retailers are facing a set of challenges that require a new set of solutions to address them. Margin declines can no longer be offset by improvements in selling, general, and administrative (SG&A) costs and reduced inventories.
Retailers will need to selectively employ a set of levers that they have utilized in the past to manage costs in the short-term, and a series of more-advanced levers to build a sustainable competitive advantage.
Notes (and Interesting Factoids)
1. Amazon.com uses collaborative filtering to generate “you might also want” prompts for each product bought or visited.
2. Neiman Markus uses behavioral segmentation, economic modeling, a multi-tier membership reward program to understand buying patterns.
3. Geographic retailing is gaining a foothold with better tools. Placecast, a provider of location-based marketing, claims that 50% of users have made purchases as a result of location-based ads (targeted ads combined with preference, profile and location)
4. Williams-Sonoma, all things kitchen and cooking, has a database of 60M households tracking variables like income, number of children, housing values, etc. They leverage these variables in e-mail targeting programs.
5. The Netflix movie recommendation contest (blending of different statistical and machine-learning techniques) has been widely followed because its crowdsourcing lessons could extend beyond improving movie picks. The researchers from around the world were grappling with a huge data set — 100+ million movie ratings — and the challenges of large-scale predictive modeling.
6. Interestingly several people think that “what your friends thought” feature to be extremely accurate in predicting and suggesting movies…more than the recommendation feature.
7. Netflix announced a second recommendation contest. Contestants were asked to model individuals’ “taste profiles,” leveraging demographic and behavioral data. The data set — 100 million entries will include information about renters’ ages, gender, ZIP codes, genre ratings and previously chosen movies. Unlike the first challenge, the contest will have no specific accuracy target. $500,000 will be awarded to the team in the lead after six months, and $500,000 to the leader after 18 months. This contest was cancelled in May 2010 after a legal challenge that it breached customer privacy with the first contest.
8. Typical problems retailers face:
- Low repeat purchase
- Fading and attrition
- No cross-selling or low expansion into new categories
- Pricing inefficiencies
- Post-promotion inventory gluts
10. As one our comments noted…. It’s important to distinguish between retail KPIs and metrics. We view KPIs as clusters of metrics that in aggregate provide a report card. Metrics on the other hand are simply snapshots of an activity. Think of KPIs as overall health vs. metrics as taking temperature or measuring shortness of breadth. KPIs are like latent variables statistics (as opposed to observable variables). KPIs are variables that are not directly observed but are rather inferred (through a mathematical model) from other variables that are observed (directly measured).
- Organizing to Anticipate BI, Analytics, and Big Data (architects.dzone.com)