Analytics: your challenge, should you choose to accept it
A couple of months ago I took on a new challenge. It all started when I was invited to write an article for the folks who just launched www.RTinsights.com, which focuses on real time technologies. When we went over the list of topics, my ears perked up at their mention of “analytics.” Interesting. After all, I am not an analytics analyst, so why did this grab my attention? It’s because analytics is critically important to all the technologies I follow, including business process management (BPM) software and customer experience management. Plus, for years I’ve managed analysts who wrote about all aspects of the data stack and I routinely edited their work. So I said “sure, I’ll write something about analytics.”
Here’s the result of that challenge—both this blog post and the article it introduces came from that research. So, you are invited to read this report and give me feedback, but with a caveat. My maiden analytics report provides an introductory overview of how analytics empower business people, describes some of the business apps for data discovery and visualization, and takes an honest look at how realistic it would be for orgs to replace their data scientists with business people powered by easy-to-use analytics. (The answer is “no, it’s neither realistic nor advisable.” ) If you are looking for a deep technical analysis of the uses for analytics, this isn’t the right report for you. But if you are interested in the business uses for analytics, how analytics is deployed in the organization and the role of data scientists, business analysts and business managers, then read on.
* * *
The Big Data Continuum: From Data Scientists to Empowered Business People
The amount of words that have been uttered and written about big data over the past four years is astounding. If big data hasn’t become a big topic in your organization, it will. And it’s probably sparked many conversations, even if the technology hasn’t arrived yet at your firm. Ironically, in a big data world, the explosive amount of data isn’t the big deal. After all, some companies analyze billions of rows in two seconds. Instead, knowing the importance of specific data, while discounting all the other data as distractions is what really matters. As a result, data discovery and visualization is the most explosive segment of the analytics market. And these new and still developing tools take BI out of IT’s hands by empowering business analysts and business people to slice and dice, visualize, and analyze their own data and then put it to use when making business decisions.
Data doesn’t lie, but data doesn’t tell the whole story
For big data to have a big impact, you need a team of data experts that can combine enormous data sets with a spectrum of analytics tools. Without them, big data is just a pipe dream. Worse, if analytics and a defined set of metrics are missing, then data alone can be spun different ways, depending on the person’s point of view. For example, consider how eyewitnesses to a crime recall a different set of facts based on the same reality. Eyewitness reports are unreliable because witnesses interpret the same facts in many different ways. Or, think about how the prosecution and defense attorneys spin completely different stories about what happened at a crime scene while using the same set of exhibits and facts. The same thing happens in businesses; without analytics, business people unwittingly create lots of different stories using the same data. The goal in analytics-driven firms is for everyone to use one agreed-upon set of metrics for business results, so that everyone can make the same conclusions, take similar actions, and tell the same stories.
The business case for data analytics is compelling
The business case for investing in analytics is persuasive. For example, a retailer that sells outdoor products was concerned about the effectiveness of its ads. Marketing decided to aggregate all the audiences and conversion rates to determine where the most effective ad buys occurred. The contribution analysis for one ad campaign uncovered $1.7M in lost conversions of ads per week. Through this analysis, the company quickly figured out which ads they should turn off and which ones they should continue. Without it, they had no statistical rigor and insight into which ads to keep and which to eliminate.
Smaller companies may be tempted to dismiss big data and analytics out of hand as tools reserved for bigger organizations, but should think again before making that decision. The analytics vendors claim (credibly) to have very sophisticated customers with less than $100M in revenues. One vendor cites an example of a B2B company with $50M in annual revenues that invests $2M per year in big data and analytics, as well as employing a team of eight data scientists and analysts. Larger companies invest even more in analytics. The same vendor told us they have customers spending $10M per year on analytics technology alone. These expenditures are more than justified through the realization of much greater savings and increased revenues.
Big Data Needs Data Scientists, Analysts, and Users to Work
The data management people hierarchy in a typical marketing organization, from the bottom up, consists of data scientists, data analysts (also called business analysts), marketers, marketing managers, and marketing executives. Or the line could lead from data scientists to business analysts to product managers and innovation executives — it varies between organizations. No matter how it is organized, this marketing hierarchy needs a continuum of huge data sets that become more simplified toward the top tiers of the org chart. For example, a data analyst may work with large data sets, while a manager or executive may have 3-5 key performance indicators (KPIs) that really matter. The continuum of data consumed up and down the firm can be correlated with the type of analytics tools and the amount of technical support needed by business people. For example, analytics tools become less technical, more visual, and easier to use the higher an individual works within the organization.
But tools are only part of the equation. Can business people use analytics tools by themselves or does the organization still need to employ a bevy of data scientists and data analysts to bridge between IT and end-users? The answer is both. Analytics tools are beginning to approach the same ease of use as Excel — at least that’s the goal. But as big data continues to churn out more data from more sources, organizations still need to employ data scientists with deep skills in math, statistics, or computer science. The answer really depends on the complexity of the application and the data aptitude of the business person. If a non-analytical marketer wants to undertake a complex analysis, he would need pre-built libraries. In this scenario, IT could run pre-built measures that are already in an analytics platform. But perhaps a more knowledgeable marketer wants to combine data sources on his own. He could consume data with an app that is built and deployed by data specialists, or he could produce his own analytics.
Think of the analytics tools this way: Imagine the difference between what an ordinary business person can do with Excel, what an ordinary accountant can do with Excel and what a statistician could do with Excel. Or what a Ph.D. in math can do with a calculator versus a regular business person or an accountant. There’s no comparison. Analytics tools are similar; the background, training, and aptitude of the user determines how much and what she can do. In the meantime, the tools for non-technical users continue to advance while organizations continue to hire data scientists and data analysts.
Data scientists are necessary for the care and feeding of big data
Data scientists are sometimes referred to affectionately as “quants.” Almost as rare as unicorns, these individuals usually have Ph.D’s in mathematics, statistics, or computer science and are employed in the business, not IT. They are hard to find and even harder to hire. Because their skill sets are rare, many companies use digital agencies or other service providers instead of employing data scientists for marketing analysis. And the companies that do hire them must retain their scientists’ interest by keeping the fascinating projects coming.
Data scientists spend their time looking for something that may be wrong and figuring out how to correct it or for something unexpected and how to capitalize on it. These talented individuals work with data quite differently from data analysts; they create equations, determine the best way to visualize data, build complex charts and libraries, develop best practices, and identify which type of data graphics would be best for the business to use. Data scientists also often build vertically oriented dashboards for business people. Because the demand for scientists outstrips the supply, dashboards may be built by vendor partners or the vendors’ engineering services team. In short, data scientists do things that other people don’t know how to do, creating a bridge between the big data world — where IT provides the muscle in security, privacy and data stores — and the business people that constantly clamor for more data.
Conclusion: Focus On Insights and Actions
This report set out to answer one simple question: Have advanced analytics like discovery and visualization, combined with real time support and mobility, and packaged with vertical and horizontal apps (like campaign management) reached the point that ordinary business people could perform all the analytics needed in the organization? And assuming that is the case, could organizations begin to reduce their numbers of data scientists and data analysts? The answer to that question is a resounding “no.”
Analytics have made great strides in the past 3-4 years. Business people capable of using pivot tables in Excel can now most likely work with some of the more user-friendly analytics tools. But it takes a hierarchy or spectrum of individuals within an organization to distill the right information from millions of data fields into the one, three or five KPIs that top executives need The reality is that data analysts and data scientists will be greatly valued for the foreseeable future, if for no other reasons than the amount of big data is exploding, the sophistication of the apps is advancing, and the number of business people using analytics (and needing the support of data analysts and data scientists) is rapidly expanding. Perhaps trying to determine if data scientists are still needed is the wrong question. Yes, they are definitely needed as we move more aggressively into a big data world. But the better question to determine is how your organization can capitalize on the explosion of analytics in a real-time world in which insight and action are the keys to survival. That’s where to focus your attention.