59% of all large enterprises are deploying data science (DS) and machine learning (ML) today.
Nearly 50% of all organizations have up to 25 or more ML models in use today.
29% of enterprises are refreshing their data science and machine learning models every day.
The higher the data literacy an enterprise can achieve before launching Data Science & Machine Learning initiatives, the higher the probability of success.
These and many other insights defining the state of the data science and machine learning market in 2021 are from Dresner Advisory Services’2021 Data Science and Machine Learning Market Study. The 7th annual report is noteworthy for its depth of analysis and insight into how data science and machine learning adoption is growing stronger in enterprises. In addition, the study explains which factors drive adoption and determine the key success factors that matter the most when deploying data science and machine learning techniques. The methodology uses crowdsourcing techniques to recruit respondents from over 6,000 organizations and vendors’ customer communities. As a result, 52% of respondents are from North America and 34% from EMEA, with the balance from Asia-Pacific and Latin America.
“The perceived importance of data science and machine learning correlates with organizational success with BI, with users that self-report as completely successful with BI almost twice as likely to rate data science as critical,” said Jim Ericson, vice president, and research director at Dresner Advisory. “The perceived level of data literacy also correlates directly and positively with the current or likely future use of data science and machine learning in 2021.”
Key insights from the study include the following:
59% of large enterprises are deploying data science and machine learning in production today. Enterprises with 10K employees or more lead all others in adopting and using DS and ML techniques, most often in R&D and Business Intelligence Competency Center (BICC)-related work. Large-scale enterprises often rely on DS and ML to identify how internal processes and workflows can be streamlined and made more cost-efficient. For example, the CEO of a manufacturing company explained on a recent conference call that DS and ML pilots bring much-needed visibility and control across multiple plants and help troubleshoot inventory management and supply chain allocation problems.
The importance of data science and ML to enterprises has doubled in eight years, jumping from 25% in 2014 to 70% in 2021. The Dresner study notes that a record level of enterprises sees data science and ML as critically important to their business in 2021. Furthermore, 90% of enterprises consider these technologies essential to their operations, rating them critically important or very important. Successful projects in Business Intelligence Competency Centers (BICC) and R&D helped data science and ML gain broad adoption across all organizations. Larger-scale enterprises with over 10K employees are successfully scaling data science and ML to improve visibility, control, and profitability in organizations today.
Enterprises dominate the recruiting and retention of data science and machine learning talent. Large-scale enterprises with over 10K employees are the most likely to have BI experts and data scientists/statisticians on staff. In addition, large-scale enterprises lead hiring and retention in seven of the nine roles included in the survey. It’s understandable how the Business Intelligence (BI) expertise of professionals in these roles is helping remove the roadblocks to getting more business value from data science and machine learning. Enterprises are learning how to scale data science and ML models to take on problems that were too complex to solve with analytics or BI alone.
80% of DS and ML respondents most want model lifecycle management, model performance monitoring, model version control, and model lineage and history at a minimum. Keeping track of the state of each model, including version control, is a challenge for nearly all organizations adopting ML today. Enterprises reach ML scale when they can manage ML models across their lifecycles using an automated system. The next four most popular features of model rollback, searchable model repository, collaborative, model co-creation tools, and model registration and certification are consistent with the feedback from Data Science teams on what they need most in an ML platform.
Financial Services prioritize model lifecycle management and model performance monitoring to achieve greater scale from the tens of thousands of models they’re using today. Consistent with other research that tracks ML adoption by industry, the Dresner study found that Financial Services leads all other industries in their need for the two most valuable features of ML platforms, model lifecycle management and model performance monitoring. Retail and Wholesale are reinventing their business models in real-time to become more virtual while also providing greater real-time visibility across supply chains. ML models in these two industries need automated model version control, model lineage and history, model rollback, collaborative, model co-creation tools, and model registration and certification. In addition, retailers and Wholesalers are doubling down on data science and machine learning to support new digital businesses, improve supply chain performance and increase productivity.
Enterprises need support for their expanding range of regression models, text analytics functions, and ensemble learning. Over the last seven years, text analytics functions and sentiment analysis’ popularity has continually grown. Martech vendors and the marketing technologists driving the market are increasing sentiment analysis’ practicality and importance. Recommendation engines and geospatial analysis are also experiencing greater adoption due to martech changing the nature of customer- and market-driven analysis and predictive modeling.
R, TensorFlow, and PyTorch are considered the three most critical open-source statistical and machine learning frameworks in 2021. Nearly 70% of respondents consider R important to getting work done in data science and ML. The R language has established itself as an industry standard and is well-respected across DevOps, and IT teams in financial services, professional services, consulting, process, and discrete manufacturing. Tensorflow and Pytorch are considered important by the majority of organizations Dresner’s research team interviewed. They’re also among the most in-demand ML frameworks today, with new applicants having experience in all three being recruited actively today.
Data literacy predicts DS and ML program success rates. 64% of organizations say they have extremely high literacy rates, implying that DS and ML have reached mainstream adoption thanks partly to BI literacy rates in the past. Enterprises that prioritize data literacy by providing training, certification, and ongoing education increase success odds with ML. A bonus is that employees will have a chance to learn marketable skills they can use in their current and future positions. Investing in training to improve data literacy is a win/win.
On-database analytics and in-memory analytics (both 91%), and multi-tenant cloud services (88%) are the three most popular technologies enterprises rely on for greater scalability. Dresner’s research team observes that the scalability of data science and machine learning often involves multiple, different requirements to address high data volumes, large numbers of users, data variety while supporting analytic throughput. Apache Spark support continues to grow in enterprises and is the fourth-most relied-on industry support for ML scalability.
IDC predicts the worldwide artificial intelligence (AI) market, including software, hardware, and services, will grow from $327.5 B in 2021 to $554.3B in 2024 with a five-year compound annual growth rate (CAGR) of 17.5%.
By 2024, 70% of enterprises will use cloud and cloud-based AI infrastructure to operationalize AI, thereby significantly alleviating concerns about integration and upscaling, according to Gartner’s data and analytics predictions for 2021.
By 2025, 50% of enterprises will have developed AI orchestration platforms to operationalize AI — up from fewer than 10% in 2020, according to a recent study by Gartner.
Demand for TensorFlow expertise is one of the leading indicators of machine learning and AI adoption globally. Kaggle’s State of Data Science and Machine Learning 2020 Survey found that TensorFlow is the second most used machine learning framework today, with 50.5% of respondents currently using it.
TensorFlow expertise continues to be one of the most marketable machine learning and AI skills in 2021, making it a reliable leading indicator of technology adoption. In 2020, there were on average 4,134 LinkedIn open positions that required TensorFlow expertise soaring to 8,414 open LinkedIn positions this year in the U.S. alone. Globally, demand for TensorFlow expertise has doubled from 12,172 open positions in 2020 to 26,958 available jobs on LinkedIn today.
Demand for machine learning expertise, as reflected in LinkedIn open positions, also shows strong growth. Increasing from 44,864 available jobs in 2020 to 78,372 in 2021 in the U.S. alone, organizations continue to staff up to support new initiatives quickly. Globally, LinkedIn’s open positions requiring machine-learning expertise grew from 98,371 in 2020 to 191,749 in 2021.
Market forecasts and projections also reflect strong growth for AI and machine learning spending globally for the long term. The following are key takeaways from the machine learning market forecasts from the last year include the following:
Forrester says the AI market will be defined and grow within four software segments, with AI maker platforms growing the fastest, reaching $13 billion by 2025, helping drive the market to $37 billion by 2025. Forrester is defining the four AI software segments as follows: AI maker platforms for general-purpose AI algorithms and data sets; AI facilitator platforms for specific AI functions like computer vision; AI-centric applications and middleware tools built around AI for specialized tasks like medical diagnosis; and AI-infused applications and middleware tools that differentiate through advanced use of AI in an existing app or tool category. New AI-centric apps built on AI functions such as medical diagnosis and risk detection solutions will be the second-largest market, valued at nearly $10 billion by 2025. Source: Sizing The AI Software Market: Not As Big As Investors Expect But Still $37 Billion By 2025, December 10, 2020.
IDC predicts worldwide revenues for the artificial intelligence (AI) market, including software, hardware, and services, will grow from $327.5 billion in 2021 to $554.3 billion in 2024, attaining a five-year compound annual growth rate (CAGR) of 17.5%. IDC further predicts that the AI Software Platforms market will be the strongest, with a five-year CAGR of 32.7%. The slowest will be AI System Infrastructure Software, with a five-year CAGR of 13.7% while accounting for roughly 36% of AI software revenues. IDC found that among the three technology categories, software represented 88% of the total AI market revenues in 2020. It’s the slowest growing category with a five-year CAGR of 17.3%. AI Applications took the largest share of revenue within the AI software category at 50% in 2020. Source: IDC Forecasts Improved Growth for Global AI Market in 2021, February 23, 2021
AI projects continued to accelerate in 2020 in the healthcare, bioscience, manufacturing, financial services, and supply chain sectors despite economic & social uncertainty. Two dominant themes emerge from the combination of 30 diverse AI technologies in this year’s Hype Cycle. The first theme is the democratization or broader adoption of AI across organizations. The greater the democratization of AI, the greater the importance of developers and DevOps to create enterprise-grade applications. The second theme is the industrialization of AI platforms. Reusability, scalability, safety, and responsible use of AI and AI governance are the catalysts contributing to the second theme. The Gartner Hype Cycle for Artificial Intelligence, 2020, is shown below: Source: Software Strategies Blog, What’s New In Gartner’s Hype Cycle For AI, 2020, October 20, 2020.
Capgemini finds that Life Sciences, Retail, Consumer Products, and Automotive industries lead in the percentage of successfully deployed AI use cases today. Life Sciences leads all interviewed industries to AI maturity, with 27% of companies saying they have deployed use cases in production and at scale. Retail is also above the industry average of 13% of companies that have deployed AI in production at scale, with 21% of companies in the industry has adopted AI successfully. 17% of companies in the Consumer Products and Automotive industries now have AI in production, running at scale. Source: Capgemini, Making AI Work For You, (The AI-powered enterprise: Unlocking the potential of AI at scale) 2021
Between 2018 and 2020, there’s been a 76% increase in sales professionals using AI-based apps and tools. Salesforce’s latest State of Sales survey found that 57% of high-performance sales organizations use AI today. High-performing sales organizations are 2.8x more likely to use AI than their peers. High-performing sales organizations rely on AI to gain new insights into customer needs, improve forecast accuracy, gain more significant visibility of rep activity, improve competitive analysis, and more. Source: Salesforce Research, 4th Edition, State of Sales, June 2020
While 24% of companies are currently using AI for recruitment, that number is expected to grow, with 56% reporting they plan to adopt AI next year. In addition, Sage’s recent survey of 500 senior HR and people leaders finds adoption of AI as an enabling technology for talent management increasing. AI is proving effective for evaluating job candidates for potential, improving virtual recruiting events, and reducing biased language in job descriptions. It’s also proving effective in helping to improve career planning and mobility. Josh Bersin, a noted HR industry analyst, educator, and technologist, recently published an interesting report on this area titled The Rise of the Talent Intelligence Platform. Leaders in the field of Talent Intelligence Platforms include Eightfold.ai. Grounded in Equal Opportunity Algorithms, the Eightfold® Talent Intelligence Platform uses deep-learning AI to help each person understand their career potential, and each enterprise understands the potential of their workforce.Sources: VentureBeat, 8 ways AI is transforming talent management in 2021, March 25, 2021, and Eightfold.ai.
84% of marketers are using AI-based apps and platforms today, up from 28% in 2018. Salesforce Research’s latest State of Marketing survey finds that high-performing marketers use an average of seven different applications or use cases. The familiarity high-performing marketers have with AI is a primary factor in 52% of them predicting they will increase their use of AI-based apps in the future. Source: Salesforce Research, 6th Edition, State of Marketing, June 2020
Marketing and Sales lead revenue increases due to AI adoption, yet lag behind other departments on cost savings. 40% of the organizations McKinsey interviewed see between a 6 and 10% increase in revenue from adopting AI in their marketing and sales departments. Adopting Ai to reduce costs delivers the best manufacturing and supply chain management results based on the McKinsey survey results. Revenue increases and cost reductions based on AI adoption are shown in the graphic below. Source: McKinsey & Company, The state of AI in 2020, November 17, 2020
AI sees the most significant adoption by marketers working in $500M to $1B companies, with conversational AI for customer service as the most dominant. Businesses with between $500M to $1B lead all other revenue categories in the number and depth of AI adoption cases. Just over 52% of small businesses with sales of $25M or less use AI for predictive analytics for customer insights. It’s interesting to note that small companies are the leaders in AI spending, at 38.1%, to improve marketing ROI by optimizing marketing content and timing. Source: The CMO Survey: Highlights and Insights Report, February 2019. Duke University, Deloitte, and American Marketing Association. (71 pp., PDF, free, no opt-in).
Three out of four companies are fast-tracking automation initiatives, including AI. Bain & Company found that executives would like to use AI to reduce costs and acquire new customers, but they’re uncertain about the ROI and cannot find the talent or solutions they need. Bain research conducted in 2019 found that 90% of tech executives view AI and machine learning as priorities that they should be incorporating into their product lines and businesses. But nearly as many (87%) also said they were not satisfied with their Company’s current approach to AI. Source: Bain & Company, Will the Pandemic Accelerate Adoption of Artificial Intelligence? May 26, 2020
Gartner’s Magic Quadrant for Data Science and Machine Learning Platforms predicts a continued glut of exciting innovations and visionary roadmaps from competing vendors. Competitors in the Data Science and Machine Learning (DSML) market focus on innovation and rapid product innovation over pure execution. Gartner said key areas of differentiation include UI, augmented DSML (AutoML), MLOps, performance and scalability, hybrid and multicloud support, XAI, and cutting-edge use cases and techniques (such as deep learning, large-scale IoT, and reinforcement learning). Please see my recent article on VentureBeat, Gartner’s 2021 Magic Quadrant cites ‘glut of innovation’ in data science and ML, March 14, 2021.
76% of enterprises are prioritizing AI & machine Learning In 2021 IT Budgets. Algorithmia’s survey finds that six in ten (64%) organizations say AI and ML initiatives’ priorities have increased relative to other IT priorities in the last twelve months. Algorithmia’s survey from last summer found that enterprises began doubling down on AI & ML spending last year. The pandemic created a new sense of urgency regarding getting AI and ML projects completed, a key point made by CIOs across the financial services and tech sectors last year during interviews for comparable research studies. Source: Algorithmia’s Third Annual Survey, 2021 Enterprise Trends in Machine Learning.
Technavio predicts the Artificial Intelligence platforms market will grow to $17.29 billion by 2025, attaining a compound annual growth rate (CAGR) of nearly 35%. The research firm cites the increased levels of AI R&D investments globally combined with accelerating adoption for pilot and proof of concept testing across industries. Technavio predicts Alibaba Group Holding Ltd., Alphabet Inc., and Amazon.com Inc. will emerge as top artificial intelligence platforms vendors by 2025. Source: Artificial Intelligence Platforms Market to grow by $ 17.29 Billion at 35% CAGR during 2021-2025. June 21, 2021
Tractica predicts the AI software market will reach $126 billion in worldwide revenue by 2025. The research firm predicts AI will grow fastest in consumer (Internet services), automotive, financial services, telecommunications, and retail industries. As a result, annual global AI software revenue is forecast to grow from $10.1 billion in 2018 to $126.0 billion by 2025. Source: T&D World, AI Software Market to Reach $126.0 Billion in Annual Worldwide Revenue by 2025.
Salesforce Q1, FY22 revenue was $5.96B, the best quarter in the company’s history.
$1M+ deals hit an all-time high and were up 120% year-over-year. New $1M+ sales are averaging four or more Clouds, with senior management calling out Service Cloud during the earnings call as gaining strong traction in enterprises. Eight of the top 10 deals included Tableau, and five included MuleSoft.
FY22 Revenue guidance raised from $25.9B to $26B, approximately 22% year-over-year growth.
Service Cloud Q1, FY22 revenue is $1.5B, growing 20% year-over-year.
Tableau sales grew 38% year-over-year, reaching $394M in sales. MuleSoft grew 49% year-over-year, reaching $380M in sales in Q1, FY22.
The Slack acquisition is expected to close at the end of Q2, FY22.
Key takeaways from their Q1, FY22 results include the following:
Q1, FY22 revenue is up 23% year-over-year to $5.96B. Operating margins reached 5.9%, with non-GAAP operating margins reaching 20.2% in Q1. Salesforce successfully capitalizes on its customers’ urgency to transform their businesses while providing them with proven, well-integrated apps and platform strategies to help them build new digital businesses. Salesforce is also well-positioned to increase revenue based on the growing interest in analytics apps, combined with strong demand for mobile and social apps and multi-cloud integration. Combining proven apps and platforms with their ongoing R&D work in machine learning, AI, and predictive intelligence shows Salesforce is well-positioned for long-term growth in an increasingly multi-cloud enterprise world.
Successful multi-cloud sales strategies are propelling double-digit growth in the platform side of the business. Five of the ten $1M+ deals Salesforce signed in Q1 included MuleSoft. The Platform business is the fastest-growing segment of Salesforce today, attaining 28% year-over-year growth. Marketing and Commerce are next at 25% year-over-year revenue growth, driven by many Salesforce customers digitally transforming their selling and service strategies online. The latest quarters’ financial results by product area show how well-integrated and revenue-generating the ExactTarget, MuleSoft, and Tableau are turning out to be today.
Salesforce will reach $50B in revenue by 2026, supported by their Total Available Market (TAM), reaching $204B by CY2025. During the Q1, FY22 earnings call, Marc Benioff predicted Salesforce would nearly double in size in four years, reaching $50B from $26B, which is the projected FY22 revenue target. During the earnings call, Marc Benioff also said, “but I’ll tell you that it’s awesome to see not just be number one in CRM, but we’re going to be the number one enterprise software applications company in the world passing SAP.” The seven core product areas Salesforce compete are combining to create a TAM growing at an 11% CAGR between 2021 and 2025.
Apple, Alphabet, Amazon, Microsoft, and Tesla are considered the five most innovative companies, according to BCG’s analysis of the 50 most innovative companies of 2021.
Abbott Labs, AstraZeneca, Comcast, Mitsubishi, and Moderna join the top 50 most innovative companies for the first time this year.
The fastest movers include Toyota, who jumped from 41st to 21st; Salesforce, who jumped from 35th to 22nd; and Coca-Cola, who jumped from 48th to 28th.
90% of companies that outperform on innovation outcomes demonstrate clear C-suite ownership of the innovation agenda.
These and many other insights are from the Boston Consulting Group’s (BCG) 15th annual report defining the world’s 50 most innovative companies in 2021. BCG surveyed 1,500 global innovation executives and found a 10% point increase, to 75%, in executives reporting that innovation is a top-three priority at their companies today. That’s the most significant year-over-year increase in the 15 global innovation surveys BCG has conducted since 2005. BCG’s Most Innovative Companies 2021: Overcoming the Innovation Readiness Gap is available for download free here (28 pp., PDF). This years’ report methodology focuses on identifying the factors causing a large innovation readiness gap between the world’s most innovative companies and their peers across industries. Please see page 23 of the study for the methodology.
Key insights from BCGs’ most innovative companies of 2020 include the following:
Creating a new COVID-19 vaccine in less than a year, inventing test kits in weeks to protect public health, and redefining online shopping and safe home delivery reflect the versatility of the world’s most innovative companies in 2021. Pzifer, Moderna, and Merck & Company’s innate ability to innovate gave everyone a decade of their lives back. Delivering a vaccine in a year when the initial projection was a decade reflects the innovative efficiency of these companies. 2021 is the first year Abbott Labs, who invented and scaled the production of COVID-19 test kits, is included in the 50 most innovative companies worldwide. Amazon and Walmart’s logistics and e-commerce expertise helped ensure safe online shopping and fast home delivery was available to millions of people under stay-at-home orders.
Five factors most differentiate the most and least innovative companies. The basis of BCG’s methodology to identify the 50 most innovative companies in 2021 centers on their innovation-to-impact (i2i) framework. The framework is designed to help companies measure the readiness of their innovation programs to operate at a consistently high level of efficiency and effectiveness. The BCG i2i scoring system identified five factors that most differentiate innovative company leaders and laggards. The five factors that best indicate how innovative a company has the potential to be are shown in the following graphic:
Lack of collaboration between sales, marketing & R&D is the major obstacle to innovation. 31% of all companies surveyed see poor collaboration between marketing and R&D as the most significant obstacle to improving the return on their innovation investments. According to BCG, the collaboration between marketing, sales, and R&D is the most challenging in the Pharmaceutical industry, where 42% of respondents say it’s the biggest hurdle to achieving more significant returns on innovation.
Digital transformation of the core business is now a top priority for 75% of CEOs, and 65% of firms are doubling down on their plans for transformation with renewed urgency. BCG identified six success factors that together—and only together—flip the odds of digital transformation success from 30% to 80%. Those six success factors are close integration of digital strategy with the business strategy, commitment from the CEO through middle management, a talent core of digital superstars, business-led and flexible technology and data platforms, agile governance, and effective monitoring of progress toward defined outcomes.
Conclusion
Companies that know how to collaborate quickly between customer and R&D teams have an inside edge on being innovation leaders. The world’s most innovative companies also have senior management teams committed to the long-term success of nascent, unproven programs. There’s greater tolerance for risk, more of a focus on customers first and innovating around their needs, and an intuitive sense of how to close innovation gaps that hold other companies back.
451 Research’s recent survey, Voice of the Enterprise: Cloud, Hosting & Managed Services, Budgets & Outlook 2021, provides new insights into how the pandemic acts as a catalyst driving organizations to pivot toward new digital business models faster than they had anticipated. The survey is based on interviews with over 300 cloud and IT professionals.
Key insights from the survey include the following:
Public cloud platforms met and exceeded expectations during the pandemic. The study found 85% of the enterprises are getting positive results from their quick shift to public cloud platforms in response to the pandemic.
Enterprises predict the most significant spending gains in 2021 will be in IaaS, PaaS, and public cloud platforms. The pandemic made adaptive intelligence, agility, time-to-launch for new apps, and speed the most urgent priorities IT teams worked to achieve. Moving more workloads to the cloud and creating a virtualized infrastructure is behind increased spending on public cloud platforms. SaaS applications trail just by 6% at 30%, showing the robust momentum cloud platforms have in enabling new app adoption.
55% of enterprises say securing cloud environments is their biggest challenge when using public cloud platforms. Enterprises also say that cost optimization (51.7%), governance & compliance (49.1%), and refactoring apps for the cloud (40.7%) are the challenges they are most focused on solving today. “Governance becomes more of a challenge for organizations the more they start using the cloud,” Posey said. “If you have two people in the IT department spinning up a cloud service, they can govern themselves. But once you get more and more workloads to the cloud and more people in the organization updating the code, you actually need to put into place some governance structure.”
Worldwide end-user spending on public cloud services is forecast to grow 23.1% in 2021 to total $332.3 billion, up from $270 billion in 2020.
Garter predicts worldwide end-user spending on public cloud services will jump from $242.6B in 2019 to $692.1B in 2025, attaining a 16.1% Compound Annual Growth Rate (CAGR).
Spending on SaaS cloud services is predicted to reach $122.6B this year, growing to $145.3B next year, attaining 19.3% growth between 2021 and 2022.
These and many other insights are from Gartner Forecasts Worldwide Public Cloud End-User Spending to Grow 23% in 2021. The pandemic created the immediate need for virtual workforces and cloud resources to support them at scale, accelerating public cloud adoption in 2020 with momentum continuing this year. Containerization, virtualization, and edge computing have quickly become more mainstream and are driving additional cloud spending. Gartner notes that CIOs face continued pressures to scale infrastructure that supports moving complex workloads to the cloud and the demands of a hybrid workforce.
Key insights from Gartner’s latest forecast of public cloud end-user spending include the following:
36% of all public cloud services revenue is from SaaS applications and services this year, projected to reach $122.6B with CRM being the dominant application category. Customer Experience and Relationship Management (CRM) is the largest SaaS segment, growing from $44.7B in 2019 to $99.7B in 2025, attaining a 12.14% CAGR. SaaS-based Enterprise Resource Planning (ERP) systems are the second most popular type of SaaS application, generating $15.7B in revenue in 2019. Gartner predicts SaaS-based ERP sales will reach $35.8B in 2025, attaining a CAGR of 12.42%.
Desktop as a Service (DaaS) is predicted to grow 67% in 2021, followed by Infrastructure-as-a-Service (IaaS) with a 38.5% jump in revenue. Platform-as-a-Service (PaaS) is the third-fastest growing area of public cloud services, projected to see a 28.3% jump in revenue this year. SaaS, the largest segment of public cloud spending at 36.9% this year, is forecast to grow 19.3% this year. The following graphic compares the growth rates of public cloud services between 2020 and 2021.
In 2021, SaaS end-user spending will grow by $19.8B, creating a $122.6B market this year. IaaS end-user spending will increase by $22.7B, the most significant revenue gain by a cloud service in 2021. PaaS will follow, with end-user spending increasing $13.1B this year. CIOs and the IT teams they lead are investing in public cloud infrastructure to better scale operations and support virtual teams. CIOs from financial services and manufacturing firms I’ve recently spoken with are accelerating cloud spending for three reasons. First, create a more virtual organization that can scale; second, extend the legacy systems’ data value by integrating their databases with new SaaS apps; and third, an urgent need to improve cloud cybersecurity.
Conclusion
CIOs and the organizations they serve are prioritizing cloud infrastructure investment to better support virtual workforces, supply chains, partners, and service partners. The CIOs I’ve spoken with also focus on getting the most value out of legacy systems by integrating them with cloud infrastructure and apps. As a result, cloud infrastructure investment starting with IaaS is projected to see end-user spending increase from $82B this year to $223B in 2025, growing 38.5% this year alone. End-user spending on Database Management Systems is projected to lead all categories of PaaS through 2025, increasing from $31.2B this year to $84.8B in 2025. The following graphic compares cloud services forecasts and growth rates:
Sage Intacct, Oracle ERP Cloud, and Microsoft Dynamics 365 ERP are the three highest-rated ERP systems by their users.
86% of Unit4 ERP users say their CRM system is the best of all vendors in the study. The survey-wide satisfaction rating for CRM is 73%, accentuating Unit4 ERP’s leadership in this area.
85% of Ramco ERP Suite users say their ERP systems’ analytics and reporting is the best of all 22 vendors evaluated.
These and many other insights are from SoftwareReview’s latest customer rankings published recently in their Enterprise Data Quadrant Report, Enterprise Resource Planning, April 2021. The report is based entirely on attitudinal data captured from verified owners of the ERP systems reviewed. 1,179 customer reviews were completed, evaluating 22 vendors. SoftwareReviews is a division of the world-class IT research and consulting firm Info-Tech Research Group. Their business model is based on providing research to enterprise buyers on subscription, alleviating the need to be dependent on vendor revenue, which helps them stay impartial in their many customer satisfaction studies. Key insights from the study include the following:
Sage Intacct, Oracle ERP Cloud, Microsoft Dynamics 365 ERP, Acumatica Cloud ERP, Unit4 ERP and FinancialForce ERP are most popular with their users. SoftwareReview found that these six ERP systems have the highest Net Emotional Footprint scores across all ERP vendors included in the study. The Net Emotional Footprint measures high-level user sentiment. It aggregates emotional response ratings across 25 questions, creating an indicator of overall user feeling toward the vendor and product. The following quadrant charts the results of the survey:
80% of Acumatica Cloud ERP users say their system helps create more business value, leading all vendors on this attribute. How effective an ERP system is at adapting to support new business and revenue models while providing greater cost visibility is the essence of how they deliver business value. The category average for this attribute is 75%. Of the 22 vendors profiled, 12 have scores at the average level or above, indicating many ERP vendors are focusing on these areas to improve the business case of adopting their systems.
86% of Sage Intacct ERP users say their system excels at ease of implementation, leading all vendors in the comparison by a wide margin. Implementing a new ERP system can be a costly and time-consuming process as it involves extensive training, change management, and integration. Ease of Implementation received a category score of 75% across the 22 vendors, indicating ERP vendors are doubling down investments to improve this area. Just 11 of the 22 ERP vendors scored above the category average.
Spotify is very much an engineering led company. It is also a product led company. Therefore when it decided it needed to get a better handle on its cloud spend, it decided to build an internal product designed to be used by its engineers. The result is Cost Insights, a plugin for Spotify’s Backstage developer portal. The idea is that engineers and engineering teams are incentivised to take more responsibility for the costs associated with the products they’re building. Modelling cost becomes part of the engineering process, rather than being a separate process for finance teams to manage.
Spotify cut its annual cloud spend by millions of dollars by helping engineers to make better decisions about resource allocation. Why was this so important? The answer is pretty simple. Infrastructure costs were outpacing user acquisition so Spotify set out to bend that curve to better align engineering and business goals.
When forecasted cloud costs are growing at a rate faster than income and revenue, it’s time to make changes. Naturally, Spotify thought it through as an engineering problem.
Spotify has been successful with its approach and is now ready to help other companies that are having the same problem. Spotify therefore open sourced Cost Insights in October 2020 as part of its broader open source and community efforts with Backstage.
Spotify certainly isn’t alone. As organisations migrate more and more workloads to the cloud, and build new applications and services there, they’re finding that cost sprawl is very real. Service ownership is a problem – who is responsible for the bill, and keeping costs down?
I spoke to the Spotify Cost Insights team in some depth to learn more. In this post we will look at Spotify’s culture, approach, technology and what it hopes to achieve with Cost Insights, but also how other companies can begin to adopt some of the same approaches based on Spotify’s practices. This research is based on interviews with Spotify employees, which are found in the companion videos, embedded below. These include a RedMonk Conversation about the issues raised, but also a video in our What Is/How To format, which includes a demo (see below for that sweet, sweet video content.)
Spotify Culture
At Spotify, speed is a core cultural principle.
CEO and founder Daniel Ek sums it up:
We aim to make mistakes faster than anyone else.
With cloud computing though, focusing on speed means costs can spiral out of control. The Spotify cost optimisation team has spent a lot of time ensuring they’re building tools that are relevant to both new product teams, and mature businesses we have at Spotify, and reconcile that with sustainable cloud costs.
Spotify has almost five hundred engineering teams today. It does about twenty thousand deployments a day across thousands of micro services. That’s a huge amount of complexity to manage, and it’s not really amenable to simply pointing at spreadsheets and saying: “we need to fix this”. Instead Spotify hacked the culture. It went back to basics and thought about cost as a problem to be solved with its engineering culture.
Spotify engineering teams are used to having a lot of autonomy, so the company couldn’t just introduce new cost guardrails as a top down concern. Autonomy is a core value at the firm, so Cost Insights was built with that in mind. Therefore the Cost team tried to foster a culture where optimisation would be fun.
More on Backstage
Cost Insights already had a natural home.
Backstage is Spotify’s developer portal. The company’s entire software delivery supply chain is managed with Backstage – all components, data, pipelines, and services are managed using the platform, from idea to production, including monitoring and observability. The core idea is to provide a single, consistent UI for all infrastructure tooling, services, and documentation. Backstage is the company’s front end for all engineering tasks. Both developer and operational workflows are integrated into one console. Backstage is built around a service catalogue enabling teams to track the infrastructure underpinning the services and products they’re building.
Core Backstage use cases include:
Creating a new microservice
Following a pull request from review to production
Centralized technical documentation
Review performance of a team’s mobile features
Backstage abstracts a lot of complexity, whether starting a Kubernetes cluster, or provisioning a pipeline. For example – Spotify uses Jenkins under the hood for CI, but engineers don’t need to know that: they just use GitHub Enterprise and Backstage.
Backstage is not just a developer portal – a better description might be a platform for building developer portals and managing software delivery. Spotify is open sourcing these components of its own Developer Experience (DX) infrastructure, for use by the engineering community. Given that many organizations see Spotify culture and processes as aspirational – the much vaunted “Spotify Model” – it makes sense that folks would adopt their tools. We’ve seen this Netflix, and open source tools such as Chaos Monkey, Spinnaker and Zuul.
Backstage was adopted as a Sandbox Project by the Cloud Native Computing Foundation in September 2020, and there has been a steady uptick in contributors and folks building their own plugins. Spotify is being intentional about building third party interest: more than 40% of pull requests are now from external, non-Spotify, contributors.
Backstage also brings an opinion about design to the table – what plugins should look like, and how they should be built, which should be helpful as Spotify grows the community. Great user experiences can definitely be supported with effective design systems.
We believe an elegant, cohesive UX is vital to what makes Backstage such a productive, end-to-end development environment.
The Stack
Backstage is a modern stack, written primarily in Typescript with a React front end. The front end code is written with React, which defines the platform’s plugin architecture. It uses the Yarn package manager and the Lerna monorepo library. Code formatting is with Prettier, with ESLint for linting. Being a modern stack the API Gateway is of course GraphQL-based.
In terms of deployment, Backstage does a solid job of supporting Kubernetes environments, notably GKE, given that’s what Spotify uses inhouse. For documentation Spotify has embraced a docs-as-code approach with MKDocs.
Cost Insights and Labels
The fact that Backstage is already central to how engineers and engineering teams work at Spotify made it the natural home for Cost Insights. Spotify wanted cost optimisation in a place where engineers are already spending the bulk of their time, as opposed to trying to encourage them to use a different third party system.
Backstage was also a place to bootstrap Cost Insights because it is based on a service catalogue model, with the graph of services already in place. Third party organisations wanting to take advantage of this functionality are going to have to do some upfront work modelling and mapping their internal APIs and services to their own products and services, but building this kind of service map is a useful exercise for any organisation.
Cost Insights also supports labels. Engineers can label Cloud provider resources so that they match their own internal terms, components and service names, rather than relying on billing info from the Cloud provider itself.
One useful aspect here is that often in an organisation a shared infrastructure service used by multiple teams is not properly represented in terms of billing. One team ends up paying for that service, rather than it being properly allocated. Backstage modelling helps with internal chargeback.
MOAR FTEs, please
Spotify largely views cost through the lens of engineering resources. If you invest X to automate a process or optimise a system, what are the returns in terms of Full time Employees (FTEs) – at Spotify that basically means Engineers – you could potentially hire with the savings.
Early experiences with Cost Insights allowed Spotify to fund the equivalent of 25 teams across the company. One interesting part of the Spotify cost optimisation journey is that it is not based on driving specific team OKRs. Engineers and teams don’t get the money back that they save, to spend on their own product, but rather it goes back into the company. The incentives are communal and cultural rather than directly financial. Cost Insights could be used as part of engineering bonus structures, at more financially oriented companies. But for Spotify at least the idea is simply to optimise so that everyone in the company benefits. This communitarianism probably reflects the fact Spotify is a Swedish company, with Scandinavian values.
Developers generally like specificity however, regardless of where they’re from. An engineer has trade-offs to make. If a particular service is becoming too expensive, for example, is there an alternative approach from the Cloud provider that might save money in the long run?
Is this migration worthwhile? How much will it save? How much engineering effort should they put in it? Spotify wants an engineer to be easily able to make a calculation based on how many engineers they need to use on a migration or automation in order to save further engineering resources.
If it’s going to take months of effort and only save half an engineer headcount then an “optimisation” is probably not worthwhile. It’s not enough to simply present cost information. It needs to be granular and mapped to services. Engineers want to know what’s the problem and where is it coming from? That way they can make better decisions.
Natural Optimisers
One facet of Cost Insights that really strikes a chord is the focus on encouraging engineers to do more of what they already like to do.
According to Janisa Anandamohan, Spotify Senior Product Manager, Cost Engineering:
We know engineers are natural optimisers when it comes to reliability, security, performance, et cetera. And now we’re telling them, hey, add costs into the mix. And they were super excited about that. We had many teams that were just able to tweak their services and data pipelines and to make them more efficient.
And we know efficiency doesn’t just help cost, but helps reliability and performance as well. So we were getting double, triple wins along the way.
Spotify further encouraged its developers in a couple of ways. One is to provide recommendations within the portal – for example, with advice about auto-scaling strategies.
The second is with an internally crowdsourced document called Our Cookbook, for engineers to share what worked for them in terms of system optimisations, to help other teams.
Spotify realised that developers were treating cost optimisation like a game, showing off their wins, and encouraging other teams to do the same thing. This helped create a virtuous circle, to “bend the curve”. Leader board functionality is therefore on the roadmap, to help the social, competitive aspects of cost control, as teams compete to reduce costs across services, and better learn from their peers.
Help from the team
While Spotify expects engineers to engage and take some responsibility for cost management, sometimes they need a nudge, and different teams take to it differently. At Spotify cost management is a conversation between engineers and the business, which is represented by a specialist cost management function.
Some teams got access to Cost Insights and immediately started using it to figure optimisations and quickly put them into place. Other teams, the Spotify Cost management team works with and asks why a service is costing so much. Often the team has a good reason for cost expansion though. For example a new service is likely to incur extra cost while it’s in startup mode. A more mature service though should see margins improving.
At times when the cost management organization feels it needs to intervene, they take a collaborative approach to optimisation, for example helping the team in question to figure out how to optimise a workflow, for example, that’s running data processing expensively. It’s a consultancy model.
Spotify argues that having these conversations, bridging bottom up and top down, helps improve alignment in the business. Cost Insights takes finger-pointing out of the equation.
Cost Insights as a product
Spotify hasn’t just built some code for itself and licensed it as open source, then hope it sticks. It is explicitly managing the Backstage development process in terms of product management, and has a full team working on it. The business model isn’t clear yet, but Spotify clearly believes it needs to build a business model for the product in order to encourage sustainable adoption. The product has its own logo.
In the case of Backstage plans arguably go further, than for example, Netflix with Chaos Monkey. Spotify wants to take more responsibility for third party use of the software. Spinnaker might be an interesting parallel, which while it began as a project at Netflix, now has a startup, Armory, building products and services around it. Or Kafka, which began as a project at Netflix, and now has companies built on the technology, notably Confluent. Commercial Open Source is a business in its own right, and Spotify is not in that business, so it will be interesting to see how this plays out. One startup has already emerged with a hosted support model for Backstage, with a cool name. Who doesn’t want to be a Roadie and have a backstage pass at a great gig?
For example, while Cost Insights–like Backstage–is intended for use by other companies, it was initially designed for use at Spotify. This creates some interesting product management challenges. Notably, Cost Insights is primarily designed for Google Cloud, given Spotify primarily runs on GCP, but AWS support is planned. Who will pay for functionality that isn’t core to Spotify’s needs? That’s where a sustainable open source model comes in.
What is FinOps?
Cost Insights is not for the average enterprise. It’s explicitly positioned for use by high scale companies like Spotify – with its 2k+ microservices and 4k data pipelines, to help numbers be attributed to specific teams, products and services.
Spotify also shares notes on a regular basis with peer cloud companies, the biggest customers of hyperscale cloud providers, about best practices in high scale operations, technology and spend. It also uses the term FinOps, shorthand for Cloud Financial Management. The FinOps Foundation is run by the Linux Foundation.
The FinOps argument is interesting, because there is quite a difference between the needs of Spotify and its peers and those of more traditional enterprises. Even the most digital enterprise is not using anywhere near the kind of capacity of a Netflix or Uber.
There are a host of FinOps vendors out there, providing Cloud Financial Management. Cloudability, CloudHealth and ParkMyCloud, are well known vendors. And of course you have AWS Cost Explorer, Amazon’s own tool.
AWS revenues in Q4 2020 were $12.7 billion. Microsoft’s Cloud revenues were $43.1 billion for the second quarter of fiscal 2021. Google Cloud’s revenues for 2020 were 13.06 billion.
Cloud is taking an increasing share of customer IT budgets in almost all product categories That’s a lot of cost to optimise, particularly given the huge number of AWS services that customers are using, and are constantly being introduced. FinOps vendors generally come at the world from the perspective of the finance person rather than the engineer, though. As Spotify looked to reduce its spend, its first initiative–before engineering-centred initiatives like Cost Insights–was top down, focusing on better forecasting and pre-booking GCP resources.
Not everyone is a fan of the “FinOps” movement. Corey Quinn, co-founder of Duckbill Group, is almost certainly the best cost optimisation expert on the planet when it comes to high scale use of AWS services. If you’re spending tens of millions of dollars a year on AWS you should contact Corey to reduce your bill before you finish reading this post. Duckbill offers fixed fee spend management advice with a solid engineering understanding of AWS services and how to get the best out of them. Corey believes that FinOps needlessly complicates things, and that good practices by developers – for example turn things off when you’re not using them! – is the best way forward.
You should be enabling engineers to better understand their total cost of service, which brings us back to tools like Cost Insights.
Label Up to Optimise
We discussed labels above. One of Spotify’s earliest efforts in the space was to apply labels to every cloud resource that it could in Spotify terms and language, rather than that of the cloud provider. This means that when an engineer looks at this tool, they can immediately see things that they understand in terms of their day to day work, rather than trying to work out what it means in GCP terms.
These data are quite granular – showing license by cloud product, so for Google Data Flow or data processing, the developer sees costs broken down by pipeline, or say for Google Compute Engine, the dev sees Spotify service names. That information can then be correlated and configured with different business metrics – e.g., daily active users, or monthly subscribers. The platform is extensible because Spotify of course expects third party users to have their own KPI and metrics.
Sustainability Goals
Cost isn’t the only factor that can be optimised. One interesting idea is that Cost Insights can also be used to optimise this sustainability of a service, deciding in what region, how and when a service runs, and ensuring that it takes advantage of a data centre running on renewable energy.
Spotify includes language in Backstage recommendations about optimisation for sustainability. The team built a feature where the user can switch between its normal FTE (engineer) conversion, to carbon offset tonnes.
RedMonk would like to see this sustainability view adopted by user organisations because it might help large cloud buyers to put more pressure on AWS to focus on renewable energy. AWS is on the record as saying more customer pressure will direct the firm to accelerate its sustainability efforts.
Cadence and Anomalies
One question about shifting cost optimisation left is – what should the cadence look like? It’s not like unit testing, for example, which happens as a constant process of improvement.
Spotify found that the best way to get involvement was to encourage engineers to use Cost Insights in a cadence alongside existing quarterly planning. If there were issues that the Cost team felt needed attention, they’d alert a team before those meetings. That said, if costs are rapidly escalating out of control for a particular service, that’s something that should generate an alert, so anomaly detection is on the Cost Insights product roadmap.
All that said, Cost Insights does have a prime place in Backstage, so the information is always there for developers to consider as they make trade-offs and decisions about infrastructure choices.
Concluding
The move to create a rich ecosystem around Backstage is going to be an interesting new challenge for Spotify, involving a lot of work and new trade offs around what are, or are not, core assets. Building for others may for example slow down development for functions that Spotify needs, which when velocity is a core company value will be an interesting challenge.
But Spotify is right that cloud costs are growing out of control at many companies as they increase their investments in the cloud. That being the case, they’ll be looking for tools that improve their effectiveness in that area. The market for Cost Insights won’t necessarily be huge, but it could be a very influential slice of the industry cloud spend.
RedMonk has done some work with Spotify, but this post was not sponsored, and is an independent piece of research. AWS, Google Cloud and Microsoft are all RedMonk clients.
In scaling the cash-flow side of SaaS, there’s almost nothing more powerful than annual contracts combined with prepaid cash. It’s just such a huge benefit. You get all the cash up-front, and your churn almost by definition goes down. Because the earliest chance the customer has to churn is 12 months hence.
So get pre-paid annual deals whenever possible
At Adobe Sign / EchoSign, half the reason we went cash-flow positive at about $5m in ARR was prepaid annual and multiyear contracts. Combine that with 100% or greater annual growth, and once you hit a few million in ARR, prepaid annual contracts bring in a lot of extra cash ahead of time. Cash you can use especially to make those accretive hires (more on that here).
Our KPI here in fact was to bring in at least 120% of our monthly MRR in cash, each month. We usually hit it. More on how here.
But … to go to annual pricing or not …
It’s a bit of a false choice for most vendors. Because the world already works certain ways here based on the context:
Bigger companies want to sign annual contracts, especially in exchange for discounts. At least, once you have enough of a mini-brand to be trusted. It’s how big company procurement and budgeting processes work. Nothing is a bigger headache in a Fortune 500 company that having to go back to procurement every single month to get an invoice approved.
and … Most small businesses and individuals generally want to pay monthly on their credit cards. Even if you provide an annual discount. Not all – but most.Think about yourself as a consumer. Very small businesses often buy more like consumers than large enterprises. How often do you yourself prepay annually? Probably only for a subset of services you are confident you want for the long term.
So there is a natural continuum of how customers like to pay, and how they are comfortable paying.
A great example is Zoom. As Zoom use exploded during Covid, even more SMBs needed Zoom. And as a result, even more chose monthly subscriptions. Zoom monthly billing exploded to 50% of ARR as SMB use grew 5x during Covid! More on that here.
If you offer bigger companies a natural incentive to sign up annually, they will. At least, once you have a brand and they know they can trust you on Day 0. And later, you can even force them to and make it the only option. But if you ask tiny businesses and individuals to pay for a whole year upfront, they will hesitate. At least the vast majority. So offer them a discount to do so (often 20%, a la the WP Engine above). And some, but not most, will.
Figure out who you sell to, and why, and offer the options each customer segment wants. And don’t push prospects to where they don’t want to go. Get the deal done.
And lastly, be flexible in the beginning. Once you have a brand that customers trust, more will prepay annually. In the beginning, at a minimum, you may need to do more paid trials and proofs-of-concepts, or quarterly/monthly deals. I know no one likes POCs and such, but you have to earn your right to skip them.
CEO: What’s the forecast?
CRO: Same as before, $3,400K.
Director 1: How do you feel about it?
CRO: Good.
Director 2: Where will we really land?
CRO: $3,400K. That’s why that’s the forecast.
Director 1: But best case, where do we land?
CRO: Best case, $3,800K.
Director 2: How do you define best case?
CRO: If the stars align.
Not very productive, is it?
I’ve already blogged about one way to solve this problem: encouraging your CRO think probabilistically about the forecast. But that’s a big ask. It’s not easy to change how sales leaders think, and it’s not always the right time to ask. So, somewhat independent of that, in this series I’ll introduce three concepts that help ensure that we have better conversations about the forecast and ultimately forecast better as a result: triangulation forecasts, to-go pipeline coverage, and this/next/all-quarter pipeline analysis. In this post, we’ll cover triangulation forecasts.
Triangulation Forecasts
The simplest way to have better conversations about the forecast is to have more than one forecast to discuss. Towards that end, much as we might take three or four bearings to triangulate our position when we’re lost in the backcountry, let’s look at three or four bearings to triangulate our position on the new annual recurring revenue (ARR) forecast for the quarter.
In this example [1] we track the forecast and its evolution along with some important context such as the plan and our actuals from the previous and year-ago quarters. We’ve placed the New ARR forecast in its leaky bucket context [2], in bold so it stands out. Just scanning across the New ARR row, we can see a few things:
We sold $3,000K in New ARR last quarter, $2,850K last year, and the plan for this quarter is $3,900K.
The CRO is currently forecasting $3,400K, or 87% of the New ARR plan. This is not great.
The CRO’s forecast has been on a steady decline since week 3, from its high of $3,800K. This makes me nervous.
The CRO is likely pressuring the VP of Customer Success to cut the churn forecast to protect Net New ARR [3].
Our growth is well below planned growth of 37% and decelerating [4].
I’m always impressed with how much information you can extract from that top block alone if you’re used to looking at it. But can we make it better? Can we enable much more interesting conversations? Yes. Look at the second block, which includes four rows:
The sum of the sales reps’ forecasts [5]
The sum of the sales managers’ forecasts [6]
The stage-weighted expected value (EV) of the pipeline [7] [8]
The forecast category-weighted expected value of the pipeline [9]
Each of these tells you something different.
The rep-level forecast tells you what you’d sell if every rep hit their current forecast. It tends to be optimistic, as reps tend to be optimistic.
The manager-level forecast tells you how much we’d sell if every CRO direct report hit their forecast. This tends to be the most accurate [10] in my experience.
The stage-weighted expected value tells you the value of pipeline when weighted by probabilities assigned to each stage. A $1M pipeline consisting of 10 stage 2 $100K oppties has a much lower EV than a $1M pipeline with 10 stage 5 $100K oppties — even though they are both “$1M pipelines.”
The stage-weighted expected value tells you the value of pipeline when weighted by probabilities assigned to each forecast category, such as commit, forecast, or upside.
These triangulation forecasts provide different bearings that can help you understand your pipeline better, know where to focus your efforts, and improve the accuracy of predicting where you’ll land.
For example, if the rep- and manager-level forecasts are well below the CRO’s, it’s often because the CRO knows about some big deal they can pull forward to make up any gap. Or, more sinisterly, because the CRO’s expense budget is automatically cut to preserve a target operating margin and thus they are choosing to be “upside down” rather face an immediate expense cut [11].
If the stage-weighted forecast is much lower than the others, it indicates that while we may have the right volume of pipeline that it’s not far enough along in its evolution, and ergo we should focus on velocity.
Now, looking at our sample data, let’s make some observations about the state of the quarter at SaaSCo.
The reps are calling $3,400K vs. a $3,900K plan and their aggregate forecast has been fairly consistently deteriorating. Not good.
The managers, who we might notice called last quarter nearly perfectly ($2,975K vs. $3,000K) have pretty consistently been calling $3,000K, or $900K below plan. Worrisome.
The stage-weighted EV was pessimistic last quarter ($2,500K vs. $3,000K) and may need updated probabilities. That said, it’s been consistently predicting around $2,600K which, if it’s 20% low (like it was last quarter), it suggests a result of $3,240K [12].
The forecast category-weighted expected value, which was a perfect predictor last quarter, is calling $2,950K. Note that it’s jumped up from earlier in the quarter, which we’ll get to later
Just by these numbers, if I were running SaaSCo I’d be thinking that we’re going to land between $2,800K and $3,200K [13]. But remember our goal here: to have better conversations about the forecast. What questions might I ask the CRO looking at this data?
Why are you upside-down relative to your manager’s forecast?
In other quarters was the manager-level forecast the most accurate, and if so, why you are not heeding it better now?
Why is the stage-weighted forecast calling such a low number?
What’s happened since week 5 such that the reps have dropped their aggregate forecast by over $600K?
Why is the churn forecast going down? Was it too high to begin with, are we getting positive information on deals, or are we pressuring Customer Success to help close the gap?
What big/lumpy deals are in these numbers that could lead to large positive or negative surprises?
Why has your forecast been moving so much across the quarter? Just 5 weeks ago you were calling $3,800K and how you’re calling $3,400K and headed in the wrong direction?
Have you cut your forecast sufficiently to handle additional bad news, or should I expect it to go down again next week?
If so, why are you not following the fairly standard rule that when you must cut your forecast you cut it deeply enough so your next move is up? You’ve broken that rule four times this quarter.
Once I complete this three-part series, I will put a link to the spreadsheet here and a link to the next post in the series here.
# # #
Notes
[1] This is the top of the weekly sheet I recommend CEOs to start their weekly staff meeting.
[2] A SaaS company is conceptualized as a leaky bucket of ARR.
[3] I cheated and look one row down to see the churn forecast was $500K in weeks 1-6 and only started coming down (i.e., improving) as the CRO continued to cut their New ARR forecast. This makes me suspicious, particularly if the VP of Customer Success reports to the CRO.
[4] I cheated and looked one row up to see starting ARR growing at 58% which is not going to sustain if New ARR is only growing at ~20%. I also had to calculate planned growth (3900/2850 = 1.37) as it’s not done for me on the sheet.
[5] Assumes a world where managers do not forecast for their reps and/or otherwise cajole reps into forecasting what the manager thinks is appropriate, instead preferring for managers to make their own forecast, loosely coupling rep-level and the manager-level forecasts.
[6] Typically, the sum of the forecasts from the CRO’s direct reports. An equally, if perhaps not more, interesting measure would be the sum of the first-line managers’ forecasts.
[7] Expected value is math-speak for probability * value. For example, if we had one $100K oppty with a 20% close probability, then its expected value would be $100K * 0.2 = $20K.
[8] A stage-weighted expected value of the (current quarter) pipeline is calculated by summing the expected value of each opportunity in the pipeline, using probabilities assigned to each stage. For example, if we had only three stages (e.g., prospect, short-list, and vendor of choice) and assigned a probability to each (e.g., 10%, 30%, 70%) and then multiplied the new ARR value of each oppty by its corresponding probability and summed them, then we would have the stage-weighted expected value of the pipeline. Note that in a more advanced world those probabilities are week-specific (and, due to quarterly seasonality, maybe week-within-quarter specific) but we’ll ignore that here for now. Typically, one way I sidestep some of that hassle is to focus my quarterly analytics by snapshotting week 3, creating in effect week 3 conversion rates which I know will work better earlier in the quarter than later. In the real world, these are often eyeballed initially and then calculated from regressions later on — i.e., in the last 8 quarters, what % of week 3, stage 2 oppties closed?
[9] The forecast category-weighted expected value of the pipeline is the same the stage-weighted one, except instead of using stage we use forecast category as the basis for the calculation. For example, if we have forecast categories of upside, forecast, commit we might assign probabilities of 0.3, 0.7, and 0.9 to each oppty in that respective category.
[10] Sometimes embarrassingly so for the CRO whose forecast thus ends up a mathematical negative value-add!
[11] This is not a great practice IMHO and thus CEOs should not inadvertently incent inflated forecasts by hard-coding expense cuts to the forecast.
[12] The point being there are two ways to fix this problem. One is to revise the probabilities via regression. The other is to apply a correction factor to the calculated result. (Methods with consistent errors are good predictors that are just miscalibrated.)
[13] In what I’d consider a 80% confidence interval — i.e., 10% chance we’re below $2,800K and 10% chance we’re above $3,200K.
