All About That Base (Rate)
Introduction
The very nature of investing requires an investor to make forecasts about future conditions and assumptions about how those conditions will impact a particular investment. While this statement may seem simple enough, there are two fundamental issues in practice:
Statistically, investors are actually pretty bad at making accurate forecasts.
Regardless of the evidence, investors still think that they are accurate.
Buffett famously eschews making macro-economic and stock market forecasts because he thinks it is impossible. He could not be clearer when he writes,
I am not in the business of predicting general stock market or business fluctuations. If you think I can do this… you should not be in the partnership.
Buffett’s view is supported by historical data. According to Prakash Loungani of the IMF, economists failed to predict 148 of the past 150 recessions, a record of failure that is “virtually unblemished.”
Interestingly, bad forecasts are not just a problem for macro analysis - the phenomenon also applies to individual public companies. In a 2005 study, Mark Bradshaw of Harvard Business School and Lawrence Brown of Georgia State University found no evidence that sell-side analysts have a sustained ability to accurately forecast price targets. The study found that only 24-45% of analysts’ target prices are actually met.
While poorly aligned incentives (positive ratings lead to investment banking engagements) could be to blame for a lack of sell-side accuracy, buy-side analysts are surprisingly even less accurate. A 2008 study in the Financial Analysts Journal concluded that buy-side analysts make even more optimistic and less accurate forecasts than their sell side counterparts.
Regardless of all the evidence to the contrary, professional investors as a group still believe that they can make accurate forecasts. Terry Smith, the successful public equities manager and founder of Fundsmith, is baffled by this:
I remain amazed (I could stop this sentence there) by the number of commentators, analysts, fund managers and investors who seem to be obsessed with trying to predict macro events on which to base their investment decisions. The fact that they are seemingly unable to predict events does not seem to stop them from trying.
And this is not a uniquely investing-related phenomenon. It jives with the work of Phil Tetlock, University of Pennsylvania professor and author of Superforecasting: The Art and Science of Prediction, who argues that more expertise doesn’t actually make you better at forecasting, it just makes you more confident.
The rejection of statistical evidence to the contrary coupled with false confidence is a dangerous combination and could lead to irrational decision-making based on an inaccurate forecast. It is likely this phenomenon that inspired Buffett to write that while “forecasts may tell you a great deal about the forecaster; they tell you nothing about the future.”
Taking all of this into consideration, we arrive at three conclusions:
The first conclusion is that we avoid attempting to forecast the broader market or economy, and instead focus on performing rigorous analysis of companies.
The second is that we need to give ourselves a wide margin for error - meaning that the valuations we pay need to allow our investments to perform reasonably well even if our forecasts are wrong to the downside.
As we cannot escape the fact that we must make assumptions about the future, we need to find a way to improve our forecasts.
The rest of this post is focused on #3.
Base Rates
In 1965, a young Warren Buffett wrote, “our business is that of ascertaining facts and then applying experience and reason to such facts to reach expectations.” A major issue with forecasting is related to exactly which “experiences” investors apply to these facts. According to Michael Mauboussin, the Director of Research at BlueMountain Capital Management and former head of Global Financial Strategies at Credit Suisse, investors typically rely heavily on their memory of prior instances as a basis for comparison.
There are two main issues with this method of analysis. First, it subjects the forecaster to a myriad of cognitive biases, including recency bias (placing too much weight on recent experiences), vivid evidence bias (placing too much weight on extreme examples or compelling stories), anchoring (relying too heavily on one piece of information), confirmation bias (focusing on information that confirms one's preconceptions), and others. Second, it limits the “experience” to only a few anecdotes, without consulting the more complete data sets of historical precedent.
The summary statistics (e.g. average result) of these historical data sets are called “base rates,” and using this data to make forecasts is called “reference class forecasting.” In his book Thinking, Fast and Slow, Nobel Prize winner psychologist Daniel Kahneman put forth a great example of the power of base rates. He proposed the following question:
Steve is very shy and withdrawn, invariably helpful but with little interest in people or in the world of reality. A meek and tidy soul, he has a need for order and structure, and a passion for detail.” Is Steve more likely to be a librarian or a farmer?
According to Kahneman, while most people assume that librarian is the correct answer, the correct answer is actually farmer, because statistically, there are 20x more male farmers than there are male librarians.
Examples like this one cause Tetlock to advise forecasters to use base rates. They also lead quantitative investor Patrick O'Shaughnessy to say that “all investors should be obsessed with base rates,” and fundamental value investor Murray Stahl to write, “the pertinent question concerns precedent. If current circumstances are merely part of ordinary historical experience, then history is a reliable guide to investment tactics.”
We agree with these conclusions, and look for ways to incorporate base rates into our investment process. Here are a few examples.
Example 1: Private Business Success Rates
Opportunities to invest in younger companies often come across our desks. These opportunities tend to look attractive from a high level - the companies offer an interesting product or service, are growing revenues very quickly, and have a long runway for growth. Based on any one example, our initial inclination might be that these companies are quite promising.
But what percentage of young companies actually survive? In other words, what is the base rate for business survival by age? This chart, built from data compiled by the Bureau of Labor Statistics from 1994 to 2017, shows one-year survival rates for businesses by age.
When we parse through the data, we find that success rates for US private businesses dramatically increase after ~5 years - from under 80% to over 90%. We use this data to derive the following rule of thumb: we try to avoid investments in businesses that have been around for less than five years, even if recent results appear promising.
Example 2: Industry Stability
For any metric, it is obviously helpful to know the average result (base rate) for a given group of companies. But it is also helpful to know how quickly outliers tend to revert to the mean.
For example, consider returns on capital for different industries. It benefits the investor to know that certain industries tend to deliver higher returns on capital than others. In 2016, Michael Mauboussin and his team at Credit Suisse compiled a comprehensive list of base rates by industry. His list of average returns on capital (defined as “cash flow return on investment” or CFROI) for various industries is republished below.
Let’s assume we found an energy company with a CFROI of 10%, significantly above the industry average. Our initial inclination might be that this company is an excellent business.
However, Mauboussin also studied the relationships between returns on investment for the same companies across different years. The strength of those relationships, or “correlation coefficient,” indicates the tendency of a company’s returns to revert towards the mean return for the industry. A stronger correlation coefficient implies a slower reversion to the mean, and a weaker correlation implies a faster reversion to the mean. Here are the four-year correlation coefficients for CFROI by industry:
As energy companies had an average CFROI of 4.9% from 1983-2015, and the 4-year correlation coefficient for the sector is 0.35, we can estimate what our outlier company's return on investment might be in four years with the following formula:
Industry Average Return + Correlation Coefficient x (Outlier Return - Industry Average Return)
Or
4.9% + 0.35 x (10.0% – 4.9%)
This gives us a projection of ~6.7%. The base rates for how quickly returns tend to revert to the mean imply a different answer than we originally thought - the abnormally high return of our energy company is statistically likely to decline significantly over the next four years. We use this data to derive the following rule of thumb: we prefer to avoid investments in low correlation coefficient sectors, even if they generate high initial returns on capital.
Limitations of Base Rates
While we are strong believers in the power of base rates to improve our forecasts, we also realize that base rates are not perfect. First, it is often hard or impossible to gather the relevant data needed to establish a base rate. Second, base rates are merely historical summaries that allow us to create a rule of thumb - but there are always exceptions to the rule. To hark back to the previous example, surely there are energy companies that have been able to maintain high returns on capital for long periods of time. Relying exclusively on base rates would cause an investor to miss out on those unique opportunities.
This is why we think it is important to use both judgement and base rates. We think Sam Hinkie, the extremely successful former GM of the Philadelphia 76ers, summed it up perfectly when he used the analogy of an airplane: do passengers prefer to trust their lives to an autopilot (which runs on historical data) or an experienced pilot (relying on judgement). “The answer is obviously both.”