Fundamental Sources of Forecast Errors and Uncertainty

When seeking to improve forecast accuracy, it is critical to understand the major sources of forecast error. Unfortunately, this is not something that is typically taught in school. And learning it the hard way can be very expensive. Hence this note.

Broadly speaking, there are four sources of forecast uncertainty and error:

1. An incorrect underlying theory or theories;
2. Poor modeling of a theory to apply it to a problem;
3. Wrong parameter values for variables in a model;
4. Calculation mistakes.

Let’s take a closer look at each of these.

Theories

When we make a forecast we are usually basing it on a theory. The problem here is twofold.

First, we often fail to consciously acknowledge the theory that underlies our forecast.

Second, even when we do this, we usually fail to reflect on the limitations of that theory when it comes to accurately forecasting real world results. Here’s a case in point: How many economic forecasts have been based on rational expectations and/or efficient market theories, despite their demonstrated weaknesses as descriptions of reality? Or, to cite an even more painful example, in the years before the 2008 Global Financial Crises, central bank policy was guided by equilibrium theories that failed to provide early warning of the impending disaster.

The forecasts we make are actually conditional on the accuracy of the theories that underlie them. In the case of high impact outcomes that we believe to have a low likelihood of occurring, failing to take into account the probability of the underlying theory’s accuracy can lead to substantial underestimates of the chance a disaster may occur (see, “Probing the Improbable: Methodological Challenges for Risks with Low Probabilities and High Stakes”, by Ord et al).

There are three other situations where the role of theory is usually obscured.

The first is forecasts based on intuition. Research has found that accurate intuition is developed through the combination of (a) repeated experience over time; (b) in a system whose structure and dynamics don’t change; (c) the receipt of repeated feedback on the accuracy of forecasts; and (d) followed by explicit reflection on this feedback that gradually sharpens intuition.

When we make a forecast based on intuition, we are (usually implicitly) making the assumption that this theory applies to the situation at hand. Yet in too many cases, it does not (e.g., because the underlying system is continually evolving). In these cases, our “intuition” very likely rests on a small number of cases that are easily recalled either because they are recent or still vivid in our memory.

The second is a forecast based on analogies. The implicit theory here is that those analogies have enough in common with the situation at hand to make them a valid basis for a forecast. In too many cases, this is only loosely true, and the resulting forecast has a higher degree of uncertainty that we acknowledge.

The third is a forecast based on the application of machine learning algorithms to a large set of data. It is often said that these forecasts are “theory free” because their predictions are based on the application of complex relationships that were found in the analysis of the training data set.

Yet theory is still very much present, including, for example, those that underlie various approaches to machine learning, and those that guide explanation of the extremely complex process that produced the forecast.

Another theoretical concern with machine learning-based forecasts is the often implicit assumption that either the system that generated the data used to train the ML algorithm will remain stable in the future (which is not the case for complex adaptive social or socio-technical systems like the economy, society, politics, and financial markets), or that it will be possible to continually update the training data and machine learning algorithm to match the speed at which the system is changing.

Models

While theories are generalized approaches to explaining and predicting observed effects, models (i.e., a specification of input and output variables and the relationships between them) apply these theories to specific real world forecasting problems.

This creates multiple sources of uncertainty. The first is the decision about which theory to include in a model, as more than one may apply. RAND’s Robert Lempert is a leading expert in this area, who advocates the construction of “ensemble” models that combine the results from applying multiple theories. Most national weather services do the same thing to guide their forecasts. However, ensemble modeling is still far from mainstream.

A second source of uncertainty is the extent to which the implications of a theory are fully captured in a model. A recent example of this was the BBC’s 24 February 2020 story, “Australia Fires Were Worse Than Any Prediction”, which noted they surpassed anything that existing fire models had simulated.

A third source of modeling uncertainty has been extensively researched by Dr. Francois Hemez, a scientist at the Los Alamos and Lawrence Livermore National Laboratories in the United States whose focus is the simulation of nuclear weapons detonations.

He has concluded that all models of complex phenomena face an inescapable tradeoff between their fidelity to historical data, robustness to lack of knowledge, and consistency of predictions.

In evolving systems, models which closely reproduce historical effects often do a poor job of predicting the future. In other words, the better a model reproduces the past, the less accurately it will predict the future, even if its forecasts are relatively consistent.

Hemez also notes that, “while unavoidable, modeling assumptions provide us with a false sense of confidence because they tend to hide our lack-of-knowledge, and the effect that this ignorance may have on predictions. The important question then becomes: ‘how vulnerable to this ignorance are our predictions?’”

“This is the reason why ‘predictability’ should not just be about accuracy, or the ability of predictions to reproduce [historical outcomes]. It is equally important that predictions be robust to the lack-of-knowledge embodied in our assumptions” (see Hemez in “Challenges in Computational Social Modeling and Simulation for National Security Decision Making” by McNamara et al).

However, making a model more robust to our lack of knowledge (e.g., by using the ensemble approach) will often reduce the consistency of its predictions about the future.

The good news is that forecast accuracy often can be increased by combining predictions made using different models and assumptions, either by simply averaging them or via a more sophisticated method (e.g., shrinkage, extremizing, etc.).

Parameter Values

The values we place on model variables is the source of uncertainty with which people are most familiar.

As such, many approaches are used to address it, including scenarios and sensitivity analysis (e.g., best, worst, and most likely cases), Monte Carlo methods (i.e., specifying input variables and results as distributions of possible outcomes, rather than point estimates), and systematic Bayesian updating of estimated values as new information becomes available.

However, even when these methods are used important sources of uncertainty can still remain. For example, in Monte Carlo modeling there is often uncertainty about the correct form of the distributions to use for different input variables. Typical defaults include the uniform distribution (where all values are equally possible), the normal (bell curve) distribution, and a triangular distribution based on the most likely value as well as those believed to be at the 10th and 90th percentiles. Unfortunately, when variable values are produced by a complex adaptive system, they often follow a power law (Pareto) distribution, and the use of traditional distributions increases forecast uncertainty.

Another common source of uncertainty is the relationship between different variables. In many models, the default decision is to assume variables are independent, which is often not true.

A final source of uncertainty is that under different conditions, the values of some model input variables may only change with varying time lags, which are rarely taken into account.

Calculations

Researchers have found that calculation errors are distressingly common, and especially in spreadsheet models (e.g., “Revisiting the Panko-Halverson Taxonomy of Spreadsheet Errors” by Raymond Panko, “Comprehensive Review for Common Types of Errors Using Spreadsheets” by Ali Aburas, and “What We Don’t Know About Spreadsheet Errors Today: The Facts, Why We don’t Believe Them, and What We Need to Do”, by Raymond Panko).

While large enterprises that create and employ complex models increasingly have independent model validation and verification (V&V) groups, and while new automated error checking technologies are appearing (e.g., see the ExcelInt add-in), their use continues to be the exception not the rule.

As a result, a large number of model calculation errors probably go undetected, at least until they produce a catastrophic result (usually a large financial loss).

Conclusion

People frequently make forecasts that assign probabilities to one or more possible future outcomes. In some cases, these probabilities are based on historical frequencies – like the likelihood of being in a car accident.

But in far more cases, forecasts reflect our subjective belief about the likelihood of the outcome in question – i.e., “I believe the probability of “X” occurring before the end of 2030 is 25%.”

What few people realize is that these forecasts are actually conditional probabilities that contain multiple sources of cumulative uncertainty.

For example, consider the probability of “X” occurring before the end of 2030 is 25% -- conditional upon (1) the probability the theory that underlies my estimate is valid; (2) the probability my model has appropriately applied this theory to the forecasting question at hand; (3) the probability my estimated value or values for the variables in my model are accurate; and (4) the probability I have not made any calculation errors.

Given what we know about these four conditioning factors, it is clear that many of the subjective forecasts we encounter are a good deal more uncertain than we usually realize.

In the absence of the opportunity to delve more deeply into the potential sources of error in a given probability forecast, the best way to improve predictive accuracy is to select and combine multiple forecasts that are made using different methodologies, and/or alternative sources of information.
Comments

Critical Uncertainties About the Wuhan Coronoavirus

Since the end of December, we have seen an expanding outbreak of a new variant of the coronavirus in China, spreading from its epicenter in Wuhan.

There are two critical uncertainties to resolve with more evidence: (1) the transmissibility of the Wuhan strain, which so far appears to be high, and (2) the pathogenicity (CFR), which at this point still appears to be relatively low. And when you hear an estimated CFR, always remember to check the denominator on which it is based (lab confirmed or just symptomatic cases).

When it comes to contagious viral diseases, there is usually a tradeoff between their transmissibility (how easily they spread) and their pathogenicity (how many people who become infected die). Viruses that quickly kill their infected hosts effectively limit their own spread.

The number of infected people who die is measured by the "Case Fatality Rate." However, this is a noisy estimate, because the denominator can be based on lab confirmed cases (which increases CFR) or just symptomatic cases (which lowers estimated CFR). Early estimates (based on very noisy reporting) have reported a preliminary CFR for the Wuhan strain of around 2%. However, this will likely change as more evidence becomes available.

To put Wuhan in perspective, the CFRs for Ebola and highly pathogenic H5N1 influenza are >60%. The 1918 pandemic flu was estimated at 10% to 20% (this strain was also relatively transmissible which is why it killed so many). The 2009 H1N1 "swine" flu CFR was estimated at 5% to 9%. By comparison, typical seasonal influenza has a CFR of one tenth of one percent or less (0.1%).

For other coronaviruses, SARS' CFR was estimated to be around 10%, while MERS' was 35%.

Transimissibility is measured using the “Basic Reproduction Number” (known as “R0” or “R-naught”), which is the number of people who will become infected by contact with one contagious person. If R is less than one (e.g., because of a high CFR), an epidemic will quickly “burn itself out”. In contrast, when R is greater than 1, a virus will spread exponentially.
Initial estimates of R for the Wuhan Novel Coronavirus are very noisy at this point. The World Health Organization has published a range of 1.4 to 2.5

For comparison, here are some historic estimated Basic Reproduction Numbers:
  • 1918 Spanish Flu = 2.3 to 3.4 (95% confidence interval)
  • SARS Coronavirus = 1.9
  • 1968 Flu = 1.80
  • 2009 Swine Flu = 1.46
  • Seasonal Influenza = 1.28
  • MERS Coronavirus = <1.0
  • Highly Pathogenic H5N1 Influenza = .90
  • Ebola = .70
The most concerning finding about the Wuhan Coronavirus are claims that it may be capable of infecting other people before a patient becomes symptomatic (i.e., shows signs that he/she has contracted the virus).

An article in the Lancet (“Nowcasting and Forecasting the Potential Domestic and International Spread of the 2019-nCoV Outbreak Originating in Wuhan, China”) found that, “Independent self-sustaining outbreaks in major cities globally could become inevitable because of substantial exportation of presymptomatic cases & the absence of large-scale public health interventions."

If it is supported by subsequent research, this initial finding will almost certainly lead to the imposition of more travel bans and quarantine measures in an attempt to limit transmission of the virus.

To end this post with a bit of good news, a very recent analysis has concluded that, because the number of new coronavirus cases in China is growing more slowly than the exponential rate implied by its Basic Reproduction Number, quarantine, travel bans, and "self-isolation" measures appear to be having a positive impact (https://arxiv.org/pdf/2002.07572.pdf).


Comments

New NACD Survey Highlights Strategy and Risk Challenges Facing Directors and Boards

As always, the recently released Public Company Governance Survey by the (US) National Association of Corporate Directors is a thought-provoking read.

Given Britten Coyne Partners’ focus on helping clients better anticipate, more accurately assess, and adapt in time to emerging threats (whether they are called strategic, disruptive, or existential risks), a few findings stood out.

(1) “Sixty-eight percent of directors report that their company can no longer count on extending its historical strategy over the next five years.” One hopes that this conclusion resulted from constructive board engagement with their management team in previous years, that challenged the continuing validity of the assumptions underlying current strategy, and thus triggered a deeper search for possible new threats and opportunities that has subsequently driven the design of new strategies and risk management processes.

(2) “Directors identified growing business model disruptions (52%) and a slowing global economy (51%) as the top trends most likely to impact their organizations over the next 12 months.” In our experience, it was possible to anticipate and assess these emerging threats long before now, and thus to time the exercise of options to adapt in time to the dangers posed by these and other emerging threats. Companies that are just now starting this process will likely struggle as the time before developing threats reach a critical threshold may be shorter than the time required to effectively adapt to them. In our work with clients, we call this the "Safety Margin."

(3) “Boards hear about risk largely from the CEO and CFO.” However, “just 56% of directors believe the risk information they receive allows their board to draw the right conclusions…Information asymmetry – the gap between what the board knows and what management knows – remains a challenge for boards.” For this very reason — and the risk blindness information asymmetry can cause — we help clients to put in place strategic risk management and governance processes that explicitly incorporate a variety of different internal and external perspectives to improve forecasting accuracy.

(4) The NACD notes that, “backward-looking risk information or information that is focused on well-known risks must be balanced with forward-looking risk reports that allow directors to peek around corners to understand emerging threats.” This aligns not only with our own experience, but also with findings from a recent EY survey of directors and CEOs, which found that “current Enterprise Risk Management processes are considered effective in assessing traditional risks, but not as effective in assessing and managing emerging and atypical risks. To address this issue, leading boards are integrating external perspectives and independent data into their [risk management processes] to expand their scope, promote fresh thinking, and challenge internal biases.”

(5) It was worrying to see in the NACD survey that over the next year, “61% of directors want to improve their board’s core oversight over strategy development, and 63% want to improve it over strategy execution.” In light of the consensus at the recently concluded World Economic Forum that we are entering a period of much higher uncertainty than has been the case over the past decade, many directors must wish that the enhanced oversight processes they desire were already in place.

(6) Finally, the NACD report also noted that, “board leaders can drive strategic board renewal by ensuring that the skills of directors in the boardroom correspond to the evolving needs of the organization.” For this reason, board members reported that director education was one of the top areas where additional time needed to be spent. In addition to consulting, Britten Coyne provides clients with a range of education offerings to help them substantially improve their capacity for anticipating, assessing, and adapting to emerging threats.

In sum, our overall conclusion after reading the NACD survey is that most directors have sensed that disruptive changes in technology, the environment, the economy, national security, society, politics, and financial markets are interacting to create more complexity, more uncertainty, and a far different set of challenges than their boards have faced in the past decade.

As the pace of these changes continues to accelerate, boards must strengthen directors' strategic risk management skills while simultaneously improving their organizations' governance and management processes if they are to successfully anticipate, assess, and adapt to the new threats they will face in the 2020s.

Comments

Chronic Illnesses Kill Organizations Too

In most of our work on Strategic Risk we examine the effects of uncertainty in a complex adaptive system world and try to understand how and why catastrophic, existential threats to businesses develop and what lessons boards of directors and executive teams can learn from these dynamics. Yet there is another class of threat to a business which can be as serious in its effect and may also eventually lead to its demise – serial underperformance.

Analogous to a chronic rather than an acute illness, businesses (or other forms of non-profit organizations) that fail to achieve health year after year, sometimes for decades, also carry a heavy price for their failure. Moreover, their long-term enfeeblement inevitably reduces organizational resilience and eliminates many options for adapting in the face of external crises. The weak die first.

This point was highlighted to us recently by the commentary of Terry Smith, iconoclastic banker, stockbroker, author* and CEO at Fundsmith. Commenting on fashions in investing Smith notes that serial underperformers will almost never represent better value, even if the stock price appears to represent a “value investment” opportunity. He cites Charlie Munger of Berkshire Hathaway:

“Over the long term, it’s hard for a stock to earn a much better return that the business which underlies it earns. If the business earns six percent on capital over forty years and you hold it for that forty years, you’re not going to make much different than a six percent return - even if you originally buy it at a huge discount. Conversely, if a business earns eighteen percent on capital over twenty or thirty years, even if you pay an expensive looking price, you’ll end up with one hell of a result.’ Our emphasis added.

Mr Munger is not offering a theory or an opinion — what he is saying is a mathematical certainty.“

To put this another way, the cost of serial failure is severe and will inevitably end up as an existential threat to an organization. That is why it represents a Strategic Risk.

Now, no doubt some readers are already objecting that they cannot be held accountable for events four decades in the future. Leaving aside the point that, at least in the UK, directors have a statutory duty to ensure the long term success of the enterprise (so perhaps that four-decade outcome is your accountability after all), serial underperformance does not require that kind of timescale to produce serious adverse effects.

Consider a firm making 2% ROCE. Over 5 years the cumulative return, with full reinvestment, is approximately 10.5%. For a firm making 5% ROCE, the comparable cumulative return is 27.6%, i.e. more than two and a half times better. A five year timespan is quite feasible as a time threshold for an existential threat. If the second company has been effectively anticipating and assessing such a potential threat, it has had five years in which to prepare for adaptation and mitigation (assuming it has been monitoring indicators of the potential event). It has also had more resources to dedicate to such a task.

When we examine the cases of corporate failure to identify the root causes and repeating patterns that have led to them, we frequently observe a chain of causality that has stretched back over time, five, ten or even twenty years. These are mostly directly attributable to failures of anticipation, risk blindness arising from reliance on faulty strategic assumptions, failures of assessment that added to blindness through the false comfort of subjective risk probabilities, and failures of action to monitor and mitigate the effects of emerging threats. When serial underperformance is added to this deadly mix, the result – the demise of the corporation – becomes almost inevitable.

When did your board last seriously seek to anticipate threats to the health of the enterprise (and how to adapt to them) over even a five year period?

* Accounting for Growth, Random House, London


Comments

Reduce Overload by Seeking High Value Information

At Britten Coyne Partners, we provide clients with advice, education, and forecasting offerings to help them better anticipate, accurately assess, and adapt in time to emerging threats to their strategy and survival.

One of the points we stress is the importance of reducing overload by searching for high value information, and using it to update prior beliefs (e.g., estimates, forecasts, models). But what constitutes high value information?

We focus on two types.

The first is "high likelihood indicators." In the face of an uncertain future, we often have a mix of probabilistic beliefs about different scenarios or outcomes that could develop in the future. High likelihood indicators are valuable because their presence (or absence) is much more likely to be observed in the case of one scenario or outcome compared to the others. Put differently, high likelihood indicators reduce our uncertainty about the future.

The second type of high value information is anything that triggers the feeling of surprise, which is usually evidence that is at odds with what our mental model of a system or situation would lead us to expect. Surprise is valuable because it increases our uncertainty about the accuracy of our mental models, and signals that we need to improve them.

Unfortunately, the feeling of surprise, and its ability to focus our attention on high value information, is very often fleeting. As Daniel Kahneman has noted, our “System 1” brain automatically seeks to make new sensory inputs cohere with our existing beliefs. This often happens very quickly; we find ourselves remembering feeling surprised, but forgetting what it was that triggered it.

For that reason, leveraging surprise requires us to call on “System 2”, our conscious deliberative brain. In our experience, the best way to do this is to write down the information that triggered the feeling of surprise, so that we can later consider its implications.



Comments