Empirical Accuracy of Travel Forecasts

Researchers and practitioners have, from time to time, reviewed the accuracy of project forecasts. Perhaps the two most known examples are the United States Department of Transportation's Urban Rail Transit Projects: Forecast Versus Actual Ridership and Costs (October 1989, a.k.a. 'The Pickrell Report') (opens new window) and, more recently, Bent Flyvbjerg et alia's How (In)Accurate Are Demand Forecasts in Public Works Projects? (opens new window) in 2005.

With minor exceptions, the authors of these types of studies have found large inaccuracies in transportation demand forecasts. Most concerning is that most authors find that forecasting accuracy is not improving over time and the forecasts are typically optimistically biased.

Assessing the empirical accuracy of travel forecasts is becoming more prevalent due to scarce public resources and the increase of private sector financing for public works projects.

Transit Projects

The Federal Transit Administration, a part of the United States Department of Transportation, has been reviewing the accuracy of demand and cost forecasts of transit capital projects funded by the federal government. The reviewed projects have been funded through FTA's "New Starts" program. The FTA has produced three Predicted versus Actual reports since 1989:

1. Urban Rail Transit Projects: Forecast Versus Actual Ridership and Costs (October 1989, a.k.a. 'The Pickrell Report' after its primary author) (opens new window),
2. Predicted and Actual Impacts of New Starts Projects: Capital Cost, Operating Cost and Ridership Data (September 2003), and
3. The Predicted and Actual Impacts of New Starts Projects - 2007: Capital Cost and Ridership (April 2008).

These Predicted versus Actual Reports are a meta-analysis of transit project forecasts, typically covering all previous capital projects constructed through the Capital Investment Program ("New Starts"). Older projects are re-assessed using the latest ridership information, while projects constructed since the previous Predicted versus Actual report are reviewed for the initial time. The reports compare the actual demand (ridership) with their forecasted values.

In 2005, the United States Congress directed the Executive Branch submit an annual report to Congress summarizing the results of all Before and After Studies (opens new window) received during the previous calendar year. This report analyzes the impact of New Starts projects in terms of transit service and demand (ridership). The information derived from these reports helps ensure that current methods provide reliable information to decision makers.

In 2015, David Schmitt noted that the accuracy of transit demand forecasts in the United States, while not spectacular, has improved significantly since 2007. He noted that many project assumptions and exogenous forecasts, which are provided to transit forecasters and typically accepted without review, are optimistically biased. Biased inputs would likely lead to biased demand forecasts, although this link has rarely been explicitly proven.

Toll Road Projects

Over the past decade, the accuracy of toll road forecasts have been subject to increasing scrutiny. Several bond rating agencies have produced meta-analyses of toll road forecast accuracy. In 2007 the National Cooperative Highway Research Program released Sythesis 364 (opens new window). This report examined the state of the toll forecasting practice at the time and provided some guidelines on improving practice.

Metrics

Accuracy is measured in different ways. Typically an equation is used that is similar to (actual observation) / (forecasted value) minus 1.0. Sometimes this value is multiplied by 100 for presentation purposes. Other metrics include Percent Root Mean Squared Error (opens new window) and Mean Absolute Percentage Error (MAPE) (opens new window), but dozens are used throughout the literature.

Ethics

Martin Wachs has written articles that describe the political and social conditions that form the environment in which forecasts of public works projects are made. His article Ethics and Advocacy in Forecasting for Public Policy (opens new window) is a must read on this subject. Nassim Taleb has also contributed to this subject, largely from economic and risk analysis perspectives.

Bias

Many authors examining project forecast accuracy find that accuracy is not improving over time and the forecasts are typically optimistically biased. Researchers such as Bent Flyvbjerg (in many papers, but this paper (opens new window) is given as an example) point out that bias takes on two forms: optimism bias and strategic misrepresentation. The former is the result of a behavioral bias first uncovered by Daniel Kahneman and Amos Tversky (opens new window), where subjects consistently under-estimated risks and over-estimated benefits. It should be noted Kahneman and Tversky's observations were not identified within a transportation project context. Kahneman won the Nobel Prize for Economics in 2002 for this work. Strategic misrepresentation is the purposeful intent to distort the analysis in order to gain a more favorable political or funding outcome. Kahneman and Tversky were the first to propose the use of Reference Class Forecasting (opens new window) to correct both types of biases in forecasts. In 2005 the American Planning Association recommended using reference class forecasts to improve the accuracy of public project revenue forecasts. In 2015 David Schmitt presented three potential Reference Classes (opens new window) for use with transit demand forecasts for projects within the United States.