Southern Maine Health Care

Jordan S. Peck

As the world continues to face the challenges related to forecasting COVID-19 and its effects on all parts of our economy,the general population is becoming familiar with a concept that was summarized by a statistician, George Box; “All models are wrong.” Modelers often frame their presentations with this phrase, usually appending, “some are useful.”

Forecasting models are a fundamental element of our daily life.While there are many complex, industry specific examples, we all use weather forecasting models.You may have experienced the situation where a weather forecast says that it should be raining right now. Yet, when you look out your window, it is sunny. How do you react to that “wrong” model? If you need to get the mail, it is unlikely that you will put on a raincoat. If you are headed to a day at the park, despite the current sun, you might bring an umbrella.

As COVID-19 rapidly spreads across the world, forecasting models are being spread nearly as quickly. The American Hospital Association haseven published a website for tracking and explaining each of these different models. Unlike weather models, which are often designed for a normal consumer, these models are not intuitive and are prone to misinterpretation.

Anyone who seeks to use one of these models must remember that models are built with a purpose and the conclusions drawn from that model should be limited to that purpose.Even when limiting conclusions in this way, we often recognize that the results of the model are true for arepresentative system that is similar to the real life system, but it is not exact. In this way the model is, by definition, “wrong.” The results of these models are used to provide a consumer with a sense of the non-linear dynamics of the live system and a magnitude for potential outcomes (i.e. does my hospital need 5 or 500 beds; as opposed to 5 or 6 beds).

The prominent models being used during the current crisis are called Susceptible, Exposed, Infectious, and Recovered or “SEIR” models. The exponential nature of disease spread causes these models to be extremely volatile. FiveThirtyEight developed a lengthy “comic strip” which describes the difficulties of developing one of these models. One might read about the complexity of developing an SEIR model and how easily the model can go “wrong” and wonder why we even do it. They are developed for a particular purpose: to provide a sense of population dynamics and advise policy making. The model may incorrectly estimate the number of COVID-19 cases by an order of thousands, but it will correctly assess the relative impact of different social distancing measures and inform the level of intervention that a community should implement. These models have driven national social distancing and have saved somewhere between thousands and millions of lives. The fact that the detail is uncleardoes not diminish the value of the model, when used for its purpose. However, many organizations have begun to draw more detailed resource planning decisions from these models and this is costly mistake.

Healthcare delivery networks are at particular risk for misusing these models. At the moment there is a lot of medical literature being rapidly produced to estimate what percentage of patients infected with COVID-19 will be hospitalized or need intensive care.

When these percentages are applied to SEIR models they suggest many extreme scenarios and a frightening outlook for the average person. However, these models were not designed for this purpose and therefore it is a poor use of the model.

" The prominent models being used during the current crisis are called Susceptible, Exposed, Infectious, and Recovered or “SEIR” models "

In contrast local healthcare systems, businesses, and the average consumer would benefit from supplementing long term SEIR models with the COVID-19 equivalent of looking out your window; statistical models. Short term decisions should be made based on what real data is telling us. A commonly cited COVID-19 model that primarily uses statisticsis published by The Institute for Health Metrics and Evaluation (IHME -). This model has been particularly popular because it shows a much more positive outlook in many areas of the country. The statistical approach of this model has drawn a lot of ire from the epidemiologists who have spent a century developing SEIR disease models. They note that by only looking at the emerging trends it ignores the possible effects of a new outbreak or the future loosening of social restrictions. This is all true, but it does not make the model any less useful and fundamentally important to a particular audience, for a particular purpose. The CDC has recognized the value of reviewing a mixed set of models but little conversation seems to have occurred about the comparative limitations, and fundamentally different applications of the two modelling approaches.

As the COVID-19 crises started, the initial issue that gained global attention was supply shortages. Supply chains can be susceptible to erratic purchasing behavior. When hospitals, nationwide, grew concerned about impending shortages they sought to order every personal protective item they could get. This had the potential to collapse the supply chain (you experienced this with your toilet paper). However, had we been looking at statistical growth rather than SEIR curves, some areas of the country may have had more logical ordering behavior which would ensure more equitable distribution of supplies. As hospitals and other industries start to re-open non-essential services they should manage resources by using statistical models to assess likely patterns for the next 1-2 weeks rather than rely on SEIR models which claim to forecast resource need. Similarly as a consumer trying to decide whether to go to the grocery store today or in two days, you are better served by a short term statistical model which may show that both days are equally “safe” rather than a long term model which may incorrectly show cases as skyrocketing in just a few days.

As the first COVID-19 wave passes over us, the SEIR models are clearly forecasting future waves of outbreak. With those in mind, now is the time for health systems, state and local governments, and businesses to engage their statistical capabilities. Many sectors of our society have mastered statistically related models that trigger responses based on constant monitoring; whether it beseismic activity or pharmacovigilance. As we move into the marathon portion of COVID-19 we need more statistical models to help us rapidly identify when a pocket of COVID-19 is emerging and generate accurate, short term,resource needs for local systems to respond in an effective coordinated way. All of this should be done in tandem to policy makers periodically checking back to the SEIR models in order to plan for our longer day at the park.