The Forecasting Power of AI with Erin Kelly – IMI BIGDataAIHUB Seminar Series

            The 2022-2023 IMI BIGDataAIHUB’s Seminar Series kicked off on November 29, featuring speaker Erin Kelly. Erin spoke to an audience of graduate and undergraduate students, faculty, staff, and community members about forecasting.  Key learnings are captured in this article, and her presentation can be found here.

            There are two types of people that predict the future: fortune tellers or data scientists and actuaries. Erin Kelly is a data scientist, CPA, and CEO of Advanced Symbolics Inc., which has been featured by national and internal media for its A.I. predictions ranging from election results to sales forecasting. The company and its patented AI, “Polly” was recently featured in the Margin of Error documentary, as the first-place contestant for predicting Canada’s 2019 Federal election results with considerable accuracy.

Principles of Good Forecasting

The seminar presented key principles of doing good forecasting, namely:

1. Pick a metric that is important and won’t change
2. Separation of measures and targets (Goodhart’s Law)
3. Measures are leading indicators (there are few of them)
4. Find measure through attribution
5. If you can’t measure it, you can still use it as a lever
6. Keep it simple (parsimony)

Metrics and Measures

            A metric, as defined by Erin Kelly, is the target you are trying to forecast. If you change the metric, you have to change the entire formula, which takes months to build. A critical example is Covid-19 modeling. The metric initially was hospitalization from Covid. However, the messaging changed and made vaccination the metric. Rather than having vaccination as the measure, it became the target and could no longer be used to predict Covid outbreaks. It is a common mistake to make a measure a metric—Goodhart’s Law.

Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes

Charles Goodhart

            A measure is a leading indicator. Measures happen before the metric, are something that cannot be gamed (such as a randomized controlled sample), and measures must be independent of each other. A measurement hypothesis must be tested and revised accordingly. Attribution is the cause of the action. Attribution will guide you to a leading indicator. Understanding attribution will become much more important, as advanced warning is needed for planning—especially with a backlogged supply chain! A lever is a factor that has an impact but not a correlation to the measure.

Issues to Consider with Modeling

Five questions to ask when modeling:

• Are there dependencies between the variables?
• Are the measures strongly correlated with the target?
• Does it predict far enough into the future for planning?
• Is it repeatable, reproducible, and reliable?
• When has it changed? And how will I know?

            Forecasting is becoming essential and has important consequences. A.I. is the superior tool for forecasting with lower margins of error and higher levels of confidence. It’s the best way to predict the future.