On model selection criteria for climate change impact studies
share
Climate change impact studies inform policymakers on the estimated damages of future climate change on economic, health and other outcomes. In most studies, an annual outcome variable is observed, e.g. annual mortality rate, along with higher-frequency regressors, e.g. daily temperature and precipitation. Practitioners use summaries of the higher-frequency regressors in fixed effects panel models. The choice over summary statistics amounts to model selection. Some practitioners use Monte Carlo cross-validation (MCCV) to justify a particular specification. However, conventional implementation of MCCV with fixed testing-to-full sample ratios tends to select over-fit models. This paper presents conditions under which MCCV, and also information criteria, can deliver consistent model selection. Previous work has established that the Bayesian information criterion (BIC) can be inconsistent for non-nested selection. We illustrate that the BIC can also be inconsistent in our framework, when all candidate models are misspecified. Our results have practical implications for empirical conventions in climate change impact studies. Specifically, they highlight the importance of a priori information provided by the scientific literature to guide the models considered for selection. We emphasize caution in interpreting model selection results in settings where the scientific literature does not specify the relationship between the outcome and the weather variables.
READ FULL TEXT