Model Selection and the Cult of AIC

Mark Brewer, BioSS Aberdeen (part of the Statistics Seminar Series)

Model selection is difficult, even in the apparently straightforward case of choosing between linear regression models. There has been a lively debate in the statistical ecology literature in recent years, where some authors have sought to evangelise AIC in this context while others have disagreed strongly.  A series of discussion articles in the journal Ecology in 2014 (e.g. Murtaugh, 2014; Burnham and Anderson, 2014) dealt with part of the issue: the distinction between AIC and p-values. But within the family of information criteria, is AIC always the best choice?  Theory suggests that AIC is optimal in terms of prediction, in the sense that it will minimise out-of-sample root mean square error of prediction. Earlier simulation studies have largely borne out this theory. However, we argue that since these studies have almost always ignored between-sample heterogeneity, the benefits of using AIC have been overstated.  Via a novel simulation framework, we show that relative predictive performance of model selection by different information criteria is heavily dependent on the degree of unobserved heterogeneity between data sets.