Battling Models of Reality

Great point by Andrew Gelman, on having your models of reality travel in groups of two or three. Winston Churchill on statistical modeling:

Image via Wikipedia

Winston Churchill said that sometimes the truth is so precious, it must be attended by a bodyguard of lies. Similarly, for a model to be believed, it must, except in the simplest of cases, be accompanied by similar models that either give similar results or, if they differ, do so in a way that can be understood.

In statistics, we call these extra models 'scaffolding,' and an important area of research (I think) is incorporating scaffolding and other tools for confidence-building into statistical practice. So far we've made progress in developing general methods for building confidence in iterative simulations, debugging Bayesian software, and checking model fit. My idea for formalizing scaffolding is to think of different models, or different versions of a model, as living in a graph, and to consider operations that move along the edges of this graph of models, both as a way to improve fitting efficiency and as a way to better understand models by making informative comparisons. The graph of models connects to some fundamental ideas in statistical computation, including parallel tempering and particle flitering.

P.S. I want to distinguish scaffolding from model selection or model averaging. Model selection and averaging address the problem of uncertainty in model choice. The point of scaffolding is that we would want to compare our results to simpler models, even if we know that our chosen model is correct. Models of even moderate complexity can be extremely difficult to understand on their own.

Image via Wikipedia

A minor criticism: a model cannot be "correct", or else it isn't a model anymore.