All models are wrong

"All models are wrong" is a common aphorism in statistics. It is generally attributed to the statistician George Box.

Quotations of George Box

The first record of Box saying "all models are wrong" is in a 1976 paper published in the Journal of the American Statistical Association.^[1] The paragraph containing the aphorism is below.

Since all models are wrong the scientist cannot obtain a "correct" one by excessive elaboration. On the contrary following William of Occam he should seek an economical description of natural phenomena. Just as the ability to devise simple but evocative models is the signature of the great scientist so overelaboration and overparameterization is often the mark of mediocrity.

Box repeated the aphorism in a paper that was published in the proceedings of a 1978 statistics workshop.^[2] The paper contains a section entitled "All models are wrong but some are useful". The section is copied below.

Now it would be very remarkable if any system existing in the real world could be exactly represented by any simple model. However, cunningly chosen parsimonious models often do provide remarkably useful approximations. For example, the law PV = RT relating pressure P, volume V and temperature T of an "ideal" gas via a constant R is not exactly true for any real gas, but it frequently provides a useful approximation and furthermore its structure is informative since it springs from a physical view of the behavior of gas molecules.

For such a model there is no need to ask the question "Is the model true?". If "truth" is to be the "whole truth" the answer must be "No". The only question of interest is "Is the model illuminating and useful?".

Box repeated the aphorism twice more in his 1987 book, Empirical Model-Building and Response Surfaces (which was co-authored with Norman Draper).^[3] The first repetition is on p. 74: "Remember that all models are wrong; the practical question is how wrong do they have to be to not be useful." The second repetition is on p. 424: "Essentially, all models are wrong, but some are useful".

Box's widely cited book Statistics for Experimenters (co-authored with William Hunter) does not include the aphorism in its first edition (published in 1978).^[4] The second edition (published in 2005; co-authored with William Hunter and his son Stuart Hunter) includes the aphorism three times: on p. 208, p. 384, and p. 440.^[5] On p. 440, the relevant sentence is this: "The most that can be expected from any model is that it can supply a useful approximation to reality: All models are wrong; some models are useful".

Comments and discussions

There have been varied comments and discussions about the aphorism. For instance, the statistician Sir David Cox has commented as follows.^[6]

... it does not seem helpful just to say that all models are wrong. The very word model implies simplification and idealization. The idea that complex physical, biological or sociological systems can be exactly described by a few formulae is patently absurd. The construction of idealized representations that capture important stable aspects of such systems is, however, a vital part of general scientific analysis and statistical models, especially substantive ones, do not seem essentially different from other kinds of model.

Burnham & Anderson, in their much-cited book on model selection,^[7] state the following (§1.2.5).

A model is a simplification or approximation of reality and hence will not reflect all of reality. ... Box noted that “all models are wrong, but some are useful.” While a model can never be “truth,” a model might be ranked from very useful, to useful, to somewhat useful to, finally, essentially useless.

The statistician J. Michael Steele has argued somewhat against the aphorism as follows.^[8]

If I say that a map is wrong, it means that a building is misnamed, or the direction of a one-way street is mislabeled. I never expected my map to recreate all of physical reality, and I only feel ripped off if my map does not correctly answer the questions that it claims to answer. My maps of Philadelphia are useful. Moreover, except for a few that are out-of-date, they are not wrong.

The statistician Andrew Gelman countered that, saying in particular the following.^[9]

I take his general point, which is that a street map could be exactly correct, to the resolution of the map.

... The saying, “all models are wrong,” is helpful because it is not completely obvious....

This is a simple point, and I can see how Steele can be irritated by people making a big point about it. But, the trouble is, many people don’t realize that all models are wrong.

The statistician David Hand made the following statement in 2014.^[10]

In general, when building statistical models, we must not forget that the aim is to understand something about the real world. Or predict, choose an action, make a decision, summarize evidence, and so on, but always about the real world, not an abstract mathematical world: our models are not the reality—a point well made by George Box in his oft-cited remark that “all models are wrong, but some are useful”.

In 2011, a workshop on model selection was held in The Netherlands. The name of the workshop was "All models are wrong...".^[11]

Additionally, the aphorism has been recommended to be a core part of the Applied Statistician's Creed.^[12]

Historical antecedents

Although the aphorism seems to have originated with George Box, the underlying idea goes back decades, perhaps centuries. For example, in 1960, Georg Rasch said the following.^[13]

… no models are [true]—not even the Newtonian laws. When you construct a model you leave out all the details which you, with the knowledge at your disposal, consider inessential…. Models should not be true, but it is important that they are applicable, and whether they are applicable for any given purpose must of course be investigated. This also means that a model is never accepted finally, only on trial.

Similarly, in 1947, John von Neumann said that "truth … is much too complicated to allow anything but approximations".^[14]

See also

• Map–territory relation
• Pragmatism
• Reification (fallacy)
• Scientific modelling
• Statistical model

Notes

Further reading

• Anderson, C. (23 June 2008), "The end of theory", Wired.
• Lavine, M. et al. (May 2015), "In Response to ‘Statistics as a Science, Not an Art: The Way to Survive in Data Science’ by Mark van der Laan", Amstat News (magazine of the American Statistical Association).

External links

• “All Models are Right, Most are Useless” —Andrew Gelman blog
• Are all models wrong? —Theory, Evolution, and Games blog
• What is the meaning of “All models are wrong, but some are useful” —Stack Exchange