Sunday, March 15, 2015

Stumbling and Mumbling: "We'll have to look at the data"

Measuring risk is only useful if that measure means anything.  "We'll have to look at the data" summarises some cases where data is misused or measures something in a way that provides excess confidence on knowledge,  

"Some stats are just bad and can give a mere illusion of knowledge. For example, in 2007-08 banks' risk models were based on data which over-sampled low volatility and under-sampled high. The upshot was that the crisis came as a shock. David Viniar, Goldman’s chief financial officer, famously said: "We were seeing things that were 25-standard deviation moves, several days in a row.” But in fact, a better inference would have been that risk was mismeasured."
If the sample is not representative of the whole population (too much attention on what has happened recently) or does not have sufficient cases of extremes that are most important in risk measurement, this will provide a feeling of confidence and knowledge that is not warranted.

No comments: