Validating the backtests of risk measures

Financial risk model evaluation or backtesting is a key part of the internal model’s
approach to market risk management as laid out by the Basle Committee on Banking
Supervision (2004). However there are a number of backtests that may be applied and
there is little guidance as to the most appropriate method. The goal of this paper is to
analyze the ability of various evaluation methodologies to gauge the accuracy of risk
models. We compare evaluation effectiveness using the standard binomial approach,
together with the interval forecast backtesting, the density forecast backtesting and the
probability forecast backtesting. Our comparison is completed for three risk measures:
Value-at-Risk (VaR), Expected Shortfall (ES) and Spectral Risk measure (SRM). We pay
special attention to applications related to ES and SRM as backtesting of these models
have not been explored in any detail thus far. Based on the Monte Carlo simulations and
the empirical study, a number of interesting results emerge. Firstly within hypothesisbased
tests, including the binomial backtesting, the interval forecast backtesting and the
density forecasts backtesting, the overall dominance of density forecast backtesting is
confirmed. In particular, the backtesting for SRM and ES is more effective than for VaR
in identifying an incorrect model from alternative models in a small sample setting.
Secondly, we propose a loss function for SRM where the probability forecast backtesting
is capable of identifying accurate models from alternative models. Thirdly, in all of the
backtesting methods examined, the choice of the distribution specification is a more
important factor in determining the evaluation performance than the choice of the
volatility specification.