MnSOST-R actuarial instrument critiqued

More questions about validity of controversial SVP tool

WARNING: This post is technical, and meant as a heads-up to professionals working in the SVP field, especially those who are still encountering (or using) the MnSOST-R. I would advise readers and subscribers who do not work in this area to skip this post – and have a nice 4th of July Holiday.

The current issue of the preeminent forensic psychology journal Law & Human Behavior has a scathing critique of the Minnesota Sex Offender Screening Tool – Revised (MnSOST-R) by University of Minnesota Professor William Grove and graduate student Scott Vrieze. Through a series of statistical analyses, the authors argue that this instrument does not result in more accurate prediction of sex offender recidivism than simply knowing the base rate for such recidivism. The instrument fails to meet basic evidentiary standards and should be excluded from SVP civil commitment trials, they argue.

Despite the fact that the MnSOST-R is used in at least 13 of the 17 states that have SVP civil commitment laws, there is little published information on its reliability or validity. The authors review the available information, which in and of itself makes the article imperative for those using the instrument.

Another contribution is the authors' critique of the recently popularized technique of using AUC's (the Area Under the Curve, from signal detection theory) as a measure of test accuracy. Recidivism rates of sex offenders would have to be about seven times higher than they are in order for AUC estimates to be reliable, the authors argue:

"An AUC statistic … can lull the clinician into thinking that, if the AUC is suitably high, the test will perform satisfactorily…. This is far from necessarily so…."

In the same issue of the journal, Douglas Mossman offers a rebuttal: "Contrary to what Vrieze and Grove suggest, ARAIs (actuarial risk assessment instruments) of modest accuracy yield probabilistic information that is more relevant to legal decision-making than just ‘betting the base rate.' "

The Vrieze and Grove critique follows a series of similar, statistically based critiques of the MnSOST-R and similar actuarials by Richard Wollert. These include:

• Wollert, R. (2002). The importance of cross-validation in actuarial test construction: Shrinkage in the risk estimates for the Minnesota Sex Offender Screening Tool- Revised. Journal of Threat Assessment. 2(1), 89-104.

• Wollert, R. (2002b). Additional flaws in the Minnesota Sex Offender Screening Tool- Revised. Journal of Threat Assessment. 2(4), 65-78

• Wollert, R. (2006) Low Base Rates Limit Expert Certainty When Current Actuarials Are Used to Identify Sexually Violent Predators: An Application of Bayes's Theorem. Psychology, Public Policy, and Law. Feb Vol 12(1) 56-85

These articles are not light reading; they amount to complicated battles among statisticians. But forensic psychologists are expected to be aware of these debates when they testify about the use of actuarial instruments in SVP proceedings.

The Vrieze and Grove abstract is here; the Mossman rebuttal abstract is here. For the full articles you either have to pay or have access to a university database. A handy medical primer on ROC/AUC statistics, complete with slidable graphics, is here.