November 27, 2011

MnSOST-3: Promising new actuarial for sex offenders to debut

Note: See below postscript for a link to the MnSOST-3 instrument and manual, now available online.

Regular readers know that I've criticized our field's overreliance on imprecise and atheoretical screening instruments to predict whether or not an individual will behave violently in the future.

As Patrick Lussier and Garth Davies of Simon Fraser University point out in the current issue of Psychology, Public Policy, and Law, the actuarialist approach of searching for external variables that distinguish individuals "is somewhat at odds with the rationale of risk assessment, which is intended to assess the risk of an individual but also takes into account any changes in the level of risk over time for a specific individual."

In their new longitudinal study, Lussier and Davies identified heterogeneous trajectories in sexual and violent offending over time. They suggest that by turning "a blind eye" to criminological research on the developmental course of offending, the actuarialists have produced measures that are a misfit for many if not most individuals, overestimating risk in some cases and underestimating it in others.

While I agree philosophically with their critique, we have to be realistic.

Legislatures and courts love the so-called actuarials, which rate an individual's risk based on the presence of various preselected risk factors. They're quick and easy to administer. And they offer an illusion of scientific certitude that legitimizes current laws and criminal justice practices.

So, until a more theoretically informed, person-oriented approach gains traction, we should at minimum insist on more accurate actuarials, and better acknowledgment of their limitations. That was the goal, for instance, of the Multisample Age-Stratified Table of Sexual Recidivism Rates (MATS-1), a collaborative project by researchers in the United States, New Zealand, and Australia to more accurately incorporate advancing age into predictions of risk for sex offenders.

With that more modest goal in mind, I am cautiously optimistic about a newly developed actuarial tool for assessing recidivism among sex offenders, the MnSOST-3.

A better actuarial?

Before you recoil in shock based on the name alone, let me reassure you that it's a completely different tool from the old Minnesota Sex Offender Screening Tool (the MnSOST or MnSOST-R). Only three of the new instrument's items are the same, and even those are measured differently, so I don't even know why they kept the tainted name. As many of you know, the original MnSOST (pronounced MIN-sauced) oversampled high-risk offenders and so produced artificially inflated estimates of risk. Also, research on its development was never published in a peer-reviewed journal.

Based on an article by developers Grant Duwe and Pamela Freske accepted for publication in the journal Sexual Abuse, the new and improved MnSOST-3 appears to have several advantages over existing actuarial instruments for assessing sex offender recidivism.

The developers took advantage of advances in statistical modeling, using a predictive logistic regression model that enables a more nuanced measurement of the effects of continuous predictors such as age and number of prior offenses. Risk is adjusted based on whether an offender will be under any kind of supervision in the community, something other actuarials do not consider. Scoring is done on an Excel spreadsheet, which should reduce data entry errors.

A major strength of the MnSOST-3 is that it was developed on a contemporary sample that included 2,315 sex offenders released from Minnesota prisons between 2003 and 2006. Given the plummeting rates of sexual offending in the Western world over the past couple of decades, this is imperative in order not to overestimate risk.

The developers report that the MnSOST-3 is well calibrated with actual recidivism rates for all but the highest-risk offenders, for whom it overestimates risk. In other words, the predicted probabilities of recidivism match up pretty closely with the actual rates of reoffending except for the very highest-risk offenders. Overall, about four percent of the released offenders were reconvicted of a new sex crime within four years, a base rate that is consistent with other recent research findings. 

Moose Lake sex offender facility, Minnesota
The authors frankly acknowledge the problem that this low base rate poses for accurate identification of recidivists. While offenders who scored in the top 10 percent on the MnSOST-3 were more likely than lower-scoring men to reoffend (their rate of reconviction was 22 percent), if you predicted that any given individual in this top bracket would reoffend, you would be wrong four times out of five.

The optimism-corrected accuracy of the MnSOST-3 for the contemporary sample, as measured by the Area Under the Curve (AUC) statistic, was .796. This means that there is about an 80% chance that a randomly selected recidivist will have a higher score on the instrument than a randomly selected non-recidivist -- although this applies only to the sample from which the instrument was developed and is not generalizable to other samples.

Although we must wait to see whether this moderate accuracy will generalize to sex offender populations outside of Minnesota, the MnSOST-3 may be about as good as it gets. After a decades-long search for the Holy Grail of risk prediction, consensus is building that the obstacles are insurmountable. Low base rates of recidivism, along with fluid and unpredictable environmental contexts, place a firm ceiling on predictive accuracy.

Which gets us back to the point made by Lussier and Davies: Consistent with a large body of criminological theory, we need to recognize the criminal career as a process with a beginning, a middle and an end. In other words, it's time to start looking at the individual offender and understanding his specific offense trajectory, rather than just continuing to amass collections of external variables to measure him against.

Oh, in case you were wondering how well the old MnSOST-R did at predicting which men in the contemporary Minnesota sample would reoffend, it had an AUC of .55. That's about as good as a coin flip.

So, if nothing else, the MnSOST-3 should seal the death warrant of its worn-out ancestors. Given their inaccurate and bloated estimates of risk, that will be a very good thing.

The articles are:

Lussier, Patrick and Davies, Garth (2011) A Person-Oriented Perspective on Sexual Offenders, Offending Trajectories, and Risk of Recidivism: A New Challenge for Policymakers, Risk Assessors, and Actuarial Prediction? Psychology, Public Policy, and Law 17 (4), 530–561. (To request a copy from the author, click HERE.)

Duwe, Grant and Freske, Pamela (In Press), Using Logistic Regression Modeling to Predict Sex Offense Recidivism: The Minnesota Sex Offender Screening Tool-3 (Mnsost-3), Sexual Abuse. (To request a copy from the author, click HERE.)

POSTSCRIPT: The MnSOST-3 is now being used by the Minnesota Department of Corrections; thus, the instrument and the scoring manual are available online -- HERE. 

Related blog posts:

No comments:

Post a Comment