MnSOST-3: Promising new actuarial for sex offenders to debut

Note: See below postscript for a link to the MnSOST-3 instrument and manual, now available online.

Regular readers know that I've criticized our field's overreliance on imprecise and atheoretical screening instruments to predict whether or not an individual will behave violently in the future.

As Patrick Lussier and Garth Davies of Simon Fraser University point out in the current issue of Psychology, Public Policy, and Law, the actuarialist approach of searching for external variables that distinguish individuals "is somewhat at odds with the rationale of risk assessment, which is intended to assess the risk of an individual but also takes into account any changes in the level of risk over time for a specific individual."

In their new longitudinal study, Lussier and Davies identified heterogeneous trajectories in sexual and violent offending over time. They suggest that by turning "a blind eye" to criminological research on the developmental course of offending, the actuarialists have produced measures that are a misfit for many if not most individuals, overestimating risk in some cases and underestimating it in others.

While I agree philosophically with their critique, we have to be realistic.

Legislatures and courts love the so-called actuarials, which rate an individual's risk based on the presence of various preselected risk factors. They're quick and easy to administer. And they offer an illusion of scientific certitude that legitimizes current laws and criminal justice practices.

So, until a more theoretically informed, person-oriented approach gains traction, we should at minimum insist on more accurate actuarials, and better acknowledgment of their limitations. That was the goal, for instance, of the Multisample Age-Stratified Table of Sexual Recidivism Rates (MATS-1), a collaborative project by researchers in the United States, New Zealand, and Australia to more accurately incorporate advancing age into predictions of risk for sex offenders.

With that more modest goal in mind, I am cautiously optimistic about a newly developed actuarial tool for assessing recidivism among sex offenders, the MnSOST-3.

A better actuarial?

Before you recoil in shock based on the name alone, let me reassure you that it's a completely different tool from the old Minnesota Sex Offender Screening Tool (the MnSOST or MnSOST-R). Only three of the new instrument's items are the same, and even those are measured differently, so I don't even know why they kept the tainted name. As many of you know, the original MnSOST (pronounced MIN-sauced) oversampled high-risk offenders and so produced artificially inflated estimates of risk. Also, research on its development was never published in a peer-reviewed journal.

Based on an article by developers Grant Duwe and Pamela Freske accepted for publication in the journal Sexual Abuse, the new and improved MnSOST-3 appears to have several advantages over existing actuarial instruments for assessing sex offender recidivism.

The developers took advantage of advances in statistical modeling, using a predictive logistic regression model that enables a more nuanced measurement of the effects of continuous predictors such as age and number of prior offenses. Risk is adjusted based on whether an offender will be under any kind of supervision in the community, something other actuarials do not consider. Scoring is done on an Excel spreadsheet, which should reduce data entry errors.

A major strength of the MnSOST-3 is that it was developed on a contemporary sample that included 2,315 sex offenders released from Minnesota prisons between 2003 and 2006. Given the plummeting rates of sexual offending in the Western world over the past couple of decades, this is imperative in order not to overestimate risk.

The developers report that the MnSOST-3 is well calibrated with actual recidivism rates for all but the highest-risk offenders, for whom it overestimates risk. In other words, the predicted probabilities of recidivism match up pretty closely with the actual rates of reoffending except for the very highest-risk offenders. Overall, about four percent of the released offenders were reconvicted of a new sex crime within four years, a base rate that is consistent with other recent research findings. 

Moose Lake sex offender facility, Minnesota

The authors frankly acknowledge the problem that this low base rate poses for accurate identification of recidivists. While offenders who scored in the top 10 percent on the MnSOST-3 were more likely than lower-scoring men to reoffend (their rate of reconviction was 22 percent), if you predicted that any given individual in this top bracket would reoffend, you would be wrong four times out of five.

The optimism-corrected accuracy of the MnSOST-3 for the contemporary sample, as measured by the Area Under the Curve (AUC) statistic, was .796. This means that there is about an 80% chance that a randomly selected recidivist will have a higher score on the instrument than a randomly selected non-recidivist -- although this applies only to the sample from which the instrument was developed and is not generalizable to other samples.

Although we must wait to see whether this moderate accuracy will generalize to sex offender populations outside of Minnesota, the MnSOST-3 may be about as good as it gets. After a decades-long search for the Holy Grail of risk prediction, consensus is building that the obstacles are insurmountable. Low base rates of recidivism, along with fluid and unpredictable environmental contexts, place a firm ceiling on predictive accuracy.

Which gets us back to the point made by Lussier and Davies: Consistent with a large body of criminological theory, we need to recognize the criminal career as a process with a beginning, a middle and an end. In other words, it's time to start looking at the individual offender and understanding his specific offense trajectory, rather than just continuing to amass collections of external variables to measure him against.

Oh, in case you were wondering how well the old MnSOST-R did at predicting which men in the contemporary Minnesota sample would reoffend, it had an AUC of .55. That's about as good as a coin flip.

So, if nothing else, the MnSOST-3 should seal the death warrant of its worn-out ancestors. Given their inaccurate and bloated estimates of risk, that will be a very good thing.

The articles are:

Lussier, Patrick and Davies, Garth (2011) A Person-Oriented Perspective on Sexual Offenders, Offending Trajectories, and Risk of Recidivism: A New Challenge for Policymakers, Risk Assessors, and Actuarial Prediction? Psychology, Public Policy, and Law 17 (4), 530–561. (To request a copy from the author, click HERE.)

Duwe, Grant and Freske, Pamela (In Press), Using Logistic Regression Modeling to Predict Sex Offense Recidivism: The Minnesota Sex Offender Screening Tool-3 (Mnsost-3), Sexual Abuse. (To request a copy from the author, click HERE.)

POSTSCRIPT: The MnSOST-3 is now being used by the Minnesota Department of Corrections; thus, the instrument and the scoring manual are available online -- HERE. 

Related blog posts:

• MnSOST-R actuarial instrument critiqued (July 3, 2008)
• SVP industry sneak peek: Problems in Actuaryland (Oct. 4, 2009)
• Study: Actuarials fail to predict sexually violent recidivism (March 5, 2010)
• Delusional campaign for a world without risk (April 3, 2010)
• New study: Do popular actuarials work? Newer instruments outperform Static-99 and RRASOR (April 20, 2010)
• Groundbreaking study of sex offender life courses: Challenge to actuarials: 4 distinct trajectories ID'd (June 4, 2010)
• Metaanalysis debunks psychopathy-violence link (Sept. 3, 2010)
• Age tables improve sex offender risk estimates (Dec. 1, 2010)

Thanksgiving roundup

Brandon McInerney

Gay panic defendant gets 21 years

The gay panic case of Brandon McInerney that we’ve been tracking here since 2008 is finally over. The defendant, who was 14 when he shot and killed classmate Larry King, agreed to a 21-year prison term after a jury deadlocked in his murder trial two months ago.

"The missing militant" pleads no contest

Ronald Bridgeforth

Ronald Bridgeforth, the man I blogged about a couple of weeks ago who spent 43 years underground before deciding to turn himself in, pleaded no contest yesterday to a 1968 charge of assault on a police officer. His sentencing is set for February. For those of you who are interested in his fascinating life, I recommend a profile (HERE) by Laura Rena Murray in Tuesday's San Francisco Chronicle. As Bridgeforth put it, "Not being in jail is not the same as being free."

From Australia: Prolonged detention and mental health

An investigative journalism program in Australia has aired a remarkable documentary on the psychiatric effects of lengthy detention of asylum seekers. ABC’s Four Corners obtained rare footage of conditions in facilities that are typically kept out of sight and out of mind. The show portrays rampant self-mutilation, suicide and psychotic decompensation among Australia's 4,000 incarcerated asylum seekers. "I have only seen darkness in life and a dark future ahead," explains a young Iranian man who has just tried to hang himself after the third rejection of his immigration petition. In a secretly filmed interview, a psychiatric nurse states that suicide attempts and grotesque self-mutilations are daily occurrences, with as many as 30 detainees at a time on one-to-one suicide watch at her facility alone. Psychiatric staff are shown responding to the overwhelming despair by overprescribing sedating medications. Dr. Suresh Sundram of the Mental Health Research Institute describes the detention sites as factories for producing mental illness, especially for detainees who are held for lengthy periods and those who have undergone torture and other traumas before fleeing their homelands. Click below to watch the 45-minute video, which is relevant not only in Australia but other countries around the world with similar immigration issues.

Juveniles: Lifelong benefits of multisystemic therapy 

In a study that's getting a bit of buzz around the Web, a researcher has found that Multisystemic Therapy's positive effects on juvenile delinquents extend for decades. An average of 22 years later, youths who were randomly selected for MST treatment had significantly fewer arrests and family problems than those who got traditional individual therapy. MST, developed by study co-author Charles Borduin of the University of Missouri, has become one of the most widely used evidence-based treatments in the world. It owes its success to the fact that it involves the offender's entire family and community, whereas traditional therapy targets only the offender without modifying his problematic environment. The new study is published in the Journal of Consulting and Clinical Psychology. The Abstract is HERE; a press release summarizing the findings is HERE.

Happy Thanksgiving!

Photo (c) Karen Franklin 2011

And finally, if you're reading this in the comfort of your warm and cozy home or office, you can be thankful you're not at the bottom of the 99 percent, living in a plywood shack being torn down just in time for the rainy season. That's the plight of the folks in one of the many homeless encampments near where I walk.

I wish all of you readers and subscribers a nice holiday. 

Psychology rife with inaccurate research findings

The case of a Dutch psychologist who fabricated experiments out of whole cloth for at least a decade is shining a spotlight on systemic flaws in the reporting of psychological research.

Diederik Stapel, a well-known and widely published psychologist in the Netherlands, routinely falsified data and made up entire experiments, according to an investigative committee.

But according to Benedict Carey of the New York Times, the scandal is just one in a string of embarrassments in "a field that critics and statisticians say badly needs to overhaul how it treats research results":

In recent years, psychologists have reported a raft of findings on race biases, brain imaging and even extrasensory perception that have not stood up to scrutiny…. 

Dr. Stapel was able to operate for so long, the committee said, in large measure because he was “lord of the data,” the only person who saw the experimental evidence that had been gathered (or fabricated). This is a widespread problem in psychology, said Jelte M. Wicherts, a psychologist at the University of Amsterdam. In a recent survey, two-thirds of Dutch research psychologists said they did not make their raw data available for other researchers to see. "This is in violation of ethical rules established in the field," Dr. Wicherts said.

In a survey of more than 2,000 American psychologists scheduled to be published this year, Leslie John of Harvard Business School and two colleagues found that 70 percent had acknowledged, anonymously, to cutting some corners in reporting data. About a third said they had reported an unexpected finding as predicted from the start, and about 1 percent admitted to falsifying data.

Also common is a self-serving statistical sloppiness. In an analysis published this year, Dr. Wicherts and Marjan Bakker, also at the University of Amsterdam, searched a random sample of 281 psychology papers for statistical errors. They found that about half of the papers in high-end journals contained some statistical error, and that about 15 percent of all papers had at least one error that changed a reported finding -- almost always in opposition to the authors' hypothesis….

Forensic implications

While inaccurate and even fabricated findings make the field of psychology look silly, they take on potentially far more serious ramifications in forensic contexts, where the stakes can include six-figure payouts or extreme deprivations of liberty.

For example, claims based on fMRI brain-scan studies are increasingly being allowed into court in both criminal and civil contexts. Yet, a 2009 analysis found that about half of such studies published in prominent scientific journals were so "seriously defective" that they amounted to voodoo science that "should not be believed."

Similarly, researcher Jay Singh and colleagues have found that meta-analyses purporting to show the efficacy of instruments used to predict who will be violent in the future are plagued with problems, including failure to adequately describe study search procedures, failure to check for overlapping samples or publication bias, failure to investigate the confound of sample heterogeneity, and use of a problematic statistical technique, the Area Under the Curve (AUC), to measure predictive accuracy.

Particularly troubling to me is a brand-new study finding that researchers' willingness to share their data is directly correlated with the strength of the evidence and the quality of reporting of statistical results. (The analysis is available online from the journal PloS ONE.)

I have heard about several researchers in the field of sex offender risk assessment who stubbornly resist efforts by other researchers to obtain their data for reanalysis. As noted by Dr. Wicherts, the University of Amsterdam psychologist, this is a violation of ethics rules. Most importantly, it makes it impossible for us to be confident about the reliability and validity of these researchers' claims. Despite this, potentially unreliable instruments -- some of them not even published -- are routinely introduced in court to establish future dangerousness.

Critics say the widespread problems in the field argue strongly for mandatory reforms, including the establishment of policies requiring that researchers archive their data to make it available for inspection and analysis by others. This reform is important for the credibility of psychology in general, but absolutely essential in forensic psychology.

Related blog posts:

• Beware "voodoo" brain science (March 10, 2009)
• Violence risk meta-meta: Instrument choice does matter (June 19, 2011)

Hat tips: Ken Pope and Jane

New article of related interest:

False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant

Psychological Science (November 2011)
Joseph Simmons, Leif Nelson, and Uri Simonsohn (click on any of the authors' names to request a copy)

From the abstract: This article show[s] that despite empirical psychologists' nominal endorsement of a low rate of false-positive findings (≤ .05), flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates. In many cases, a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not. We present computer simulations and a pair of actual experiments that demonstrate how unacceptably easy it is to accumulate (and report) statistically significant evidence for a false hypothesis.

Backlash against penal excess hits New Zealand this weekend

When I was in Australia over the summer, giving a keynote at their national forensic psychology conference, I got the distinct impression that Aussie practitioners were a wee bit savvier about criminal justice excesses than the average American.

Now comes evidence that neighboring New Zealanders are equally astute: A joint conference this weekend of Australian and New Zealanders is entitled: Crime and Punishment: The Rising Punitiveness.


The Wellington conference, co-hosted by the Australian and New Zealand Association of Psychiatry, Psychology and Law and the Royal Australian and New Zealand College of Psychiatrists, features intriguing keynotes on:

• Law and psychiatry, from cooperation to contamination (my favorite title in a great lineup) -- Law and Ethics Professor Nigel Eastman, St George’s University of London

• New Zealand's three strikes legislation: Sound policy or penal excess -- Law Professor Warren Brookbanks, Auckland University Law School

• Off with their heads…said the Queen (runner-up for best title) -- Forensic Psychiatry Professor Emeritus Paul Mullen, Monash University

• While it may not be criminalisation, it is criminal: The plight of people with mental illnesses in the criminal justice system -- Forensic Psychology Professor James Ogloff, Monash University and Director of the Centre for Forensic Behavioural Science and Director of Psychological Services at the Victorian Institute of Forensic Mental Health

• Contrasts in punishment: An examination of Anglophone excess and Nordic exceptionalism (one I would especially like to hear) -- Criminology Professor John Pratt, Institute of Criminology, Victoria University

• Reconceptualizing psychopathy to promote effective intervention -- Psychology and Social Behavior Professor Jennifer Skeem, Centers for Psychology and Law and Evidence-based Corrections at the University of California

• Politics and punitiveness – Limiting the rush to punish -- Kim Workman, retired public servant with the New Zealand Department of Maori Affairs and Ministry of Health

 If you happen to be in or near Wellington this weekend, check it out. I am excited to see this growing backlash against penal excesses, and I sure wish I could be there!

Predicting behavior: The case of the missing militant

There is an oft-repeated axiom in our field that the most reliable predictor of future behavior is what a person has done in the past.

But is this axiom valid?

Let's take the example of Ronald Stanley Bridgeforth, featured on America's Most Wanted.

Forty-three years ago, while being detained on suspicion of trying to buy merchandise with a stolen credit card, the 24-year-old Black militant pulled out a revolver and shot at police. He jumped bail and, three years later, became a suspect in the killing of a police sergeant during an armed invasion of a police station in San Francisco.

Would this information lead you to predict that he was likely to engage in more violence in the future?

If so, you would be wrong.

Bridgeforth vanished from the radar screen, and eventually police figured he might have died. But the former community activist was far from dead. After fleeing to Africa, he moved to Ann Arbor, Michigan, married, and raised two sons. Under the assumed name of Cole Jordan, he worked as a janitor, earned a bachelor’s degree from Wayne State University and a master's degree in counseling from Eastern Michigan University, became a licensed professional counselor, and eventually worked his way up to the rank of a full-time faculty member at Washtenaw Community College.

Bridgeforth with attorney Paul Harris (L) and wife Diane (R)

Last week, Bridgeforth finally turned himself in. Authorities were not closing in, but he had a troubled conscience. He plans to plead guilty in the assault case, in which he faces a maximum of five years in prison. Prosecutors announced they will not prosecute him in the infamous murder of Sgt. John Young at San Francisco's Ingleside Station on Aug. 29, 1971. That case unraveled two years ago in part due to allegations (aired in a documentary, Legacy of Torture) that police used torture with electric shock, cattle prods, beatings, sensory deprivation and asphyxiation to obtain confessions from three of the nine suspects.

One might argue that Bridgeforth is an exception to the rule. Only, he's not. Time and again, we hear about a long-time fugitive who lived a quiet life, surrounded by friends and co-workers who had no clue about his or her violent past.

David Gonzales, William Walter Asher III, Katherine Ann Power, Claude Daniel Marks and Donna Jean Willmott, to name just a few.

These cases are testament to the weak validity of the axiom that past behavior is a good predictor of the future. There are several flaws with the theory, among them:

• The base rate: Most serious crimes have a low base rate of recidivism. That makes us most likely to be correct if we predict that the behavior will NOT reoccur. For example, because of the base rate of rearrest for murder, we would be wrong in the broad majority of cases if we predicted that someone who has killed once will most likely kill again. The same is true for sexual offending. In one recent study, 95 out of 100 people arrested on sex charges had no prior sex crimes; an inordinate focus on the five percent lends an illusion of a higher base rate of reoffending than the evidence warrants.

• Desistance: A second major problem with static predictions is that people change. In fact, even hard-core criminals almost universally desist from crime as they age. This holds true across all eras and cultures. As scholars Shadd Maruna and Laub and Sampson have shown, crime is mainly a young man's game. As they age, offenders settle down and become less impulsive. Or, they simply burn out.

• Environmental context: The axiom of past as prelude also ignores the circumstances that contribute to offending. Criminologists have long known about the critical importance of context. For example, peer influence is critical to crime by adolescents and young adults, who have the highest rates of offending. Lifestyle circumstances that can -- and often do -- change over time influence other types of crimes as well, such as robberies and drug offenses. For Bridgeforth and others of the 1960s-1970s era, the context was a militant revolutionary movement that swept up many idealistic young people.

• Unproven allegations: The Bridgeforth case also highlights the problem of relying on allegations of past misconduct that may not be reliable. Bridgeforth has denied the charge that he was the getaway car driver in the San Francisco police killing, and now prosecutors have chosen not to prosecute him.   

Ultimately, the past-as-prelude axiom may hold better for some behaviors than others. Perhaps it is more reliable when predicting scripted or compulsive acts that a person engages in with high frequency over a lengthy period of time. However, it is less reliable when applied to context-influenced behaviors with low base rates of reoccurrence.

And never should we ignore the influence of aging. Bridgeforth is not the same man at 67 as he was at 24. Think back to your own adolescence or early adulthood; are you the same person now as you were then?

The viewpoint that past is prelude is fundamentally pessimistic, leaving little room to acknowledge that human beings are highly adaptive, and often capable of learning from mistakes and changing our lives.

Call for papers on violence risk assessment

The field of violence risk assessment has expanded rapidly over the past several decades. But despite a plethora of new risk assessment tools, confusion abounds as to how to understand their accuracy and utility. And controversy is growing over how well these tools actually predict violence in the individual case.

To address these gaps, forensic scholars John Petrila and Jay Singh of the University of South Florida have teamed up to edit a special issue of the respected journal, Behavioral Sciences and the Law on the topic of "measuring and interpreting the predictive validity of violence risk assessment."

The goal of the special issue is to provide a comprehensive and accessible resource for researchers, clinicians, and policymakers interested in the measurement of predictive validity or the use of such findings in clinical or legal practice.

The editors invite empirical and conceptual papers on the measurement of predictive validity as it relates to violence risk assessment. In addition, papers focusing on the implications of the measurement of predictive validity for public protection and individual liberty are also welcome, as are legal perspectives on these issues.

Papers should be no longer than 35 pages, including tables, figures and references. The deadline for submissions is July 1, 2012. Authors should send two electronic copies of any submission, one blinded for peer review, to John Petrila, JD or Jay P. Singh, PhD.