September 23, 2010

Static redux: Sandgropers jumping off rickety ship

Inventors of actuarial tools to assess sex offender risk recently described their process as "somewhat like rebuilding a ship at sea, continually replacing one plank at a time when we sprang a leak."

But while they viewed this method "as a strength, not a problem," others might reasonably regard staying aboard such a ship as a sure-fire way to drown.

Among those jumping ship are the Western Australians. Courts in Australia's largest state are increasingly rebuffing the Static-99, the world's leading actuarial tool for assessing sex offender risk.

In the latest of a string of cases, Justice John McKechnie has rejected an attempt by the Director of Public Prosecutions to civilly commit a convicted sex offender based on the man's high Static score. In his ruling, the judge criticized the "uncritical acceptance" of actuarial risk tools, saying they do not take into account reductions in risk that can accrue from prison-based treatment programs.

The ruling in the case of Leslie Fred Free is part of a "national backlash" against the Static by courts as well as legal and psychological researchers, according to a news report at The report quotes Bernadette McSherry, a law professor at Monash University who has been tracking cases in which Australian courts have rejected the Static-99.

Many question whether an instrument developed in North America is relevant or reliable for forensic use Down Under. In a 2007 decision, another Western Australian judge focused on the problem with applying the Static-99 to aboriginal offenders. Noted Judge Hasluck in the case of Robert Mangolamara, a young aboriginal man:
[T]he facts and assumptions underlying the assessment tools and related manuals have not been proved…. Moreover, [because] the tools were not devised for and do not necessarily take account of the social circumstances of indigenous Australians in remote communities, I harbour grave reservations as to whether a person of the respondent's background can be easily fitted within the categories of appraisal presently allowed for by the assessment tools.….
Sandgropers, as Western Australians are known, think of themselves as especially good-looking and intelligent. But even here in the less enlightened United States the mainstream forensic community is expressing growing skepticism about the science underlying the actuarials.

Forensic psychiatry journal issues scathing critique

The American Academy of Psychiatry and the Law has just published a scathing critique of the Static-99, saying it lacks each of two elements required for an accurate statistical method of calculating risk: representative samples and uniform measures of outcome.

The must-read article, "Alice in Actuarial-Land" (freely available online), provides a detailed overview of the dizzying changes in recommended norms and procedures for using and interpreting the Static family of instruments over the past decade.

The authors, Shoba Sreenivasan, Linda Weinberger, Allen Frances, and Sarah Cusworth-Walker, illustrate the resultant problems through the hypothetical case of "Mr. X," a 62-year-old rapist.
  • Using the original Static-99 norms, Mr. X's risk of sexually reoffending within 5 years of release falls somewhere between 9.1 percent and 39 percent. Qualitatively, in other words, he falls "somewhere between a low and a high risk."
  • Using the norms provided with the revised Static-99R instrument, Mr. X's risk lies between 8.7 percent and 29.6 percent by 10 years. Again, "somewhere between low and high risk."
Although the case of Mr. X is hypothetical, the judge in the Western Australia case of Mr. Free commented on similar discrepancies between the risk assessments of two forensic psychologists who both relied upon the Static-99. Wrote Judge McKechnie in his insightful analysis:
Dr. [Tara] Yewer's conclusion that [Mr. Free's] risk of re-offending over a five year period is nearly 10 percent more than Ms. [Kirstin] Bouse's conclusion over the same time span is unexplained in the evidence. There is no reason on the evidence why I should reject Ms. Bouse's assessment of 'medium-high' risk or accept in preference Dr. Yewer's assessment of 'high' risk. This application highlights the limitations of STATIC-99.

Any judge who engages in sentencing offenders undertakes some prediction of risk as part of the sentencing process, and does so against a background of principle and experience which suggests that, for example, many young people mature and grow out of crime. Others forsake illicit substances and alcohol and the criminal lifestyle that accompanied them. For some, a period of imprisonment has an actual deterrent effect....

Uncritical acceptance of the STATIC-99 score also negates the whole purpose of [sex offender treatment]. If the programme, to which significant resources are given, has no effect on the risk that participants might then pose to the community, why does the Department of Corrective Services bother with it? … In the present case, I am unable to accept uncritically the risk assessment undertaken by coding the STATIC-99 score, because of earlier reports suggesting the possibility of effective treatment … coupled with the respondent's actual performance [in treatment].
Astonishingly, high-stakes forensic decisions such as criminal sentencing and civil commitment are being made based on such wildly disparate interpretations of data. Also, the authors of the Journal of the American Academy of Psychiatry and Law article point out, most of the data underlying the Static norms derive from master's or doctoral-level papers that were never published or subjected to the peer-review process:
Apart from the dizzying number of risk scores and qualifications, the validity of the risk scores themselves is dubious, given different definitions of recidivism in the norming samples, lack of clarity in statistical methods, and an overreliance on unpublished manuscripts and presentations to document methods….

The expression of risk in numerical form, whether it is a risk percentage, a d-statistic, or a receiver operating characteristic (ROC) or risk ratio, gives the trier of fact the impression of the precision of risk to a greater degree of accuracy than actually exists.

The serious nature of the sentencing decisions being made using these norms requires that these risk estimates are getting it right….

Although they purport to be empirically based, the current Static-99 and its newer iteration, the Static-99R, violate the basic tenets of evidence-based medicine that require reasoned, not mechanical, application of group findings to the individual.
Lead authors Sreenivasan and Weinberger are core faculty at the Institute of Psychiatry and Law of the University of Southern California. That's more than 10,000 miles from Western Australia. But perhaps they recently vacationed Down Under, and drank a bit of the Perth water.

If they sailed, I just hope they first checked the ship's safety record. Replacing planks at sea can be risky business. Especially if you hit a patch of rough weather.

Of related interest:
  • For readers interested in learning more about the Dangerous Sexual Offenders Act of Western Australia, as well as the rules of expert evidence admissibility more generally in Australia, I recommend the written decision in the 2007 case of Robert Anthony Mangolamara, available HERE.