December 14, 2012

Judge bars Static-99R risk tool from SVP trial

Developers staunchly refused requests to turn over data
For several years now, the developers of the most widely used sex offender risk assessment tool in the world have refused to share their data with independent researchers and statisticians seeking to cross-check the  instrument's methodology.

Now, a Wisconsin judge has ordered the influential Static-99R instrument excluded from a sexually violent predator (SVP) trial, on the grounds that failure to release the data violates a respondent's legal right to due process.

The ruling may be the first time that the Static-99R has been excluded altogether from court. At least one prior court, in New Hampshire, barred an experimental method that is currently popular among government evaluators, in which Static-99R risk estimates are artificially inflated by comparing sex offenders to a specially selected "high-risk" sub-group, a procedure that has not been empirically validated in any published research. 

In the Wisconsin case, the state was seeking to civilly commit Homer Perren Jr. as a sexually dangerous predator after he completed a 10-year prison term for an attempted sexual assault on a child age 16 or under. The exclusion of the Static-99R ultimately did not help Perren.  This week, after a 1.5-day trial, a jury deliberated for only one hour before deciding that he met the criteria for indefinite civil commitment at the Sand Ridge Secure Treatment Center.*

Dec. 18 note: After publishing this post, I learned that the judge admitted other "actuarial" risk assessment instruments, including the original Static-99 and the MnSOST-R, which is way less accurate than the Static-99R and vastly overpredicts risk. He excluded the RRASOR, a four-item ancestor of the Static-99. In hindsight, for the defense to get the Static-99R excluded was a bit like cutting off one's nose to spite one's face.

The ruling by La Crosse County Judge Elliott Levine came after David Thornton, one of the developers of the Static-99R and a government witness in the case, failed to turn over data requested as part of a Daubert challenge by the defense. Under the U.S. Supreme Court's 1993 ruling in Daubert v. Merrell Dow Pharmaceuticals, judges are charged with the gatekeeper function of filtering evidence for scientific reliability and validity prior to its admission in court.

Defense attorney Anthony Rios began seeking the data a year ago so that his own expert, psychologist Richard Wollert, could directly compare the predictive accuracy of the Static-99R with that of a competing instrument, the Multisample Age-Stratified Table of Sexual Recidivism Rates," or MATS-1. Wollert developed the MATS-1 in an effort to improve the accuracy of risk estimation by more precisely considering the effects of advancing age. It incorporates recidivism data on 3,425 offenders published by Static-99R developer Karl Hanson in 2006, and uses the statistical method of Bayes's Theorem to calculate likelihood ratios for recidivism at different levels of risk.

The state's attorney objected to the disclosure request, calling the data "a trade secret."

Hanson, the Canadian psychologist who heads the Static-99 enterprise, has steadfastly rebuffed repeated requests to release data on which the family of instruments is based. Public Safety Canada, his agency, takes the position that it will not release data on which research is still being conducted, and that "external experts can review the data set only to verify substantive claims (i.e., verify fraud), not to conduct new analyses,"  according to a document filed in the case.

Thornton estimated that the raw data will remain proprietary for another five years, until the research group finishes its current projects and releases the data to the public domain.

While declining to release the data to the defense, Hanson agreed to release it to Thornton, the government's expert and a co-developer of the original Static-99, so that Thornton could analyze the relative accuracy of the two instruments. 

The American Psychological Association's Ethics Code requires psychologists to furnish data, after their research results are published, to "other competent professionals who seek to verify the substantive claims through reanalysis" (Section 8.14).

At least three five researchers have been rebuffed in their attempts to review Static-99 data over the past few years, for purposes of research replication and reanalysis. As described in their 2008 article, Hanson's steadfast refusals to share data required Wollert and his colleagues, statisticians Elliot Cramer and Jacqueline Waggoner, to perform complex statistical manipulations to develop their alternate methodology. (Correspondence between Hanson and Cramer can be viewed HERE.) Hanson also rejected a request by forensic psychologists Brian Abbott and Ted Donaldson; see comments section, below.

Since the Static-99 family of instruments (which include the Static-99, Static-99R, and Static-2000) began to be developed more than a decade ago, they have been in a near-constant state of flux, with risk estimates and instructions for interpretation subject to frequent and dizzying changes.

It is unfortunate, with the stakes so high, that all of these researchers cannot come together in a spirit of open exchange. I'm sure that would result in more scientifically sound, and defensible, risk estimations in court.

The timing of this latest brouhaha is apropos, as reports of bias, inaccuracy and outright fraud have  shaken the psychological sciences this year and led to more urgent calls for transparency and sharing of data by researchers. Earlier this year, a large-scale project was launched to systematically try to replicate studies published in three prominent psychological journals.

A special issue of Perspectives on Psychological Science dedicated to the problem of research bias in psychology is available online for free (HERE).

*Hat tip to blog reader David Thompson for alerting me that the trial had concluded. 


Jeffrey C. Singer, PhD. said...

One step closer to stopping statistical phrenology!

Anonymous said...

I wonder if they were accused of a sexual assault and were subjected to their own test, how would they fare? If they were to take their own test now, how would they rate?

I am very pleased to see judges keeping researchers and those that create tests like this accountable. If they want their test used widely as it has been then there is no reason to not share how they came up with the test. It isn't like it is some secret recipe how to make Hostess treats, but a test that can have drastic affects on a person's life, and the rest of their lives.

Unknown said...

Hanson and his research group refused to turn over data we requested related to our research on the Static-99R (see Donaldson and Abbott, 2011). While I agree with you about APA ethics for release of data, I wonder if Thornton is immune from following them since he is not a licensed psychologist in Wisconsin. Don't know if he is a member of APA, though.

Karen Franklin, Ph.D. said...

Dr. Abbott,
Thanks for sharing your experience of attempting to obtain Static-99R data. I have updated the blog post (in red) to reflect that.

Karen Franklin, Ph.D. said...

My understanding is that Dr. Thornton did attempt to obtain release of the data, but Dr. Hanson rejected the request. Searching the electronic membership database, neither Hanson nor Thornton appear to be members of the APA. However, in many states, such as California, one need not be a member of the American Psychological Association to be bound by its Ethics Code, as it is codified into state licensing regulations. Someone who is not licensed at all would obviously not face any licensure sanctions for violating the Ethics Code.

Unknown said...

Great blog post, Karen!

Regarding the issue of professionals working together, I want to point out that I invited Dr. Hanson to co-author my 2006 article, "Low Baserates Limit Expert Certainty When Current Actuarial Tests Are Used to Identify Sexually Violent Predators."

He declined.

Jan said...

I think that it is unfair to criticize Hanson for complying with restrictions place on him by his employer. If he is unable to release the data he is unable to release the data. I feel compelled to point out that he and his team collected data from other researchers and performed a variety of analyses on those data. The wrote in detail about the datasets and the analyses they performed. If anyone is so inclined they could contact the same researchers and make the same request.

Unknown said...

I think the reason you attribute to Hanson refusing to turn over data requested by Wollert’s research group and by Ted and I is based on an unfounded assumption. In the responses to the three requests mentioned in the blog, Hanson did not state his employer imposed restrictions that prevented him from releasing the data. In the case of the requests by Ted and me and by Elliot Cramer, Hanson refused to release the data because he did not think the areas we were researching were worthwhile enough. In the Static-99R issue in Wisconsin, Hanson released only part of the requested data and refused to release the data most critical to Wollert’s analysis. These examples make it obvious Hanson’s employer did not prohibit release of the data.

The three requests for data at issue did not ask for release of the entire data set. Ted and I simply requested correlational data for items and total score. The two requests by Wollert’s research team were for frequency data at specific age groups. To say independent researchers would have to request data-sets from all the contributors under the circumstances of the limited requests for data is simply ludicrous. Moreover, Helmus extensively cleaned the 23 data sets using standards that have not been published so it would not be possible to have the exact data upon which they have published results.

Two examples come to mind regarding researchers releasing data upon request. Wollert & Cramer (2011) requested data from Ray Blanchard related to his publication on pedohebephilia. Dr. Blanchard clearly disagreed with the thesis of their research but graciously provided the data. When I was investigating the item correlation issue with the Static-99, Dr. Jan Looman generously provided the data I requested from his sample and he even ran a specific statistical test of the data for me. It is inconceivable to me and inconsistent with the customs of scientific investigation and debate that the Static-99R research group would stymie legitimate efforts to verify their published results or to extend the use of the Static-99/99R data in other areas not explored by the researchers.

Jan said...

I based "assumption" about the reason for Hanson's refusal on the statement in Franklin's blog

" Public Safety Canada, his agency, takes the position that it will not release data on which research is still being conducted, and that "external experts can review the data set only to verify substantive claims (i.e., verify fraud), not to conduct new analyses," according to a document filed in the case."

So it is not "unfounded".

Karen Franklin, Ph.D. said...

Hi Jan,
Perhaps the confusion is because in the blog post I quote a document from the Perren case attributing the decision to agency policy, whereas Dr. Abbott, Dr. Wollert, Dr. Cramer and others have had direct interactions with Dr. Hanson in which he apparently denied requests to share data, without referencing institutional rules.

Unknown said...

Hi Jan:

Perhaps a timeline of events might be helpful. I described the desired data on many occasions (1.04.12, 2.01.12, 10.24.12, and 11.19.12). Attorney Rios filed motions to compel the release of the data on 2.01.12 and 8.13.12. Dr. Thornton replied that he didn't understand the data request on 2.01.12. On 2.02.12 he stated that the data couldn't be released because of Public Safety Canada Policies. On 8.17.12 AAG Dufour filed a motion indicating, among other things, that a) the data may not exist; b) Dr. Thornton doesn't have the raw data; c) the data is privileged as a trade secret; and d) the STATIC team couldn't release the data because of confidentiality agreements with their employers. A hearing on the issue was set for 10.31.12. On 10.10.12 AAG Dufour wrote a letter to Mr. Rios indicating that Leslie Helmus and Dr. Thornton were writing a paper using the data and that "the data you are looking for will be available on November 12, 2012." Mr. Dufour also requested that the hearing be rescheduled. On 11.07.12 Dr. Thornton sent a cover letter and a rough draft of his article with Ms. Helmus to Mr. Dufour. The letter indicated that Dr. Thornton requested the data from Dr. Hanson but that Dr. Hanson initially refused to accede to Dr. Thornton's request. He reportedly agreed to a study that would lead to the data's release after Dr. Thornton proposed to write a paper with Ms. Helmus that compared the Static-99R and the MATS-1. On 11.07.12 Mr. Rios sent me the letter that included the cover letter and rough draft. The draft did not include all of the requested data. On 11.17.12 I sent a letter and an attachment to Mr. Rios indicating "what we asked for, why we asked for it, and why the information we received was not sufficient." On 11.28.12 Judge Levine signed an Order that stated the following: "On October 31, 2012, the petitioner or state informed the court that it talked to Dr. David Thornton, one of the developers of the Static-99R and state witness, and that he agreed to provide the requested data to the respondent. On November 20, 2012, after being advised that Dr. David Thornton failed to provide the state or the respondent with the requested data the court made the following finding: That any testimony or evidence pertaining to the Static-99R, without the disclosure of the requested data, would violate the respondent’s due process rights, specifically, his right to present a defense. IT IS HEREBY ORDERED, that any evidence or testimony pertaining to the actuarial instrument Static-99R is inadmissible."

The foregoing record clearly shows that the process of requesting the Static-99R data was respectful and deliberate, that the AAG held a good-faith belief that Dr. Thornton was committed to releasing the requested data, and that Dr. Thornton did not so.

I agree with Karen that it would be better for everyone if the researchers in our field could work together in a more collaborative way.


Karen Franklin, Ph.D. said...

At Dr. Wollert’s request, here are links to a letter from David Thornton explaining the reasons for denying access to the the data, and the judge’s subsequent order excluding the Static-99R.

Phil Rich said...

Great blog!