Reading time: 2 minutes, 48 seconds
Research on the tests and measures used in forensic mental health evaluations focuses mainly on their reliability and validity in a carefully controlled research environment. Fewer studies field-test their reliability in real-life cases with consequences for people. Fewer still look at the differences among the evaluators using these tools.
Recent research has found that the reliability of those evaluations is highest in controlled environments, decreases when used in the field, and varies among evaluators, particularly in adversarial settings.
Fortunately, you can improve the reliability of your forensic evaluations by taking comprehensive training and minimizing unconscious biases.
Dr. Daniel Murrie covered these issues and more in his April 10 presentation to the International Forensic Psychiatry Lecture Series.
AI-generated image using Adobe Photoshop
Reliability and Objectivity
He explained 2 key concepts.
- Reliability refers to reaching the same conclusions in response to the same case details as other clinicians, and over time.
- Objectivity refers to whether clinicians can reach the same conclusions regardless of contextual biases or pressures.
“We all want to believe that our judgment is good, that we’re not really vulnerable to biases, and that we reach what is objectively the right answer,” Dr. Murrie said.
“So reliability really matters for justice. A person’s fate shouldn’t depend on which professional handled their case. And so each of us wants to develop insight into our own patterns, our own idiosyncrasies, our own biases.”
Field Studies
Hawaii provides a natural field-reliability study because all court cases involving competence, sanity, and risk after release need 3 independent evaluations. Across 254 competency cases, the evaluators agreed 68% of the time. That decreased to 55% for the 165 sanity cases and 53% in the 62 risk assessments. These numbers are lower than most forensic mental health professionals would want.
A field-reliability study of sexually violent predator evaluations in California had much better results with 93% agreement. That study likely reflected rigorous, uniform training for the evaluators using the STATIC 99R. However, a study with less training had poorer reliability.
Regarding evaluations of trial competence, Dr. Murrie’s team of researchers also looked for patterns among the evaluators in 2 statewide samples. While most evaluators found about 40% of the people to be incompetent, one evaluator found only 9% while another found 77%. “That suggests there is something about the evaluators — not just the defendants — that is influencing the results.”
A field-reliability study in Texas found that 34% of the variance in examinees’ Psychopathy Checklist (PCL-R) scores was due to clinicians, not the examinees, which strongly suggests clinician influence on the scores, Dr. Murrie said.
Evaluator Differences
An evaluator’s training can affect their procedures, including which information they collect and use.
Differences in personalities and values can also affect evaluation outcomes. Research has found that clinicians who were more agreeable or had some psychopathy traits tended to give lower scores. Sociopolitical, racial, ethnic, and other biases have also been shown to affect evaluations.
An empathetic interview style may influence evaluations. Some believe that empathy creates a false sense of alliance, leading defendants to disclose incriminating information they might not otherwise disclose. However, research has not supported that concern. When the researchers instructed evaluators to show greater empathy, there was little effect on the person being evaluated. It was the evaluators who were affected as they tended to see the person more favourably and more sympathetically.
Adversarial Systems
The Hawaii and California studies all had court-appointed evaluators. Things change in the adversarial context of evaluators being hired by the prosecution or defence.
Lawyers typically try to choose forensic mental health evaluators who are more sympathetic to their side or case arguments. The evaluators may also prefer to work with prosecutors or defence lawyers. To remove both biases in selection, the researchers randomly assigned evaluators to each side, providing each with the same case files.
The evaluators tended to select and interpret the information in a biased way that favoured the side that retained them. Evaluators who believed they were working for the prosecution assigned higher risk scores than those who believed they were working for the defence.
For more details on reliability and objectivity in forensic evaluations, watch the recording of the April 3 lecture, “How Reliable and Objective Are Forensic Mental Health Evaluations?” on our website.
Written by Virginia St-Denis and approved by Dr. Daniel Murrie.
