"DSM-5 Has Flunked Its Reliability Tests" - Newsflash From APA Meeting -SOME RELIABILITY SCORES ARE UNACCEPTABLY LOW ACCORDING TO PROFESSOR ALLEN FRANCES - May 2012 - Courtesy of the Huffington Post website.

 http://www.huffingtonpost.com/allen-frances/dsm-5-reliability-tests_b_1490857.html

Click above to get full article and clearer table of results.

Newsflash From APA Meeting: DSM-5 Has Flunked Its Reliability Tests- So why is it still being published in May 2013??

Posted: 05/08/2012 1:24 pm by Allen Frances M.D.

The whole purpose of having a manual of psychiatric diagnosis is to promote diagnostic agreement. The great value to the field of DSM-III was that it established reliability and preserved the credibility of psychiatry at a time when it was becoming irrelevant because it seemed that psychiatrists could not agree on diagnoses. Everyone knew that the reliability achieved in DSM field testing far exceeds what is possible in clinical practice, but DSM-III took the major step of proving that reliability could be achieved at all. Until now the DSMs have facilitated communication across the clinical/research interface, promoted research, and provided credibility in the court room.

But bad news was recently reported from the annual meeting of the American Psychiatric Association in Philadelphia. The hard-won credibility of psychiatric diagnosis is compromised by the abysmal results reported by the DSM-5 field trials. This failure was clearly predictable from the start:

1. The writing of the DSM-5 criteria sets was far too raw and imprecise to be ready for the rigors of field testing. The ambiguity cried out for expert editing, without which reasonable reliability is impossible.
2. The design of the field trial was byzantine in complexity and could never be done on schedule.
3. Constant delays in starting and completing Stage 1 of the study forced DSM-5 to cancel the planned Stage 2, which was meant to clean up the poorly performing criteria sets identified in the first stage.
4. With Stage 2 cancelled without explanation, it looks like even the worst diagnoses are being given a social pass.
5. Most absurdly, the design was totally off-point, failing to ask the only question that really counted: the impact of DSM-5 on rates.
6.                       DSM5 ; DSM4; ICD10; DSM3
   
   
   GAD.             .2             .65.         .30           .72
   PTSD.           .67           .59.         .76           .55
   Schizophr..   .46           .76.          .79           .81
   Bipolar          .1            .54.          .69
   MDD.            .32          .59.          .53           .80
   Maj neuro    .78.                           .6           .91
   Mild. ".         .50
   Alc use.          .4.                           .71.          .8
   Hoarding.     .59
   BED.              .56
   Bipolar 2.      .40
   MADD.         .06
   APSS.            .46
   OCD              .31
   Antisoc pd.   .22
   Autis spec.    .69         .85.        .77.            .01
   ADHD.          .61.        .59.         .85            .50
   DMDD.         .50
   ODD.            .41.        .55.                         .66
   Conduct.      .48         .57         .78.           .61
   Some of the reliability scores are shamefully poor e.g. antisocial disorder, ODD, OCD, MADD, MDD, GAD and Bipolar 1 + 2.  THIS UNDERMINES THE SCIENTIFIC VALIDITY OF USING DSM5 OUTCOMES AS A RELIABLE INDICATOR OF THESE CONDITIONS

The results of the DSM-5 field trials are a disgrace to the field. For context, in previous DSMs a diagnosis had to have a kappa reliability of about 0.6 or above to be considered acceptable. A reliability of 0.2 to 0.4 has always been considered completely unacceptable, not much above chance agreement.

No predetermined publication date justifies business as usual in the face of these terrible field trial results, which are even more striking given that they were obtained in academic settings with trained and skilled interviewers, highly selected patients, and no time pressure (the results in real-world settings would be much lower). Reliability this low for so many diagnoses gravely undermines the credibility of DSM-5 as a basis for administrative coding, treatment selection, and clinical research.

What can be done to salvage this deplorable mess?

1. DSM-5 has never had anyone on board who could write a clean, consistent, unambiguous criteria set. DSM-5 appears to have received either no editing at all, or amateur editing at best. Getting the words right is certainly not enough, but if you can't even get them right, nothing else can ever be safe.
2. For DSM-5 to retrieve credibility, it must complete the second planned stage of its field testing. If doing the job right must delay publication, so be it. Public trust must trump private publishing profits, and it is self-defeating for APA to publish a book no one can trust.

I have been consistently pessimistic and critical about DSM-5 since my first piece on it three years ago. The sad thing is I can still be so surprised. Each step of the way I predict it will fail in one or another way. But then I discover that DSM-5 has managed to fail in ways that go beyond my poor imagination. This assault on reliability was predicted, but its scope exceeds even my jaundiced fears and creates a DSM-5 emergency.