The limitations of the systematic approach

Qualitative information

Systematic reviews focusing on the value of treatments given to those with mental health problems usually involves summation of data from randomized trials. Incorporating the great wealth of information from more qualitative approaches in an unbiased way is problematic.

Trial content and quality

Systematic reviews are limited by trial content and quality. For example, it is feasible that, on average, those taking a new drug may have a statistically significant 10-point-greater decrease in a modified Brief Psychiatric Rating Scale (23> score than those taking the comparison treatment. First, this finding is difficult to put into clinically meaningful terms. Second, most scales do not provide 'interval' data. A 10-point change for someone who started with a very high score may not mean the same as the same change for a person entering the study with a lower rating. Third, more problems stem from the modification of the scale. This may well not be published and so validity is questionable. The use of such data is associated with an overestimate of effect.

Undertaking a systematic review of poor-quality trials is an important prerequisite for the design of good studies, but clinical interpretation can be problematic. Rare outcomes

Randomized trials are not a powerful means of identifying rare but important outcomes. For example, large cohorts of those taking the 'atypical' antipsychotic drug, clozapine, suggest a rate of about 1 per cent for agranulocytosis, a serious adverse effect. (24> However, a systematic review of all relevant randomized trials finds a much lower incidence.(6) As the most vulnerable period for the occurrence of agranulocytosis is from weeks 6 to 18 of treatment,(24) and most studies in the systematic review were of shorter duration, the incidence was underestimated.

Trials have limited power to identify rare outcomes and, although systematic reviews may increase this power, reviews of studies of different methodologies may be needed to quantify these important effects.

Limited statistical methods

The statistics used to summate data within meta-analyses are still evolving. For example, much of the continuous scale data, seen so frequently in mental health trials, is not normally distributed. How robust the commonly used methods of meta-analysis are for these non-parametric data is not clear. In addition, as mental health begins to evaluate preventive interventions then cluster randomization, where communities or institutions are randomized rather than individuals, will become more common. The statistics for a meta-analysis of these studies is still a matter of debate. Frequently a systematic review of mental health trials must present, but not summate, all data.

Funny Wiring Autism

Funny Wiring Autism

Autism is a developmental disorder that manifests itself in early childhood and affects the functioning of the brain, primarily in the areas of social interaction and communication. Children with autism look like other children but do not play or behave like other children. They must struggle daily to cope and connect with the world around them.

Get My Free Ebook


Post a comment