A recent analysis of mindfulness research studies (known as a meta-analysis) was published by the Association for Health and Research Quality (AHRQ). This government agency does major reviews of various therapies.

The good news is that mindfulness has come of age to attract such a review. The bad news, if you will, is that the modest results appear to run counter to the current wild enthusiasm for mindfulness. How can we understand this apparent incongruity?

The main reason is methodology. AHRQ reviews only include studies that meet stringent criteria. The vast majority of mindfulness studies did not meet these criteria. In fact, only .2 percent were included. Not all of these excluded studies focused on mindfulness, many implemented transcendental meditation.

Peer review versus gold standard. Grant supported versus Many of the mindfulness studies published over the last thirty-five years have appeared in peer-reviewed journals. However, peer review is not the most stringent criteria. The AHRQ study concludes:

After reviewing 18 753 citations, we included 47 trials with 3515 participants. Mindfulness meditation programs had moderate evidence of improved anxiety (effect size, 0.38 at 8 weeks and 0.22  at 3-6 months), depression 0.30 at 8 weeks and 0.23 at 3-6 month, and pain (0.33) and low evidence of improved stress/distress and mental health–related quality of life. We found low evidence of no effect or insufficient evidence of any effect of meditation programs on positive mood, attention, substance use, eating habits, sleep, and weight. We found no evidence that meditation programs were better than any active treatment (ie, drugs, exercise, and other behavioral therapies).

Understanding this study is part understanding scientific literacy, part philosophy of science, and part epistemology.For more on research methodology read the interview with Willoughby Britton, a neuroscience researcher at Brown University Medical School.

The popular consumption of science contains much misunderstanding. If you read that something has been “proven” be careful. It is very difficult to prove anything. What science can do, with well constructed studies, is disprove things. This is the principle of falsifiability.

If a study gets a positive result that is evidence that supports the hypothesis that this treatment, let’s say, works better than chance. But you never know when future evidence may contradict what you’ve found. Accumulated evidence is good but it is not definitive. A study that finds a negative result may be more definitive if the methodology of the study is robust enough to support that claim. this is known as “power” in statistics and a study must have a sufficient number of subjects and other design characteristics before you conclude that the difference between the treatment group and the non-treatment group (placebo) were no different than chance variations. If scientists replicate such studies and consistently find a non-result, it is reasonable to conclude that the treatment offers nothing beyond a placebo response.

With mindfulness, we simply don’t know enough yet. There is much to encourage that with sufficient methodological rigor, mindfulness intervention studies will continue to demonstrate benefit. There are many other effective forms of treatment, such as cognitive behavioral therapies. Mindfulness may not be superior to these other forms of treatment and this does not mean that it not effective, just not incrementally more effective. Studies study groups and the results of a study report group averages. Individual results will vary.

Efficacy is not the same as effectiveness and a placebo response is always part of a positive result. Studies attempt to remove sources of variability and create a controlled labaroatory envireonment. These conditions do not replicate real life. There is a difference between participating in a research study on stress and being referred by your physician to a mindfulness-based stress reduction group in the community.

The small mindfulness database does not mean that mindfulness does not work. It means that there is not enough rigorous scientific studies yet. It costs more to run a high level study. It requires more subjects and an active control group. this adds complexity and expense. Mindfulness studies are just now getting funded.

In the “real” world outside the laboratory, when people seek treatment, they bring with them a set of expectations for response. This is the power of the placebo response and it is active in almost every treatment, including drug treatments (and especially with drugs that treat anxiety and depression). The goal in clinical practice is not to eliminate the placebo effect. In fact, the opposite. What you don’t want is a treatment that works only via placebo. The results of the AHRQ study suggest this not the case for mindfulness, but more research needs to be done. When people undergo mindfulness-based interventions, they receive benefit.