Los Angeles, CA - When Dr Steven Nissen and Kathy Wolski (Cleveland Clinic, OH) published their 42-trial meta-analysis in the New England Journal of Medicine (NEJM) examining the cardiovascular safety profile of rosiglitazone (Avandia, GlaxoSmithKline [GSK]), they found a 43% statistically significant increase in MI and a 64% increase in CV death that just missed statistical significance [1]. But in a new analysis based on the same trials, but using several different statistical methods, Dr George Diamond (Cedars-Sinai Medical Center, Los Angeles, CA) and colleagues found no significant increase in either end point [2]. They also detail what they say are critical flaws in the way Nissen and Wolski conducted their analysis, although one author agrees there is a "suggestion" of increased risk.

Nissen, in response, says the reanalysis of his work is now neither "useful" nor "relevant" because the FDA has subsequently conducted its own patient-level analysis, available through the FDA's website. Nissen's meta-analysis, as well as the analyses performed by Diamond et al, used study-level data. "We performed a study-level analysis because the company would not provide patient-level data," Nissen said, but he pointed out that the FDA's analysis turned up similar results to his own.

Dr Sanjay Kaul (Cedars-Sinai Medical Center), one of the authors of the new analysis, told heartwire that he and his coauthors were concerned about how Nissen and Wolski selected the studies they included in their analysis, how they were combined, and, in particular, about the fact that their methodological approach, known as the Peto fixed-effect model, could not account for the exclusion of studies in which no cardiovascular events occurred. Given these questions, Nissen's gloomy predicaments about rosiglitazone in the mainstream media following the publication of the original analysis were unjustified, Kaul said.

"You need to utilize every tool you have in your armamentarium to explore the robustness of the data. If different methods are giving you the same answer, you know the data are quite robust. But if different methods are giving you different answers, both qualitatively and quantitatively, you need to be more measured in your pronouncements."

Diamond and Kaul also presented their analysis during the open public hearing at the July 30 FDA advisory panel hearing on rosiglitazone. Their concerns, and the results of their reanalyses of the data, appear online August 6, 2007 in the Annals of Internal Medicine.

Diamond et al take issue with how Nissen and Wolski selected the 42 trials for inclusion in their study, despite having first identified 48 by their predefined inclusion criteria (having a randomized comparator group, plus at least 24 weeks of drug exposure). Six trials, however, were excluded from the start because they did not report any CV events.

"Right off the bat, that is not appropriate," Kaul told heartwire. "When you define a selection strategy, you don't take a peek at the data and then make up your mind whether or not you're going to include that trial. It's an axiom in science that the hypothesis should drive the data; the data should never drive the hypotheses."

It's an axiom in science that the hypothesis should drive the data; the data should never drive the hypotheses.

The problem of null events also complicated the analysis of trials that were included, Diamond et al point out. "Zero-total-event trials may provide relevant information by showing that event rates for both the intervention and control groups are low and relatively equal," they write. By default, the meta-analysis software program used by Nissen and Wolski excluded any trials that had no events of interest in both the treatment and comparator groups—this meant excluding four trials from the MI analysis and 19 trials from the CV death analysis.

"When we saw the analysis in its published form, we were struck by the number of zeros in the database; it was just a very sparse database," Diamond told heartwire. "And they reported only a single meta-analytic method, the Peto method. There are other methods to use, and other things you can do, to adjust for values of zero; they simply dropped the studies out of consideration. . . . So we took it upon ourselves to do the alternative analyses and correction factors, and when we did that we got numbers that were all lower than the single numbers that Nissen and Wolski reported."

In their paper, Diamond et al report six models using continuity corrections for dealing with the zero events. With these, the odds ratio for MI ranged from 1.26 to 1.34, and for CV death from 1.17 to 1.51, with wide confidence intervals, none statistically significant. According to Diamond, he and his colleagues also used Bayesian methods to examine the data, but the journal editors decided it would be "overkill" to include these additional analyses.

"The ones we included are the most conventional," Diamond told heartwire. "Some of them aren't that appropriate. The inverse-variance method I think most people would agree is not suited to this particular purpose. But the problem is there are a lot of zeros in the data, and all the methods have limitations. It's somewhat analogous to having a cheap camera and going out at night and taking some snapshots without a flash attached. Whatever pictures you take are not going to show very much, and it doesn't matter how many pictures you take. The fundamental problem is, there aren't that many events, and the methods are insensitive to the issue at hand when there are so few events."

Diamond continued, "The Peto method, by dropping out the studies with zero values, produces the highest estimate of risk of all the possible alternatives. And I don't know if they knew that at the time, or if they did any other exploratory analyses and reported only the Peto analysis, or if they just didn't think to do them. But our point was to say that the conclusions about increased risk are far more tenuous than are implied in the article."

When you have patient-level data, a study-level analysis is not useful or relevant.

Asked to comment on Diamond et al's analysis, Nissen was dismissive, insisting that the FDA's analysis, presented at the July 30 hearing, was better powered statistically to address the question of cardiovascular risk and included relevant data that neither he nor Diamond et al had access to. Moreover, the FDA's meta-analysis yielded odds ratios closer to those published in the NEJM than those reported by Diamond et al.

"The analysis published in the Annals is a reanalysis of the study-level data for rosiglitazone using more conservative statistical methods; it is no longer relevant," Nissen stated. "A much more statistically powerful, patient-level analysis was presented by the FDA at the advisory board meeting. The FDA analysis reported virtually identical results compared with our NEJM manuscript: a 40% increase in risk with rosiglitazone. When you have patient-level data, a study-level analysis is not useful or relevant."

Diamond et al also take issue with the heterogeneity in study designs and in the populations combined for the analysis, particularly since Nissen and Wolski justified combining the data on the grounds that the studies lacked statistical heterogeneity by the Cochran Q test.

When you combine apples and oranges, sometimes you come up with no more than a hill of beans.

"We believe that the decision to pool all studies despite design and population heterogeneity probably led to artificial inflation and precision of the risk estimate," Diamond, Bax, and Kaul write.

According to Kaul, the Cochran Q test "has low statistical power when the event rates are low. It's usually designed for large sample sizes, large event rates."

Kaul also argues that sufficient clinical heterogeneity may be enough to preclude pooling the trials. "We believe that both clinical as well as statistical heterogeneity should be considered when you're pooling data; otherwise, as somebody once said, when you combine apples and oranges, sometimes you come up with no more than a hill of beans."

Diamond et al point to the fact that some of the trials in the meta-analysis were of diabetics, while others were in nondiabetics, including one study of people with Alzheimer's disease and two with psoriasis patients. One trial included patients with congestive heart failure—a group already identified as a contraindication. Forty of the 42 trials were small and of shorter duration, while the remaining two enrolled more than 4000 patients with a follow-up of three to five years. Then there were differences in treatment groups: some trials were placebo-controlled against rosiglitazone monotherapy, some were studying rosiglitazone monotherapy against active controls, and others were add-on rosiglitazone on a background of standard antidiabetic therapy.

Indeed, during the FDA hearing, panel members saw evidence to suggest that rosiglitazone's cardiovascular risk profile is likely greater in studies that were placebo controlled but that the drug shows little or no suggestion of increased risk in trials that compared rosiglitazone with metformin or with sulfonylurea.

"These meta-analyses work best when they are used as tools, not as weapons, so we have to be very circumspect about these," Kaul said. "I will be the first person to acknowledge there is no single best or correct approach to meta-analyses, especially with sparse data. So when that is the situation, we need to explore alternative methods to examine the sensitivity of the data to the operative assumptions. And the results of the sensitivity analyses should be reported in a transparent manner."

Asked by the FDA advisory panel member Dr Nancy Geller (National Heart, Lung, and Blood Institute, Bethesda, MD) during the hearing whether he had conducted any sort of continuity corrections to account for the zero events, Nissen said that he and Wolski had indeed done so and obtained "essentially identical results."

It's a response that sits ill with Diamond and Kaul, both of whom raised the issue with heartwire. Kaul called it "a mathematical impossibility," while Diamond deemed it "inconsistent with our analyses."

"If you include the zeros, the numbers will change," Diamond said. "If he did do that, he would have had to do that with some other statistical package than the one he reports in the paper, because that package doesn't permit that."

In response, Nissen told heartwire that he and Wolski are providing alternative analyses in an upcoming response to letters to the editor in the NEJM. Moreover, he added, "The question of studies with zero events is not particularly important. These were small short-term studies that contribute almost nothing to the analysis. The fact that the FDA analysis showed nearly identical results validates our approach."

To heartwire, Kaul emphasized that their reanalysis was focused solely on Nissen and Wolski's paper, but he doesn't accept the idea that the FDA's analysis or a meta-analysis performed by GSK is in any way "vindication" for Nissen.

"The FDA and the GSK database analyzed all myocardial ischemic events, which includes serious as well as nonserious events," Kaul said. "When you focus your attention only on the serious events, neither the FDA nor the GSK meta-analysis showed a statistically significant increase in risk."

But one of the FDA reviewers, Dr David Graham, argued during the hearing that even in the absence of statistical significance, the "compass points" for rosiglitazone consistently point in the wrong direction.

"I think there is some validity to that statement," Kaul admitted to heartwire. "If you were to ask me, is there increased risk? I would say yes, there is a suggestion of increased risk, but the data are weak and there is some uncertainty associated with it. Is it certain enough to warrant yanking it off the market? No."