Comparative Effectiveness Research: Just How Effective Is It?

With health care expenditures constituting more than 17 percent of the U.S. gross domestic product, lawmakers and physicians continue to search for ways to control costs while providing effective, evidence-based care. Comparative effectiveness research (CER) is touted as a way to investigate and identify measures that can answer important clinical questions and improve the quality of care for patients.

The Journal of the American Medical Association (JAMA) recently devoted an entire special issue to the topic. Recent news coverage has been peppered with discussion of the value of randomized clinical trials, observational studies, and efforts such as the Choosing Wisely campaign, spearheaded by the American Board of Internal Medicine Foundation, to help physicians and patients make better choices about medical tests and procedures.

And as part of the next phase of federally backed CER efforts, last month the Patient-Centered Outcomes Research Institute (PCORI) announced they will spend $120 million to fund comparative clinical effectiveness research.

Given the attention that CER has generated, what’s behind the rapid growth in this field and how can clinicians evaluate and use the results from these studies to inform their current practice and provide the best care to their patients?

Comparative effectiveness research

Only recently known as CER, this established methodology has long been used to evaluate the safety and effectiveness of prescription medications. Its goal is to provide evidence on the harms, benefits, and effectiveness of different treatments to help patients and physicians “…make informed decisions that will improve health care at both the individual and population levels,” according to the Institute of Medicine.

Although CER can be performed prospectively, the majority of these studies are retrospective in nature. Unlike randomized controlled trials (RCTs), retrospective observational studies draw upon information in databases or registries and apply statistical tools to weigh the merits of different treatments. Observational investigations—and CER as a whole—also focus on pertinent clinical issues that lack evidence in the literature or cannot be answered with an RCT.

Because they utilize preexisting data, observational studies use standard and novel statistical methods, such as propensity scores, to reduce confounding variables and identify correlations between treatments and outcomes. The biggest threat to retrospective CER is confounding by indication “in which certain patients may preferentially receive one treatment or another based on their characteristics,” said Wolfgang Winkelmayer, MD, a long-time proponent of CER. With advanced statistical analysis, employing diverse methodologies each with their own advantages and disadvantages, the confounding can be controlled and researchers can identify relevant results. “Ideally, you would like to apply a number of different analytical techniques, and if the analyses yield similar results, then we have greater faith in those findings,” said Winkelmayer.

Increasing interest in CER

Rising health care spending has prompted the federal government to examine options to rein in costs, such as value-based purchasing and quality improvement programs. Between 2009 and 2010 the U.S. government added more than $1 billion of funding specifically for CER through the American Recovery and Reinvestment Act and the Affordable Care Act (ACA).

“The motivation for federal funding for CER is the cost and quality lapses brought on by immense variability in medical practice patterns across the country,” said Carolyn Engelhard, director of health policy at the University of Virginia’s department of public health sciences. She noted research from the Dartmouth Atlas Group has demonstrated “that Medicare spending varies by as much as 30 percent in different parts of the country even after the Medicare data has been controlled for severity of illness and other demographic factors.” Another impetus was the frustration some felt with the well-funded “clinical research that didn’t pertain to a lot of the clinical questions or the types of patients physicians encountered,” Winkelmayer said.

Outside the United States, regulatory agencies use CER to assess new therapeutics and best practices for physicians. In the United Kingdom, the National Institute for Health and Clinical Excellence uses CER along with economic models to evaluate new drugs and devices. Performance of the new treatment in comparison with established therapies determines coverage by the National Health Service. Other countries apply CER to make similar recommendations on diagnostic tests, and to control medication costs to ensure equal access.

The first wave of U.S. government–funded CER studies was overseen by the Agency for Healthcare Research and Quality (AHRQ) and covered such areas as hypertension, spinal disease, and stroke. The next phase of research will be administered through the newly formed PCORI.

Putting the patient first

Patient-centered outcomes research reorients CER by focusing on patient priorities and integrating their perspective at each step in the process. The main distinction between the two approaches is “…the extent to which the preferences, decision-making needs, and characteristics of patients are addressed,” notes PCORI (1).

“PCORI was established under the ACA to give CER a home and to invite various experts and stakeholders in the health industry to participate in the effort,” said Engelhard. With the funding initiatives for PCORI and CER, the government expects “a movement toward consensus, based on scientific evidence, regarding the most effective treatments for various medical conditions.” With that consensus, protocols can be established by professional medical societies to begin standardizing care nationwide, and “once medical care is more standardized it can be measured and managed, which will bring quality improvement,” she said. “Whether it will save money remains to be seen, but certainly it will bring greater value as appropriate care replaces less efficacious care patterns.”

Using multiple criteria to ensure patient involvement in the process, PCORI recently issued its five national priorities for research: 1) assessing options for prevention, diagnosis, and treatment; 2) improving health care systems; 3) researching the best ways to disseminate and communicate findings and recommendations; 4) addressing disparities (which ASN recommended as “a core research priority”); and 5) accelerating patient-centered outcomes research and methodology.

Kidney disease and CER

Because of the multiple facets of kidney disease, there are large gaps in evidence needed to guide many aspects of renal care. A study in the recent JAMA special issue examined one such area in kidney cancer that was missing data to inform treatment. Hung-Jui Tan, MD, and colleagues performed an observational study to compare outcomes after partial or radical nephrectomy for patients 65 years or older with early-stage kidney cancer (2). “In light of new trial data generating new uncertainty regarding the benefits of partial nephrectomy, we became interested in the comparative effectiveness of these treatment options. We elected to use SEER-Medicare data, and chose an instrumental variable approach because it offered the potential to balance both the measured and unmeasured confounders,” said Tan. Their analysis demonstrated that for these patients with kidney cancer, partial nephrectomy was associated with improved survival.

New fields of investigation with CER in nephrology include the comparative effectiveness and safety of certain drug regimens to treat kidney disease, said Winkelmayer. With the inclusion of Medicare Part D data in the United States Renal Data System, researchers now have the opportunity to study medication-based therapeutic strategies for patients with end stage renal disease.

Potential for improving care

Because of the primacy of RCTs, some clinicians may be dissuaded from considering and implementing findings from observational studies.

“Randomized trials often impact medical practice immediately and relatively strongly, and although CER studies may not carry the same weight, they can still be very influential,” said Winkelmayer. Tan also noted that “a well-designed RCT is going to continue to be the gold standard, but they may not provide clear insight into the clinical scenario and they face their own limitations, especially for surgical interventions.” Small patient populations, a long follow-up, or ethical considerations could also make them impractical and/or impossible to perform.

Yet, as a recent JAMA editorial (3) noted, there are important considerations when conducting CER and interpreting the results. Physicians must weigh the statistical methods, effect sizes, and the origins of the data before applying the findings to their practice. A draft protocol for conducting observational CER was released last month by AHRQ and should be finalized this year.

Another concern with CER is using study results to individualize patient care. In their JAMA article (4), David M. Kent, MD, and Nilay D. Shah, PhD, stated that “inferring individual effects from average group effects is an example of the fallacy of division.” However, Mullins et al. (5) noted that involving different groups of patients in the research process could yield “…CER results that go beyond ‘average treatment effects’ and produce results that are applicable to specific patient subgroups.”

Perhaps the greatest challenge for CER is the effective communication and implementation of the recommendations, as several JAMA articles noted. No matter how studies are conducted, if physicians are unaware of new guidelines their clinical practice may remain unaltered. Research to find innovative approaches for disseminating findings and encouraging their incorporation will be funded by both AHRQ and PCORI.

Despite these concerns, CER and PCORI’s approach of engaging populations to identify research goals that are meaningful to patients could help fill gaps in clinical knowledge and improve the health of individuals and communities. Regardless of the study design, Tan concludes “it’s going to take a combination of methods to answer the questions needed to deliver the best care for patients.”


[1] ASN will offer a two-day course entitled “Update on Patient-Centered Outcomes Research in Kidney Disease” October 30–31 as part of Kidney Week 2012.


1.Methodology Committee of the Patient-Centered Outcomes Research Institute. Methodological standards and patient-centeredness in comparative effectiveness research. The PCORI perspective. JAMA 2012; 307:1636–1640.

2.Tan HJ, et al. Long-term survival following partial vs radical nephrectomy among older patients with early-stage kidney cancer. JAMA 2012; 307:1629–1635.

3.Concato J. Is it time for medicine-based evidence? JAMA 2012; 307:1641–1643.

4.Kent DM, Shah ND. Risk models and patient-centered evidence. Should physicians expect one right answer? JAMA 2012; 307:1585–1586.

5.Mullins CD, et al. Continuous patient engagement in comparative effectiveness research. JAMA 2012; 307:1587–1588.

June 2012 (Vol. 4, Number 6)