The apples and oranges of cost-effectiveness: A rejoinder

Health care delivery is perennially resource-constrained, perhaps never more so than in these times of severe economic distress. Yet the introduction of new medical technologies and therapies (some of dubious benefit) continues unabated. Consequently, the search for how best to deploy limited health care resources continues to engender much interest.

In that light, the recent commentary on cost-effectiveness studies by Dr. Vinay Prasad in the June 2012 of this journal,¹ which attempted to highlight some of the pitfalls of such studies, is commendable. Unfortunately, the comments, which largely focused on the methodology of cost-effectiveness studies, end up merely as a straw man debate. To the less well-informed reader, the commentary might appear as an indictment of cost-effectiveness research.

It is thus crucial to correct those potentially misleading comments and to point out that recommendations for the proper conduct of cost-effectiveness studies were published as far back as 1996 by the Panel on Cost-effectiveness in Health and Medicine.² This panel was convened by the US Public Health Service and included members with demonstrated expertise in cost-effectiveness analysis, clinical medicine, ethics, and health outcomes measurement. The recommendations addressed all the issues raised in the commentary and more, and are well worth a read, as they enable readers to understand how to conduct these studies, how to judge the quality of these studies, and how the findings might be applied.² Nonetheless, it is worthwhile to address the logical inaccuracies in the specific examples in the commentary.

IF A TREATMENT IS INEFFECTIVE, IT IS COST-INEFFECTIVE TOO

First, the author discusses the case of vertebroplasty for osteoporotic vertebral fractures. Vertebroplasty had previously been estimated to be cost-effective relative to 12 months of medical therapy. However, a subsequent clinical study found it was no better than a sham procedure, thus setting up the uncomfortable possibility that a sham procedure is more cost-effective than both vertebroplasty and medical therapy.

This can hardly be blamed on the earlier cost-effectiveness study. If any given therapy does not effectively achieve the desired outcomes for the condition for which it is being used, then that therapy ought not to be used at all for that condition. In that context, a cost-effectiveness study is rendered moot in the first place, as the therapy of interest is not effective. Using a more broadly related example, why would anyone conduct a cost-effectiveness study of antibiotics for the treatment of the common cold? Indeed, the vertebroplasty example merely highlights the limitations of the original clinical studies that erroneously deemed it effective for osteoporotic vertebral fractures.

The possibility that a sham procedure might be more cost-effective than vertebroplasty or medical intervention is unsettling to the extent that one has a pro-intervention bias for all diseases. Perhaps the lesson may be that none of the current therapies for this condition is useful, and that until there is a truly beneficial therapy, patients may best be served by doing nothing. To paraphrase one of the author’s rather obvious recommendations, knowing that a therapy is efficacious (toward achieving our desired end point, whatever that may be) should be a prerequisite to adopting it into clinical practice, let alone determining its cost-effectiveness.

Furthermore, cost-effectiveness studies by their nature cannot and should not be static but need to be adjusted over time. For all analyses, it is anticipated that future amendments will be required to adjust for changes in effectiveness (including the disproving of efficacy), changes in relevant strategies available, changes in cost, and changes in population parameters.

WE ALL DIE EVENTUALLY

Secondly, using the example of exemestane (Aromasin) for primary prevention of breast cancer in postmenopausal women, the author raises issues about how to determine the net benefit of preventive therapies in terms of deaths avoided or life-years gained. The particular concern relates to what extent the benefit of deaths avoided by exemestane is negated by deaths that are caused by other non-breast-cancer-related diseases. This implies that using exemestane to prevent death by breast cancer is possibly useless, as those women would go on to die of other causes eventually.

But is that not the case for every preventive or therapeutic intervention? Curing bacterial pneumonia with antibiotics surely saves patients who nonetheless will eventually die some day from another cause. Does this make the use of antibiotics for bacterial pneumonia cost-ineffective? No. The point is that life ultimately ends in death, but along the spectrum of life we utilize various interventions to prolong life and improve its quality as long as is meaningfully possible—either by preventing some diseases or by treating others.

Thus, the implicit assumption ab initio is that prevention or treatment of any particular disease is intrinsically a desirable proposition on its own merits and deserving of some expense of resources. As such, for any given disease, the cost-effectiveness of preventive or therapeutic measures must necessarily be confined to deaths avoided and life-years gained (or other such suitable measures) that are directly attributable to that disease process or to side effects of the particular therapy. Attempting to expand beyond that measure would lead to absurdities such that no intervention would ever be cost-effective because we all eventually die.

REAL-WORLD DATA TAKE YEARS

Finally, using the case of cyclooxygenase 2 inhibitors, the author raises the issue of sourcing data for cost-effectiveness studies.

There is some validity to this point regarding using only real-world experiential data versus data from randomized controlled clinical trials, as vastly different estimates of cost per unit of benefit can be found. However, strict adherence to this recommendation creates a dilemma: real-world data take years to accumulate after an intervention is approved for clinical use based on clinical trial data. But front-line clinicians and payers need to know whether the new intervention is worth adopting into daily clinical practice—particularly because new brand-name, patent-protected therapies generally cost much more early on than later, when patents expire and economies of scale induce drops in prices.

If high acquisition costs without supporting cost-effectiveness data preclude the adoption of the new therapy, then real-world experience cannot be accumulated. On the flip side, unfettered adoption would certainly consume significant resources that may turn out to have been wasted if, years later, real-world experience reveals that the effectiveness was significantly less than estimated by the clinical trial.

However, this is not a problem inherent in cost-effectiveness studies, but rather a result of the uncertainties and difficulties involved in translating findings from clinical trials to the real world, where patients are not as closely monitored to ensure proper compliance and to minimize side effects and uncontrolled interactions. Health economists are well aware of this problem of uncertainty and other limitations of randomized controlled trials.

These limitations have precipitated the development of decision analytic modeling for economic evaluation. This research method is now highly sophisticated and widely accepted as the gold standard. Decision analytic modeling allows data from a trial to be extrapolated beyond the trial period, intermediate clinical outcomes to be linked to final outcomes, clinical trial results to be generalized to other settings, head-to-head comparisons of interventions to be made where relevant clinical trial data do not exist, and economic evaluations to be performed for trials in which economic outcomes were not collected.³

Furthermore, decision analytic modeling in part exists to overcome the data issues raised by the commentary. By using probabilistic sensitivity analyses to account for uncertainties and assure robustness of the results, the reliability of the results is enhanced, regardless of the source of data. In fact, with today’s more powerful computers and software and the limited financial resources available for large randomized controlled clinical trials, the use of economic modeling continues to grow as an indispensable means of economic evaluation.

AN INDISPENSABLE TOOL

In conclusion, properly conducted cost-effectiveness studies are an increasingly important and indispensable tool as we strive to improve the efficiency and effectiveness of health care delivery, particularly in this time of health system changes, the aging of the population, and increasingly limited budgets. Economic modeling allows researchers to explore different scenarios, overcome many of the limitations of clinical trials, identify thresholds at which estimated cost-effectiveness ratios may change, and provide valuable information to health policy makers, providers, and patients to guide the efficient allocation and utilization of health care resources.

Health care delivery is perennially resource-constrained, perhaps never more so than in these times of severe economic distress. Yet the introduction of new medical technologies and therapies (some of dubious benefit) continues unabated. Consequently, the search for how best to deploy limited health care resources continues to engender much interest.

In that light, the recent commentary on cost-effectiveness studies by Dr. Vinay Prasad in the June 2012 of this journal,¹ which attempted to highlight some of the pitfalls of such studies, is commendable. Unfortunately, the comments, which largely focused on the methodology of cost-effectiveness studies, end up merely as a straw man debate. To the less well-informed reader, the commentary might appear as an indictment of cost-effectiveness research.

It is thus crucial to correct those potentially misleading comments and to point out that recommendations for the proper conduct of cost-effectiveness studies were published as far back as 1996 by the Panel on Cost-effectiveness in Health and Medicine.² This panel was convened by the US Public Health Service and included members with demonstrated expertise in cost-effectiveness analysis, clinical medicine, ethics, and health outcomes measurement. The recommendations addressed all the issues raised in the commentary and more, and are well worth a read, as they enable readers to understand how to conduct these studies, how to judge the quality of these studies, and how the findings might be applied.² Nonetheless, it is worthwhile to address the logical inaccuracies in the specific examples in the commentary.

IF A TREATMENT IS INEFFECTIVE, IT IS COST-INEFFECTIVE TOO

First, the author discusses the case of vertebroplasty for osteoporotic vertebral fractures. Vertebroplasty had previously been estimated to be cost-effective relative to 12 months of medical therapy. However, a subsequent clinical study found it was no better than a sham procedure, thus setting up the uncomfortable possibility that a sham procedure is more cost-effective than both vertebroplasty and medical therapy.

This can hardly be blamed on the earlier cost-effectiveness study. If any given therapy does not effectively achieve the desired outcomes for the condition for which it is being used, then that therapy ought not to be used at all for that condition. In that context, a cost-effectiveness study is rendered moot in the first place, as the therapy of interest is not effective. Using a more broadly related example, why would anyone conduct a cost-effectiveness study of antibiotics for the treatment of the common cold? Indeed, the vertebroplasty example merely highlights the limitations of the original clinical studies that erroneously deemed it effective for osteoporotic vertebral fractures.

The possibility that a sham procedure might be more cost-effective than vertebroplasty or medical intervention is unsettling to the extent that one has a pro-intervention bias for all diseases. Perhaps the lesson may be that none of the current therapies for this condition is useful, and that until there is a truly beneficial therapy, patients may best be served by doing nothing. To paraphrase one of the author’s rather obvious recommendations, knowing that a therapy is efficacious (toward achieving our desired end point, whatever that may be) should be a prerequisite to adopting it into clinical practice, let alone determining its cost-effectiveness.

Furthermore, cost-effectiveness studies by their nature cannot and should not be static but need to be adjusted over time. For all analyses, it is anticipated that future amendments will be required to adjust for changes in effectiveness (including the disproving of efficacy), changes in relevant strategies available, changes in cost, and changes in population parameters.

WE ALL DIE EVENTUALLY

Secondly, using the example of exemestane (Aromasin) for primary prevention of breast cancer in postmenopausal women, the author raises issues about how to determine the net benefit of preventive therapies in terms of deaths avoided or life-years gained. The particular concern relates to what extent the benefit of deaths avoided by exemestane is negated by deaths that are caused by other non-breast-cancer-related diseases. This implies that using exemestane to prevent death by breast cancer is possibly useless, as those women would go on to die of other causes eventually.

But is that not the case for every preventive or therapeutic intervention? Curing bacterial pneumonia with antibiotics surely saves patients who nonetheless will eventually die some day from another cause. Does this make the use of antibiotics for bacterial pneumonia cost-ineffective? No. The point is that life ultimately ends in death, but along the spectrum of life we utilize various interventions to prolong life and improve its quality as long as is meaningfully possible—either by preventing some diseases or by treating others.

Thus, the implicit assumption ab initio is that prevention or treatment of any particular disease is intrinsically a desirable proposition on its own merits and deserving of some expense of resources. As such, for any given disease, the cost-effectiveness of preventive or therapeutic measures must necessarily be confined to deaths avoided and life-years gained (or other such suitable measures) that are directly attributable to that disease process or to side effects of the particular therapy. Attempting to expand beyond that measure would lead to absurdities such that no intervention would ever be cost-effective because we all eventually die.

REAL-WORLD DATA TAKE YEARS

Finally, using the case of cyclooxygenase 2 inhibitors, the author raises the issue of sourcing data for cost-effectiveness studies.

There is some validity to this point regarding using only real-world experiential data versus data from randomized controlled clinical trials, as vastly different estimates of cost per unit of benefit can be found. However, strict adherence to this recommendation creates a dilemma: real-world data take years to accumulate after an intervention is approved for clinical use based on clinical trial data. But front-line clinicians and payers need to know whether the new intervention is worth adopting into daily clinical practice—particularly because new brand-name, patent-protected therapies generally cost much more early on than later, when patents expire and economies of scale induce drops in prices.

If high acquisition costs without supporting cost-effectiveness data preclude the adoption of the new therapy, then real-world experience cannot be accumulated. On the flip side, unfettered adoption would certainly consume significant resources that may turn out to have been wasted if, years later, real-world experience reveals that the effectiveness was significantly less than estimated by the clinical trial.

However, this is not a problem inherent in cost-effectiveness studies, but rather a result of the uncertainties and difficulties involved in translating findings from clinical trials to the real world, where patients are not as closely monitored to ensure proper compliance and to minimize side effects and uncontrolled interactions. Health economists are well aware of this problem of uncertainty and other limitations of randomized controlled trials.

These limitations have precipitated the development of decision analytic modeling for economic evaluation. This research method is now highly sophisticated and widely accepted as the gold standard. Decision analytic modeling allows data from a trial to be extrapolated beyond the trial period, intermediate clinical outcomes to be linked to final outcomes, clinical trial results to be generalized to other settings, head-to-head comparisons of interventions to be made where relevant clinical trial data do not exist, and economic evaluations to be performed for trials in which economic outcomes were not collected.³

Furthermore, decision analytic modeling in part exists to overcome the data issues raised by the commentary. By using probabilistic sensitivity analyses to account for uncertainties and assure robustness of the results, the reliability of the results is enhanced, regardless of the source of data. In fact, with today’s more powerful computers and software and the limited financial resources available for large randomized controlled clinical trials, the use of economic modeling continues to grow as an indispensable means of economic evaluation.

AN INDISPENSABLE TOOL

In conclusion, properly conducted cost-effectiveness studies are an increasingly important and indispensable tool as we strive to improve the efficiency and effectiveness of health care delivery, particularly in this time of health system changes, the aging of the population, and increasingly limited budgets. Economic modeling allows researchers to explore different scenarios, overcome many of the limitations of clinical trials, identify thresholds at which estimated cost-effectiveness ratios may change, and provide valuable information to health policy makers, providers, and patients to guide the efficient allocation and utilization of health care resources.

Health care delivery is perennially resource-constrained, perhaps never more so than in these times of severe economic distress. Yet the introduction of new medical technologies and therapies (some of dubious benefit) continues unabated. Consequently, the search for how best to deploy limited health care resources continues to engender much interest.

In that light, the recent commentary on cost-effectiveness studies by Dr. Vinay Prasad in the June 2012 of this journal,¹ which attempted to highlight some of the pitfalls of such studies, is commendable. Unfortunately, the comments, which largely focused on the methodology of cost-effectiveness studies, end up merely as a straw man debate. To the less well-informed reader, the commentary might appear as an indictment of cost-effectiveness research.

It is thus crucial to correct those potentially misleading comments and to point out that recommendations for the proper conduct of cost-effectiveness studies were published as far back as 1996 by the Panel on Cost-effectiveness in Health and Medicine.² This panel was convened by the US Public Health Service and included members with demonstrated expertise in cost-effectiveness analysis, clinical medicine, ethics, and health outcomes measurement. The recommendations addressed all the issues raised in the commentary and more, and are well worth a read, as they enable readers to understand how to conduct these studies, how to judge the quality of these studies, and how the findings might be applied.² Nonetheless, it is worthwhile to address the logical inaccuracies in the specific examples in the commentary.

IF A TREATMENT IS INEFFECTIVE, IT IS COST-INEFFECTIVE TOO

First, the author discusses the case of vertebroplasty for osteoporotic vertebral fractures. Vertebroplasty had previously been estimated to be cost-effective relative to 12 months of medical therapy. However, a subsequent clinical study found it was no better than a sham procedure, thus setting up the uncomfortable possibility that a sham procedure is more cost-effective than both vertebroplasty and medical therapy.

This can hardly be blamed on the earlier cost-effectiveness study. If any given therapy does not effectively achieve the desired outcomes for the condition for which it is being used, then that therapy ought not to be used at all for that condition. In that context, a cost-effectiveness study is rendered moot in the first place, as the therapy of interest is not effective. Using a more broadly related example, why would anyone conduct a cost-effectiveness study of antibiotics for the treatment of the common cold? Indeed, the vertebroplasty example merely highlights the limitations of the original clinical studies that erroneously deemed it effective for osteoporotic vertebral fractures.

The possibility that a sham procedure might be more cost-effective than vertebroplasty or medical intervention is unsettling to the extent that one has a pro-intervention bias for all diseases. Perhaps the lesson may be that none of the current therapies for this condition is useful, and that until there is a truly beneficial therapy, patients may best be served by doing nothing. To paraphrase one of the author’s rather obvious recommendations, knowing that a therapy is efficacious (toward achieving our desired end point, whatever that may be) should be a prerequisite to adopting it into clinical practice, let alone determining its cost-effectiveness.

Furthermore, cost-effectiveness studies by their nature cannot and should not be static but need to be adjusted over time. For all analyses, it is anticipated that future amendments will be required to adjust for changes in effectiveness (including the disproving of efficacy), changes in relevant strategies available, changes in cost, and changes in population parameters.

WE ALL DIE EVENTUALLY

Secondly, using the example of exemestane (Aromasin) for primary prevention of breast cancer in postmenopausal women, the author raises issues about how to determine the net benefit of preventive therapies in terms of deaths avoided or life-years gained. The particular concern relates to what extent the benefit of deaths avoided by exemestane is negated by deaths that are caused by other non-breast-cancer-related diseases. This implies that using exemestane to prevent death by breast cancer is possibly useless, as those women would go on to die of other causes eventually.

But is that not the case for every preventive or therapeutic intervention? Curing bacterial pneumonia with antibiotics surely saves patients who nonetheless will eventually die some day from another cause. Does this make the use of antibiotics for bacterial pneumonia cost-ineffective? No. The point is that life ultimately ends in death, but along the spectrum of life we utilize various interventions to prolong life and improve its quality as long as is meaningfully possible—either by preventing some diseases or by treating others.

Thus, the implicit assumption ab initio is that prevention or treatment of any particular disease is intrinsically a desirable proposition on its own merits and deserving of some expense of resources. As such, for any given disease, the cost-effectiveness of preventive or therapeutic measures must necessarily be confined to deaths avoided and life-years gained (or other such suitable measures) that are directly attributable to that disease process or to side effects of the particular therapy. Attempting to expand beyond that measure would lead to absurdities such that no intervention would ever be cost-effective because we all eventually die.

REAL-WORLD DATA TAKE YEARS

Finally, using the case of cyclooxygenase 2 inhibitors, the author raises the issue of sourcing data for cost-effectiveness studies.

There is some validity to this point regarding using only real-world experiential data versus data from randomized controlled clinical trials, as vastly different estimates of cost per unit of benefit can be found. However, strict adherence to this recommendation creates a dilemma: real-world data take years to accumulate after an intervention is approved for clinical use based on clinical trial data. But front-line clinicians and payers need to know whether the new intervention is worth adopting into daily clinical practice—particularly because new brand-name, patent-protected therapies generally cost much more early on than later, when patents expire and economies of scale induce drops in prices.

If high acquisition costs without supporting cost-effectiveness data preclude the adoption of the new therapy, then real-world experience cannot be accumulated. On the flip side, unfettered adoption would certainly consume significant resources that may turn out to have been wasted if, years later, real-world experience reveals that the effectiveness was significantly less than estimated by the clinical trial.

However, this is not a problem inherent in cost-effectiveness studies, but rather a result of the uncertainties and difficulties involved in translating findings from clinical trials to the real world, where patients are not as closely monitored to ensure proper compliance and to minimize side effects and uncontrolled interactions. Health economists are well aware of this problem of uncertainty and other limitations of randomized controlled trials.

These limitations have precipitated the development of decision analytic modeling for economic evaluation. This research method is now highly sophisticated and widely accepted as the gold standard. Decision analytic modeling allows data from a trial to be extrapolated beyond the trial period, intermediate clinical outcomes to be linked to final outcomes, clinical trial results to be generalized to other settings, head-to-head comparisons of interventions to be made where relevant clinical trial data do not exist, and economic evaluations to be performed for trials in which economic outcomes were not collected.³

Furthermore, decision analytic modeling in part exists to overcome the data issues raised by the commentary. By using probabilistic sensitivity analyses to account for uncertainties and assure robustness of the results, the reliability of the results is enhanced, regardless of the source of data. In fact, with today’s more powerful computers and software and the limited financial resources available for large randomized controlled clinical trials, the use of economic modeling continues to grow as an indispensable means of economic evaluation.

AN INDISPENSABLE TOOL

In conclusion, properly conducted cost-effectiveness studies are an increasingly important and indispensable tool as we strive to improve the efficiency and effectiveness of health care delivery, particularly in this time of health system changes, the aging of the population, and increasingly limited budgets. Economic modeling allows researchers to explore different scenarios, overcome many of the limitations of clinical trials, identify thresholds at which estimated cost-effectiveness ratios may change, and provide valuable information to health policy makers, providers, and patients to guide the efficient allocation and utilization of health care resources.