Adherence of Health Economic Evaluations to Country Specific Guidelines: A Protocol for Systematic Review

Introduction

Healthcare decision makers often face the dilemma to allocate finite resources against infinite demands. Health Technology Assessment (HTA) is an informative tool to support such resource allocation decisions [1]. Economic evaluations are the most important component of HTA. The quality of economic evaluations has remained questionable. Hutter et al reported that about 76 methodological reviews were undertaken from 1990 to 2010, to assess the quality of economic evaluations [2]. These reviews identified important gaps in the basic conduct of economic evaluations. Numerous efforts have been undertaken to standardize the methods of economic evaluations which include development of reference case and standard methodological guidelines.

U.S. Panel on Cost-Effectiveness in Health and Medicine in 1996 had proposed the first ever Reference Case [3]. In 2016, the Second Panel released the revised version of the Reference Case [4]. This Panel Reference Case was developed for use by researchers in the United States. International Decision Support Initiative (iDSI) developed similar document for low and middle-income settings [5]. European region and other high-income countries have also developed their own guidelines for economic evaluations [6]. The overall objective of these guidelines is to define best practices and guide the analysts on how to design, conduct and report economic evaluations. Compliance to these guidelines or reference cases would ultimately produce better quality economic evaluations.

Over the years, an increasing number of countries have developed guidelines, however, the relevance of these guidelines to the analysts has rarely been evaluated. It is unclear whether the analysts adhere to these guidelines and if yes to what an extent? In the late 1990s, experiences and applicability of the early versions of the Canadian and Australian guidelines were reviewed [7,8]. Such evaluations are largely missing for other country specific guidelines. We thus propose to undertake a systematic review to assess the adherence of the published economic evaluations to existing country specific guidelines.

Methodology

Around 44 countries have published guidelines for the conduct of economic evaluations either as part of HTA or for pharmacoeconomic evaluation [6]. Out of these 44, around 31 guidelines are available online in English language. Majority of the guidelines (26) belong to high-income countries, only five are from middle-income countries. Further, none of the low-income countries has developed a guideline. We randomly selected, one country each from the high income, upper-middle income, and lower-middle income group having pre-developed guidelines namely Canada, South Africa and Egypt.

Objectives

The primary objective of the systematic review will be to estimate the adherence of the published economic evaluations to the principles of economic evaluation as elaborated in the country specific guidelines. The secondary objective will be to identify individual principles showing poor adherence. In addition, independent factors affecting the adherence will also be identified.

The systematic review will be undertaken in accordance to the systematic review guidelines given by the Cochrane Collaboration for Reviews and Centre for Reviews and Dissemination (CRD) [9].

Literature Search

A comprehensive literature search will be carried out to identify all health economic evaluations pertaining to Canada, South Africa and Egypt. The searches will be performed using the following databases: PubMed, EMBASE and York CRD and the individual websites of the HTA agencies the three countries. Further, bibliometric search of included articles will also be undertaken to identify any additional studies.

These databases will be searched from one year after the publication of the country specific guidelines. The latest version of the guidelines for Canada, South Africa and Egypt have been developed in April 2017 (fourth version), December 2012 (first version) and August 2013 (first version) respectively [10–12]. Accordingly, the searches for economic evaluations will be undertaken from April 2018 for Canada, December 2013 for South Africa and August 2014 for Egypt.

The potential search terms will include those relating to economic evaluations and the selected countries such as ‘cost-effectiveness analysis’; ‘cost-benefit analysis’; ‘cost-utility analysis’, ‘cost-minimization analysis’; ‘pharmacoeconomic evaluation’; ‘health technology assessment’; ‘economic evaluation’; ‘incremental cost effectiveness ratio’ and ‘Canada’; ‘Egypt’ and ‘South Africa’.

Study selection

Two reviewers (DS, ASC) will independently screen title and abstract of individual studies. If considered appropriate full-texts of the selected studies will be screened to check if they meet the inclusion criteria.

A study would be included if it meets the following inclusion criteria:

1. Full economic evaluation that is evaluating both costs and consequences of two or more interventions
2. Economic evaluations pertaining to the health sector
3. Economic evaluation undertaken in any of the three selected countries-Canada, South Africa and Egypt
4. Economic evaluations published one year after the development of the country specific guideline.

A study will be excluded if:

1. It is a partial economic evaluation-cost analysis, cost comparisons, effectiveness studies will be excluded
2. Economic evaluation for non-health related sectors will also be excluded
3. Economic evaluations published in countries other than Canada, South Africa and Egypt will be excluded
4. Economic evaluations published before April 2018 (for Canada), December 2013 (for South Africa) and August 2014 (for Egypt) will also be excluded since we assume that it would take at least one year for the researchers to apply to a particular guideline in their economic evaluation and then publish it.

Quality Appraisal

The quality of the included studies will be assessed by two reviewers (DS, ASC) using Drummonds checklist [13]. This checklist is one of the most widely used checklist for appraising the quality of economic evaluations [14]. It contains a total of 38 criteria divided into 10 main criteria and 28 sub-criteria. The responses to these criteria are to be marked as yes, no, not clear or not applicable.

Data Extraction

A specific data extraction form will be developed to collect general and methodological data. The general information section will include: i) year of publication ii) disease area of the study, iii) type of intervention (pharmaceutical, public health programme, service delivery etc.), iv) author details and v) funding details. The methodological section will include: i) type of economic evaluation, ii) study design, iii) intervention and comparator(s) iv) characteristics of target population v) the perspective adopted vi) study time horizon used vii) discount rates used to discount costs and outcomes, viii) types of costs included ix) outcome measures x) key model parameter if a model was used and xi) Incremental cost-effectiveness ratios (ICERs) and presentation of results.

Three separate adherence questionnaires will be designed for Canada, South Africa and Egypt based on the recommendations provided in the respective country-specific guidelines. Table 1 illustrates some of the key principles and recommendations given by the three guidelines.

Data Analysis

Microsoft Excel will be used for data entry and analysis. Descriptive statistical analysis, including frequency and percentages, will be used to describe the characteristics of the studies. All principles of the economic evaluation will be weighted equally. A score of ‘1’ will be given if the economic evaluations adheres to the recommendation as specified in the guideline, alternately a score of ‘0’ will be given for non-adherence. Total adherence score will be calculated by adding individual scores. Further, individual variables showing poor adherence will be identified.

Independent variables such as type of disease, type of intervention, type of economic evaluation, background and affiliation of the lead author and the source of funding which could possibly impact the adherence to the national guidelines will be identified. The difference between mean adherence score for these variables will be tested using one-way ANOVA.

Conclusion and Policy Implications

Economic evaluations are useful to inform policy decisions, however their applicability is highly dependent on their quality. In order to produce high quality and methodologically sound economic evaluations, researchers attempt to comply with standard best practices as elaborated in the country-specific guidelines. The aim of the proposed systematic review is to assess the adherence of the researchers to three such guidelines developed for Canada, South Africa and Egypt. The findings of our systematic review will help to identify the extent to which these guidelines are accepted among the users, and additionally highlight independent factors, which affect compliance. This in turn would have important implications for countries that are targeting to develop new guidelines or revise their existing guidelines in the near future.

References

1. World Health Assembly. Health intervention and technology assessment in support of universal health coverage. WHA Resolution; Sixty seventh World Health Assembly, 2014. 2014. Available from: http://apps.who.int/medicinedocs/en/m/abstract/Js21463e. Last accessed 2020 Aug 29.
2. Hutter MF, Rodríguez-Ibeas R, Antonanzas F. Methodological reviews of economic evaluations in health care: what do they target?. Eur J Health Econ. 2014;15(8):829-840. doi:10.1007/s10198-013-0527-7. [Google Scholar]
3. Weinstein MC, Siegel JE, Gold MR, Kamlet MS, Russell LB. Recommendations of the Panel on Cost-effectiveness in Health and Medicine. JAMA. 1996 Oct 16;276(15):1253-8. [doi:10.1001/jama.1996.03540150055031]
4. Sanders GD, Neumann PJ, Basu A, Brock DW, Feeny D, Krahn M, et al. Recommendations for Conduct, Methodological Practices, and Reporting of Cost-effectiveness Analyses: Second Panel on Cost-Effectiveness in Health and Medicine [published correction appears in JAMA. 2016 Nov 8;316(18):1924]. JAMA. 2016;316(10):1093-1103. doi:10.1001/jama.2016.12195 [doi:10.1001/jama.2016.12195]
5. Wilkinson T, Sculpher MJ, Claxton K, Revill P, Briggs A, Cairns JA, et al. The International Decision Support Initiative Reference Case for Economic Evaluation: An Aid to Thought. Value Health. 2016;19(8):921-928. doi:10.1016/j.jval.2016.04.015 [doi:10.1016/j.jval.2016.04.015]
6. International Society for Pharmacoeconomics and Outcomes Research [ISPOR]. Pharmacoeconomic guidelines around the world. ISPOR. 2020. Available from: http://www.ispor.org/PEguidelines/index.asp. Last accessed 2020 Aug 29.
7. Baladi JF, Menon D, Otten N. Use of economic evaluation guidelines: 2 years' experience in Canada. Health economics. 1998 May;7(3):221-7. [doi:10.1002/(sici)1099-1050(199805)7:3<221::aid-hec341>3.3.co;2-e]
8. Aristides M, Mitchell A. Applying the Australian guidelines for the reimbursement of pharmaceuticals. Pharmacoeconomics. 1994;6(3):196-201. doi:10.2165/00019053-199406030-00003 [doi:10.2165/00019053-199406030-00003]
9. Higgins JP, Green S. Cochrane handbook for systematic reviews of interventions. Wiley; University of Florida, USA; 2011. [Google Scholar]
10. Canadian Agency for Drugs and Technologies in Health. Guidelines for the economic evaluation of health technologies: Canada. 4th ed. Ottawa: Canadian Agency for Drugs and Technologies in Health; 2017. Available from: https://www.cadth.ca/dv/guidelines-economic-evaluation-health-technologies-canada-4th-edition. Accessed 9 Jul 2020. Last accessed 2020 Aug 29.
11. National Department of Health. Guidelines for Pharmacoeconomic Submissions. December 2012. Available from: https://www.ispor.org/PEguidelines/source/PEGazette_February2013_SouthAfrica.pdf. Last accessed 2020 Aug 29.
12. Pharmacoeconomic Unit, Central Administration for Pharmaceutical Affairs. Guidelines for reporting pharmacoeconomic evaluations in Egypt. 2013. Available from: http://www.eda.mohp.gov.eg/Files/402_Egyptian_Pharmacoeconomic_guidelines.pdf Last accessed 2020 Aug 29.
13. Drummond MF, Sculpher MJ, Claxton K, Stoddart GL, Torrance GW. Methods for the economic evaluation of health care programmes. Oxford university press; 2015 Sep 25 [Google Scholar]
14. Watts RD, Li IW. Use of Checklists in Reviews of Health Economic Evaluations, 2010 to 2018. Value in Health. 2019 Mar 1;22(3):377-82. [doi:10.1016/j.jval.2018.10.006]