Methods for Optimizing Evidence Syntheses of Complex Interventions: Case Study of a Systematic Review and Meta-Analysis of Diabetes Quality Improvement Trials

• Main Author: Danko, Kristin Julianna
• Other Authors: Grimshaw, Jeremy
• Format: Others
• Language: en
• Published: Université d'Ottawa / University of Ottawa 2018
• Subjects: systematic review; complex interventions; meta-analysis; author contact; hierarchical meta-regression; implementation science; quality improvement; evidence synthesis methodology; cluster randomized trials
• Online Access: http://hdl.handle.net/10393/38225; http://dx.doi.org/10.20381/ruor-22479

Healthcare decision-makers need high quality evidence to inform policy and practice decisions. Systematic reviews of randomized controlled trials (RCTs), including meta- analyses of study effects, are considered one of the highest forms of evidence to inform such decisions. Most applications of systematic reviews and meta-analyses are based on a standardized cannon of methods that seek to collect, abstract, assess, and synthesize evidence from primary studies to produce a comprehensive and unbiased summary of the evidence. While useful, standard synthesis methods tend to assume simple data structures (e.g., two-arm comparison of a single intervention vs. a similar control evaluated in a parallel individual randomized design) and some practices (e.g., author contact) may not always be supported by empirical evidence. Complex interventions are of increasing focus in healthcare and public health and pose challenges to the standard methods of systematic review and meta-analysis. While different definitions of complex interventions have been proposed, most definitions assume: i) multiple intervention ‘components’ that may or may not interact with each other to increase or decrease observed intervention effects and ii) effect modification by study-specific characteristics (e.g., healthcare setting, patient population). At least three challenges may result from this complexity. First, reviewers will likely have to contact authors for additional information about intervention components and contextual factors that may operate as effect modifiers. Unfortunately, evidence supporting optimal strategies for achieving response from author contact is lacking. Second, complex interventions are often evaluated using a cluster randomized trial (CRT) design that randomize units of patients to different healthcare/health policy interventions. Analyses from CRTs that are not adjusted for the clustering effect are said to have unit of analysis errors, which if incorporated in meta-analyses could lead to biased summary estimates and overly precise confidence intervals (CIs). Methods for reviewers to appropriately appraise abstract evidence from CRTs are limited. Thirdly, standard meta-analyses estimate an overall effect of a singular ‘complex intervention’. Such analyses answer the question “Do complex interventions as a whole lead to a difference in observed outcomes?” and tend to exhibit high statistical heterogeneity since variation in intervention components and effect modifiers are not accounted for. Hierarchical multivariate meta-regression models have been proposed as an alternative synthesis approach for complex interventions to better account for observed heterogeneity and answer the question decision-makers are really interested in; that is “What component(s) (or combination of components) work and under what conditions?”. Hierarchical multivariate meta-regression models however have yet to be applied in the review of complex healthcare interventions. The overall aim of my doctoral research was to explore the utility of three methodological approaches to address these challenges and optimize the synthesis of complex interventions using a large systematic review of diabetes quality improvement interventions as a case study. The first objective of this thesis was to do an RCT evaluation of the effect of telephone call versus repeated email contact of non-responding authors for additional study information on response rates and research costs. We found authors contacted by telephone call were more likely to complete requests for additional information (response rate 36.7% vs. 20.2%; adjusted odds ratio 2.26 [95% CI 1.10-4.76]) but the intervention took more time to deliver in total (20 vs. 10 hours over several months vs. one month) and was more expensive overall (approximately $505 vs. $253). The second objective of this thesis was to better account for evidence from CRTs and involved a descriptive study and a methodological study. The descriptive study described the proportion of studies with unit of analysis errors and the nature of the error (inappropriate analysis versus unclear or incomplete reporting). The methodological study investigated the utility of building a database of intracluster correlation coefficients (ICCs) and use of an ICC posterior predictive distribution model to correct unit of analysis errors identified in the descriptive study. We found that although trials often adjusted for the cluster effect (67% across outcomes; range 25%-81%), most did not report enough information to extract adjusted effect estimates required for meta-analysis (an average of 77% of studies with remaining unit of analysis errors across outcomes; range 42%-100%). We were able to construct a posterior predictive distribution of the ICC for most outcomes in our review using estimates of the ICC obtained from the descriptive study combined with external estimates and use these distributions to impute missing ICCs to correct unit of analysis errors. Finally, the third objective of this thesis was to illustrate the use of hierarchical multivariate meta-regression for quantitative synthesis when estimating the effects of complex interventions and exploring effect heterogeneity. Using an arm-based analysis of post-treatment means of one continuous outcome, we demonstrated that hierarchical multivariate meta-regression models can be used to estimate a ‘response surface’ that accounts for complex intervention multiple components and study characteristics, and these models can be used to infer estimates of component effects, interactions among components, and effect modification by study covariates. Collectively the results from this thesis suggest three methodological approaches (contacting authors by telephone, imputing missing ICCs using a predictive distribution, estimating complex intervention effects using a hierarchical multivariate meta-regression) can be used to optimize the processes of synthesizing complex interventions. Further work is needed to evaluate the impact of additional study-covariates on explaining residual heterogeneity and testing these methods in other reviews of complex interventions.