Developing better methods for monitoring performance in vascular surgery

• Main Author: Jibawi, Abdullah
• Published: University of Brighton 2016
• Subjects: 617.4; A000 Medicine
• Online Access: https://ethos.bl.uk/OrderDetails.do?uin=uk.bl.ethos.694879

Background - Traditional audit methods are limited in their ability to provide short feedback loop to identify underperforming surgical units in time for them to respond appropriately. Moreover, case mix and other confounding factors limit the usefulness of crude mortality figures. More advanced industrial methods such as cumulative sum method (CUSUM) have therefore become of interest to surgeons. Hypothesis – Continuous monitoring of outcome in aortic aneurysm surgery using CUSUM technique (with optimisation using fractional polynomial mathematical models) can be applied, and do provide significant higher and more accurate detection rate of outliers when compared to traditional audit methods. Methods – Using anonymised records from National Vascular Database (NVD), three monitoring systems were applied in real time: Cumulative mortality (reflecting traditional audit process), funnel plot, and CUSUM (SPRT). VBOHM risk score was used to adjust for case-mix. Outliers were detected using different detection levels (h) and odds ratios (OR) with variable mortality rates (p). Performance of the three monitoring models was compared using direct alarm signals, sensitivity and specificity analysis, receiver operating curve (ROC), and average run length (ARL). Choosing control limits to maximise efficiency was approximated using direct simulation, Markov chain, and fractional polynomial techniques. Results –In-hospital mortality following elective Abdominal Aortic Aneurysm (AAA) repair between 1995 and 2011 in 140 centers were monitored. Compared to traditional audit methods, CUSUM has significant sensitivity to the outlier status of each vascular unit, with average number of CUSUM alerts of 0.89 when there is no outlier status, rising up to 23 alerts when there is an outlier status. Maximising the sensitivity and specificity of detecting outliers by CUSUM technique (also called incontrol ARL) while minimising false alarms (also called out-of-control ARL) was achieved using different range of values for control limits (h) and odds ratios (OR). For best CUSUM performance, values of OR=3, p=3, and h=1.25 has been shown to detect outliers correctly in 53% of case, and reject correctly in 59% of cases. This corresponds with CUSUM sensitivity of 80% and specificity of 80%. CUSUM has a positive predictive value of 78% and negative predictive values of 82%, with accuracy reaching 80%. Fractional polynomial technique and CUSUM simulation behavior were shown to correlate well (R > 0.88, p < 0.05) to the real-time NVD data analysis. Conclusion - To the best of our knowledge, our results demonstrated for the first time that using CUSUM (SPRT) model is both feasible and beneficial when used to comprehensively analyze a national dataset, validating the performance of all contributing Units, triggering alarm signals where necessary, and testing the sensitivity and specificity of each detection method with different decision making thresholds.