Real-time surgical instrument detection in robot-assisted surgery using a convolutional neural network cascade

• Main Authors: Zijian Zhao, Tongbiao Cai, Faliang Chang, Xiaolin Cheng
• Format: Article
• Language: English
• Published: Wiley 2019-10-01
• Series: Healthcare Technology Letters
• Subjects: object detection; medical image processing; image colour analysis; medical robotics; regression analysis; surgery; learning (artificial intelligence); convolutional neural nets; robot vision; convolutional neural network cascade; robot-assisted surgery videos; vision component; single-tool detection; cascading convolutional neural network; cnn; real-time multitool detection; hourglass network; modified vgg network; detection heatmaps; tool tip areas; bounding-box regression; authors; mainstream detection methods; rgb image frames; frame-by-frame detection method; deep learning methods; endovis challenge dataset; atlas dione dataset; real-time surgical instrument detection; real-time multi-tool detection
• Online Access: https://digital-library.theiet.org/content/journals/10.1049/htl.2019.0064

Surgical instrument detection in robot-assisted surgery videos is an import vision component for these systems. Most of the current deep learning methods focus on single-tool detection and suffer from low detection speed. To address this, the authors propose a novel frame-by-frame detection method using a cascading convolutional neural network (CNN) which consists of two different CNNs for real-time multi-tool detection. An hourglass network and a modified visual geometry group (VGG) network are applied to jointly predict the localisation. The former CNN outputs detection heatmaps representing the location of tool tip areas, and the latter performs bounding-box regression for tool tip areas on these heatmaps stacked with input RGB image frames. The authors’ method is tested on the publicly available EndoVis Challenge dataset and the ATLAS Dione dataset. The experimental results show that their method achieves better performance than mainstream detection methods in terms of detection accuracy and speed.