Leveraging the crowd for annotation of retinal images

• Main Authors: Leifman, George (Contributor), Swedish, Tristan (Contributor), Roesch, Karin (Contributor), Raskar, Ramesh (Contributor)
• Other Authors: Massachusetts Institute of Technology. Media Laboratory (Contributor), Program in Media Arts and Sciences (Massachusetts Institute of Technology) (Contributor)
• Format: Article
• Language: English
• Published: Institute of Electrical and Electronics Engineers (IEEE), 2017-07-07T20:35:57Z.
• Subjects: Article
• Online Access: Get fulltext

 Medical data presents a number of challenges. It tends to be unstructured, noisy and protected. To train algorithms to understand medical images, doctors can label the condition associated with a particular image, but obtaining enough labels can be difficult. We propose an annotation approach which starts with a small pool of expertly annotated images and uses their expertise to rate the performance of crowd-sourced annotations. In this paper we demonstrate how to apply our approach for annotation of large-scale datasets of retinal images. We introduce a novel data validation procedure which is designed to cope with noisy ground-truth data and with non-consistent input from both experts and crowd-workers.