3D-Reconstruction and Semantic Segmentation of Cystoscopic Images

摘要

Bladder cancer (BCa) is the fourth most common cancer and the eighth most common cause of cancer-related mortality in men. Although roughly 75% of patients are diagnosed with non-muscle invasive bladder cancer (NMIBC), recurrence rates are high even at a low stage and grade. Transurethral resection (TURB) is required for establishing the pathological diagnosis and clinical staging of patients. In daily clinical practice, however, conventional tumor documentation after TURB lacks accuracy, posing a major limitation for patient follow-up and surveillance. Novel technologies to facilitate data documentation and interpre-tation processes are imperative to maximize patients' outcomes. As part of the RaVeNNA-4pi initiative, our contribution is twofold: first, we propose a blad-der 3D-reconstruction method using Structure-from-Motion (SfM). Second, we propose deep convolutional neural networks (DCNN) for cystoscopic image seg-mentation to improve the interpretation of cystoscopic findings and localization of tumors. 3D reconstruction of endoscopic images assists physicians in navigating the bladder and monitoring successive resections. Nevertheless, this process is challenging due to an endoscope's narrow field of view (FoV), illumination con-ditions and the bladder's highly dynamic structure. So far in our project, the SfM approach has been tested on a bladder phantom, demonstrating that the processing sequence permits a 3D reconstruction. Subsequently, we will test our approach on bladder images from patients generated in real-time with a rigid cystoscope. In recent years, deep learning (DL) has enabled significant progress in medical image analysis. Accurate localization of structures such as tumors is of partic-ular interest in processing medical images. In this work, we apply a DCNN for multi-class semantic segmentation of cystoscopic images. Moreover, we introduce a new training dataset for evaluating state-of-the-art DL models on cystoscopic images. Our results show that on average a 0.67 Dice score coefficient (DSC) can be achieved.