AUTOMATED, MULTI-CLASS GROUND-TRUTH LABELING OF HE IMAGES FOR DEEP LEARNING USING MULTIPLEXED FLUORESCENCE MICROSCOPY

摘要

Manual annotation of Hematoxylin and Eosin (H&E) stained tissue images for deep learning classification isdifficult, time consuming, and error-prone, particularly for multi-class and rare-class problems. Chemical probes inimmunohistochemistry (IHC) or immunofluorescence (IF) can automatically tag cellular structures; however,chemical labeling is difficult to use in training a deep learning models for H&E images (e.g. through serialsectioning and registration). In this work, we leverage the novel Multiplexed Immuno-Fluorescencent (MxIF)microscopy method developed by General Electric Global Research Center (GE GRC) which allows sequential,stain-image-bleach (SSB) application of protein markers on formalin-fixed, paraffin-embedded (FFPE) samplesfollowed by traditional H&E staining to build chemically-annotated tissue maps for a variety of biomarkers. Thisallows us to use targeted labels for precise, automated creation of ground truth class-label maps for training anH&E-based tissue classifier. In this study, a tissue microarray consisting of 149 breast cancer and normal tissuecores were stained using MxIF for biomarkers DAPI, ribosomalS6, NaKATPase, and Ki67 followed by traditionalH&E staining. The MxIF stains for each TMA core were combined to create a “Virtual H&E” image, which isregistered with the corresponding real H&E images. Each MxIF stained spot was then segmented to obtain a classlabelmap for each analyte, which in turn was used as segmentation ground truth to train a U-net model on H&Eimages. We found that the model performed well in the segmentation of more trivial structures such as nuclei andcytoplasmic regions, but had difficulty in segmenting more subtle structures on H&E such as ki67 positive nucleiand membranous regions.