RO134298A2 - Method for automatic segmentation of anomalies in mammographic images - Google Patents

Abstract

The invention relates to a method for automatic segmentation of anomalies of mass type and of microcalcification type in mammographic images using automatic learning techniques. According to the invention, the method consists in training two models of binary classification of pixels in the mammographic image, one for segmentation of masses and other for segmentation of microcalcifications and, based on the so trained models, a pixel-range probability that they belong to an anomaly or to a normal tissue zone may be determined.

Description

DESCRIPTION

OFFICE OFI STAT PBNTRU INVWT »V TRADEMARKS Patent Application No. ^ .... ^. 213 ..... 22.2. /. ^ .... Date of filing .... Q.» P. .?I WOULD..

The invention relates to a "Method for the automatic segmentation of abnormalities in mammographic images".

The present invention aims at developing a method and a set of algorithms for automatic segmentation of “mass” and “micro-calcifications” ^anomalies t ^Ike2016 l di _n mammographic images, using machine learning techniques (“machine leaming”).

Breast cancer is one of the leading diseases affecting the lives of women worldwide. Statistical data published by the World Health Organization (WHO) show that 23% of all cancers are breast and 14% lead to death [ ^Jem2008 l Among the most effective ways to reduce the mortality rate caused by this disease are programs screening that use mammograms as an imaging method of primary investigation and that allow the detection of the disease from the asymptomatic phase. Dual interpretation (simultaneous and independent visualization of the mammogram by two radiologists) has become the standard in most screening programs in order to reduce the rate of false-negative detection of pathological cases [ ^Tab2005 l. However, this technique requires the allocation of time and costs additional, a digitized mammographic image generally having a size of tens of MegaPixels, and manual visual scanning of the 4 views (Medio-Lateral-Oblice - MLO and Cranio-Caudal - CC of the left breast-L, respectively right-R) associated with an investigation lasting approx. 20 .. 30 min.

The main difficulty in detecting these anomalies (especially masses and microcalcifiers) ^[Ike2016] is given by the process of acquiring 2D mammograms (film or digital): a 3D structure is designed on a 2D plane which leads to structural noise due to overlap of mammographic tissues, especially dense glandular tissues. Because of this 10% -15% of cancers are mammographically invisible, the dense glandular tissue can hide up to 30% ... 50% of cancers nke20i6]

In this context, the present invention proposes the development of methods for automatic segmentation of “mass” and “microcalcifications” type anomalies ^ ^{2m (>} \ * NOTE - with the superscript symbol [] refers to the Bibliography section on the last page (example [1]) ** NOTE - the symbol () refers to components in Figures in the Drawings section

Two models of binary classification of the pixels in the image were trained (one for mass segmentation and one for microcalcification segmentation) which will provide a number of 2019 00018 16/01/2019.% (Description of the invention either an abnormality or belong to a normal tissue area.The segmentation operations of the two types of abnormalities can be run / applied independently.The developed methods are integrated in a software tool that will provide the radiologist with a prediction of the corresponding areas anomalies, thus dramatically reducing the time of examination of mammographic images.

The purpose of this invention is to develop a method and a set of algorithms for automatic segmentation faults of the "mass" and "micro-calcifications" ^ ² ^ of mammographic images using machine learning techniques ( "machine Leaming"). The segmentation operations of the two types of anomalies will be able to run / apply independently. The developed method and algorithms are integrated in a software tool that will provide radiologists with predictions on the areas corresponding to the anomalies, thus dramatically reducing the time of examination of mammographic images and decreasing rate (FNR - false negative rates), given that this process by a human operator is slow and meticulous (about 20 .. 30 minutes) due to the very high resolutions of the images (order of tens of MegaPixels). The final decision on the pathology of the detected abnormalities will be made by the radiologist. In addition to reducing the examination time, the method will have the effect of reducing the false-negative detection rates (NRFs) of mass and microcalcium anomalies compared to the manual visual scanning process.

The problem solved by the invention is a method for automatic segmentation faults of the "mass" and "microcalcifieri" ^ ²⁰ ^, using two different binary classification models are generated through machine learning techniques of "deep Leaming". The developed methods are integrated in a software tool that will provide radiologists with a prediction of the areas that correspond to the anomalies. The components of the software tool referred to in the description are: training data preprocessing routines (3) and testing (4), pixel classification model training routines (5), a routine that implements the prediction / inference mechanism at the level of pixel (6), a command line input / output interface for entering and displaying numeric data (8) and visual display of the segmentation / prediction result (7). One part of these components (1) are used in the training phase (off-line) and the other part (2) in the test / prediction / inference phase (on-line). The two phases are applied separately on the two data sets, "masses" and "microcalcifications", resulting in different training patterns and predictions for the 2 pathologies.

Working principle: Automatic segmentation of mammography images for the detection of the 2 types of abnormalities ("masses" and "microcalcifications") is a process that has 2019 00018

Description of the invention>

performs in two stages: a training stage (1) followed by an inference / prediction stage (2) -fig · 1.

The training process is performed off-line (1) using data sets annotated by radiologists containing 4 images for each examination (2 MedioLateral-Oblique views of the left and right breasts and 2 Cranio-Caudal views of the left breast, respectively Right). The drive process is performed separately on 2 data sets: the data set containing "mass" type anomalies and the data set containing "microcalcifications" type anomalies. following the entrainment process, two distinct models of binary classification of pixels in the image are obtained: a model for classifying pixels in the classes “normal” / “mass” and a model for classifying pixels in the classes “abnormal;’ 7 ”microcalcification”.

The inference / prediction process is performed online (2) by providing a set of new mammographic images, the tool generating at the output for each image a probability map for the detection of "mass" type anomalies and a probability map for the detection of "microcalcification" type. The end result of segmentation is to view these probability maps or binary images obtained by binarizing the probability maps with a fixed threshold provided by the user.

To demonstrate the functionality of the proposed method using a set of reference data from the literature: CBIS-DDSM [ ^Lee20l6 l. I _n total CBIS-DDSM set contains a training set of 1546 images annotated with calcification anomalies and 1318 cases with mass-type abnormalities and a test set with 284 images annotated with calcification-type abnormalities and 348 images with mass-type abnormalities. For each view, the original image ("full grayscale") in 16-bit / pixel TIFF format stored in a DICOM file is provided. The binary masks (Ground Truth) in the DDSM set corresponding to the mass and micro-calcification anomalies were reviewed by radiologists and then their contour was refined by the “level-set” ^method l ^Lee2017 l These images are in 8-bit TIFF format / pixel, stored as a DICOM image. If a view (LMLO, RMLO, LCC, or RCC) contains multiple anomalies, the binary mask set will contain one binary image / mask for each anomaly.

The pre-processing methods (3), (4) applied in both the off-line phase (1) and the on-line phase (2) are:

- Restructuring the training and testing data set used from the original format and structure (DICOM) into a structure and format appropriate to the training process by renaming the files corresponding to full grayscale images and binary masks associated with distinct numbers that index the patient case identifier ( * F_XXXXXX * .dcm, a 2019 00018

16/01/2019 53 (Description of the invention}

respectively and * M_XXXXXX * .dcm), structured in folders related to the anomaly (mass / calcification) and sub-folders related to the processing phase (training / testing)

- Eliminate those pairs of files (full grayscale / binary mask) that do not have the same spatial resolution (height / width) and consequently there are chances that the grayscale visual information and the binary mask are not correlated. This verification is necessary because the existence of such cases can generate ambiguities in the classification models that will be involved and consistently in the results of the inferences on the test images.

- Detection of the real orientation of the mammography image because it was found that the name of the orientation of the mammographic images (Left or Right) encoded in the name of the images in the CBIS-DDSM set does not always correspond to the real orientation. Thus, the orientation of the mammogram was automatically determined by comparative examination of the averages of the intensities in two vertical strips in the image with the limits [kifWidth .. kn * Width \, respectively [(l-kH) * Width .. (1kL) * Width \, the strip with the mean higher also giving the real orientation of the mammogram (fig. 2).

- Injury-centered cutting of training patches: the purpose of generating these types of regions is to provide the classification process of the classifier as many positive examples as possible, to capture the characteristics of the lesions globally (shape / contour). As input is needed the set of mammographic images fu.ll and binary masks associated with each anomaly (no_binary masks = no_full_images x no / lesion / imagefull). The proposed procedure iterates the set of binary masks, and for each anomaly (white object - unique to a mask) is found the midpoint of the rectangle surrounding the lesion. Calculate the coordinates of the square of dimension dim x dim (dim = 2 ⁿ ) centered at the midpoint, and cut from the binary mask. Simultaneously cut the same square from the fiillgrayscale image. The binary masks and the corresponding square regions (ROI) in the grayscale image are stored in folders with suggestive names, in TIFF files with 8 bits / pixel for masks, respectively 16 bits / pixel for ROL. The whole process is illustrated graphically in fig. 3.

Preprocessing methods (3) specific only to the drive / off-line phase (1) are:

- Cutting through the sliding window the drive patches ·, the purpose of generating these types of patches is to provide the drive process of the classifier as many positive examples (translated from the centered ones) and as many negative examples. As input is needed the set of fiill-grayscale mammographic images and binary masks obtained by merging all masks associated with each anomaly (no binary masks = no_full_images x no_injury / no_full_image). This fusion is performed by an operation of or logically at the level of each pixel in the binary masks that are associated with each anomaly that appears in a view (fig. 4.a). At the output, square binary masks will be obtained (TIFF type files, dim = 2, a2019 00018 16/01/2019 (Description of the invention _________________> ________________________________ bit / pixel) and corresponding grayscale regions of interest (ROI) (TIFF type files, dim = 2, 16 bits / pixel - Fig. 4, b) As the parameters of the sliding scheme, the sliding step ("stride") of the window is provided in both directions (horizontal and vertical) which can have any value contained in interval 1 .. dim and a Thmean threshold used to ignore areas in the image that are irrelevant (quasi-black, with lower intensities than Thmean).

The training of the pixel classification models in the image (5) was done to generate pixel classification models for the 2 types of anomalies (masses or microcalcifications), depending on the training set used. The training was performed using “deep leaming” machine learning techniques implemented through convolutional neural networks (CNN). Two approaches were chosen: the first is based on the U-NET architecture, which proved useful for generating binary models giving pixel classification with applications in medical imaging (segmentation of dermatological images, segmentation of tumor cancer cells (CTC) in microscopic images of blood [ ^Moc2018 l; the second approach is based on the ERFNet architecture, which has been successfully used in the multi-class semantic segmentation of images, adapted in this case for the problem of binary segmentation. As input images for the training process used the regions of interest and the corresponding binary masks generated in the preprocessing stage of the training data set (3).

U-Net networks ^Ron2015 l are based on an “encoder-decoder” architecture. For the implementation of the binary pixel classification model, a modified U-NET network was used, which contains 7 levels of encoding and 7 levels of decoding. A coding level has the following structure: convolution with 3x3 size kemel, batch normalization and nonlinear ReLU activation, convolution with 3x3 size kemel, batch normalization and nonlinear ReLU activation, max pooling with 2x2 kemel, A decoding level contains the following operations: deconvolution with kemel 2x2, concatenation with the corresponding layer in the coding, deconvolution with kemel size 3 x 3, batch normalization and nonlinear activation ReLU, deconvolution with kemel size 3x3, batch normalization and nonlinear activation ReLU, deconvolution with kemel size 3x3, normalization batch and nonlinear ReLU activation. The drive and inference mechanism was implemented in the Python programming language, using the TensorFlow and Keras software libraries. Also; the OpenCV procedure library ^[OCV2018] was also used to read and preprocess the information.

The second approach is based on the ERFNet network architecture [ ^Rom2018 l. And ERFNet is an encoder / decoder network. The network architecture has been modified as seen in 2019 00018

16/01/2019 5 /.

(Description of the invention figure 5, so as to fit the size of the drive images. The network consists of an encoder comprising levels of convolutions which reduce the feature maps and make convolutions with different types of filters. The encoder comprises 16 residual levels and reduction but also combined expanded convolutions.Their role is to allow the levels of the network to learn information about the context from the images and at the same time the expanded convolutions have the role of reducing the execution time.The decoder enlarges the feature maps obtained by the encoder until it reaches the original size. decoder levels are deconvolution layers with a displacement factor of 2 pixels.

For the implementation of the inference mechanism (6) we opted for a sliding window scheme („sliding window”) the image scrolling being similar to that of generating the drive patches (3), (fig. 4). The full grayscale image of the test mammogram is needed as input (fig. 6.a and fig. 7.a). The size of the sliding window must be identical to the size of the window used to drive the model (dim = 2 ⁿ ). For the current sliding window, the same intensity normalization operations that were applied in the training process will be applied and the prediction method is used based on the trained classification model. finally, the entire image from the predictions obtained on each patch is reassembled (fig. 6.d and fig. 7.d) in the form of a grayscale image with 8 bits / pixel. Optionally, the result can be binarized to obtain an output image. binary (fig. 6.c and fig. 7.c) with a binary threshold Th jnf For the quantitative evaluation of the result one can calculate the metrics IoU ^[IOU2018] or DSC ^[Fls2018 l by reference to the reference binary mask (Ground Trouth / GT - Fig. 6.b and Fig. 7.b).

Product description. The operating parameters that can be controlled are:

> Absolute paths to folders containing the original training image sets - specified as a string, independently of the two training sets: masses and micro-calipers (eg E: \\ Data_Sets \\ Mammography \ \ CBISDDSM \\ 0MASS \\ Train \\ FULL4sw)> Absolute paths to folders containing preprocessed training image sets - specified as a string, independently for the two training sets: masses and microcalicifiers (e.g. : E: \\ Data_Sets \\ Mammography \\ CBISDDSM \\ OMASS \\ Train \\ 256_ROI, E: \\ Data_Sets \\ Mammography \\ CBISDDSM \\ 0MASS \\ Train \\ 256_MASK) »2019 00018

16/01/2019 (Description of the invention f> The two constants (ki, kn) used for cutting the two vertical strips in the image with the limits [ki * Width .. kH * Width], respectively [(l-kn) * Width. . (l-kL) * Width \, used to detect the real orientation of the mammogram (fig. 2), are subunit values, (ki, <kn), with typical values: ki = 0 ..0.1 and kn ^ 0.02 .. 0.025.

> Sliding window size for generating regions of interest (patches) for both drag and inference: dim = 2 ⁿ - can be any integer, power of 2. It is necessary that the size of the window used in the inference process be identical to that used for training.

> The step of moving the sliding window (stride) can be any integer value: stride = 1 ... dim.

> Thmean threshold is used to ignore in the process of generating entrainment images (ROI and masks) the areas in the image that are irrelevant (quasi-black - fig. 4.b), and which have lower average intensities than Th mean . Typical values for 16-bit / pixel grayscale drive images can be between 1000 ... 10000.

> The binarization threshold of the prediction result to obtain a binary segmented image (Fig. 6.c and fig. 7.c) is an integer value Th_inf = 1 .... 254.

Using the user interface

User interaction with the input interface is done via the command line using alphanumeric characters. The main entries are:

The absolute path to the training image set (off-line phase) or the path to the test image (online phase).

Sliding window size: dim = 2 ⁿ

Sliding window step: stride

Threshold used to ignore areas of the image that are irrelevant in the training process: Th mean

Binary threshold of the prediction result: Th jnf

The output interface consists of messages displayed at the command line regarding the status and results of the execution of the program routines and the graphical visualization of the prediction / segmentation result (fig. 6 and fig. 7).

Open-source software tools used to implement the processing methods of 2019 00018 16/01/2019 description of the invention _______________________________________________________________________________> ____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

Python 3.5 was used to write the routines. The IDE proposed for the development of the application was PyCharm (JetBrains).

Regarding the software tools used, the following dependencies are required: Anaconda3 (for Python 3.7), 64-bit version. With the installation of Anaconda3, Python 3.7 will be installed automatically. To avoid conflicts, it is recommended to uninstall any previous version of Python; Tensorflow does not work with Python 3.7, so downgrade is required. For this, the command is called: “conda install python = 3.5”;

CUDA 9 - downloaded from the official Nvidia website. It is required for the subsequent installation of Tensorflow;

- Tensorflow - is an open source software library for numerical calculations with increased performance. The flexible architecture allows its use on a wide range of platforms (CPU, GPU, etc.). Tensorflow comes with strong support for machine leaming and deep leaming. in this case, the GPU version of Tensorflow was used, on a workstation with the Nvidia video card (for the advantages offered by the CUDA parallel computing platform). The installation is done with the command: “pip install tensorflow-gpu”;

Pyhamcrest - recommended for smooth operation and avoid errors when installing packages. It is installed using the command: “conda install -c conda-forge pyhamcrest”

Keras - is a high-level API for neural networks, capable of running over Tensorflow. Its focus is on fast experiments, which is why it is most often used in deep leaming. The following command is used for installation: “conda install -c conda-forge keras”.

The additional libraries for the Python environment that are needed are the following: Pydicom - for reading DICOM medical images (extension * .dcm)

- NumPy - fundamental package for scientific calculations. Contains Dimensional vectors, fast functions for linear algebra calculations, transforms, random number generator, etc. Also, the data (pixel array) in the image is stored in a NumPy array.

OpenCv Python - was built for increased computational efficiency with a focus for real-time applications. in the application is used for reading images other than the DICOM format, and for operations on the matrices in which the images are stored.

Other libraries used come with the installation of Anaconda3 and can be imported directly into scripts (eg, bone - for parsing files and organizing them).

and 2019 00018

16/01/2019 (Description of the invention ______________>

Technical specifications of system components

Format of input data for preprocessing routines (3), (4)

Mammography format

Binary Mask Format (GT)

Input data format for training routines (5)

Grayscale Region of Interest (ROI) format

The format of the corresponding binary masks (GT)

Output format for training routines (5)

Classification models

Model weights

Input routine format for inference routine (6)

Format of test mammograms

Classification models

Model weights

Input routine output data format

DICOM file (* dcm)

Grayscale images with 16 bti / pixel

Grayscale images with 8 bti / pixel

TIFF image files (* .tiff), LZW compression (no loss)

16-bit / pixel grayscale images

Grayscale images with 8 bits / pixel

Files with the extension * .json and * .h5 Files with the extension * .model

DICOM files containing 16-bit / pixel images

Files with the extension * .json and * .h5

Files with the extension * .model

TIFF image files (* .tiff), LZW compression (no loss), 8 bits / pixel of 2019 00018 16/01/2019 (Description of the invention

Bibliography:

[Fls2018] FI score (Sorensen-Dice coefficient), https://en.wikipedia.org/wiki/S%C3%B8rensen%E2%80%93Dice coefficient, cited nov. 2018.

[Ike2016] Ikeda, D, Kanae K. “Breast Imaging: The Requisites”, Elsevier Health Sciences, 2016.

[IoU2018] Intersection over Union (Jacard index) https://en.wikipedia.org/wiki/Jaccard index, cited nov. 2018.

[Jem2008] Jemal, et al. Cancer statistics CA: a cancer joumal for clinicians 58.2 (2008): 7196.

[Lee2016] R. S. Lee, F. Gimenez, A. Hoogi, D. Rubin (2016). Curated Breast Imaging Subset of DDSM. The Cancer Imaging Archive. http://dx.doi.org/10.7937/K9/TCIA.2016.7Q02S9CY

[Lee2017] Lee, R.S., et al. A curated mammography data set for use in computer-aided detection and diagnosis research. Scientific data 4 (2017): 170177.

[Moc2018] Mocan, L, Itu, R., Ciurte, A., Danescu, R., & Buiga, R. (2018, September). Automatic Detection of Tumor Cells in Microscopic Images of Unstained Blood using Convolutional Neural Networks. In 2018 IEEE 14th International Conference on Intelligent Computer Communication and Processing (ICCP) (pp. 319-324). IEEE.

[OCV2018] Open Computer Vision Library, https://opencv.org/. cited nov. 2018.

[Rom2018] E. Romera, J. M. Âlvarez, L. M. Bergasa and R. Arroyo, ERFNet: Efficient Residual Factorized ConvNet for Real-Time Semantic Segmentation, in IEEE Transactions on Intelligent Transportation Systems, voi. 19, no. 1, pp. 263-272, Jan. 2018.

[Ron2015] Ronneberger, O., Fischer, P., & Brox, T. (2015, October). U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention (pp. 234-241). Springer, Cham, https://lmb.informatik.uni-freiburg.de/people/ronneber/u-net/

[Tab2005] Tabăr, L., Tot, T., & Dean, P. B. (2005). Breast cancer: the art and Science of early detection with mammography: perception, interpretation, histopathologic correlation (pp. 405-438). Stuttgart: Thieme.

a2019 00018

16/01/2019

DRAWINGS

Figure 1. The architecture of the tool for the automatic segmentation of the anomalies in the mammographic images Figure 2. The illustration of the automatic detection of the real orientation of the mammogram: a. Pro-l>pro-r; b. proIL ^{<PROi-rFigura} 3. Illustration of the cross cutting process binary masks of the regions of interest (ROI) corresponding (grayscale)

Fig. 4. Illustration of the procedure for dividing the fat image and the binary mask merged according to a sliding-window scheme: a. Merging the binary masks that correspond to the same views; b. the actual cutting of the regions of interest.

Figure 5. Illustration of the ERFNet network architecture for input images (regions of interest and binary masks) with the size dim = 256.

Figure 6. Example of segmentation of a “mass” type anomaly using the inference technique based on sliding windows

Figure 7. Example of segmentation of a “microcalcification” type anomaly using inference technique based on sliding windows

Claims (6)

claims CLAIMS "Method for automatic segmentation of anomalies in mammographic images" that allows the automatic classification / segmentation of pixels of mammographic images in one of the following 3 categories: healthy, mass or micro-calcification. The method of automatic segmentation of anomalies in mammographic images is characterized by the following novelty elements introduced in the present invention: 1. An architecture of the software system (figure 1) composed of: • An “off-line” phase called “training” (1) and an online phase called inference / prediction (2) • A set of software preprocessing software routines for training (3) and testing phases ( 4) • A set of software routines for training classification models (5) • A software routine for performing the inference / prediction operation for classifying pixels in the image (6) • A user interface such as command line and graphics for entering data and viewing the results (7); 2. A software routine for automatic detection of mammogram orientation (left / right) by analyzing visual information 3. A software routine for the automatic generation of a set of Regions of Interest (ROI) and binary masks (Mask), of variable resolution, centered around the anomalies, for training the classification models with positive examples, starting from a database with annotated mammographic images; 4. A software routine for the automatic generation of a set of Regions of Interest (ROI) and binary masks (Mask), of variable resolution, by the technique of sliding windows with variable step ("stride"), set by the user to drive the classification models with positive and negative examples, starting from a database with annotated mammographic images; 5. A set of software routines for training the pixel classification models in the image, which generate pixel classification models for the two types of anomalies (masses or micro-calcifications), depending on the training set used. 6. A software routine for predicting the membership class for each pixel in the test image