KR20220095342A - The diagnostic method and system of lymph node metastasis in thyroid cancer using ct image - Google Patents

Abstract

Provided is a method for diagnosing lymph node metastasis of thyroid cancer based on deep learning. The method includes: a step of acquiring a CT image that photographed the thyroid of a subject; a step of generating a crop image by extracting a lesion from the CT image; a step of augmenting the crop image; a step of generating learning information by using a convolutional neural network (CNN) and learning the augmented image; and a step of diagnosing whether thyroid cancer is metastasized to the lymph node based on the learning information.

Description

Method and system for diagnosing lymph node metastasis of thyroid cancer using CT image

The present application relates to a method and system for diagnosing lymph node metastasis of thyroid cancer using a CT image.

The incidence of thyroid cancer has increased rapidly in recent decades. It is known that the diagnosis of lymph node metastasis is the most important in the establishment of a surgical plan for thyroidectomy and the success of the operation. Lymph node metastases are one of the major prognostic factors of thyroid cancer, reported in 60% to 70% of patients, and are an important risk factor for local recurrence.

Therefore, in addition to ultrasound before thyroidectomy, computed tomography (CT) is currently recommended. However, diagnosing cervical lymph node metastasis in thyroid cancer patients by CT imaging has a problem of low sensitivity and specificity.

In thyroidectomy, if the incision is reduced due to a lack of diagnosis, the recurrence rate may increase, and if the incision is excessively incision, there is a problem that increases the surgical complications and reduces the quality of life after surgery. Therefore, it is very important to accurately determine the degree of lymph node dissection before surgery, and the need for related research is emerging.

In this regard, recent attempts to analyze complex imaging and anatomical images using computer aided diagnosis (CAD) systems and deep learning algorithms for accurate and consistent medical images have been continuously attempted to be used for diagnosis. is becoming However, in the CT image, lymph node metastasis and the remaining tissue area are generally affected by noise artifacts, and there is a problem that may overlap with other areas. There are limitations to its use in deep learning algorithms.

Republic of Korea Patent Publication No. 10-2020-0087427, which is the background technology of the present application, relates to a method for diagnosing lymph node metastasis of thyroid cancer using deep learning, and specifically, a pre-processing step in which a multi-dimensional mask image is generated from a radioiodine plane image, In the radioiodine plane image, the likelihood of residual tissue or lymph node metastasis for each pixel is calculated. ) relates to a method for diagnosing lymph node metastasis of thyroid cancer using deep learning, which includes a step in which a learning model is applied and learning is performed, and a step in which lymph node metastasis of thyroid cancer is diagnosed based on the result of neural network learning. However, the above patent does not mention a method for diagnosing lymph node metastasis of thyroid cancer using a CT image.

The present application is to solve the problems of the prior art, and an object of the present application is to provide a method for diagnosing lymph node metastasis of thyroid cancer based on deep learning.

Another object of the present application is to provide a system for diagnosing lymph node metastasis of thyroid cancer based on deep learning.

Another object of the present application is to provide a computer-readable recording medium in which a program for implementing the diagnostic method is recorded.

Another object of the present application is to provide a platform system for diagnosing lymph node metastasis of thyroid cancer based on deep learning.

However, the technical problems to be achieved by the embodiments of the present application are not limited to the technical problems described above, and other technical problems may exist.

As a technical means for achieving the above technical problem, a first aspect of the present application includes the steps of acquiring a CT image of the thyroid gland of a subject, extracting a lesion from the CT image to generate a crop image; Augmenting the cropped image, generating learning information by learning the augmented image using a convolutional neural network (CNN), and whether lymph node metastasis of thyroid cancer based on the learning information It provides a method for diagnosing lymph node metastasis of thyroid cancer based on deep learning, including the step of diagnosing.

According to the exemplary embodiment of the present application, the CT image may be obtained from a patient before thyroidectomy is performed, but is not limited thereto.

According to one embodiment of the present application, the generating of the crop image includes the step of recommending a lesion area of the cervical lymph node from the CT image using a segmentation algorithm or an object detection algorithm, and the recommended location information of the lesion area. It may include performing masking and cropping the masked image, but is not limited thereto.

According to one embodiment of the present application, the step of augmenting the image includes cropping, rotating, flipping, rescale, zooming, zero filling, rolling. (rolling), color jittering, noise addition, contrast adjustment, brightness adjustment (gamma conversion), smoothing (averaging filter), scaling, invert (left/right/up/down), shift (horizontal/vertical), Partial mask (Cut out and Random Erasing), trimming (Random Crop), transformation, discoloration, background replacement, Gaussian transformation, shear, affine, and combinations thereof selected from the group consisting of may be performed, but is not limited thereto.

According to one embodiment of the present application, the method will further include a post-processing step of generating a heat map and removing a region erroneously determined in the step of diagnosing whether thyroid cancer has metastasized to a lymph node by using the heat map However, the present invention is not limited thereto.

A second aspect of the present application provides an acquisition unit for acquiring a CT image of the thyroid gland of a subject, a preprocessing unit for extracting a lesion from the CT image to generate a crop image, and augmentation for augmenting the crop image A unit, a learning unit that generates learning information by learning the augmented image using a convolutional neural network (CNN), and a diagnostic prediction unit that diagnoses whether lymph node metastasis of thyroid cancer is based on the learning information We provide a system for diagnosing lymph node metastasis of thyroid cancer based on deep learning.

According to one embodiment of the present application, the acquisition unit may acquire the CT image from the patient before thyroidectomy is performed, but is not limited thereto.

According to one embodiment of the present application, the preprocessor extracts a lesion area of a cervical lymph node from the CT image using a segmentation algorithm or an object detection algorithm, and performs image cropping on the extracted lesion area to generate the cropped image may be, but is not limited thereto.

According to an embodiment of the present application, the augmentation unit is cropped, rotated, flipped, rescaled, zoomed, zero filling, rolling, Color jittering, Noise Addition, Contrast Adjustment, Brightness Adjustment (Gamma Conversion), Smoothing (Averaging Filter), Scaling, Invert (Left/Left/Up/Down), Shift (Horizontal/Vertical), Partial Mask (Cut) out and Random Erasing), trimming (Random Crop), transformation, discoloration, background replacement, Gaussian transformation, shear, affine, and combinations thereof by performing an operation selected from the group consisting of It may be to enhance the image, but is not limited thereto.

According to one embodiment of the present application, it may further include a generator for generating a heat map and a post-processing unit for receiving the heat map and removing the region erroneously determined in the stage of diagnosing whether the thyroid cancer has metastasized to the lymph node. , but is not limited thereto.

A third aspect of the present application provides a computer-readable recording medium in which a program for implementing the method according to the first aspect of the present application is recorded.

A fourth aspect of the present application is to obtain a CT image data set obtained by photographing the thyroid gland of a subject to learn, extract lesions from the CT image data set to generate a crop image data set, and use the crop image data set to generate a crop image data set. By augmenting (augmentation), using a convolutional neural network (CNN) to learn the augmented image data set to generate learning information, based on the learning information to diagnose whether the lymph node metastasis of thyroid cancer, the A deep learning-based automatic learning unit for receiving and storing and processing a diagnostic information data set, receives a CT image for diagnosing whether thyroid cancer has metastasized to a lymph node from a user, and extracts lesions from the received CT image to generate a cropped image Deep learning for generating learning information by learning the crop image using a convolutional neural network, diagnosing whether lymph node metastasis of thyroid cancer is based on the learning information, and transmitting the diagnosis result to the user terminal Lymph node metastasis of thyroid cancer based on deep learning comprising an API engine for diagnosing lymph node metastasis of thyroid cancer based on an API and a user terminal that obtains the diagnosis result value from the API engine and outputs the diagnosis result of lymph node metastasis of thyroid cancer on the screen in real time Provides a diagnostic platform system.

The above-described problem solving means are merely exemplary, and should not be construed as limiting the present application. In addition to the exemplary embodiments described above, additional embodiments may exist in the drawings and detailed description.

According to the above-described problem solving means of the present application, in the case of a conventional CT image, lymph node metastasis and residual tissue regions are generally affected by noise artifacts and overlap with other regions, and it is impossible to identify them from adjacent tissues. There was a problem in that there was a metastasis case, so there was a limitation in that it was difficult to use the CT image directly in the deep learning algorithm. By learning the cropped image of the extracted lesion region, it is possible to provide a deep learning-based method for diagnosing lymph node metastasis of thyroid cancer with high sensitivity, specificity and accuracy.

In addition, the deep learning-based method for diagnosing lymph node metastasis of thyroid cancer according to the present application can reliably predict whether or not lymph node metastasis occurs, and thus can be very usefully used to establish a treatment strategy, such as the surgical range of thyroidectomy.

In addition, the deep learning-based method for diagnosing lymph node metastasis of thyroid cancer according to the present application provides evidence for an appropriate surgical range for lymph node dissection, thereby reducing the recurrence rate and postoperative morbidity caused by excessive incision. This has the advantage of increasing the treatment efficiency.

In addition, the deep learning-based method for diagnosing lymph node metastasis of thyroid cancer according to the present application can increase the confidence levels of diagnosis, so it can provide clinical usefulness as a training tool to help doctors have confidence in the diagnosis.

In addition, conventional methods for diagnosing lymph node metastasis of thyroid cancer have a limitation in that they cannot provide objective and consistent diagnostic results depending on the skill, experience, etc. of a doctor, but the deep learning-based lymph node of thyroid cancer according to the present application The metastasis diagnosis method has the advantage of being able to predict the lymph node metastasis of thyroid cancer objectively and consistently quickly and conveniently by using deep learning.

In addition, in the case of the conventional diagnostic method using deep learning, it is difficult to explain the internal relationship between the input value and the result predicted by deep learning, so it is difficult to evaluate the diagnostic accuracy of deep learning. The learning-based method for diagnosing lymph node metastasis of thyroid cancer generates and provides a heat map to infer which part of the CT image the deep learning model is focusing on. An improved diagnostic method can be provided.

In addition, the deep-learning-based thyroid cancer lymph node metastasis diagnosis system according to the present application performs a post-processing step of removing erroneously judged areas through a heat map, integrating reports on erroneous results to upgrade the CAD system's performance. It is possible to provide a diagnostic system having a high scalability.

In addition, the deep learning-based thyroid cancer lymph node metastasis diagnosis platform system according to the present application provides real-time whether or not lymph node metastasis of thyroid cancer is present to the user terminal using the API engine, so that the user can easily check whether the diagnosis is made without specialized knowledge. system can be provided.

In addition, the deep learning-based thyroid cancer lymph node metastasis diagnosis platform system according to the present application automatically generates a crop image based on the CT image received from the user, so that even if a user lacking specialized knowledge does not know exactly the lesion site, lymph node metastasis It is possible to know whether or not to provide a platform system with increased user convenience.

In addition, the deep learning-based thyroid cancer lymph node metastasis diagnosis platform system according to the present application can continuously learn new images provided by various users by utilizing the API engine, so that the diagnosis accuracy and diagnosis speed can be continuously improved. .

In addition, the deep learning-based thyroid cancer lymph node metastasis diagnosis platform system according to the present application is supplied to a variety of users to provide services related to thyroid cancer lymph node metastasis diagnosis, so it is useful as a platform for developing various application services. can be

However, the effects obtainable herein are not limited to the above-described effects, and other effects may exist.

1 is a flowchart illustrating a method for diagnosing lymph node metastasis of thyroid cancer based on deep learning according to an embodiment of the present application.
2 is a flowchart of a method for diagnosing lymph node metastasis of thyroid cancer based on deep learning according to an embodiment of the present application.
3 is an image showing a system for diagnosing lymph node metastasis of thyroid cancer based on deep learning according to an embodiment of the present application.
4 is a flowchart of a system for diagnosing lymph node metastasis of thyroid cancer based on deep learning according to an embodiment of the present application.
5 is a graph showing a heat map of a method for diagnosing lymph node metastasis of thyroid cancer based on deep learning according to an embodiment of the present application.
6 is a table showing the results of evaluating the diagnostic performance of the CAD model of the method for diagnosing lymph node metastasis of thyroid cancer based on deep learning according to an embodiment of the present application.
7 is a graph illustrating a receiver operating characteristic curve of a CAD model of a method for diagnosing lymph node metastasis of thyroid cancer based on deep learning according to an embodiment of the present application.
8 is a graph showing a Sankey plot of a change in confidence levels before and after the use of the deep learning-based method for diagnosing lymph node metastasis of thyroid cancer according to an embodiment of the present application.

Hereinafter, embodiments of the present application will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art to which the present application pertains can easily carry out. However, the present application may be implemented in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present application in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout this specification, when a part is "connected" with another part, this includes not only the case where it is "directly connected" but also the case where it is "electrically connected" with another element interposed therebetween. do.

Throughout this specification, when a member is positioned “on”, “on”, “on”, “on”, “under”, “under”, or “under” another member, this means that a member is positioned on the other member. It includes not only the case where they are in contact, but also the case where another member exists between two members.

Throughout this specification, when a part "includes" a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated.

As used herein, the terms "about," "substantially," and the like are used in a sense at or close to the numerical value when the manufacturing and material tolerances inherent in the stated meaning are presented, and to aid in the understanding of the present application. It is used to prevent an unconscionable infringer from using the mentioned disclosure in an unreasonable way. Also, throughout this specification, "step to" or "step to" does not mean "step for".

Throughout this specification, the term "combination of these" included in the expression of the Markush form means one or more mixtures or combinations selected from the group consisting of the components described in the expression of the Markush form, and the components It is meant to include one or more selected from the group consisting of.

Throughout this specification, reference to “A and/or B” means “A, B, or A and B”.

Hereinafter, a method and system for diagnosing lymph node metastasis of thyroid cancer using a CT image of the present application will be described in detail with reference to embodiments, examples, and drawings. However, the present application is not limited to these embodiments and examples and drawings.

As a technical means for achieving the above technical problem, a first aspect of the present application includes the steps of acquiring a CT image of the thyroid gland of a subject, extracting a lesion from the CT image to generate a crop image; Augmenting the cropped image, generating learning information by learning the augmented image using a convolutional neural network (CNN), and whether lymph node metastasis of thyroid cancer based on the learning information It provides a method for diagnosing lymph node metastasis of thyroid cancer based on deep learning, including the step of diagnosing.

1 is a flowchart illustrating a method for diagnosing lymph node metastasis of thyroid cancer based on deep learning according to an embodiment of the present application, and FIG. 2 is a flowchart illustrating a method for diagnosing lymph node metastasis of thyroid cancer based on deep learning according to an embodiment of the present application .

In order to perform the deep learning-based method for diagnosing lymph node metastasis of thyroid cancer, a step of acquiring a CT image of the thyroid gland of the subject is first performed (S100).

According to the exemplary embodiment of the present application, the CT image may be obtained from a patient before thyroidectomy is performed, but is not limited thereto.

In thyroidectomy, if the incision is reduced due to a lack of diagnosis, the recurrence rate may increase, and if the incision is excessively incision, there is a problem that increases the surgical complications and reduces the quality of life after surgery. Therefore, it is very important to accurately determine the degree of lymph node dissection before surgery.

The deep learning-based method for diagnosing lymph node metastasis of thyroid cancer according to the present application can reliably predict whether or not lymph node metastasis occurs, and thus can be very usefully used to establish a treatment strategy such as the surgical range of thyroidectomy.

In addition, the deep learning-based method for diagnosing lymph node metastasis of thyroid cancer according to the present application provides evidence for an appropriate surgical range for lymph node dissection, thereby reducing the recurrence rate and postoperative morbidity caused by excessive incision. This has the advantage of increasing the treatment efficiency.

In addition, the deep learning-based method for diagnosing lymph node metastasis of thyroid cancer according to the present application can increase the confidence levels of diagnosis, so it can provide clinical usefulness as a training tool to help doctors have confidence in the diagnosis.

In addition, conventional methods for diagnosing lymph node metastasis of thyroid cancer have a limitation in that they cannot provide objective and consistent diagnostic results depending on the skill, experience, etc. of a doctor, but the deep learning-based lymph node of thyroid cancer according to the present application The metastasis diagnosis method has the advantage of being able to predict the lymph node metastasis of thyroid cancer objectively and consistently quickly and conveniently by using deep learning.

Then, a step of generating a crop image by extracting the lesion from the CT image is performed (S200).

In the case of conventional CT images, lymph node metastasis and remaining tissue regions are generally affected by noise artifacts and overlap with other regions. Although there was a limitation in that it was difficult to directly use the deep learning algorithm, the deep learning-based thyroid cancer lymph node metastasis diagnosis method according to the present application extracts the lesion area of the cervical lymph node from the CT image and learns the crop image of the extracted lesion area. By doing so, it is possible to provide a method for diagnosing lymph node metastasis of thyroid cancer based on deep learning with high sensitivity, specificity and accuracy.

According to one embodiment of the present application, the generating of the crop image includes the step of recommending a lesion area of the cervical lymph node from the CT image using a segmentation algorithm or an object detection algorithm, and the recommended location information of the lesion area. It may include performing masking and cropping the masked image, but is not limited thereto.

For example, the location of the lesion region of the expected cervical lymph node may be recommended by performing various image processing, such as Gaussian filter, affine transformation, contrast adjustment, etc., in addition to the segmentation algorithm or the object detection algorithm. .

The location of the lesion region of the predicted cervical lymph node may be recommended by applying the artificial intelligence algorithm. The lesion area of the predicted cervical lymph node may be a two-dimensional definition of the boundary of the lesion in a rectangular shape. That is, the location of the lesion can be recommended by displaying the lesion area of the predicted cervical lymph node in a rectangular shape at the location predicted as the lesion of the cervical lymph node. When a plurality of predicted lesions exist, a plurality of predicted lesion regions corresponding thereto may be recommended.

Next, performing masking in consideration of the recommended location information of the lesion area (eg, x, y coordinate location, horizontal length, vertical length) and the CT image, and performing cropping on the masked image It may include, but is not limited to.

Next, a step of augmenting the cropped image is performed (S300).

Since the training data has a higher number of benign lymph nodes than the metastatic lymph node image, the data can be augmented so that the ratio of the two labels is similar, preventing overfitting of the model and improving the model performance. That is, by augmenting the cropped image, the amount of training images is effectively increased based on a limited training data set.

According to one embodiment of the present application, the step of augmenting the image includes cropping, rotating, flipping, rescale, zooming, zero filling, rolling. (rolling), color jittering, noise addition, contrast adjustment, brightness adjustment (gamma conversion), smoothing (averaging filter), scaling, invert (left/right/up/down), shift (horizontal/vertical), Partial mask (Cut out and Random Erasing), trimming (Random Crop), transformation, discoloration, background replacement, Gaussian transformation, shear, affine, and combinations thereof selected from the group consisting of may be performed, but is not limited thereto.

For example, the step of augmenting the image includes randomly setting the image angle within about ±20º, randomly changing the size of the area around the lymph node by about 20%, and randomly changing the image horizontally and vertically including the lymph node itself. It may be to perform inversion, but is not limited thereto.

Next, a step of generating learning information by learning the augmented image using a convolutional neural network (CNN) is performed (S400).

For example, the convolutional neural network may be a model selected from the group consisting of VGG16, VGG19, Xception, InceptionV3, InceptionResNetV2, DenseNet121, DenseNet169, ResNet, EfficientNet, MobileNet, and combinations thereof, but is not limited thereto. .

Next, a step of diagnosing whether thyroid cancer has metastasized to a lymph node based on the learning information is performed (S500).

According to one embodiment of the present application, the method will further include a post-processing step of generating a heat map and removing a region erroneously determined in the step of diagnosing whether thyroid cancer has metastasized to a lymph node by using the heat map However, the present invention is not limited thereto.

In the case of the conventional diagnosis method using deep learning, it was difficult to explain the internal relationship between the input value and the result predicted by deep learning, so it was difficult to evaluate the diagnostic accuracy of deep learning. of thyroid cancer lymph node metastasis diagnosis method generates and provides a heat map to infer which part of the CT image the deep learning model is focusing on. method can be provided.

Specifically, the heat map may show the location where the CNN model is focused to differentiate between benign and metastatic lymph nodes. This suggests that the abnormal characteristics of malignant tumors were learned by CNN and used as the basis for the diagnosis of lymph node metastasis of thyroid cancer.

Since the generated heat map is superimposed on the original CT image image, the location of the actual lymph node and the area used by the CNN model for metastasis diagnosis can be compared, the user can compare and remove the area judged incorrectly by the deep learning model. There is an advantage of further increasing specificity and sensitivity.

A second aspect of the present application provides an acquisition unit for acquiring a CT image of the thyroid gland of a subject, a preprocessing unit for extracting a lesion from the CT image to generate a crop image, and augmentation for augmenting the crop image A unit, a learning unit that generates learning information by learning the augmented image using a convolutional neural network (CNN), and a diagnostic prediction unit that diagnoses whether lymph node metastasis of thyroid cancer is based on the learning information We provide a system for diagnosing lymph node metastasis of thyroid cancer based on deep learning.

With respect to the deep learning-based thyroid cancer lymph node metastasis diagnosis system of the second aspect of the present application, detailed descriptions of parts overlapping with the first aspect of the present application are omitted, but even if the description is omitted, the first aspect of the present application The described content is equally applicable to the second aspect of the present application.

According to one embodiment of the present application, the acquisition unit may acquire the CT image from the patient before thyroidectomy is performed, but is not limited thereto.

According to one embodiment of the present application, the preprocessor extracts a lesion area of a cervical lymph node from the CT image using a segmentation algorithm or an object detection algorithm, and performs image cropping on the extracted lesion area to generate the cropped image may be, but is not limited thereto.

According to an embodiment of the present application, the augmentation unit is cropped, rotated, flipped, rescaled, zoomed, zero filling, rolling, Color jittering, Noise Addition, Contrast Adjustment, Brightness Adjustment (Gamma Conversion), Smoothing (Averaging Filter), Scaling, Invert (Left/Left/Up/Down), Shift (Horizontal/Vertical), Partial Mask (Cut) out and Random Erasing), trimming (Random Crop), transformation, discoloration, background replacement, Gaussian transformation, shear, affine, and combinations thereof by performing an operation selected from the group consisting of It may be to enhance the image, but is not limited thereto.

According to one embodiment of the present application, it may further include a generator for generating a heat map and a post-processing unit for receiving the heat map and removing the region erroneously determined in the stage of diagnosing whether the thyroid cancer has metastasized to the lymph node. , but is not limited thereto.

The deep learning-based thyroid cancer lymph node metastasis diagnosis system according to the present application performs a post-processing step to remove the wrongly judged area through the heat map, thereby integrating the report on the wrong result and expanding the CAD system's performance to be upgraded. It is possible to provide a diagnostic system having characteristics.

A third aspect of the present application provides a computer-readable recording medium in which a program for implementing the method according to the first aspect of the present application is recorded.

With respect to the computer-readable recording medium in which the program of the third aspect of the present application is recorded, detailed descriptions of parts overlapping with the first and second aspects of the present application are omitted, but even if the description is omitted, the first aspect of the present application The contents described in the first aspect and the second aspect may be equally applied to the third aspect of the present application.

A fourth aspect of the present application is to obtain a CT image data set obtained by photographing the thyroid gland of a subject to learn, extract lesions from the CT image data set to generate a crop image data set, and use the crop image data set to generate a crop image data set. By augmenting (augmentation), using a convolutional neural network (CNN) to learn the augmented image data set to generate learning information, based on the learning information to diagnose whether the lymph node metastasis of thyroid cancer, the A deep learning-based automatic learning unit for receiving and storing and processing a diagnostic information data set, receives a CT image for diagnosing whether thyroid cancer has metastasized to a lymph node from a user, and extracts lesions from the received CT image to generate a cropped image Deep learning for generating learning information by learning the crop image using a convolutional neural network, diagnosing whether lymph node metastasis of thyroid cancer is based on the learning information, and transmitting the diagnosis result to the user terminal Lymph node metastasis of thyroid cancer based on deep learning comprising an API engine for diagnosing lymph node metastasis of thyroid cancer based on an API and a user terminal that obtains the diagnosis result value from the API engine and outputs the diagnosis result of lymph node metastasis of thyroid cancer on the screen in real time Provides a diagnostic platform system.

With respect to the deep learning-based thyroid cancer lymph node metastasis diagnosis platform system of the fourth aspect of the present application, detailed descriptions of parts overlapping with the first to third aspects of the present application are omitted, but even if the description is omitted, the present application The contents described in the first aspect to the third aspect can be equally applied to the fourth aspect of the present application.

3 is an image showing a deep learning-based thyroid cancer lymph node metastasis diagnosis platform system according to an embodiment of the present application, and FIG. 4 is a deep learning-based thyroid cancer lymph node metastasis diagnosis platform system according to an embodiment of the present application. It is a flow chart.

The deep learning-based thyroid cancer lymph node metastasis diagnosis platform system according to the present application automatically generates a crop image based on the CT image received from the user, so that even if a user lacking specialized knowledge does not know exactly the lesion site, it can detect whether lymph node metastasis or not. Therefore, it is possible to provide a platform system with increased user convenience.

In addition, the deep learning-based thyroid cancer lymph node metastasis diagnosis platform system according to the present application can continuously learn new images provided by various users by utilizing the API engine, so that the diagnosis accuracy and diagnosis speed can be continuously improved. .

In addition, the deep learning-based thyroid cancer lymph node metastasis diagnosis platform system according to the present application is supplied to a variety of users to provide services related to thyroid cancer lymph node metastasis diagnosis, so it is useful as a platform for developing various application services. can be

The present invention will be described in more detail through the following examples, but the following examples are for illustrative purposes only and are not intended to limit the scope of the present application.

[Example 1] Acquisition of CT image

The protocol was approved by the Ethics Committee of the Institutional Review Board, Ajou University Medical Center, Suwon, Korea, from 698 thyroid cancer patients who underwent CT for surgical planning between January and August 2018, benign (n = 3606) and malignant (n = 232) a total of 3838 axial CT images of lymph nodes (>8 mm to 10 mm) were collected.

All CT images were acquired using a 64-channel to 128-channel multi-detector CT scanner (SOMATOM Definition Flash, Siemens Healthineers, Brilliance, Philips Healthcare, Aquilion One GENESIS, Canon Medical Systems). CT images were obtained from 0.5 mm to 0.6 mm and were reconstructed as axial images every 2.0 mm in a 512 × 512 matrix using an iterative reconstruction algorithm associated with each vendor's CT scanner.

[Example 2] Crop image generation

Crop images were generated using segmentation algorithm or object detection algorithm.

[Example 3] Lymph node metastasis diagnosis of thyroid cancer

Since the training data had a higher number of positive lymph nodes than the metastatic lymph node image, the crop image obtained in Example 2 was augmented so that the ratio of the two labels was similar.

Specifically, in order to prevent overfitting of the model and improve model performance, the image angle was randomly set within ±20º to enhance the CT image, and the size of the area around the lymph node was randomly changed up to 20%, and the image was Randomly inverted horizontally and vertically, including the lymph node itself.

Then, standard deviation normalization for each feature was applied while reading the augmented image by dividing the standard deviation. After that, all augmented training datasets were resized to 224 x 224 pixels to standardize the distance scale, and images were converted to 8 CNN models: VGG16, VGG19, Xception, InceptionV3, InceptionResNetV2, DenseNet121, DenseNet169, ResNet, EfficientNet, MobileNet. After input into , it was trained.

Specifically, all training sessions were performed using the deep learning package Keras (http://keras.io/) with a tensor flow machine learning backend. The batch size was set to 30 and the learning rate was set to 0.005. To optimize the categorical cross entropy, an ADAM optimizer with momentum set to 0.9 and weight reduction set to 0.00005 was used. Since the CNN network has a large number of parameters, training on the training data continued until the accuracy of the validation set decreased.

Eight deep learning algorithms were trained on the same training sample, and model weight values were generated for each generation (iteration through the entire data set). Training was stopped after 100 epochs because the accuracy and cross entropy did not improve any further.

[Example 4] Heat map generation

5 is a graph showing a heat map of a method for diagnosing lymph node metastasis of thyroid cancer based on deep learning according to an embodiment of the present application.

Specifically, a CAM (class activation mapping) method was used to replace the connected layer with a global average pooling (GAP) layer. This is an efficient way to capture in most representative regions to predict labels using image-level annotations without location information.

Specifically, a model was created that classifies and localizes data in a single model at the same time without using location information through the CAM method, so that the predicted label is connected with the focal region indicated on the heat map. Therefore, the GAP layer was applied to the last convolutional layer of the backbone model because the activation includes the model detail function.

Activation maps generated by the CAM in the test data set were applied to evaluate regions of interest for further clinical review. Layer dropout is a method of removing neurons from fully connected layers to prevent overfitting at each training iteration.

Specifically, we set up two dropout layers after the base model between two fully connected layers with the final soft max activation layer.

Subsequently, the VGG model was used together with the GAP method to understand how the CNN model arrives at the prediction result. Instead of adopting the existing fully connected layer and flattening the last convolutional layer, the GAP method was used to directly input the spatial average of the feature map of each label in the last convolutional layer. Therefore, it was possible to draw a heat map related to lymph node malignancy. This is the weighted sum of the spatial mean of the functional map.

Through this, it was possible to confirm the location where the CNN model focused to distinguish the metastatic lymph node by the red region. It was confirmed that the abnormal characteristics of malignant tumors were learned by CNN and used as the basis for the diagnosis of lymph node metastasis of thyroid cancer.

This is because the deep learning-based method for diagnosing lymph node metastasis of thyroid cancer according to the present application generates and provides a heat map to infer which part of the CT image the deep learning model focuses on. This suggests that a diagnostic method with increased specificity and sensitivity can be provided.

[Example 5] Establishment of a thyroid cancer lymph node metastasis diagnosis platform system

A platform system for diagnosing lymph node metastasis of thyroid cancer according to the present application was constructed using Vue. Node.js framework and server communication was developed using RESTful application program interface. The backend was built in Python using the FLASK micro-framework with an embedded development server.

[Experimental Example 1]

6 is a table showing the results of evaluating the diagnostic performance of the CAD model of the method for diagnosing lymph node metastasis of thyroid cancer based on deep learning according to an embodiment of the present application, and FIG. This is a graph showing the receiver operating characteristic curve of the CAD model of the method for diagnosing lymph node metastasis of thyroid cancer.

Specifically, in the validation set for each algorithm, the most accurate model for distinguishing between benign and metastatic lymph nodes based on preoperative CT images was evaluated. The performance of the eight algorithms was measured as the area under the receiver operating characteristic curve (AUROC) and sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV) and accuracy of the validation data set. The parameters were compared using McNemar's test. Receiver operating characteristic curve plots and analysis of results were performed using the pROC package of Rstatistical software (https://cran.r-project.org/web/packages/pROC). AUROC was calculated with a 95% confidence interval.

Through this, it was confirmed that the area under the receiver operating characteristic curve (AUROC) of the eight deep learning algorithms was 0.846 (range 0.784 to 0.884). It was confirmed that the best performing model was Xception with an AUROC of 0.884, and the diagnostic accuracy, sensitivity, specificity, positive predictive value, and negative predictive value of Xception were 82.8%, 80.2%, 83.0%, 83.0%, and 80.2%, respectively. there was.

This is the deep learning-based method for diagnosing lymph node metastasis of thyroid cancer according to the present application by extracting the lesion area of the cervical lymph node from the CT image and learning the crop image for the extracted lesion area. This suggests that a learning-based method for diagnosing lymph node metastasis of thyroid cancer can be provided.

[Experimental Example 2]

8 is a graph showing a Sankey plot of a change in confidence levels before and after the use of the deep learning-based method for diagnosing lymph node metastasis of thyroid cancer according to an embodiment of the present application.

Specifically, Pearson correlation analysis was used to evaluate the change in diagnosis by 6 trainees before and after providing CAD system information. Confidence levels before and after CAD information disclosure were compared using paired Student t tests.

Through this, the confidence level for thyroid cancer lymph node metastasis diagnosis of all trainees with CAD system support was 4.51 to 4.63 (p = 0.002) for Reviewer 1, 4.22 to 4.52 (p <0.001) for Reviewer 2, and Reviewer 3 3.62 to 4.14 for Reviewer 4 (p < 0.001), 4.26 to 4.62 for Reviewer 4 (p < 0.001), 3.00 to 3.76 for Reviewer 5 (p <0.001), and 3.77 to 4.14 for Reviewer 6 (p < 0.001). 0.001) was confirmed to increase. It was confirmed that the overall diagnostic confidence level increased significantly from 3.90 to 4.30 with CAD support for both benign and malignant lymph node analysis (p <0.001).

This suggests that the deep learning-based method for diagnosing lymph node metastasis of thyroid cancer according to the present application can increase the confidence levels of diagnosis, so it can provide clinical usefulness as a training tool to help doctors have confidence in the diagnosis. will do

The foregoing description of the present application is for illustration, and those of ordinary skill in the art to which the present application pertains will understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present application. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a dispersed form, and likewise components described as distributed may be implemented in a combined form.

The scope of the present application is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present application.

Claims (12)

acquiring a CT image of the thyroid gland of the subject;
generating a crop image by extracting the lesion from the CT image;
augmenting the cropped image;
generating learning information by learning the augmented image using a convolutional neural network (CNN); and
diagnosing whether thyroid cancer has metastasized to a lymph node based on the learning information;
A method for diagnosing lymph node metastasis of thyroid cancer based on deep learning, including
The method of claim 1,
The method for diagnosing lymph node metastasis of thyroid cancer based on deep learning, wherein the CT image is obtained from a patient before thyroidectomy is performed.
The method of claim 1,
The generating of the crop image may include: recommending a lesion region of a cervical lymph node from the CT image using a segmentation algorithm or an object detection algorithm; and
performing masking in consideration of the recommended location information of the lesion region, and performing cropping on the masked image;
A method for diagnosing lymph node metastasis of thyroid cancer based on deep learning, comprising a.
The method of claim 1,
The step of augmenting the image includes cropping, rotating, flipping, rescale, zooming, zero filling, rolling, color jittering), Noise Addition, Contrast Adjustment, Brightness Adjustment (Gamma Transformation), Smoothing (Averaging Filter), Scaling, Invert (Left/Left/Up/Down), Shift (Horizontal/Vertical), Partial Masking (Cut out and Random Erasing) ), trimming (Random Crop), transformation, discoloration, background replacement, Gaussian transformation, shear, affine, and combinations thereof, deep learning to perform A method for diagnosing lymph node metastasis in thyroid cancer based on
The method of claim 1,
The method includes generating a heat map; and
a post-processing step of removing an area erroneously determined in the step of diagnosing whether thyroid cancer has metastasized to a lymph node using the heat map;
A method for diagnosing lymph node metastasis of thyroid cancer based on deep learning further comprising a.
an acquisition unit configured to acquire a CT image of the thyroid gland of the subject;
a preprocessor for extracting a lesion from the CT image and generating a crop image;
an augmentation unit for augmenting the cropped image;
a learning unit for generating learning information by learning the augmented image using a convolutional neural network (CNN); and
a diagnosis prediction unit for diagnosing whether thyroid cancer has metastasized to a lymph node based on the learning information;
A deep learning-based lymph node metastasis diagnosis system for thyroid cancer.
7. The method of claim 6,
The deep learning-based lymph node metastasis diagnosis system for thyroid cancer, wherein the acquisition unit acquires the CT image from the patient before thyroidectomy is performed.
7. The method of claim 6,
The preprocessor extracts the lesion region of the cervical lymph node from the CT image using a segmentation algorithm or an object detection algorithm, and performs image cropping on the extracted lesion region to generate the cropped image. Lymph node metastasis diagnostic system of thyroid cancer.
7. The method of claim 6,
The augmentation unit crops, rotates, flips, rescales, zooms, zero filling, rolling, color jittering, noise Addition (Noise Addition), contrast adjustment, brightness adjustment (gamma conversion), smoothing (averaging filter), scaling, invert (left/right/up/down), shift (horizontal/vertical), partial mask (Cut out and Random Erasing), trimming ( Random Crop), transformation, discoloration, background replacement, Gaussian transformation, shear, affine, and combinations thereof to enhance the image by performing a deep A learning-based thyroid cancer lymph node metastasis diagnosis system.
7. The method of claim 6,
a generator for generating a heat map; and
a post-processing unit that receives the heat map and removes a region erroneously determined in the step of diagnosing whether the thyroid cancer has metastasized to a lymph node;
Further comprising a, deep learning-based lymph node metastasis diagnosis system of thyroid cancer.
A computer-readable recording medium in which a program for implementing the method according to any one of claims 1 to 5 is recorded.
A CT image data set obtained by photographing the thyroid gland of an object to be studied is acquired, a lesion is extracted from the CT image data set, a crop image data set is generated, and the crop image data set is augmented to convolution A neural network (convolutional neural network, CNN) is used to learn the augmented image data set to generate learning information, to diagnose whether lymph node metastasis of thyroid cancer is based on the learning information, and to receive the diagnostic information data set. Deep learning-based automatic learning unit for storage processing;
Receive a CT image for diagnosing whether thyroid cancer has metastasized to a lymph node from a user, extract a lesion from the received CT image to generate a cropped image, and learn the cropped image using a convolutional neural network to generate learning information, an API engine for diagnosing lymph node metastasis of thyroid cancer based on deep learning for diagnosing whether thyroid cancer has metastasized to a lymph node based on the learning information and transmitting the diagnosis result to a corresponding user terminal; and
a user terminal for obtaining the diagnosis result value from the API engine and outputting the diagnosis result of lymph node metastasis of thyroid cancer on the screen in real time;
A deep learning-based thyroid cancer lymph node metastasis diagnosis platform system that includes

Images