KR102043672B1 - System and method for lesion interpretation based on deep learning - Google Patents

Abstract

The present invention relates to a system and a method for deciphering a lesion based on deep learning. According to the present invention, the system for deciphering a lesion comprises: a patch generation unit to generate a plurality of image patches for deciphering different from each other on an area to be deciphered in a medical image to be deciphered; a deep learning-based deciphering unit which has a deciphering model pre-learned with lesion images and normal images as learning data, and receives the image patches for deciphering to decipher the status of a legion of the image patches for deciphering by the deciphering model; a patch overlapping unit to give weights to the image patches for deciphering in accordance with deciphering results of the deep learning-based deciphering unit, and match the image patches for deciphering to correspond to each other to overlap the image patches for deciphering; and a deciphering control unit to sum up the weights between pixels corresponding to each other of the image patches for deciphering overlapped by the patch overlapping unit to calculate a weight sum value, and compare the weight sum value of the pixels and a preset reference value to detect a legion in the area to be deciphered. Accordingly, the medical image to be deciphered can be robustly and accurately deciphered in spite of changes of the photographing position, size, and direction of the medical image to be deciphered to detect the status of a legion and the position of the legion.

Description

Deep learning-based lesion reading system and method {SYSTEM AND METHOD FOR LESION INTERPRETATION BASED ON DEEP LEARNING}

The present invention relates to a deep learning-based lesion reading system and method, and more particularly, deep learning that can more accurately read a medical image to be read even when the position, size, or direction of the medical image to be read is changed. To a lesion based reading system and method.

In modern medicine, medical imaging is a very important tool for effective diagnosis and treatment of patients. In addition, advances in imaging technology have enabled the acquisition of more sophisticated medical image data. In exchange for the sophistication, the amount of data is gradually increasing, which makes it difficult to analyze medical image data depending on human vision. Thus, in recent decades, clinical decision support systems and computer-aided reading systems have played an essential role in automated medical image analysis.

Conventional clinical decision support systems or computer-aided reading systems detect and display lesion areas or present reading information to medical personnel or healthcare practitioners and the like (hereinafter, users).

For example, in the method and apparatus for calculating medical diagnosis based on medical image disclosed in Korean Patent Laid-Open No. 10-2017-0017614, a region of interest in which an object to be analyzed is photographed is detected, a coefficient of variation is calculated, and a coefficient of variation It includes the step of creating an image and comparing it with a reference sample, and mentions the effect of diagnosing the degree of disease of the patient using medical images acquired through CT, MRI, and ultrasound imaging apparatus.

In particular, artificial intelligence (AI) technology based on machine learning, such as deep learning, has recently been the basis for a dramatic advance in the reading of patients' diseases using medical images. .

Deep learning refers to a machine learning method based on an artificial neural network that simulates a human neuron and allows a machine to learn. Recently, deep learning technology has been rapidly developed in the field of image recognition and widely used in the field of reading using medical images.

In the deep learning technology of a medical image, machine learning is performed using a plurality of medical images including a disease and the disease as learning data, and a machine learning model (hereinafter referred to as a 'reading model') is generated, and a medical image to be read Input to this reading model will diagnose the lesion.

As described above, the deep learning-based machine learning method collects training data used to generate a reading model, for example, a plurality of lesion images and a plurality of normal images, and a read model generated by learning the collected lesion images and the normal images. Since the newly inputted medical object image is read as a lesion, it is possible to increase the accuracy of the reading result when a variety of learning data are used for learning. In addition, when the medical image to be read is similar to the training data, the accuracy of the reading result may be increased.

However, in the case of a medical image, even if the same patient is photographed according to a photographing environment at the time of photographing the medical image, for example, the posture of the patient, the distance between the photographing equipment and the patient, the photographing angle of the photographing equipment, and the like, This can be taken. In addition, the shape of the lesion also varies from person to person, and the shape may vary depending on the photographing direction.

Therefore, when the original medical image is input directly into the reading model, it may have different characteristics from the training data due to the above factors, and thus there is a problem that the accuracy of the reading result cannot be guaranteed.

Accordingly, the present invention has been made to solve the above problems, deep learning-based lesions that can read the medical image to be read more accurately, even robustly to the change in the recording position, size, direction, etc. of the medical image to be read Its purpose is to provide a reading system and method.

According to an aspect of the present invention, there is provided a deep learning-based lesion reading system, comprising: a patch generator for generating a plurality of reading image patches different from each other in a region to be read in a medical image to be read; A deep learning-based reading unit having a reading model pre-trained from the lesion image and the normal image as training data, and receiving each of the reading image patches and reading the lesions of each of the reading image patches through the reading model; Wow; A patch overlapping unit which weights each of the reading image patches according to a reading result of the deep learning based reading unit, matches and overlaps each of the reading image patches to correspond to each other; The weighted sum is calculated by summing the weights between the corresponding pixels of the read image patches superimposed by the patch overlapping unit, and comparing the weighted sum of the respective pixels with a preset reference value. A deep learning based lesion reading system comprising a read control for detecting lesions in the area.

Wherein the patch generator generates the read image patches such that at least one of the extraction position, the size, and the rotation angle of each read image patch is different; The patch overlapping part may be overlapped by readjusting the extraction position, the size, and the rotation angle of the read image patch.

The patch superimposing unit may assign a positive weight to each pixel of the read image patch when the read image patch is read as a lesion by the deep learning based read unit, and the read image patch is read normally. In this case, a negative weight may be assigned to the pixel of the read image patch.

The medical image to be read includes a 3D image; The patch generator may generate a three-dimensional image patch for reading.

The medical image to be read may include a three-dimensional cerebrovascular image, and the patch generator may generate the three-dimensional readout image patch so as to overlap each other along the cerebrovascular in the cerebrovascular image.

According to another aspect of the present invention, there is provided a deep learning-based lesion reading method comprising the steps of: (a) registering a readout model pre-learned with training data of a lesion image and a normal image; (b) generating a plurality of reading image patches different from each other from a region to be read within the medical image to be read; (c) inputting each read image patch to the read model to read whether a read image patch is lesioned or not; (d) weighting each of the reading image patches according to whether or not each of the reading image patches is lesioned; (e) matching and overlapping the weighted reading image patches to correspond to each other; (f) calculating a weighted sum value by summing the weights of the pixels corresponding to each other of the overlapping readout image patches; and (g) comparing the weighted sum value of each pixel with a predetermined reference value to detect the lesion of the region to be read, which is achieved by the deep learning based lesion reading method.

Wherein in step (b), the readout image patch is generated such that at least one of the extraction position, the size, and the rotation angle of each readout image patch is different; In the step (e), the extraction position, size and rotation angle of the read image patch may be readjusted and overlapped.

In the step (d), when the reading image patch is read as a lesion in step (c), a positive weight is assigned to each pixel of the reading image patch, and in step (c), the reading When the image patch is read normally, a negative weight may be assigned to the pixels of the read image patch.

The medical image to be read includes a 3D image; In step (b), a three-dimensional image patch for reading may be generated.

In addition, the medical image to be read includes a three-dimensional cerebrovascular image, and in the step (b), the three-dimensional image patch for reading may be generated to overlap each other along the cerebrovascular in the cerebrovascular image.

According to the configuration as described above, according to the present invention, a dip that can more accurately read the medical image to be read, even if the location of the medical image to be read, the position, size, direction, etc. more accurately, to detect whether the lesion and even its position Learning-based lesion reading systems and methods are provided.

1 is a view showing the configuration of a deep learning-based lesion reading system according to the present invention,
2 is a view showing an example of a medical image to be read in the deep learning-based lesion reading system according to the present invention,
3 is a view showing examples of the image patch for reading in the deep learning-based lesion reading system according to the present invention,
4 to 6 are views for explaining a process of extracting lesions in the deep learning-based lesion reading system according to the present invention,
7 is a view for explaining a deep learning-based lesion reading method according to the present invention,
FIG. 8 is a diagram for explaining an example of a method of extracting a lesion from a 3D image by a deep learning-based lesion reading system according to the present invention.

As the inventive concept allows for various changes and numerous embodiments, particular embodiments will be illustrated in the drawings and described in detail in the written description.

However, this is not intended to limit the present invention to specific embodiments, it should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention.

The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting of the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, the terms "comprise" or "have" are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, and one or more other features. It is to be understood that the present invention does not exclude the possibility of the presence or the addition of numbers, steps, operations, components, components, or a combination thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in the commonly used dictionaries should be construed as having meanings consistent with the meanings in the context of the related art and shall not be construed in ideal or excessively formal meanings unless expressly defined in this application. Do not.

Hereinafter, with reference to the accompanying drawings will be described embodiments of the present invention;

1 is a view showing the configuration of a deep learning-based lesion reading system according to the present invention. Referring to FIG. 1, a deep learning based lesion reading system according to the present invention includes a patch generator 130, a deep learning based reader 140, a patch overlapping unit 150, and a read controller 110. In addition, the deep learning-based lesion reading system according to the present invention may include an original image storage unit 120.

The medical image to be read is transmitted from the medical imaging apparatus, a hospital having the medical imaging apparatus, or the like, and is stored in the original image storage unit 120. The medical image to be read according to the present invention may include a three-dimensional medical image such as a two-dimensional X-ray image, a CT image or an MRI image capable of reading a lesion on a deep learning basis.

The patch generator 130 generates a plurality of image patches for reading in the region to be read in the medical image to be read. Here, the plurality of reading image patches are generated differently from each other. In the present invention, for example, at least one or more of the extraction position, the size, and the rotation angle of the reading image patch are different in the reading target area.

2 is a view showing an example of a medical image to be read in the deep learning-based lesion reading system according to the present invention. The image shown in FIG. 2 shows a cross section taken as a CT image of a human brain. In FIG. 2, assuming that the red rectangular box is a read target region, the patch generator 130 extracts and generates a plurality of read image patches from the read target region.

FIG. 3 shows an example in which four read image patches are generated from the read target area in FIG. In the example shown in Fig. 3, an example in which a patch for reading an image is generated with different extraction positions in a region to be read is shown. In the example shown in FIG. 3, the white part is the lesion area, and it can be seen that the position is at a different position in each read image patch.

Here, as described above, in the deep learning process using the training data, when the image in the red box of FIG. 2 is used as the training data as the lesion image, a plurality of lesion images are generated in the same manner as the generation of the read image patch. By using it for learning, the reading accuracy of the lesion can be finally improved.

Meanwhile, as described above, when a plurality of read image patches are generated by the patch generator 130, the deep learning-based read unit 140 may identify lesions of each read image patch using a pre-registered read model. Will be read. In the case of the readout image patch extracted in one read target area, the readout of the lesion may vary according to the extraction position, the rotation angle, the size, and the like. For example, the readout image patch shown in FIG. In this case, only a part of the lesion area may be included and thus may not be read as a lesion.

The patch overlapping unit 150 weights each patch image for reading according to the reading result of the deep learning based reading unit 140. In the present invention, when the patch overlap unit 150 reads the image patch for reading by the deep learning-based reading unit 140 as a lesion, a positive weight, for example, a weight of +1 is applied to the image patch for reading. Can be given to each pixel. When the reading patch is normally read by the deep learning reading unit 140, the patch overlapping unit 150 assigns a negative weight, for example, a weight of -1 to each pixel of the reading patch. Grant.

When the weighting is completed for all the read image patches, the patch overlapping unit 150 matches and overlaps the plurality of read image patches to correspond to each other. More specifically with reference to FIG. 4, assuming that (a) of FIG. 4 is one read image patch, and that each cell inside is one pixel, FIG. 4 (b) is one read. A plurality of reading image patches are extracted at different extraction positions in the target area, and then matched to each position to show an example of overlapping.

In this case, as described above, in the case of extracting by extracting a different extraction position in the extraction process of the read image patch, as shown in FIG. If the patch is extracted and the rotation angle is adjusted or the size is changed, the patch is readjusted before the overlap and applied to the overlap.

Meanwhile, the read control unit 110 calculates a weighted sum value for each pixel by summing weights among pixels corresponding to each other in the read image patches superimposed by the patch overlapping unit 150. The read control unit 110 compares the weighted sum value of each pixel with a preset reference value to detect lesions in the read object area.

Fig. 5 shows an example of values in which a weighted sum is added to each pixel in the read object area. That is, the plurality of reading image patches have a weight of -1 or +1 as a weight depending on the reading result of the deep learning based reading unit 140. As the number of reading image patches read as lesions overlaps, The weighted sum is increased.

In addition, when the reading control unit 110 sets the reference value to 35 in the example of FIG. 5 and detects the position of the pixel having a weighted sum value of 35 or more as the lesion position, not only whether the lesion exists in the reading target area but also Detection up to that position is possible. FIG. 6 illustrates an example in which a region marked in red in a red box is detected as a lesion as an example extracted from the region to be read through the above process.

Hereinafter, a deep learning based lesion reading method according to the present invention will be described with reference to FIG. 7.

First, as described above, in a state in which the read model is registered (S70), when a read target medical image is input (S71), a plurality of read image patches are generated from the read target region of the read target medical image (S72). .

Thereafter, whether or not the reading image patch is lesioned is sequentially read through the reading model (S73), and weighted according to whether the reading image patch is lesioning (S74). Then, steps S73 and S74 are repeatedly performed until the reading of all the read image patches is completed (S75).

Then, when reading and weighting of all the read image patches are completed, as described above, all read image patches are superimposed (S76) to calculate the weighted sum value for each pixel, whereby the read target area The presence of the lesion and the location of the lesion in the detection (S77).

In the above-described embodiment, the medical image to be read is described as an example of a two-dimensional image. In addition, the medical image to be read according to the present invention may include a 3D image. 8 is a diagram illustrating an example of a three-dimensional cerebrovascular image of cerebrovascular (BV) with blood vessels extracted from a CT image.

Herein, in the case of a 3D image of the medical image to be read, the patch generator 130 generates a 3D image patch for reading. In FIG. 8, a patch 3D_CP is formed in a cube. In this case, the reading model is also trained using the three-dimensional image as the training data.

At this time, the patch generation unit 130 may generate a three-dimensional image patch for reading so as to overlap each other along the cerebrovascular (BV) in the cerebrovascular image, and sequentially read along the cerebrovascular (BV) Through this, it is possible to detect whether a cerebrovascular disease such as a cerebral aneurysm has occurred along the cerebrovascular (BV) and its location.

Although some embodiments of the invention have been shown and described, it will be apparent to those skilled in the art that the embodiments may be modified without departing from the spirit or spirit of the invention. . It is intended that the scope of the invention be defined by the claims appended hereto and their equivalents.

110: reading control unit 120: original image storage unit
130: patch generation unit 140: deep learning based reading unit
150: patch overlap

Claims (10)

In a deep learning-based lesion reading system,
A patch generator for generating a plurality of read image patches different from each other in a read target region in the read target medical image;
A deep learning-based reading unit having a reading model pre-trained from the lesion image and the normal image as training data, and receiving each of the reading image patches and reading the lesions of each of the reading image patches through the reading model; Wow;
A patch overlapping unit which weights each of the reading image patches according to a reading result of the deep learning based reading unit, matches and overlaps each of the reading image patches to correspond to each other;
The weighted sum is calculated by summing the weights between the corresponding pixels of the readout image patches superimposed by the patch overlapping unit, and comparing the weighted sum of each pixel with a preset reference value. Deep learning-based lesion reading system comprising a read control for detecting lesions in the area.
The method of claim 1,
The patch generation unit generates the read image patches such that at least one of an extraction position, a size, and a rotation angle of each read image patch is different;
The deep overlap-based lesion reading system of claim 1, wherein the patch overlapping unit re-adjusts the extraction position, size, and rotation angle of the read image patch.
The method of claim 1,
The patch superimposing unit assigns a positive weight to each pixel of the read image patch when the read image patch is read as a lesion by the deep learning based read unit, and when the read image patch is read normally. Deep learning based lesion reading system, characterized in that the negative weight is assigned to the pixels of the image patch for reading.
The method of claim 1,
The medical image to be read includes a three-dimensional image;
The patch generation unit deep learning-based lesion reading system, characterized in that for generating a three-dimensional image patch for reading.
The method of claim 4, wherein
The medical image to be read includes a three-dimensional cerebrovascular image,
The patch generation unit deep learning-based lesion reading system, characterized in that for generating the three-dimensional image patch for reading so as to overlap each other along the cerebrovascular in the cerebrovascular image.
In the deep learning-based lesion reading method,
(a) registering a readout model previously trained on the lesion image and the normal image as learning data;
(b) generating a plurality of reading image patches different from each other from a region to be read within the medical image to be read;
(c) inputting each read image patch to the read model to read whether a read image patch is lesioned or not;
(d) weighting each of the reading image patches according to whether or not each of the reading image patches is lesioned;
(e) matching and overlapping the weighted reading image patches to correspond to each other;
(f) calculating a weighted sum value by summing the weights of the pixels corresponding to each other in the overlapping readout image patches;
and (g) comparing the weighted sum of each pixel with a predetermined reference value to detect lesions in the region to be read.
The method of claim 6,
In the step (b), the reading image patch is generated such that at least one of an extraction position, a size, and a rotation angle of each reading image patch is different;
In the step (e), deep learning-based lesion reading method, characterized in that the extraction position, size and rotation angle of the read image patch is readjusted and superimposed.
The method of claim 6,
In step (d)
In the step (c), when the read image patch is read as a lesion, a positive weight is assigned to each pixel of the read image patch,
Deep learning-based lesion reading method characterized in that the negative weight is given to the pixels of the read image patch when the read image patch is read normally in step (c).
The method of claim 6,
The medical image to be read includes a three-dimensional image;
In the step (b), deep learning-based lesion reading method characterized in that for generating a three-dimensional image patch for reading.
The method of claim 9,
The medical image to be read includes a three-dimensional cerebrovascular image,
In the step (b), deep learning-based lesion reading method characterized in that the three-dimensional image patch for reading is generated so as to overlap each other along the cerebrovascular in the cerebrovascular image.

Images