KR102226897B1 - Method for medical diagnosis using neural network trained by multi-phase biomedical image, medical diagnosis system performing the same - Google Patents

Abstract

Disclosed are a disease diagnosis method using a neural network learned using a multi-phase biometric image and a disease diagnosis system performing the same. According to an aspect of the present invention, a diagnostic system using a neural network generates a diagnostic neural network for predicting a diagnosis result for a predetermined disease using a biometric image, and the diagnostic system using the neural network comprises: Obtaining a training biometric image of-each of the plurality of training biometric images is labeled with a diagnosis result for the disease corresponding thereto, and a diagnosis system using the neural network, wherein the plurality of training biometric images are And training the diagnostic neural network using an image, wherein the training of the diagnostic neural network includes: (a) different characteristics of the training biometric images for each of the plurality of training biometric images Generating K noise-inserted images corresponding to the training biometric images (where K is an integer greater than or equal to 2) by inserting noise having a (b) K noise-inserted images corresponding to the training biometric images There is provided a method comprising the step of training the diagnostic neural network by inputting the diagnostic neural network.

Description

[Method for medical diagnosis using neural network trained by multi-phase biomedical image, medical diagnosis system performing the same}

The present invention relates to a disease diagnosis system and method using a neural network. More specifically, it relates to a disease diagnosis method using a neural network learned using a multi-phase biometric image, and a disease diagnosis system performing the same.

One of the major tasks performed in the pathology or pathology department is to read a patient's biological slide (for example, a tissue image slide through a biopsy, etc.) to perform a diagnosis to determine a condition or symptom for a specific disease. This diagnosis is a method that is dependent on the experience and knowledge of a skilled medical practitioner for a long time.

In recent years, due to the development of machine learning, attempts to automate tasks such as recognizing or classifying images by computer systems have been actively made. In particular, attempts have been made to automate the diagnosis performed by skilled medical personnel using a neural network (eg, a deep learning method using a convolution neural network, CNN), which is a kind of machine learning.

In particular, diagnosis through deep learning using neural networks (e.g., CNN, deep learning network) does not simply automate the experience and knowledge of experienced medical personnel, but finds characteristic elements through self-learning to derive desired answers. In that sense, there are cases in which the characteristics of disease factors that experienced medical personnel were not aware of are found in images.

In general, diagnosis of a disease through a neural network using a biometric image, which is a scanned image of a biometric slide, uses a biometric image or a patch (also referred to as a tile), which is a fragment of the biometric image. That is, a medical practitioner who is skilled in the biometric image or patch annotates the state of a specific disease (eg, whether cancer has been expressed), and learns a neural network using a plurality of these annotated tiles as training data. do. In this case, a convolutional neural network may be used as the neural network.

Meanwhile, most current diagnosis methods through neural networks use already scanned data for learning, and predict a diagnosis result based on the scanned data. That is, the existing diagnostic method through neural networks works relatively well for fixed data. However, when there is noise, it shows a weak appearance. For example, in the case of image data photographed in real time through a microscope, noise generated by a camera of the microscope is heavily embedded in the image, but the noise of the camera itself causes a problem in that the diagnosis result may change continuously.

Meanwhile, since the diagnosis rate by the neural network is generally slower than the frame rate of the camera, there is a problem that the screen by the camera is cut off when the part to be diagnosed through the microscope continues to move.

Korean Patent Laid-Open Publication No. 10-2016-0034814 "Client device with neural network and system including the same"

A technical problem to be achieved by the present invention is to provide a diagnosis system and method using a neural network capable of stably performing diagnosis regardless of noise caused by a camera.

In addition, the present invention provides a diagnosis system and method that enables smooth screen movement by detecting movement of a camera and performing diagnosis only when a specific part is fixed and viewed without performing diagnosis when moving or when the screen is blank.

Furthermore, it provides a neural network structure capable of distinguishing not only whether a disease has occurred through a specific biometric image, but also an area where a disease has occurred in the corresponding biometric image. In particular, by adding a sub-architecture for segmentation to the classical neural network architecture, which can determine whether a disease exists in the patch, a neural network structure that can efficiently perform segmentation to distinguish the diseased area is established. To provide.

According to an aspect of the present invention, a diagnostic system using a neural network generates a diagnostic neural network for predicting a diagnosis result for a predetermined disease using a biometric image, and the diagnostic system using the neural network comprises: Obtaining a training biometric image of-each of the plurality of training biometric images is labeled with a diagnosis result for the disease corresponding thereto, and a diagnosis system using the neural network, wherein the plurality of training biometric images are And training the diagnostic neural network using an image, wherein the training of the diagnostic neural network includes: (a) different characteristics of the training biometric images for each of the plurality of training biometric images Generating K noise-inserted images corresponding to the training biometric images (where K is an integer greater than or equal to 2) by inserting noise having a (b) K noise-inserted images corresponding to the training biometric images There is provided a method comprising the step of training the diagnostic neural network by inputting the diagnostic neural network.

In one embodiment, the step (b) is the step of generating one training data corresponding to the training biometric image by concatenating all K noise-inserted images corresponding to the training biometric image. Training data corresponding to the training biometric image is labeled as a diagnosis result for the training biometric image and training the diagnostic neural network by inputting training data corresponding to the training biometric image into the diagnosis neural network. Can include.

In one embodiment, the method further includes a diagnosis step, wherein the diagnosis step comprises: obtaining K diagnosis subject biometric images continuously photographed by an image sensor, and the K diagnosis subject biometric images as the diagnosis neural It may include inputting into a network and predicting a diagnosis result for the disease.

In one embodiment, the diagnostic step further comprises determining whether the image sensor is moving based on the K diagnostic target biometric images, wherein the K diagnostic target biometric images are input to the diagnostic neural network. The predicting of the diagnosis result for the disease may include, when it is determined that the image sensor is not moving, inputting the K diagnosis target biometric images into the diagnosis neural network to predict the diagnosis result for the disease. Can include.

In one embodiment, the method further includes a diagnosis step, wherein the diagnosis step comprises: acquiring K diagnosis subject biometric images continuously photographed by an image sensor, and connecting all the K diagnosis subject biometric images Generating one diagnosis target data and inputting the diagnosis target data into the diagnosis neural network may include predicting a diagnosis result for the disease.

In one embodiment, the diagnosis neural network is a segmentation neural network that receives the diagnosis target data as an input layer and specifies a region in which a disease exists in the diagnosis target biometric image, and the segmentation neural network includes the diagnosis target data Generated in each of two or more feature map extraction layers among the classical neural network that receives as an input layer and outputs a classification result of whether the disease is present in the biological image, and the hidden layers included in the classical neural network. It may include a segmentation architecture for specifying a region in which a disease exists in the biometric image by receiving a feature map.

In one embodiment, the segmentation architecture is a convolution sub-architecture including a convolution node corresponding to each of the two or more feature extraction layers-each of the convolution nodes is in a feature map input from a corresponding feature extraction layer. A segmentation sub-architecture specifying a region in which a disease exists in the patch based on a convolution result of the convolution or two or more different convolutions- and a convolution result generated by the convolution sub-architecture may be included.

In one embodiment, the disease may be characterized in that prostate cancer.

According to another aspect of the present invention, there is provided a computer program installed in a data processing apparatus and recorded in a computer-readable medium for performing the above-described method.

According to another aspect of the present invention, as a diagnostic system using a neural network, including a processor and a memory for storing a computer program, the computer program, when executed by the processor, the diagnostic system using the neural network A diagnostic system using a neural network for performing the above-described method is provided.

According to another aspect of the present invention, a storage module that stores a diagnostic neural network for predicting a diagnosis result for a predetermined disease using a biometric image, and a diagnosis system using the neural network includes a plurality of training biometric images- In each of the plurality of training biometric images, an acquisition module for acquiring that a diagnosis result for the disease corresponding thereto is labeled and a diagnosis system using the neural network, by using the plurality of training biometric images, And a learning module for training the diagnostic neural network, wherein the learning module includes, for each of the plurality of training biometric images, (a) inserting noises having different characteristics into the training biometric images, Generating K noise-inserted images corresponding to the biometric images (where K is an integer greater than or equal to 2) and (b) inputting K noise-inserted images corresponding to the training biometric images into the diagnostic neural network, A diagnostic system using a neural network that performs a step of training a diagnostic neural network is provided.

In one embodiment, the step (b) is the step of generating one training data corresponding to the training biometric image by concatenating all K noise-inserted images corresponding to the training biometric image. Training data corresponding to the training biometric image is labeled as a diagnosis result for the training biometric image, and training the diagnosis neural network by inputting training data corresponding to the training biometric image into the diagnosis neural network. It may include.

In one embodiment, the diagnostic system using the neural network acquires K diagnostic target biometric images continuously photographed by an image sensor, and inputs the K diagnostic target biometric images to the diagnostic neural network to detect the disease. It may further include a diagnostic module for predicting the diagnosis result.

In one embodiment, the diagnostic module determines whether the image sensor is moving based on the K diagnostic target biometric images, and when it is determined that the image sensor is not moving, the K biometric images to be diagnosed are By inputting into the diagnostic neural network, a diagnosis result for the disease may be predicted.

In one embodiment, the diagnostic system using the neural network acquires K diagnostic target biometric images continuously photographed by an image sensor, and generates one diagnosis target data by connecting all the K diagnostic target biometric images. And a diagnosis module for predicting a diagnosis result for the disease by inputting the diagnosis target data into the diagnosis neural network.

According to the technical idea of the present invention, it is possible to provide a diagnosis system and method using a neural network capable of stably performing diagnosis irrespective of noise caused by a camera.

In addition, it is possible to provide a diagnosis system and method that enables smooth screen movement by detecting movement of a camera and performing diagnosis only when a specific part is fixed and viewed without performing diagnosis when moving or when the screen is blank.

In addition, it is possible to provide an efficient neural network structure capable of performing a segmentation capable of distinguishing not only the occurrence of a disease in each biological image, but also a region in which a disease has occurred among the corresponding biological images.

A brief description of each drawing is provided in order to more fully understand the drawings cited in the detailed description of the present invention.
1 is a diagram illustrating an operating environment of a method of learning a neural network using a multi-phase biometric image and diagnosing a disease using the multi-phase biometric image according to the technical idea of the present invention.
2 is a diagram showing a schematic configuration of a disease diagnosis system according to an embodiment of the present invention.
3 is a flowchart illustrating a process of learning a diagnostic neural network by using a plurality of training biometric images by a diagnostic system according to an embodiment of the present invention.
4 is a diagram illustrating an example in which two noise-added images are combined and used for learning.
5 is a flowchart illustrating a process of diagnosing a biometric image to be diagnosed by a diagnostic system according to an embodiment of the present invention.
6 is a diagram for explaining a process of performing a diagnosis using two consecutively photographed biometric images to be diagnosed.
7 is a diagram illustrating an overall structure of a diagnostic neural network for segmentation according to an embodiment of the present invention.
8 is a diagram for explaining the overall structure of a segmentation architecture according to an embodiment of the present invention.
9 is a diagram illustrating a specific example of a diagnostic neural network for segmentation according to an embodiment of the present invention.

Since the present invention can apply various transformations and have various embodiments, specific embodiments are illustrated in the drawings and will be described in detail in the detailed description. However, this is not intended to limit the present invention to a specific embodiment, it should be understood to include all conversions, equivalents, and substitutes included in the spirit and scope of the present invention. In describing the present invention, when it is determined that a detailed description of a related known technology may obscure the subject matter of the present invention, a detailed description thereof will be omitted.

Terms such as first and second may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another component.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise.

In the present specification, terms such as "comprise" or "have" are intended to designate the presence of features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and one or more other It should be understood that the presence or addition of features, numbers, steps, actions, components, parts, or combinations thereof, does not preclude the possibility of preliminary exclusion.

In addition, in the present specification, when one component'transmits' data to another component, the component may directly transmit the data to the other component, or through at least one other component. This means that the data may be transmitted to the other component. Conversely, when one component'directly transmits' data to another component, it means that the data is transmitted from the component to the other component without passing through the other component.

Hereinafter, the present invention will be described in detail based on embodiments of the present invention with reference to the accompanying drawings. The same reference numerals shown in each drawing indicate the same members.

Since the present invention can apply various transformations and have various embodiments, specific embodiments are illustrated in the drawings and will be described in detail in the detailed description. However, this is not intended to limit the present invention to a specific embodiment, it should be understood to include all conversions, equivalents, and substitutes included in the spirit and scope of the present invention. In describing the present invention, when it is determined that a detailed description of a related known technology may obscure the subject matter of the present invention, a detailed description thereof will be omitted.

Terms such as first and second may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another component.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise.

In the present specification, terms such as "comprise" or "have" are intended to designate the presence of features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and one or more other It should be understood that the presence or addition of features, numbers, steps, actions, components, parts, or combinations thereof, does not preclude the possibility of preliminary exclusion.

In addition, in the present specification, when one component'transmits' data to another component, the component may directly transmit the data to the other component, or through at least one other component. This means that the data may be transmitted to the other component. Conversely, when one component'directly transmits' data to another component, it means that the data is transmitted from the component to the other component without passing through the other component.

Hereinafter, the present invention will be described in detail based on embodiments of the present invention with reference to the accompanying drawings. The same reference numerals shown in each drawing indicate the same members.

1 is a diagram illustrating an operating environment of a method of learning a neural network using a multi-phase biometric image and diagnosing a disease by using the multi-phase biometric image according to the technical idea of the present invention. Referring to FIG. 1, a method according to the technical idea of the present invention may be performed by the diagnostic system 100.

In the spirit of the present invention, the diagnostic system 100 may be a computing system. Alternatively, it may be installed on a predetermined server 10 to implement the technical idea of the present invention. The server 10 refers to a data processing device having computing power to implement the technical idea of the present invention, and generally, a personal computer, a portable terminal as well as a data processing device accessible to a client (terminal; 20) through a network An average expert in the technical field of the present invention can easily infer that any device capable of performing a specific service, such as, can be defined as a server.

The server 10 may include a processor and a storage device. The processor may mean an operation device capable of driving a program for implementing the technical idea of the present invention. The storage device may mean a data storage means capable of storing programs and various types of data necessary to implement the technical idea of the present invention, and may be implemented as a plurality of storage means according to implementation examples. In addition, the storage device may mean not only a main memory device included in the server 10 but also a temporary storage device or a memory that may be included in the processor.

When the diagnostic system 100 is included in a predetermined server 10 and implemented, the diagnostic system 100 may communicate with at least one client (eg, 20) accessible to the server 10. have. In this case, the client (eg, 20) may transmit a biometric image to the diagnosis system 100, and the diagnosis system 100 may perform diagnosis according to the technical idea of the present invention on the transmitted biometric image. . In addition, the diagnosis result may be transmitted to the client (eg, 20).

The diagnosis system 100 may learn a neural network for diagnosing a biometric image using a plurality of training biometric images, and after learning the neural network, the client 20 provides it using the learned neural network. It is possible to predict the diagnosis result for the image to be diagnosed.

The client 20 may include an image sensor 21. The image sensor 21 may be a sensor for photographing or scanning a living body image. The image sensor 21 may continuously capture a living body image. For example, the image sensor 21 may capture a biometric image at 60 frames per second.

In one embodiment, the image sensor 21 may be a camera.

In another embodiment, the image sensor 21 may be a microscope. In this case, the image sensor 21 may enlarge and take a picture of a specific part of the biological slide, and a diagnostician who wants to diagnose a biological image by using the client 20 moves the photographing area on the slide and moves it to a specific area of the biological slide. Diagnosis can be performed.

Although the disease diagnosis system 100 is illustrated as being implemented as any one physical device in FIG. 1, a plurality of physical devices are organically combined as needed to implement the diagnosis system 100 according to the technical idea of the present invention. It will be easily inferred by an average expert in the technical field of the present invention.

In addition, according to an embodiment, the disease diagnosis system 100 may be installed and operated directly on the client 20 instead of the server 10.

2 is a diagram showing a schematic configuration of a disease diagnosis system 100 according to an embodiment of the present invention.

Referring to FIG. 2, the system 100 may include a storage module 110, a generation module 130, an acquisition module 140, a learning module 150, and a diagnosis module 160. Depending on the embodiment of the present invention, some of the above-described components may not necessarily correspond to the components essential to the implementation of the present invention, and according to the embodiment, the diagnostic system 100 may be It goes without saying that more components may be included. For example, the system 100 includes other components of the system 100 (for example, the storage module 110, the generation module 130, the acquisition module 140, the learning module 150), and the diagnosis module 160 ), etc.) may further include a control module (not shown) for controlling functions and/or resources. In addition, according to an embodiment, the system 100 may further include a database (DB) 300 for storing various types of information and/or data necessary to implement the technical idea of the present invention.

The system 100 may refer to a logical configuration including hardware resources and/or software necessary to implement the technical idea of the present invention, and necessarily refers to one physical component or one device. Does not mean that. That is, the system 100 may mean a logical combination of hardware and/or software provided to implement the technical idea of the present invention. If necessary, the system 100 is installed in a device separated from each other to perform each function. It may be implemented as a set of logical configurations for implementing the technical idea of the present invention. In addition, the system 100 may refer to a set of components implemented separately for each function or role for implementing the technical idea of the present invention. For example, each of the storage module 110, the generation module 130, the acquisition module 140, the learning module 150, and the diagnosis module 160 may be located in different physical devices or in the same physical device. have. In addition, depending on the implementation example, the combination of software and/or hardware constituting each of the storage module 110, the generation module 130, the acquisition module 140, the learning module 150, and the diagnostic module 160 Components located in different physical devices and components located in different physical devices may be organically combined with each other to implement the respective modules.

In addition, in the present specification, a module may mean a functional and structural combination of hardware for performing the technical idea of the present invention and software for driving the hardware. For example, the module may mean a predetermined code and a logical unit of a hardware resource for executing the predetermined code, and does not necessarily mean a physically connected code or a single type of hardware. Can be easily inferred by an average expert in the technical field of the present invention.

The DB 200 may store a plurality of training biometric images. The training biometric image may be various biometric images including tissue images and biopsy images.

In addition, the training biological image may be a biological slide or a patch in which the biological slide is divided.

A disease diagnosis result may be labeled on each of the plurality of training biological images. The labeled diagnosis result may be the same type as the diagnosis result diagnosed by the disease diagnosis system 100. The diagnosis result labeled on each training biometric image may be a result of determining a disease previously made by a medical professional based on the corresponding biometric image.

For example, when the disease diagnosis system 100 is a system that predicts a classification result for the presence or absence of a disease, the diagnosis result labeled on each of the plurality of training biometric images may also indicate the presence or absence of the disease. Or, depending on the embodiment, the diagnosis result for the biometric image is, for example, the diagnosis result for the biometric image, not only whether a specific disease has occurred (negative/positive), but also the degree of progression (or probability corresponding to the degree of progression) of a specific disease. Can include. For example, when the technical idea of the present invention is used for diagnosis of prostate cancer, a Gleason Pattern or Gleason Score, an index indicating the progression of prostate cancer, may be included in the diagnosis result. For example, the Gleason score has a value of 2 to 10, and usually a value of 6 to 10 can be considered as cancer, and the higher the number, the more severe the degree of prostate cancer is. Gleason patterns can be classified into classes 1 to 5.

If the disease diagnosis system 100 is a system that predicts a disease incidence area or a diagnosis result in units of pixels (i.e., a system that performs segmentation), the diagnosis result labeled on each of the plurality of training biological images is also It may relate to the onset area or the diagnosis result in units of pixels.

On the other hand, the disease may be prostate cancer, and the description will focus on prostate cancer hereinafter, but it will be easily understood by those of ordinary skill in the field to which the present invention belongs that the technical idea of the present invention is not limited to prostate cancer.

The storage module 110 may store a diagnostic neural network 120, and the diagnostic neural network 120 may be generated by the generation module 130. The diagnostic neural network 120 may be a neural network for predicting a diagnosis result for a predetermined disease using a biometric image.

The storage module 110 may mean a data storage means capable of storing the diagnostic neural network 120, and may be implemented as a plurality of storage means according to implementation examples. In addition, the storage module 110 may mean not only a main memory device included in the server 10 but also a temporary storage device or a memory.

The diagnostic neural network 120 may be a neural network for diagnosing a biometric image in units of patches. In this case, the patch may be a partial image in which a predetermined slide is divided into a predetermined size. That is, one slide may be divided into a plurality of patches, and the diagnostic neural network 120 may receive a patch and perform a patch level diagnosis. Before performing diagnosis, the diagnosis neural network 120 may be learned in advance by the learning module 150.

On the other hand, the information output from the diagnostic neural network 120 is information that enables determination of whether a specific disease (eg, a specific type of cancer) has been expressed in the tissue corresponding to the patch or a region where a specific disease has occurred. Can be For example, the information output by the diagnostic neural network 160 may be information indicating a probability of whether a specific disease (eg, a specific type of cancer) has been expressed in a tissue corresponding to the patch. In this case, when the probability value output from the diagnostic neural network 120 is greater than or equal to a specific reference value (threshold value), it may be determined that a disease (eg, prostate cancer) is expressed in the patch. Of course, according to an exemplary embodiment, the diagnostic neural network 120 may output not only whether or not a specific disease has occurred, but also information indicating the degree of progression of a specific disease (or a probability corresponding to the degree of progression). For example, when the technical idea of the present invention is used for diagnosis of prostate cancer, the Gleason Pattern or Gleason Score, which is an index indicating the progression of prostate cancer, is output by the diagnostic neural network 120. Can be included in the information. On the other hand, the threshold value used by the diagnostic neural network 120 may be variously set, and a specific patch may be determined as a disease patch, that is, a disease patch, or a normal patch according to the threshold value. Of course.

Alternatively, the diagnostic neural network 120 may be a neural network for receiving a patch and predicting a classification result of the disease for each pixel constituting the patch. The diagnostic neural network 120 that outputs the classification result for each pixel may perform segmentation to classify an onset region of a disease. In this case, the diagnostic neural network 120 may output the occurrence of a disease or the probability of a disease for each pixel constituting the patch, and a pixel determined to have an onset or a pixel having an onset probability exceeding a predetermined threshold value is The constituting area may be determined as an onset area.

Depending on the embodiment, the diagnostic neural network 120 may predict and output whether a disease occurs (or a disease probability) and/or a disease progression degree (or a probability corresponding to the progression degree) for each pixel. have.

Meanwhile, before segmentation (ie, prediction of a diagnosis result for each pixel) is performed, the diagnosis neural network 120 may be learned in advance by the learning module 150.

In the present specification, a neural network may mean a set of information representing a series of design items defining a neural network. In an embodiment, the diagnostic neural network 120 may be a convolutional neural network.

As is well known, the convolutional newelul network may include an input layer, a plurality of hidden layers, and an output layer. Each of the plurality of hidden layers may include a convolution layer and a pooling layer (or sub-sampling layer).

The convolutional newelite network may be defined by a function, a filter, a stride, a weight factor, or the like for defining each of these layers. In addition, the output layer may be defined as a fully connected feedforward layer.

The design details of each layer constituting the convolutional newelul network are widely known. For example, known functions may be used for each of the number of layers to be included in a plurality of layers, a convolution function, a pooling function, and an activation function for defining the plurality of layers, and to implement the technical idea of the present invention. Separately defined functions may be used.

An example of a convolution function is a discrete convolution sum. As an example of the pooling function, max pooling, average pooling, or the like may be used. An example of the activation function may be a sigmoid, a tangent hyperbolic (tanh), a rectified linear unit (ReLU), or the like.

When the design items of such a convolutional neural network are defined, the convolutional neural network in which the design items are defined may be stored in a storage device. In addition, when the convolutional neural network is learned, a weight factor corresponding to each layer may be specified.

That is, learning of the convolutional newelul network may refer to a process in which weight factors of each layer are determined. In addition, when the convolutional neural network is trained, the learned convolutional neural network may receive input data into an input layer and output output data through a predefined output layer.

The neural network according to an embodiment of the present invention may be defined by selecting any one or a plurality of widely known design items as described above, or an independent design item may be defined for the neural network.

Referring back to FIG. 2, the acquisition module 140 may acquire training data for machine learning for the diagnostic neural network 120. That is, the acquisition module 140 may acquire a plurality of training biometric images, and each of the plurality of training biometric images may be labeled with a corresponding diagnosis result. For example, the acquisition module 140 may acquire a training biometric image from the DB 200.

In addition, although the present specification has mainly described an example in which the technical idea of the present invention is applied to prostate cancer, not only a specific tissue but also the state of the surrounding tissues is taken into consideration, and the diagnosis of the specific tissue needs to be performed. Also, when the technical idea of the present invention is applied, an average expert in the technical field of the present invention can easily infer that accurate diagnosis may be possible.

Meanwhile, the learning module 150 may train the diagnostic neural network 120 using the plurality of training biometric images.

In more detail, the learning module 150 inserts noises having different characteristics into the training biometric images for each of the plurality of training biometric images, so that K pieces corresponding to the training biometric images (here, K is an integer greater than or equal to 2) noise-inserted images are generated, and K noise-inserted images corresponding to the training biometric images are input to the diagnostic neural network to train the diagnostic neural network.

3 is a flowchart illustrating a process in which the diagnostic system 100 learns the diagnostic neural network 120 using a plurality of training biometric images according to an embodiment of the present invention.

Referring to FIG. 3, the acquisition module 140 of the diagnosis system 100 may acquire N (N is an integer equal to or greater than 2) training biometric images each labeled with a diagnosis result for a disease (S100). .

Thereafter, the diagnosis module 150 of the diagnosis system 100 may perform the processes S120 to S150 for each of the N training biometric images (S110).

For a J-th training the learning process by the biometric image S _j for, K dog by inserting a noise having different characteristics in the living body image S _j for training at the S120 step corresponding to a biometric image S _j for the train (here, K Is an integer greater than or equal to 2) of noise-inserted images T _{(J, 1)} to T _{(J, K)} (S120).

For example, the noise insertion images T _{(J, 1)} to T _{(J, K)} may be images in which Gaussian noise having different averages and/or variances from each other is added to the _{training biometric image S j.} . When Gaussian noise is added, in addition to the average and/or variance, various parameters may be modified, and there may be embodiments in which various types of noise are added to the biometric image in addition to Gaussian noise.

In step S130, the training biometric image of the K noise insert image corresponding to the S _j for the T _{(J, 1)} to T _{(J, K)} to connect (concatenation) all of the one corresponding to the living body image S _j for the training We can generate the training data U _{j of.} In this case, the training data U _j may be labeled with the label of the training biometric image S _{j (S130).}

In step S140, _{training data U j} corresponding to the training _{biometric image S j} may be input to the diagnostic neural network 120 to train the diagnostic neural network 120.

4 is a diagram illustrating an example in which two noise-added images generated by adding two different noises to one training body image are combined and used for learning (that is, an example in which K is 2).

As shown in FIG. 4, noise 1 and noise 2 are respectively added to the original image to be learned to generate two different noise-inserted images, and after combining them, input to the diagnostic neural network 120 to perform learning. You can do it.

Referring back to FIG. 2, the diagnostic module 160 acquires K diagnostic-target biometric images that are continuously photographed, and the K diagnostic-target biometric images are learned by the learning module 150. 120), the diagnosis result for the disease can be predicted. In an embodiment, the K number of biometric images to be diagnosed may be biometric images captured by the image sensor 21.

5 is a flowchart illustrating a process of diagnosing a biometric image to be diagnosed by the diagnosis system 100 according to an embodiment of the present invention.

Referring to FIG. 5, the diagnosis module 160 may _{acquire K consecutively photographed biometric images O 1} to O _K to be diagnosed (S200).

The diagnosis module 160 may determine whether a predetermined diagnosis condition is satisfied based on the _{K diagnosis target biometric images O 1} to O K (S210).

In one embodiment, the diagnosis module 160 may determine whether the image sensor 21 that is photographing the diagnosis target image is moving based on the K diagnosis target biometric images, and the image sensor 21 If it is determined that) is not moving, it can be determined that the diagnosis condition is satisfied. That is, the diagnosis module 160 may determine that the diagnosis condition is satisfied when the image sensor 21 is not in motion but continues to photograph a specific part of the biometric image in a stationary state.

For example, the diagnostic module 160 may calculate a numerical value or numerical values representing a relationship between two consecutive images, and determine that the calculated numerical value or numerical values are within a predetermined limit range, it is not in motion.

Alternatively, according to an embodiment, the diagnosis module 160 may determine that the diagnosis condition is satisfied when it is determined that the K diagnosis target biometric images are photographing a living body tissue while the image sensor 21 is not moving. have. In addition to this, various diagnostic conditions may be set in advance.

If the diagnosis condition is not satisfied, the diagnosis of the disease is not performed and K biometric images photographed subsequently may be acquired (see S210 and S200).

If the diagnosis condition is satisfied, the diagnosis module 160 may _{concatenate K biometric images O 1} to O K to be diagnosed to generate the diagnosis target data P (S220), and the diagnosis target data P By inputting into the diagnostic neural network 120, a diagnosis result for a disease may be predicted.

6 is a diagram for explaining a process of performing a diagnosis using two consecutively photographed biometric images to be diagnosed.

As shown in FIG. 6, two diagnostic target images (ie, image 1 and image 2) acquired from an input device (eg, image sensor 21) are combined to determine whether or not the diagnosis condition is satisfied. It can be entered into the algorithm for doing. If the diagnosis condition is satisfied, the combined data (ie, diagnosis target data) may be input to the learned diagnosis neural network 120, and the diagnosis neural network 120 may predict a diagnosis result.

As described above, according to an embodiment, the diagnostic neural network 12 may be a segmentation neural network that performs segmentation for specifying an onset region of a disease among a living body image (for example, a patch). Networks can have very unique structures.

The diagnostic neural network 120 for segmentation according to an embodiment of the present invention uses a neural network (a'classification neural network' to be described later) that performs classification to determine whether a disease exists in a living body image. As a base, it can be implemented in a form in which a sub-architecture for segmentation is combined. The structure of such a diagnostic neural network 120 is illustrated in FIG. 7.

7 is a diagram illustrating an overall structure of a diagnostic neural network 120 for segmentation according to an embodiment of the present invention.

As shown in FIG. 7, the diagnostic neural network 120 according to an embodiment of the present invention may include a classification neural network 200 and a segmentation architecture 260.

The classification neural network 200 receives the diagnosis target data as an input layer, and a classification result regarding whether a disease exists in the diagnosis target image corresponding to the diagnosis target data (for example, as shown in FIG. 7 ). The same score) can be output. This is called classification. During the classification process, the classification neural network 200 may generate a feature for an input (ie, a patch) as an intermediate product in some hidden layers included therein. In particular, when a matrix of two or more dimensions, such as an image, is received as an input, the feature created is in the form of a two-dimensional matrix, so the term feature map is sometimes used. Meanwhile, hereinafter, a layer for generating a feature map among hidden layers included in the classification neural network 200 will be referred to as a feature map extraction layer.

On the other hand, the segmentation architecture 260 is a feature map generated from each of two or more feature map extraction layers among hidden layers included in the classification neural network 200 (e.g., f1, f2 shown in FIG. 4 ). , f3) may be input to specify and output a region in which a disease exists among the patches.

7 shows an example of generating three feature maps (f1, f2, f3) in the process of the classification neural network 200 performing the classification. However, depending on the embodiment, more or less number of feature maps are generated. Of course, feature maps can be created.

Meanwhile, according to an embodiment of the present invention, the classification neural network 200 for performing classification uses a known densenet, and at this time, as disclosed in the previous application, not only a specific patch to be diagnosed but also a peripheral patch It can be designed to take even into account. In addition, various neural networks may be used, and in any case, the classification neural network 200 may be defined to receive a specific patch as an input and output a feature value corresponding to a disease occurrence probability of the specific patch.

Meanwhile, FIG. 8 is a diagram for explaining the overall structure of the segmentation architecture 260.

Referring to FIG. 8, the segmentation architecture 260 may include a convolution sub-architecture 261 and a segmentation sub-architecture 263, and may further include a cropping sub-architecture 264 according to embodiments. .

As described above, feature maps (f1, f2, f3) may be generated by each feature extraction layer in the classification process performed in the classification neural network 200, each of which is the convolution It may be input to the convolution nodes 262-1 to 262-3 included in the sub-architecture 261.

Each of the convolution nodes 262-1 to 262-3 corresponds to each of the two or more feature extraction layers included in the manifestation neural network 200, and a feature map (f1) input from the corresponding feature extraction layer. To f3) convolution or two or more different convolutions may be performed. Meanwhile, according to an embodiment, each of the convolution nodes 262-1 to 263-3 may perform convolution after upscaling (eg, upsampling) or downscaling (eg, downsampling). .

Each of the convolution nodes 262-1 to 262-3 may generate one or two or more results by performing one or two or more convolutions. According to an embodiment, the convolution performed by the convolution nodes 262-1 to 262-3 may be dialed convolution (also referred to as atrous convolution). Unlike conventional convolution, dialed convolution is a method of performing convolution at predetermined intervals rather than extracting features from adjacent pixels. For example, any one of the convolution nodes (e.g., 511-2) is 1x1 dialed convolution, 3x3 dialed convolution of rate 6, 3x3 dialed convolution of rate 12, rate 18 It is possible to generate 4 convolution products (features) by performing 3x3 dialed convolution of.

Meanwhile, the segmentation sub-architecture 263 may specify a region in which a disease exists among the patches based on a convolution result generated by the convolution sub-architecture 261.

The segmentation sub-architecture 263 may perform a predetermined operation on a convolution result generated by the convolution sub-architecture 261. The operation performed by the segmentation sub-architecture 263 may be defined as a combination of concatenation and/or convolution. Combinations and convolutions can be combined in various ways, depending on the embodiment.

Meanwhile, according to an embodiment, the cropping sub-architecture 264 may generate a final result for segmentation by performing center-cropping on the result output from the segmentation sub-architecture 263. This is because in the results of the convolutional sub-architecture 261 and the segmentation sub-architecture 263, the center portion tends to reflect the result more accurately.

9 is a diagram illustrating a specific example of a segmentation diagnostic neural network 120 according to an embodiment of the present invention.

Referring to FIG. 9, the classification neural network 200 included in the diagnostic neural network may perform classification. For classification, the classification neural network 200 may receive a patch as an input layer, and a 1/4-sized raw feature map through convolution and pooling operations in the first feature extraction layer (L1). (f1) can be created. Then, through a first density block and a first transition operation (denseblock1, transition1) and a second density block and a second transition operation (denseblock2, transition2), a middle feature map of 1/16 size in the second feature extraction layer L2 ( f2) can be created. Then, the third feature extraction layer (L3) is sequentially subjected to a third density block and a third transition operation (denseblock1, transition1), a fourth density block (denseblock4), and a 1x1 convolution, and an end feature map (f3) having a size of 1/16 in the third feature extraction layer (L3). ) Can be created. After that, the score for disease status can be output through the average pooling.

Meanwhile, in the segmentation architecture 260, segmentation may be performed using each feature generated by the classification neural network 200.

In more detail, each convolution node 510 included in the convolution sub-architecture 261 performs at least one convolution in a predefined manner for a feature map input from a feature extraction layer corresponding thereto. I can. In the example of FIG. 9, the first convolution node 262-1 may perform 1x1 convolution on the feature map f1 input from the first feature extraction layer L1 corresponding thereto. The second convolution node 262-2 is a 1x1 convolution, a 3x3 dialed convolution of a rate 6, and a rate of 12 with respect to the feature map f2 input from the corresponding second feature extraction layer L2. It is possible to generate 4 convolution products (features) by performing 3x3 dialed convolution and 3x3 dialed convolution with a rate of 18. The third convolution node 262-3 may perform a 1x1 convolution after twice upsampling the feature map f3 input from the third feature extraction layer L3 corresponding thereto.

Meanwhile, the segmentation sub-architecture 263 may receive a result generated by the convolution sub-architecture 261 and perform a predefined operation. In the example of FIG. 9, the segmentation sub-architecture 263 concatenates all convolution results (features) generated by the second convolution node 511-2 and the third convolution node 511-3. Thus, 1x1 convolution may be performed, and a 3x3 convolution may be performed after combining the convolution result (feature) generated in the first convolution node 511-1.

Thereafter, center cropping may be performed in the cropping sub-architecture 264.

By using a neural network having the technical features of the present invention, the accuracy of segmentation can be greatly improved. It is well known from experience that in order to increase segmentation accuracy in a typical neural network, features must be well extracted from input data, and the initial weight of the neural network must be well set before training and training must be performed. However, as described above, according to the technical idea of the present invention, segmentation is performed by a neural network in a manner that combines a segmentation architecture having a specific structure based on a classification neural network, and is extracted in the patch level classification process. These features reflect the characteristics of the input data very well. Therefore, it is possible to increase the accuracy of the segmentation by using it as it is in the segmentation process.

Meanwhile, in the present specification, an example to which the technical idea of the present invention is applied to prostate cancer has been mainly described, but not only a specific tissue but also other diseases in which the corresponding tissue needs to perform diagnosis of the specific tissue considering the state of the surrounding tissue. Also, when the technical idea of the present invention is applied, an average expert in the technical field of the present invention can easily infer that accurate diagnosis may be possible.

Meanwhile, according to an implementation example, the diagnostic system 100 may include a processor and a memory storing a program executed by the processor. The processor may include a single-core CPU or a multi-core CPU. The memory may include high-speed random access memory and may include non-volatile memory such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid state memory devices. Access to memory by processors and other components can be controlled by a memory controller.

Meanwhile, the multi-color model and the diagnosis method through a neural network according to an embodiment of the present invention may be implemented in the form of a computer-readable program command and stored in a computer-readable recording medium. The corresponding control program and the target program may also be stored in a computer-readable recording medium. The computer-readable recording medium includes all types of recording devices that store data that can be read by a computer system.

The program instructions recorded on the recording medium may be specially designed and configured for the present invention, or may be known and usable to those skilled in the software field.

Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and floptical disks. Hardware devices specially configured to store and execute program instructions such as magneto-optical media and ROM, RAM, flash memory, and the like are included. In addition, the computer-readable recording medium is distributed over a computer system connected through a network, so that computer-readable codes can be stored and executed in a distributed manner.

Examples of program instructions include not only machine language codes such as those produced by a compiler, but also devices that process information electronically using an interpreter, for example, high-level language codes that can be executed by a computer.

The above-described hardware device may be configured to operate as one or more software modules to perform the operation of the present invention, and vice versa.

The above description of the present invention is for illustrative purposes only, and those of ordinary skill in the art to which the present invention pertains will be able to understand that other specific forms can be easily modified without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative and non-limiting in all respects. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. .

Claims (15)

delete delete delete Generating, by a diagnostic system using a neural network, a diagnostic neural network for predicting a diagnosis result for a predetermined disease by using a living body image;
Acquiring, by the diagnosis system using the neural network, a plurality of training biometric images-each of the plurality of training biometric images is labeled with a corresponding diagnosis result for the disease; And
A method comprising the step of, by the diagnostic system using the neural network, training the diagnostic neural network using the plurality of training biometric images,
Training the diagnostic neural network may include, for each of the plurality of training biometric images,
(a) generating K noise-inserted images corresponding to the training biometric images by inserting noises having different characteristics into the training biometric images (where K is an integer greater than or equal to 2); And
(b) training the diagnostic neural network by inputting K noise-inserted images corresponding to the training biometric images into the diagnostic neural network,
The method further comprises a diagnostic step,
The diagnostic step,
Obtaining K biometric images to be diagnosed continuously photographed by the image sensor; And
Predicting a diagnosis result for the disease by inputting the K biometric images to be diagnosed into the diagnosis neural network,
The diagnostic step,
Further comprising determining whether the image sensor is moving based on the K diagnostic target biometric images,
Predicting a diagnosis result for the disease by inputting the K biometric images to be diagnosed into the diagnosis neural network,
And predicting a diagnosis result for the disease by inputting the K diagnosis target biometric images into the diagnosis neural network when it is determined that the image sensor is not moving.
Generating, by a diagnostic system using a neural network, a diagnostic neural network for predicting a diagnosis result for a predetermined disease by using a living body image;
Acquiring, by the diagnosis system using the neural network, a plurality of training biometric images-each of the plurality of training biometric images is labeled with a corresponding diagnosis result for the disease; And
A method comprising the step of, by the diagnostic system using the neural network, training the diagnostic neural network using the plurality of training biometric images,
Training the diagnostic neural network may include, for each of the plurality of training biometric images,
(a) generating K noise-inserted images corresponding to the training biometric images by inserting noises having different characteristics into the training biometric images (where K is an integer greater than or equal to 2); And
(b) training the diagnostic neural network by inputting K noise-inserted images corresponding to the training biometric images into the diagnostic neural network,
The step (b),
Generating one training data corresponding to the training biometric image by concatenating all K noise-inserted images corresponding to the training biometric image-The training data corresponding to the training biometric image is the training Labeled with diagnostic results for biometric images; And
Training the diagnostic neural network by inputting training data corresponding to the training biological image to the diagnostic neural network,
The method further comprises a diagnostic step,
The diagnostic step,
Obtaining K biometric images to be diagnosed continuously photographed by the image sensor;
Generating one diagnosis subject data by connecting all the K diagnosis subject biometric images; And
And predicting a diagnosis result for the disease by inputting the diagnosis target data into the diagnosis neural network.
The method of claim 5,
The diagnostic neural network,
A segmentation neural network that receives the diagnosis target data as an input layer and specifies a region where a disease exists in the diagnosis target biometric image,
The segmentation neural network,
A classification neural network for receiving the diagnosis target data as an input layer and outputting a classification result regarding whether the disease is present in the biometric image; And
A method comprising a segmentation architecture that receives a feature map generated from each of two or more feature map extraction layers among hidden layers included in the classification neural network and specifies a region in which a disease exists in the biometric image.
The method of claim 6, wherein the segmentation architecture comprises:
Convolution sub-architecture including convolution nodes corresponding to each of the two or more feature extraction layers-each of the convolution nodes is a convolution for a feature map input from a corresponding feature extraction layer or two or more different convolutions To perform; And
A method comprising a segmentation sub-architecture for specifying a region in which a disease exists in a patch based on a convolution result generated by the convolution sub-architecture.
The method of claim 4, wherein the disease is
The method characterized in that it is prostate cancer.
A computer program installed in a data processing apparatus and recorded on a computer-readable medium for performing the method according to any one of claims 4 to 8.
As a diagnostic system using a neural network,
Processor; And a memory for storing a computer program,
When the computer program is executed by the processor, the diagnostic system using a neural network causes the diagnostic system using the neural network to perform the method according to any one of claims 4 to 8.
delete delete A storage module for storing a diagnostic neural network for predicting a diagnosis result for a predetermined disease by using the biometric image;
An acquisition module for acquiring a plurality of training biometric images, in which each of the plurality of training biometric images is labeled with a corresponding diagnosis result for the disease; And
A diagnostic system using a neural network including a learning module for training the diagnostic neural network by using the plurality of training biometric images,
The learning module, for each of the plurality of training biometric images,
(a) generating K noise-inserted images corresponding to the training biometric images by inserting noises having different characteristics into the training biometric images (where K is an integer greater than or equal to 2); And
(b) performing the step of training the diagnostic neural network by inputting K noise-inserted images corresponding to the training biometric images into the diagnostic neural network,
The diagnostic system using the neural network,
Acquires K biometric images to be diagnosed continuously photographed by an image sensor,
Diagnosis system using a neural network further comprising a diagnosis module for predicting a diagnosis result for the disease by inputting the K diagnosis target biometric images into the diagnosis neural network.
The method of claim 13,
The diagnostic module,
Determine whether the image sensor is moving based on the K diagnostic target biometric images,
When it is determined that the image sensor is not in motion, the diagnosis system using a neural network for predicting a diagnosis result for the disease by inputting the K diagnosis target biometric images into the diagnosis neural network.
A storage module for storing a diagnostic neural network for predicting a diagnosis result for a predetermined disease by using the biometric image;
An acquisition module for acquiring a plurality of training biometric images, in which each of the plurality of training biometric images is labeled with a corresponding diagnosis result for the disease; And
A diagnostic system using a neural network including a learning module for training the diagnostic neural network by using the plurality of training biometric images,
The learning module, for each of the plurality of training biometric images,
(a) generating K noise-inserted images corresponding to the training biometric images by inserting noises having different characteristics into the training biometric images (where K is an integer greater than or equal to 2); And
(b) performing the step of training the diagnostic neural network by inputting K noise-inserted images corresponding to the training biometric images into the diagnostic neural network,
The step (b),
Generating one training data corresponding to the training biometric image by concatenating all K noise-inserted images corresponding to the training biometric image-The training data corresponding to the training biometric image is the training Labeled with diagnostic results for biometric images; And
Training the diagnostic neural network by inputting training data corresponding to the training biometric image to the diagnostic neural network,
The diagnostic system using the neural network,
Acquires K biometric images to be diagnosed continuously photographed by an image sensor,
One diagnostic target data is generated by connecting all the K diagnostic target biometric images,
Diagnosis system using a neural network further comprising a diagnosis module for predicting a diagnosis result for the disease by inputting the diagnosis target data into the diagnosis neural network.

Images