JP6621117B1 - Image processing apparatus, image processing system, and image processing program - Google Patents

Abstract

An image processing apparatus, an image processing system, and an image processing program capable of extracting an abnormal part from a medical image. An image processing apparatus includes: an auto encoder configured to restore and output a medical image having no abnormality with respect to an input medical image; A variance estimator configured to output the variance of the image that does not exist, an image input receiving unit that receives an input of a medical image of the diagnostic subject, and a difference between the input image and the restored image output by the auto encoder. A difference map generating means for calculating and generating a difference map; a standard deviation calculating means for calculating a standard deviation based on the estimated variance output from the variance estimator; A diagnostic result image generating unit that extracts a difference portion of the image and generates a diagnosis result image in which the difference portion other than the characteristic difference portion included in the input image is reduced. [Selection diagram] FIG.

Description

  The present invention relates to an image processing apparatus, an image processing system, and an image processing program.

  The following biological image processing apparatuses are known. In this biological image processing apparatus, an input training image that is a biological image including a target object, an output training image that is a biological image not including the target object, and an input image that is a biological image including the target object are received and input Using a training image and an output training image, learn a neural network that performs image processing to generate a biological image that does not include an object from a biological image that includes the object, and use the image processing of the neural network that is the learning result Then, an output image is generated by removing the object from the input image (for example, Patent Document 1).

JP-A-2018-89301

  In a conventional biological image processing apparatus, a neural network that performs image processing for generating a biological image that does not include a target object from a biological image that includes a target object is learned, and an input image is obtained using the neural network image processing that is a learning result. An output image in which the target object is removed from is generated. However, in diagnosis using images, it is often required to find a part that may be abnormal from medical images. For this reason, there is a need for a mechanism capable of assisting in identifying an abnormal part when an image of a person with no abnormality is learned and the result of learning is used to input an image of a person with an abnormality. However, the conventional technique has not been studied at all about this point.

An image processing apparatus according to the present invention includes an auto encoder configured to learn an image pattern of a medical image having no abnormality, and to restore and output a medical image having no abnormality with respect to the input medical image. The variance estimator is configured to learn the variance of the medical image without any abnormality and output the variance of the image with no abnormality with respect to the input medical image, and the medical image of the person to be diagnosed is input. A difference map is generated by calculating a difference between an image input receiving unit that receives an input as an image, an input image input by the image input receiving unit, and a restored image output by an auto encoder as a result of inputting the input image A difference map generating means; a standard deviation calculating means for calculating a standard deviation based on the estimated variance output from the variance estimator as a result of inputting the input image; and a difference map. Based on the difference map generated by the generation means and the standard deviation calculated by the standard deviation calculation means, the difference portion between the input image and the restored image is extracted and the difference portion other than the characteristic difference portion included in the input image And a diagnostic result image generating means for generating a diagnostic result image with reduced noise.
An image processing system according to the present invention includes an auto encoder configured to learn an image pattern of a medical image having no abnormality and to restore and output a medical image having no abnormality with respect to the input medical image. The variance estimator is configured to learn the variance of the medical image without any abnormality and output the variance of the image with no abnormality with respect to the input medical image, and the medical image of the person to be diagnosed is input. A difference map is generated by calculating a difference between an image input receiving unit that receives an input as an image, an input image input by the image input receiving unit, and a restored image output by an auto encoder as a result of inputting the input image A difference map generating means; a standard deviation calculating means for calculating a standard deviation based on the estimated variance output from the variance estimator as a result of inputting the input image; and a difference Based on the difference map generated by the loop generation means and the standard deviation calculated by the standard deviation calculation means, the difference portion between the input image and the restored image is extracted, and other than the characteristic difference portion included in the input image And a diagnostic result image generating means for generating a diagnostic result image with reduced differences.
An image processing program according to the present invention learns an image input acceptance procedure for accepting an input using a medical image of a person to be diagnosed as an input image, and an image pattern of a medical image having no abnormality. A restored image acquisition procedure in which an input image received by the image input acceptance procedure is input to an auto encoder configured to restore and output a medical image having no abnormality, and a restored image is obtained by the auto encoder, and an image The difference map generation procedure for generating a difference map by calculating the difference between the input image received in the input reception procedure and the restored image acquired in the restored image acquisition procedure, and learning of the dispersion of the medical image without abnormality, input image the input of which is accepted by the image input accepting procedure configured dispersed estimator to output the variance of abnormal there is no image on the input medical image To obtain the estimated variance obtained by the variance estimator, the standard deviation calculation procedure for calculating the standard deviation based on the estimated variance obtained in the estimated variance acquisition procedure, and the difference generated in the difference map generation procedure Based on the standard deviation calculated by the map and the standard deviation calculation procedure, the difference part between the input image and the restored image is extracted, and the diagnostic result image in which the difference part other than the characteristic difference part included in the input image is reduced It is a program for causing a computer to execute a diagnostic result image generation procedure to be generated.

  According to the present invention, by using an auto encoder configured to learn an image pattern of a medical image having no abnormality and restore and output a medical image having no abnormality with respect to the input medical image Since it is possible to extract a difference part between the input image and the restored image and reduce a difference part other than the characteristic difference part included in the input image, an abnormality using a medical image can be generated. The identification of the part can be supported.

1 is a block diagram illustrating a configuration of an embodiment of an image processing apparatus 100. FIG. It is the figure which showed the flow of the process by an auto encoder typically. It is the figure which showed typically the flow of the process by a dispersion | distribution estimator. It is the figure which showed typically the production method of a difference map. It is the figure which showed typically the method for reducing difference parts other than an abnormal site | part using a difference map and an estimated standard deviation. It is the figure which put together the flow of the training step and the estimation step, and showed typically. It is a flowchart figure which shows the flow of a training step. It is a flowchart figure which shows the flow of an estimation step.

  FIG. 1 is a block diagram illustrating a configuration of an embodiment of an image processing apparatus 100 according to the present embodiment. The image processing apparatus 100 executes a process for extracting and visualizing an abnormal site from a medical image using medical images taken in advance, for example, an MRI image, a CT image, and an X-ray image. As the image processing apparatus 100, for example, a server apparatus or a personal computer is used. FIG. 1 is a block diagram showing a configuration of an embodiment when a personal computer is used as the image processing apparatus 100 in the present embodiment.

  The image processing apparatus 100 includes an operation member 101, a control device 102, a storage medium 103, and a display device 104.

  The operation member 101 includes various devices that are operated by an operator of the image processing apparatus 100, such as a keyboard and a mouse.

  The control device 102 includes a CPU, a memory, and other peripheral circuits, and controls the entire image processing device 100. In addition, the memory which comprises the control apparatus 102 is volatile memories, such as SDRAM, for example. This memory is used as a work memory for the CPU to expand the program when the program is executed and a buffer memory for temporarily recording data. For example, data read via the connection interface 102 is temporarily recorded in the buffer memory.

  The storage medium 103 is a storage medium for recording various data stored by the image processing apparatus 100, data of a program to be executed by the control apparatus 102, and the like, for example, HDD (Hard Disk Drive) or SSD (Solid State). Drive) or the like is used. The program data recorded in the storage medium 103 is provided by being recorded on a recording medium such as a CD-ROM or DVD-ROM, or provided via a network, and stores the program data acquired by the operator. By installing in the medium 103, the control device 102 can execute the program. In the present embodiment, programs and various data used in the processing described below are recorded in the storage medium 103.

  The display device 104 is a liquid crystal monitor, for example, and displays various display data output from the control device 102.

  In this embodiment, a medical image in which no abnormality exists, that is, a medical image in which a specific part of a patient having no abnormality is captured in advance, and the image pattern and pixel distribution of the medical image in which no abnormality exists are distributed. A training step for learning is performed. A medical image that does not have an abnormality is an image that does not include a portion that is recognized as being medically abnormal in the image, for example, an image obtained by photographing a specific portion of a person who does not have a medical abnormality, that is, a normal image An image obtained by photographing a specific part of a patient is used. Hereinafter, training steps in the present embodiment will be described.

  In the training step, as described above, a medical image having no abnormality is used as an input image. When a medical image is input, the control device 102 inputs the medical image to an auto encoder prepared in advance, and causes the auto encoder to learn an image pattern of medical image data having no abnormality. In addition, since the auto encoder can learn the neural network so that the input and the output are the same, in this embodiment, in the training step, the image pattern of the medical image in which no abnormality exists in the auto encoder is learned. Thus, when an image is input in an estimation step described later, an output image that does not have an abnormality can be output. In addition, since an auto encoder is a well-known technique, detailed description is abbreviate | omitted in this Embodiment, but here, for example, a deep auto encoder is assumed.

  FIG. 2 is a diagram schematically showing a flow of processing by an auto encoder (Auto-encoder). When the input image Xh is input to the auto encoder, the encoder Enc converts the high-dimensional input image Xh into a low-dimensional latent representation z, and the decoder Dec performs a high-dimensional reconstruction of the low-dimensional latent representation z. Convert to image X'h.

  The control device 102 repeats the above conversion process while adjusting the weight, which is a parameter used for the calculation of the auto encoder, until the loss L = d (X′h, Xh) is equal to or less than a preset threshold value. The control device 102 sets the weight value set when the loss L is equal to or less than a preset threshold as the weight of the auto encoder. As a result, the auto encoder can be made to learn to output an image having no abnormality with respect to the input image.

  The latent expression z converted by the encoder Enc is input to the variance estimator DecV shown in FIG. FIG. 3 is a diagram schematically showing the flow of processing by the variance estimator DecV. The variance estimator DecV converts the input latent expression z to obtain a restored image X′h, and estimates the variance Vh of the restored image X′h.

  The control device 102 adjusts the variance while adjusting the weight, which is a parameter used for the calculation of the variance estimator DecV, until the loss Lv = d (Vh, d (X′h, Xh)) is equal to or less than a preset threshold value. Repeat the estimation process. The control device 102 sets the weight value set when the loss Lv is equal to or less than a preset threshold as the weight of the scatter estimator DecV. As a result, the variance estimator DecV can be trained to output an image in which the variance of an image having no abnormality with respect to the input image is estimated. As shown in FIG. 3, if the variance of an image having no abnormality can be estimated, the standard deviation can be easily calculated therefrom, and therefore the estimated standard deviation Sh can be easily output.

  When the training step described above is completed, it is possible to execute an estimation step for inputting a medical image of a patient to be diagnosed and outputting an image in which an abnormal region is extracted from the medical image. That is, since the training step described above needs to be completed in order to execute the estimation step, the auto encoder shown in FIG. 2 receives the input image Xh when executing the estimation step. The learning is completed so that an image having no abnormality is output as the restored image X′h, and the variance estimator DecV shown in FIG. 3 restores the restored image X′h in which no abnormality exists with respect to the input image Xh. Learning is completed so that the variance of the output is output as the estimated variance Vh.

  For example, as shown in FIG. 4, when an input image Xa is input to a brain image of a patient having an abnormal part 4a in an auto encoder, an image having no abnormal part is output as a restored image X′a. For this reason, if the difference between the input image Xa and the restored image X′a is taken, the difference image Da from which the abnormal part 4a is extracted can be generated. In the present embodiment, the difference image Da is referred to as a difference map Da.

  Even if this difference map Da is used, the abnormal part 4a can be estimated from the brain image of the patient, but there is a difference between the input image Xa and the restored image X′a in addition to the abnormal part 4a. Therefore, in the difference map Da, a difference portion other than the abnormal part 4a that is a characteristic difference portion included in the input image Xa is also extracted. For example, in brain image data, since a normal part also has a different shape depending on a person, the difference in shape appears as a difference on the difference map Da. Also in FIG. 4, a difference appears on the difference map Da in addition to the abnormal part 4a. For this reason, in the estimation step in the present embodiment, the estimated standard deviation Sa obtained based on the output of the variance estimator DecV is used to reduce the difference portion other than the characteristic difference portion included in the input image Xa. Perform the process.

FIG. 5 is a diagram schematically illustrating a method for reducing a difference portion other than the abnormal part 4a using the difference map Da and the estimated standard deviation Sa. As shown in FIG. 5, the control device 102 performs an operation on the difference map Da and the estimated standard deviation Sa using the calculation formula shown in the following formula (1), so that a difference portion other than the abnormal part 4 a appearing on the image is obtained. Can be reduced. In FIG. 5, a standard score map ZSa is obtained as a diagnostic result image indicating the result.
ZSa = Da / Sa (1)

  The control device 102 outputs the standard score map ZSa obtained by the calculation to the display device 104 and displays it. As a result, the operator of the image processing apparatus 100 captures and inputs a medical image of a patient, for example, a brain image, into the image processing apparatus 100, so that a difference portion from a normal brain image without an abnormal portion is defined as an abnormal portion. Since the extracted standard score map can be confirmed on the display device 104, it is possible to diagnose whether there is an abnormality in the brain of the patient.

  FIG. 6 is a diagram schematically showing the flow of the training step and the estimation step described above. Note that the processing flow shown in FIG. 6 is the same as that described with reference to FIGS.

  FIG. 7 is a flowchart showing a flow of training steps in the present embodiment. The process shown in FIG. 7 is executed by the control device 102 as a program that starts when a normal medical image without an abnormal site, that is, a brain image of a healthy person is input.

  In step S10, the control device 102 inputs the input image Xh to the auto encoder, and the encoder Enc converts the high-dimensional input image Xh into a low-dimensional latent representation z. Then, it progresses to step S20.

  In step S20, the low-dimensional latent representation z is converted into a high-dimensional restored image X′h by the decoder Dec. Then, it progresses to step S30.

  In step S30, the control device 102 calculates the above-described loss L = d (X′h, Xh). Thereafter, the process proceeds to step S40.

  In step S40, the control apparatus 102 inputs the latent expression z converted by the encoder Enc to the variance estimator DecV, and the variance estimator DecV calculates the variance of the restored image X′h obtained by converting the latent expression z. Estimated estimated variance Vh is calculated. Thereafter, the process proceeds to step S50.

  In step S50, the control device 102 calculates the above-described loss Lv = d (Vh, d (X′h, Xh)). Thereafter, the process proceeds to step S60.

  In step S60, the control apparatus 102 determines whether or not the loss L and the loss Lv are equal to or less than preset threshold values. When an affirmative determination is made in step S60, the weight at that time is adopted and the process is terminated. On the other hand, if a negative determination is made in step S60, the process proceeds to step S70.

  In step S70, the control apparatus 102 adjusts the weights of the auto encoder and the variance estimator as described above, and returns to step S10.

  FIG. 8 is a flowchart showing the flow of the estimation step in the present embodiment. The process shown in FIG. 8 is executed by the control device 102 as a program that is activated when a medical image of a patient to be diagnosed is input.

  In step S110, the control apparatus 102 inputs the input image Xa to the auto encoder, and the encoder Enc converts the high-dimensional input image Xa into a low-dimensional latent representation (latent representation) z. Then, it progresses to step S120.

  In step S120, the low-dimensional latent representation z is converted into a high-dimensional restored image X′a by the decoder Dec. Thereafter, the process proceeds to step S130.

  In step S130, as described above, the control apparatus 102 takes the difference between the input image Xa and the restored image X′a to generate the difference map Da. Thereafter, the process proceeds to step S140.

  In step S140, the control apparatus 102 inputs the latent representation z converted by the encoder Enc to the variance estimator DecV, and the variance estimator DecV estimates the variance of the restored image X′a obtained by converting the latent representation z. Va is calculated. Thereafter, the process proceeds to step S150.

  In step S150, the control apparatus 102 calculates the estimated standard deviation Sa based on the calculated estimated variance Va. Thereafter, the process proceeds to step S160.

  In step S160, the control apparatus 102 calculates the standard score map ZSa by the equation (1). Thereafter, the process proceeds to step S170.

  In step S170, the control device 102 outputs the standard score map ZSa obtained by the calculation to the display device 104 and displays it. Thereafter, the process ends.

According to the embodiment described above, the following operational effects can be obtained.
(1) The control device 102 receives an input of a medical image of a person to be diagnosed as an input image, learns an image pattern of a medical image having no abnormality, and has no abnormality with respect to the inputted medical image The input image is input to the auto encoder configured to restore and output the image, the restored image is obtained by the auto encoder, the difference between the input image and the restored image is calculated, and the difference map is generated. did. The control device 102 learns the variance of a medical image having no abnormality and inputs the input image to a variance estimator configured to output the variance of an image having no abnormality with respect to the input medical image. Obtain the estimated variance from the variance estimator, calculate the standard deviation based on the obtained estimated variance, extract the difference between the input image and the restored image based on the difference map and standard deviation, and include them in the input image A diagnostic result image in which a difference portion other than the characteristic difference portion is reduced is generated. Accordingly, it is possible to assist a doctor or the like to identify and diagnose an abnormal region that is not found in a normal patient from the input medical image. In addition, although it is possible to extract an abnormal part from the difference map obtained by calculating the difference between the input image and the restored image, a diagnostic result image that takes into account the standard deviation is generated by the above-described equation (1). Therefore, it is possible to reduce the influence of a different normal part for each patient appearing on the image and generate a diagnosis result image in which a part that is likely to be an abnormal part is clarified.

(2) The control device 102 displays the generated diagnosis result image on the display device 104. Accordingly, an operator such as a doctor can check the diagnosis result image on the display device 104.

(3) The auto-encoder repeats the process while changing the value of the parameter used for the calculation until the difference between the input medical image and the output image reaches a predetermined threshold value. The image pattern was learned. As a result, the learning accuracy of the auto encoder can be improved, and a medical image without any abnormality can be output as a restored image even when various images are input.

-Modification-
Note that the image processing system of the above-described embodiment can be modified as follows.
(1) In the above-described embodiment, the example in which the image processing apparatus 100 is a personal computer and the control apparatus 102 executes the above-described processing has been described. However, the terminal operated by the operator can be connected to the image processing apparatus 100 via a communication line such as the Internet, and the input images Xh and Xa described above are input by being transmitted from the operating terminal to the image processing apparatus 100. May be. In that case, the standard score map ZSa displaying the abnormal part on the diagnosis subject's image may be transmitted to the operation terminal and displayed on the operation terminal. As a result, a client server type or cloud type image processing system can be constructed by connecting the operation terminal and the image processing apparatus 100 via a communication line, while the image processing apparatus 100 is used alone. Therefore, it can be used as a stand-alone device.

(2) In the above-described embodiment, the example in which the processing for the training step is executed by the control apparatus 102 in the image processing apparatus 100 has been described. However, the processing for the training step may be executed by another device, and the data obtained by the training step may be recorded in the storage medium 103. In this case, processing for the training step in the image processing apparatus 100 is not necessary.

  Note that the present invention is not limited to the configurations in the above-described embodiments as long as the characteristic functions of the present invention are not impaired. Moreover, it is good also as a structure which combined the above-mentioned embodiment and a some modification.

DESCRIPTION OF SYMBOLS 100 Image processing apparatus 101 Operation member 102 Control apparatus 103 Storage medium 104 Display apparatus

Claims (9)

An auto encoder configured to learn an image pattern of a medical image having no abnormality and to restore and output a medical image having no abnormality with respect to the input medical image;
A variance estimator configured to learn the variance of medical images without anomalies and output the variance of images without anomalies with respect to the input medical images;
Image input accepting means for accepting an input as a medical image of a person to be diagnosed as an input image;
A difference map generating unit that calculates a difference between the input image input by the image input receiving unit and a restored image output by the auto encoder as a result of inputting the input image, and generates a difference map;
A standard deviation calculating means for calculating a standard deviation based on the estimated variance output from the variance estimator as a result of inputting the input image;
Based on the difference map generated by the difference map generating means and the standard deviation calculated by the standard deviation calculating means, a difference portion between the input image and the restored image is extracted and included in the input image. An image processing apparatus comprising: a diagnosis result image generation unit that generates a diagnosis result image in which a difference portion other than a characteristic difference portion is reduced. The image processing apparatus according to claim 1.
An image processing apparatus, further comprising an image display means for displaying the diagnosis result image generated by the diagnosis result image generation means on a display device. The image processing apparatus according to claim 1 or 2,
The auto-encoder repeats the process while changing the value of the parameter used for the calculation until the difference between the input medical image and the output image reaches a predetermined threshold value, so that the image pattern of the medical image having no abnormality is obtained. An image processing apparatus characterized by learning. An auto encoder configured to learn an image pattern of a medical image having no abnormality and to restore and output a medical image having no abnormality with respect to the input medical image;
A variance estimator configured to learn the variance of medical images without anomalies and output the variance of images without anomalies with respect to the input medical images;
Image input accepting means for accepting an input as a medical image of a person to be diagnosed as an input image;
A difference map generating unit that calculates a difference between the input image input by the image input receiving unit and a restored image output by the auto encoder as a result of inputting the input image, and generates a difference map;
A standard deviation calculating means for calculating a standard deviation based on the estimated variance output from the variance estimator as a result of inputting the input image;
Based on the difference map generated by the difference map generating means and the standard deviation calculated by the standard deviation calculating means, a difference portion between the input image and the restored image is extracted and included in the input image. An image processing system comprising: a diagnosis result image generating unit that generates a diagnosis result image in which a difference portion other than a characteristic difference portion is reduced. The image processing system according to claim 4,
An image processing system, further comprising: an image display unit that displays the diagnostic result image generated by the diagnostic result image generation unit on a display device. The image processing system according to claim 4 or 5,
The auto-encoder repeats the process while changing the value of the parameter used for the calculation until the difference between the input medical image and the output image reaches a predetermined threshold value, so that there is no abnormality in the image pattern of the medical image An image processing system characterized by learning. An image input acceptance procedure for accepting an input as a medical image of a diagnostic subject as an input image;
The image input reception procedure is used to input an auto encoder configured to learn an image pattern of a medical image having no abnormality and to restore and output a medical image having no abnormality with respect to the input medical image. A restored image acquisition procedure for inputting the received input image and obtaining a restored image by the auto encoder;
A difference map generation procedure for generating a difference map by calculating a difference between the input image received in the image input reception procedure and the restored image acquired in the restored image acquisition procedure;
The variance input estimator configured to learn the variance of a medical image having no abnormality and output the variance of an image having no abnormality with respect to the input medical image, the input accepted in the image input acceptance procedure An estimated variance acquisition procedure for inputting an input image and obtaining an estimated variance by the variance estimator;
A standard deviation calculation procedure for calculating a standard deviation based on the estimated variance acquired in the estimated variance acquisition procedure;
Based on the difference map generated in the difference map generation procedure and the standard deviation calculated in the standard deviation calculation procedure, a difference portion between the input image and the restored image is extracted and is included in the input image An image processing program for causing a computer to execute a diagnostic result image generation procedure for generating a diagnostic result image in which a difference portion other than a different difference portion is reduced. The image processing program according to claim 7,
An image processing program further comprising an image display procedure for displaying the diagnostic result image generated in the diagnostic result image generation procedure on a display device. The image processing program according to claim 7 or 8,
The auto-encoder repeats the process while changing the value of the parameter used for the calculation until the difference between the input medical image and the output image reaches a predetermined threshold value, so that there is no abnormality in the image pattern of the medical image An image processing program characterized by learning.

Images