JP2021086560A - Medical image processing apparatus, medical image processing method, and program - Google Patents

Abstract

To provide a medical image processing apparatus which can generate an image more suitable for image diagnosis than before.SOLUTION: A medical image processing apparatus includes: an acquisition unit which acquires a first medical image obtained by a first imaging apparatus imaging a predetermined section of a subject; and an image quality enhancing unit which generates a second medical image having image quality higher than the first medical image, from the first medical image, by use of a learned model obtained with learning data using the medical image obtained by the first imaging apparatus imaging the predetermined section of the subject as input data, and a medical image obtained by a second imaging apparatus different from the first imaging apparatus and imaging the predetermined section of the subject as output data.SELECTED DRAWING: Figure 7

Description

The present invention relates to a medical image processing apparatus, a medical image processing method and a program.

In the medical field, in order to identify a disease of a subject and observe the degree of the disease, images are acquired by various imaging devices, and image diagnosis is performed by a medical worker. The types of imaging equipment include, for example, in the field of radiology, X-ray imaging equipment, X-ray computed tomography (CT) equipment, magnetic resonance imaging (MRI) equipment, positron emission tomography (PET) equipment, and single photon emission. There are tomography (SPECT) devices and the like. Further, for example, in the field of ophthalmology, there are a fundus camera, a scanning laser ophthalmoscope (SLO), an optical coherence tomography (OCT) device, and an OCT angiography (OCTA) device.

In order to perform accurate image diagnosis and complete it in a short time, it is important to have low noise in the image acquired by the imaging device, high resolution / spatial resolution, and high image quality such as appropriate gradation. It becomes. Images that emphasize the area or lesion you want to observe may also be useful.

However, in many photographing devices, some compensation is required in order to acquire an image suitable for image diagnosis, such as high image quality. For example, there is a way to purchase a high-performance imaging device to obtain a high-quality image, but it often requires more investment than a low-performance one.

Further, for example, in CT, it may be necessary to increase the exposure dose of the subject in order to acquire an image with less noise. Further, for example, in MRI, a contrast medium having a risk of side effects may be used in order to obtain an image in which the portion to be observed is emphasized. Further, in OCT, for example, when the area to be photographed is wide or a high spatial resolution is required, the photographing time may be longer. Further, for example, in some photographing devices, it is necessary to acquire an image a plurality of times in order to acquire an image having high image quality, and it takes time to take an image accordingly.

Patent Document 1 discloses a technique for converting a previously acquired image into a higher resolution image by an artificial intelligence engine in order to cope with rapid progress in medical technology and simple shooting in an emergency. .. According to such a technique, for example, an image acquired by simple shooting with low cost can be converted into an image having a higher resolution.

JP-A-2018-5841

Here, in the artificial intelligence engine, when a high-resolution image obtained by image processing is used as the teacher data of the training data, the image obtained by using the artificial intelligence engine is an image having a higher resolution than the input image. However, it may not be an image suitable for diagnostic imaging.

On the other hand, one of the objects of the present invention is to provide a medical image processing apparatus, a medical image processing method and a program capable of generating an image more suitable for image diagnosis than before.

The medical image processing apparatus according to one embodiment of the present invention is
An acquisition unit that acquires a first medical image obtained by photographing a predetermined part of a subject with a first imaging device, and
The medical image obtained by photographing a predetermined part of the subject with the first imaging device was used as input data, and the predetermined part of the subject was photographed with a second imaging device different from the first imaging device. Using the trained model obtained by using the training data using the medical image as the output data, the second medical image has a higher image quality than the first medical image from the acquired first medical image. It is provided with a high image quality unit for generating an image.

According to one of the present inventions, it is possible to generate an image more suitable for diagnostic imaging than before.

It is a figure which shows an example of the schematic structure of the OCT system which concerns on 1st Embodiment. It is a figure which shows an example of the schematic structure of the OCT system which concerns on 1st Embodiment. It is a figure which shows an example of the schematic structure of the image processing part which concerns on 1st Embodiment. It is a figure which shows an example of the 1st SLO image and a tomographic image which concerns on 1st Embodiment. It is a figure which shows an example of the 2nd SLO image and the tomographic image which concerns on 1st Embodiment. It is a figure which shows the size conversion of the tomographic image which concerns on 1st Embodiment. It is a figure which shows an example of the structure of the neural network concerning the high image quality processing. It is a figure which shows an example of the display of the tomographic image which concerns on 1st Embodiment. It is a figure which shows an example of the case where the tomographic image which concerns on 2nd Embodiment is divided and used. It is a figure which shows an example of the structure of the neural network concerning the high image quality processing. It is a flow chart which shows an example of the flow of image processing which concerns on 1st Embodiment. It is a flow figure which shows another example of the flow of image processing which concerns on 1st Embodiment. It is a figure which shows an example of the structure of the neural network used as the machine learning engine which concerns on modification 6. It is a figure which shows an example of the structure of the neural network used as the machine learning engine which concerns on modification 6. It is a figure which shows an example of the user interface which concerns on modification 1. It is a figure which shows an example of the user interface which concerns on modification 1. An example of an input image related to high image quality processing is shown.

Hereinafter, exemplary embodiments for carrying out the present invention will be described in detail with reference to the drawings. However, the dimensions, materials, shapes, relative positions of the components, etc. described in the following embodiments are arbitrary and can be changed according to the configuration of the device to which the present invention is applied or various conditions. Also, in the drawings, the same reference numerals are used between the drawings to indicate elements that are the same or functionally similar.

<Explanation of terms>
First, terms used in the present specification will be described.

In the network herein, each device may be connected by a wired or wireless line. Here, the lines connecting each device in the network are, for example, a dedicated line, a local area network (hereinafter referred to as LAN) line, a wireless LAN line, an Internet line, Wi-Fi (registered trademark), and Bluetooth (registered trademark). ) Etc. are included.

The medical image processing device may be composed of two or more devices capable of communicating with each other, or may be composed of a single device. Further, each component of the medical image processing apparatus may be composed of a software module executed by a processor such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). In addition, each component may be configured by a circuit or the like that performs a specific function such as an ASIC. Further, it may be configured by a combination of any other hardware and any software.

Further, the medical image processed by the medical image processing apparatus or the medical image processing method according to the following embodiment includes an image acquired by using an arbitrary modality (imaging apparatus, imaging method). The medical image to be processed may include a medical image acquired by an arbitrary imaging device or the like, a medical image processing device according to the following embodiment, or an image created by a medical image processing method.

Further, the medical image to be processed is an image of a predetermined part of the subject (subject), and the image of the predetermined part includes at least a part of the predetermined part of the subject. In addition, the medical image may include other parts of the subject. Further, the medical image may be a still image or a moving image, and may be a black-and-white image or a color image. Further, the medical image may be an image showing the structure (morphology) of a predetermined part or an image showing the function thereof. The image showing the function includes, for example, an OCTA image, a Doppler OCT image, an fMRI image, and an image showing hemodynamics (blood flow volume, blood flow velocity, etc.) such as an ultrasonic Doppler image. The predetermined part of the subject may be determined according to the subject to be imaged, and the human eye (eye to be examined), brain, lung, intestine, heart, pancreas, kidney, liver and other organs, head, chest, etc. Includes any part such as legs and arms.

Further, the medical image may be a tomographic image of the subject or a frontal image. The frontal image is, for example, a frontal image of the fundus, a frontal image of the anterior segment of the eye, a fluorescently photographed fundus image, and at least a part of the range of data acquired by OCT (three-dimensional OCT data) in the depth direction of the object to be imaged. Includes En-Face images generated using the data from. The En-Face image is an OCTA En-Face image (motion contrast) generated by using at least a part of the data in the depth direction of the shooting target for the three-dimensional OCTA data (three-dimensional motion contrast data). Front image) may be used. Further, three-dimensional OCT data and three-dimensional motion contrast data are examples of three-dimensional medical image data.

Here, the motion contrast data is data indicating a change between a plurality of volume data obtained by controlling the measurement light to be scanned a plurality of times in the same region (same position) of the eye to be inspected. At this time, the volume data is composed of a plurality of tomographic images obtained at different positions. Then, motion contrast data can be obtained as volume data by obtaining data showing changes between a plurality of tomographic images obtained at substantially the same position at different positions. The motion contrast frontal image is also referred to as an OCTA frontal image (OCTA En-Face image) relating to OCTA angiography (OCTA) for measuring the movement of blood flow, and the motion contrast data is also referred to as OCTA data. The motion contrast data can be obtained, for example, as a decorrelation value, a variance value, or a maximum value divided by a minimum value (maximum value / minimum value) between two tomographic images or corresponding interference signals. , It may be obtained by any known method. At this time, the two tomographic images can be obtained, for example, by controlling the measurement light to be scanned a plurality of times in the same region (same position) of the eye to be inspected.

The En-Face image is, for example, a front image generated by projecting data in the range between two layer boundaries in the XY directions. At this time, the front image is at least a part of the depth range of the volume data (three-dimensional tomographic image) obtained by using optical interference, and is the data corresponding to the depth range determined based on the two reference planes. Is projected or integrated on a two-dimensional plane. The En-Face image is a frontal image generated by projecting the data corresponding to the depth range determined based on the detected retinal layer of the volume data onto a two-dimensional plane. As a method of projecting data corresponding to a depth range determined based on two reference planes on a two-dimensional plane, for example, a representative value of data within the depth range is set as a pixel value on the two-dimensional plane. Techniques can be used. Here, the representative value can include a value such as an average value, a median value, or a maximum value of pixel values within a range in the depth direction of a region surrounded by two reference planes. Further, the depth range related to the En-Face image is, for example, a range including a predetermined number of pixels in a deeper direction or a shallower direction with respect to one of the two layer boundaries relating to the detected retinal layer. May be good. Further, the depth range related to the En-Face image may be, for example, a range changed (offset) according to an operator's instruction from a range between two layer boundaries related to the detected retinal layer. Good.

The imaging device is a device for capturing an image used for diagnosis. The photographing device detects, for example, a device that obtains an image of a predetermined part by irradiating a predetermined part of the subject with radiation such as light or X-rays, electromagnetic waves, ultrasonic waves, or the like, or radiation emitted from the subject. This includes a device for obtaining an image of a predetermined part. More specifically, the imaging devices according to the following embodiments are at least an X-ray imaging device, a CT device, an MRI device, a PET device, a SPECT device, an SLO device, an OCT device, an OCTA device, a fundus camera, and an endoscope. Includes mirrors, etc.

The OCT apparatus may include a time domain OCT (TD-OCT) apparatus and a Fourier domain OCT (FD-OCT) apparatus. Further, the Fourier domain OCT apparatus may include a spectral domain OCT (SD-OCT) apparatus and a wavelength sweep type OCT (SS-OCT) apparatus. In addition, the OCT apparatus may include a Doppler-OCT apparatus. Further, the SLO device and the OCT device may include a wave surface compensation SLO (AO-SLO) device using a wave surface compensation optical system, a wave surface compensation OCT (AO-OCT) device, and the like. Further, the SLO device and the OCT device may include a polarized SLO (PS-SLO) device, a polarized OCT (PS-OCT) device, and the like for visualizing information on polarization phase difference and polarization elimination. Further, the SLO device and the OCT device may include a pathological microscope SLO device, a pathological microscope OCT device, and the like. Further, the SLO device and the OCT device may include a handheld type SLO device, a handheld type OCT device, and the like. Further, the SLO device and the OCT device may include a catheter SLO device, a catheter OCT device and the like.

An image management system is a device and system that receives and stores an image taken by a photographing device or an image processed image. In addition, the image management system may transmit an image in response to a request from the connected device, perform image processing on the saved image, or request an image processing request from another device. it can. The image management system can include, for example, an image storage communication system (PACS). In particular, the image management system according to the following embodiment includes a database capable of storing various information such as subject information and shooting time associated with the received image. In addition, the image management system is connected to a network and can send and receive images, convert images, and send and receive various information associated with saved images in response to requests from other devices. ..

The shooting conditions are various information at the time of shooting the image acquired by the shooting device. The shooting conditions include, for example, information about the shooting device, information about the facility where the shooting was performed, information about the inspection related to shooting, information about the photographer, information about the subject, and the like. The shooting conditions include, for example, shooting date and time, shooting part name, shooting area, shooting angle of view, shooting method, image resolution and gradation, image size, applied image filter, information on image data format, and radiation. Includes information about quantity, etc. The photographing area may include a peripheral area deviated from a specific photographing part, an area including a plurality of imaging parts, and the like.

The shooting conditions can be saved in the data structure constituting the image, saved as shooting condition data different from the image, or saved in a database or an image management system related to the shooting device. Therefore, the photographing conditions can be acquired by a procedure corresponding to the means for storing the photographing conditions of the photographing apparatus. Specifically, the shooting conditions are, for example, for analyzing the data structure of the image output by the shooting device, acquiring the shooting condition data corresponding to the image, and acquiring the shooting conditions from the database related to the shooting device. Obtained by accessing the interface of.

Depending on the photographing device, there are some photographing conditions that cannot be acquired because they are not saved. For example, the imaging device does not have the ability to acquire or save specific imaging conditions, or such a function is disabled. Further, for example, it may not be saved because it is a shooting condition unrelated to the shooting device or shooting. Further, for example, the shooting conditions may be hidden, encrypted, or cannot be acquired without the right. However, it may be possible to acquire even shooting conditions that have not been saved. For example, by performing image analysis, it is possible to specify the name of the imaged part and the imaged area.

The machine learning model is a model in which training (learning) is performed on an arbitrary machine learning algorithm using appropriate teacher data (learning data) in advance. The teacher data is composed of one or more pairs of input data and output data (correct answer data). The format and combination of the input data and output data of the pair group that composes the teacher data may be one of which is an image and the other of which is a numerical value, or one of which is composed of a plurality of image groups and the other of which is a character string. May be an image or the like, which is suitable for a desired configuration.

Specifically, for example, teacher data (hereinafter, first teacher data) composed of a pair group of an image acquired by OCT and a imaging site label corresponding to the image can be mentioned. The photographed part label is a unique numerical value or character string representing the part. Further, as an example of other teacher data, it is composed of a pair group of a noisy low-quality image acquired by normal OCT shooting and a high-quality image shot multiple times by OCT to improve the image quality. Teacher data (hereinafter, second teacher data) and the like can be mentioned.

When input data is input to the machine learning model, output data according to the design of the machine learning model is output. The machine learning model outputs output data that is likely to correspond to the input data, for example, according to the tendency trained with the teacher data. Further, the machine learning model can output, for example, the possibility of corresponding to the input data as a numerical value for each type of output data according to the tendency of training using the teacher data. Specifically, for example, when an image acquired by OCT is input to a machine learning model trained with the first teacher data, the machine learning model outputs a imaging site label of the imaging site captured in the image. Or output the probability for each imaging site label. Further, for example, when a low-quality image with a lot of noise acquired by normal shooting of OCT is input to the machine learning model trained with the second teacher data, the machine learning model is shot multiple times by OCT to improve the image quality. Outputs a high-quality image equivalent to the processed image. From the viewpoint of quality maintenance, the machine learning model can be configured so that the output data output by itself is not used as teacher data.

Machine learning algorithms also include techniques for deep learning such as convolutional neural networks (CNNs). In the method related to deep learning, if the parameter settings for the layers and nodes constituting the neural network are different, the degree to which the tendency trained using the teacher data can be reproduced in the output data may be different. For example, in a deep learning machine learning model using the first teacher data, if more appropriate parameters are set, the probability of outputting a correct imaging site label may be higher. Further, for example, in a deep learning machine learning model using the second teacher data, if more appropriate parameters are set, a higher quality image may be output.

Specifically, the parameters in the CNN are, for example, the kernel size of the filter, the number of filters, the stride value, and the dilation value, and the number of nodes output by the fully connected layer, which are set for the convolution layer. Etc. can be included. The parameter group and the number of training epochs can be set to preferable values for the usage pattern of the machine learning model based on the teacher data. For example, based on the teacher data, it is possible to set a parameter group and the number of epochs that can output the correct imaged part label with a higher probability and output a higher quality image.

One of the methods for determining such a parameter group and the number of epochs will be illustrated. First, 70% of the pair group constituting the teacher data is used for training, and the remaining 30% is randomly set for evaluation. Next, the machine learning model is trained using the training pair group, and the training evaluation value is calculated using the evaluation pair group at the end of each training epoch. The training evaluation value is, for example, the average value of a group of values in which the output when the input data constituting each pair is input to the machine learning model during training and the output data corresponding to the input data are evaluated by the loss function. is there. Finally, the parameter group and the number of epochs when the training evaluation value becomes the smallest are determined as the parameter group and the number of epochs of the machine learning model. In this way, by dividing the pair group that constitutes the teacher data into training and evaluation and determining the number of epochs, the machine learning model overfits the training pair group. Can be prevented.

The high image quality engine (learned model for high image quality) is a module that outputs a high image quality image obtained by improving the high image quality of the input low image quality image. Here, high image quality in the present specification means converting an input image into an image having an image quality more suitable for image diagnosis, and a high image quality image is converted into an image having an image quality more suitable for image diagnosis. Refers to the image. The low-quality image is, for example, a two-dimensional image or a three-dimensional image acquired by X-ray photography, CT, MRI, OCT, PET, SPECT, or the like, or a three-dimensional moving image of CT continuously photographed. It was taken without setting the image quality. Specifically, low-quality images include, for example, images acquired by low-dose radiography using an X-ray imaging device or CT, MRI imaging without using a contrast medium, short-time OCT imaging, and a small number of imaging images. Includes OCTA images acquired by the number of times.

Further, the content of the image quality suitable for the image diagnosis depends on what is desired to be diagnosed by various image diagnoses. Therefore, it cannot be said unconditionally, but for example, the image quality suitable for image diagnosis has less noise, high contrast, the image to be photographed is shown in colors and gradations that are easy to observe, and the image size is large. Includes image quality such as high resolution. In addition, it is possible to include an image quality in which objects and gradations that do not actually exist that have been drawn in the process of image generation are removed from the image.

Further, when a high-quality image having less noise or high contrast is used for image analysis such as blood vessel analysis processing of an image such as OCTA or area segmentation processing of an image such as CT or OCT, a low-quality image can be obtained. In many cases, analysis can be performed more accurately than using it. Therefore, the high-quality image output by the high-quality engine may be useful not only for image diagnosis but also for image analysis.

In the image processing method constituting the image quality improvement method in the following embodiment, processing using various machine learning algorithms such as deep learning is performed. In the image processing method, in addition to the processing using the machine learning algorithm, various existing optional image processing, matching processing using a database of high-quality images corresponding to similar images, and knowledge-based image processing are performed. May be processed.

In particular, there is a configuration shown in FIG. 10 as a configuration example of a CNN that improves the image quality of a two-dimensional image. The configuration of the CNN includes a plurality of convolution processing blocks 1000 groups. The convolution processing block 1000 includes a convolution layer 1010, a batch normalization layer 1020, and an activation layer 1030 using a rectifier linear unit (ReLU) function. The CNN configuration also includes a synthetic (Merge) layer 1040 and a final convolution layer 105. The composite layer 1040 synthesizes by connecting or adding the output value group of the convolution processing block 1000 and the pixel value group constituting the image. The final convolution layer 1050 outputs a group of pixel values constituting the high-quality image Im1200 synthesized by the composite layer 104. In such a configuration, the value group in which the pixel value group constituting the input image Im1100 is output via the convolution processing block 1000 group and the pixel value group constituting the input image Im1100 are formed in the composite layer 1040. It is synthesized. After that, the combined pixel value group is formed into a high-quality image Im1200 at the final convolution layer 1050.

For example, by setting the number of convolution processing blocks 1000 to 16 and setting the kernel size of the filter to 3 pixels in width and 3 pixels in height and the number of filters to 64 as parameters of the convolution layer 1010 group, a constant high image quality is obtained. You can get the effect of conversion. However, in reality, as described in the above description of the machine learning model, a better parameter group can be set by using the teacher data according to the usage pattern of the machine learning model. If it is necessary to process a three-dimensional image or a four-dimensional image, the kernel size of the filter may be expanded to three-dimensional or four-dimensional.

Further, as another example of the configuration example of the CNN in the image quality improving unit 225 according to the present embodiment, FIG. 7 will be described. FIG. 7 shows an example of the machine learning model configuration in the image quality improving unit 225. The configuration shown in FIG. 7 is composed of a plurality of layers that are responsible for processing and outputting the input value group. As shown in FIG. 7, the types of layers included in the above configuration include a convolution layer, a downsampling layer, an upsampling layer, and a composite layer. The convolution layer is a layer that performs convolution processing on the input value group according to parameters such as the kernel size of the set filter, the number of filters, the stride value, and the dilation value. The number of dimensions of the kernel size of the filter may be changed according to the number of dimensions of the input image. The downsampling layer is a process of reducing the number of output value groups to be smaller than the number of input value groups by thinning out or synthesizing the input value groups. Specifically, for example, there is a Max Polling process. The upsampling layer is a process of increasing the number of output value groups to be larger than the number of input value groups by duplicating the input value group or adding the interpolated value from the input value group. Specifically, for example, there is a linear interpolation process. The composite layer is a layer in which a value group such as an output value group of a certain layer or a pixel value group constituting an image is input from a plurality of sources, and the processing is performed by concatenating or adding them. In such a configuration, the value group in which the pixel value group constituting the input image 701 is output through the convolution processing block and the pixel value group constituting the input image 701 are combined in the composite layer. After that, the combined pixel value group is formed into a high-quality image 702 in the final convolution layer. Although not shown, as an example of changing the configuration of the CNN, for example, a batch normalization (Batch Normalization) layer and an activation layer using a normalization linear (ReLU: Rectifier Liner Unit) function are incorporated after the convolution layer. Etc. may be done.

Here, the GPU can perform efficient calculations by processing more data in parallel. Therefore, when learning is performed a plurality of times using a learning model such as deep learning, it is effective to perform processing on the GPU. Therefore, in the present embodiment, the GPU is used in addition to the CPU for the processing by the image processing unit 220, which is an example of the learning unit (not shown). Specifically, when executing a learning program including a learning model, learning is performed by the CPU and the GPU collaborating to perform calculations. The processing of the learning unit may be performed only by the CPU or GPU. Further, the image quality improving unit 225 may also use the GPU in the same manner as the learning unit. Further, the learning unit may include an error detecting unit and an updating unit (not shown). The error detection unit obtains an error between the output data output from the output layer of the neural network and the correct answer data according to the input data input to the input layer. The error detection unit may use the loss function to calculate the error between the output data from the neural network and the correct answer data. Further, the update unit updates the coupling weighting coefficient between the nodes of the neural network based on the error obtained by the error detection unit so that the error becomes small. This updating unit updates the coupling weighting coefficient and the like by using, for example, the backpropagation method. The error backpropagation method is a method of adjusting the coupling weighting coefficient between the nodes of each neural network so that the above error becomes small.

When using some image processing methods such as image processing using CNN, it is necessary to pay attention to the image size. Specifically, it should be noted that different image sizes may be required for the input low-quality image and the output high-quality image in order to deal with the problem that the peripheral part of the high-quality image is not sufficiently high-quality. Should be.

For the sake of clarity, although not specified in the embodiments described later, when a high image quality engine that requires different image sizes for the image input to the high image quality engine and the output image is adopted, an appropriate image It is assumed that the size is adjusted. Specifically, padding is performed on an input image such as an image used for teacher data for training a machine learning model or an image input to a high image quality engine, and a shooting area around the input image is set. Adjust the image size by combining them. The area to be padded is filled with a fixed pixel value, filled with neighboring pixel values, or mirror padded according to the characteristics of the high image quality method so that the image quality can be effectively improved.

Further, the image quality improving method may be implemented by only one image processing method, or may be implemented by combining two or more image processing methods. In addition, a plurality of high image quality improvement method groups may be performed in parallel to generate a plurality of high image quality image groups, and then the highest image quality high image quality image may be finally selected as the high image quality image. The selection of the highest image quality image may be automatically performed using the image quality evaluation index, or a plurality of high image quality image groups are displayed on the user interface provided in an arbitrary display unit or the like. It may be performed according to the instruction of the examiner (user).

Since the input image without high image quality may be more suitable for image diagnosis, the input image may be added to the final image selection target. Further, parameters may be input to the high image quality engine together with the low image quality image. A parameter for specifying the degree of high image quality and a parameter for specifying the image filter size used in the image processing method may be input to the high image quality engine together with the input image.

<First Embodiment>
Hereinafter, the medical image processing apparatus according to the first embodiment will be described with reference to FIGS. 1 to 3. Here, FIG. 3 shows an example of the schematic configuration of the image processing unit 220 as an example of the schematic configuration of the medical image processing apparatus according to the present embodiment.

<Outline configuration of OCT system>
FIG. 1 shows an example of a schematic configuration of an OCT system (ophthalmic imaging system) according to the present embodiment. The photographing apparatus 100 included in the OCT system according to the present embodiment is an example of an SS-OCT apparatus that disperses light in time using a wavelength sweeping light source. The OCT system according to the present embodiment includes an imaging device 100, a control device 200 (an example of a medical image processing device), and a display unit 300. Here, these may be configured as separate bodies and connected to each other so as to be communicable with each other. Moreover, these may be configured as one. Hereinafter, the configurations of the photographing device 100 and the control device 200 will be described in order with reference to FIGS. 1 to 3.

<Configuration of imaging device 100>
The imaging device 100 has a built-in measurement optical system for capturing a two-dimensional image and a tomographic image of the anterior segment Ea of the eye to be inspected E and the fundus Er of the eye to be inspected E. Hereinafter, the configuration arranged inside the photographing device 100 will be described. A part of the optical system (for example, the reference optical system, the detection optical path of the OCT optical system, etc.) of the optical system arranged inside the photographing apparatus 100 may be configured as a separate body. At this time, these optical systems may be configured to be connected to each other by an optical fiber or the like.

The objective lens 101-1 is installed at a position facing the eye E to be inspected in the photographing apparatus 100. A first dichroic mirror 102 and a second dichroic mirror 103 are arranged on the optical axis of the objective lens 101-1, and the optical path from the eye to be inspected is branched by these. That is, the optical path from the eye E to be inspected is the measurement optical path L1 of the OCT optical system, the measurement optical path L2 of the fundus observation optical path and the fixation lamp (hereinafter referred to as the measurement of the SLO optical system), and the anterior eye observation optical path L3 by these optical members. It is branched for each wavelength band. More specifically, the front eye observation optical path L3 is arranged in the transmission direction of the first dichroic mirror 102, and the second dichroic mirror 103 is arranged in the reflection direction. The measurement optical path L1 of the OCT optical system is arranged in the transmission direction of the second dichroic mirror 103, and the measurement optical path L2 of the SLO optical system is arranged in the reflection direction. The arrangement of these optical paths is an example, and the optical paths arranged in the transmission direction and the reflection direction of each dichroic mirror can be appropriately replaced.

The optical path L2 includes an optical path to the APD (Avalanche photodiode) 115 for observing the fundus of the eye, and an optical path to the light source 120 for observing the fundus and the fixation lamp 116 by a prism 118 on which a perforated mirror or a hollow mirror is further deposited. It is branched for each wavelength band. A third dichroic mirror 119 is arranged at a position close to the prism 118. The light from the fundus observation light source 120 and the light from the fixation lamp 116 having different wavelength bands are guided to the optical path L2 toward the eye E to be inspected by the third dichroic mirror 119. Lens 101-2, mirror 113-1, Y scanner 117, X scanner 114, mirror 113-2, shutter 110, lenses 111, 112, and prism 118 are arranged on the optical path L2 in this order from the second dichroic mirror 103. Will be done. The light source 120 has a photodetector (PD) for measuring the amount of light from a light source (not shown) and a current detection circuit for measuring the light source current, and each value can be monitored. The lens 111 is driven in the direction of the optical axis indicated by the arrow in the figure by a focus driving means including a motor (not shown) or the like for focusing adjustment for fixation and fundus observation. The shutter 110 can block the optical path L2 by a driving means including a solenoid (not shown) or the like. The APD 115 arranged in the transmission direction of the prism 118 is a single detector having a sensitivity in the wavelength of the illumination light emitted by the fundus observation light source 120, specifically around 780 nm, and the light scattered / reflected from the fundus Er and returned. Is detected. On the other hand, the fixation lamp 116 arranged in the reflection direction of the prism 118 generates visible light to promote fixation of the subject.

Further, in the optical path L2, an X scanner 114 (for the main scanning direction) and a Y scanner 117 (for the main scanning direction) for scanning the light emitted from the light source for observing the fundus of the eye on the fundus Er of the eye E to be inspected intersect with the main scanning direction. (For sub-scanning direction) is arranged. The lens 101-2 is arranged with the vicinity of the center position of the X scanner 114 and the Y scanner 117 as the focal position. In the present embodiment, the scanners 114 and 117 are composed of polygon mirrors and resonance type galvanometric mirrors, respectively, but may be composed of only galvanometric mirrors. Further, as the scanning unit, a single scanner capable of scanning light in an arbitrary direction such as MEMS may be used by combining these scanners. The vicinity of the center position of the X scanner 114 and the Y scanner 117 and the position of the pupil of the eye E to be inspected are optically in a substantially conjugate relationship.

In the optical path L3, a lens 161 and a monochrome CMOS sensor unit 162 having high sensitivity to near-infrared light for frontal eye observation are arranged in order from the first dichroic mirror 102. The CMOS sensor unit 162 has a sensitivity in the wavelength of the illumination light for frontal eye observation (not shown), specifically in the vicinity of 970 nm. Further, the illumination light for front eye observation has a current detection circuit for measuring the light source current, and the light source current value can be monitored.

The optical path L1 forms an OCT optical system as described above, and is used for taking a tomographic image of the fundus Er of the eye to be inspected. More specifically, it is used to obtain an interference signal for forming a tomographic image. The OCT optical system has a lens 101-3, a mirror 121, a scanning unit, a shutter 126, and lenses 123 and 124, which are arranged in order from the second dichroic mirror 103. The lens 123 is driven in the direction of the optical axis indicated by the arrow in the figure by a focus driving means including a motor (not shown) or the like for focusing adjustment for observing the fundus. The shutter 126 can block the optical path L1 by opening and closing the optical path by a driving means including a solenoid (not shown) or the like. The scanning unit is arranged including an OCT X scanner 122-1 and an OCT Y scanner 122-2 that function to scan the measurement light on the fundus Er of the eye E to be inspected. The OCT X-scanner 122-1 and the OCT Y-scanner 122-2 are arranged so that the vicinity of the center position thereof is the focal position of the lens 101-3. It is said to be an optical conjugate relationship.

With the above configuration, the measurement light deflected by the scanning of each scanner is kept horizontal with respect to the optical axis between the first dichroic mirror 102 and the second dichroic mirror 103 regardless of the scanning state. As a result, the incident angle of the measurement light incident on each dichroic mirror does not change. By adopting this configuration, it is possible to prevent interference of wavelength bands in the optical path provided for each of the above applications. In this embodiment, a galvano scanner is used as the OCT X scanner 122-1 and the OCT Y scanner 122-2. However, it can be changed as appropriate according to the required scanning conditions such as scanning speed, and a resonance scanner or the like may be used. Further, as the scanning unit, a single scanner capable of scanning light in any direction such as MEMS as described above may be used.

Next, the periphery of the light source 130 will be described. The light source 130 is a wavelength sweep type (SS; Swept Source) light source, and emits light while sweeping at, for example, a sweep center wavelength of 1050 nm and a sweep width of 100 nm. The light source 130 has a resonator (not shown). The resonator may be provided inside the light source or outside the light source. The resonator length has a length of, for example, 50 mm.

Next, the configuration of the optical path from the light source 130 and the reference optical system will be described. The light emitted from the light source 130 is guided to the ND filter 132, the fiber 131-2, and the optical coupler 133 through the fiber 131-1. The light transmittance of the ND filter 132 can be adjusted by a motor and a drive mechanism (not shown), and the transmittance can be known from the position. Further, the ND filter 132 is installed at an angle of 45 degrees with respect to the optical path in order to suppress reflection. The OCT optical coupler 133 is branched into the optical fibers 133-1, 2 at a branching ratio of 5:95 to the measurement light, and is guided to the photodetector (PD) 134 and the optical coupler 125 that measure the amount of light. The optical coupler 125 is branched into 125-1 as the measurement light and 125-2 as the reference light at a branching ratio of 70:30. The measurement light is emitted to the optical path L1 through the lens 124.

On the other hand, the reference light is emitted to the reference optical system via the optical fiber 125-2. The reference optical system includes lenses 141-1 and 141-2, dispersion compensating glass 142, and reference mirrors 143, mirrors 144 and 145, and an ND filter 146. The reference light reaches the reference mirror 143 via the lens 141-1, the ND filter 146, the mirrors 144 and 145, and is reflected. The reflected light is emitted to the optical fiber 151-1 via the mirror 145, 144 and the lens 141-2. The position of the reference mirror 143 can be adjusted in the direction of the optical axis indicated by the arrow in the figure by an optical path length adjusting mechanism composed of a motor and a drive mechanism (not shown). Further, the ND filter 146 can adjust the light transmittance by a motor and a drive mechanism (not shown), and the transmittance can be known from the position.

The reference light guided to the optical fiber 151-1 is combined with the optical fiber 151-2 that is reflected by the eye to be inspected and guides the returned measurement light by the optical coupler 151. The light combined by the optical coupler is branched into optical fibers 151-3 and 4 at a branching ratio of 50:50 and emitted to the detector. The detector 150 is a differential detector, and when two interference signals whose phases are inverted by 180 degrees are input, the DC component is removed and only the interference component is output. The interference signal detected by the detector 150 is output as an electric signal according to the light intensity, input to the control device 200, and signal processing is performed to generate a tomographic image.

The motor and drive mechanism (not shown) used in this embodiment are also provided with a sensor mechanism such as a photo interrupter, so that the drive position can be monitored and the step-out of the motor can be known.

The photographing apparatus 100 having the above configuration further includes a temperature / humidity sensor 171. A plurality of temperature / humidity sensors 171 are installed, for example, in a place where the temperature / humidity of the outside air is easily reflected, around the light source 130, around the X scanner 122-1 and the Y scanner 122-2, and on other control boards, and the installation location and number are arbitrary. You can choose.

<Configuration of control device 200>
Next, an example of a schematic configuration of the OCT system according to the present embodiment will be described with reference to FIG. 2, which is a functional block diagram. The control device 200 controls each part by being communicably connected to each part of the photographing device 100 via a circuit or a network, performs a process of generating a tomographic image or the like, and acquires various information of the device. It can be carried out.

The control device 200 is provided with a control unit 211, a first information acquisition unit 212, a storage unit 213, and an image processing unit 220. The control unit 211 controls each component of the photographing apparatus 100. The first information acquisition unit 212 acquires tomographic image data, fundus image data, anterior eye image data, and the state of each component from the photographing apparatus 100. In addition, the first information acquisition unit 212 can acquire various data and images from the photographing device 100 and other devices, and can acquire input from the examiner via an input device (not shown). As the input device, a mouse, a keyboard, a touch panel, or any other input device may be adopted. Further, the display unit 300 may be configured as a touch panel display. The image processing unit 220 processes the image data acquired by the first information acquisition unit 212 to generate a tomographic image. The storage unit 213 stores a program for realizing the processing executed by the control unit 211, data acquired by the first information acquisition unit 212, data such as a tomographic image and shooting information generated by the image processing unit 220, and the like. can do. In addition, the trained model described later can be stored.

Next, the image processing unit 220, which is an example of the medical image processing device according to the present embodiment, will be described with reference to FIG. The image processing unit 220 includes a second information acquisition unit 221, an input unit 222, an output unit 223, a high image quality improvement possibility determination unit 224, and a high image quality improvement unit 225. The second information acquisition unit 221 acquires information such as data acquired by the first information acquisition unit 212 and a tomographic image, data stored in the storage unit 214 and the external storage device 320, and a tomographic image. The input unit 222 inputs the tomographic image acquired by the second information acquisition unit 221 to the high image quality improvement unit 225. The high image quality improvement unit 225 implements a machine learning algorithm corresponding to a trained model that outputs a tomographic image after high image quality when a tomographic image is input as an input. The output unit 223 outputs a tomographic image that has been improved in image quality by the image quality improving unit 225. The high image quality determination unit 224 determines, for example, whether or not the input tomographic image can be improved in image quality from the data obtained by the second information acquisition unit 221. A plurality of trained models are generated in advance according to the tomographic image and stored in the storage unit 214. The high image quality improvement unit 225 analyzes the tomographic image which is the input data of the input unit 222 by using a plurality of trained models, and outputs the high image quality image from the output unit 223.

A display unit 300 such as a monitor and an input unit 222 composed of a pointing device such as a mouse are connected to the control device 200. The display unit 300 displays various images generated by the image processing unit 220, subject information, and other maintenance information output by the image processing unit 220. The input unit 222 functions as an input means when the user makes various inputs to the control device 200. The external storage device 320 can store data in the storage unit of the control device, output images, information output by the image processing unit 220, and the like.

The control device 200 can be configured as a general-purpose computer or a computer dedicated to the photographing device 100. Each component of the control device 200 may be composed of a software module executed by an arithmetic unit such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). Further, each component of the control device 200 may be configured by a circuit or the like that performs a specific function such as an ASIC. Further, the storage unit 214 may be composed of an arbitrary memory, an arbitrary storage medium such as an optical disk, or the like. Further, although the imaging device 100, the control device 200, the display unit 300, and the like are shown as separate bodies, all or each of these may be partially integrated.

A switch or the like that controls the photographing device 100 from other than the input unit 222 may be installed in the photographing device 100 or the control device 200. The switch can have functions such as turning on / off the power supply, shooting, adjusting the focus, and controlling various devices. In addition, the types of switches include push type, rotary type, and toggle type, and the types are not limited.

<Learning model and learning data>
Further, for the learning model related to the machine learning algorithm, for example, the above-mentioned CNN or the like can be used. As parameters set in the convolution layer group, for example, by setting the kernel size of the filter to 3 pixels in width and 3 pixels in height and the number of filters to 64, image processing with a certain accuracy can be performed. However, it should be noted that if the parameter settings for the layers and nodes that make up the neural network are different, the degree to which the tendency trained from the teacher data can be reproduced in the teacher data may differ. That is, in many cases, the appropriate parameters differ depending on the embodiment, and therefore, the values can be changed to preferable values as needed. In addition to the method of changing the parameters as described above, there are cases where the CNN can obtain better characteristics by changing the configuration of the CNN. Better characteristics include, for example, high learning accuracy of the fundus image, short image processing time, and short training time for the trained model.

Further, training of a learning model related to a machine learning algorithm requires one or more pairs of input data and teacher data (output data). In this embodiment, a wide angle of view tomographic image taken by the SS-OCT apparatus shown in FIG. 1 is used as the input data. Further, as the teacher data, a tomographic image of the same subject at the same position taken with high resolution using an SD-OCT device different from the input data photographing device is used. As the pair of the input data and the teacher data, a plurality of pairs such as a plurality of pairs obtained from the same subject and a pair obtained from a plurality of different subjects are used as learning data. When obtaining a trained model, training is performed in advance using the training data consisting of these tomographic images.

When photographing the fundus, the light source of the SS-OCT apparatus mainly uses a wavelength band of 1 μm, while the light source of the SD-OCT apparatus mainly uses a wavelength band of 800 nm. At this time, if the wavelength widths are substantially the same, the SD-OCT device can acquire a tomographic image having a higher resolution (vertical resolution) in the depth direction than the SS-OCT device. That is, the SD-OCT apparatus has a higher resolution (vertical resolution) in the depth direction than the SS-OCT apparatus. When photographing the anterior segment of the eye, a wavelength band of 1.3 μm is mainly used as the light source of the SS-OCT device, so that the relationship of the resolutions of the tomographic images obtained by using these devices is the same. The imaging speed of the SS-OCT apparatus mainly depends on the sweep speed of the wavelength sweep light source, and the A scan rate is about 100 KHz or more. On the other hand, the photographing speed of the SD-OCT apparatus mainly depends on the reading speed of the line sensor, and the A scan rate is about 50 KHz to about 85 KHz. That is, the SS-OCT device can shoot at a higher speed than the SD-OCT device. Here, since the eye to be imaged moves, we want to complete the image as quickly as possible. Therefore, as an apparatus for acquiring a tomographic image having a wide angle of view, an SS-OCT apparatus capable of photographing at a higher speed is mainly used. The A scan rate is the acquisition speed of the A scan image. At this time, the SS-OCT device is an example of the first photographing device, and the SD-OCT device is an example of the second photographing device different from the first photographing device. For example, the first photographing device and the second photographing device may be of different models. Further, for example, the first photographing device and the second photographing device may have different photographing methods. Further, for example, the second imaging device may be capable of acquiring a medical image having a higher resolution than that of the first imaging device. Further, for example, the second photographing device may be capable of photographing with a resolution higher than the resolution of the first photographing device. The tomographic image as input data and the tomographic image as teacher data included in the learning data may be aligned (positional deviation correction) by various methods. For example, the alignment may be performed by using the correlation between a plurality of partial regions set for the tomographic image and the reference image. Further, for example, the alignment may be performed by using the correlation between the characteristic region such as the macula in the tomographic image and the reference image. Further, for example, the alignment of the retinal layer boundary in the tomographic image may be performed. Further, not only the translational misalignment but also the rotational misalignment may be corrected.

Next, the above-mentioned pair of input data and teacher data will be described with reference to FIGS. 4 to 6. FIG. 4 is an example of an image corresponding to the input data, and FIG. 4A shows an SLO fundus image taken by the SS-OCT apparatus shown in FIG. 401 represents a tomographic image imaging region (imaging position), and FIG. 4B represents a tomographic image in the imaging region 401. FIG. 5 shows an example of a teacher data image taken by an SD-OCT device different from the SS-OCT device that captured the input data, and FIGS. 5 (a), 5 (b), and 5 (c) show. The SLO fundus images obtained by moving the fixation lamp and dividing the images are shown, and 501 to 503 represent the imaging areas of the respective tomographic images. Further, FIGS. 5 (d), 5 (e), and 5 (f) represent tomographic images of sites corresponding to the respective SLO fundus images, and FIG. 5 (g) shows FIGS. 5 (d) and 5 (f). 5 (e) and FIG. 5 (f) are combined to obtain a high-quality image at the same position as in FIG. 4 (b). Note that FIG. 5 (g) schematically shows that the image quality is higher than that of the tomographic image of FIG.

In the case of this embodiment, it is desirable that the input data and the teacher data have the same eye and the same position. Therefore, even if the fixation and the like are set to the same position between different devices, the positions may not always be the same. Therefore, in order to create the pair image of the present embodiment, it is necessary to determine the imaging region by using, for example, the relative distance from the feature point of each SLO fundus image. As an example of the feature points, by using the papilla of 402 in FIG. 4A, the macula of 403, or the running state of the blood vessel, it is possible to photograph substantially the same position. FIG. 6 shows that the tomographic image 601 to the tomographic image 602 are obtained by, for example, enlarging and padding the tomographic image of FIG. 4 (b) and the tomographic image of FIG. 5 (g) so that the image sizes are the same. It shows that it can be obtained. As a result, the tomographic image 602 and the tomographic image of FIG. 5 (g) can be used as a pair image for learning.

When a tomographic image of the fundus is input to the trained model obtained by using such training data, a tomographic image of the fundus according to the design of the trained model is output. When the tomographic image 701 of the fundus is input using the trained model obtained by using the training data described above, the image quality improving unit 225 according to the present embodiment follows the tendency of training using the training data. A tomographic image 702 of the fundus can be output.

<Image processing flow>
Next, a series of image processing according to the present embodiment will be described with reference to the flow chart of FIG. FIG. 11 is a flow chart of a series of image processing according to the present embodiment. First, when a series of image processing according to the present embodiment is started, the processing proceeds to step S510.

In step S510, the second information acquisition unit 221 acquires an image captured by the photographing device 100 as an input image from the photographing device 100 connected via a circuit or a network. The second information acquisition unit 221 may acquire an input image in response to a request from the photographing device 100. Such a request is, for example, when the photographing device 100 generates an image, the saved image is displayed on the display unit 300 before or after the image generated by the photographing device 100 is stored in the recording device included in the photographing device 100. It may be issued when displaying on the image, when using a high-quality image for image analysis processing, or the like.

The second information acquisition unit 221 acquires data for generating an image from the photographing device 100, and the control device 200 or the image processing unit 220 acquires an image generated based on the data as an input image. May be good. In this case, any existing image generation method may be adopted as the image generation method for the control device 200 or the image processing unit 220 to generate various images.

In step S520, the second information acquisition unit 221 acquires a group of shooting conditions for the input image. Specifically, the shooting condition group saved in the data structure constituting the input image is acquired according to the data format of the input image. As described above, when the shooting conditions are not saved in the input image, the second information acquisition unit 221 obtains the shooting information group including the shooting condition group from the shooting device 100 or the image management system (not shown). Can be obtained.

In step S530, the image quality improvement possibility determination unit 224 determines whether or not the input image can be improved in image quality by the image quality improvement engine provided in the image quality improvement unit 225 using the acquired shooting condition group. To do. Specifically, the high image quality improvement possibility determination unit 224 determines whether or not the shooting portion, shooting method, shooting angle of view, and image size of the input image match the conditions that can be dealt with by the high image quality improvement engine. ..

When the image quality improvement possibility determination unit 224 determines all the shooting conditions and determines that it can be dealt with, the process proceeds to step S540. On the other hand, when the high image quality improvement possibility determination unit 224 determines that the high image quality improvement engine cannot deal with the input image based on these shooting conditions, the process proceeds to step S550.

Depending on the settings and mounting form of the control device 200 or the image processing unit 220, it is determined that the input image cannot be processed based on a part of the photographing portion, the photographing method, the photographing angle of view, and the image size. Even so, the high image quality processing in step S540 may be performed. For example, it is assumed that the high image quality engine can comprehensively handle any imaging part of the subject, and even if the input data contains an unknown imaging part, it seems to be able to deal with it. Such processing may be performed when it is implemented in. In addition, the image quality improvement possibility determination unit 224 is a condition that at least one of the image pickup portion, the image pickup method, the image angle of view, and the image size of the input image can be dealt with by the image quality improvement engine according to the desired configuration. It may be determined whether or not they match.

In step S540, the image quality enhancement unit 225 uses the image quality enhancement engine to improve the image quality of the input image and generate a high image quality image more suitable for image diagnosis than the input image. Specifically, the high image quality unit 225 inputs the input image to the high image quality engine and generates a high image quality image with high image quality. The high image quality engine uses a machine learning model that has been machine-learned using teacher data, and uses a camera that has a higher resolution than the resolution of the camera that obtained the input image from the input image. To generate a high-quality image like this. Therefore, the high image quality engine can generate a high image quality image suitable for image diagnosis rather than the input image.

Depending on the settings and mounting form of the control device 200 or the image processing unit 220, the image quality improving unit 225 inputs parameters together with the input image to the image quality improving engine according to the shooting condition group to improve the image quality. The degree and the like may be adjusted. Further, the image quality improving unit 225 may input parameters according to the input of the examiner into the image quality improving engine together with the input image to adjust the degree of image quality improving and the like.

In step S550, if the high-quality image is generated in step S540, the output unit 223 outputs the high-quality image and displays it on the display unit 300. On the other hand, if it is determined in step S530 that the high image quality processing is impossible, the input image is output and displayed on the display unit 300. Instead of displaying the output image on the display unit 300, the output unit 223 may display or store the output image on the photographing device 100 or another device. Further, the output unit 223 can process the output image so that it can be used by the photographing device 100 or another device, or transmit it to an image management system or the like, depending on the setting or mounting form of the control device 200 or the image processing unit 220. The data format may be converted as follows. Here, FIG. 8 shows a state in which the output result is displayed on the display unit 300. On the left side of FIG. 8, the SLO fundus image and the tomographic image captured by the SS-OCT apparatus shown in FIG. 1 are displayed, the center is the captured input tomographic image, and the right side is improved in image quality. The tomographic image is displayed.

As described above, the medical image processing apparatus according to the present embodiment includes a second information acquisition unit 221 and a high image quality improvement unit 225. The second information acquisition unit 221 acquires an input image (first image) which is an image of a predetermined portion of the subject. The high image quality improvement unit 225 generates a high image quality image (second image) from the input image by using the high image quality improvement engine including the machine learning engine. The high image quality engine includes a machine learning engine that uses an image obtained by using a photographing device having a resolution higher than that of the photographing device that obtained the input image as training data.

With this configuration, the medical image processing apparatus according to the present embodiment can output a high-quality image suitable for image diagnosis from the input image. For this reason, the medical image processing device makes images suitable for image diagnosis, such as clearer images and images in which the site or lesion to be observed is emphasized, more invasive to the photographer or the subject than in the past. It can be acquired at a lower price without increasing or increasing labor.

Further, the medical image processing device further includes a high image quality improvement possibility determination unit 224 for determining whether or not a high image quality image can be generated for the input image by using the high image quality improvement engine. The image quality improvement possibility determination unit 224 makes the determination based on at least one of the photographing portion, the photographing method, the photographing angle of view, and the image size of the input image.

With this configuration, the medical image processing apparatus according to the present embodiment can omit the input image that cannot be processed by the image quality improving unit 225 from the high image quality processing, and reduces the processing load and the occurrence of errors of the medical image processing apparatus. be able to.

In the present embodiment, the output unit 223 (display control unit) is configured to display the generated high-quality image on the display unit 300, but the operation of the output unit 223 is not limited to this. For example, the output unit 223 can output a high-quality image to another device connected to the photographing device 100, the control device 200, or the like. Therefore, the high-quality image can be displayed on the user interface of these devices, stored in an arbitrary recording device, used for arbitrary image analysis, or transmitted to an image management system.

In the present embodiment, the image quality improvement possibility determination unit 224 determines whether or not the input image can be improved by the image quality improvement engine, and if the input image can be improved in image quality, the image quality is improved. Part 225 improved the image quality. On the other hand, when the photographing device 100 takes a picture only under the shooting conditions capable of improving the image quality, the image obtained from the photographing device 100 may be unconditionally improved in image quality. In this case, as shown in FIG. 12, the processing of step S520 and step S530 can be omitted, and step S540 can be performed after step S510.

In this embodiment, the output unit 223 is configured to display a high-quality image on the display unit 300. However, the output unit 223 may display a high-quality image on the display unit 300 in response to an instruction from the examiner. For example, the output unit 223 may display a high-quality image on the display unit 300 in response to the examiner pressing an arbitrary button on the user interface of the display unit 300. In this case, the output unit 223 may switch to the input image to display the high-quality image, or may display the high-quality image side by side with the input image.

Further, the output unit 223 displays a high-quality image indicating that the displayed image is a high-quality image generated by processing using a machine learning algorithm when the display unit 300 displays the high-quality image. May be displayed with. In this case, the user can easily identify from the display that the displayed high-quality image is not the image itself acquired by shooting, so that misdiagnosis can be reduced or the diagnosis efficiency can be improved. Can be done. The display indicating that the image is a high-quality image generated by the process using the machine learning algorithm has any mode as long as the input image and the high-quality image generated by the process can be distinguished from each other. It may be one.

Further, the output unit 223 displays a display indicating what kind of teacher data the machine learning algorithm has learned for the display indicating that the image is a high-quality image generated by processing using the machine learning algorithm. May be displayed on the display unit 300. The display may include an explanation of the types of input data and output data of the teacher data, and an arbitrary display regarding the teacher data such as the imaging part included in the input data and the output data.

Although omitted in the description of the present embodiment, the high-quality image generated from the plurality of images used as the output data of the teacher data can be generated from the plurality of aligned images. As the alignment process, for example, one of a plurality of images is selected as a template, the similarity with other images is obtained while changing the position and angle of the template, and the amount of misalignment with the template is obtained. Each image may be corrected based on the amount of misalignment. Moreover, other existing arbitrary alignment processing may be performed.

When aligning a three-dimensional image, the three-dimensional image is decomposed into a plurality of two-dimensional images, and the aligned ones are integrated to perform the alignment of the three-dimensional image. May be good. Further, the two-dimensional image may be aligned by decomposing the two-dimensional image into a one-dimensional image and integrating the aligned images for each one-dimensional image. It should be noted that these alignments may be performed on the data for generating the image instead of the image.

Further, in the present embodiment, when the image quality improvement unit 224 determines that the input image can be dealt with by the image quality improvement unit 225, the process shifts to step S540, and the image quality improvement unit 225 improves the image quality. Processing has started. On the other hand, the output unit 223 may display the determination result by the image quality improvement possibility determination unit 224 on the display unit 300, and the image quality improvement unit 225 may start the image quality improvement process in response to an instruction from the examiner. .. At this time, the output unit 223 can display the input image and the imaging conditions such as the imaging portion acquired for the input image on the display unit 300 together with the determination result. In this case, since the high image quality processing is performed after the examiner determines whether or not the determination result is correct, it is possible to reduce the high image quality processing based on the erroneous determination.

Further, the output unit 223 displays the input image and the imaging conditions such as the imaged portion acquired for the input image on the display unit 300 without making a determination by the image quality improvement possibility determination unit 224, and the image quality improvement unit 225 is displayed by the examiner. The high image quality processing may be started according to the instruction of.

(Second Embodiment)
Regarding the pair image (pair of input data and teacher data) for generating the trained model used in the image quality improving unit 225 in the medical image processing apparatus according to the second embodiment, FIGS. 4, 5, and 9 are shown. It will be described using. The medical image processing apparatus according to the present embodiment is the same as the apparatus configuration described in the first embodiment.

The tomographic images of the input data of FIG. 4 (b) are shown in FIGS. 9 (b) and 9 (c) so as to correspond to the teacher data of FIGS. 5 (d), 5 (e), and 5 (f). ), Region division is performed as shown in FIG. 9 (d). Further, the plurality of tomographic images obtained by dividing the area are subjected to enlargement processing and padding processing as in the tomographic images of FIGS. 9 (e), 9 (f), and 9 (g). At this time, the plurality of tomographic images are processed so that the image sizes are the same. As a result, the image is divided, and a pair image for learning suitable for each shooting area can be generated.

In the case of this embodiment as well, it is desirable that the input data and the teacher data are at the same position of the same eye to be inspected, as in the first embodiment. Therefore, even if the fixation and the like are set to the same position between different devices, the positions may not always be the same. Therefore, in order to create the pair image of the present embodiment, it is necessary to determine the photographing area by using, for example, the relative distance from the feature point of each SLO image. As an example of the feature points, by using the papilla of 402 in FIG. 4A, the macula of 403, or the running state of the blood vessel, it is possible to photograph substantially the same position. The flow of image processing according to this embodiment is the same as that of the first embodiment.

(Modification example 1)
Next, the medical image processing apparatus according to the present modification will be described with reference to FIGS. 15 to 17. In the above-described embodiment, the high image quality improving unit 225 performs high image quality processing on the tomographic image using a learned model (high image quality high quality model) for high image quality. On the other hand, the high image quality unit 225 may perform high image quality processing on other images using the high image quality model, and the output unit 223 displays various high image quality images on the display unit 300. You may let me. For example, the high image quality improving unit 225 may perform high image quality processing on an En-Face image of brightness, an OCTA front image, or the like. Further, the output unit 223 can display at least one of the tomographic image, the brightness En-Face image, and the OCTA front image processed by the image quality improving unit 225 on the display unit 300. The image to be displayed with high image quality may be an SLO fundus image, a fundus image acquired by a fundus camera (not shown), a fluorescent fundus image, or the like.

Here, the learning data of the high image quality model for processing the high image quality of various images is the image before the high image quality processing for the various images, similarly to the learning data of the high image quality model according to the above-described embodiment. Is used as input data, and the image after high image quality processing is used as output data. Regarding the high image quality processing related to the training data, for example, the output data of the teacher data related to the high image quality model is higher than that of the OCT device when the image which is the input data is taken, as in the above-described embodiment. It may be an image taken by using a high-performance OCT device or an image taken by a high load setting.

Further, the high image quality model may be prepared for each type of image to be subjected to the high image quality processing. For example, a high image quality model for a tomographic image, a high image quality model for an En-Face image of brightness, and a high image quality model for an OCTA front image may be prepared. Further, the high image quality model for the En-Face image of brightness and the high image quality model for the OCTA front image are learned by comprehensively learning the images of different depth ranges with respect to the depth range (generation range) related to the image generation. It may be a finished model. Images of different depth ranges may include, for example, images of the surface layer (Im2910), the deep layer (Im2920), the outer layer (Im2930), and the choroidal vascular network (Im2940), as shown in FIG. 17A. .. Further, as the high image quality model for the En-Face image of brightness and the high image quality model for the OCTA front image, a plurality of high image quality models in which images for different depth ranges are learned may be prepared. The high image quality model that performs high image quality processing on images other than tomographic images is not limited to the high image quality model that performs different image processing for each area, and the same image processing is performed on the entire image. It may be a model.

Further, when preparing a high image quality model for a tomographic image, it may be a trained model in which the tomographic images obtained at positions in different sub-scanning directions (Y-axis directions) are comprehensively learned. The tomographic images Im2951 to Im2953 shown in FIG. 17B are examples of tomographic images obtained at positions in different sub-scanning directions. However, in the case of images taken at different imaging sites (for example, the center of the macula and the center of the optic nerve head), learning may be performed separately for each imaging site, and the imaging site may not be a concern. You may try to study together. The tomographic image with higher image quality may include a tomographic image of brightness and a tomographic image of motion contrast data. However, since the image feature amount is significantly different between the tomographic image of brightness and the tomographic image of motion contrast data, learning may be performed separately as a high image quality improvement model for each.

In this modified example, an example in which the output unit 223 displays the image obtained by the image quality improving unit 225 on the display unit 300 will be described. In this modified example, the description will be given with reference to FIGS. 15 (a) and 15 (b), but the display screen is not limited to this. Similarly, the high image quality processing can be applied to a display screen that displays a plurality of images obtained at different dates and times side by side, such as follow-up observation. Similarly, the high image quality processing can be applied to a display screen such as a shooting confirmation screen in which the examiner confirms the success or failure of shooting immediately after shooting. The output unit 223 can display a plurality of high-quality images generated by the high-quality image unit 225 and low-quality images that have not been high-quality on the display unit 300. Further, the output unit 223 selects a low-quality image and a high-quality image selected according to the instruction of the examiner with respect to the plurality of high-quality images displayed on the display unit 300 and the low-quality images that have not been improved in image quality. Each can be displayed on the display unit 300. In addition, the medical image processing device can also output a low-quality image and a high-quality image selected according to the instruction of the examiner to the outside.

Hereinafter, with reference to FIGS. 15A and 15B, an example of the table z screen 3400 of the interface according to this modification is shown. 3400 represents the entire screen, 3401 represents the patient tab, 3402 represents the imaging tab, 3403 represents the report tab, 3404 represents the settings tab, and diagonal lines in the report tab of 3403 represent the active state of the report screen. In this modified example, an example of displaying a report screen will be described. Im3405 is an SLO image, and Im3406 is an OCTA En-Face image shown in Im3407, which is superimposed and displayed on the SLO image Im3405. Here, the SLO image is a frontal image of the fundus acquired by an SLO (Scanning Laser Ophthalmoscope) optical system. Im3407 and Im3408 show En-Face images of OCTA, Im3409 show En-Face images of brightness, and Im3411 and Im3412 show tomographic images. In 3413 and 3414, the boundary line of the upper and lower ranges of the En-Face image of OCTA shown in Im3407 and Im3408, respectively, is superimposed and displayed on the tomographic image. The button 3420 is a button for designating the execution of the high image quality processing. Of course, as will be described later, the button 3420 may be a button for instructing the display of a high-quality image.

In this modified example, the high image quality processing is executed by designating the button 3420, or whether or not the processing is executed is determined based on the information stored (stored) in the database. First, an example of switching between the display of a high-quality image and the display of a low-quality image by designating the button 3420 according to an instruction from the examiner will be described. The target image for the high image quality processing will be described as an OCTA En-Face image. When the examiner specifies the report tab 3403 and transitions to the report screen, the low-quality OCTA En-Face images Im3407 and Im3408 are displayed. After that, when the examiner specifies the button 3420, the image quality improving unit 225 executes the image quality improving process on the images Im3407 and Im3408 displayed on the screen. After the high image quality processing is completed, the output unit 223 displays the high image quality image generated by the high image quality improvement unit 225 on the report screen. Since Im3406 superimposes Im3407 on the SLO image Im3405, Im3406 also displays an image that has been subjected to high image quality processing. Then, the display of the button 3420 is changed to the active state, and a display indicating that the high image quality processing has been executed is displayed. Here, the execution of the process in the image quality improving unit 225 does not have to be limited to the timing when the examiner specifies the button 3420. Since the types of OCTA En-Face images Im3407 and Im3408 to be displayed when the report screen is opened are known in advance, high image quality processing may be executed when transitioning to the report screen. Then, the output unit 223 may display a high-quality image on the report screen at the timing when the button 3420 is pressed. Further, it is not necessary that there are two types of images for which high image quality processing is performed in response to an instruction from the examiner or when transitioning to the report screen. For images that are likely to be displayed, for example, En-Face images of multiple OCTAs such as the surface layer (Im2910), the deep layer (Im2920), the outer layer (Im2930), and the choroidal vascular network (Im2940) as shown in FIG. May be processed. In this case, the image obtained by the high image quality processing may be temporarily stored in the memory or stored in the database.

Next, a case where the high image quality processing is executed based on the information stored (stored) in the database will be described. When the state in which the high image quality processing is executed is saved in the database, the high quality image obtained by executing the high image quality processing is displayed by default when the report screen is displayed. Then, by displaying the button 3420 as the active state by default, it is possible to configure the examiner to know that the high-quality image obtained by executing the high-quality processing is displayed. .. When the examiner wants to display the low-quality image before the high-quality processing, the examiner can display the low-quality image by designating the button 3420 and canceling the active state. If the examiner wants to return to a high-quality image, the examiner specifies the button 3420. Whether or not to execute the high image quality processing in the database shall be specified for each layer, such as common to all the data stored in the database and for each shooting data (for each inspection). For example, when the state in which the high image quality processing is executed is saved for the entire database, the state in which the examiner does not execute the high image quality processing is saved for the individual shooting data (individual inspection). In this case, when the shooting data is displayed next time, it is displayed without executing the high image quality processing. A user interface (for example, a save button) (not shown) may be used to save the execution state of the high image quality processing for each shooting data (for each inspection). In addition, when transitioning to other imaging data (other examinations) or other patient data (for example, changing to a display screen other than the report screen in response to an instruction from the examiner), the display state (for example, button 3420) is displayed. The state in which the high image quality processing is executed may be saved based on the state). As a result, if it is not specified whether or not to execute the high image quality processing in the shooting data unit (inspection unit), the processing is performed based on the information specified for the entire database, and the shooting data unit (inspection unit) is used. If specified, the process can be executed individually based on the information.

In this modification, an example of displaying Im3407 and Im3408 as the En-Face image of OCTA is shown, but the En-Face image of OCTA to be displayed can be changed by the specification of the examiner. Therefore, the change of the image when the execution of the high image quality processing is specified (the button 3420 is in the active state) will be described.

The image to be displayed is changed by using a user interface (for example, a combo box) (not shown). For example, when the examiner changes the type of image from the surface layer to the choroidal vascular network, the high image quality improving unit 225 executes the high image quality processing on the choroidal vascular network image, and the output unit 223 has the high image quality improving unit 225. Display the generated high-quality image on the report screen. That is, the output unit 223 displays the high-quality image in the first depth range according to the instruction from the examiner, and displays the high-quality image in the second depth range that is at least partially different from the first depth range. You may change to the display of. At this time, the output unit 223 displays a high-quality image in the first depth range by changing the first depth range to the second depth range in response to an instruction from the examiner. You may change to display a high-quality image in the depth range. As described above, for an image that is likely to be displayed at the time of transition of the report screen, if a high-quality image has already been generated, the output unit 223 may display the generated high-quality image. The method of changing the image type is not limited to the above, and it is also possible to generate an En-Face image of OCTA in which different depth ranges are set by changing the reference layer and the offset value. In that case, when the reference layer or the offset value is changed, the high image quality improving unit 225 executes the high image quality processing on the En-Face image of an arbitrary OCTA, and the output unit 223 performs the high image quality image. Is displayed on the report screen. The reference layer and the offset value can be changed by using a user interface (for example, a combo box or a text box) (not shown). Further, by dragging (moving the layer boundary) any of the boundary lines 3413 and 3414 superimposed on the tomographic images Im3411 and Im3412, the generation range of the En-Face image of OCTA can be changed. When the boundary line is changed by dragging, the execution command of the high image quality processing is continuously executed. Therefore, the image quality improving unit 225 may always process the execution instruction, or may execute it after changing the layer boundary by dragging. Alternatively, although the execution of the high image quality processing is continuously instructed, the previous instruction may be canceled and the latest instruction may be executed when the next instruction arrives. It should be noted that the high image quality processing may take a relatively long time. Therefore, no matter what timing the command is executed as described above, it may take a relatively long time for the high-quality image to be displayed. Therefore, from the time when the depth range for generating the En-Face image of OCTA is set according to the instruction from the examiner until the high-quality image is displayed, the OCTA corresponding to the set depth range is displayed. En-Face image (low image quality image) may be displayed. That is, when the depth range is set, an OCTA En-Face image (low-quality image) corresponding to the set depth range is displayed, and when the high-quality processing is completed, the OCTA En-Face image is displayed. The display of (the low-quality image) may be configured to be changed to the display of a high-quality image. In addition, information indicating that the high image quality processing is being executed may be displayed from the time when the depth range is set until the high image quality image is displayed. It should be noted that these are not only in the case where the execution of the high image quality processing has already been specified (the button 3420 is in the active state), but also, for example, the execution of the high image quality processing in response to an instruction from the examiner. Can be applied even until a high-quality image is displayed when is instructed.

In this modification, different layers are displayed on Im3407 and Im3408 as En-Face images of OCTA, and low-quality and high-quality images are switched and displayed, but the present invention is not limited to this. For example, the Im3407 may display a low-quality OCTA En-Face image, and the Im3408 may display a high-quality OCTA En-Face image side by side. When the images are switched and displayed, the images are switched at the same place, so that it is easy to compare the parts where there is a change.

Next, the execution of the high image quality processing in the screen transition will be described with reference to FIGS. 15A and 15B. FIG. 15A is an enlarged screen example of the OCTA image in FIG. 15B. Also in FIG. 15 (a), the button 3420 is displayed in the same manner as in FIG. 15 (b). The screen transition from FIG. 15 (b) to FIG. 15 (a) is made by, for example, double-clicking the OCTA image, and the transition from FIG. 15 (a) to FIG. 15 (b) is performed by the close button 3430. The screen transition is not limited to the method shown here, and a user interface (not shown) may be used.

When execution of high image quality processing is specified at the time of screen transition (button 3420 is active), that state is maintained even at the time of screen transition. That is, when transitioning to the screen of FIG. 15 (a) while displaying the high-quality image on the screen of FIG. 15 (b), the high-quality image is also displayed on the screen of FIG. 15 (a). Then, the button 3420 is activated. The same applies to the transition from FIG. 15 (a) to FIG. 15 (b). In FIG. 15A, the button 3420 can be specified to switch the display to a low-quality image.

Regarding the screen transition, not only the screen shown here, but also the display state of the high-quality image was maintained if the transition was to a screen displaying the same shooting data such as a display screen for follow-up observation or a display screen for panorama. Make the transition as it is. That is, on the display screen after the transition, an image corresponding to the state of the button 3420 on the display screen before the transition is displayed. For example, if the button 3420 on the display screen before the transition is in the active state, a high-quality image is displayed on the display screen after the transition. Further, for example, if the active state of the button 3420 on the display screen before the transition is released, a low-quality image is displayed on the display screen after the transition. When the button 3420 on the follow-up display screen becomes active, a plurality of images obtained at different dates and times (different inspection dates) displayed side by side on the follow-up display screen are switched to high-quality images. You may. That is, when the button 3420 on the display screen for follow-up observation becomes active, it may be configured to be collectively reflected on a plurality of images obtained at different dates and times.

Here, an example of a display screen for follow-up observation is shown in FIG. When tab 3801 is selected in response to an instruction from the examiner, a display screen for follow-up observation is displayed as shown in FIG. At this time, the depth range of the measurement target area can be changed by the examiner selecting from the default depth range sets (3802 and 3803) displayed in the list box. For example, in the list box 3802, the surface layer of the retina is selected, and in the list box 3803, the deep layer of the retina is selected. The analysis result of the motion contrast image of the surface layer of the retina is displayed in the upper display area, and the analysis result of the motion contrast image of the deep layer of the retina is displayed in the lower display area. That is, when the depth range is selected, the analysis results of the plurality of motion contrast images in the selected depth range are collectively displayed in parallel for the plurality of images having different dates and times.

At this time, if the display of the analysis result is set to the non-selected state, the display may be changed to the parallel display of a plurality of motion contrast images having different dates and times at once. Then, when the button 3420 is specified in response to the instruction from the examiner, the display of the plurality of motion contrast images is changed to the display of the plurality of high-quality images at once.

Further, when the display of the analysis result is in the selected state, when the button 3420 is specified in response to the instruction from the examiner, the display of the analysis result of the plurality of motion contrast images is the analysis result of the plurality of high-quality images. It is changed to the display of. Here, the analysis result may be displayed by superimposing the analysis result on the image with arbitrary transparency. At this time, the change to the display of the analysis result may be changed to a state in which the analysis result is superimposed on the displayed image with arbitrary transparency, for example. Further, the change to the display of the analysis result may be, for example, a change to the display of an image (for example, a two-dimensional map) obtained by blending the analysis result and the image with an arbitrary transparency.

In addition, the type of layer boundary and the offset position used to specify the depth range can be collectively changed from a user interface such as 3805 and 3806, respectively. By displaying the tomographic image together and moving the layer boundary data superimposed on the tomographic image according to the instruction from the examiner, the depth range of multiple motion contrast images at different dates and times can be changed at once. May be done. At this time, when a plurality of tomographic images having different dates and times are displayed side by side and the above movement is performed on one tomographic image, the layer boundary data may be similarly moved on another tomographic image. Further, the presence / absence of the image projection method or the projection artifact suppression process may be changed by selecting from a user interface such as a context menu. Further, the selection button 3807 may be selected to display the selection screen, and the image selected from the image list displayed on the selection screen may be displayed. The arrow 3804 displayed at the upper part of FIG. 16 is a mark indicating that the inspection is currently selected, and the reference inspection (Baseline) is the inspection selected at the time of Follow-up imaging (one of FIG. 16). The image on the far left). Of course, a mark indicating a reference inspection may be displayed on the display unit 300.

When the "Show Difference" check box 3808 is specified, the measured value distribution (map or sector map) with respect to the reference image is displayed on the reference image. Further, in this case, a difference measurement value map between the measurement value distribution calculated for the reference image and the measurement distribution calculated for the image displayed in the area is displayed in the area corresponding to the other inspection dates. To do. As the measurement result, a trend graph (a graph of the measured values for the image of each inspection day obtained by the time-dependent change measurement) may be displayed on the report screen. That is, time-series data (for example, a time-series graph) of a plurality of analysis results corresponding to a plurality of images having different dates and times may be displayed. At this time, the analysis results related to the date and time other than the plurality of date and time corresponding to the plurality of displayed images can be discriminated from the plurality of analysis results corresponding to the plurality of displayed images (for example, time series). The color of each point on the graph differs depending on whether or not the image is displayed). It may be displayed as time series data. Further, the regression line (curve) of the trend graph and the corresponding mathematical formula may be displayed on the report screen.

In this modification, the motion contrast image has been described, but the present invention is not limited to this. The image related to processing such as display, high image quality, and image analysis according to this modification may be a tomographic image. Further, not only a tomographic image but also a different image such as an SLO image, a fundus photograph, or a fluorescent fundus photograph may be used. In that case, the user interface for executing the high image quality processing is one that instructs the execution of the high image quality processing for a plurality of different types of images, and an arbitrary image is selected from a plurality of different types of images. There may be something that instructs the execution of the high image quality processing.

For example, when displaying a tomographic image obtained by B-scan with high image quality, the tomographic images Im3411 and Im3412 shown in FIG. 15A may be displayed with high image quality. Further, a high-quality tomographic image may be displayed in the area where the OCTA front images Im3407 and Im3408 are displayed. Further, the tomographic image by B-scan may be not only a tomographic image of brightness but also a tomographic image by B-scan obtained by using motion contrast data. It should be noted that only one tomographic image may be displayed or a plurality of tomographic images may be displayed with high image quality. When only one is displayed, the tomographic image obtained by, for example, a circle scan may be displayed with high image quality. Further, when a plurality of tomographic images are displayed, the tomographic images acquired at positions in different sub-scanning directions may be displayed, and the plurality of tomographic images obtained by, for example, cross-scanning may be displayed in high image quality. When the images are displayed in different directions, images in different scanning directions may be displayed. Since the image features of a plurality of tomographic images obtained by cross-scanning or the like are often similar, for example, using a common trained model obtained by learning these tomographic images as training data, each of them is used. The image quality in the scanning direction may be improved. Further, when displaying a plurality of tomographic images obtained by, for example, a radial scan with high image quality, a plurality of partially selected tomographic images (for example, two tomographic images at positions symmetrical with respect to a reference line) are displayed. ) May be displayed respectively. Further, a plurality of tomographic images are displayed on a display screen for follow-up observation as shown in FIG. 16, and instructions for improving the image quality and analysis results (for example, the thickness of a specific layer, etc.) are obtained by the same method as the above method. ) May be displayed. Further, the tomographic image may be subjected to high image quality processing based on the information stored in the database by the same method as the above method.

Similarly, when displaying the SLO fundus image with high image quality, for example, the SLO fundus image Im3405 may be displayed with high image quality. Further, when the brightness En-Face image is displayed with high image quality, for example, the brightness En-Face image Im3409 may be displayed with high image quality. Further, a plurality of SLO fundus images and En-Face images having brightness are displayed on a follow-up display screen as shown in FIG. 16, and instructions and analysis results for improving the image quality (for example, by the same method as the above method). , The thickness of a specific layer, etc.) may be displayed. Further, high image quality processing may be executed on the SLO fundus image and the En-Face image of brightness based on the information stored in the database by the same method as the above method. The display of the tomographic image, the SLO fundus image, and the brightness En-Face image is an example, and these images may be displayed in any manner depending on the desired configuration. Further, at least two or more of the OCTA front image, the tomographic image, the SLO fundus image, and the brightness En-Face image may be displayed in high quality with a single instruction. With such a configuration, the output unit 223 can display the image processed by the image quality improving unit 225 according to the present modification on the display unit 300. At this time, as described above, when at least one of a plurality of conditions relating to the display of the high-quality image, the display of the analysis result, the depth range of the displayed front image, and the like is selected, the display screen is displayed. Even if it is transitioned, the selected state may be maintained.

Further, as described above, when at least one of the plurality of conditions is in the selected state, the selected state of at least one is maintained even if the other conditions are changed to the selected state. You may. For example, when the display of the analysis result is in the selected state, the output unit 223 displays the analysis result of the low-quality image in high quality according to the instruction from the examiner (for example, when the button 3420 is specified). You may change to display the analysis result of the image. Further, when the display of the analysis result is in the selected state, the output unit 223 displays the analysis result of the high-quality image according to the instruction from the examiner (for example, when the designation of the button 3420 is canceled). The display may be changed to display the analysis result of a low-quality image.

Further, the output unit 223 analyzes the low-quality image according to the instruction from the examiner (for example, when the designation of displaying the analysis result is canceled) when the display of the high-quality image is not selected. The display of the result may be changed to the display of a low-quality image. Further, when the display of the high-quality image is not selected, the output unit 223 reduces the display of the low-quality image according to the instruction from the examiner (for example, when the display of the analysis result is specified). You may change to display the analysis result of the image quality image. Further, the output unit 223 responds to an instruction from the examiner (for example, when the designation of displaying the analysis result is canceled) when the display of the high-quality image is selected, and the analysis result of the high-quality image is released. The display of may be changed to the display of a high-quality image. Further, when the display of the high-quality image is selected, the output unit 223 displays the high-quality image with high image quality in response to an instruction from the examiner (for example, when the display of the analysis result is specified). You may change to display the analysis result of the image.

Further, consider the case where the display of the high-quality image is in the non-selected state and the display of the analysis result of the first type is in the selected state. In this case, the output unit 223 displays the analysis result of the first type of the low-quality image according to the instruction from the examiner (for example, when the display of the analysis result of the second type is specified). May be changed to display the analysis result of the second type of the low image quality image. Further, consider the case where the display of the high-quality image is in the selected state and the display of the analysis result of the first type is in the selected state. In this case, the output unit 223 displays the analysis result of the first type of the high-quality image according to the instruction from the examiner (for example, when the display of the analysis result of the second type is specified). May be changed to display the analysis result of the second type of the high-quality image.

As described above, the display screen for follow-up observation may be configured so that these display changes are collectively reflected on a plurality of images obtained at different dates and times. Here, the analysis result may be displayed by superimposing the analysis result on the image with arbitrary transparency. At this time, the change to the display of the analysis result may be changed to a state in which the analysis result is superimposed on the displayed image with arbitrary transparency, for example. Further, the change to the display of the analysis result may be, for example, a change to the display of an image (for example, a two-dimensional map) obtained by blending the analysis result and the image with an arbitrary transparency.

In this modified example, the image quality improving unit 225 generated a high image quality image in which the image quality of the tomographic image was improved by using the image quality improving model. However, the component that generates a high-quality image using the high-quality model is not limited to the high-quality unit 225. For example, a second image quality improving unit different from the image quality improving unit 225 may be provided, and the second image quality improving unit may generate a high image quality image using the image quality improving model. In this case, the second high-quality image unit is not a high-quality image in which different image processing is performed for each region using the trained model, but a high-quality image in which the same image processing is performed on the entire image. May be generated. At this time, the output data of the trained model may be an image in which the same high image quality processing has been performed on the entire image. The high image quality model used by the second high image quality unit and the second high image quality unit may be composed of a software module or the like executed by a processor such as a CPU, MPU, GPU, or FPGA. It may be configured by a circuit or the like that performs a specific function such as an ASIC.

(Modification 2)
In the above-described modification, the output unit 223 may display the high-quality image and the input image generated by the high-quality image-enhancing unit 225 on the display unit 300 in response to an instruction from the examiner. it can. Further, the output unit 223 may switch the display on the display unit 300 from a captured image (input image) to a high-quality image in response to an instruction from the examiner. That is, the output unit 223 may change the display of the low-quality image to the display of the high-quality image in response to an instruction from the examiner. Further, the output unit 223 may change the display of the high-quality image to the display of the low-quality image in response to an instruction from the examiner.

Further, the high image quality unit 225 in the control device 200 (or the image processing unit 220) starts the high image quality processing by the high image quality engine (learned model for high image quality) (images to the high image quality engine). (Input) may be executed in response to an instruction from the examiner, and the output unit 223 may display the high-quality image generated by the high-quality image-enhancing unit 225 on the display unit 300. On the other hand, when the input image is taken by the photographing device 100, the high image quality engine automatically generates a high image quality image based on the input image, and the output unit 223 is set to be high according to the instruction from the examiner. The image quality image may be displayed on the display unit 300. Here, the high image quality engine includes a trained model that performs the high image quality processing described above.

Note that these processes can be performed in the same manner for the output of the analysis result. That is, the output unit 223 may change the display of the analysis result of the low-quality image to the display of the analysis result of the high-quality image in response to the instruction from the examiner. Further, the output unit 223 may change the display of the analysis result of the high-quality image to the display of the analysis result of the low-quality image in response to the instruction from the examiner. Of course, the output unit 223 may change the display of the analysis result of the low-quality image to the display of the low-quality image in response to the instruction from the examiner. Further, the output unit 223 may change the display of the low-quality image to the display of the analysis result of the low-quality image in response to an instruction from the examiner. Further, the output unit 223 may change the display of the analysis result of the high-quality image to the display of the high-quality image in response to an instruction from the examiner. Further, the output unit 223 may change the display of the high-quality image to the display of the analysis result of the high-quality image in response to an instruction from the examiner.

Further, the output unit 223 may change the display of the analysis result of the low-quality image to the display of the analysis result of another type of the low-quality image in response to the instruction from the examiner. Further, the output unit 223 may change the display of the analysis result of the high-quality image to the display of the analysis result of another type of the high-quality image in response to the instruction from the examiner.

Here, the display of the analysis result of the high-quality image may be a display in which the analysis result of the high-quality image is superimposed and displayed on the high-quality image with arbitrary transparency. Further, the display of the analysis result of the low-quality image may be a display in which the analysis result of the low-quality image is superimposed and displayed on the low-quality image with arbitrary transparency. At this time, the change to the display of the analysis result may be changed to a state in which the analysis result is superimposed on the displayed image with arbitrary transparency, for example. Further, the change to the display of the analysis result may be, for example, a change to the display of an image (for example, a two-dimensional map) obtained by blending the analysis result and the image with an arbitrary transparency.

In this modified example, the image quality improving unit 225 generated a high image quality image in which the image quality of the tomographic image was improved by using the image quality improving model. However, the component that generates a high-quality image using the high-quality model is not limited to the high-quality unit 225. For example, a second image quality improving unit different from the image quality improving unit 225 may be provided, and the second image quality improving unit may generate a high image quality image using the image quality improving model. In this case, the second high-quality image unit is not a high-quality image in which different image processing is performed for each region using the trained model, but a high-quality image in which the same image processing is performed on the entire image. May be generated. At this time, the output data of the trained model may be an image in which the same high image quality processing has been performed on the entire image. The high image quality model used by the second high image quality unit and the second high image quality unit may be composed of a software module or the like executed by a processor such as a CPU, MPU, GPU, or FPGA. It may be configured by a circuit or the like that performs a specific function such as an ASIC.

(Modification example 3)
The output unit 223 in the various embodiments and modifications described above may display analysis results such as a desired layer thickness and various blood vessel densities on the report screen of the display screen. In addition, the optic nerve head, macular region, vascular region, nerve fiber bundle, vitreous region, macular region, choroidal region, scleral region, lamina cribrosa region, retinal layer boundary, retinal layer boundary edge, photoreceptor cells, blood cells, The value (distribution) of the parameter relating to the site of interest including at least one such as the vascular wall, the vascular inner wall boundary, the vascular lateral boundary, the ganglion cell, the corneal region, the corner region, and Schlemm's canal may be displayed as the analysis result. At this time, for example, by analyzing the medical image to which the reduction processing of various artifacts is applied, it is possible to display the analysis result with high accuracy. The artifacts are, for example, a false image region generated by light absorption by a blood vessel region or the like, a projection artifact, a band-shaped artifact in a front image generated in the main scanning direction of the measured light depending on the state of the eye to be inspected (movement, blinking, etc.). You may. Further, the artifact may be any image loss region as long as it is randomly generated for each image taken on a medical image of a predetermined portion of the subject, for example. In addition, the value (distribution) of the parameter relating to the region including at least one of various artifacts (copy loss region) as described above may be displayed as the analysis result. In addition, the value (distribution) of the parameter relating to the region including at least one such as an abnormal site such as drusen, new blood vessel, vitiligo (hard vitiligo), and pseudo-drusen may be displayed as an analysis result. Further, the comparison result obtained by comparing the standard value or standard range obtained by using the standard database with the analysis result may be displayed.

Further, the analysis result may be displayed in an analysis map, a sector showing statistical values corresponding to each divided area, or the like. The analysis result may be generated by using a trained model (analysis result generation engine, trained model for analysis result generation) obtained by learning the analysis result of the medical image as training data. .. At this time, the trained model is trained using training data including a medical image and an analysis result of the medical image, training data including a medical image and an analysis result of a medical image of a type different from the medical image, and the like. It may be obtained by. Further, the trained model is obtained by training using training data including input data in which a plurality of medical images of different types of predetermined parts are set, such as a luminance front image and a motion contrast front image. May be good. Here, the brightness front image corresponds to the brightness En-Face image, and the motion contrast front image corresponds to the OCTA En-Face image. Further, the analysis result obtained by using the high-quality image generated by the high-quality engine may be displayed.

Further, the input data included in the learning data may be a high-quality image generated by the high-quality image engine, or may be a set of a low-quality image and a high-quality image. Further, the training data includes, for example, at least analysis values (for example, average value, median value, etc.) obtained by analyzing the analysis area, a table including the analysis values, an analysis map, the position of the analysis area such as a sector in the image, and the like. The information including one may be the data labeled (annotated) with the input data as the correct answer data (for supervised learning). In addition, according to the instruction from the examiner, the analysis result obtained by the trained model for generating the analysis result may be displayed. For example, a medical image processing apparatus uses a trained model for generating analysis results (different from a trained model for high image quality) to generate image analysis results related to the medical image from various medical images. be able to. Further, for example, the output unit 223 can display the image analysis result obtained by using the trained model for generating the analysis result from various medical images on the display unit 300.

In addition, the output unit 223 in the various embodiments and modifications described above may display various diagnostic results such as glaucoma and age-related macular degeneration on the report screen of the display screen. At this time, for example, by analyzing the medical image to which the various artifact reduction processes as described above are applied, it is possible to display an accurate diagnostic result. Further, in the diagnosis result, the position of the specified abnormal portion may be displayed on the image, or the state of the abnormal portion may be displayed by characters or the like. Further, the classification result of the abnormal part or the like (for example, Curtin classification) may be displayed as the diagnosis result. Further, as the classification result, for example, information indicating the certainty of each abnormal part (for example, a numerical value indicating the ratio) may be displayed. In addition, information necessary for the doctor to confirm the diagnosis may be displayed as a diagnosis result. As the necessary information, for example, advice such as additional shooting can be considered. For example, when an abnormal site is detected in the blood vessel region in the OCTA image, it may be displayed that fluorescence imaging using a contrast medium capable of observing the blood vessel in more detail than OCTA is performed. In addition, the diagnosis result may be information on the future medical treatment policy of the subject. In addition, the diagnosis result is, for example, the diagnosis name, the type and state (degree) of the lesion (abnormal site), the position of the lesion in the image, the position of the lesion with respect to the region of interest, the findings (interpretation findings, etc.), and the basis of the diagnosis name (affirmation). Medical support information, etc.), grounds for denying the diagnosis name (negative medical support information, etc.), etc. may be included in the information. At this time, for example, a diagnosis result that is more probable than the diagnosis result such as the diagnosis name input in response to the instruction from the examiner may be displayed as medical support information. Further, when a plurality of types of medical images are used, for example, the types of medical images that can be the basis of the diagnosis result may be displayed in an identifiable manner.

The diagnosis result may be generated by using a trained model (diagnosis result generation engine, trained model for generation of diagnosis result) obtained by learning the diagnosis result of the medical image as training data. .. At this time, the trained model is trained using training data including a medical image and a diagnosis result of the medical image, training data including a medical image and a diagnosis result of a medical image of a type different from the medical image, and the like. It may be obtained by. Further, the diagnosis result obtained by using the high-quality image generated by the high-quality engine may be displayed. For example, a medical image processing apparatus uses a trained model for generating diagnostic results (different from a trained model for high image quality) to generate diagnostic results related to the medical image from various medical images. Can be done. Further, for example, the output unit 223 can display the diagnosis result obtained by using the learned model for generating the diagnosis result from various medical images on the display unit 300.

Further, the input data included in the learning data may be a high-quality image generated by the high-quality image engine, or may be a set of a low-quality image and a high-quality image. In addition, the learning data includes, for example, the diagnosis name, the type and state (degree) of the lesion (abnormal site), the position of the lesion in the image, the position of the lesion with respect to the region of interest, the findings (interpretation findings, etc.), and the basis of the diagnosis name (affirmation). Information including at least one such as (general medical support information, etc.) and grounds for denying the diagnosis name (negative medical support information), etc. are labeled (annotated) in the input data as correct answer data (for supervised learning). It may be data. In addition, according to the instruction from the examiner, the diagnosis result obtained by the trained model for generating the diagnosis result may be displayed.

Further, the output unit 223 in the various embodiments and modifications described above can be used to display an object recognition result (object detection result) of a partial region such as a region of interest, an artifact region, or an abnormal region as described above on the report screen of the display screen. The segmentation result may be displayed. At this time, for example, a rectangular frame or the like may be superimposed and displayed around the object on the image. Further, for example, colors and the like may be superimposed and displayed on the object in the image. The object recognition result and the segmentation result are learned models (object recognition engine, for object recognition) obtained by learning the learning data in which the information indicating the object recognition and the segmentation is used as the correct answer data and labeled (annotated) on the medical image. It may be generated using a trained model, a segmentation engine, a trained model for segmentation). For example, a medical image processing device may use a trained model for segmentation or object recognition (different from the trained model for high image quality) to perform segmentation results or objects related to the medical image from various medical images. The recognition result can be generated. Further, for example, the output unit 223 can display the segmentation result or the object recognition result obtained from various medical images by using the trained model for segmentation or object recognition on the display unit 300. The above-mentioned analysis result generation and diagnosis result generation may be obtained by using the above-mentioned object recognition result and segmentation result. For example, analysis result generation or diagnosis result generation processing may be performed on a region of interest obtained by object recognition or segmentation processing.

In addition, when detecting an abnormal site, the medical image processing apparatus may use a hostile generation network (GAN: Generative Adversarial Networks) or a variational autoencoder (VAE: Variational Auto-Encoder). For example, a DCGAN (Deep Convolutional GAN) consisting of a generator obtained by learning the generation of a tomographic image and a discriminator obtained by learning the discrimination between a new tomographic image generated by the generator and a real frontal image of the fundus of the eye. ) Can be used as a machine learning model.

When DCGAN is used, for example, the discriminator encodes the input tomographic image into a latent variable, and the generator generates a new tomographic image based on the latent variable. After that, the difference between the input tomographic image and the generated new tomographic image can be extracted as an abnormal part. When VAE is used, for example, the input tomographic image is encoded by an encoder to be a latent variable, and the latent variable is decoded by a decoder to generate a new tomographic image. After that, the difference between the input tomographic image and the generated new tomographic image can be extracted as an abnormal part. Although a tomographic image has been described as an example of the input data, a fundus image, a frontal image of the anterior eye, or the like may be used.

Further, the medical image processing apparatus may detect an abnormal site by using a convolutional autoencoder (CAE). When CAE is used, the same image is learned as input data and output data at the time of learning. As a result, when an image with an abnormal part is input to CAE at the time of estimation, an image without an abnormal part is output according to the learning tendency. After that, the difference between the image input to the CAE and the image output from the CAE can be extracted as an abnormal portion. In this case as well, not only the tomographic image but also the fundus image, the frontal image of the anterior eye, and the like may be used as input data.

In these cases, the medical image processing apparatus obtains information on the difference between the medical image obtained by using the hostile generation network or the autoencoder and the medical image input to the hostile generation network or the autoencoder, and information on the abnormal part. Can be generated as. As a result, the medical image processing apparatus can be expected to detect abnormal parts at high speed and with high accuracy. Here, the autoencoder includes VAE, CAE, and the like. For example, a medical image processing apparatus obtains information on a difference between a medical image obtained from various medical images using a hostile generation network or an autoencoder and a medical image input to the hostile generation network or the autoencoder. , Can be generated as information about abnormal parts. Further, for example, the output unit 223 provides information on the difference between the medical image obtained from various medical images by using the hostile generation network or the autoencoder and the medical image input to the hostile generation network or the autoencoder. Can be displayed on the display unit 300 as information regarding the abnormal portion.

Further, in the diseased eye, the image features differ depending on the type of disease. Therefore, the trained models used in the various examples and modifications described above may be generated and prepared for each type of disease or for each abnormal site. In this case, for example, the medical image processing apparatus can select a trained model to be used for processing according to an input (instruction) of the type of disease of the eye to be inspected, an abnormal part, or the like from the operator. The trained model prepared for each type of disease or abnormal site is not limited to the trained model used for detecting the retinal layer and generating a region label image, for example, for an engine for image evaluation or for analysis. It may be a trained model used in the engine of the above. At this time, the medical image processing apparatus may identify the type of disease or abnormal site of the eye to be inspected from the image by using a separately prepared learned model. In this case, the medical image processing apparatus may automatically select the trained model to be used for the above processing based on the type of disease or abnormal site identified by using the trained model prepared separately. it can. The trained model for identifying the disease type and abnormal site of the eye to be inspected is the training data in which the tomographic image, the fundus image, etc. are input data, and the disease type and the abnormal site in these images are output data. Learning may be performed using pairs. Here, as the input data of the training data, a tomographic image, a fundus image, or the like may be used alone as input data, or a combination thereof may be used as input data.

Further, in particular, the trained model for generating the diagnosis result may be a trained model obtained by learning from the training data including the input data including a set of a plurality of medical images of different types of the predetermined part of the subject. Good. At this time, as the input data included in the training data, for example, data in which a motion contrast front image of the fundus and a luminance front image (or a luminance tom image) are set can be considered. Further, as the input data included in the training data, for example, data in which a tomographic image (B scan image) of the fundus and a color fundus image (or a fluorescent fundus image) are set can be considered. Further, the plurality of medical images of different types may be anything as long as they are acquired by different modality, different optical systems, different principles, or the like.

Further, the trained model for generating the diagnosis result may be a trained model obtained by learning from the training data including the input data including a plurality of medical images of different parts of the subject. At this time, as the input data included in the training data, for example, data in which a tomographic image of the fundus of the eye (B scan image) and a tomographic image of the anterior segment of the eye (B scan image) are set can be considered. Further, as input data included in the training data, for example, data in which a three-dimensional OCT image (three-dimensional tomographic image) of the macula of the fundus and a circle scan (or raster scan) tomographic image of the optic nerve head of the fundus are set are also included. Conceivable.

The input data included in the learning data may be different parts of the subject and a plurality of different types of medical images. At this time, the input data included in the training data may be, for example, input data in which a tomographic image of the anterior segment of the eye and a color fundus image are set. Further, the various trained models described above may be trained models obtained by learning from training data including input data including a set of a plurality of medical images having different shooting angles of view of a predetermined part of the subject. Good. Further, the input data included in the learning data may be a combination of a plurality of medical images obtained by time-dividing a predetermined portion into a plurality of regions, such as a panoramic image. At this time, by using a wide angle-of-view image such as a panoramic image as learning data, there is a possibility that the feature amount of the image can be accurately acquired because the amount of information is larger than that of the narrow angle-of-view image. The result of processing can be improved. For example, when an abnormal portion is detected at a plurality of positions in a wide angle-of-view image at the time of estimation (prediction), an enlarged image of each abnormal portion can be sequentially displayed. As a result, it is possible to efficiently confirm the abnormal portion at a plurality of positions, so that the convenience of the examiner can be improved, for example. At this time, for example, the examiner may be configured to select each position on the wide angle-of-view image in which the abnormal portion is detected, and an enlarged image of the abnormal portion at the selected position may be displayed. .. Further, the input data included in the learning data may be input data in which a plurality of medical images of different dates and times of a predetermined part of the subject are set.

Further, the display screen on which at least one of the above-mentioned analysis result, diagnosis result, object recognition result, and segmentation result is displayed is not limited to the report screen. Such a display screen is, for example, at least one display screen such as a shooting confirmation screen, a display screen for follow-up observation, and a preview screen for various adjustments before shooting (a display screen on which various live moving images are displayed). It may be displayed in. For example, by displaying at least one result obtained by using the various trained models described above on the shooting confirmation screen, the examiner can confirm the accurate result even immediately after shooting. Further, for example, when a specific object is recognized, a frame surrounding the recognized object may be configured to be superimposed and displayed on the live moving image. At this time, if the information indicating the certainty of the object recognition result (for example, the numerical value indicating the ratio) exceeds the threshold value, the color of the frame surrounding the object may be changed or highlighted. Good. This allows the examiner to easily identify the object on the live video. Further, the above-mentioned change in the display between the low-quality image and the high-quality image may be, for example, a change in the display between the analysis result of the low-quality image and the analysis result of the high-quality image.

Here, the various trained models described above can be obtained by machine learning using the training data. Machine learning includes, for example, deep learning consisting of a multi-layer neural network. Further, for at least a part of the multi-layer neural network, for example, a convolutional neural network (CNN) can be used as a machine learning model. Further, a technique related to an autoencoder (self-encoder) may be used for at least a part of a multi-layer neural network. Further, a technique related to backpropagation (error backpropagation method) may be used for learning. Further, for learning, a method (dropout) in which each unit (each neuron) is randomly inactivated for each learning may be used. Further, for learning, a method (batch normalization) may be used in which the data transmitted to each layer of the multi-layer neural network is normalized before the activation function (for example, the ReLu function) is applied. However, the machine learning is not limited to deep learning, and any learning using a model capable of extracting (expressing) the features of learning data such as images by learning may be used. Here, the machine learning model refers to a learning model based on a machine learning algorithm such as deep learning. The trained model is a model in which a machine learning model by an arbitrary machine learning algorithm is trained (learned) in advance using appropriate learning data. However, the trained model does not require further learning, and additional learning can be performed. The learning data is composed of a pair of input data and output data (correct answer data). Here, the learning data may be referred to as teacher data, or the correct answer data may be referred to as teacher data.

The GPU can perform efficient calculations by processing more data in parallel. Therefore, when learning is performed a plurality of times using a learning model such as deep learning, it is effective to perform processing on the GPU. Therefore, in this modification, a GPU is used in addition to the CPU for processing by the image processing unit 220, which is an example of the learning unit (not shown). Specifically, when executing a learning program including a learning model, learning is performed by the CPU and the GPU collaborating to perform calculations. The processing of the learning unit may be performed only by the CPU or GPU. Further, the processing unit (estimation unit) that executes the processing using the various trained models described above may also use the GPU as in the learning unit. Further, the learning unit may include an error detecting unit and an updating unit (not shown). The error detection unit obtains an error between the output data output from the output layer of the neural network and the correct answer data according to the input data input to the input layer. The error detection unit may use the loss function to calculate the error between the output data from the neural network and the correct answer data. Further, the update unit updates the coupling weighting coefficient between the nodes of the neural network based on the error obtained by the error detection unit so that the error becomes small. This updating unit updates the coupling weighting coefficient and the like by using, for example, the backpropagation method. The error backpropagation method is a method of adjusting the coupling weighting coefficient between the nodes of each neural network so that the above error becomes small.

In addition, as machine learning models used for high image quality and segmentation, there are an encoder function consisting of a plurality of layers including a plurality of downsampling layers and a decoder function consisting of a plurality of layers including a plurality of upsampling layers. A U-net type machine learning model having the above is applicable. In the U-net type machine learning model, position information (spatial information) that is ambiguous in a plurality of layers configured as encoders is displayed in layers of the same dimension (layers corresponding to each other) in a plurality of layers configured as a decoder. ) (For example, using a skip connection).

Further, as a machine learning model used for high image quality, segmentation, etc., for example, FCN (Full Convolutional Network), SegNet, or the like can be used. Further, a machine learning model that recognizes an object in a region unit according to a desired configuration may be used. As a machine learning model for performing object recognition, for example, RCNN (Region CNN), fastRCNN, or fasterRCNN can be used. Further, as a machine learning model for recognizing an object in a region unit, YOLO (You Only Look Object) or SSD (Single Shot Detector or Single Shot MultiBox Detector) can also be used.

Further, the machine learning model may be, for example, a capsule network (Capsule Network; CapsNet). Here, in a general neural network, each unit (each neuron) is configured to output a scalar value, so that, for example, spatial information regarding the spatial positional relationship (relative position) between features in an image can be obtained. It is configured to be reduced. Thereby, for example, learning can be performed so as to reduce the influence of local distortion and translation of the image. On the other hand, in the capsule network, each unit (each capsule) is configured to output spatial information as a vector, so that, for example, spatial information is retained. Thereby, for example, learning can be performed in which the spatial positional relationship between the features in the image is taken into consideration.

Further, the high image quality engine (trained model for high image quality) may be a trained model obtained by additionally learning learning data including at least one high image quality image generated by the high image quality engine. Good. At this time, whether or not to use the high-quality image as the learning data for additional learning may be configured to be selectable according to the instruction from the examiner. It should be noted that these configurations can be applied not only to the trained model for high image quality but also to the various trained models described above. Further, in the generation of the correct answer data used for learning the various trained models described above, the trained model for generating the correct answer data for generating the correct answer data such as labeling (annotation) may be used. At this time, the trained model for generating correct answer data may be obtained by (sequentially) additionally learning the correct answer data obtained by labeling (annotation) by the examiner. That is, the trained model for generating correct answer data may be obtained by additional training of training data in which the data before labeling is used as input data and the data after labeling is used as output data. Further, in a plurality of consecutive frames such as a moving image, the result of the frame judged to have low accuracy is corrected in consideration of the results of object recognition and segmentation of the preceding and following frames. May be good. At this time, according to the instruction from the examiner, the corrected result may be additionally learned as correct answer data.

In the various embodiments and modifications described above, a partial region of the eye to be inspected (for example, a region of interest, an artifact region, an abnormal region, etc.) is detected using a trained model for object recognition and a trained model for segmentation. In this case, predetermined image processing can be performed for each detected area. For example, consider the case of detecting at least two regions of the vitreous region, the retinal region, and the choroid region. In this case, when performing image processing such as contrast adjustment on at least two detected regions, adjustments suitable for each region can be performed by using different image processing parameters. By displaying the image adjusted suitable for each area, the operator can more appropriately diagnose the disease or the like in each area. Note that the configuration using different image processing parameters for each detected region may be similarly applied to the region of the eye to be inspected detected without using the trained model, for example.

(Modification example 4)
In the preview screens in the various embodiments and modifications described above, the various trained models described above may be used for at least one frame of the live video. At this time, when a plurality of live moving images of different parts or different types are displayed on the preview screen, the trained model corresponding to each live moving image may be used. As a result, for example, even if it is a live moving image, the processing time can be shortened, so that the examiner can obtain highly accurate information before the start of shooting. Therefore, for example, the failure of re-imaging can be reduced, so that the accuracy and efficiency of diagnosis can be improved.

The plurality of live moving images may be, for example, a moving image of the anterior segment for alignment in the XYZ direction, a frontal moving image of the fundus for focus adjustment or OCT focus adjustment of the fundus observation optical system. Further, the plurality of live moving images are at least one moving image such as a tomographic moving image of the fundus for coherence gate adjustment of OCT (adjustment of the optical path length difference between the measured optical path length and the reference optical path length). May be good. At this time, the above-mentioned various adjustments may be performed so that the region detected by using the above-described trained model for object recognition or the above-mentioned trained model for segmentation satisfies a predetermined condition. For example, a value (for example, a contrast value or an intensity value) related to a predetermined retinal layer such as a vitreous region or RPE detected using a trained model for object recognition or a trained model for segmentation exceeds a threshold value (for example, a contrast value or an intensity value). Alternatively, various adjustments such as OCT focus adjustment may be performed so as to reach a peak value). Further, for example, the OCT so that a predetermined retinal layer such as a vitreous region or RPE detected by using a trained model for object recognition or a trained model for segmentation is at a predetermined position in the depth direction. The coherence gate adjustment may be configured to be performed.

In these cases, the image quality improving unit 225 in the medical image processing device can perform high image quality processing on the moving image using the trained model to generate a high quality moving image. Further, the control unit 211 has a photographing range of the reference mirror 143 or the like so that the partial area such as the attention portion obtained by the segmentation process or the like is at a predetermined position in the display area in the state where the high-quality moving image is displayed. It is possible to drive and control the optical member for changing the. In such a case, the control unit 211 can automatically perform the alignment process so that the desired area becomes a predetermined position of the display area based on the highly accurate information. The optical member for changing the photographing range may be, for example, an optical member for adjusting the coherence gate position, and specifically, a reference mirror 143 or the like. Further, the coherence gate position can be adjusted by an optical member that changes the optical path length difference between the measurement optical path length and the reference optical path length, and the optical member can change, for example, the optical path length of the measurement light (not shown). It may be a mirror or the like. The optical member for changing the photographing range may be, for example, a stage portion (not shown). Further, the control unit 211 scans (rescans) the partial area such as the artifact area obtained by the segmentation process or the like during the shooting or at the end of the shooting in response to the instruction regarding the start of shooting. The unit may be driven and controlled. Further, for example, when the information indicating the certainty of the object recognition result regarding the region of interest (for example, the numerical value indicating the ratio) exceeds the threshold value, each adjustment, shooting start, etc. may be automatically performed. Good. Further, for example, when the information indicating the certainty of the object recognition result regarding the region of interest (for example, the numerical value indicating the ratio) exceeds the threshold value, each adjustment, the start of imaging, etc. can be executed according to the instruction from the examiner. It may be configured to change to the above state (release the execution prohibited state).

Further, the moving image to which the various trained models described above can be applied is not limited to the live moving image, and may be, for example, a moving image stored (stored) in the storage unit. At this time, for example, the moving image obtained by aligning each frame of the tomographic moving image of the fundus stored (stored) in the storage unit may be displayed on the display screen. For example, when it is desired to observe the vitreous body preferably, first, a reference frame based on a condition such as the presence of the vitreous body on the frame as much as possible may be selected. At this time, each frame is a tomographic image (B scan image) in the XZ direction. Then, a moving image in which another frame is aligned in the XZ direction with respect to the selected reference frame may be displayed on the display screen. At this time, for example, a high-quality image (high-quality frame) sequentially generated by the high-quality engine may be continuously displayed for each at least one frame of the moving image.

Here, during various adjustments, it is possible that the imaged object such as the retina of the eye to be inspected has not yet been successfully imaged. Therefore, since there is a large difference between the medical image input to the trained model and the medical image used as the training data, there is a possibility that a high-quality image cannot be obtained with high accuracy. Therefore, when the evaluation value such as the image quality evaluation of the tomographic image (B scan) exceeds the threshold value, the display of the high-quality moving image (continuous display of the high-quality frame) may be automatically started. Further, when the evaluation value such as the image quality evaluation of the tomographic image (B scan) exceeds the threshold value, the image quality enhancement button may be configured to be changed to a state (active state) that can be specified by the examiner.

Further, a different high image quality engine may be prepared for each shooting mode having a different scanning pattern or the like, and the high image quality engine corresponding to the selected shooting mode may be selected. Further, one high image quality engine obtained by learning learning data including various medical images obtained in different imaging modes may be used.

As the above-mentioned method for aligning between frames, the same method may be applied to the method for aligning in the X direction and the method for aligning in the Z direction (depth direction), or all different methods may be applied. May be applied. Further, the alignment in the same direction may be performed a plurality of times by different methods. For example, after performing the coarse alignment, the precise alignment may be performed. Further, as a method of alignment, for example, a plurality of alignments obtained by segmenting a tomographic image (B scan image) using a retinal layer boundary obtained by performing segmentation processing (coarse in the Z direction) and dividing the tomographic image. Alignment using the correlation information (similarity) between the region and the reference image (precise in the X and Z directions), and using the one-dimensional projection image generated for each tomographic image (B scan image) (in the X direction). ) Alignment, etc. Alignment (in the X direction) using a two-dimensional front image. Further, it may be configured so that the alignment is roughly performed in pixel units and then the precise alignment is performed in sub-pixel units.

(Modification 5)
In the various embodiments and modifications described above, when the trained model is executing the additional learning, it may be difficult to output (infer / predict) using the trained model itself during the execution of the additional learning. There is. Therefore, it is preferable to prohibit the input of medical images other than the training data to the trained model during the execution of the additional learning. Further, another trained model that is the same as the trained model before the execution of the additional learning may be prepared as another preliminary trained model. At this time, during the execution of the additional learning, it is preferable that the input of the medical image other than the training data to the preliminary trained model can be executed. Then, after the additional learning is completed, the trained model after the execution of the additional learning is evaluated, and if there is no problem, the preliminary trained model may be replaced with the trained model after the execution of the additional learning. Also, if there is a problem, a preliminary trained model may be used. As an evaluation of the trained model obtained by additional learning, for example, a trained model for classification for classifying a high-quality image obtained by the trained model for high image quality with another type of image is used. It may be used. The trained model for classification uses, for example, a plurality of images including a high-quality image and a low-quality image obtained by the trained model for high image quality as input data, and the types of these images are labeled (annotation). It may be a trained model obtained by training training data including the obtained data as correct answer data. At this time, the image type of the input data at the time of estimation (prediction) is displayed together with the information (for example, a numerical value indicating the ratio) indicating the certainty of each type of image included in the correct answer data at the time of learning. May be good. In addition to the above images, the input data of the trained model for classification includes superposition processing of a plurality of low-quality images (for example, averaging processing of a plurality of low-quality images obtained by alignment) and the like. It may include a high-quality image in which high contrast, noise reduction, etc. are performed. Further, as an evaluation of the trained model after the execution of the additional learning, for example, a plurality of trained models obtained from the same image using the trained model after the execution of the additional learning and the trained model before the execution of the additional learning are used. High-quality images may be compared, or the analysis results of the plurality of high-quality images may be compared. At this time, for example, the comparison result of the plurality of high-quality images (an example of change due to additional learning) or the comparison result of the analysis result of the plurality of high-quality images (an example of change due to additional learning) is within a predetermined range. It may be determined whether or not, and the determination result may be displayed.

In addition, the trained model obtained by learning for each imaging site may be selectively used. Specifically, learning including a first learned model obtained by using learning data including a first imaging site (lung, eye to be examined, etc.) and a second imaging site different from the first imaging site. A second trained model obtained using the data and a plurality of trained models including the second trained model can be prepared. Then, any of these plurality of trained models may be configured to be selected (using a selection means (not shown). At this time, the medical image processing apparatus may have a control means (not shown) that executes additional learning on the selected trained model. The control means searches for data in which the imaged part corresponding to the selected trained model and the photographed image of the imaged part are paired according to the instruction from the examiner, and the data obtained by the search is the learning data. Can be executed as additional learning for the selected trained model. The imaging site corresponding to the selected trained model may be acquired from the information in the header of the data or manually input by the examiner. Further, the data search may be performed from a server of an external facility such as a hospital or a research institute via a network, for example. As a result, additional learning can be efficiently performed for each imaged part by using the photographed image of the imaged part corresponding to the trained model.

The selection means and the control means may be composed of a software module executed by a processor such as a CPU or an MPU of a medical image processing device. Further, the selection means and the control means may be composed of a circuit that performs a specific function such as an ASIC, an independent device, or the like.

In addition, when acquiring learning data for additional learning from a server of an external facility such as a hospital or research institute via a network, it is desired to reduce reliability deterioration due to falsification or system trouble during additional learning. Therefore, the correctness of the learning data for additional learning may be detected by confirming the consistency by digital signature or hashing. As a result, the learning data for additional learning can be protected. At this time, if the validity of the training data for additional learning cannot be detected as a result of confirming the consistency by digital signature or hashing, a warning to that effect is given and additional learning is performed using the training data. Absent. The server may be in any form, for example, a cloud server, a fog server, an edge server, or the like, regardless of its installation location. Further, the data protection by confirming the consistency as described above can be applied not only to the learning data for additional learning but also to the data including the medical image. Further, the image management system may be configured so that the transaction of data including medical images between servers of a plurality of facilities is managed by a distributed network. Further, the image management system may be configured to connect a plurality of blocks in which the transaction history and the hash value of the previous block are recorded together in a time series. As a technique for confirming consistency, even if a cipher that is difficult to calculate even using a quantum computer such as a quantum gate method (for example, lattice-based cryptography, quantum cryptography by quantum key distribution, etc.) is used. Good.

(Modification 6)
In the various embodiments and modifications described above, the instruction from the examiner may be an instruction by voice or the like in addition to a manual instruction (for example, an instruction using a user interface or the like). At this time, for example, a machine learning engine including a voice recognition engine (speech recognition model, trained model for voice recognition) obtained by machine learning may be used. Further, the manual instruction may be an instruction by character input or the like using a keyboard, a touch panel, or the like. At this time, for example, a machine learning engine including a character recognition engine (character recognition model, trained model for character recognition) obtained by machine learning may be used. Further, the instruction from the examiner may be an instruction by a gesture or the like. At this time, a machine learning engine including a gesture recognition engine (gesture recognition model, learned model for gesture recognition) obtained by machine learning may be used.

Further, the instruction from the examiner may be an eye-gaze detection result of the examiner on the display screen of the display unit 300 or the like. The line-of-sight detection result may be, for example, a pupil detection result using a moving image of the examiner obtained by photographing from the periphery of the display screen on the display unit 300. At this time, the object recognition engine as described above may be used for the pupil detection from the moving image. Further, the instruction from the examiner may be an instruction by an electroencephalogram, a weak electric signal flowing through the body, or the like.

In such a case, for example, as the training data, various trained data such as character data or voice data (waveform data) indicating instructions for displaying the results obtained by processing the various trained models as described above are used as input data. It may be learning data in which the execution instruction for actually displaying the result or the like obtained by the processing of the model on the display unit 300 is the correct answer data. Further, as the training data, for example, character data or audio data indicating an instruction for displaying a high-quality image obtained by a trained model for high image quality is used as input data, and an execution command and a button for displaying the high-quality image are used as input data. It may be learning data in which the execution instruction for changing the 3420 to the active state is the correct answer data. Of course, the learning data may be any data as long as the instruction content and the execution instruction content indicated by the character data, the voice data, or the like correspond to each other. Further, the voice data may be converted into character data by using an acoustic model, a language model, or the like. Further, the waveform data obtained by the plurality of microphones may be used to perform a process of reducing the noise data superimposed on the voice data. Further, the instruction by characters or voice and the instruction by a mouse, a touch panel or the like may be configured to be selectable according to the instruction from the examiner. Further, the on / off of the instruction by characters or voice may be selectably configured according to the instruction from the examiner.

Here, machine learning includes deep learning as described above, and for at least a part of a multi-layer neural network, for example, a recurrent neural network (RNN) can be used. Here, as an example of the machine learning engine according to this modified example, RNN, which is a neural network that handles time series information, will be described with reference to FIGS. 13 (a) and 13 (b). Further, a Long short-term memory (hereinafter referred to as LSTM), which is a kind of RNN, will be described with reference to FIGS. 14 (a) and 14 (b).

FIG. 13A shows the structure of the RNN, which is a machine learning engine. The RNN 3520 has a loop structure in the network, ^{inputs data x t} 3510 at time t, and outputs ^{data h t 3530.} Since the RNN3520 has a loop function in the network, the current state can be inherited to the next state, so that time-series information can be handled. FIG. 13B shows an example of input / output of the parameter vector at time t. The data x ^t 3510 contains N pieces of data (Params1 to ParamsN). Further, the data h ^t 3530 output from the RNN 3520 includes N data (Params1 to ParamsN) corresponding to the input data.

However, since RNN cannot handle long-term information at the time of error back propagation, LSTM may be used. The LSTM can learn long-term information by providing a forgetting gate, an input gate, and an output gate. Here, FIG. 14A shows the structure of the LSTM. In the RSTM3540, the information that the network takes over at the next time t is the internal state c ^{t-1 of the} network called the cell and the output data h ^t-1 . The lowercase letters (c, h, x) in the figure represent vectors.

Next, FIG. 14 (b) shows the details of RSTM3540. In FIG. 14B, FG indicates a forgetting gate network, IG indicates an input gate network, and OG indicates an output gate network, each of which is a sigmoid layer. Therefore, a vector in which each element has a value of 0 to 1 is output. The oblivion gate network FG determines how much past information is retained, and the input gate network IG determines which value to update. The CU is a cell update candidate network and is an activation function tanh layer. This creates a vector of new candidate values to be added to the cell. The output gate network OG selects the cell candidate element and selects how much information to convey at the next time.

Since the above-mentioned LSTM model is a basic model, it is not limited to the network shown here. You may change the coupling between the networks. QRNN (Quasi Recurrent Neural Network) may be used instead of RSTM. Further, the machine learning engine is not limited to the neural network, and boosting, a support vector machine, or the like may be used. Further, when the instruction from the examiner is input by characters, voice, or the like, a technique related to natural language processing (for example, Sequence to Sequence) may be applied. Further, a dialogue engine (dialogue model, trained model for dialogue) that responds to the examiner by outputting characters or voices may be applied.

Further, as a technique related to natural language processing, a trained model obtained by pre-learning document data by unsupervised learning may be used. Further, as a technique related to natural language processing, a trained model obtained by further transfer learning (or fine tuning) of a trained model obtained by pre-learning may be used. Further, as a technique related to natural language processing, for example, BERT (Bidirectional Encoder Representations from Transfermers) may be applied. Further, as a technique related to natural language processing, a model capable of extracting (expressing) a context (feature amount) by itself by predicting a specific word in a sentence from both the left and right contexts may be applied. Further, as a technique related to natural language processing, a model capable of determining the relationship (continuity) of two sequences (sentences) in the input time series data may be applied. Further, as a technique related to natural language processing, a Transformer Encoder is used for the hidden layer, and a model in which a vector sequence is input and output may be applied.

Here, the instruction from the examiner to which this modification is applicable is for changing the display of various images and analysis results as described in the various examples and modifications described above, and for generating an En-Face image. Selection of depth range, selection of whether to use as training data for additional learning, selection of trained model, output (display, transmission, etc.) and storage of results obtained using various trained models, etc. Any instruction may be used as long as it is at least one instruction. Further, the instruction from the examiner to which this modification is applicable may be an instruction before photography as well as an instruction after photography. For example, an instruction regarding various adjustments and an instruction regarding setting of various imaging conditions. , It may be an instruction regarding the start of shooting. Further, the instruction from the examiner to which this modification is applicable may be an instruction regarding a change (screen transition) of the display screen.

(Modification 7)
Further, in the various embodiments and modifications described above, the high-quality image, the composite image, or the like may be stored in the storage unit according to an instruction from the examiner. At this time, after an instruction from the examiner to save a high-quality image or a composite image, etc., when registering the file name, one of the file names (for example, the first part) is recommended as the file name. , The last part), the file name containing information (for example, characters) indicating that the image is generated by processing using the trained model for high image quality (high image quality processing) is given by the examiner. It may be displayed in an editable state according to the instruction of.

Further, on various display screens such as a report screen, as described above, when displaying a high-quality image on the display unit 300, the displayed image is generated by processing using a trained model for high image quality. A display indicating that the image is a high-quality image may be displayed together with the high-quality image. In this case, the user can easily identify from the display that the displayed high-quality image is not the image itself acquired by shooting, so that misdiagnosis can be reduced or the diagnosis efficiency can be improved. Can be done. The display indicating that the image is a high-quality image generated by the process using the trained model for high image quality is a display that can distinguish the input image and the high-quality image generated by the process. It may be of any aspect. Further, not only the processing using the trained model for high image quality but also the processing using various trained models as described above is the result generated by the processing using the trained model of the same type. A display indicating that there is may be displayed with the result.

At this time, the display screen such as the report screen may be saved in the storage unit as image data according to the instruction from the examiner. For example, a report screen as one image in which a high-quality image, a composite image, or the like and a display indicating that these images are high-quality images generated by processing using a trained model for high image quality are arranged. May be stored in the storage unit.

In addition, what kind of training data was used by the trained model for high image quality to learn the display indicating that the image is a high quality image generated by processing using the trained model for high image quality. A display indicating the presence or absence may be displayed on the display unit 300. The display may include a display of an explanation of the input data of the learning data and the type of the correct answer data, an arbitrary display of the correct answer data such as the imaging part included in the input data and the correct answer data, and the like. It should be noted that not only the processing using the trained model for high image quality but also the processing using the various trained models as described above is trained by what kind of training data the trained model of that type uses. A display indicating whether or not the data is used may be displayed on the display unit 300.

In addition, information (for example, characters) indicating that the image is generated by processing using a trained model for high image quality is displayed or saved in a state of being superimposed on a high image quality image or a composite image. It may be configured in. At this time, the portion to be superimposed on the image may be any region (for example, the edge of the image) that does not overlap with the region where the region of interest to be photographed is displayed. Further, the non-overlapping areas may be determined and superimposed on the determined areas.

In addition, when the button 3420 is set to the active state (high image quality processing is on) by default as the initial display screen of the report screen, a high image quality image or composition is performed according to the instruction from the examiner. The report image corresponding to the report screen including the image and the like may be configured to be transmitted to the server. In addition, when the button 3420 is set to the active state by default, at the end of the inspection (for example, when the shooting confirmation screen or the preview screen is changed to the report screen according to the instruction from the inspector). In addition, the report image corresponding to the report screen including the high-quality image, the composite image, and the like may be configured to be (automatically) transmitted to the server. At this time, various settings in the default settings (for example, the depth range for generating the En-Face image on the initial display screen of the report screen, the presence / absence of superimposition of the analysis map, whether or not the image is high quality, and the display screen for follow-up observation. The report image generated based on (settings related to at least one such as whether or not) may be configured to be transmitted to the server.

(Modification 8)
Further, in the various embodiments and modifications described above, among the various trained models described above, the analysis results of images (for example, high-quality images, analysis maps, etc.) obtained by the first type of trained model. (Image showing the image, the image showing the object recognition result, the image showing the segmentation result) may be input to the trained model of the second type different from the first type. At this time, the result (for example, analysis result, diagnosis result, object recognition result, segmentation result) of the processing of the second type of trained model may be generated.

Further, among the various trained models as described above, the result of processing the trained model of the first type (for example, analysis result, diagnosis result, object recognition result, segmentation result) is used to use the first type. An image to be input to a second type of trained model different from the first type may be generated from the image input to the trained model of. At this time, the generated image is likely to be an image suitable as an image to be processed by the second type of trained model. Therefore, an image obtained by inputting the generated image into the second type of trained model (for example, a high-quality image, an image showing an analysis result such as an analysis map, an image showing an object recognition result, and a segmentation result are displayed. The accuracy of the image shown) can be improved.

Further, the various trained models as described above may be trained models obtained by learning learning data including a two-dimensional medical image of the subject, or may be a three-dimensional medical model of the subject. It may be a trained model obtained by learning training data including images.

Further, a similar case image search using an external database stored in a server or the like may be performed using the analysis result, the diagnosis result, etc. obtained by the processing of the learned model as described above as a search key. If a plurality of images stored in the database are already managed by machine learning or the like with the feature amount of each of the plurality of images attached as incidental information, the image itself is used as a search key. A similar case image search engine (similar case image search model, trained model for similar case image search) may be used. For example, a medical image processing apparatus uses a trained model for searching for similar case images (different from the trained model for high image quality) to search various medical images for similar case images related to the medical image. It can be performed. Further, for example, the output unit 223 can display the similar case image obtained from various medical images by using the learned model for searching the similar case image on the display unit 300. At this time, the similar case image is, for example, an image having a feature amount similar to the feature amount of the medical image input to the trained model. Further, a plurality of similar case images may be searched, and a plurality of similar case images may be displayed so that the order in which the feature amounts are similar can be identified. In addition, a trained model for searching similar case images is additionally learned by using learning data including an image selected according to an instruction from an examiner and a feature amount of the image among a plurality of similar case images. It may be configured to be.

(Other embodiments)
The present invention is also realized by executing the following processing. That is, software (program) that realizes one or more functions of the various embodiments and modifications described above is supplied to the system or device via a network or various storage media, and the computer (or CPU) of the system or device is supplied. This is a process in which an MPU or the like reads and executes a program.

Further, the present invention supplies software (program) that realizes one or more functions of the various embodiments and modifications described above to a system or device via a network or storage medium, and the computer of the system or device supplies the software (program). It can also be realized by the process of reading and executing the program. A computer may have one or more processors or circuits and may include multiple separate computers or a network of separate processors or circuits to read and execute computer executable instructions.

At this time, the processor or circuit may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), or a field programmable gateway (FPGA). Also, the processor or circuit may include a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU).

Claims (18)

An acquisition unit that acquires a first medical image obtained by photographing a predetermined part of a subject with a first imaging device, and
The medical image obtained by photographing a predetermined part of the subject with the first imaging device was used as input data, and the predetermined part of the subject was photographed with a second imaging device different from the first imaging device. Using the trained model obtained by using the training data using the medical image as the output data, the second medical image has a higher image quality than the first medical image from the acquired first medical image. High image quality section that generates images and
A medical image processing device. The medical image processing device according to claim 1, wherein the first imaging device and the second imaging device are of different models. The medical image processing device according to claim 1 or 2, wherein the first imaging device and the second imaging device have different imaging methods. The medical image processing device according to any one of claims 1 to 3, wherein the second imaging device can acquire a medical image having a higher resolution than that of the first imaging device. The first photographing device can shoot with a shooting angle of view wider than the shooting angle of view of the second shooting device.
The medical image processing apparatus according to any one of claims 1 to 4, wherein the second imaging device can image with a resolution higher than that of the first imaging device. The first imaging device is an SS-OCT device.
The medical image processing apparatus according to any one of claims 1 to 5, wherein the second imaging apparatus is an SD-OCT apparatus. The medical image processing apparatus according to any one of claims 1 to 6, wherein the subject is an eye to be inspected. A medical image processing apparatus further comprising a display control unit for displaying the second medical image on the display unit. The display control unit is an analysis result generated by using a trained model for generating an analysis result obtained by learning a medical image of a subject, and is the first medical image or the second medical image. The medical image processing apparatus according to claim 8, wherein the analysis result relating to the above is displayed on the display unit. The display control unit is a diagnostic result generated by using a trained model for generating a diagnostic result obtained by learning a medical image of a subject, and is the first medical image or the second medical image. The medical image processing apparatus according to claim 8 or 9, wherein the diagnostic result relating to the above is displayed on the display unit. The display control unit inputs an image generated by using the hostile generation network or autoencoder into which the first medical image or the second medical image is input and the hostile generation network or autoencoder. The medical image processing apparatus according to any one of claims 8 to 10, wherein information on a difference from the medical image is displayed on the display unit as information on an abnormal portion. The display control unit is a similar case image searched by using a learned model for searching a similar case image obtained by learning a medical image of a subject, and is the first medical image or the second medical image. The medical image processing apparatus according to any one of claims 8 to 11, wherein a similar case image relating to a medical image is displayed on the display unit. The display control unit is an object detection result or a segmentation result generated by using a trained model for object recognition or a trained model for segmentation obtained by learning a medical image of a subject, and is the first. The medical image processing apparatus according to any one of claims 8 to 12, wherein an object detection result or a segmentation result relating to the medical image or the second medical image is displayed on the display unit. The display control unit has the display unit display the image, information, or result obtained by inputting the first medical image and the second medical image into the trained model, according to claims 8 to 13. The medical image processing apparatus according to any one of the following items. The instruction from the examiner regarding the image quality improvement is information obtained by using at least one trained model of a trained model for character recognition, a trained model for voice recognition, and a trained model for gesture recognition. The medical image processing apparatus according to any one of claims 1 to 14. The first to second claims, wherein the file name of the second medical image includes information indicating that the image is an image generated by improving the image quality in a state in which the image can be edited according to an instruction from the examiner. The medical image processing apparatus according to any one of 15. A step of acquiring a first medical image obtained by photographing a predetermined part of a subject with a first imaging device, and
The medical image obtained by photographing a predetermined part of the subject with the first imaging device was used as input data, and the predetermined part of the subject was photographed with a second imaging device different from the first imaging device. Using the trained model obtained by using the training data using the medical image as the output data, the second medical image has a higher image quality than the first medical image from the acquired first medical image. The process of generating an image and
Medical image processing methods, including. A program that, when executed by a processor, causes the processor to perform each step of the medical image processing method according to claim 17.

Images