JP2020058629A - Medical image processing apparatus, medical image processing method, and program - Google Patents

Abstract

To provide a medical image processing apparatus capable of generating images more suitable for image diagnosis than before.SOLUTION: The medical image processing apparatus includes: an acquisition unit for acquiring a first image, which is a motion contrast front image of a subject eye; and an image quality enhancing unit for generating, from the first image, a second image having a more enhanced image quality compared with the first image, using an image quality enhancing engine including a machine learning engine.SELECTED DRAWING: Figure 5

Description

  The present invention relates to a medical image processing device, a medical image processing method, and a program.

  In the medical field, in order to identify a disease of a subject and observe the degree of the disease, images are acquired by various imaging devices, and image diagnosis is performed by a medical staff. The types of imaging devices include, for example, in the field of radiology, X-ray imaging devices, X-ray computed tomography (CT) devices, magnetic resonance imaging (MRI) devices, positron emission tomography (PET) devices, and single photon emission. There is a tomography (SPECT) device and the like. Further, for example, in the field of ophthalmology, there are a fundus camera, a scanning laser ophthalmoscope (SLO), an optical coherence tomography (OCT) device, and an OCT angiography (OCTA) device.

  In order to perform image diagnosis accurately and to complete it in a short time, high image quality such as low noise, high resolution / spatial resolution, and appropriate gradation is important for the images acquired by the imaging device. Becomes Also, an image in which the region or lesion to be observed is emphasized may be useful.

  However, in many imaging devices, some compensation is required to acquire an image suitable for image diagnosis, such as high image quality. For example, there is a method of purchasing a high-performance image capturing device in order to obtain a high-quality image, but it often requires more investment than a low-performance image capturing device.

  Further, for example, in CT, it may be necessary to increase the exposure dose of the subject in order to acquire an image with less noise. In addition, for example, in MRI, a contrast agent having a risk of side effects may be used in order to acquire an image in which a region to be observed is emphasized. Further, for example, in OCT, when a region to be imaged is wide or a high spatial resolution is required, the image taking time may be longer. In addition, for example, some image capturing devices need to acquire images multiple times in order to acquire images with high image quality, and thus it takes a long time to shoot.

  Patent Document 1 discloses a technique of converting an image acquired previously into an image of higher resolution by an artificial intelligence engine in order to cope with rapid progress of medical technology and simple imaging in an emergency. . According to such a technique, for example, an image acquired by simple shooting with low cost can be converted into an image with higher resolution.

JP, 2018-5841, A

  However, even a high-resolution image may not be suitable for image diagnosis. For example, even if the image has a high resolution, the object to be observed may not be properly grasped when there is a lot of noise or the contrast is low.

  On the other hand, one of the objects of the present invention is to provide a medical image processing apparatus, a medical image processing method, and a program capable of generating an image more suitable for image diagnosis than ever before.

  A medical image processing apparatus according to an embodiment of the present invention uses an acquisition unit that acquires a first image that is a motion-contrast front image of an eye to be inspected, and an image quality enhancement engine that includes a machine learning engine to perform the first image processing. Image quality improvement section that generates a second image quality-improved from the first image quality compared to the first image quality.

  Further, a medical image processing method according to another embodiment of the present invention uses the image quality improving engine including a machine learning engine to obtain a first image which is a motion contrast front image of an eye to be examined, and Generating from the first image a second image of which the quality is higher than that of the first image.

  According to one aspect of the present invention, it is possible to generate an image more suitable for image diagnosis than ever before.

An example of the configuration of the neural network relating to the image quality improvement processing is shown. An example of the configuration of the neural network relating to the shooting location estimation processing is shown. An example of the configuration of a neural network for image authenticity evaluation processing is shown. 1 shows an example of a schematic configuration of an image processing apparatus according to a first embodiment. It is a flowchart which shows an example of the flow of the image processing which concerns on 1st Embodiment. It is a flow figure showing another example of a flow of image processing concerning a 1st embodiment. It is a flowchart which shows an example of the flow of the image processing which concerns on 2nd Embodiment. It is a figure for demonstrating the image processing which concerns on 4th Embodiment. It is a flow figure showing an example of the flow of image quality improvement processing concerning a 4th embodiment. It is a figure for explaining image processing concerning a 5th embodiment. It is a flow figure showing an example of the flow of image quality improvement processing concerning a 5th embodiment. It is a figure for demonstrating the image processing which concerns on 6th Embodiment. It is a flow figure showing an example of the flow of image quality improvement processing concerning a 6th embodiment. It is a figure for demonstrating the image processing which concerns on 6th Embodiment. An example of a schematic configuration of an image processing apparatus according to a seventh embodiment is shown. It is a flowchart which shows an example of the flow of the image processing which concerns on 7th Embodiment. An example of the user interface which concerns on 7th Embodiment is shown. An example of a schematic configuration of an image processing apparatus according to a ninth embodiment will be shown. It is a flowchart which shows an example of the flow of the image processing which concerns on 9th Embodiment. An example of a schematic configuration of an image processing apparatus according to a twelfth embodiment is shown. It is a flow figure showing an example of the flow of image quality improvement processing concerning a 13th embodiment. It is a flowchart which shows another example of the flow of the image quality improvement process which concerns on 13th Embodiment. An example of a schematic structure of the image processing apparatus which concerns on 17th Embodiment is shown. It is a flowchart which shows an example of the flow of the image processing which concerns on 17th Embodiment. An example of the configuration of the neural network relating to the image quality improvement processing is shown. An example of the schematic structure of the image processing apparatus which concerns on 19th Embodiment is shown. It is a flowchart which shows an example of the flow of the image processing which concerns on 19th Embodiment. It is a flowchart which shows an example of the flow of the image processing which concerns on 21st Embodiment. An example of the teacher image regarding the high image quality processing is shown. An example of the input image regarding an image quality improvement process is shown. An example of the schematic structure of the image processing apparatus which concerns on 22nd Embodiment is shown. It is a flowchart which shows an example of the flow of the image processing which concerns on 22nd Embodiment. It is a figure for demonstrating the wide-angle image which concerns on 22nd Embodiment.

  Hereinafter, exemplary embodiments for carrying out the present invention will be described in detail with reference to the drawings. However, dimensions, materials, shapes, relative positions of constituent elements, and the like described in the following embodiments are arbitrary, and can be changed according to the configuration of the apparatus to which the present invention is applied or various conditions. Also, in the drawings, the same reference numbers are used in the drawings to denote the same or functionally similar elements.

<Explanation of terms>
First, terms used in this specification will be described.

  In the network herein, each device may be connected by a wired or wireless line. Here, the line connecting each device in the network is, for example, a dedicated line, a local area network (hereinafter, referred to as LAN) line, a wireless LAN line, an Internet line, Wi-Fi (registered trademark), and Bluetooth (registered trademark). ) Etc. are included.

  The medical image processing apparatus may be composed of two or more devices that can communicate with each other, or may be composed of a single device. Further, each component of the medical image processing apparatus may be configured by a software module executed by a processor such as a CPU (Central Processing Unit) and an MPU (Micro Processing Unit). In addition, each of the constituent elements may be configured by a circuit or the like that performs a specific function such as an ASIC. Further, it may be configured by a combination of other arbitrary hardware and arbitrary software.

  Further, the medical image processed by the medical image processing apparatus or the medical image processing method according to the following embodiments includes an image acquired by using an arbitrary modality (imaging apparatus, imaging method). The medical image to be processed can include a medical image acquired by an arbitrary imaging device or the like, or an image created by the medical image processing device or the medical image processing method according to the following embodiments.

  Further, the medical image to be processed is an image of a predetermined part of the subject, and the image of the predetermined part includes at least a part of the predetermined part of the subject. In addition, the medical image may include other parts of the subject. Further, the medical image may be a still image or a moving image, and may be a monochrome image or a color image. Further, the medical image may be an image showing the structure (morphology) of a predetermined part or an image showing its function. The image representing the function includes images representing blood flow dynamics (blood flow rate, blood flow velocity, etc.) such as an OCTA image, a Doppler OCT image, an fMRI image, and an ultrasonic Doppler image. The predetermined part of the subject may be determined according to the imaging target, human eyes (inspection eye), brain, lungs, intestines, heart, pancreas, kidneys, organs such as liver, head, chest, It includes any part such as legs and arms.

  Further, the medical image may be a tomographic image of the subject or a front image. The front image is, for example, a front image of the fundus of the eye, a front image of the anterior segment of the eye, a fundus image captured by fluorescence, and at least a part of a range (three-dimensional OCT data) acquired by OCT in the depth direction of the imaging target. The En-Face image generated by using the data of 1. It should be noted that the En-Face image is an OCTA En-Face image (motion contrast) generated by using data of at least a partial range in the depth direction of the imaging target for the three-dimensional OCTA data (three-dimensional motion contrast data). Front image).

  The image capturing device is a device for capturing an image used for diagnosis. The imaging device detects, for example, a device that obtains an image of a predetermined region by irradiating a predetermined region of a subject with light, radiation such as X-rays, electromagnetic waves, or ultrasonic waves, or radiation emitted from a subject. It includes a device for obtaining an image of a predetermined part by doing so. More specifically, the imaging apparatus according to the following embodiments includes at least an X-ray imaging apparatus, a CT apparatus, an MRI apparatus, a PET apparatus, a SPECT apparatus, an SLO apparatus, an OCT apparatus, an OCTA apparatus, a fundus camera, and an endoscopic apparatus. Including a mirror.

  The OCT device may include a time domain OCT (TD-OCT) device and a Fourier domain OCT (FD-OCT) device. Further, the Fourier domain OCT device may include a spectral domain OCT (SD-OCT) device and a wavelength swept OCT (SS-OCT) device. The SLO device and the OCT device may include a wavefront compensation SLO (AO-SLO) device using a wavefront compensation optical system, a wavefront compensation OCT (AO-OCT) device, and the like.

  The image management system is a device and a system for receiving and storing an image captured by an image capturing device or an image processed by the image capturing device. Further, the image management system may transmit an image in response to a request from a connected device, perform image processing on a stored image, or request an image processing request to another device. it can. The image management system may include, for example, an image storage communication system (PACS). In particular, the image management system according to the following embodiment includes a database that can store various information such as the information of the subject and the shooting time associated with the received image. Further, the image management system is connected to a network and can transmit and receive images, convert images, and transmit and receive various information associated with stored images in response to a request from another device. .

  The shooting conditions are various kinds of information at the time of shooting an image acquired by the shooting device. The imaging conditions include, for example, information about the imaging device, information about the facility where the imaging was performed, information about the examination related to the imaging, information about the photographer, and information about the subject. Further, the imaging conditions include, for example, imaging date / time, imaging site name, imaging region, imaging angle of view, imaging method, image resolution and gradation, image size, applied image filter, information regarding image data format, and radiation. Includes information about quantity. The imaging region may include a peripheral region deviated from a specific imaging region, a region including a plurality of imaging regions, and the like.

  The shooting conditions can be stored in a data structure forming the image, as shooting condition data different from the image, or stored in a database or an image management system associated with the shooting device. Therefore, the photographing condition can be acquired by a procedure corresponding to the photographing condition storage unit of the photographing device. Specifically, the shooting condition is, for example, for analyzing the data structure of the image output by the shooting device, obtaining shooting condition data corresponding to the image, or obtaining the shooting condition from a database related to the shooting device. It is obtained by accessing the interface of.

  Depending on the image capturing device, there are also image capturing conditions that cannot be acquired because they are not saved. For example, there is a case where the image capturing apparatus does not have a function of acquiring or storing a specific image capturing condition, or such a function is disabled. Further, for example, in some cases, it may not be saved because the shooting condition is not related to the shooting device or shooting. Furthermore, for example, there are cases where the shooting conditions are hidden, encrypted, or cannot be acquired without the right. However, it may be possible to acquire even shooting conditions that have not been saved. For example, by performing image analysis, it is possible to specify the imaged region name and the imaged region.

  The machine learning model is a model in which an arbitrary machine learning algorithm is trained (learned) in advance using appropriate teacher data (learning data). The teacher data is composed of one or more pairs of pairs of input data and output data. The format and combination of the input data and the output data of the pair group that constitutes the teacher data are such that one is an image and the other is a numerical value, or one is composed of a plurality of image groups and the other is a character string. May be an image or the like, and may be suitable for a desired configuration.

  Specifically, for example, teacher data (hereinafter, first teacher data) configured by a pair group of an image acquired by OCT and an imaged region label corresponding to the image can be cited. The imaged part label is a unique numerical value or character string representing the part. Further, as another example of the teacher data, it is composed of a pair group of a low-quality image with a lot of noise, which is acquired by normal OCT imaging, and a high-quality image that has been captured multiple times by OCT and has undergone high-quality processing. Examples include teacher data (hereinafter, second teacher data) and the like.

  When input data is input to the machine learning model, output data according to the design of the machine learning model is output. The machine learning model outputs output data that is highly likely to correspond to the input data, for example, according to the tendency trained using the teacher data. Further, the machine learning model can output the possibility of corresponding to the input data as a numerical value for each type of the output data according to the tendency of being trained using the teacher data, for example. Specifically, for example, when the image acquired by OCT is input to the machine learning model trained with the first teacher data, the machine learning model outputs the imaged part label of the imaged part imaged in the image. Or output the probability for each imaged part label. Further, for example, when a noisy low-quality image acquired by normal photographing of OCT is input to a machine learning model trained with the second teacher data, the machine learning model is photographed multiple times by OCT to improve the image quality. A high quality image equivalent to the processed image is output. Note that the machine learning model can be configured so that the output data output by itself is not used as teacher data from the viewpoint of maintaining quality.

  Further, the machine learning algorithm includes a method related to deep learning such as convolutional neural network (CNN). In the method related to deep learning, the degree of reproducibility of the training tendency using the teacher data to the output data may be different when the setting of the parameters for the layer group and the node group forming the neural network is different. For example, in the deep learning machine learning model using the first teacher data, if more appropriate parameters are set, the probability of outputting a correct imaging region label may be higher. Further, for example, in a deep learning machine learning model using the second teacher data, higher quality images may be output when more appropriate parameters are set.

  Specifically, the CNN parameters are, for example, the filter kernel size, the number of filters, the stride value, and the dilation value, which are set for the convolutional layer, and the number of nodes output by the fully connected layer. Etc. can be included. It should be noted that the parameter group and the number of training epochs can be set to values preferable for the usage form of the machine learning model based on the teacher data. For example, it is possible to set a parameter group and the number of epochs that can output a correct imaging region label with a higher probability and output a higher quality image based on the teacher data.

  One example of such a parameter group and a method of determining the number of epochs will be illustrated. First, 70% of the pair groups constituting the teacher data are used for training, and the remaining 30% are randomly set for evaluation. Next, the machine learning model is trained using the training pair group, and at the end of each epoch of training, the training evaluation value is calculated using the evaluation pair group. The training evaluation value is, for example, the average value of the value group evaluated by the loss function of the output when the input data forming each pair is input to the machine learning model during training and the output data corresponding to the input data. is there. Finally, the parameter group and the number of epochs when the training evaluation value becomes the smallest are determined as the parameter group and the number of epochs of the machine learning model. In this way, the machine learning model overtrains the pair group for training by determining the number of epochs by dividing the pair group forming the teacher data for training and for evaluation. Can be prevented.

  The high-quality image engine is a module that outputs a high-quality image obtained by improving the input low-quality image. Here, the term "higher image quality" in this specification means that an input image is converted into an image having an image quality more suitable for image diagnosis, and the high image quality is converted to an image having an image quality more suitable for image diagnosis. Image. In addition, the low-quality image is, for example, a two-dimensional image or a three-dimensional image acquired by X-ray imaging, CT, MRI, OCT, PET, SPECT, or the like, or a three-dimensional moving image of continuous CT. The image was taken without being set to achieve high image quality. Specifically, the low-quality image is, for example, an image acquired by low-dose imaging by an X-ray imaging apparatus or CT, imaging by MRI without using a contrast agent, short-time imaging by OCT, or low imaging. It includes OCTA images acquired by the number of times.

  Further, the content of image quality suitable for image diagnosis depends on what one wants to diagnose with various image diagnoses. Therefore, although it cannot be said unequivocally, for example, the image quality suitable for image diagnosis is low in noise, has high contrast, shows the shooting target in colors and gradations that are easy to observe, and has a large image size, Includes image quality such as high resolution. Further, it is possible to include an image quality in which an object or gradation that does not actually exist and which is drawn in the process of image generation is removed from the image.

  In addition, when a high-quality image with little noise or high contrast is used for blood vessel analysis processing of images such as OCTA and image analysis such as area segmentation processing of images such as CT and OCT, low-quality images are obtained. In many cases, the analysis can be performed more accurately than using it. Therefore, the high quality image output by the high quality engine may be useful not only for image diagnosis but also for image analysis.

  In the image processing method that constitutes the image quality improving method in the following embodiments, processing using various machine learning algorithms such as deep learning is performed. In addition, in the image processing method, in addition to the processing using the machine learning algorithm, various existing image filtering processing, matching processing using a database of high-quality images corresponding to similar images, and knowledge base image processing are available. You may perform the process of.

  In particular, the configuration shown in FIG. 1 is an example of the configuration of the CNN that improves the quality of a two-dimensional image. The CNN configuration includes a plurality of convolution processing block 100 groups. The convolution processing block 100 includes a convolution layer 101, a batch normalization layer 102, and an activation layer 103 using a normalized linear function (Rectifier Linear Unit). The CNN configuration also includes a merge (Merger) layer 104 and a final convolutional layer 105. The synthesizing layer 104 synthesizes the output value group of the convolution processing block 100 and the pixel value group forming the image by connecting or adding them. The final convolutional layer 105 outputs the pixel value group that is combined in the combining layer 104 and forms the high-quality image Im120. In such a configuration, the pixel value group forming the input image Im110 is output through the convolution processing block 100 group, and the pixel value group forming the input image Im110 is combined in the combining layer 104. Is synthesized. After that, the combined pixel value group is formed into the high-quality image Im120 at the final convolutional layer 105.

  Note that, for example, by setting the number of convolution processing blocks 100 to 16 and setting the kernel size of the filter to 3 pixels wide, 3 pixels high, and 64 filters as parameters of the convolutional layer 101 group, a constant high image quality is obtained. You can get the effect of. However, actually, as described in the description of the machine learning model above, it is possible to set a better parameter group by using the teacher data according to the usage pattern of the machine learning model. When it is necessary to process a three-dimensional image or a four-dimensional image, the kernel size of the filter may be expanded to three-dimensional or four-dimensional.

  When using some image processing methods such as image processing using CNN, it is necessary to pay attention to the image size. Specifically, it should be noted that different image sizes may be required for the input low-quality image and the output high-quality image in order to deal with the problem that the peripheral portion of the high-quality image is not sufficiently improved in image quality. Should be.

  For the sake of clear explanation, although not specified in the embodiments described below, if an image quality improvement engine that requires different image sizes for the image input to the image quality improvement engine and the output image is adopted, an appropriate image It is assumed that the size is adjusted. Specifically, padding is performed on an input image such as an image used as teacher data for training a machine learning model or an image input to the image quality enhancement engine, or a shooting area around the input image is set. Adjust the image size by combining them. The area to be padded is filled with a fixed pixel value, filled with a neighboring pixel value, or mirror-padded according to the characteristics of the image quality improving method so that the image quality can be effectively improved.

  Further, the image quality improving method may be implemented by only one image processing method, or may be implemented by combining two or more image processing methods. In addition, a plurality of high image quality method groups may be performed in parallel to generate a plurality of high quality image groups, and then the highest quality image may be finally selected as the high quality image. It should be noted that the selection of the highest quality image may be automatically performed using the image quality evaluation index, or a plurality of high quality image groups may be displayed on the user interface provided in an arbitrary display unit or the like. Then, it may be performed according to an instruction from an examiner (user).

  In some cases, the input image that has not been improved in image quality is more suitable for image diagnosis. Therefore, the input image may be added to the final image selection target. Further, parameters may be input to the high image quality engine together with the low image quality image. For example, a parameter that specifies the degree of image quality enhancement or a parameter that specifies the image filter size used in the image processing method may be input to the image quality enhancement engine together with the input image.

  The shooting location estimation engine is a module that estimates a shooting location and a shooting area of an input image. The image capturing position estimation engine determines where the image capturing region or image capturing region drawn in the input image is, or for each required detail level image capturing region label or image capturing region label, the probability of being the image capturing region or image capturing region. Can be output.

  Depending on the image capturing device, the image capturing part or the image capturing region may not be stored as image capturing conditions, or the image capturing device may not be able to acquire the image capturing region and may not store it. Further, even if the imaged region or the imaged region is stored, the imaged region or the imaged region at a necessary detail level may not be stored. For example, it is simply stored as "the posterior segment of the eye" as an imaged part, and in detail, it is "macular part", "optic papilla", or "macular and optic papilla", " There are times when I don't know if it is "other." In another example, only "breast" is stored as an imaged part, and it is not known in detail whether it is "right breast", "left breast", or "both". is there. Therefore, by using the imaging part estimation engine, the imaging part and the imaging region of the input image can be estimated in these cases.

  In the image and data processing method that constitutes the estimation method of the shooting location estimation engine, processing using various machine learning algorithms such as deep learning is performed. In addition, in the image and data processing method, in addition to or in place of the processing using the machine learning algorithm, any existing existing processing such as natural language processing, matching processing using a database of similar images and similar data, knowledge base processing, etc. The estimation process may be performed. Note that the teacher data for training the machine learning model constructed by using the machine learning algorithm can be an image labeled with the imaged region or the imaged region. In this case, the image of the teacher data is used as the input data, and the label of the imaged region or the imaged region is used as the output data.

  In particular, there is a configuration shown in FIG. 2 as a configuration example of the CNN that estimates the shooting location of the two-dimensional image. The configuration of the CNN includes a plurality of convolution processing blocks 200 including a convolutional layer 201, a batch normalization layer 202, and an activation layer 203 using a normalized linear function. Further, the CNN configuration includes a final convolutional layer 204, a full connection layer 205, and an output layer 206. The total combination layer 205 fully combines the output value groups of the convolution processing block 200. Further, the output layer 206 uses the Softmax function to output the probability of each assumed imaging region label for the input image Im 210 as an estimation result (Result) 207. In such a configuration, for example, if the input image Im210 is an image of a “macular part”, the highest probability is output for the “imaged part label corresponding to the macular part”.

  Note that, for example, the number of convolution processing blocks 200 is 16, the kernel size of the filter is 3 pixels wide, 3 pixels high, and the number of filters is 64 as parameters of the convolutional layer 201 group. The site can be estimated. However, actually, as described in the description of the machine learning model above, it is possible to set a better parameter group by using the teacher data according to the usage pattern of the machine learning model. When it is necessary to process a three-dimensional image or a four-dimensional image, the kernel size of the filter may be expanded to three-dimensional or four-dimensional. Note that the estimation method may be performed by only one image and data processing method, or may be performed by combining two or more image and data processing methods.

  The image quality evaluation engine is a module that outputs an image quality evaluation index for an input image. In the image quality evaluation processing method for calculating the image quality evaluation index, processing using various machine learning algorithms such as deep learning is performed. In the image quality evaluation processing method, any existing evaluation processing such as image noise measurement algorithm and matching processing using a database of image quality evaluation indexes corresponding to similar images and base images may be performed. Note that these evaluation processes may be performed in addition to or in place of the process using the machine learning algorithm.

  For example, the image quality evaluation index can be obtained from a machine learning model constructed using a machine learning algorithm. In this case, the input data of the pair forming the teacher data for training the machine learning model is an image group including a low image quality image group and a high image quality image group captured in advance under various image capturing conditions. Further, the output data of the pair forming the teacher data for training the machine learning model is, for example, the image quality evaluation index group set for each image group of the input data by the examiner who performs image diagnosis.

  The authenticity evaluation engine in the description of the present invention is a module that evaluates the drawing of an input image and evaluates with a certain degree of accuracy whether or not the image is captured and acquired by the target imaging device. In the authenticity evaluation processing method, processing using various machine learning algorithms such as deep learning is performed. In the authenticity evaluation processing method, any existing evaluation processing such as knowledge base processing may be performed in addition to or instead of the processing using the machine learning algorithm.

  For example, the authenticity evaluation process can be performed by a machine learning model constructed using a machine learning algorithm. First, the teacher data of the machine learning model will be described. The teacher data includes a pair group of a high-quality image group previously captured under various image capturing conditions and a label (hereinafter, a genuine label) indicating that the image is captured and acquired by the target image capturing apparatus. Further, the teacher data represents that a high-quality image group generated by inputting a low-quality image to the high-quality image engine (first-level high-quality image engine) and that the target image capturing apparatus has not captured and acquired the image data. A group of pairs with labels (hereinafter, counterfeit labels) is included. The machine learning model trained using such teacher data outputs a counterfeit label when a high-quality image generated by the first-level image quality improving engine is input.

  In particular, there is a configuration shown in FIG. 3 as a configuration example of the CNN that performs the authenticity evaluation processing of a two-dimensional image. The configuration of the CNN includes a plurality of convolution processing blocks 300 including a convolutional layer 301, a batch normalization layer 302, and an activation layer 303 using a normalized linear function. The CNN configuration also includes a final convolutional layer 304, a fully coupled layer 305, and an output layer 306. The fully connected layer 305 fully combines the output value groups of the convolution processing block 300. Further, the output layer 306 uses the sigmoid function to set a value of 1 (true) indicating a genuine label or a value of 0 (false) indicating a false label to the input image Im 310 as a result of the falsehood evaluation processing. (Result) 307 is output.

  Note that the number of convolution processing blocks 300 is 16, the filter kernel size is 3 pixels wide, 3 pixels high, and the number of filters is 64 as parameters of the convolutional layer 301 group. The processing result can be obtained. However, actually, as described in the description of the machine learning model above, it is possible to set a better parameter group by using the teacher data according to the usage pattern of the machine learning model. When it is necessary to process a three-dimensional image or a four-dimensional image, the kernel size of the filter may be expanded to three-dimensional or four-dimensional.

  The authenticity evaluation engine produces a genuine label when a high-quality image generated by a high-quality image engine (second-level image quality improvement engine) that produces higher image quality than the first-level image quality improvement engine is input. May be output. In other words, the authenticity evaluation engine cannot reliably evaluate whether or not the input image is an image captured and acquired by the image capturing device, but whether the input image is an image having image-likeness captured and acquired by the image capturing device. Can be evaluated. By using this characteristic, input the high quality image generated by the high quality image generation engine to the authenticity evaluation engine, and evaluate whether the high quality image generated by the high quality image generation engine is sufficiently high quality or not. it can.

  Further, the machine learning model of the image quality improving engine and the machine learning model of the authenticity evaluation engine may be trained in cooperation with each other to improve the efficiency and accuracy of both engines. In this case, first, the machine learning model of the image quality improving engine is trained so that the authenticity evaluation engine outputs a high quality image generated by the image quality improving engine so that a true label is output. Further, in parallel, the machine learning model of the authentication evaluation engine is trained so that the authentication label is output when the authentication evaluation engine evaluates the image generated by the image quality enhancement engine. Further, in parallel, the machine learning model of the authenticity evaluation engine is trained so that the authenticity evaluation engine outputs an authentic label when the image acquired by the image capturing apparatus is evaluated. This improves the efficiency and accuracy of the image quality improvement engine and the authenticity evaluation engine.

<First Embodiment>
Hereinafter, the medical image processing apparatus according to the first embodiment will be described with reference to FIGS. 4 and 5. FIG. 4 shows an example of a schematic configuration of the image processing apparatus according to this embodiment.

  The image processing device 400 is connected to the imaging device 10 and the display unit 20 via a circuit or a network. Further, the image capturing device 10 and the display unit 20 may be directly connected. Although these devices are separate devices in the present embodiment, some or all of these devices may be integrally configured. Further, these devices may be connected to any other device via a circuit or a network, or may be configured integrally with any other device.

  The image processing apparatus 400 is provided with an acquisition unit 401, a shooting condition acquisition unit 402, an image quality improvement possibility determination unit 403, an image quality improvement unit 404, and an output unit 405 (display control unit). The image processing apparatus 400 may be composed of a plurality of devices provided with some of these components. The acquisition unit 401 can acquire various data and images from the imaging device 10 and other devices, and can acquire input from an examiner via an input device (not shown). A mouse, a keyboard, a touch panel, or any other input device may be used as the input device. Further, the display unit 20 may be configured as a touch panel display.

  The imaging condition acquisition unit 402 acquires the imaging conditions of the medical image (input image) acquired by the acquisition unit 401. Specifically, the imaging condition group stored in the data structure forming the medical image is acquired according to the data format of the medical image. If the medical image does not store the shooting conditions, the shooting unit 401 can acquire the shooting information group including the shooting condition group from the shooting apparatus 10 or the image management system.

  The image quality improvement possibility determination unit 403 determines whether or not the medical image can be dealt with by the image quality improvement unit 404 using the imaging condition group acquired by the imaging condition acquisition unit 402. The image quality enhancement unit 404 enhances the image quality of a medical image that can be dealt with, and generates a high quality image suitable for image diagnosis. The output unit 405 causes the display unit 20 to display the high-quality image generated by the high-quality image generation unit 404, the input image, various information, and the like. The output unit 405 may store the generated high-quality image or the like in a storage device connected to the image processing device 400.

  Next, the image quality improving unit 404 will be described in detail. The image quality improving unit 404 includes an image quality improving engine. The image quality improving method provided in the image quality improving engine according to the present embodiment performs processing using a machine learning algorithm.

  In the present embodiment, in training of a machine learning model according to a machine learning algorithm, input data that is a low-quality image having a specific shooting condition assumed as a processing target and output data that is a high-quality image corresponding to the input data. The teacher data composed of a pair group of is used. It should be noted that the specific shooting condition specifically includes a predetermined shooting site, shooting method, shooting angle of view, image size, and the like.

  In the present embodiment, the input data of the teacher data is a low-quality image acquired by the same model as the image capturing apparatus 10 and the same settings as the image capturing apparatus 10. The output data of the teacher data is a high-quality image acquired by the setting and image processing provided in the same model as the image capturing apparatus 10. Specifically, the output data is, for example, a high-quality image (superimposed image) obtained by performing superimposing processing such as averaging on a group of images (original images) acquired by photographing multiple times. is there. Here, the high-quality image and the low-quality image will be described using OCTA motion contrast data as an example. Here, the motion contrast data is data used in OCTA or the like, which is obtained by repeatedly photographing the same portion of the photographing target and detecting a temporal change of the photographing target during the photographing. At this time, an en-face image of OCTA (motion contrast front image) is generated by generating a front image using data of a desired range in the depth direction of the imaging target among the calculated motion contrast data. You can It should be noted that in the following, repetitive imaging of OCT data at the same location will be referred to as NOR (Number Of Repeat).

  In the present embodiment, two different methods will be described with reference to FIG. 28 as examples of generating a high-quality image and a low-quality image by superimposing processing.

  The first method will be described with reference to FIG. 28A with regard to motion contrast data generated from OCT data obtained by repeatedly photographing the same portion of a photographing target as an example of a high-quality image. In FIG. 28A, Im2810 indicates three-dimensional motion contrast data, and Im2811 indicates two-dimensional motion contrast data forming the three-dimensional motion contrast data. Then, Im2811-1 to Im2811-1 indicate OCT tomographic images (B scan) for generating Im2811. Here, NOR means the number of OCT tomographic images in Im2811-1 to Im2811-1-3 in FIG. 28A, and NOR is 3 in the example of the figure. Im2811-1 to Im2813-1 are photographed at predetermined time intervals (Δt). Note that the same location means one line in the front direction (X-Y) of the eye to be inspected, and corresponds to the location of Im2811 in FIG. 28 (a). Since the motion contrast data is data in which a temporal change is detected, it is necessary to set the NOR at least twice in order to generate this data. For example, when NOR is 2, one motion contrast data is generated. When the NOR is 3, when the motion contrast data is generated only by the OCT of the adjacent time intervals (first and second times, second and third times), two data are generated. When motion contrast data is also generated using OCT data at distant time intervals (first time and third time), a total of three data is generated. That is, when NOR is increased three times, four times, ..., The number of motion contrast data at the same location also increases. It is possible to generate high-quality motion contrast data by aligning a plurality of motion contrast data obtained by repeatedly photographing the same location and performing an overlay process such as an averaging process. Therefore, it is desirable to set NOR to at least 3 times or more and 5 times or more. On the other hand, as an example of a low-quality image corresponding to this, the motion contrast data before the superimposing process such as the averaging is performed. In this case, it is desirable that the low image-quality image be used as a reference image when performing overlay processing such as arithmetic mean. When performing the overlay processing, if the position and shape of the target image are deformed and aligned with respect to the reference image, there is almost no spatial misalignment between the reference image and the image after the overlay processing. . Therefore, a low-quality image and a high-quality image can be easily paired. It should be noted that the target image, which has undergone the image transformation process for alignment, may be the low-quality image, instead of the reference image. By using each of the original image groups (reference image and target image) as input data and the corresponding superimposed image as output data, a plurality of pair groups can be generated. For example, in the case of obtaining one superimposed image from 15 original image groups, a pair of the first original image of the original image group and the superimposed image, a second original image of the original image group, A pair with the superimposed image can be generated. In this way, when one superposed image is obtained from the fifteen original image group, one pair of the original image group and fifteen pairs of superposed images can be generated. Note that three-dimensional high-quality image data can be generated by repeatedly photographing the same portion in the main scanning (X) direction and scanning it while shifting it in the sub scanning (Y) direction.

  The second method will be described with reference to FIG. 28 (b), regarding processing for generating a high-quality image by superimposing motion contrast data obtained by shooting the same region of the shooting target multiple times. In addition, the same region refers to a region such as 3 × 3 mm or 10 × 10 mm in the front direction (X-Y) of the eye to be inspected, and includes three-dimensional motion contrast data including the depth direction of the tomographic image. Means to get. When the same region is photographed a plurality of times and the overlapping process is performed, it is desirable to set NOR to 2 or 3 times in order to shorten the photographing per time. Further, in order to generate high-quality three-dimensional motion contrast data, at least two or more pieces of three-dimensional data in the same area are acquired. FIG. 28B shows an example of a plurality of three-dimensional motion contrast data. Im2820 to Im2840 are three-dimensional motion contrast data as described with reference to FIG. Using the three-dimensional motion contrast data of two or more data, alignment processing in the front direction (X-Y) and depth direction (Z) is performed, and the data that becomes an artifact in each data is excluded, and then the averaging processing is performed. I do. This makes it possible to generate one high-quality three-dimensional motion contrast data from which artifacts are excluded. A high quality image is obtained by generating an arbitrary plane from the three-dimensional motion contrast data. On the other hand, it is preferable that the low-quality image corresponding to this is an arbitrary plane generated from the reference data when performing the overlay processing such as the averaging. As described in the first method, there is almost no spatial positional deviation between the reference image and the image after the addition and averaging, so that a low-quality image and a high-quality image can be easily paired. It should be noted that any plane generated from the target data that has undergone the image transformation process of alignment instead of the reference data may be the low image quality image.

  In the first method, the burden on the subject is small because the shooting itself is completed once. However, as the number of NORs increases, the shooting time for one shooting becomes longer. In addition, a good image is not always obtained when an artifact such as cloudiness of the eyes or eyelashes is included in the shooting. In the second method, the burden on the subject is slightly increased because the images are taken multiple times. However, one shooting time is short, and even if an artifact is included in one shooting, if the artifact is not captured in another shooting, a beautiful image with few artifacts can be finally obtained. Considering these characteristics, when collecting data, an arbitrary method is selected according to the subject's situation.

  Although the present embodiment has been described by taking the motion contrast data as an example, the present invention is not limited to this. Since the OCT data is captured to generate the motion contrast data, the same method can be applied to the OCT data by the above method. Further, although the description of the tracking process is omitted in the present embodiment, it is desirable to perform imaging while tracking the eye to be inspected because the same area or the same area of the eye to be inspected is imaged.

  In the present embodiment, since a pair of three-dimensional high-quality image data and low-quality image data is created, an arbitrary pair of two-dimensional images can be generated from this pair. This will be described with reference to FIG. For example, when the target image is an OCTA En-Face image, an OCTA En-Face image is generated in a desired depth range from the three-dimensional data. The desired depth range refers to the range in the Z direction in FIG. An example of the OCTA En-Face image generated here is shown in FIG. As an OCTA En-Face image, learning is performed using OCTA En-Face images generated in different depth ranges such as a surface layer (Im2910), a deep layer (Im2920), an outer layer (Im2930), and a choroidal vascular network (Im2940). . Note that the types of OCTA En-Face images are not limited to this, and the types may be increased by generating OCTA En-Face images in which different depth ranges are set by changing the reference layer and the offset value. When performing learning, learning may be performed separately for each En-Face image of OCTA having a different depth, or a plurality of images in different depth ranges may be combined (for example, divided on the surface side and the deep side). Alternatively, the En-Face images of OCTA in all depth ranges may be learned together. Also in the case of the En-Face image of the brightness generated from the OCT data, similarly to the En-Face of OCTA, learning is performed using a plurality of En-Face images generated from an arbitrary depth range. For example, consider a case where the image quality improving engine includes a machine learning engine obtained by using learning data including a plurality of motion contrast front images corresponding to different depth ranges of the eye to be inspected. At this time, the acquisition unit can acquire, as the first image, a motion contrast front image corresponding to a part of the depth ranges of a long depth range including different depth ranges. That is, a motion contrast front image corresponding to a depth range different from the plurality of depth ranges corresponding to the plurality of motion contrast front images included in the learning data can be used as an input image for high image quality. Of course, a motion contrast front image in the same depth range as during learning may be used as the input image for high image quality. Further, a part of the depth range may be set according to the examiner pressing an arbitrary button on the user interface, or may be set automatically. Note that the above-described contents are not limited to the motion contrast front image, and can be applied to, for example, an En-Face image of brightness.

  When the image to be processed is a tomographic image, learning is performed using an OCT tomographic image that is a B scan or a tomographic image of motion contrast data. This will be described with reference to FIG. In FIG. 29B, Im2951 to Im2953 are tomographic images of OCT. The images are different in FIG. 29B because they show tomographic images at different positions in the sub-scanning (Y) direction. In the tomographic image, the learning may be performed together without worrying about the difference in the position in the sub-scanning direction. However, in the case of an image taken at a place where the imaged part (for example, the center of the macula, the center of the optic papilla) is imaged, learning may be done separately for each part, or the imaged part may be taken into consideration. You may also study together. Since the image feature amounts of the OCT tomographic image and the tomographic image of the motion contrast data are greatly different, it is better to perform learning separately.

  The superimposed image subjected to the superimposing processing is a high-quality image suitable for image diagnosis because the pixels commonly drawn in the original image group are emphasized. In this case, the generated high-quality image is a high-contrast image in which the difference between the low-luminance region and the high-luminance region is clear as a result of the commonly drawn pixels being emphasized. In addition, for example, in the superimposed image, random noise that occurs each time the image is captured can be reduced, or a region that was not drawn well in the original image at a certain point can be interpolated by another original image group.

  Further, when the input data of the machine learning model needs to be composed of a plurality of images, a required number of original image groups can be selected from the original image group and used as the input data. For example, in the case of obtaining one superposed image from 15 original image groups, if 2 images are required as input data of the machine learning model, 105 (15C2 = 105) pair groups can be generated.

  It should be noted that, of the pair group forming the teacher data, the pair that does not contribute to the high image quality can be removed from the teacher data. For example, if the high-quality image that is the output data forming the pair of teacher data has an image quality that is not suitable for image diagnosis, the image output by the high-quality image learning engine learned using the teacher data is also suitable for image diagnosis. There is a possibility that the image quality will not be obtained. Therefore, it is possible to reduce the possibility that the image quality improving engine will generate an image with an image quality not suitable for image diagnosis by removing from the teacher data a pair whose output data has an image quality not suitable for image diagnosis.

  Further, when the average brightness and the brightness distribution of the pair of images are significantly different, the image quality improving engine learned by using the teacher data is an image not suitable for image diagnosis having a brightness distribution significantly different from the low image quality image. May be output. Therefore, a pair of input data and output data having a large difference in average luminance or luminance distribution can be removed from the teacher data.

  Further, when the structure or position of the shooting target drawn in the pair of images is significantly different, the image quality improvement engine learned using the teacher data causes the shooting target to have a structure or position significantly different from the low quality image. There is a possibility to output an image that is not suitable for image diagnosis. Therefore, a pair of input data and output data, which differ greatly in the structure or position of the imaged object to be drawn, can be removed from the teacher data. Further, the high quality image engine can be configured not to use the high quality image output by itself as the teacher data from the viewpoint of maintaining the quality.

  By using the image quality improving engine that has been machine-learned in this way, the image quality improving unit 404 can increase the contrast and noise by superimposing processing when a medical image acquired in one shooting is input. It is possible to output a high-quality image that has undergone reduction and the like. Therefore, the image quality improving unit 404 can generate a high quality image suitable for image diagnosis based on the low quality image which is the input image.

  Next, a series of image processing according to the present embodiment will be described with reference to the flowchart of FIG. FIG. 5 is a flowchart of a series of image processing according to this embodiment. First, when the series of image processes according to the present embodiment is started, the process proceeds to step S510.

  In step S510, the acquisition unit 401 acquires an image captured by the image capturing apparatus 10 as an input image from the image capturing apparatus 10 connected via a circuit or a network. The acquisition unit 401 may acquire the input image in response to a request from the imaging device 10. Such a request is, for example, when the image capturing apparatus 10 generates an image, before or after saving the image generated by the image capturing apparatus 10 in a recording device included in the image capturing apparatus 10, the saved image is displayed on the display unit 20. It may be issued when a high-quality image is used for image analysis processing, or the like.

  In addition, the acquisition unit 401 may acquire data for generating an image from the image capturing apparatus 10 and acquire an image generated by the image processing apparatus 400 based on the data as an input image. In this case, as an image generation method for the image processing apparatus 400 to generate various images, any existing image generation method may be adopted.

  In step S520, the shooting condition acquisition unit 402 acquires a shooting condition group of the input image. Specifically, the shooting condition group stored in the data structure forming the input image is acquired according to the data format of the input image. As described above, when the shooting condition is not stored in the input image, the shooting condition acquisition unit 402 acquires a shooting information group including a shooting condition group from the shooting device 10 or an image management system (not shown). be able to.

  In step S530, the image quality improvement possibility determination unit 403 determines whether or not the image quality of the input image can be improved by the image quality improvement engine provided in the image quality improvement unit 404 using the acquired shooting condition group. To do. Specifically, the image quality improvement availability determination unit 403 determines whether or not the imaging region, imaging method, imaging angle of view, and image size of the input image match the conditions that can be handled by the image quality improvement engine. .

  The image quality improvement availability determination unit 403 determines all the shooting conditions, and when it is determined that it is possible to deal with the processing, the process proceeds to step S540. On the other hand, when the image quality improvement determination unit 403 determines that the image quality improvement engine cannot handle the input image based on these shooting conditions, the process proceeds to step S550.

  It should be noted that, depending on the settings and implementation of the image processing apparatus 400, even if it is determined that the input image cannot be processed based on a part of the imaging region, the imaging method, the imaging angle of view, and the image size, The image quality enhancement process in step S540 may be performed. For example, it is assumed that the high image quality engine can comprehensively support any imaging part of the subject, and it is possible to cope even if the input data includes an unknown imaging part. Such a process may be performed when it is installed in. In addition, the image quality improvement possibility determination unit 403 determines that at least one of the image capturing part, the image capturing method, the image capturing angle of view, and the image size of the input image can be handled by the image quality enhancing engine according to the desired configuration. You may judge whether it corresponds.

  In step S540, the image quality improving unit 404 uses the image quality improving engine to improve the image quality of the input image, and generates a high image quality image more suitable for image diagnosis than the input image. Specifically, the image quality improving unit 404 inputs the input image to the image quality improving engine and generates a high quality image with high image quality. The image quality improving engine generates a high quality image that is obtained by performing a superimposing process using an input image based on a machine learning model in which machine learning is performed using teacher data. Therefore, the image quality improving engine can generate a high quality image in which noise is reduced or contrast is emphasized more than the input image.

  It should be noted that, depending on the settings and implementation of the image processing apparatus 400, the image quality improving unit 404 adjusts the degree of image quality improvement by inputting parameters together with the input image to the image quality improving engine according to the shooting condition group. You may. Further, the image quality improving unit 404 may input a parameter according to the input of the examiner to the image quality improving engine together with the input image to adjust the degree of image quality improvement and the like.

  In step S550, the output unit 405 outputs the high-quality image and displays it on the display unit 20 if the high-quality image is generated in step S540. On the other hand, if it is determined in step S530 that the image quality enhancement process cannot be performed, the input image is output and displayed on the display unit 20. Note that the output unit 405 may display or store the output image on the imaging device 10 or another device, instead of displaying the output image on the display unit 20. In addition, the output unit 405 processes the output image so that it can be used by the image capturing apparatus 10 or another apparatus, or outputs the data so that it can be transmitted to the image management system or the like, depending on the setting of the image processing apparatus 400 or the mounting mode. You may convert the format.

  As described above, the image processing device 400 according to this embodiment includes the acquisition unit 401 and the image quality improvement unit 404. The acquisition unit 401 acquires an input image (first image) that is an image of a predetermined region of the subject. The image quality improving unit 404 uses the image quality improving engine including the machine learning engine to perform high quality image (second image) in which at least one of noise reduction and contrast enhancement is performed on the input image as compared with the input image. ) Is generated. The image quality improving engine includes a machine learning engine in which an image obtained by the superposition processing is used as learning data.

  With this configuration, the image processing apparatus 400 according to the present embodiment can output a high-quality image in which noise is reduced or contrast is emphasized from the input image. For this reason, the image processing apparatus 400 makes an image suitable for image diagnosis such as a clearer image or an image in which a region to be observed or a lesion is emphasized more invasive to the photographer and the subject than the conventional image. It can be obtained at a lower price without increasing or increasing labor.

  The image processing apparatus 400 further includes an image quality improvement possibility determination unit 403 that determines whether or not a high quality image can be generated using an image quality enhancement engine for an input image. The image quality improvement determination unit 403 makes the determination based on at least one of the imaged region of the input image, the image capturing method, the image capturing angle of view, and the image size.

  With this configuration, the image processing apparatus 400 according to the present embodiment can omit an input image that cannot be processed by the image quality enhancement unit 404 from the image quality enhancement processing, and reduce the processing load of the image processing apparatus 400 and the occurrence of errors. be able to.

  Although the output unit 405 (display control unit) is configured to display the generated high-quality image on the display unit 20 in the present embodiment, the operation of the output unit 405 is not limited to this. For example, the output unit 405 can output the high-quality image to another device connected to the imaging device 10 or the image processing device 400. Therefore, the high-quality image can be displayed on the user interface of these devices, stored in any recording device, used for any image analysis, or transmitted to the image management system.

  In this embodiment, the image quality improvement determination unit 403 determines whether or not the input image can be image-quality-improved by the image quality improvement engine, and if the input image can be image-quality-improved, the image quality is improved. The section 404 improved the image quality. On the other hand, in the case where the image capturing apparatus 10 performs image capturing only under image capturing conditions capable of achieving high image quality, the image acquired from the image capturing apparatus 10 may be unconditionally made high in image quality. In this case, as shown in FIG. 6, the processes of steps S520 and S530 can be omitted and step S540 can be performed after step S510.

  In the present embodiment, the output unit 405 is configured to display a high quality image on the display unit 20. However, the output unit 405 may display the high-quality image on the display unit 20 according to an instruction from the examiner. For example, the output unit 405 may display the high-quality image on the display unit 20 in response to the examiner pressing an arbitrary button on the user interface of the display unit 20. In this case, the output unit 405 may switch the input image to display the high quality image, or may display the high quality image side by side with the input image.

  Further, the output unit 405, when displaying the high quality image on the display unit 20, displays a display indicating that the displayed image is the high quality image generated by the process using the machine learning algorithm. May be displayed together with. In this case, the user can easily identify by the display that the displayed high-quality image is not the image itself acquired by shooting, and therefore, it is possible to reduce false diagnosis or improve diagnosis efficiency. You can Note that the display indicating that the high-quality image generated by the process using the machine learning algorithm is in any form as long as the display can distinguish the input image from the high-quality image generated by the process. It may be one.

  Further, the output unit 405 displays a display indicating what kind of teacher data the machine learning algorithm has learned about the display indicating that the image is a high-quality image generated by the process using the machine learning algorithm. May be displayed on the display unit 20. The display may include a description of the types of the input data and the output data of the teacher data, and an arbitrary display regarding the teacher data such as the imaging region included in the input data and the output data.

  In the image quality improving engine according to the present embodiment, the superimposed image is used as the output data of the teacher data, but the teacher data is not limited to this. As the output data of the teacher data, a high-quality image obtained by performing at least one of a superimposing process, a process group described later, and a photographing method described later, which is a means for obtaining a high-quality image, may be used. .

  For example, as the output data of the teacher data, a high-quality image obtained by performing the maximum posterior probability estimation process (MAP estimation process) on the original image group may be used. In the MAP estimation processing, a likelihood function is obtained from the probability density of each pixel value in a plurality of low-quality images, and a true signal value (pixel value) is estimated using the obtained likelihood function.

  The high-quality image obtained by the MAP estimation process becomes a high-contrast image based on the pixel value close to the true signal value. In addition, since the estimated signal value is obtained based on the probability density, noise that is randomly generated is reduced in the high-quality image obtained by the MAP estimation process. Therefore, by using the high-quality image obtained by the MAP estimation process as the teacher data, the high-quality image engine can reduce the noise from the input image and have a high contrast, which is suitable for image diagnosis. A high quality image can be generated. A method of generating a pair of input data and output data of the teacher data may be the same as the method of using the superimposed image as the teacher data.

  Alternatively, a high-quality image obtained by applying a smoothing filter process to the original image may be used as the output data of the teacher data. In this case, the image quality improving engine can generate a high quality image with reduced random noise from the input image. Further, an image obtained by applying the gradation conversion process to the original image may be used as the output data of the teacher data. In this case, the image quality enhancement engine can generate a high-quality image with contrast enhancement from the input image. A method of generating a pair of input data and output data of the teacher data may be the same as the method of using the superimposed image as the teacher data.

  It should be noted that the input data of the teacher data may be an image acquired from a photographing device having the same image quality tendency as the photographing device 10. The output data of the teacher data may be a high-quality image obtained by high-cost processing such as the successive approximation method, or the subject corresponding to the input data may have higher performance than the imaging device 10. It may be a high-quality image acquired by shooting with a shooting device. Furthermore, the output data may be a high-quality image acquired by performing noise reduction processing based on a rule. Here, the noise reduction process can include, for example, a process of replacing a high-luminance pixel of only one pixel, which is apparently noise appearing in the low-luminance region, with an average value of neighboring low-luminance pixel values. . For this reason, the high image quality engine processes an image captured by an image capturing device having a higher performance than the image capturing device used to capture the input image, or an image captured in a capturing process that requires more man-hours than the capturing process of the input image. It may be used as learning data. For example, when a motion contrast front image is used as an input image, the high image quality engine uses an image obtained by OCTA imaging by an OCT imaging device having a higher performance than an OCT imaging device used for OCTA imaging of the input image, or the input image. The images acquired and obtained in the OCTA imaging process, which requires more man-hours than the OCTA imaging process in step S1, may be used as the learning data.

  Although omitted in the description of this embodiment, a high-quality image generated from a plurality of images, which is used as output data of teacher data, can be generated from a plurality of images that have already been aligned. As the alignment processing, for example, one of a plurality of images is selected as a template, the similarity with other images is obtained while changing the position and angle of the template, and the amount of positional deviation from the template is obtained. Each image may be corrected based on the amount of displacement. Further, any other existing alignment processing may be performed.

  When aligning a 3D image, the 3D image is decomposed into a plurality of 2D images, and the 3D images are aligned by integrating the 2D images. Good. Further, the two-dimensional images may be aligned by decomposing the two-dimensional images into one-dimensional images and integrating the one-dimensional images aligned with each other. Note that these alignments may be performed not on the image but on the data for generating the image.

  In addition, in the present embodiment, when the image quality improvement determination unit 403 determines that the image quality improvement unit 404 can handle the input image, the process proceeds to step S540, and the image quality improvement unit 404 improves the image quality. The process has started. On the other hand, the output unit 405 may cause the display unit 20 to display the determination result of the image quality improvement determination unit 403, and the image quality enhancement unit 404 may start the image quality enhancement process in response to an instruction from the examiner. . At this time, the output unit 405 can cause the display unit 20 to display the imaging conditions such as the input image and the imaging site acquired for the input image together with the determination result. In this case, the image quality improving process is performed after the examiner determines whether or not the determination result is correct, so that the image quality improving process based on the erroneous determination can be reduced.

  Further, the image quality improvement possibility determination unit 403 does not perform the determination, and the output unit 405 causes the display unit 20 to display the imaging conditions such as the input image and the imaging site acquired for the input image. The image quality improving process may be started in response to the instruction.

<Second Embodiment>
Next, an image processing apparatus according to the second embodiment will be described with reference to FIGS. In the first embodiment, the image quality improving unit 404 includes one image quality improving engine. On the other hand, in the present embodiment, the image quality improving unit includes a plurality of image quality improving engines that perform machine learning using different teacher data, and generates a plurality of high quality images for the input image.

  Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 400 according to the first embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below focusing on the differences from the image processing apparatus according to the first embodiment. Since the configuration of the image processing apparatus according to the present embodiment is similar to that of the image processing apparatus according to the first embodiment, the same reference numerals are used for the configuration illustrated in FIG. 4, and description thereof will be omitted. To do.

  The image quality improving unit 404 according to the present embodiment includes two or more image quality improving engines in which machine learning is performed using different teacher data. Here, a method of creating the teacher data group according to the present embodiment will be described. Specifically, first, a pair group of an original image as input data and a superimposed image as output data in which various imaging regions are imaged is prepared. Next, the teacher data group is created by grouping the pair groups for each imaging region. For example, the first teacher data composed of a pair group acquired by photographing the first imaged region and the second teacher data composed of a pair group acquired by photographing the second imaged region. In this way, the teacher data group is created.

  After that, machine learning is performed by different image quality improving engines using each teacher data. For example, a first image quality improvement engine corresponding to a machine learning model trained with first teacher data, a second image quality improvement engine corresponding to a machine learning model trained with second teacher data, and so on. Prepare a high-quality image engine group.

  Since such high image quality engine has different teacher data used for training the corresponding machine learning model, the degree to which the high quality of the input image can be achieved differs depending on the image capturing conditions of the image input to the high image quality engine. . Specifically, the first image-quality improving engine has a high degree of image quality improvement with respect to the input image acquired by capturing the first imaged region, and acquires the image by capturing the second imaged region. The degree of high image quality of the captured image is low. Similarly, the second image quality improving engine has a high degree of image quality improvement with respect to the input image acquired by capturing the second imaged region, and is acquired by capturing the first imaged region. The degree of image quality improvement is low for images.

  Since each of the teacher data is made up of a pair group that is grouped according to the imaged site, the image quality tendencies of the image groups forming the pair group are similar. Therefore, the high image quality engine can perform higher image quality more effectively than the high image quality engine according to the first embodiment as long as it is a corresponding imaging region. Note that the shooting conditions for grouping pairs of teacher data are not limited to the shooting region, and may be the shooting angle of view, the image resolution, or a combination of two or more of these. Good.

  Hereinafter, a series of image processing according to this embodiment will be described with reference to FIG. 7. FIG. 7 is a flowchart of a series of image processing according to this embodiment. Note that the processes of steps S710 and S720 are the same as steps S510 and S520 according to the first embodiment, and thus description thereof will be omitted. When the image quality of the input image is unconditionally improved, the process of step S730 may be omitted after the process of step S720 and the process may proceed to step S740.

  When the shooting condition of the input image is acquired in step S720, the process proceeds to step S730. In step S730, the image quality improvement availability determination unit 403 can handle the input image by using one of the image quality improvement engine groups included in the image quality improvement unit 404 using the shooting condition group acquired in step S720. Or not.

  When the image quality improvement determination unit 403 determines that none of the image quality improvement engine groups can handle the input image, the process proceeds to step S760. On the other hand, when the image quality improvement availability determination unit 403 determines that one of the image quality enhancement engine groups can handle the input image, the process proceeds to step S740. Note that, depending on the setting and implementation of the image processing apparatus 400, even if the image quality improving engine determines that some image capturing conditions cannot be handled, as in the first embodiment, step S740 is performed. You may.

  In step S740, the image quality enhancement unit 404 performs the image quality enhancement process from the image quality enhancement engine group based on the shooting condition of the input image acquired in step S720 and the information of the teacher data of the image quality enhancement engine group. Select the image quality engine. Specifically, for example, with respect to the imaged part of the imaged condition group acquired in step S720, there is teacher data information about the imaged part or surrounding imaged parts, and the image quality is high. Select an optimization engine. In the above example, when the imaged part is the first imaged part, the image quality improving unit 404 selects the first image quality improving engine.

  In step S750, the image quality improving unit 404 uses the image quality improving engine selected in step S740 to generate a high quality image obtained by improving the quality of the input image. Then, in step S760, the output unit 405 outputs the high-quality image and displays it on the display unit 20 if the high-quality image is generated in step S750. On the other hand, if it is determined in step S730 that the image quality enhancement process cannot be performed, the input image is output and displayed on the display unit 20. The output unit 405 may display, when displaying the high-quality image on the display unit 20, that the high-quality image is generated by using the high-quality image engine selected by the high-quality image generation unit 404. .

  As described above, the image quality improving unit 404 according to the present embodiment includes a plurality of image quality improving engines that perform learning using different learning data. Here, each of the plurality of image quality enhancement engines performs learning by using different learning data regarding at least one of the imaging region, the imaging angle of view, the front image with different depth, and the resolution of the image. Is. The image quality improving unit 404 generates a high quality image by using an image quality improving engine according to at least one of a captured region of an input image, a captured field angle, a front image with different depths, and image resolution. .

  With such a configuration, the image processing device 400 according to the present embodiment can generate a more effective high quality image.

  In the present embodiment, the image quality improving unit 404 selects the image quality improving engine to be used for the image quality improving process based on the shooting condition of the input image, but the image quality improving engine selection process is not limited to this. For example, the output unit 405 causes the captured image of the acquired input image and the group of image quality enhancement engines to be displayed on the user interface of the display unit 20, and the image quality enhancement unit 404 performs the image quality enhancement process in response to an instruction from the examiner. The image quality enhancement engine used for the above may be selected. The output unit 405 may cause the display unit 20 to display the information of the teacher data used for learning of each image quality improvement engine together with the image quality improvement engine group. The display mode of the information of the teacher data used for learning of the high image quality engine may be arbitrary. For example, even if the name of the teacher data used for learning is used to display the high image quality engine group. Good.

  Further, the output unit 405 may display the image quality improving engine selected by the image quality improving unit 404 on the user interface of the display unit 20 and receive an instruction from the examiner. In this case, the image quality improving unit 404 may determine whether or not to finally select the image quality improving engine as the image quality improving engine used for the image quality improving processing, in response to an instruction from the examiner. .

  Note that the output unit 405 may output the generated high-quality image to another device connected to the image capturing device 10 or the image processing device 400, as in the first embodiment. Further, the output data of the teacher data of the image quality improving engine is not limited to the high quality image subjected to the superimposing process, as in the first embodiment. That is, at least one of a processing group and an imaging method such as superposition processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing. You may use the high quality image obtained by performing.

<Third Embodiment>
Next, an image processing device according to the third embodiment will be described with reference to FIGS. In the first and second embodiments, the shooting condition acquisition unit 402 acquires a shooting condition group from the data structure of the input image or the like. On the other hand, in the present embodiment, the imaging condition acquisition unit estimates the imaging site or imaging region of the input image based on the input image by using the imaging site estimation engine.

  Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 400 according to the second embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below focusing on the differences from the image processing apparatus according to the second embodiment. Note that the configuration of the image processing apparatus according to the present embodiment is similar to the configuration of the image processing apparatus according to the first and second embodiments, so the configuration illustrated in FIG. Is omitted.

  The shooting condition acquisition unit 402 according to the present embodiment includes a shooting position estimation engine that estimates a shooting region or a shooting region drawn in the input image acquired by the acquisition unit 401. In the method of estimating a shooting location included in the shooting location estimation engine according to the present embodiment, an estimation process using a machine learning algorithm is performed.

  In the present embodiment, the training of the machine learning model related to the imaging part estimation method using the machine learning algorithm is performed by using input data that is an image and output data that is an imaging part label or imaging region label corresponding to the input data. Teacher data composed of pairs are used. Here, the input data is an image having specific shooting conditions assumed as a processing target (input image). The input data is preferably an image acquired from an image capturing apparatus having the same image quality tendency as the image capturing apparatus 10, and more preferably the same model with the same settings as the image capturing apparatus 10. The type of the imaged part label or the imaged region label that is the output data may be the imaged part or the imaged region whose input data includes at least a part. The type of the imaging part label that is the output data may be, for example, in the case of OCT, “macular part”, “optic nerve head”, “macular part and optic nerve head”, and “other”.

  The learning part estimation engine according to the present embodiment can output the imaging part and the imaging region drawn in the input image by performing the learning using the teacher data. . Further, the imaged part estimation engine can output the probability of being the imaged part or the imaged region for each of the imaged part labels and the imaged region labels of a required detail level. By using the shooting location estimation engine, the shooting condition acquisition unit 402 can estimate the shooting site or shooting area of the input image based on the input image, and obtain the shooting condition or the shooting condition for the input image. When the imaging location estimation engine outputs the probability of being the imaging area or imaging area for each imaging area label or imaging area label, the imaging condition acquisition unit 402 determines the imaging area or imaging area with the highest probability. It is acquired as the shooting condition of the input image.

  Next, similar to the second embodiment, a series of image processing according to the present embodiment will be described with reference to the flowchart of FIG. 7. Note that the processes of step S710 and steps S730 to S760 according to the present embodiment are the same as those of the second embodiment, so description thereof will be omitted. When the image quality of the input image is unconditionally improved, the process of step S730 may be omitted after the process of step S720 and the process may proceed to step S740.

  When the input image is acquired in step S710, the process proceeds to step S720. In step S720, the shooting condition acquisition unit 402 acquires the shooting condition group of the input image acquired in step S710.

  Specifically, the shooting condition group stored in the data structure forming the input image is acquired according to the data format of the input image. Further, when the image capturing condition group does not include information about the image capturing site or the image capturing region, the image capturing condition acquiring unit 402 inputs the input image to the image capturing site estimating engine, and the input image is acquired by capturing which image capturing site. Estimate what it is. Specifically, the imaging condition acquisition unit 402 inputs the input image to the imaging part estimation engine, evaluates the probability output to each of the imaging part label groups, and determines the imaging part having the highest probability as the input image. Set / get as shooting conditions.

  Note that when the input image does not store the shooting conditions other than the shooting region or the shooting region, the shooting condition acquisition unit 402 acquires the shooting information group including the shooting condition group from the shooting device 10 or an image management system (not shown). Can be obtained.

  Subsequent processing is the same as the series of image processing according to the second embodiment, and therefore description thereof will be omitted.

  As described above, the shooting condition acquisition unit 402 according to the present embodiment functions as an estimation unit that estimates at least one of the shooting region and the shooting region of the input image. The imaging condition acquisition unit 402 includes an imaging part estimation engine that uses images labeled with imaging parts and imaging regions as learning data. By inputting an input image to the imaging part estimation engine, the imaging part of the input image Estimate the shooting area.

  As a result, the image processing apparatus 400 according to the present embodiment can acquire the shooting conditions for the shooting region and the shooting area of the input image based on the input image.

  In the present embodiment, the imaging condition acquisition unit 402 uses the imaging part estimation engine to estimate the imaging part or imaging region of the input image when the imaging condition group does not include information about the imaging part or imaging region. went. However, the situation in which the imaging site and the imaging region are estimated using the imaging site estimation engine is not limited to this. The imaging condition acquisition unit 402 uses the imaging part estimation engine to detect the imaging part and the imaging part even when the information about the imaging part or the imaging region included in the data structure of the input image is insufficient as the information of the necessary detail level. You may estimate about a photography area.

  In addition, regardless of whether or not the data structure of the input image includes information about the imaged region or the imaged region, the image capturing condition acquisition unit 402 uses the imaged region estimation engine to estimate the imaged region or the imaged region of the input image. You may. In this case, the output unit 405 causes the display unit 20 to display the estimation result output from the imaging location estimation engine and the information about the imaging region and the imaging region included in the data structure of the input image, and the imaging condition acquisition unit 402 detects the information. These shooting conditions may be determined according to a person's instruction.

  Note that the output unit 405 may output the generated high-quality image to another device connected to the image capturing device 10 or the image processing device 400, as in the first embodiment. Further, the output data of the teacher data of the image quality improving engine is not limited to the high quality image subjected to the superimposing process, as in the first embodiment. That is, at least one of a processing group and an imaging method such as superposition processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing. You may use the high quality image obtained by performing.

<Fourth Embodiment>
Next, an image processing apparatus according to the fourth embodiment will be described with reference to FIGS. In the present embodiment, the image quality improving unit enlarges or reduces the input image so that the input image has an image size that can be handled by the image quality improving engine. The image quality improving unit reduces or enlarges the output image from the image quality improving engine so that the image size of the output image becomes the image size of the input image to generate a high quality image.

  Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 400 according to the first embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below focusing on the differences from the image processing apparatus according to the first embodiment. Since the configuration of the image processing apparatus according to the present embodiment is similar to that of the image processing apparatus according to the first embodiment, the same reference numerals are used for the configuration illustrated in FIG. 4, and description thereof will be omitted. To do.

  The image quality improving unit 404 according to the present embodiment includes an image quality improving engine similar to the image quality improving engine according to the first embodiment. However, in the present embodiment, as the teacher data used for learning of the image quality enhancement engine, the input data composed of an image group of the input data image and the output data image enlarged or reduced to have a constant image size It uses a pair of output data pairs.

  Here, the teacher data of the image quality improving engine according to the present embodiment will be described with reference to FIG. 8. As shown in FIG. 8, let us consider a case where, for example, there are a low image quality image Im810 and a high image quality image Im820 that are smaller than the fixed image size set for the teacher data. In this case, each of the low-quality image Im810 and the high-quality image Im820 is enlarged so that the fixed image size set for the teacher data is obtained. Then, the enlarged low-quality image Im811 and the enlarged high-quality image Im821 are used as a pair, and the pair is used as one of the teacher data.

  As in the first embodiment, the input data of the teacher data is an image having a specific shooting condition assumed as a processing target (input image), but the specific shooting condition is determined in advance. The photographing part, photographing method, and photographing angle of view. That is, unlike the first embodiment, the specific shooting condition according to the present embodiment does not include the image size.

  The image quality improving unit 404 according to this embodiment uses the image quality improving engine learned with such teacher data to improve the quality of the input image and generate a high quality image. At this time, the image quality improving unit 404 generates a modified image obtained by enlarging or reducing the input image to have a constant image size set for the teacher data, and inputs the modified image to the image quality improving engine. Further, the image quality improving unit 404 reduces or enlarges the output image from the image quality improving engine so as to have the image size of the input image, and generates a high quality image. Therefore, the image quality improving unit 404 according to the present embodiment generates an image with high image quality by the image quality improving engine even if the input image has an image size that cannot be dealt with in the first embodiment. be able to.

  Next, a series of image processing according to the present embodiment will be described with reference to FIGS. FIG. 9 is a flowchart of the image quality improving process according to the present embodiment. Note that the processes of step S510, step S520, and step S550 according to the present embodiment are the same as those of the first embodiment, so description thereof will be omitted. When the image quality of the input image is unconditionally improved under shooting conditions other than the image size, the process of step S530 may be omitted after the process of step S520, and the process may proceed to step S540.

  In step S520, as in the first embodiment, when the shooting condition acquisition unit 402 acquires the shooting condition group of the input image, the process proceeds to step S530. In step S530, the image quality improvement availability determination unit 403 determines whether the image quality improvement engine included in the image quality improvement unit 404 can handle the input image using the acquired shooting condition group. Specifically, the image quality improvement availability determination unit 403 determines whether or not the imaging condition of the input image is an imaging region, an imaging method, and an imaging angle of view that can be handled by the image quality improvement engine. Unlike the first embodiment, the image quality improvement availability determination unit 403 does not determine the image size.

  If it is determined that the input image can be dealt with, the process proceeds to step S540. On the other hand, when the image quality improvement determination unit 403 determines that the image quality improvement engine cannot handle the input image based on these shooting conditions, the process proceeds to step S550. Even if it is determined that the input image cannot be processed based on a part of the imaging region, the imaging method, and the imaging angle of view, depending on the setting or the mounting mode of the image processing apparatus 400, in step S540. Image quality improvement processing may be performed.

  When the process proceeds to step S540, the image quality improving process according to the present embodiment shown in FIG. 9 is started. In the image quality improving process according to the present embodiment, first, in step S910, the image quality improving unit 404 enlarges or reduces the input image to a certain image size set for the teacher data to generate a deformed image.

  Next, in step S920, the image quality improving unit 404 inputs the generated deformed image to the image quality improving engine and acquires the image quality improved deformed image.

  After that, in step S930, the image quality improving unit 404 reduces or enlarges the high-quality deformed image to the image size of the input image to generate a high-quality image. When the image quality improving unit 404 generates a high quality image in step S930, the image quality improving process according to the present embodiment ends, and the process proceeds to step S550. The process of step S550 is the same as step S550 of the first embodiment, and thus the description thereof is omitted.

  As described above, the image quality improving unit 404 according to the present embodiment adjusts the image size of the input image to an image size that can be handled by the image quality improving engine and inputs the image size to the image quality improving engine. Further, the image quality improving unit 404 adjusts the output image from the image quality improving engine to the original image size of the input image to generate a high quality image. As a result, the image processing apparatus 400 according to the present embodiment uses the image quality improving engine to improve the image quality of an input image having an image size that cannot be dealt with in the first embodiment, and achieves a high image quality suitable for image diagnosis. Images can be generated.

  Note that the output unit 405 may output the generated high-quality image to another device connected to the image capturing device 10 or the image processing device 400, as in the first embodiment. Further, the output data of the teacher data of the image quality improving engine is not limited to the high quality image subjected to the superimposing process, as in the first embodiment. That is, at least one of a processing group and an imaging method such as superposition processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing. You may use the high quality image obtained by performing.

<Fifth Embodiment>
Next, an image processing apparatus according to the fifth embodiment will be described with reference to FIGS. In the present embodiment, the image quality improving unit generates a high quality image by the image quality improving process based on a fixed resolution by the image quality improving engine.

  Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 400 according to the first embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below focusing on the differences from the image processing apparatus according to the first embodiment. Since the configuration of the image processing apparatus according to the present embodiment is similar to that of the image processing apparatus according to the first embodiment, the same reference numerals are used for the configuration illustrated in FIG. 4, and description thereof will be omitted. To do.

  The image quality improvement unit 404 according to the present embodiment includes the same image quality improvement engine as in the first embodiment. However, in this embodiment, the teacher data used for learning of the image quality enhancement engine is different from the teacher data in the first embodiment. Specifically, after enlarging or reducing the image group to an image size such that the resolution of the image group forming the pair group of the input data and the output data of the teacher data becomes a constant resolution, a sufficiently large constant It is padded to the image size. Here, the resolution of the image group refers to, for example, the spatial resolution of the imaging device or the resolution for the imaging region.

  Here, the teacher data of the image quality enhancement engine according to the present embodiment will be described with reference to FIG. 10. As shown in FIG. 10, for example, consider a case where there is a low image quality image Im1010 and a high image quality image Im1020 having a resolution lower than a certain resolution set for teacher data. In this case, each of the low-quality image Im1010 and the high-quality image Im1020 is enlarged so that the constant resolution set for the teacher data is obtained. Further, each of the enlarged low-quality image Im1010 and high-quality image Im1020 is padded so as to have a constant image size set for the teacher data. Then, the enlarged and padded low-quality image Im1011 and high-quality image Im1021 are paired, and the pair is used as one of the teacher data.

  It should be noted that the fixed image size set for the teacher data is the maximum image size that can be obtained when the image assumed as the processing target (input image) is enlarged or reduced to have a fixed resolution. If the certain image size is not large enough, when the image input to the image quality enhancement engine is enlarged, the machine learning model may have an image size that cannot be handled.

  In addition, the padding area is filled with a fixed pixel value, a neighboring pixel value, or mirror padded according to the characteristics of the machine learning model so that the image quality can be effectively improved. As in the first embodiment, an image having a specific imaging condition assumed as a processing target is used as the input data. The specific imaging condition is a predetermined imaging part, imaging method, This is the shooting angle of view. That is, unlike the first embodiment, the specific shooting condition according to the present embodiment does not include the image size.

  The image quality improving unit 404 according to this embodiment uses the image quality improving engine learned with such teacher data to improve the quality of the input image and generate a high quality image. At this time, the image quality improving unit 404 generates a deformed image obtained by enlarging or reducing the input image so as to have a constant resolution set for the teacher data. Further, the image quality improving unit 404 performs padding on the deformed image so that the image size becomes a constant image size set for the teacher data, generates a padding image, and inputs the padding image to the image quality improving engine.

  Further, the image quality improving unit 404 trims only the padded area of the high quality padding image output from the image quality improving engine to generate a high quality deformed image. After that, the image quality improving unit 404 reduces or enlarges the generated high quality deformed image so as to have the image size of the input image, and generates a high quality image.

  Therefore, the image quality improving unit 404 according to the present embodiment generates an image with high image quality by the image quality improving engine even if the input image has an image size that cannot be dealt with in the first embodiment. be able to.

  Next, a series of image processing according to the present embodiment will be described with reference to FIGS. FIG. 11 is a flowchart of the image quality improving process according to this embodiment. Note that the processes of step S510, step S520, and step S550 according to the present embodiment are the same as those of the first embodiment, so description thereof will be omitted. When the image quality of the input image is unconditionally improved under shooting conditions other than the image size, the process of step S530 may be omitted after the process of step S520, and the process may proceed to step S540.

  In step S520, as in the first embodiment, when the shooting condition acquisition unit 402 acquires the shooting condition group of the input image, the process proceeds to step S530. In step S530, the image quality improvement availability determination unit 403 determines whether the image quality improvement engine included in the image quality improvement unit 404 can handle the input image using the acquired shooting condition group. Specifically, the image quality improvement availability determination unit 403 determines whether or not the imaging condition of the input image is an imaging region, an imaging method, and an imaging angle of view that can be handled by the image quality improvement engine. Unlike the first embodiment, the image quality improvement availability determination unit 403 does not determine the image size.

  If it is determined that the input image can be dealt with, the process proceeds to step S540. On the other hand, when the image quality improvement determination unit 403 determines that the image quality improvement engine cannot handle the input image based on these shooting conditions, the process proceeds to step S550. Even if it is determined that the input image cannot be processed based on a part of the imaging region, the imaging method, and the imaging angle of view, depending on the setting or the mounting mode of the image processing apparatus 400, in step S540. Image quality improvement processing may be performed.

  When the process proceeds to step S540, the image quality improving process according to the present embodiment shown in FIG. 11 is started. In the image quality improving process according to the present embodiment, first, in step S1110, the image quality improving unit 404 enlarges or reduces the input image to have a constant resolution set for the teacher data, and generates a deformed image. .

  Next, in step S1120, the image quality improving unit 404 performs padding on the generated deformed image so that the image size is set for the teacher data to generate a padding image. At this time, the image quality improving unit 404 fills a region to be padded with a constant pixel value, a neighboring pixel value, or a mirror pixel value according to the characteristics of the machine learning model so as to effectively improve the image quality. Padding

  In step S1130, the image quality improving unit 404 inputs the padding image to the image quality improving engine and acquires the high quality padding image.

  Next, in step S1140, the high quality image generation unit 404 trims the high quality padding image by the area padded in step S1120 to generate a high quality deformed image.

  After that, in step S1150, the image quality improving unit 404 reduces or enlarges the high quality deformed image to the image size of the input image to generate a high quality image. When the image quality improving unit 404 generates the high image quality image in step S1130, the image quality improving process according to the present embodiment ends, and the process proceeds to step S550. The process of step S550 is the same as step S550 of the first embodiment, and thus the description thereof is omitted.

  As described above, the image quality improving unit 404 according to the present embodiment adjusts the image size of the input image so that the input image has a predetermined resolution. Further, the image quality improving unit 404 generates a padding image that has been padded so that the adjusted image size becomes an image size that can be handled by the image quality improving engine, for the input image whose image size has been adjusted, Input the padding image to the high quality image engine. After that, the image quality improving unit 404 trims the output image from the image quality improving engine only for the padded area. Then, the image quality improving unit 404 generates a high quality image by adjusting the image size of the trimmed image to the original image size of the input image.

  As a result, the image quality improving unit 404 of the present embodiment generates an image with a high image quality by the image quality improving engine even if the input image has an image size that cannot be dealt with by the first embodiment. You can Further, by using the image quality improving engine learned with the teacher data based on the resolution, the input image can be more efficiently processed than the image quality improving engine according to the fourth embodiment which simply processes images of the same image size. May be able to improve the image quality.

  Note that the output unit 405 may output the generated high-quality image to another device connected to the image capturing device 10 or the image processing device 400, as in the first embodiment. Further, the output data of the teacher data of the image quality improving engine is not limited to the high quality image subjected to the superimposing process, as in the first embodiment. That is, at least one of a processing group and an imaging method such as superposition processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing. You may use the high quality image obtained by performing.

<Sixth Embodiment>
Next, an image processing apparatus according to the sixth embodiment will be described with reference to FIGS. In the present embodiment, the image quality improving unit generates a high quality image by improving the image quality of the input image for each area having a constant image size.

  Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 400 according to the first embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below focusing on the differences from the image processing apparatus according to the first embodiment. Since the configuration of the image processing apparatus according to the present embodiment is similar to that of the image processing apparatus according to the first embodiment, the same reference numerals are used for the configuration illustrated in FIG. 4, and description thereof will be omitted. To do.

  The image quality improvement unit 404 according to the present embodiment includes the same image quality improvement engine as in the first embodiment. However, in this embodiment, the teacher data used for learning of the image quality enhancement engine is different from the teacher data in the first embodiment. Specifically, a group of pairs of low-quality image input data and high-quality image output data, which form teacher data, is assigned a fixed image size corresponding to the positional relationship between the low-quality image and the high-quality image. The rectangular area image of The rectangular area is an example of the partial area and does not have to be rectangular and may have any shape.

  Here, the teacher data of the image quality improving engine according to the present embodiment will be described with reference to FIG. As shown in FIG. 12, let us consider a case where, for example, one of the pair groups forming the teacher data includes an original image Im1210 that is a low-quality image and a superimposed image Im1220 that is a high-quality image. In this case, in the first embodiment, the input data of the teacher data is Im1210 and the output data is Im1220.

  On the other hand, in the present embodiment, the rectangular area image R1211 of the original image Im1210 is used as input data, and the rectangular area image R1221 that is the same shooting area as the rectangular area image R1211 in the superimposed image Im1220 is used as output data. Then, the rectangular area image R1211, which is the input data, and the rectangular area image R1221, which is the output data, form a pair of teacher data (hereinafter, a first rectangular area image pair). Here, the rectangular area image R1211 and the rectangular area image R1221 are images having a constant image size. The original image Im1210 and the superimposed image Im1220 may be aligned by any method. Further, the corresponding positional relationship between the rectangular area image R1211 and the rectangular area image R1221 may be specified by an arbitrary method such as template matching. The image size and the number of dimensions of the input data and the output data may be different depending on the design of the high image quality engine. For example, when the processing target is an OCT image, when the input data is a part of the B scan image (two-dimensional image), the output data may be a part of the A scan image (one-dimensional image). .

  The constant image size of the rectangular area images R1211 and R1221 can be determined from, for example, the common divisor of the pixel number group of each dimension for the image size group of the image assumed as the processing target (input image). In this case, it is possible to prevent the positional relationships of the rectangular area image groups output by the image quality improving engine from overlapping. Specifically, for example, the image supposed to be processed is a two-dimensional image, the first image size in the image size group is 500 pixels wide and 500 pixels high, and the second image size is wide. Consider the case of 100 pixels and height 100 pixels. Here, a fixed image size for the rectangular area images R1211 and R1221 is selected from the common divisor of each side. In this case, for example, a fixed image size is selected from width 100 pixels, height 100 pixels, width 50 pixels, height 50 pixels, width 25 pixels, height 25 pixels, and the like.

  When the image supposed to be processed is three-dimensional, the number of pixels is determined in terms of width, height and depth. It should be noted that a plurality of rectangular areas can be set for one of a pair of a low-quality image corresponding to input data and a high-quality image corresponding to output data. Therefore, for example, the rectangular area image R1212 of the original image Im1210 is used as input data, and the rectangular area image R1222 that is the same shooting area as the rectangular area image R1212 in the superimposed image Im1220 is used as output data. Then, the rectangular area image R1212 that is the input data and the rectangular area image R1222 that is the output data form a pair of teacher data. Thereby, a rectangular area image pair different from the first rectangular area image pair can be created.

  It should be noted that it is possible to enhance the group of pairs forming the teacher data by creating a large number of pairs of rectangular area images while changing the image of the rectangular area to images of different coordinates, and the learning was performed using the teacher pairs. Efficient image quality can be expected with the image quality improvement engine. However, pairs that do not contribute to improving the image quality of the machine learning model can be excluded from the teacher data. For example, when the rectangular area image created from the high-quality image that is the output data forming the pair has an image quality not suitable for diagnosis, the image-quality improving engine that has performed learning using such teacher data outputs the image. Images may also have an image quality that is not suitable for image diagnosis. Therefore, the pair including such a high quality image can be removed from the teacher data.

  Further, for example, even when the average brightness and the brightness distribution of a rectangular area image created from a low-quality image and a rectangular area image created from a high-quality image that are a pair are significantly different, such a pair can be removed from the teacher data. it can. If learning is performed using such teacher data, the image quality improving engine may output an image having a luminance distribution that is significantly different from the input image, which is not suitable for image diagnosis.

  Further, for example, consider a case where the structure and the position of the imaging target drawn in the rectangular area image created from the low-quality image and the rectangular area image created from the high-quality image that are paired are significantly different. In this case, there is a possibility that an image quality improvement engine that has performed learning using such teacher data may output an image that is not suitable for image diagnosis in which the imaging target is drawn at a structure or position that is significantly different from the input image. is there. Therefore, such a pair can be removed from the teacher data.

  Note that, as in the first embodiment, an image having a specific image capturing condition assumed as a processing target is used as the input data of the teacher data, and the specific image capturing condition is a predetermined image capturing part, The shooting method and the shooting angle of view. That is, unlike the first embodiment, the specific shooting condition according to the present embodiment does not include the image size.

  The image quality improving unit 404 according to this embodiment uses the image quality improving engine learned with such teacher data to improve the quality of the input image and generate a high quality image. At this time, the image quality improving unit 404 divides the input image into a rectangular area image group that is continuous with no gap and has a constant image size set for the teacher data. The image quality improving unit 404 improves the image quality of each of the divided rectangular area image groups by the image quality improving engine, and generates a high image quality rectangular area image group. After that, the image quality improving unit 404 arranges the generated high-quality rectangular area image groups according to the positional relationship of the input images and combines them to generate a high-quality image. Here, at the time of learning, if the mutual positional relationship between the input data and the output data, which are pair images, corresponds to each other, each rectangular area is cut out (extracted) from an arbitrary position in the low-quality image and the high-quality image. Good). On the other hand, when the image quality is improved, the input image may be divided into continuous rectangular area image groups without a gap. Further, the image size of each pair image at the time of learning and the image size of each rectangular area image at the time of high image quality may be set to correspond to each other (for example, be the same). As a result, it is possible to improve the learning efficiency and prevent the problem that an image is not formed when useless calculation or a lacking portion appears.

  As described above, the image quality improving unit 404 of the present embodiment improves the image quality of the input image in units of rectangular areas, and combines the images of high image quality, so that the image that cannot be dealt with in the first embodiment. It is possible to generate a high quality image by improving the quality of a size image.

  Next, a series of image processing according to the present embodiment will be described with reference to FIGS. FIG. 13 is a flowchart of the image quality improving process according to this embodiment. Note that the processes of step S510, step S520, and step S550 according to the present embodiment are the same as those of the first embodiment, so description thereof will be omitted. When the image quality of the input image is unconditionally improved under shooting conditions other than the image size, the process of step S530 may be omitted after the process of step S520, and the process may proceed to step S540.

  In step S520, as in the first embodiment, when the shooting condition acquisition unit 402 acquires the shooting condition group of the input image, the process proceeds to step S530. In step S530, the image quality improvement availability determination unit 403 determines whether the image quality improvement engine included in the image quality improvement unit 404 can handle the input image using the acquired shooting condition group. Specifically, the image quality improvement availability determination unit 403 determines whether or not the imaging condition of the input image is an imaging region, an imaging method, and an imaging angle of view that can be handled by the image quality improvement engine. Unlike the first embodiment, the image quality improvement availability determination unit 403 does not determine the image size.

  If it is determined that the input image can be dealt with, the process proceeds to step S540. On the other hand, when the image quality improvement determination unit 403 determines that the image quality improvement engine cannot handle the input image based on these shooting conditions, the process proceeds to step S550. Even if it is determined that the input image cannot be processed based on a part of the imaging region, the imaging method, and the imaging angle of view, depending on the setting or the mounting mode of the image processing apparatus 400, in step S540. Image quality improvement processing may be performed.

  When the process proceeds to step S540, the image quality improving process according to the present embodiment shown in FIG. 13 is started. This will be described with reference to FIG. In the image quality improving process according to the present embodiment, first, in step S1310, as shown in FIG. 14A, a constant image size (size indicated by R1411) set for the teacher data, in which the input images are continuous without gaps. ) Rectangular area image group. Here, FIG. 14A shows an example in which the input image Im1410 is divided into rectangular area images R1411 to R1426 having a constant image size. As described above, depending on the design of the image quality improving engine, the image size and the number of dimensions of the input image and the output image of the image quality improving engine may be different. In this case, the divided positions of the input images can be adjusted to be overlapped or separated so that the combined high-quality images generated in step S1320 have no defects. FIG. 14B shows an example in which division positions are overlapped. In FIG. 14B, the regions where R1411 'and R1412' overlap are shown. Although not shown because of complexity, it is assumed that R1413 to R1426 also have similar overlapping regions R1413 'to R1426'. The rectangular area size set for the teacher data in the case of FIG. 14B is the size shown in R1411 '. Since data does not exist around the periphery (upper, left, right ends) of the input image Im1410 outside the image, the pixel is padded with a constant pixel value, neighboring pixel values, or mirror padded. In addition, depending on the image quality improving engine, the accuracy of image quality improvement may be reduced at the periphery (upper, left, right edges) inside the image due to the filtering process. Therefore, as shown in FIG. 14 (b), the division positions may be overlapped to set the rectangular area image, and a part of the rectangular area image may be trimmed and combined as a final image. The size of the rectangular area is set according to the characteristics of the image quality enhancement engine. 14A and 14B illustrate OCT tomographic images, the input image (Im1450) is an OCTA En-Face image as illustrated in FIGS. 14C and 14D. A front image may be used, and similar processing is possible. Note that the size of the rectangular area image is appropriately set according to the target image and the type of image quality enhancement engine.

  Next, in step S1320, the image quality improving unit 404 enhances each of the rectangular area images R1411 to R1426, or the rectangular area images R1411 ′ to R1426 ′ group when the overlapping area is set by the image quality improving engine. Image quality is improved and a high-quality rectangular area image group is generated.

  Then, in step S1330, image quality improving section 404 arranges and combines each of the generated high-quality rectangular area image groups in the same positional relationship as each of the divided rectangular area images R1411 to R1426. Then, a high quality image is generated. When the overlapping area is set, the rectangular area images R1411 'to R1426' are arranged in the same positional relationship as each other, and then the rectangular area images R1411 to R1426 are cut out and combined to generate a high quality image. The brightness values of the rectangular area images R1411 'to R1426' may be corrected using the overlapping area. For example, a rectangular area image as a reference is arbitrarily set. Then, by measuring the brightness value of the same coordinate point in the adjacent rectangular image having an area overlapping the reference rectangular image, the difference (ratio) of the brightness values between the adjacent images can be known. Similarly, by obtaining the difference (ratio) of the brightness values in the overlapping areas in all the images, it becomes possible to perform correction so as to eliminate the unevenness of the brightness values as a whole. Note that it is not necessary to use the entire overlapping area for brightness value correction, and it is not necessary to use part of the overlapping area (a few pixels in the peripheral portion).

  As described above, the image quality improving unit 404 according to the present embodiment divides the input image into a plurality of rectangular area images (third images) R1411 to R1426 having a predetermined image size. After that, the image quality improving unit 404 inputs the plurality of divided rectangular area images R1411 to R1426 to the image quality improving engine to generate a plurality of fourth images, and integrates the plurality of fourth images. Generate high quality images. In addition, when the positional relationships among the rectangular area groups overlap at the time of integration, the pixel value groups of the rectangular area groups can be integrated or overwritten.

  As a result, the image quality improving unit 404 of the present embodiment generates an image with a high image quality by the image quality improving engine even if the input image has an image size that cannot be dealt with by the first embodiment. You can Further, when the teacher data is created from a plurality of images obtained by dividing the low-quality image and the high-quality image into predetermined image sizes, a large number of teacher data can be created from a small number of images. Therefore, in this case, the number of low-quality images and high-quality images for creating the teacher data can be reduced.

  Note that the output unit 405 may output the generated high-quality image to another device connected to the image capturing device 10 or the image processing device 400, as in the first embodiment. Further, the output data of the teacher data of the image quality improving engine is not limited to the high quality image subjected to the superimposing process, as in the first embodiment. That is, at least one of a processing group and an imaging method such as superposition processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing. You may use the high quality image obtained by performing.

<Seventh Embodiment>
Next, an image processing apparatus according to the seventh embodiment will be described with reference to FIGS. In the present embodiment, the image quality evaluation unit selects the highest quality image among the plurality of high quality images output from the plurality of quality enhancement engines according to the instruction of the examiner.

  Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 400 according to the first embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below focusing on the differences from the image processing apparatus according to the first embodiment.

  FIG. 15 shows a schematic configuration of the image processing apparatus 1500 according to this embodiment. The image processing apparatus 1500 according to the present embodiment is provided with an image quality evaluation unit 1506 in addition to the acquisition unit 401, the shooting condition acquisition unit 402, the image quality improvement possibility determination unit 403, the image quality improvement unit 404, and the output unit 405. Has been. The image processing apparatus 1500 may be composed of a plurality of devices provided with some of these components. Here, the acquisition unit 401, the shooting condition acquisition unit 402, the image quality improvement possibility determination unit 403, the image quality improvement unit 404, and the output unit 405 have the same configurations as the image processing apparatus according to the first embodiment. 4, the same reference numerals are used for the configuration shown in FIG. 4, and description thereof is omitted.

  Further, the image processing apparatus 1500 may be connected to the image capturing apparatus 10, the display unit 20, and other devices (not shown) similarly to the image processing apparatus 400 according to the first embodiment via an arbitrary circuit or network. . Further, these devices may be connected to any other device via a circuit or a network, or may be configured integrally with any other device. Although these devices are separate devices in the present embodiment, some or all of these devices may be integrally configured.

  The image quality improving unit 404 according to the present embodiment includes two or more image quality improving engines in which machine learning is performed using different teacher data. Here, a method of creating the teacher data group according to the present embodiment will be described. Specifically, first, a pair group of input data, which is a low-quality image, and output data, which is a high-quality image, which are shot under various shooting conditions, is prepared. Next, the teacher data group is created by grouping the pair groups according to an arbitrary combination of shooting conditions. For example, the first teacher data composed of the pair group acquired by the combination of the first shooting conditions, the second teacher data composed of the pair group acquired by the combination of the second shooting conditions, and so on. , Create as a teacher data group.

  After that, machine learning is performed by different image quality improving engines using each teacher data. For example, a first image quality improvement engine corresponding to a machine learning model trained with first teacher data, a first image quality improvement engine corresponding to a machine learning model trained with first teacher data, and so on. Prepare a high-quality image engine group.

  Since such high image quality engine has different teacher data used for training the corresponding machine learning model, the degree to which the high quality of the input image can be achieved differs depending on the image capturing conditions of the image input to the high image quality engine. . Specifically, the first image-quality improving engine has a high degree of image quality improvement for the input image obtained by shooting under the combination of the first shooting conditions, and the first image-quality improving engine under the combination of the second shooting conditions. The degree of image quality improvement is low for images captured and acquired. Similarly, the second image quality improving engine has a high degree of image quality improvement with respect to the input image captured and acquired under the second image capturing condition, and is captured and acquired under the first image capturing condition. The degree of image quality improvement is low for images.

  Since each of the teacher data is composed of a pair group grouped by a combination of shooting conditions, image quality tendencies of the image groups forming the pair group are similar. Therefore, the image quality improving engine can achieve higher image quality more effectively than the image enhancing engine according to the first embodiment as long as the combination of corresponding shooting conditions is used. It should be noted that the combination of shooting conditions for grouping the pair of teacher data may be arbitrary, and may be, for example, a combination of two or more of a shooting region, a shooting angle of view, and an image resolution. Further, the teacher data may be grouped based on one shooting condition as in the second embodiment.

  The image quality evaluation unit 1506 selects the highest image quality image from the plurality of high image quality images generated by the image quality improvement unit 404 using the plurality of image quality improvement engines according to an instruction from the examiner.

  The output unit 405 can display the high-quality image selected by the image quality evaluation unit 1506 on the display unit 20 or output it to another device. The output unit 405 can display a plurality of high-quality images generated by the high-quality image generation unit 404 on the display unit 20, and the image quality evaluation unit 1506 responds to an instruction from the examiner who has confirmed the display unit 20. It is possible to select a high quality image having the highest image quality.

  As a result, the image processing apparatus 1500 can output the highest quality image in accordance with the instruction from the examiner among the plurality of quality images generated by using the plurality of quality enhancement engines.

  Hereinafter, a series of image processing according to the present embodiment will be described with reference to FIGS. FIG. 16 is a flowchart of a series of image processing according to this embodiment. Note that the processing of steps S1610 and S1620 according to the present embodiment is the same as the processing of steps S510 and S520 in the first embodiment, so description will be omitted. In addition, when the image quality of an input image is unconditionally improved, the process of step S1630 may be omitted after the process of step S1620, and the process may proceed to step S1640.

  In step S1620, as in the first embodiment, when the shooting condition acquisition unit 402 acquires the shooting condition group of the input image, the process proceeds to step S1630. In step S1630, as in the second embodiment, the image quality improvement possibility determination unit 403 uses the acquired shooting condition group to cause any of the image quality improvement engines included in the image quality improvement unit 404 to input the input image. It is determined whether it can be dealt with.

  If the image quality improvement determination unit 403 determines that none of the image quality improvement engine groups can handle the input image, the process proceeds to step S1660. On the other hand, when the image quality improvement availability determination unit 403 determines that one of the image quality enhancement engine groups can handle the input image, the process proceeds to step S1640. Note that, depending on the setting and implementation of the image processing apparatus 400, even if the image quality enhancement engine determines that some image capturing conditions cannot be dealt with, step S1640 is performed, as in the first embodiment. You may.

  In step S1640, the image quality improving unit 404 inputs the input image acquired in step S1610 to each of the image quality improving engine groups to generate a high quality image group.

  In step S1650, the image quality evaluation unit 1506 selects the highest quality image from the high quality image group generated in step S1640. Specifically, first, the output unit 405 displays the high-quality image group generated in step S1640 on the user interface of the display unit 20.

  Here, FIG. 17 shows an example of the interface. The input image Im1710 and each of the high-quality images Im1720, Im1730, Im1740, and Im1750 output by each of the high-quality image engine groups are displayed on the interface. The examiner operates an arbitrary input device (not shown) to indicate the highest image quality among the image groups (high quality images Im1720 to Im1750), that is, the image most suitable for image diagnosis. Since an input image that has not been improved in image quality by the image quality improvement engine may be more suitable for image diagnosis, the input image may be added to the image group to be instructed by the examiner.

  Then, the image quality evaluation unit 1506 selects the high quality image designated by the examiner as the highest quality image.

  In step S1660, output unit 405 causes display unit 20 to display the image selected in step S1650 or outputs the image to another device. However, if it is determined in step S1630 that the input image cannot be processed, the output unit 405 outputs the input image as the output image. It should be noted that the output unit 405 causes the display unit 20 to display that the output image is the same as the input image when the examiner instructs the input image or when the input image cannot be processed. Good.

  As described above, the image quality improving unit 404 according to the present embodiment uses the plurality of image quality improving engines to generate a plurality of high quality images from the input image, and the output unit 405 of the image processing apparatus 1500 detects the image quality. At least one image out of the plurality of high-quality images is output according to the instruction of the person. In particular, in this embodiment, the output unit 405 outputs the image with the highest image quality in accordance with the instruction from the examiner. As a result, the image processing apparatus 1500 can output a high-quality image having a high image quality in accordance with an instruction from the examiner, out of the plurality of high-quality images generated by using the plurality of high-quality images engines.

  Note that the output unit 405 may output the generated high-quality image to another device connected to the image capturing device 10 or the image processing device 1500, as in the first embodiment. Further, the output data of the teacher data of the image quality improving engine is not limited to the high quality image subjected to the superimposing process, as in the first embodiment. That is, at least one of a processing group and an imaging method such as superposition processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing. You may use the high quality image obtained by performing.

<Eighth Embodiment>
Next, an image processing apparatus according to the eighth embodiment will be described with reference to FIGS. In the present embodiment, the image quality evaluation unit uses the image quality evaluation engine to select the highest quality image among the plurality of high quality images output from the plurality of quality improvement engines.

  Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 1500 according to the seventh embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below focusing on the differences from the image processing apparatus according to the seventh embodiment. Since the configuration of the image processing apparatus according to the present embodiment is similar to that of the image processing apparatus according to the seventh embodiment, the same reference numerals are used for the configuration illustrated in FIG. 15, and description thereof will be omitted. To do.

  The image quality evaluation unit 1506 according to this embodiment includes an image quality evaluation engine that evaluates the image quality of an input image. The image quality evaluation engine outputs an image quality evaluation index for the input image. The image quality evaluation processing method for calculating the image quality evaluation index in the image quality evaluation engine according to the present embodiment uses a machine learning model constructed using a machine learning algorithm. The input data of a pair forming the teacher data for training the machine learning model is an image group including a low image quality image group and a high image quality image group that have been captured in advance under various image capturing conditions. Further, the output data of the pair forming the teacher data for training the machine learning model is, for example, the image quality evaluation index group set for each image group of the input data by the examiner who performs image diagnosis.

  Next, with reference to FIG. 16, a series of image processing according to the present embodiment will be described. Note that the processes of step S1610, step S1620, step S1630, and step S1660 according to the present embodiment are the same as those of the seventh embodiment, so description thereof will be omitted. In addition, when the image quality of an input image is unconditionally improved, the process of step S1630 may be omitted after the process of step S1620, and the process may proceed to step S1640.

  In step S1630, if the image quality improvement availability determination unit 403 determines that one of the image quality enhancement engine groups can handle the input image, as in the seventh embodiment, the process proceeds to step S1640. Transition. Note that, depending on the setting and implementation of the image processing apparatus 400, even if the image quality enhancement engine determines that some image capturing conditions cannot be dealt with, step S1640 is performed, as in the first embodiment. You may.

  In step S1640, the image quality improving unit 404 inputs the input image acquired in step S1610 to each of the image quality improving engine groups to generate a high quality image group.

  In step S1650, the image quality evaluation unit 1506 selects the highest quality image from the high quality image group generated in step S1640. Specifically, first, the image quality evaluation unit 1506 inputs the high quality image group generated in step S1640 to the image quality evaluation engine. The image quality evaluation engine calculates an image quality evaluation index for each input high quality image based on learning. The image quality evaluation unit 1506 selects the high quality image for which the highest image quality evaluation index has been calculated among the calculated image quality evaluation indexes. Since the input image that has not been improved in image quality by the image quality improvement engine may be more suitable for image diagnosis, the image quality evaluation unit 1506 also inputs the input image to the image quality evaluation engine and An evaluation index may also be added to the selection. Since step S1660 is similar to step S1660 of the seventh embodiment, description thereof will be omitted.

  As described above, the image processing apparatus 1500 according to this embodiment further includes the image quality evaluation unit 1506 that evaluates the image quality of the high quality image. The image quality improving unit 404 uses the plurality of image quality improving engines to generate a plurality of high image quality images from the input image, and the output unit 405 of the image processing apparatus 1500 outputs the image quality evaluation unit 1506 according to the evaluation result. At least one of the plurality of high quality images is output. In particular, the image quality evaluation unit 1506 according to the present embodiment includes an image quality evaluation engine that uses the evaluation value obtained by a predetermined evaluation method as learning data. The image quality evaluation unit 1506 selects a high quality image that has the highest evaluation result by the image quality evaluation unit 1506 using the image quality evaluation engine from the plurality of high quality images. The output unit 405 outputs the high quality image having the highest evaluation value selected by the image quality evaluation unit 1506.

  As a result, the image processing apparatus 1500 according to the present embodiment can easily output a high-quality image most suitable for image diagnosis from a plurality of high-quality images based on the output of the image quality evaluation engine.

  In this embodiment, the image quality evaluation unit 1506 selects the high quality image having the highest image quality evaluation index among the image quality evaluation indexes output by the image quality evaluation engine, and the output unit 405 displays the selected high quality image. It was displayed on 20. However, the configuration of the image quality evaluation unit 1506 is not limited to this. For example, the image quality evaluation unit 1506 selects the high quality images of the top several image quality evaluation indexes of the image quality evaluation indexes output by the image quality evaluation engine, and the output unit 405 displays the selected high quality images on the display unit 20. You may let me. In addition, the output unit 405 causes the image quality evaluation index output by the image quality evaluation engine to be displayed on the display unit 20 together with the corresponding high quality image, and the image quality evaluation unit 1506 displays the highest quality image according to the instruction of the examiner. You may choose.

  Note that the output unit 405 may output the generated high-quality image to another device connected to the image capturing device 10 or the image processing device 1500, as in the first embodiment. Further, the output data of the teacher data of the image quality improving engine is not limited to the high quality image subjected to the superimposing process, as in the first embodiment. That is, at least one of a processing group and an imaging method such as superposition processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing. You may use the high quality image obtained by performing.

<Ninth Embodiment>
Next, an image processing apparatus according to the ninth embodiment will be described with reference to FIGS. In the present embodiment, the authenticity evaluation unit uses an authenticity evaluation engine to evaluate whether or not the high-quality image generated by the image quality improvement unit 404 is sufficiently high quality.

  Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 400 according to the first embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below focusing on the differences from the image processing apparatus according to the first embodiment.

  FIG. 18 shows a schematic configuration of an image processing device 1800 according to this embodiment. The image processing apparatus 1800 according to the present embodiment is provided with an authenticity evaluation unit 1807 in addition to the acquisition unit 401, the imaging condition acquisition unit 402, the image quality improvement possibility determination unit 403, the image quality improvement unit 404, and the output unit 405. Has been. The image processing device 1800 may be configured by a plurality of devices provided with some of these components. Here, the acquisition unit 401, the shooting condition acquisition unit 402, the image quality improvement possibility determination unit 403, the image quality improvement unit 404, and the output unit 405 have the same configurations as the image processing apparatus according to the first embodiment. 4, the same reference numerals are used for the configuration shown in FIG. 4, and description thereof is omitted.

  Further, the image processing device 1800 may be connected to the image capturing device 10, the display unit 20, and other devices (not shown) similarly to the image processing device 400 according to the first embodiment via an arbitrary circuit or network. . Further, these devices may be connected to any other device via a circuit or a network, or may be configured integrally with any other device. Although these devices are separate devices in the present embodiment, some or all of these devices may be integrally configured.

  The authenticity evaluation unit 1807 includes an authenticity evaluation engine. The authenticity evaluation unit 1807 evaluates whether or not the high quality image generated by the high quality image engine is sufficiently high quality by using the authenticity evaluation engine. The authenticity evaluation processing method in the authenticity evaluation engine according to the present embodiment uses a machine learning model constructed by using a machine learning algorithm.

  The teacher data for training the machine learning model has a pair of high-quality images that were captured in advance under various capturing conditions and a label (hereinafter, a true label) representing that the image was captured and captured by the target image capturing device. Groups are included. Further, in the teacher data, a high-quality image group generated by inputting a low-quality image into a high-quality image engine with poor accuracy in image quality and a label indicating that the target image capturing device has not captured and acquired , Counterfeit label) and a group of pairs.

  The authenticity evaluation engine learned by using such teacher data cannot evaluate whether or not the input image is an image captured and acquired by the image capturing device, but it does not depend on the image capturing device. It is possible to evaluate whether or not the image has the image-likeness that is captured and acquired. By utilizing this characteristic, the authenticity evaluation unit 1807 inputs the high-quality image generated by the high-quality image generation unit 404 to the authenticity evaluation engine, so that the high-quality image generated by the high-quality image generation unit 404 is sufficiently high in quality. It is possible to evaluate whether or not

  When it is determined by the authenticity evaluation unit 1807 that the high quality image generated by the high quality image generation unit 404 is sufficiently high quality, the output unit 405 displays the high quality image on the display unit 20. On the other hand, the output unit 405 causes the display unit 20 to display the input image when the authenticity evaluation unit 1807 determines that the high quality image generated by the high quality image generation unit 404 is not sufficiently high quality. The output unit 405, when displaying the input image, confirms that the high-quality image generated by the high-quality image generation unit 404 has not been sufficiently improved in quality, and that the displayed image is the input image. It can be displayed on the display unit 20.

  Hereinafter, a series of image processing according to the present embodiment will be described with reference to FIG. FIG. 19 is a flowchart of a series of image processing according to this embodiment. Note that the processing of steps S1910 to S1940 according to the present embodiment is the same as the processing of steps S510 to S540 in the first embodiment, so description will be omitted. In addition, when the image quality of the input image is unconditionally improved with respect to the shooting condition, the process of step S1930 may be omitted after the process of step S1920 and the process may proceed to step S1940.

  In step S1940, when the image quality improving unit 404 generates a high quality image group, the process proceeds to step S1950. In step S1950, the authenticity evaluation unit 1807 inputs the high-quality image generated in step S1940 into the authenticity evaluation engine and evaluates the authenticity based on the output of the authenticity evaluation engine. Specifically, the authenticity evaluation unit 1807 evaluates that the generated high quality image is sufficiently high quality when the authenticity label (true) is output from the authenticity evaluation engine. On the other hand, when an authentication label (false) is output from the authentication evaluation engine, the authentication evaluation unit 1807 evaluates that the generated high quality image is not sufficiently high quality.

  In step S1960, when the output unit 405 determines that the high quality image generated by the high quality image generation unit 404 by the authenticity evaluation unit 1807 is sufficiently high quality, the high quality image is displayed on the display unit 20. Let On the other hand, the output unit 405 causes the display unit 20 to display the input image when the authenticity evaluation unit 1807 determines that the high quality image generated by the high quality image generation unit 404 is not sufficiently high quality.

  As described above, the image processing apparatus 1800 according to this embodiment further includes the authenticity evaluation unit 1807 that evaluates the image quality of a high-quality image, and the authenticity evaluation unit 1807 includes an authenticity evaluation engine that evaluates the authenticity of the image. The authenticity evaluation engine includes a machine learning engine in which an image generated by an image quality improving engine whose image quality improving processing is lower (bad) than the image quality improving engine of the image quality improving unit 404 is used as learning data. The output unit 405 of the image processing device 1800 outputs a high-quality image when the output from the authenticity evaluation engine of the authenticity evaluation unit is true.

  As a result, in the image processing device 1800 according to the present embodiment, the examiner can efficiently check the high-quality image with sufficiently high image quality.

  Further, the machine learning model of the image quality improving engine and the machine learning model of the authenticity evaluation engine may be trained in cooperation with each other to improve the efficiency and accuracy of both engines.

  In the present embodiment, the image quality improving unit 404 generates one high quality image, and the authenticity evaluation unit 1807 evaluates one generated high quality image. However, the evaluation by the authenticity evaluation unit 1807 is not limited to this. Is not limited to this. For example, when the image quality improving unit 404 generates a plurality of high image quality images using a plurality of image quality improving engines as in the second embodiment, the authenticity evaluation unit 1807 generates a plurality of high image quality images. The evaluation may be performed on at least one of the images. In this case, for example, the authenticity evaluation unit 1807 may evaluate all of the generated plurality of high quality images, or may evaluate only the image instructed by the examiner among the plurality of high quality images. Good.

  Further, the output unit 405 causes the display unit 20 to display the determination result of whether or not the high quality image by the authenticity evaluation unit 1807 is sufficiently high quality, and outputs the high quality image according to the instruction of the examiner. You may.

  Note that the output unit 405 may output the generated high-quality image to another device connected to the image capturing device 10 or the image processing device 1800, as in the first embodiment. Further, the output data of the teacher data of the image quality improving engine is not limited to the high quality image subjected to the superimposing process, as in the first embodiment. That is, at least one of a processing group and an imaging method such as superposition processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing. You may use the high quality image obtained by performing.

<Tenth Embodiment>
Next, an image processing apparatus according to the tenth embodiment will be described with reference to FIGS. In the present embodiment, the image quality improving unit divides the three-dimensional input image into a plurality of two-dimensional images, inputs the images into the image quality improving engine, and combines the output images from the image quality improving engine to combine the three-dimensional image. Generate high quality images.

  Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 400 according to the first embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below focusing on the differences from the image processing apparatus according to the first embodiment. Since the configuration of the image processing apparatus according to the present embodiment is similar to that of the image processing apparatus according to the first embodiment, the same reference numerals are used for the configuration illustrated in FIG. 4, and description thereof will be omitted. To do.

  The acquisition unit 401 according to the present embodiment acquires a three-dimensional image composed of a structurally continuous two-dimensional image group. Specifically, the three-dimensional image is, for example, a three-dimensional OCT volume image composed of a B scan image (tomographic image) group of OCT. In addition, for example, it is a three-dimensional CT volume image composed of a group of axial tomographic images.

  The image quality improving unit 404 is provided with an image quality improving engine as in the first embodiment. A pair group of input data and output data, which is teacher data of the image quality improving engine, is composed of an image group of two-dimensional images. The image quality improving unit 404 divides the acquired three-dimensional image into a plurality of two-dimensional images, and inputs each two-dimensional image to the image quality improving engine. Thereby, the image quality improving unit 404 can generate a plurality of two-dimensional high quality images.

  The output unit 405 combines the plurality of two-dimensional high-quality images generated for each two-dimensional image of the three-dimensional image by the high-quality rendering unit 404 and outputs the three-dimensional high-quality image.

  Next, a series of image processing according to the present embodiment will be described with reference to FIG. Note that the processes of steps S510 to S530 and step S550 according to the present embodiment are the same as those of the first embodiment, so description thereof will be omitted. However, in step S510, the acquisition unit 401 acquires a three-dimensional image. Note that when the image quality of the input image is unconditionally improved, the process of step S530 may be omitted after the process of step S520, and the process may proceed to step S540.

  When the image quality improvement determination unit 403 determines in step S530 that the input image can be handled by the image quality improvement engine, the process proceeds to step S540. The image quality improvement determination unit 403 may make the determination based on the shooting conditions of the three-dimensional image, or may make the determination based on the shooting conditions of a plurality of two-dimensional images forming the three-dimensional image. May be. In step S540, the image quality improving unit 404 divides the acquired three-dimensional image into a plurality of two-dimensional images. The image quality improving unit 404 inputs each of the plurality of divided two-dimensional images to the image quality improving engine and generates a plurality of two-dimensional high quality images. The image quality improving unit 404 combines the plurality of generated two-dimensional high-quality images based on the acquired three-dimensional image to generate a three-dimensional high-quality image.

  In step S550, the output unit 405 causes the display unit 20 to display the generated three-dimensional high quality image. The display mode of the three-dimensional high-quality image may be arbitrary.

  As described above, the image quality improving unit 404 according to the present embodiment divides the three-dimensional input image into a plurality of two-dimensional images and inputs them to the image quality improving engine. The image quality improving unit 404 combines a plurality of two-dimensional image quality images output from the image quality improving engine to generate a three-dimensional image quality image.

  As a result, the image quality improving unit 404 according to the present embodiment can improve the image quality of the three-dimensional image using the image quality improving engine that has been learned using the teacher data of the two-dimensional image.

  Note that the output unit 405 may output the generated high-quality image to another device connected to the image capturing device 10 or the image processing device 400, as in the first embodiment. Further, the output data of the teacher data of the image quality improving engine is not limited to the high quality image subjected to the superimposing process, as in the first embodiment. That is, at least one of a processing group and an imaging method such as superposition processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing. You may use the high quality image obtained by performing.

<Eleventh Embodiment>
Next, an image processing apparatus according to the eleventh embodiment will be described with reference to FIGS. In the present embodiment, the image quality improving section divides the three-dimensional input image into a plurality of two-dimensional images, and the plurality of two-dimensional images are image-processed in parallel by a plurality of image quality improving engines. By combining the output images, a three-dimensional high quality image is generated.

  Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 400 according to the tenth embodiment. Therefore, in the following, the image processing apparatus according to the present embodiment will be described focusing on the differences from the image processing apparatus according to the tenth embodiment. Since the configuration of the image processing apparatus according to the present embodiment is similar to the configuration of the image processing apparatus according to the first and tenth embodiments, the configuration illustrated in FIG. Is omitted.

  The image quality improving unit 404 according to the present embodiment includes a plurality of image quality improving engines similar to those of the tenth embodiment. Note that the plurality of image quality improvement engine groups provided in the image quality improvement unit 404 may be mounted so as to be able to perform distributed processing in two or more device groups via a circuit or a network. It may be mounted on the device.

  The image quality improving unit 404 divides the acquired three-dimensional image into a plurality of two-dimensional images, as in the tenth embodiment. The image quality improving unit 404 uses a plurality of image quality improving engines to share (in parallel) the image quality of a plurality of two-dimensional images to generate a plurality of two-dimensional image quality images. The image quality improving unit 404 combines a plurality of two-dimensional image quality images output from a plurality of image quality improvement engines based on the processing target three-dimensional image to generate a three-dimensional image quality image.

  Next, a series of image processing according to the present embodiment will be described with reference to FIG. Note that the processes of steps S510 to S530 and step S550 according to the present embodiment are the same as those of the tenth embodiment, so description thereof will be omitted. Note that when the image quality of the input image is unconditionally improved, the process of step S530 may be omitted after the process of step S520, and the process may proceed to step S540.

  When the image quality improvement determination unit 403 determines in step S530 that the input image can be handled by the image quality improvement engine, the process proceeds to step S540. The image quality improvement determination unit 403 may make the determination based on the shooting conditions of the three-dimensional image, or may make the determination based on the shooting conditions of a plurality of two-dimensional images forming the three-dimensional image. May be.

  In step S540, the image quality improving unit 404 divides the acquired three-dimensional image into a plurality of two-dimensional images. The image quality improvement unit 404 inputs each of the plurality of divided two-dimensional images to a plurality of image quality improvement engines, performs image quality improvement processing in parallel, and generates a plurality of two-dimensional image quality images. The image quality improving unit 404 combines the plurality of generated two-dimensional high-quality images based on the acquired three-dimensional image to generate a three-dimensional high-quality image.

  In step S550, the output unit 405 causes the display unit 20 to display the generated three-dimensional high quality image. The display mode of the three-dimensional high-quality image may be arbitrary.

  As described above, the image quality improving unit 404 according to the present embodiment includes a plurality of image quality improving engines. The image quality improving unit 404 divides the three-dimensional input image into a plurality of two-dimensional images and uses a plurality of image quality improving engines in parallel to generate a plurality of two-dimensional high quality images. The image quality improving unit 404 generates a three-dimensional high quality image by integrating a plurality of two-dimensional high quality images.

  As a result, the image quality improving unit 404 according to the present embodiment can improve the image quality of the three-dimensional image using the image quality improving engine that has been learned using the teacher data of the two-dimensional image. In addition, the quality of the three-dimensional image can be improved more efficiently than in the tenth embodiment.

  The teacher data of the plurality of image quality improving engines may be different teacher data depending on the processing target to be processed by each image quality improving engine. For example, the first image-quality improving engine may perform learning using teacher data for the first image capturing area, and the second image-quality improving engine may perform learning using teacher data for the second image capturing area. In this case, each image quality improving engine can improve the image quality of the two-dimensional image with higher accuracy.

  Moreover, the output unit 405 may output the generated high-quality image to another device connected to the image capturing device 10 or the image processing device 400, as in the first embodiment. Further, the output data of the teacher data of the image quality improving engine is not limited to the high quality image subjected to the superimposing process, as in the first embodiment. That is, at least one of a processing group and an imaging method such as superposition processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing. You may use the high quality image obtained by performing.

<Twelfth Embodiment>
Next, an image processing apparatus according to the twelfth embodiment will be described with reference to FIGS. In the present embodiment, the acquisition unit 401 acquires an input image from the image management system 2000 instead of the image capturing device.

  Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 400 according to the first embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below focusing on the differences from the image processing apparatus according to the first embodiment. Note that the configuration of the image processing apparatus according to the present embodiment is similar to the configuration of the image processing apparatus 400 according to the first embodiment, so the same reference numerals are used for the configuration shown in FIG.

  FIG. 20 shows a schematic configuration of the image processing device 400 according to this embodiment. The image processing apparatus 400 according to the present embodiment is connected to the image management system 2000 and the display unit 20 via an arbitrary circuit or network. The image management system 2000 is an apparatus and system that receives and saves an image captured by an arbitrary image capturing device or an image that has undergone image processing. Further, the image management system 2000 may transmit an image in response to a request from a connected device, perform image processing on a stored image, or request an image processing request from another device. You can The image management system may include, for example, an image storage communication system (PACS).

  The acquisition unit 401 according to the present embodiment can acquire an input image from the image management system 2000 connected to the image processing apparatus 400. Further, the output unit 405 can output the high quality image generated by the high quality image generation unit 404 to the image management system 2000.

  Next, a series of image processing according to the present embodiment will be described with reference to FIG. Note that the processes of steps S520 to S540 according to the present embodiment are the same as those of the first embodiment, so description thereof will be omitted. Note that when the image quality of the input image is unconditionally improved, the process of step S530 may be omitted after the process of step S520, and the process may proceed to step S540.

  In step S510, the acquisition unit 401 acquires an image stored in the image management system 2000 as an input image from the image management system 2000 connected via a circuit or a network. The acquisition unit 401 may acquire the input image in response to a request from the image management system 2000. Such a request may be issued, for example, when the image management system 2000 saves an image, before transmitting the saved image to another device, or when displaying the saved image on the display unit 20. Further, the request is, for example, when a user operates the image management system 2000 to make a request for high image quality processing, or when the high image quality is used for the image analysis function of the image management system 2000. May be issued.

  The processing of steps S520 to S540 is the same as the processing in the first embodiment. When the image quality improving unit 404 generates a high quality image in step S540, the process proceeds to step S550. In step S550, if the high quality image is generated in step S540, the output unit 405 outputs the high quality image to the image management system 2000 as an output image. If a high quality image has not been generated in step S540, the input image is output to the image management system 2000 as an output image. Note that the output unit 405 may process the output image so that it can be used by the image management system 2000 or convert the data format of the output image, depending on the settings and implementation of the image processing apparatus 400.

  As described above, the acquisition unit 401 according to the present embodiment acquires the input image from the image management system 2000. Therefore, the image processing apparatus 400 according to the present embodiment, based on the images stored in the image management system 2000, provides a high-quality image suitable for image diagnosis to enhance the invasiveness of the photographer or the subject, It can be output without increasing labor. Further, the output high quality image can be stored in the image management system 2000 or can be displayed on a user interface included in the image management system 2000. Further, the output high-quality image can be used for an image analysis function included in the image management system 2000 or can be transmitted to another device connected to the image management system 2000 via the image management system 2000. .

  The image processing device 400, the image management system 2000, and the display unit 20 may be connected to other devices (not shown) via a circuit or a network. Further, although these devices are separate devices in the present embodiment, some or all of these devices may be integrally configured.

  The output unit 405 may output the generated high-quality image to another device connected to the image management system 2000 or the image processing device 400, as in the first embodiment.

<Thirteenth Embodiment>
Next, with reference to FIGS. 4, 5, 21A, and 21B, an image processing apparatus according to the thirteenth embodiment will be described. In the present embodiment, the image quality improving unit uses a plurality of images as input images to generate one high quality image.

  Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 400 according to the first embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below focusing on the differences from the image processing apparatus according to the first embodiment. Since the configuration of the image processing apparatus according to the present embodiment is similar to that of the image processing apparatus according to the first embodiment, the same reference numerals are used for the configuration illustrated in FIG. 4, and description thereof will be omitted. To do.

  The acquisition unit 401 according to the present embodiment acquires a plurality of images as input data to be processed from the image capturing device 10 or another device.

  The image quality improvement unit 404 according to the present embodiment includes the same image quality improvement engine as in the first embodiment. Further, the teacher data may be the same as in the first embodiment. The image quality improvement unit 404 inputs each of the plurality of images acquired by the acquisition unit 401 to the image quality improvement engine, superimposes the plurality of output high quality images, and outputs a final high quality image. To generate. Note that the image quality improving unit 404 may align the plurality of high quality images by an arbitrary method before performing the superimposing process on the plurality of high quality images.

  The output unit 405 causes the display unit 20 to display the final high quality image generated by the high quality image generation unit 404. The output unit 405 may display a plurality of input images on the display unit 20 together with the final high quality image. Further, the output unit 405 may display the plurality of generated high quality images on the display unit 20 together with the final high quality image and the input image.

  Next, a series of image processing according to this embodiment will be described with reference to FIGS. 5 and 21A. FIG. 21A is a flowchart of the image quality improving process according to this embodiment. Note that the processes of steps S510 to S530 according to the present embodiment are the same as those of the first embodiment, so description thereof will be omitted.

  However, in step S510, the acquisition unit 401 acquires a plurality of images, and in steps S520 and S530, the shooting conditions are acquired for each of the plurality of images, and it is determined whether or not the image quality improvement engine can deal with the shooting conditions. To be done. Note that when the image quality of the input image is unconditionally improved, the process of step S530 may be omitted after the process of step S520, and the process may proceed to step S540. In addition, when a part of the plurality of images is determined to be incapable of being dealt with by the image quality improving engine, the image can be excluded from the subsequent processing.

  In step S530, when the image quality improvement determination unit 403 determines that the plurality of input images can be dealt with by the image quality improvement engine, the process proceeds to step S540. When the process proceeds to step S540, the image quality improving process according to the present embodiment shown in FIG. 21A is started. In the image quality improving process according to the present embodiment, first, in step S2110, the image quality improving unit 404 inputs each of the plurality of input images to the image quality improving engine to generate a high image quality image group.

  Next, in step S2120, the high quality image generation unit 404 performs a superimposing process on the generated high quality image group to generate a final high quality image. The superposition processing may be performed by averaging processing such as arithmetic averaging or any other existing processing. Further, upon superimposing, the image quality improving unit 404 may align a plurality of high quality images by an arbitrary method and then superimpose them. When the image quality improving unit 404 generates the final high quality image, the process proceeds to step S550.

  In step S550, the output unit 405 causes the display unit 20 to display the generated final high-quality image.

  As described above, the image quality improving unit 404 according to the present embodiment generates one final high quality image from a plurality of input images. Since the image quality improvement by the image quality improvement engine is based on the input image, for example, when the lesion area or the like is not properly displayed in a certain input image, the high image quality image obtained by improving the image quality of the input image has a low pixel value. turn into. On the other hand, a lesion part or the like is appropriately displayed in another input image obtained by photographing the same portion, and a high pixel value may be obtained in a high-quality image obtained by improving the quality of the other input image. Therefore, by superimposing these high-quality images, it becomes possible to appropriately display the portion having the low or high pixel value, and it is possible to generate a high-contrast high-quality image. Note that the number of input images is set to be smaller than the number required for conventional superimposition, so that the cost such as a longer shooting time as in the conventional case can be further reduced.

  It should be noted that the action becomes remarkable when an input image using motion contrast data such as OCTA is used.

  Since the motion contrast data is obtained by detecting the temporal change of the shooting target at the time intervals at which the same part of the shooting target is repeatedly shot, for example, only a slight movement of the shooting target can be detected at a certain time interval. There are cases. On the other hand, when another time interval shooting is performed, the motion of the shooting target may be detected as a larger motion. Therefore, by superimposing the images obtained by improving the quality of the motion contrast images in each case, it is possible to interpolate the motion contrast that did not occur or was only slightly detected at a specific timing. Therefore, according to such processing, it is possible to generate a motion contrast image in which contrast enhancement is performed for more movements of the imaging target, and the examiner can grasp a more accurate state of the imaging target. it can.

  Therefore, when an image that depicts a temporally changing portion, such as an OCTA image, is used as an input image, high-quality images acquired at different times are superposed so that a predetermined part of the subject can be more accurately identified. It can be imaged in detail.

  In this embodiment, a high-quality image is generated from each of a plurality of input images, and the high-quality images are superposed to generate one final high-quality image. The method of generating the high quality image of is not limited to this. For example, in another example of the image quality improving process of the present embodiment illustrated in FIG. 21B, when the image quality improving process is started in step S540, the image quality improving unit 404 superimposes the input image groups in step S2130, Generate a single superimposed input image.

  After that, in step S2140, the image quality improving unit 404 inputs one superimposed input image to the image quality improving engine, and generates one high quality image. Even in such an image quality improving process, similarly to the above-described image quality improving process, it becomes possible to appropriately display a portion having a low or high pixel value in a plurality of input images, and to obtain a high contrast. A high quality image can be generated. This process can also exert a remarkable effect when a motion contrast image such as the OCTA image is used as an input image.

  In the case of performing the high image quality processing, a superimposition image of the same number of input images as the plurality of input images to be processed is used as the input data of the teacher data of the high image quality improvement engine. As a result, the image quality improving engine can perform an appropriate image quality improving process.

  Further, regarding the image quality improvement processing according to the present embodiment and the above-described other image quality improvement processing, the processing of combining the high image quality image group or the input image group is not limited to superposition. For example, one image may be generated by applying the MAP estimation process to these image groups. Further, one image may be generated by combining the high quality image group or the input image group.

  When a high-quality image group or an input image group is combined to generate one image, for example, an image having a wide gradation in a high-luminance region and an image having a wide gradation in a low-luminance region are used as input images. May be used. In this case, for example, an image having a high image quality of an image having a wide gradation in the high luminance region and an image having a high image quality of an image having a wide gradation in the low luminance region are combined. This makes it possible to generate an image capable of expressing a wider brightness range (dynamic range). In this case, the input data of the teacher data of the image quality improving engine is an image to be processed, which has a wide gradation in a high brightness region and a low image quality image having a wide gradation in a low brightness region. be able to. Further, the output data of the teacher data of the image quality improving engine can be a high quality image corresponding to the input data.

  Further, an image having a wide gradation in the high-luminance region and an image having a wide gradation in the low-luminance region may be combined, and the combined image may be rendered high in image quality by the image quality improving engine. Also in this case, it is possible to generate an image that can express a wider brightness range. In this case, the input data of the teacher data of the image quality improving engine is the low image quality image having a wide gradation in the high brightness region and the low image quality image having a wide gradation in the low brightness region, which are the processing targets. Can be a composite image. Further, the output data of the teacher data of the image quality improving engine can be a high quality image corresponding to the input data.

  In these cases, the high image quality engine can be used to improve the image quality of an image that can express a wider range of brightness, and processing can be performed with a smaller number of images or the like compared to the conventional case. , It is possible to provide an image suitable for image analysis with less cost.

  As an image capturing method for an image having a wide gradation in the high-luminance region and an image having a wide gradation in the low-luminance region, an arbitrary method such as a shorter or longer exposure time of the photographing device is adopted. You can do it. Further, the method of dividing the gradation width is not limited to the low-luminance region and the high-luminance region, and may be arbitrary.

  In addition, in the image quality improving process according to the present embodiment, a plurality of image quality improving engines may be used to process a plurality of input images in parallel. Note that the output unit 405 may output the generated high-quality image to another device connected to the image capturing device 10 or the image processing device 400, as in the first embodiment. Further, the output data of the teacher data of the image quality improving engine is not limited to the high quality image subjected to the superimposing process, as in the first embodiment. That is, at least one of a processing group and an imaging method such as superposition processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing. You may use the high quality image obtained by performing.

<Fourteenth Embodiment>
Next, an image processing apparatus according to the fourteenth embodiment will be described with reference to FIGS. In the present embodiment, the high quality image generation unit generates a high quality image by using a medium quality image generated from a plurality of low quality images as an input image.

  Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 400 according to the first embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below focusing on the differences from the image processing apparatus according to the first embodiment. Since the configuration of the image processing apparatus according to the present embodiment is similar to that of the image processing apparatus according to the first embodiment, the same reference numerals are used for the configuration illustrated in FIG. 4, and description thereof will be omitted. To do.

  The acquisition unit 401 according to the present embodiment acquires a medium-quality image obtained by superimposing a plurality of low-quality images as input data to be processed from the imaging device 10 or another device. Note that an arbitrary alignment process may be performed when superimposing low-quality images.

  The image quality improvement unit 404 according to the present embodiment includes the same image quality improvement engine as in the first embodiment. However, the image quality improving engine of the present embodiment is designed to input a medium image quality image having a medium image quality and output a high image quality image. The medium-quality image is a superimposed image generated by superimposing a plurality of low-quality image groups. A high quality image is a higher quality image than a medium quality image. Also, regarding the pair group that constitutes the teacher data used for the training of the image quality enhancement engine, the input data that configures each pair is a medium image quality image generated in the same way as the medium image quality image, and the output data is high. It is a high quality image.

  The output unit 405 causes the display unit 20 to display the high quality image generated by the high quality image generation unit 404. The output unit 405 may display the input image on the display unit 20 together with the high-quality image. In this case, the output unit 405 determines that the input image is an image generated from a plurality of low-quality images. May be displayed on the display unit 20.

  Next, a series of image processing according to the present embodiment will be described with reference to FIG. Note that the processes of steps S520 to S550 according to the present embodiment are the same as those of the first embodiment, so description thereof will be omitted.

  In step S510, the acquisition unit 401 acquires a medium-quality image as an input image from the imaging device 10 or another device. It should be noted that the acquisition unit 401 may acquire, as an input image, a medium image quality image generated by the image capturing apparatus 10 in response to a request from the image capturing apparatus 10. Such a request is, for example, when the image capturing apparatus 10 generates an image, before or after saving the image generated by the image capturing apparatus 10 in a recording device included in the image capturing apparatus 10, the saved image is displayed on the display unit 20. It may be issued when a high-quality image is used for image analysis processing, or the like.

  Subsequent processing is the same as the processing in the first embodiment, so description will be omitted.

  As described above, the acquisition unit 401 according to the present embodiment acquires, as an input image, a medium image quality image that is an image generated using a plurality of images of a predetermined region of the subject. In this case, since the input image is a clearer image, the image quality improving engine can generate the high quality image with higher accuracy. Note that the number of low-quality images used to generate a medium-quality image may be smaller than the number of images used to generate a conventional superimposed image.

  The medium image quality image is not limited to an image in which a plurality of low image quality images are superimposed, and may be an image obtained by applying MAP estimation processing to a plurality of low image quality images or an image obtained by combining a plurality of low image quality images. It may be. When a plurality of low-quality images are combined, images having different gradations may be combined.

  Moreover, the output unit 405 may output the generated high-quality image to another device connected to the image capturing device 10 or the image processing device 400, as in the first embodiment. Further, the output data of the teacher data of the image quality improving engine is not limited to the high quality image subjected to the superimposing process, as in the first embodiment. That is, at least one of a processing group and an imaging method such as superposition processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing. You may use the high quality image obtained by performing.

<Fifteenth Embodiment>
Next, an image processing device according to the fifteenth embodiment will be described with reference to FIGS. In the present embodiment, the image quality improving unit performs the image quality improvement (enlargement) of the input image together with the image quality improvement according to the first embodiment and the like.

  Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 400 according to the first embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below focusing on the differences from the image processing apparatus according to the first embodiment. Since the configuration of the image processing apparatus according to the present embodiment is similar to that of the image processing apparatus according to the first embodiment, the same reference numerals are used for the configuration illustrated in FIG. 4, and description thereof will be omitted. To do.

  The acquisition unit 401 according to the present embodiment acquires an image having a low image size (low size image) as an input image. The low-size image is an image having a smaller number of pixels forming the image than an image of a high image size (high-size image) output by an image quality improving engine described later. Specifically, for example, when the image size of the high size image is 1024 pixels in width, 1024 pixels in height, and 1024 pixels in depth, the image size of the low size image is 512 pixels, 512 pixels in height, and 512 pixels in depth. This is the case. In this context, increasing the image size in this specification means a process of increasing the number of pixels per image to increase the image size.

  The image quality improving unit 404 according to the present embodiment is provided with an image quality improving engine as in the first embodiment. However, the image quality improving engine of the present embodiment is configured to reduce the noise and enhance the contrast of the input image and increase the image size of the input image. Therefore, the image quality enhancement engine of the present embodiment is configured to input a low size image and output a high size image.

  In this regard, with respect to the pair group forming the teacher data of the image quality improving engine, the input data forming each pair is a low size image and the output data is a high size image. The high-size image used for output data can be acquired from a device having higher performance than the image-capturing device that acquired the low-size image, or can be acquired by changing the setting of the image-capturing device. If a high-size image group already exists, the low-size image group used as input data is acquired by reducing the high-size image group to the image size of the image expected to be acquired from the image capturing apparatus 10. May be. Further, as the high-size image, as in the first embodiment and the like, a superimposition of low-size images is used.

  Note that, regarding the enlargement of the image size of the input image by the image quality improving unit 404 according to the present embodiment, it is acquired as teacher data from a device having a higher performance than the photographing device 10 or the setting of the photographing device 10 is changed. Since it is acquired by this, it is different from the simple image enlargement. Specifically, the image size enlargement processing of the input image by the image quality improving unit 404 according to the present embodiment can reduce the deterioration of resolution as compared with the case where the image is simply enlarged.

  With such a configuration, the image quality improving unit 404 according to the present embodiment can generate a high quality image in which noise reduction and contrast enhancement are performed on the input image and the image size is increased.

  Next, a series of image processing according to the present embodiment will be described with reference to FIG. Note that the processes of step S520, step S530, and step S550 according to the present embodiment are the same as those of the first embodiment, so description thereof will be omitted. Note that when the image quality of the input image is unconditionally improved, the process of step S530 may be omitted after the process of step S520, and the process may proceed to step S540.

  In step S510, the acquisition unit 401 acquires a low-size image from the imaging device 10 or another device as input data to be processed. In addition, the acquisition unit 401 may acquire a low-size image generated by the image capturing apparatus 10 as an input image in response to a request from the image capturing apparatus 10. Such a request is, for example, when the image capturing apparatus 10 generates an image, before or after saving the image generated by the image capturing apparatus 10 in a recording device included in the image capturing apparatus 10, the saved image is displayed on the display unit 20. It may be issued when a high-quality image is used for image analysis processing, or the like.

  The processing of steps S520 and S530 is the same as the processing of the first embodiment, and therefore description thereof is omitted. In step S540, the image quality improving unit 404 inputs the input image to the image quality improving engine and generates an image having a high image size with noise reduction and contrast enhancement performed as the high quality image. Subsequent processing is the same as that of the first embodiment, and therefore description thereof is omitted.

  As described above, the image quality improving unit 404 according to the present embodiment generates a high quality image in which at least one of noise reduction and contrast enhancement is performed as compared with the input image and the image size is enlarged. . As a result, the image processing apparatus 400 according to the present embodiment can output a high-quality image suitable for image diagnosis without increasing the invasiveness of the photographer or the subject or increasing the labor.

  In this embodiment, one high image quality engine generates a high image quality image that has been subjected to the image quality improvement process and the high resolution process according to the first embodiment etc., but the configuration for performing these processes is It is not limited to this. For example, the image quality improving unit may include an image quality improving engine that performs the image quality improving process according to the first embodiment and the like, and another image quality improving engine that performs the image size increasing process.

  In this case, the image quality improving engine that performs the image quality improving process according to the first embodiment or the like may use a machine learning model that has been learned as in the image quality improving engine according to the first embodiment or the like. it can. Further, as the input data of the teacher data of the image quality improving engine that performs the image size increasing process, the high quality image generated by the image quality improving engine according to the first embodiment or the like is used. Further, as the output data of the teacher data of the image quality improving engine, the high quality image generated by the image quality improving engine according to the first embodiment or the like is used for the image acquired by the high performance image capturing apparatus. As a result, the high image quality engine that performs the high image size processing can generate a final high quality image that is a high image size of the high quality image that has been subjected to the high image quality processing according to the first embodiment and the like. You can

  Further, the image size increasing process by the image quality improving engine may be performed before the image quality improving process by the image quality increasing process engine according to the first embodiment or the like. In this case, the teacher data for the high image quality engine that performs the high image size processing is composed of a pair group of input data, which is a low size image, and output data, which is a high size image, acquired by the image capturing apparatus. Further, as the teacher data of the image quality enhancement engine that performs the image quality enhancement processing according to the first embodiment and the like, a high-size image is composed of input data and an image in which the high-size image is superimposed is composed of a pair group of output data. To do.

  With such a configuration as well, the image processing apparatus 400 can perform at least one of noise reduction and contrast enhancement as compared with the input image and can generate an image with an enlarged image size as a high-quality image. it can.

  In the present embodiment, the image quality improving process according to the first embodiment and the like has been described with respect to the configuration in which the superimposed image is used as the output data of the teacher data, but the output data is the same as in the first embodiment. It is not limited to this. That is, at least one of a processing group and an imaging method such as superposition processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing. You may use the high quality image obtained by performing.

  Note that the output unit 405 may output the generated high-quality image to another device connected to the image capturing device 10 or the image processing device 400, as in the first embodiment.

<Sixteenth Embodiment>
Next, an image processing apparatus according to the sixteenth embodiment will be described with reference to FIGS. In the present embodiment, the image quality improving unit performs high spatial resolution as well as high image quality according to the first embodiment and the like.

  Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 400 according to the first embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below focusing on the differences from the image processing apparatus according to the first embodiment. Since the configuration of the image processing apparatus according to the present embodiment is similar to that of the image processing apparatus according to the first embodiment, the same reference numerals are used for the configuration illustrated in FIG. 4, and description thereof will be omitted. To do.

  The acquisition unit 401 according to the present embodiment acquires a low spatial resolution image as an input image. The low spatial resolution image is an image having a lower spatial resolution than the high spatial resolution image output by the image quality improving unit 404.

  The image quality improving unit 404 is provided with an image quality improving engine as in the first embodiment. However, the image quality enhancement engine of the present embodiment is configured to reduce noise and enhance the contrast of the input image and to increase the spatial resolution of the input image. Therefore, the image quality enhancement engine according to the present embodiment is configured to input a low spatial resolution image and output a high spatial resolution image.

  In this regard, with respect to the pair group forming the teacher data of the image quality improving engine, the input data forming each pair is a low spatial resolution image and the output data is a high spatial resolution image. It should be noted that the high spatial resolution image can be acquired from a device having a higher performance than the photographing device that has acquired the low spatial resolution image, or by changing the setting of the photographing device. Further, as the high spatial resolution image, an image in which low spatial resolution images are superimposed is used as in the first embodiment and the like.

  With such a configuration, the image quality improving unit 404 according to the present embodiment can generate a high quality image in which noise reduction and contrast enhancement are performed on the input image and the spatial resolution is increased.

  Next, a series of image processing according to the present embodiment will be described with reference to FIG. Note that the processes of step S520, step S530, and step S550 according to the present embodiment are the same as those of the first embodiment, so description thereof will be omitted. Note that when the image quality of the input image is unconditionally improved, the process of step S530 may be omitted after the process of step S520, and the process may proceed to step S540.

  In step S510, the acquisition unit 401 acquires a low spatial resolution image as input data to be processed from the imaging device 10 or another device. Note that the acquisition unit 401 may acquire the low spatial resolution image generated by the imaging device 10 as an input image in response to a request from the imaging device 10. Such a request is, for example, when the image capturing apparatus 10 generates an image, before or after saving the image generated by the image capturing apparatus 10 in a recording device included in the image capturing apparatus 10, the saved image is displayed on the display unit 20. It may be issued when a high-quality image is used for image analysis processing, or the like.

  The processing of steps S520 and S530 is the same as the processing of the first embodiment, and therefore description thereof is omitted. In step S540, the image quality improving unit 404 inputs the input image to the image quality improving engine and generates an image with high spatial resolution while noise reduction and contrast enhancement are performed as the high quality image. Subsequent processing is the same as that of the first embodiment, and therefore description thereof is omitted.

  As described above, the image quality improving unit 404 according to the present embodiment performs at least one of noise reduction and contrast enhancement as compared to the input image and generates an image with improved spatial resolution as a high quality image. To do. As a result, the image processing apparatus 400 according to the present embodiment can output a high-quality image suitable for image diagnosis without increasing the invasiveness of the photographer or the subject or increasing the labor.

  In this embodiment, one high image quality engine generates a high image quality image that has been subjected to the image quality improvement process and the high resolution process according to the first embodiment etc., but the configuration for performing these processes is It is not limited to this. For example, the image quality improving unit may include an image quality improving engine for performing the image quality improving process according to the first embodiment and the like, and another image quality improving engine for performing the high resolution process.

  In this case, the image quality improving engine that performs the image quality improving process according to the first embodiment or the like may use a machine learning model that has been learned as in the image quality improving engine according to the first embodiment or the like. it can. Further, as the input data of the teacher data of the image quality enhancement engine that performs the resolution enhancement process, the high quality image generated by the image quality enhancement engine according to the first embodiment or the like is used. Further, as the output data of the teacher data of the image quality improving engine, the high quality image generated by the image quality improving engine according to the first embodiment or the like is used for the image acquired by the high performance image capturing apparatus. As a result, the high image quality engine that performs the high spatial resolution processing can generate a final high quality image with high spatial resolution for the high image quality image that has been subjected to the high image quality processing according to the first embodiment and the like. You can

  Further, the high spatial resolution enhancement processing by the high image quality enhancement engine may be performed before the high image quality enhancement processing by the high resolution enhancement engine according to the first embodiment or the like. In this case, the teacher data about the high image quality engine that performs the high spatial resolution processing is composed of a pair group of input data that is a low spatial resolution image and output data that is a high spatial resolution image acquired by the image capturing device. . Further, as the teacher data of the image quality improving engine that performs the image quality improving process according to the first embodiment and the like, a pair group of the input data of the high spatial resolution image and the output data of the image obtained by superimposing the high spatial resolution image It consists of.

  With such a configuration as well, the image processing apparatus 400 can generate at least one of noise reduction and contrast enhancement as compared with the input image and an image with improved spatial resolution as a high-quality image. .

  In the present embodiment, the image quality improving process according to the first embodiment and the like has been described with respect to the configuration in which the superimposed image is used as the output data of the teacher data, but the output data is the same as in the first embodiment. It is not limited to this. That is, at least one of a processing group and an imaging method such as superposition processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing. You may use the high quality image obtained by performing.

  Further, the image quality improving unit 404 may use the image quality improving engine to perform the image quality improving process according to the fifteenth embodiment in addition to the high spatial resolution process. In this case, at least one of noise reduction and contrast enhancement compared to the input image is performed, and an image having a higher image size and higher spatial resolution than the input image is generated as a high quality image. You can As a result, the image processing apparatus 400 according to the present embodiment can output a high-quality image suitable for image diagnosis without increasing the invasiveness of the photographer or the subject or increasing the labor.

  Note that the output unit 405 may output the generated high-quality image to another device connected to the image capturing device 10 or the image processing device 400, as in the first embodiment.

<17th Embodiment>
Next, an image processing device according to the seventeenth embodiment will be described with reference to FIGS. In the present embodiment, the analysis unit performs image analysis on the high quality image generated by the high quality image generation unit.

  Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 400 according to the first embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below focusing on the differences from the image processing apparatus according to the first embodiment.

  FIG. 22 shows a schematic configuration of the image processing apparatus 2200 according to this embodiment. The image processing apparatus 2200 according to the present embodiment is provided with an analysis unit 2208 in addition to the acquisition unit 401, the shooting condition acquisition unit 402, the image quality improvement possibility determination unit 403, the image quality improvement unit 404, and the output unit 405. ing. The image processing apparatus 2200 may be composed of a plurality of devices provided with some of these components. Here, the acquisition unit 401, the shooting condition acquisition unit 402, the image quality improvement possibility determination unit 403, the image quality improvement unit 404, and the output unit 405 have the same configurations as the image processing apparatus according to the first embodiment. 4, the same reference numerals are used for the configuration shown in FIG. 4, and description thereof is omitted.

  The analysis unit 2208 applies a predetermined image analysis process to the high quality image generated by the high quality image generation unit 404. The image analysis processing is, for example, in the field of ophthalmology, segmentation of retinal layers, layer thickness measurement, papillary three-dimensional shape analysis, lamina cribrosa analysis, blood vessel density measurement of OCTA images, and corneal shape analysis for images acquired by OCT. Etc., including any existing image analysis processing. Further, the image analysis processing is not limited to the analysis processing in the field of ophthalmology, and includes, for example, any existing analysis processing in the radiation field such as diffusion tensor analysis and VBL (Voxel-based Morphometry) analysis.

  The output unit 405 can display the high-quality image generated by the image quality improving unit 404 on the display unit 20 and also display the analysis result of the image analysis processing by the analysis unit 2208. The output unit 405 may display only the image analysis result by the analysis unit 2208 on the display unit 20, or may output the image analysis result to the imaging device 10, the image management system, other devices, or the like. The display form of the analysis result may be arbitrary according to the image analysis processing performed by the analysis unit 2208, and may be displayed as an image, a numerical value, or a character, for example.

  Hereinafter, with reference to FIG. 23, a series of image processing according to the present embodiment will be described using an OCTA En-Face image as an example. FIG. 23 is a flowchart of a series of image processing according to this embodiment. Note that the processing of steps S2310 to S2340 according to the present embodiment is the same as the processing of steps S510 to S540 in the first embodiment, so description thereof will be omitted. Note that when the image quality of the input image is unconditionally improved, the process of step S2330 may be omitted after the process of step S2320, and the process may proceed to step S2340.

  In step S2340, the image quality enhancing unit 404 enhances the image quality of the OCTA En-Face image, and the process proceeds to step S2350. In step S2350, the analysis unit 2208 performs image analysis on the high quality image generated in step S2340. As image analysis in an OCTA En-Face image with high image quality, an arbitrary binarization process can be applied to detect a portion corresponding to a blood vessel (blood vessel region) from the image. The area density can be analyzed by obtaining the ratio of the detected blood vessel-corresponding portion to the image. In addition, by thinning a portion corresponding to the binarized blood vessel, an image having a line width of 1 pixel can be obtained, and a ratio (also called skeleton density) of blood vessels that does not depend on the thickness can be obtained. You may make it analyze the area and shape (circularity etc.) of a blood-free area | region (FAZ) using these images. As a method of analysis, the above-mentioned numerical values may be calculated from the entire image, or a user interface (not shown) may be used to determine a specified region of interest (ROI) based on instructions from an examiner (user). Alternatively, the numerical value may be calculated. The ROI setting is not necessarily designated by the examiner, but may be such that a predetermined area is designated automatically. Here, the various parameters described above are examples of analysis results regarding blood vessels, and may be any parameters as long as they are parameters regarding blood vessels. The analysis unit 2208 may perform a plurality of image analysis processes. That is, although an example of analyzing the OCTA En-Face image is shown here, not only this but also the retinal layer segmentation, layer thickness measurement, papilla three-dimensional shape analysis, lamina cribrosa with respect to images acquired by OCT at the same time. Analysis or the like may be performed. In this regard, the analysis unit 2208 may perform some or all of the plurality of image analysis processes according to an instruction from the examiner via an arbitrary input device.

  In step S2360, the output unit 405 causes the display unit 20 to display the high-quality image generated by the image quality enhancement unit 404 and the analysis result of the analysis unit 2208. The output unit 405 may output the high-quality image and the analysis result to different display units or devices. Further, the output unit 405 may display only the analysis result on the display unit 20. Furthermore, when the analysis unit 2208 outputs a plurality of analysis results, the output unit 405 may output a part or all of the plurality of analysis results to the display unit 20 or another device. For example, the analysis result regarding the blood vessel in the OCTA En-Face image may be displayed on the display unit 20 as a two-dimensional map. Further, a value indicating the analysis result regarding the blood vessel in the OCTA En-Face image may be displayed on the display unit 20 while being superimposed on the OCTA En-Face image.

  As described above, the image processing apparatus 2200 according to the present embodiment further includes the analysis unit 2208 that analyzes the high quality image, and the output unit 405 causes the display unit 20 to display the analysis result of the analysis unit 2208. As described above, in the image processing apparatus 2200 according to the present embodiment, the high-quality image is used for image analysis, so that the accuracy of analysis can be improved.

  The output unit 405 may output the generated high-quality image to another device connected to the image capturing device 10 or the image processing device 2200, as in the first embodiment. Further, the output data of the teacher data of the image quality improving engine is not limited to the high quality image subjected to the superimposing process, as in the first embodiment. That is, at least one of a processing group and an imaging method such as superposition processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing. You may use the high quality image obtained by performing.

<Eighteenth embodiment>
Next, an image processing apparatus according to the eighteenth embodiment will be described with reference to FIG. In the present embodiment, an example will be described in which the image quality improving unit generates a high quality image by adding noise to the image during learning and learning the noise component.

  Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 400 according to the first embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below focusing on the differences from the image processing apparatus according to the first embodiment. Since the configuration of the image processing apparatus according to the present embodiment is similar to that of the image processing apparatus according to the first embodiment, the same reference numerals are used for the configuration illustrated in FIG. 4, and description thereof will be omitted. To do.

  The acquisition unit 401 according to the present embodiment acquires an image as input data to be processed from the imaging device 10 or another device. An example of the configuration of the CNN in the image quality improving unit according to this embodiment will be described with reference to FIG. FIG. 24 shows an example of a machine learning model configuration in the image quality improving unit 404. The configuration shown in FIG. 24 is configured by a plurality of layer groups that are responsible for processing the input value group and outputting it. As the types of layers included in the above configuration, there are a convolution layer, a downsampling layer, an upsampling layer, and a merging layer, as shown in FIG. The convolutional layer is a layer that performs convolution processing on an input value group according to parameters such as a set filter kernel size, the number of filters, a stride value, and a dilation value. The number of dimensions of the kernel size of the filter may be changed according to the number of dimensions of the input image. The down-sampling layer is a process of reducing the number of output value groups than the number of input value groups by thinning out or combining the input value groups. Specifically, for example, there is Max Pooling processing. The upsampling layer is a process for making the number of output value groups larger than the number of input value groups by duplicating the input value group or adding a value interpolated from the input value group. Specifically, for example, there is linear interpolation processing. The combining layer is a layer that inputs a value group such as an output value group of a certain layer or a pixel value group forming an image from a plurality of sources, and performs a process of combining and adding the value groups. With such a configuration, the pixel value group forming the input image Im2410 is output through the convolution processing block, and the pixel value group forming the input image Im2410 is combined in the combining layer. After that, the combined pixel value group is formed into a high-quality image Im2420 by the final convolution layer. Although not shown, examples of modification of the configuration of the CNN include, for example, incorporating a batch normalization layer after the convolutional layer and an activation layer using a normalized linear function (Rectifier Linear Unit). You may.

  The high image quality engine of the present embodiment inputs a low image quality image obtained by adding a first noise component to an image obtained from the image capturing device 10 or another device, and outputs as output data from the image capturing device 10 or another device. The image obtained by adding the second noise component to the obtained image is trained as a high quality image. That is, as the teacher image at the time of learning of the present embodiment, the low-quality image and the high-quality image are common, and the noise components in the respective images are different. Since the same images are used, it is not necessary to perform the alignment when forming the paired images.

  As the noise component, Gaussian noise, modeled noise specific to the target image, or the like is added as noise. However, the first and second noises are different noises. The different noises mean that spatial locations (pixel positions) to which noises are added are different or noise values are different. As the noise peculiar to the target image, for example, in the case of OCT, it is possible to estimate the noise based on the data captured without placing the model eye or the eye to be inspected and use them as a noise model. In the case of OCTA, it can be used as a noise model based on noise that appears in the range of avascular region (FAZ) and noise that appears in an image of a model eye that schematically reproduces the flow of blood.

  In the case of Gaussian noise, the standard deviation or variance is defined as the size of noise, and noise is randomly given to the image based on these numerical values. The average value as a whole may not change as a result of giving random noise. That is, the average value of the noise added to each pixel of one image is set to zero. Here, it is not necessary that the average value be 0, as long as noises having different patterns can be added to the input data and the output data. Further, it is not necessary to add noise to both the input data and the output data, and noise may be added to either one. Here, when noise is not added, for example, a false image of a blood vessel may occur in the image after high image quality improvement, but this is considered to occur when the difference between the images before and after high image quality improvement is relatively large. It is also possible. Therefore, the difference between the images before and after the high image quality may be reduced. At this time, at the time of learning, two images obtained by adding different patterns of noise to the low-quality image and the high-quality image may be paired images, or different patterns may be used for the high-quality image. The two images obtained by adding the noise of 1 may be paired images.

  The output unit 405 causes the display unit 20 to display the high quality image generated by the high quality image generation unit 404. The output unit 405 may display the input image on the display unit 20 together with the high-quality image.

  Subsequent processing is the same as the processing in the first embodiment, so description will be omitted.

  In the present embodiment, a high-quality image is obtained by using an image obtained by adding a first noise component and a second noise component different from the first noise component to a low-quality image obtained from the image capturing device 10 or another device. However, the configuration for performing these processes is not limited to this. For example, as the image to which noise is added, the first and second noise components may be added to the high-quality image subjected to the superimposing processing shown in the first embodiment. That is, the image obtained by adding the first noise component to the superposition processed image may be learned as a low image quality image, and the image obtained by adding the second noise component to the superposition processed image may be learned as a high quality image.

  Furthermore, in the present embodiment, an example in which learning is performed using the first and second noise components has been described, but the present invention is not limited to this. For example, the first noise component may be added only to the low quality image, and the learning may be performed to the high quality image without adding the noise component. The image at that time may be an image obtained from the photographing device 10 or another device, or may be an image obtained by superposing the images.

  In the present embodiment, the photographing conditions of the image are not specified, but learning is performed by using images with different photographing ranges and different numbers of scans, front images with different photographing regions and different depths, and the like.

  In the above, an image obtained from the image capturing device 10 or another device, a noise image in which noise is added to the image, a superimposition processing image, and an image in which noise is added to the superimposition processing image have been described. However, these combinations are not limited to those described above, and low-quality images and high-quality images may be combined in any way.

<19th Embodiment>
Next, an image processing apparatus according to the nineteenth embodiment will be described with reference to FIGS. In the present embodiment, the image quality improving unit includes a plurality of image quality improving engines and generates a plurality of high quality images for the input image. Then, an example in which the combining unit 2505 combines a plurality of high quality images output from a plurality of high quality engines will be described.

  Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 400 according to the first embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below focusing on the differences from the image processing apparatus according to the first embodiment. Since the configuration of the image processing apparatus according to the present embodiment is similar to that of the image processing apparatus according to the first embodiment, the same reference numerals are used for the configuration illustrated in FIG. 4, and description thereof will be omitted. To do.

  The acquisition unit 401 according to the present embodiment acquires an image as input data to be processed from the imaging device 10 or another device.

  The image quality improvement unit 404 according to the present embodiment includes a plurality of image quality improvement engines as in the second embodiment. Here, each of the plurality of image quality enhancement engines performs learning by using different learning data regarding at least one of the imaging region, the imaging angle of view, the front image with different depth, the noise component, and the resolution of the image. I went there. The image quality improving unit 404 uses a plurality of image quality improving engines corresponding to at least one of a captured region of the input image, a captured field angle, a front image with different depths, a noise component, and an image resolution, and thus a high image quality is achieved. Generate an image.

  FIG. 26 is a flowchart of a series of image processing according to this embodiment. Note that the processes of steps S2610 and S2620 according to the present embodiment are the same as the processes of steps S510 and S520 in the first embodiment, so description thereof will be omitted. In addition, when the image quality of an input image is unconditionally improved, the process of step S2630 may be omitted after the process of step S2620 and the process may proceed to step S2640.

  In step S2620, as in the first embodiment, when the shooting condition acquisition unit 402 acquires the shooting condition group of the input image, the process proceeds to step S2630. In step S2630, as in the second embodiment, the image quality improvement determination unit 403 uses the acquired image capturing condition group to input the input image by any of the image quality improvement engines included in the image quality improvement unit 404. It is determined whether it can be dealt with.

  If the image quality improvement availability determination unit 403 determines that none of the image quality enhancement engine groups can handle the input image, the process proceeds to step S2660. On the other hand, when the image quality improvement availability determination unit 403 determines that one of the image quality enhancement engine groups can handle the input image, the process proceeds to step S2640. Note that, depending on the setting and implementation of the image processing apparatus 400, as in the first embodiment, even if the image quality enhancement engine determines that some shooting conditions cannot be handled, step S2640 is performed. You may.

  In step S2640, image quality improvement unit 404 inputs the input image acquired in step S2610 to each of the image quality improvement engine groups to generate a high image quality image group.

  In step S2650, the synthesizing unit 2405 synthesizes some high-quality images of the high-quality image group generated in step S2640. Specifically, for example, as shown in the first embodiment, a low image quality image acquired from the image capturing apparatus 10 and an image group acquired by capturing the low image quality image a plurality of times are overlapped with an arithmetic mean or the like. Learning is performed using a first image quality enhancement engine learned using a paired image with a high quality image obtained by the matching process, and a paired image obtained by adding noise to the image as shown in the eighteenth embodiment. The result of the two high-quality images is combined with the second high-quality image engine. As a synthesizing method, averaging or weighted averaging can be used for synthesizing.

  In step S2660, output unit 405 causes display unit 20 to display the image combined in step S2650 and outputs the image to another device. However, if it is determined in step S2630 that the input image cannot be processed, the output unit 405 outputs the input image as the output image. It should be noted that the output unit 405 causes the display unit 20 to display that the output image is the same as the input image when the examiner instructs the input image or when the input image cannot be processed. Good.

<Twentieth Embodiment>
Next, an image processing apparatus according to the twentieth embodiment will be described with reference to FIG. In the present embodiment, an example will be described in which the image quality improving unit uses the output result of the first image quality improving engine and the second image quality improving engine generates a high quality image.

  Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 400 according to the first embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below focusing on the differences from the image processing apparatus according to the first embodiment. Since the configuration of the image processing apparatus according to the present embodiment is similar to that of the image processing apparatus according to the first embodiment, the same reference numerals are used for the configuration illustrated in FIG. 4, and description thereof will be omitted. To do.

  The acquisition unit 401 according to the present embodiment acquires an image as input data to be processed from the imaging device 10 or another device.

  The image quality improving unit 404 according to the present embodiment includes a plurality of image quality improving engines similar to those in the first embodiment. The image quality improving unit of the present embodiment learns the low image quality image acquired as the input data from the image capturing device 10 or another device and the medium image quality image generated from the plurality of low image quality images as the output data. Equipped with an image quality engine. Further, it is provided with a second image quality improving engine that has learned an image output from the first image quality improving engine and an image having a higher image quality than the medium image quality image as output data. Note that the medium-quality image is the same as that in the fourteenth embodiment, so description thereof will be omitted.

  The output unit 405 causes the display unit 20 to display the high quality image generated by the high quality image generation unit 404. The output unit 405 may display the input image on the display unit 20 together with the high-quality image. In this case, the output unit 405 determines that the input image is an image generated from a plurality of low-quality images. May be displayed on the display unit 20.

  Next, a series of image processing according to the present embodiment will be described with reference to FIG. Note that the processes of steps S510 to S530 according to the present embodiment are the same as those of the first embodiment, so description thereof will be omitted.

  In step S540, the image quality improving unit 404 uses the image quality improving engine to improve the image quality of the input image, and generates a high image quality image more suitable for image diagnosis than the input image. Specifically, the image quality improving unit 404 inputs the input image to the first image quality improving engine and generates the first image quality improving image. Further, the first high quality image is input to the second high quality image engine to obtain the second high quality image. The image quality improving engine generates a high quality image that is obtained by performing a superimposing process using an input image based on a machine learning model in which machine learning is performed using teacher data. Therefore, the image quality improving engine can generate a high quality image in which noise is reduced or contrast is emphasized more than the input image.

  Subsequent processing is the same as the processing in the first embodiment, so description will be omitted.

  It should be noted that in the present embodiment, the first high image quality engine and the first high image quality image and the high image quality image obtained by learning a pair of the low image quality image and the medium image quality image obtained from the image capturing device 10 or another device are used. Although the high-quality image is generated using the second high-quality image engine learned by the pair, the configuration for performing these processes is not limited to this. For example, the pair of images to be learned by the first image quality improving engine is the engine for learning the noise described in the eighteenth embodiment, and the second image quality improving engine is the image quality of the first image quality and the image quality. You may make it learn by pairing with an image. In the opposite configuration, the first high-quality image engine that learned the low-quality image and the medium-quality image as a pair, and the second high-quality image engine generates an image with noise added to the first high-quality image. It may be a learned engine.

  Furthermore, both the first image-quality improving engine and the second image-quality improving engine may be noise learning engines described in the eighteenth embodiment. In this case, for example, the first image-quality improving engine learns a pair of images in which the first and second noises are added to the high-quality image generated by the superimposed processing image, and the second image-quality improving engine , A pair of images obtained by adding the first and second noises to the first high-quality image generated by the first high-quality image engine. In the present embodiment, two image quality improving engines have been described, but the present invention is not limited to this, and the third and fourth engines may be further connected to perform processing. By cleaning the images used for learning, a network that can easily generate smoother and sharper images is constructed.

<Twenty-first embodiment>
Next, an image processing apparatus according to the 21st embodiment will be described with reference to FIGS. In the first embodiment, the image quality improving unit 404 includes one image quality improving engine. On the other hand, in the present embodiment, the image quality improving unit includes a plurality of image quality improving engines that perform machine learning using different teacher data, and generates a plurality of high quality images for the input image.

  Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 400 according to the second embodiment. Therefore, in the following, the image processing apparatus according to the present embodiment will be described focusing on differences from the image processing apparatuses according to the first and second embodiments. Since the configuration of the image processing apparatus according to the present embodiment is similar to the configuration of the image processing apparatus according to the first and second embodiments, the configuration illustrated in FIG. The description is omitted.

  The image quality improving unit 404 according to the present embodiment includes two or more image quality improving engines in which machine learning is performed using different teacher data. Here, a method of creating the teacher data group according to the present embodiment will be described. First, a pair group of an original image as input data and a superposed image as output data, which are captured by images with various imaging ranges and different scan numbers, is prepared. Explaining OCT and OCTA as an example, for example, a pair of the first image group captured by 300 A scans and 300 B scans in a range of 3 × 3 mm and 500 A in a range of 10 × 10 mm. A pair of a second image group captured by scanning and 500 B-scans is used. At this time, the scan densities of the first image group pair and the second image group pair are twice different. Therefore, these image groups are separately grouped. Then, if there is an image group obtained by taking 600 A scans and 600 B scans in a range of 6 × 6 mm, the same group as the first image group is set. That is, here, image groups having the same or almost the same scan density (error of about 10%) are grouped in the same group.

  Next, the teacher data group is created by grouping the pair groups for each scan density. For example, the first teacher data composed of a pair group captured and acquired at a first scan density, and the second teacher data composed of a pair group acquired and captured at a second scan density. In this way, the teacher data group is created.

  After that, machine learning is performed by different image quality improving engines using each teacher data. For example, a first image enhancement engine corresponding to a machine learning model trained with first teacher data, a second image enhancement engine corresponding to a machine learning model trained with second teacher data, and so on. Prepare a high-quality image engine group.

  Since such high image quality engine has different teacher data used for training the corresponding machine learning model, the degree to which the high quality of the input image can be achieved differs depending on the image capturing conditions of the image input to the high image quality engine. . Specifically, the first image quality improvement engine has a high degree of image quality improvement for the input image acquired at the first scan density and for the image acquired at the second scan density. Has a low degree of high image quality. Similarly, the second image quality improvement engine has a high degree of image quality improvement for the input image acquired at the second scan density and a high image quality improvement for the image acquired at the first scan density. The degree of image quality is low.

  On the other hand, there are cases in which a sufficient number of images having different shooting ranges and different scan densities cannot be collected as teacher data during learning. In that case, for those image groups, as shown in the eighteenth embodiment, an image quality improving engine that learns noise components is prepared.

  The image quality enhancement engine that learned the noise component is not easily affected by the scan density at the time of shooting, so this is applied when an image with a scan density that has not been learned is input.

  Since each of the teacher data is composed of a pair group grouped according to the scan density, the image quality tendency of the image groups forming the pair group is similar. Therefore, the high image quality engine can achieve higher image quality more effectively than the high image quality engine according to the first embodiment as long as the corresponding scan density is achieved. The imaging condition for grouping the pair of teacher data is not limited to the scan density, but may be an imaging region, images of different depths in the front image, or a combination of two or more of these. You may.

  Hereinafter, a series of image processing according to the present embodiment will be described with reference to FIG. FIG. 27 is a flowchart of a series of image processing according to this embodiment. Note that the processing of steps S2710 and S2720 is the same as steps S510 and S520 according to the first embodiment, so description thereof will be omitted.

  When the shooting condition of the input image is acquired in step S2720, the process proceeds to step S2730. In step S2730, the image quality improvement possibility determination unit 403 can handle the input image by using one of the image quality improvement engine groups included in the image quality improvement unit 404 using the shooting condition group acquired in step S2720. Or not.

  When the image quality improvement determination unit 403 determines that the image capturing conditions are not satisfied, the process proceeds to step S2770. On the other hand, if the image quality improvement determination unit 403 determines that the image capturing conditions are satisfied, the process proceeds to step S2740.

  In step S2740, the image quality improving unit 404 performs the image quality improving process from the image quality improving engine group based on the shooting condition of the input image acquired in step S2720 and the information of the teacher data of the image quality improving engine group. Select the image quality engine. Specifically, for example, for the scan density of the imaging condition group acquired in step S2720, the high image quality engine having the information of the teacher data regarding the scan density and having a high degree of high image quality is selected. In the above example, when the scan density is the first scan density, the image quality improving unit 404 selects the first image quality improving engine.

  On the other hand, in step S2770, image quality enhancement unit 404 selects the image quality enhancement engine that learned the noise component.

  In step S2750, the image quality improving unit 404 uses the image quality improving engine selected in steps S2740 and S2770 to generate a high quality image obtained by improving the quality of the input image. After that, in step S2760, the output unit 405 outputs the high-quality image in step S2750 and displays it on the display unit 20. The output unit 405 may display, when displaying the high-quality image on the display unit 20, that the high-quality image is generated by using the high-quality image engine selected by the high-quality image generation unit 404. .

  As described above, the image quality improving unit 404 according to the present embodiment includes a plurality of image quality improving engines that perform learning using different learning data. Here, each of the plurality of image quality enhancement engines performs learning by using different learning data regarding at least one of the imaging region, the imaging angle of view, the front image with different depth, and the resolution of the image. Is. Furthermore, for the data for which the correct answer data was not collected sufficiently, the learning was performed using the noise component. The image quality improving unit 404 uses the image quality improving engine corresponding to at least one of these to generate a high quality image.

  With such a configuration, the image processing device 400 according to the present embodiment can generate a more effective high quality image.

<Twenty-second Embodiment>
Next, an image processing apparatus according to the twentieth embodiment will be described with reference to FIGS. In the present embodiment, the wide-angle-of-view image generating unit generates a wide-angle-of-view image by using the plurality of high-quality images generated by the high-quality improving unit.

  Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 400 according to the first embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below focusing on the differences from the image processing apparatus according to the first embodiment. Since the configuration of the image processing apparatus according to the present embodiment is similar to that of the image processing apparatus according to the first embodiment, the same reference numerals are used for the configuration illustrated in FIG. 4, and description thereof will be omitted. To do.

  FIG. 31A is a flowchart of a series of image processing according to this embodiment. In step S3110, the acquisition unit 401 acquires a plurality of images (at least two images) as input data from the imaging device 10 or another device. The plurality of images are images in which different positions of the same subject (such as an eye to be inspected) are captured, and images in which portions of the images are not completely overlapped with the subject are captured. Taking the case of photographing the eye to be examined as an example, the position of the fixation lamp is changed at the time of photographing, and the eye to be examined pays attention to the fixation lamp, thereby acquiring images of different places photographed in the same eye. You can It should be noted that at the time of image capturing, it is desirable to change the position of the fixation lamp so that at least about 20% of overlapping areas of adjacent images are the same. FIG. 32A shows an example of an OCTA En-Face image captured by changing the position of the fixation lamp so that adjacent images partially overlap. FIG. 32A shows an example in which the position of the fixation lamp is changed and five different locations are photographed. Although FIG. 32 shows five images as an example, the number of images is not limited to five and may be two or more.

  Note that the process of step S3120 according to the present embodiment is the same as the process of step S520 in the first embodiment, so description will be omitted. In addition, when the image quality of the input image is unconditionally improved with respect to the shooting conditions, the process of step S3130 may be omitted after the process of step S3120, and the process may proceed to step S3140.

  In step S3120, as in the first embodiment, when the shooting condition acquisition unit 402 acquires the shooting condition group of the input image, the process proceeds to step S3130. In step S3130, the image quality improvement possibility determination unit 403 can handle the input image by the image quality improvement engine included in the image quality improvement unit 404 using the acquired shooting condition group, as in the first embodiment. Determine if there is.

  When the image quality improvement determination unit 403 determines that the image quality improvement engine cannot handle a plurality of input images, the process proceeds to step S3160. On the other hand, when the image quality improvement determination unit 403 determines that the image quality improvement engine can handle a plurality of input images, the process proceeds to step S3140. Note that, depending on the settings and implementation of the image processing apparatus 400, even if it is determined by the image quality enhancement engine that some shooting conditions cannot be dealt with, step S3140 is performed, as in the first embodiment. You may.

  In step S3140, the image quality improving unit 404 performs processing on the plurality of input images acquired in step S3110 to generate a plurality of high quality images.

  In step S3150, the wide-angle-of-view image generation unit 3005 combines some high-quality images of the high-quality image group generated in step S3140. Specifically, an OCTA En-Face image will be described as an example. Although the plurality of images do not completely overlap, adjacent images are OCTA En-Face images captured such that some areas overlap each other. Therefore, the wide-angle-of-view image generation unit 3005 detects overlapping regions from a plurality of OCTA En-Face images, and performs alignment using the overlapping regions. By deforming the OCTA En-Face image based on the alignment parameter and combining the images, it is possible to generate an OCTA En-Face image having a wider range than one OCTA En-Face image. At this time, since the plurality of OCTA En-Face images to be input have been made high in image quality in step S3140, the wide-angle OCTA En-Face images output in step S3150 have already been made high in image quality. . FIG. 32B shows an example of an OCTA En-Face image with a wide field of view, which is generated by the wide field of view image generation unit 3005. FIG. 32B is an example in which the five images shown in FIG. 32A are aligned and generated. FIG. 32 (c) shows the positional correspondence between FIG. 32 (a) and FIG. 32 (b). As shown in FIG. 32C, Im3220 to 3250 are arranged around the Im3210 as a center. It should be noted that the OCTA En-Face image can generate a plurality of OCTA En-Face images by setting different depth ranges from the three-dimensional motion contrast data. Therefore, although an example of the surface image with a wide angle of view is shown in FIG. 32, the invention is not limited to this. For example, the surface OCTA En-Face image (Im2910) shown in FIG. 29 is used for alignment, and the OCTA En-Face images in other depth ranges are deformed using the parameters obtained there. You may Alternatively, a composite color image is generated in which the input image for alignment is a color image, the RG component of the RGB components is the En-Face of OCTA on the surface layer, and the B component is the En-Face image of OCTA to be aligned. Then, the En-Face image of the composite color OCTA in which the layers in the plurality of depth ranges are composited into one sheet may be aligned. As a result, if only the B component is extracted from the color OCTA En-Face image that has been aligned, a wide-angle OCTA En-Face image with the target OCTA En-Face image aligned is obtained. You can Note that the target for high image quality is not limited to the two-dimensional OCTA En-Face image, but may be the three-dimensional OCT or the three-dimensional motion contrast data itself. In that case, the alignment may be performed using three-dimensional data to generate a wide range of three-dimensional data. By cutting out an arbitrary cross section (any plane of XYZ is possible) or an arbitrary depth range (range in the Z direction) from a wide range of three-dimensional data, it is possible to generate a wide-angle image with high image quality.

  In step S3160, the output unit 405 causes the display unit 20 to display the image combined from the plurality of images in step S3150, and outputs the image to another device. However, if it is determined in step S3130 that the input image cannot be processed, the output unit 405 outputs the input image as the output image. It should be noted that the output unit 405 causes the display unit 20 to display that the output image is the same as the input image when the examiner instructs the input image or when the input image cannot be processed. Good.

  In the present embodiment, a high-quality image is generated from each of a plurality of input images, and the high-quality images are aligned to generate a final high-quality wide-angle image of one image. The method of generating one high quality image from the input image is not limited to this. For example, in another example of the image quality improvement processing of the present embodiment shown in FIG. 31B, one wide-angle image is first generated, and the image quality improvement processing is executed on the wide-angle image. Finally, one high-quality wide-angle image may be generated.

  This process will be described with reference to FIG. 31B, but the description of the same process as in FIG. 31A will be omitted.

  In step S3121, the wide view angle image generation unit 3005 combines the plurality of images acquired in step S3110. The wide-angle image generation is the same as the description in step S3150, except that the input image is an image acquired from the image capturing device 10 or another device and is an image before being improved in image quality.

  In step S3151, the high quality image generation unit 404 performs processing on the high quality image generated by the wide angle image generation unit 3005 to generate one high quality wide angle image.

  With such a configuration, the image processing device 400 according to the present embodiment can generate a high-quality image with a wide angle of view.

  Regarding the above-described first to twenty-second embodiments, the display of the high-quality image on the display unit 20 by the output unit 405 is basically performed by the high-quality image generation unit 404 to generate the high-quality image and the analysis unit 2208 to output the analysis result. It is done automatically accordingly. However, the display of the high quality image may be performed in response to an instruction from the examiner. For example, the output unit 405 may cause the display unit 20 to display an image selected from the high-quality image generated by the high-quality image generation unit 404 and the input image, according to an instruction from the examiner. In addition, the output unit 405 may switch the display on the display unit 20 from the captured image (input image) to the high-quality image in response to an instruction from the examiner. Further, the image quality improving unit 404 executes the image quality improving process by the image quality improving engine (input of the image to the image quality improving engine) according to the instruction from the examiner, and the output unit 405 outputs the high image quality. The high-quality image generated by the conversion unit 404 may be displayed on the display unit 20. On the other hand, when the image capturing apparatus 10 captures an input image, the image quality improving engine automatically generates a high quality image based on the input image, and the output unit 405 outputs a high quality image in response to an instruction from the examiner. The quality image may be displayed on the display unit 20. It should be noted that these processes can be similarly performed for the output of the analysis result. Furthermore, the image processing apparatus may be configured to start the processing by the image capturing position estimation engine, the image quality evaluation engine, the authenticity evaluation engine, and the evaluation unit in response to an instruction from the examiner. Note that, regarding the above-described first to twenty-second embodiments, the display mode in which the output unit 405 displays the high-quality image on the display unit 20 may be arbitrary. For example, the output unit 405 may display the input image and the high-quality image side by side, or may switch and display them. Further, the output unit 405 may sequentially display the input image and the high-quality image according to the imaged site, the imaged date and time, the facility where the image was taken, and the like. Similarly, the output unit 405 may sequentially display the image analysis result and the like using the high-quality image according to an arbitrary shooting condition of the high-quality image or the input image corresponding to the high-quality image. Further, the output unit 405 may display the image analysis result using the high quality image in order for each analysis item.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. It can also be realized by the processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions. Although the present invention has been described with reference to the exemplary embodiments, the present invention is not limited to the above exemplary embodiments. The present invention includes inventions modified within a range not departing from the spirit of the present invention and inventions equivalent to the present invention. Further, the above-described respective embodiments can be appropriately combined within a range not departing from the spirit of the present invention.

Claims (36)

An acquisition unit that acquires a first image that is a motion contrast front image of the eye to be inspected;
An image quality improving unit for generating a second image having a higher image quality compared to the first image from the first image by using an image quality improving engine including a machine learning engine;
A medical image processing apparatus comprising:   The medical image processing apparatus according to claim 1, wherein the image quality improvement engine includes a machine learning engine obtained by using learning data including an image obtained by the superposition processing.   The image quality improving engine is more effective than an image obtained by OCTA imaging by an OCT imaging apparatus having a higher performance than an OCT imaging apparatus used for OCTA imaging of the first image, or an OCTA imaging step of the first image. The medical image processing apparatus according to claim 1, further comprising a machine learning engine obtained by using learning data including an image obtained by an OCTA imaging process that requires a lot of man-hours.   The image quality improving unit divides the first image into a plurality of two-dimensional images, inputs the image into the image quality improving engine, and integrates a plurality of output images from the image quality improving engine, The medical image processing apparatus according to claim 1, wherein the medical image processing apparatus generates a second image. The image quality improving engine includes a machine learning engine obtained by using learning data including pair images in which mutual positional relationships correspond to each other,
The image quality improving unit divides the first image into the plurality of two-dimensional images with an image size corresponding to the image size of the pair of images and inputs the image into the image quality improving engine. The medical image processing apparatus described.   The image quality improving engine outputs learning data including images of a plurality of partial regions set so that a part of adjacent partial regions overlap each other with respect to a region including an image and a periphery of the image. The medical image processing apparatus according to claim 4, comprising a machine learning engine obtained by using the medical image processing apparatus.   The medical image processing apparatus according to claim 1, wherein the image quality improving engine includes a machine learning engine obtained by using learning data including an image obtained by adding noise.   7. The medical image according to claim 1, wherein the image quality improving engine includes a machine learning engine obtained by using learning data including paired images obtained by adding noises having different patterns from each other. Processing equipment.   The image quality improving engine includes a machine learning engine obtained by using learning data including pair images obtained by adding noises of different patterns to images obtained by the superposition processing. 9. The medical image processing apparatus according to any one of items 8 to 8.   A plurality of the first images obtained by OCTA imaging at different positions of the eye to be inspected so that some areas of adjacent motion contrast front images overlap each other, and the first images are obtained from the plurality of first images. The medical image processing apparatus according to claim 1, further comprising: a wide-angle-of-view image generation unit that generates a wide-angle-of-view image using the plurality of second images. The image quality improving engine includes a machine learning engine obtained by using learning data including a plurality of motion contrast front images corresponding to different depth ranges of the eye to be inspected,
The said acquisition part acquires the motion contrast front image corresponding to a part of depth range among the long depth ranges containing the said different depth range as the said 1st image, The any one of Claim 1 thru | or 10. Medical image processing device. A determination unit that determines whether or not the second image can be generated using the image quality enhancement engine for the first image;
The medical device according to any one of claims 1 to 11, wherein the determination unit makes the determination based on at least one of an imaged region, an imaging method, an imaging angle of view, and an image size of the first image. Image processing device. The image quality improving unit includes a plurality of image quality improving engines that have performed learning using different learning data,
Each of the plurality of image quality enhancement engines performs learning using different learning data for at least one of the imaging region, the imaging angle of view, and the image resolution,
The image quality improving unit generates the second image by using the image quality improving engine according to at least one of a photographing region of the first image, a photographing angle of view, and a resolution of the image. The medical image processing apparatus according to any one of claims 1 to 12. The image quality improving unit includes a plurality of image quality improving engines that have performed learning using different learning data,
13. The image quality improving unit generates the second image by using the image quality improving engine according to an instruction from an examiner among the plurality of image quality improving engines. The medical image processing apparatus according to item. The image quality improving unit includes a plurality of image quality improving engines that have performed learning using different learning data,
The image quality improving unit uses the plurality of image quality improving engines to generate a plurality of the second images from the first image,
The medical image processing apparatus according to any one of claims 1 to 12, wherein the medical image processing apparatus outputs at least one image of the plurality of second images according to an instruction from an examiner. Further comprising an evaluation unit for evaluating the image quality of the second image,
The image quality improving unit includes a plurality of image quality improving engines that have performed learning using different learning data,
The image quality improving unit uses the plurality of image quality improving engines to generate a plurality of the second images from the first image,
The medical image processing apparatus according to any one of claims 1 to 12, wherein the medical image processing apparatus outputs at least one image of the plurality of second images according to an evaluation result by the evaluation unit. apparatus. The evaluation unit includes a machine learning engine in which an evaluation value by a predetermined evaluation method is used as learning data,
The medical image processing apparatus according to claim 16, wherein the medical image processing apparatus outputs the second image having the highest evaluation value by the evaluation unit among the plurality of second images. The image quality improving unit is
The image size of the first image is adjusted to an image size that can be handled by the high image quality engine, and is input to the high image quality engine.
The medical image processing according to any one of claims 1 to 17, wherein the second image is generated by adjusting an output image from the image quality improving engine to an original image size of the first image. apparatus. The image quality improving unit is
The image size of the first image is adjusted so that the resolution of the first image becomes a predetermined resolution,
The padded image is input to the image quality improvement engine so that the adjusted image size becomes an image size that can be handled by the image quality improvement engine for the first image with the image size adjusted. ,
The output image from the high quality engine is trimmed only for the padded area,
The medical image according to any one of claims 1 to 18, wherein the second image is generated by adjusting the image size of the trimmed image to the original image size of the first image. Image processing device. The image quality improving unit is
Dividing the first image into a plurality of third images of a predetermined image size,
Inputting the plurality of third images to the image quality enhancement engine to generate a plurality of fourth images,
The medical image processing apparatus according to claim 1, wherein the second image is generated by integrating the plurality of fourth images. The image quality improving unit is
Dividing the first image into a plurality of two-dimensional images, including a plurality of the image quality enhancement engines;
Generating a plurality of second images from the plurality of two-dimensional images by using the plurality of image quality enhancement engines in parallel,
The medical image processing apparatus according to claim 1, wherein the second image is generated by integrating the plurality of second images.   The image quality improving unit performs the image processing in which at least one of noise reduction and contrast enhancement is performed as compared with the first image, and at least one of image size expansion and spatial resolution improvement is performed. The medical image processing device according to claim 1, wherein the medical image processing device is generated as a second image.   23. The medical image processing apparatus according to claim 1, further comprising a display control unit that displays the second image on a display unit.   The medical image processing apparatus according to claim 23, wherein the display control unit causes the display unit to display the second image according to an instruction from an examiner.   25. The display control unit causes the display unit to display, together with the second image, a display indicating that the second image is an image generated by the image quality enhancement engine. Medical image processing device. The display control unit causes the display unit to display the second image,
The medical image processing apparatus according to claim 23, wherein the medical image processing apparatus outputs the second image in response to an input from an examiner with respect to the displayed second image. . Further comprising an analysis unit that analyzes the image of the second image,
27. The medical image processing apparatus according to claim 23, wherein the display control unit causes the display unit to display the analysis result regarding the blood vessel in the second image.   The medical image processing apparatus according to any one of claims 1 to 27, wherein the image quality improving unit generates the one second image from a plurality of the first images.   29. The medical image processing apparatus according to claim 1, wherein the first image is one motion contrast front image generated using a plurality of motion contrast front images of the eye to be inspected.   30. The medical image processing apparatus according to claim 1, wherein the image quality improving unit generates a plurality of the second images and averages the plurality of the second images.   The medical image processing apparatus according to claim 1, wherein the acquisition unit acquires the first image from an OCT imaging apparatus.   The acquisition unit acquires three-dimensional motion contrast data of the eye to be inspected from an OCT imaging apparatus, and uses data of at least a part of the three-dimensional motion contrast data in the depth direction of the eye to be inspected, The medical image processing apparatus according to claim 1, wherein the first image that is the motion contrast front image is generated.   31. The medical image processing apparatus according to claim 1, wherein the acquisition unit acquires the first image from an image management system.   The medical image processing apparatus according to claim 1, wherein the image quality enhancement engine does not use the second image as learning data. Acquiring a first image that is a motion contrast front image of the eye to be inspected;
Generating a second image having a higher image quality compared to the first image from the first image using an image quality improving engine including a machine learning engine;
A medical image processing method including: A program that, when executed by a processor, causes the processor to execute the steps of the medical image processing method according to claim 35.