JP2022145732A - Medical image processing apparatus, medical image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a medical image processing apparatus capable of generating an image more suitable for diagnostic imaging than a conventional one.

SOLUTION: The medical image processing apparatus comprises: an acquisition unit which acquires a first image that is an image of a predetermined portion of a subject; and an image quality improvement unit that uses an image quality improvement engine including a machine learning engine and generates from the first image a second image which noise-reduced or/and contrast-enhanced as compared with the first image.

SELECTED DRAWING: Figure 5

COPYRIGHT: (C)2023,JPO&INPIT

Description

The present invention relates to a medical image processing apparatus, a medical image processing method, and a program.

2. Description of the Related Art In the medical field, in order to identify a disease of a subject and to observe the degree of the disease, images are acquired by various imaging devices and image diagnosis is performed by medical professionals. Types of imaging devices include, for example, in the field of radiology, X-ray imaging devices, X-ray computed tomography (CT) devices, magnetic resonance imaging (MRI) devices, positron emission tomography (PET) devices, and single photon emission devices. There is a tomography (SPECT) device and the like. Also, for example, in the field of ophthalmology, there are fundus cameras, scanning laser ophthalmoscopes (SLO), optical coherence tomography (OCT) devices, and OCT angiography (OCTA) devices.

In order to perform diagnostic imaging accurately and complete it in a short period of time, it is important to have high image quality, such as low noise, high resolution/spatial resolution, and appropriate gradation, in the images acquired by the imaging device. becomes. Images that highlight the area or lesion of interest may also be helpful.

However, in many imaging devices, some trade-off is required to obtain images suitable for diagnostic imaging, such as high image quality. For example, there is a method of purchasing a high-performance imaging device in order to acquire a high-quality image, but in many cases, a larger investment is required than a low-performance one.

In addition, for example, in CT, there are cases where the exposure dose of the subject must be increased in order to obtain an image with less noise. Further, for example, in MRI, there are cases where a contrast medium with a risk of side effects is used to obtain an image in which a site to be observed is emphasized. In addition, for example, in OCT, if the area to be imaged is wide or high spatial resolution is required, the imaging time may be longer. Further, for example, some photographing apparatuses need to obtain images multiple times in order to obtain an image with high image quality, which takes a long time to photograph.

Patent Literature 1 discloses a technique for converting a previously acquired image into a higher-resolution image by an artificial intelligence engine in order to respond to rapid advances in medical technology and simple imaging in emergencies. . According to such a technique, for example, an image acquired by simple photography with little cost can be converted into an image with higher resolution.

JP 2018-5841 A

However, even a high-resolution image may not be suitable for diagnostic imaging. For example, even if the image has a high resolution, if there is a lot of noise or if the contrast is low, the object to be observed may not be properly grasped.

On the other hand, one of the objects of the present invention is to provide a medical image processing apparatus, a medical image processing method, and a program capable of generating images more suitable for image diagnosis than conventional ones.

A medical image processing apparatus according to an embodiment of the present invention uses an acquisition unit that acquires a first image, which is an image of a predetermined region of a subject, and an image quality enhancement engine that includes a machine learning engine to obtain the first image. and an image quality enhancer for generating from the first image a second image having at least one of noise reduction and contrast enhancement as compared to the first image.

Further, a medical image processing method according to another embodiment of the present invention acquires a first image that is an image of a predetermined part of a subject, and uses an image quality enhancement engine including a machine learning engine to perform generating from the first image a second image having at least one of noise reduction and contrast enhancement relative to the first image.

According to one aspect of the present invention, it is possible to generate an image more suitable for image diagnosis than conventionally.

An example of the configuration of a neural network for image quality enhancement processing is shown. 1 shows an example of the configuration of a neural network relating to imaging location estimation processing. An example of the configuration of a neural network for image authenticity evaluation processing is shown. 1 shows an example of a schematic configuration of an image processing apparatus according to a first embodiment; 4 is a flow chart showing an example of the flow of image processing according to the first embodiment; FIG. FIG. 7 is a flowchart showing another example of the flow of image processing according to the first embodiment; FIG. 11 is a flowchart showing an example of the flow of image processing according to the second embodiment; FIG. 12 is a diagram for explaining image processing according to the fourth embodiment; FIG. FIG. 14 is a flow diagram showing an example of the flow of image quality improvement processing according to the fourth embodiment; FIG. 12 is a diagram for explaining image processing according to the fifth embodiment; FIG. FIG. 14 is a flow diagram showing an example of the flow of image quality improvement processing according to the fifth embodiment; FIG. 12 is a diagram for explaining image processing according to the sixth embodiment; FIG. FIG. 20 is a flow diagram showing an example of the flow of image quality improvement processing according to the sixth embodiment; FIG. 12 is a diagram for explaining image processing according to the sixth embodiment; FIG. FIG. 11 shows an example of a schematic configuration of an image processing apparatus according to a seventh embodiment; FIG. FIG. 14 is a flow diagram showing an example of the flow of image processing according to the seventh embodiment; 11 shows an example of a user interface according to the seventh embodiment; FIG. 11 shows an example of a schematic configuration of an image processing apparatus according to a ninth embodiment; FIG. FIG. 22 is a flow diagram showing an example of the flow of image processing according to the ninth embodiment; 12 shows an example of a schematic configuration of an image processing apparatus according to a twelfth embodiment; FIG. 22 is a flow diagram showing an example of the flow of image quality improvement processing according to the thirteenth embodiment; FIG. 17 is a flowchart showing another example of the flow of image quality improvement processing according to the thirteenth embodiment; FIG. 11 shows an example of a schematic configuration of an image processing apparatus according to a seventeenth embodiment; FIG. FIG. 22 is a flow diagram showing an example of the flow of image processing according to the seventeenth embodiment;

Exemplary embodiments for carrying out the present invention will now be described in detail with reference to the drawings. However, the dimensions, materials, shapes, relative positions of components, etc. described in the following embodiments are arbitrary and can be changed according to the configuration of the device to which the present invention is applied or various conditions. Also, the same reference numbers are used in the drawings to indicate identical or functionally similar elements.

<Description of terms>
First, the terms used in this specification will be explained.

In the network herein, each device may be connected by wired or wireless links. Here, the lines connecting each device in the network are, for example, dedicated lines, local area network (hereinafter referred to as LAN) lines, wireless LAN lines, Internet lines, Wi-Fi (registered trademark), and Bluetooth (registered trademark). ) etc.

A medical image processing apparatus may be composed of two or more devices capable of communicating with each other, or may be composed of a single device. Also, each component of the medical image processing apparatus may be configured by a software module executed by a processor such as a CPU (Central Processing Unit) or MPU (Micro Processing Unit). Also, each component may be configured by a circuit or the like that performs a specific function, such as an ASIC. Also, it may be configured by a combination of any other hardware and any software.

Further, medical images processed by a medical image processing apparatus or a medical image processing method according to the following embodiments include images acquired using any modality (imaging device, imaging method). Medical images to be processed can include medical images acquired by any imaging device or the like and images created by a medical image processing apparatus or a medical image processing method according to the following embodiments.

Furthermore, the medical image to be processed is an image of a predetermined region of the subject, and the image of the predetermined region includes at least part of the predetermined region of the subject. Also, the medical image may include other parts of the subject. Also, the medical image may be a still image or a moving image, and may be a black-and-white image or a color image. Further, the medical image may be an image representing the structure (morphology) of a predetermined site, or an image representing its function. Images representing functions include, for example, images representing blood flow dynamics (blood flow, blood flow velocity, etc.) such as OCTA images, Doppler OCT images, fMRI images, and ultrasound Doppler images. In addition, the predetermined part of the subject may be determined according to the object to be imaged. Includes optional parts such as legs and arms.

Further, the medical image may be a tomographic image or a front image of the subject. The front image is generated using, for example, the front image of the fundus, the front image of the anterior segment, the fundus image obtained by fluorescence imaging, and the data of at least a part of the range in the depth direction of the imaging target for the data acquired by OCT. Contains En-Face images.

Also, the imaging device is a device for capturing an image used for diagnosis. The imaging device is, for example, a device that obtains an image of a predetermined portion of a subject by irradiating the predetermined portion of the subject with radiation such as light, X-rays, electromagnetic waves, or ultrasonic waves, or detects radiation emitted from the subject. It includes a device for obtaining an image of a predetermined site by means of More specifically, the imaging apparatus according to the following embodiments includes at least an X-ray imaging apparatus, a CT apparatus, an MRI apparatus, a PET apparatus, a SPECT apparatus, an SLO apparatus, an OCT apparatus, an OCTA apparatus, a fundus camera, and an endoscope. Including mirrors, etc.

The OCT apparatus may include a time domain OCT (TD-OCT) apparatus and a Fourier domain OCT (FD-OCT) apparatus. Fourier-domain OCT devices may also include spectral-domain OCT (SD-OCT) devices and wavelength-swept OCT (SS-OCT) devices. Further, the SLO device and the OCT device may include a wavefront compensation SLO (AO-SLO) device and a wavefront compensation OCT (AO-OCT) device using a wavefront compensation optical system.

An image management system is a device and system that receives and stores images captured by an image capturing device and images that have undergone image processing. In addition, the image management system can transmit images in response to requests from connected devices, perform image processing on stored images, and request other devices to perform image processing. can. The image management system can include, for example, a picture archival communication system (PACS). In particular, the image management system according to the embodiment described below includes a database capable of storing various types of information such as subject information and imaging time associated with received images. Also, the image management system is connected to a network and can send and receive images, convert images, and send and receive various information associated with saved images in response to requests from other devices. .

The shooting conditions are various information at the time of shooting the image acquired by the shooting device. The imaging conditions include, for example, information about the imaging device, information about the facility where the imaging was performed, information about the examination related to imaging, information about the photographer, and information about the subject. Imaging conditions include, for example, imaging date and time, imaging site name, imaging area, imaging angle of view, imaging method, image resolution and gradation, image size, applied image filter, information on image data format, and radiation Including information on quantity, etc. Note that the imaging region can include a peripheral region shifted from a specific imaging region, a region including a plurality of imaging regions, and the like.

The imaging conditions can be stored in the data structure that constitutes the image, stored as imaging condition data separate from the image, or stored in a database or image management system associated with the imaging device. Therefore, the imaging conditions can be acquired by a procedure corresponding to the storage means of the imaging conditions of the imaging device. Specifically, the imaging conditions are used, for example, to analyze the data structure of an image output by the imaging device, to acquire imaging condition data corresponding to the image, or to acquire the imaging conditions from a database related to the imaging device. It is obtained by accessing the interface of

Depending on the imaging device, there are imaging conditions that cannot be obtained because they are not saved. For example, if the imaging device does not have the ability to acquire or store specific imaging conditions, or if such a function is disabled. Further, for example, there are cases in which the photographing conditions are not stored because they are unrelated to the photographing device or photographing. Furthermore, for example, there are cases where the shooting conditions are hidden, encrypted, or cannot be obtained without the right. However, there are cases where even shooting conditions that have not been saved can be acquired. For example, by performing image analysis, it is possible to specify the imaging site name and imaging region.

A machine learning model is a model that has been trained (learned) in advance using appropriate teacher data (learning data) for any machine learning algorithm. The teacher data consists of one or more pairs of input data and output data. The format and combination of the input data and the output data of the paired group that constitutes the training data may be one of which is an image and the other is a numerical value, or one is composed of a plurality of image groups and the other is a character string, or both. may be suitable for a desired configuration, such as an image.

Specifically, for example, teacher data (hereinafter referred to as first teacher data) configured by a pair group of an image obtained by OCT and an imaging site label corresponding to the image can be used. Note that the imaging region label is a unique numerical value or character string representing the region. Another example of training data is a pair group of a low-quality image with a lot of noise obtained by normal OCT imaging and a high-quality image that has been imaged multiple times by OCT and processed to improve the image quality. Teacher data (hereinafter referred to as second teacher data) and the like can be mentioned.

When input data is input to a machine learning model, output data according to the design of the machine learning model is output. A machine learning model outputs output data that is likely to correspond to the input data, for example, according to trends trained using teacher data. In addition, the machine learning model can, for example, output the possibility of corresponding to the input data as a numerical value for each type of output data according to the tendency trained using the teacher data. Specifically, for example, when an image acquired by OCT is input to a machine learning model trained with the first teacher data, the machine learning model outputs the imaging part label of the imaging part photographed in the image. or output the probability for each imaging site label. In addition, for example, when a low-quality image with a lot of noise obtained by normal OCT imaging is input to the machine learning model trained with the second teacher data, the machine learning model is imaged multiple times by OCT to improve the image quality. A high-quality image equivalent to the processed image is output. From the viewpoint of maintaining quality, the machine learning model can be configured not to use its own output data as teacher data.

Machine learning algorithms also include deep learning techniques such as convolutional neural networks (CNN). In methods related to deep learning, different parameter settings for layers and nodes that make up a neural network may result in different degrees of reproducibility in output data of tendencies trained using teacher data. For example, in a deep learning machine learning model using the first teacher data, setting more appropriate parameters may increase the probability of outputting a correct imaging region label. Further, for example, in a deep learning machine learning model using the second teacher data, if more appropriate parameters are set, it may be possible to output a higher quality image.

Specifically, the parameters in the CNN are, for example, the kernel size of the filter, the number of filters, the value of the stride, the value of the dilation, and the number of nodes output by the fully connected layer, which are set for the convolution layer etc. Note that the parameter group and the number of training epochs can be set to values that are preferable for the form of use of the machine learning model based on teacher data. For example, based on teacher data, it is possible to set a parameter group and the number of epochs that can output a correct imaging site label with a higher probability or output an image with higher image quality.

One method for determining such a parameter group and the number of epochs will be exemplified. First, 70% of the pairs constituting the teacher data are randomly set for training and the remaining 30% for evaluation. Next, the training pair group is used to train the machine learning model, and at the end of each epoch of training, the evaluation pair group is used to calculate a training evaluation value. The training evaluation value is, for example, the average value of a group of values obtained by evaluating the output when inputting each pair of input data into the machine learning model under training and the output data corresponding to the input data using a loss function. be. Finally, the parameter group and epoch number when the training evaluation value is the smallest are determined as the parameter group and epoch number of the machine learning model. It should be noted that the machine learning model may overfit the pair group for training by dividing the pairs constituting the training data into those for training and those for evaluation and determining the number of epochs in this way. can be prevented.

The image quality enhancement engine is a module that outputs a high quality image obtained by enhancing the image quality of an input low quality image. Here, the term “improvement in image quality” as used herein refers to conversion of an input image into an image having a quality more suitable for image diagnosis. image. In addition, a low-quality image is, for example, a two-dimensional image or a three-dimensional image obtained by X-ray imaging, CT, MRI, OCT, PET, or SPECT, or a three-dimensional moving image of continuous CT. It was taken without any settings for high image quality. Specifically, low-quality images include, for example, low-dose imaging by X-ray imaging equipment or CT, imaging by MRI that does not use a contrast agent, images acquired by short-time imaging of OCT, etc., and low-dose imaging. Including OCTA images etc. acquired at the number of times.

Further, the content of image quality suitable for image diagnosis depends on what is desired to be diagnosed in various image diagnoses. For this reason, it cannot be said unconditionally, but for example, the image quality suitable for image diagnosis has little noise, high contrast, shows the object to be photographed in colors and gradations that are easy to observe, has a large image size, etc. Includes image quality, which may be high resolution. In addition, it is possible to include an image quality in which non-existent objects and gradations that have been drawn in the process of image generation are removed from the image.

In addition, if high-quality images with little noise or high contrast are used for image analysis such as blood vessel analysis processing of images such as OCTA, and region segmentation processing of images such as CT and OCT, low-quality images can be used. In many cases, analysis can be performed with higher accuracy than using it. Therefore, the high-quality image output by the image quality enhancement engine may be useful not only for image diagnosis but also for image analysis.

In the image processing method constituting the image quality improvement method in the following embodiments, processing using various machine learning algorithms such as deep learning is performed. In addition to processing using machine learning algorithms, this image processing method includes various image filter processing, matching processing using a database of high-quality images corresponding to similar images, and existing arbitrary image processing such as knowledge-based image processing. may be processed.

In particular, there is a configuration shown in FIG. 1 as a configuration example of a CNN that enhances the image quality of a two-dimensional image. The configuration of the CNN includes multiple convolution processing blocks 100 . The convolution processing block 100 includes a convolution layer 101, a batch normalization layer 102, and an activation layer 103 using a rectifier linear unit. The configuration of the CNN also includes a Merger layer 104 and a final convolutional layer 105 . The synthesizing layer 104 synthesizes by connecting or adding the output value group of the convolution processing block 100 and the pixel value group constituting the image. The final convolutional layer 105 outputs a group of pixel values that make up the high-quality image Im120 synthesized in the synthesis layer 104 . In such a configuration, the value group output from the pixel value group forming the input image Im110 through the convolution processing block 100 group and the pixel value group forming the input image Im110 are combined in the synthesis layer 104. synthesized. The combined pixel values are then shaped in a final convolutional layer 105 into a high quality image Im120.

Note that, for example, the number of convolution processing blocks 100 is 16, and the filter kernel size is 3 pixels wide, 3 pixels high, and the number of filters is 64 as parameters of the convolutional layer 101 group. You can get the effect of transformation. However, in practice, as described in the explanation of the machine learning model above, it is possible to set a better parameter group using teacher data according to the usage form of the machine learning model. Note that if it is necessary to process a three-dimensional image or a four-dimensional image, the kernel size of the filter may be extended to three or four dimensions.

Note that when using some image processing methods such as image processing using CNN, it is necessary to pay attention to the image size. Specifically, it should be noted that different image sizes may be required for the input low-quality image and the output high-quality image in order to deal with the problem that the peripheral part of the high-quality image is not sufficiently high-quality. should.

For the sake of clarity, it will not be specified in the embodiments described later, but if an image quality enhancement engine that requires different image sizes for an image input to the image quality enhancement engine and an image output from the image quality enhancement engine is adopted, the Assume that the size is adjusted. Specifically, for input images such as images used as teacher data for training machine learning models and images input to the image quality improvement engine, padding is performed, and the shooting area around the input image is adjusted. Combine and adjust the image size. Note that the area to be padded is filled with a constant pixel value, filled with neighboring pixel values, or mirror-padding in accordance with the characteristics of the image quality improvement method so as to effectively improve the image quality.

Also, the image quality improvement method may be implemented by only one image processing method, or may be implemented by combining two or more image processing methods. Alternatively, a plurality of image quality improvement method groups may be performed in parallel to generate a plurality of high quality image groups, and then the highest quality image may be finally selected as the high quality image. The selection of the high-quality image with the highest image quality may be performed automatically using the image quality evaluation index, or a plurality of high-quality image groups may be displayed on a user interface provided on an arbitrary display unit. , may be performed according to the instructions of the examiner (user).

In some cases, an input image whose image quality has not been enhanced is more suitable for image diagnosis, so the input image may be added to the final selection of images. Also, a parameter may be input to the high image quality engine together with the low image quality image. For example, a parameter designating the degree of image quality improvement or a parameter designating the image filter size used in the image processing technique may be input together with the input image to the image quality improvement engine.

The imaging location estimation engine is a module for estimating the imaging part and imaging area of the input image. The imaging location estimation engine determines where the imaging location or imaging region drawn in the input image is, or determines the probability of being the imaging location or imaging region for each imaging location label or imaging region label with a required level of detail. can be output.

Depending on the imaging device, the imaging site and imaging region may not be stored as imaging conditions, or may not be stored because the imaging device cannot acquire them. Further, even if the imaging parts and imaging regions are stored, there are cases where the imaging parts and imaging regions with the required level of detail are not stored. For example, only "posterior segment of the eye" is stored as an imaging region, but in detail, is it "macula", "optical papilla", or "macula and optic papilla"? Sometimes I don't know if it's "other". In another example, only "breast" is stored as the imaged region, but in detail, it is not known whether it is "right breast", "left breast", or "both". be. Therefore, by using the imaging location estimation engine, it is possible to estimate the imaging part and imaging area of the input image in these cases.

The image and data processing method that constitutes the estimation method of the shooting location estimation engine performs processing using various machine learning algorithms such as deep learning. In the image and data processing method, in addition to or instead of processing using machine learning algorithms, any existing arbitrary processing such as natural language processing, matching processing using a database of similar images and similar data, knowledge base processing, etc. An estimation process may be performed. The training data for training a machine learning model constructed using a machine learning algorithm can be images labeled with imaging sites and imaging regions. In this case, the image of the teacher data is used as input data, and the labels of the imaging parts and imaging regions are used as output data.

In particular, there is a configuration shown in FIG. 2 as a configuration example of a CNN for estimating a shooting location of a two-dimensional image. The configuration of the CNN includes a group of convolution processing blocks 200 composed of a convolution layer 201, a batch normalization layer 202, and an activation layer 203 using a normalization linear function. The configuration of the CNN also includes a final convolution layer 204 , a full connection layer 205 and an output layer 206 . A fully connected layer 205 fully connects the output values of the convolution processing block 200 . In addition, the output layer 206 uses the Softmax function to output the probability for each assumed imaging site label for the input image Im 210 as an estimation result (Result) 207 . In such a configuration, for example, if the input image Im210 is an image obtained by imaging the "macula", the highest probability is output for the imaging site label corresponding to the "macula.

Note that, for example, the number of convolution processing blocks 200 is 16, the filter kernel size is 3 pixels wide, 3 pixels high, and the number of filters is 64 as parameters of the convolutional layer 201 group. The site can be estimated. However, in practice, as described in the explanation of the machine learning model above, it is possible to set a better parameter group using teacher data according to the usage form of the machine learning model. Note that if it is necessary to process a three-dimensional image or a four-dimensional image, the kernel size of the filter may be extended to three or four dimensions. Note that the estimation technique may be implemented using only one image and data processing technique, or may be implemented using a combination of two or more image and data processing techniques.

An image quality evaluation engine is a module that outputs an image quality evaluation index for an input image. The image quality evaluation processing method for calculating the image quality evaluation index performs processing using various machine learning algorithms such as deep learning. In the image quality evaluation processing method, any existing evaluation processing such as matching processing using an image noise measurement algorithm and a database of image quality evaluation indexes corresponding to similar images and base images may be performed. Note that these evaluation processes may be performed in addition to or instead of the process using the machine learning algorithm.

For example, the image quality evaluation index can be obtained from a machine learning model constructed using a machine learning algorithm. In this case, the pair of input data constituting teacher data for training the machine learning model is an image group composed of a low-quality image group and a high-quality image group captured in advance under various shooting conditions. A pair of output data constituting teacher data for training a machine learning model is, for example, an image quality evaluation index group set for each image group of input data by an examiner who performs image diagnosis.

The authenticity evaluation engine in the description of the present invention is a module that evaluates the rendering of an input image and evaluates with a certain degree of accuracy whether or not the image was captured and acquired by the target imaging device. The authenticity evaluation processing method performs processing using various machine learning algorithms such as deep learning. In the authenticity evaluation processing method, any existing evaluation processing such as knowledge base processing may be performed in addition to or instead of the processing using the machine learning algorithm.

For example, the authenticity evaluation process can be implemented by a machine learning model constructed using a machine learning algorithm. First, the teacher data of the machine learning model will be explained. The training data includes a group of pairs of high-quality images captured in advance under various shooting conditions and a label indicating that the image was captured and acquired by the target camera (hereinafter referred to as a genuine label). In addition, the training data includes a group of high-quality images generated by inputting low-quality images into the image-quality enhancement engine (first-level image-quality enhancement engine) and a group of high-quality images that have not been captured by the target camera. A group of pairs with labels (hereinafter referred to as counterfeit labels) is included. A machine learning model trained using such teacher data outputs a counterfeit label when a high-quality image generated by a first-level image quality enhancement engine is input.

In particular, there is a configuration shown in FIG. 3 as a configuration example of a CNN that performs authenticity evaluation processing of a two-dimensional image. The configuration of the CNN includes a group of convolutional processing blocks 300 composed of a convolutional layer 301, a batch normalization layer 302, and an activation layer 303 using a normalized linear function. The configuration of the CNN also includes a final convolutional layer 304 , a fully connected layer 305 and an output layer 306 . A fully connected layer 305 fully connects the output values of the convolution processing block 300 . In addition, the output layer 306 uses the sigmoid function to convert the input image Im310 to a value of 1 (true) representing a genuine label or a value of 0 (false) representing a counterfeit label as a result of the authenticity evaluation process. (Result) 307 is output.

By setting the number of convolution processing blocks 300 to 16, the filter kernel size to be 3 pixels wide, 3 pixels high, and the number of filters to be 64 as parameters of the convolution layer 301 group, correct authentication evaluation can be performed with a certain degree of accuracy. You can get the result of processing. However, in practice, as described in the explanation of the machine learning model above, it is possible to set a better parameter group using teacher data according to the usage form of the machine learning model. Note that if it is necessary to process a three-dimensional image or a four-dimensional image, the kernel size of the filter may be extended to three or four dimensions.

When a high-quality image generated by an image quality enhancement engine (second-level image quality enhancement engine) that produces higher image quality than the first-level image quality enhancement engine is input, the authenticity evaluation engine identifies the authentic label. output. In other words, the authenticity evaluation engine cannot evaluate whether or not the input image is definitely captured and acquired by the image capturing device, but whether or not the image is likely to be captured and captured by the image capturing device. can be evaluated. Using this characteristic, by inputting the high-quality image generated by the high-quality image engine to the authenticity evaluation engine, it is evaluated whether the high-quality image generated by the high-quality image engine is sufficiently high. can.

Also, the machine learning model of the image quality enhancement engine and the machine learning model of the authenticity evaluation engine may be trained in cooperation to improve the efficiency and accuracy of both engines. In this case, first, the machine learning model of the image quality enhancement engine is trained so that the authenticity evaluation engine outputs a genuine label when the high quality image generated by the image quality enhancement engine is evaluated by the authenticity evaluation engine. In parallel, a machine learning model of the authentication evaluation engine is trained so that the authentication evaluation engine outputs a counterfeit label when the image generated by the image quality enhancement engine is evaluated by the authentication evaluation engine. Furthermore, in parallel, a machine learning model of the authenticity evaluation engine is trained so as to output a counterfeit label when the image acquired by the photographing device is evaluated by the authenticity evaluation engine. This improves the efficiency and accuracy of the image quality improvement engine and the authenticity evaluation engine.

<First embodiment>
The medical image processing apparatus according to the first embodiment will be described below with reference to FIGS. 4 and 5. FIG. FIG. 4 shows an example of a schematic configuration of an image processing apparatus according to this embodiment.

The image processing device 400 is connected to the imaging device 10 and the display unit 20 via circuits and networks. Alternatively, the imaging device 10 and the display unit 20 may be directly connected. Although these devices are separate devices in this embodiment, some or all of these devices may be integrally configured. Also, these devices may be connected to any other device via a circuit or network, or may be integrally configured with any other device.

The image processing apparatus 400 includes an acquisition unit 401, an imaging condition acquisition unit 402, an image quality enhancement determination unit 403, an image quality enhancement unit 404, and an output unit 405 (display control unit). Note that the image processing device 400 may be composed of a plurality of devices provided with some of these components. The acquisition unit 401 can acquire various data and images from the imaging device 10 and other devices, and can acquire input from the examiner via an input device (not shown). As the input device, a mouse, keyboard, touch panel, and other arbitrary input devices may be adopted. Also, the display unit 20 may be configured as a touch panel display.

The imaging condition acquisition unit 402 acquires imaging conditions for the medical image (input image) acquired by the acquisition unit 401 . Specifically, the imaging condition group stored in the data structure forming the medical image is obtained according to the data format of the medical image. If imaging conditions are not stored in the medical image, the imaging information group including the imaging condition group can be acquired from the imaging apparatus 10 or the image management system via the acquisition unit 401 .

The image quality enhancement determination unit 403 uses the imaging condition group acquired by the imaging condition acquisition unit 402 to determine whether or not the medical image can be processed by the image quality enhancement unit 404 . The image quality enhancing unit 404 enhances the image quality of a medical image that can be handled, and generates a high quality image suitable for image diagnosis. The output unit 405 causes the display unit 20 to display the high-quality image generated by the image quality enhancement unit 404, the input image, various information, and the like. Also, the output unit 405 may store the generated high-quality image or the like in a storage device connected to the image processing apparatus 400 .

Next, the image quality improving unit 404 will be described in detail. The image quality improving unit 404 is provided with an image quality improving engine. The image quality enhancement method provided by the image quality enhancement engine according to the present embodiment performs processing using a machine learning algorithm.

In this embodiment, for training a machine learning model related to a machine learning algorithm, input data that is a low-quality image having specific shooting conditions assumed to be processed, and output data that is a high-quality image corresponding to the input data. We use teacher data composed of pairs of The specific imaging conditions specifically include a predetermined imaging region, imaging method, imaging angle of view, image size, and the like.

In the present embodiment, the input data of the teacher data is a low-quality image acquired with the same model as the imaging device 10 and with the same settings as the imaging device 10 . Also, the output data of the teacher data is a high-quality image acquired by setting and image processing of the same model as the imaging device 10 . Specifically, the output data is, for example, a high-quality image (superimposed image) obtained by performing superimposition processing such as averaging on a group of images (original images) obtained by photographing multiple times. be.

A superimposed image that has undergone the superimposing process has a high-quality image that is suitable for image diagnosis because the pixels commonly drawn in the original image group are emphasized. In this case, the generated high-quality image is a high-contrast image in which the difference between the low-brightness region and the high-brightness region is clear as a result of the commonly drawn pixels being emphasized. Also, for example, in the superimposed image, random noise that occurs each time an image is captured can be reduced, and an area that was not well rendered in the original image at a certain point in time can be interpolated with another original image group.

By using each of the original image groups as input data and the corresponding superimposed image as output data, it is possible to generate a plurality of pair groups. For example, when obtaining one superimposed image from a group of 15 original images, a pair of the first original image and the superimposed image in the group of original images, the second original image in the group of original images, and A pair of superimposed images can be generated. Thus, when one superimposed image is obtained from a group of 15 original images, 15 pair groups of one image in the group of original images and the superimposed image can be generated.

Further, when it is necessary to configure the input data of the machine learning model with a plurality of images, a necessary number of original image groups can be selected from the original image group and used as input data. For example, when obtaining one superimposed image from a group of 15 original images, if two images are required as input data for the machine learning model, 105 (15C2=105) pair groups can be generated.

Note that pairs that do not contribute to high image quality can be removed from the training data in the pair group that constitutes the training data. For example, if a high-quality image, which is output data that constitutes a pair of training data, is not suitable for image diagnosis, the image output by the high-quality image engine that has learned using the training data is also suitable for image diagnosis. image quality may be poor. Therefore, by removing pairs whose output data has an image quality unsuitable for image diagnosis from the training data, it is possible to reduce the possibility that the image quality enhancement engine generates an image with an image quality unsuitable for image diagnosis.

In addition, when the average brightness and brightness distribution of the paired images are significantly different, the image quality improvement engine trained using the training data may detect an image that is not suitable for image diagnosis with a brightness distribution that is significantly different from that of the low-quality image. may be output. Therefore, pairs of input data and output data having significantly different average luminances and luminance distributions can be removed from the teacher data.

Furthermore, if the structures and positions of the objects to be captured that are drawn in the pair of images are significantly different, the image quality enhancement engine that has learned using the training data will be able to draw the images in a structure and position that are significantly different from those of the low-quality images. There is a possibility of outputting an image that is not suitable for diagnostic imaging. For this reason, pairs of input data and output data in which the structure and position of the object to be drawn are largely different can be removed from the teacher data. In addition, from the viewpoint of maintaining quality, the image quality improvement engine can be configured not to use the high quality image it outputs as training data.

By using the image quality improvement engine that performs machine learning in this way, the image quality improvement unit 404 performs superimposition processing to increase contrast and reduce noise when a medical image acquired in a single imaging process is input. It is possible to output a high-quality image that looks like it has been reduced. Therefore, the image quality enhancing unit 404 can generate a high image quality image suitable for image diagnosis based on the low image quality image that is the input image.

Next, a series of image processing according to this embodiment will be described with reference to the flowchart of FIG. FIG. 5 is a flowchart of a series of image processing according to this embodiment. First, when a series of image processing according to this embodiment is started, the process proceeds to step S510.

In step S510, the acquisition unit 401 acquires an image captured by the imaging device 10 as an input image from the imaging device 10 connected via a circuit or network. Note that the acquisition unit 401 may acquire an input image in response to a request from the imaging device 10 . For example, when the image capturing device 10 generates an image, such a request is made before or after saving the image generated by the image capturing device 10 in a recording device included in the image capturing device 10, and displaying the saved image on the display unit 20. may be issued when displaying a high quality image, when using a high quality image for an image analysis process, and so on.

Note that the obtaining unit 401 may obtain data for generating an image from the imaging device 10 and obtain an image generated based on the data by the image processing device 400 as an input image. In this case, any existing image generation method may be employed as an image generation method for the image processing apparatus 400 to generate various images.

In step S520, the imaging condition acquisition unit 402 acquires a group of imaging conditions for the input image. Specifically, according to the data format of the input image, the imaging condition group stored in the data structure forming the input image is acquired. It should be noted that, as described above, when the imaging conditions are not saved in the input image, the imaging condition acquisition unit 402 acquires the imaging information group including the imaging condition group from the imaging apparatus 10 or an image management system (not shown). be able to.

In step S530, the image quality enhancement determination unit 403 determines whether the image quality enhancement engine provided in the image quality enhancement unit 404 can enhance the image quality of the input image using the acquired group of shooting conditions. do. Specifically, the image quality enhancement determination unit 403 determines whether or not the imaging region, imaging method, imaging angle of view, and image size of the input image match conditions that can be handled by the image quality enhancement engine. .

The image quality improvement possibility determination unit 403 determines all the shooting conditions, and when it is determined that the condition can be handled, the process proceeds to step S540. On the other hand, if the image quality enhancement determination unit 403 determines that the image quality enhancement engine cannot process the input image based on these shooting conditions, the process proceeds to step S550.

Note that even if it is determined that the input image cannot be processed based on part of the imaging region, imaging method, imaging angle of view, and image size, depending on the settings and implementation of the image processing apparatus 400, Image quality enhancement processing in step S540 may be performed. For example, it is assumed that the image quality enhancement engine can comprehensively handle any imaging part of the subject, and even if the input data contains an unknown imaging part, it can be handled. Such a process may be performed if it is implemented in In addition, the image quality enhancement determination unit 403 determines, according to a desired configuration, that at least one of the imaging region of the input image, the imaging method, the imaging angle of view, and the image size is a condition that can be handled by the image quality enhancement engine. It may be determined whether or not they match.

In step S540, the image quality enhancement unit 404 uses the image quality enhancement engine to enhance the image quality of the input image to generate a high quality image that is more suitable for image diagnosis than the input image. Specifically, the image quality enhancement unit 404 inputs the input image to the image quality enhancement engine to generate a high quality image with enhanced image quality. The image quality enhancement engine generates a high quality image that looks like it has been superimposed using the input image, based on a machine learning model that has been machine-learned using the teacher data. Therefore, the image quality enhancement engine can generate a high quality image with noise reduced or contrast enhanced compared to the input image.

Note that depending on the settings and implementation of the image processing apparatus 400, the image quality enhancement unit 404 inputs parameters together with the input image to the image quality enhancement engine according to the imaging condition group, and adjusts the degree of image quality enhancement. You may Further, the image quality improving unit 404 may input a parameter corresponding to the input of the examiner to the image quality improving engine together with the input image to adjust the degree of image quality improvement.

In step S550, the output unit 405 outputs the high-quality image and causes the display unit 20 to display it if the high-quality image is generated in step S540. On the other hand, if it is determined in step S530 that the image quality enhancement process is not possible, the input image is output and displayed on the display unit 20 . Note that the output unit 405 may cause the image capturing apparatus 10 or another apparatus to display or store the output image instead of displaying the output image on the display unit 20 . The output unit 405 also processes the output image so that it can be used by the image capturing apparatus 10 or other apparatuses, or converts the output image into data so that it can be transmitted to an image management system or the like, depending on the settings and implementation of the image processing apparatus 400 . You can convert the format.

As described above, the image processing apparatus 400 according to this embodiment includes the acquisition unit 401 and the image quality enhancement unit 404 . Acquisition unit 401 acquires an input image (first image) that is an image of a predetermined region of a subject. The image quality enhancement unit 404 uses an image quality enhancement engine including a machine learning engine to generate a high quality image (second image ). The image quality improvement engine includes a machine learning engine that uses images obtained by superimposition processing as learning data.

With this configuration, the image processing apparatus 400 according to this embodiment can output a high-quality image with reduced noise or enhanced contrast from an input image. For this reason, the image processing apparatus 400 can produce images suitable for image diagnosis, such as clearer images and images in which a site to be observed or a lesion is emphasized, with less invasiveness to the operator or the subject than in the past. It can be obtained at a lower cost without increasing the cost or effort.

The image processing apparatus 400 further includes an image quality enhancement determination unit 403 that determines whether or not a high quality image can be generated from the input image using the image quality enhancement engine. The image quality improvement possibility determination unit 403 performs the determination based on at least one of the captured part of the input image, the imaging method, the imaging angle of view, and the image size.

With this configuration, the image processing apparatus 400 according to the present embodiment can omit an input image that cannot be processed by the image quality enhancement unit 404 from the image quality enhancement process, thereby reducing the processing load of the image processing apparatus 400 and the occurrence of errors. be able to.

In this embodiment, the output unit 405 (display control unit) is configured to display the generated high-quality image on the display unit 20, but the operation of the output unit 405 is not limited to this. For example, the output unit 405 can also output high-quality images to other devices connected to the imaging device 10 and the image processing device 400 . As such, high quality images can be displayed on the user interface of these devices, stored on any recording device, used for any image analysis, or sent to an image management system.

In this embodiment, the image quality enhancement determination unit 403 determines whether or not the input image can be enhanced in image quality by the image quality enhancement engine. The unit 404 performed image quality enhancement. On the other hand, if the imaging device 10 performs imaging only under imaging conditions that allow high image quality, the image acquired from the imaging device 10 may be unconditionally enhanced in image quality. In this case, as shown in FIG. 6, steps S520 and S530 can be omitted, and step S540 can be performed after step S510.

In this embodiment, the output unit 405 is configured to display a high quality image on the display unit 20 . However, the output unit 405 may display a high-quality image on the display unit 20 according to an instruction from the examiner. For example, the output unit 405 may cause the display unit 20 to display a high-quality image in response to the examiner pressing any button on the user interface of the display unit 20 . In this case, the output unit 405 may display a high-quality image by switching to the input image, or may display a high-quality image side by side with the input image.

Further, when causing the display unit 20 to display a high-quality image, the output unit 405 displays a high-quality image indicating that the displayed image is a high-quality image generated by processing using a machine learning algorithm. may be displayed together with In this case, the user can easily identify from the display that the displayed high-quality image is not the image itself acquired by photography, so that misdiagnosis can be reduced and diagnosis efficiency can be improved. can be done. Note that the display indicating that the image is a high-quality image generated by processing using a machine learning algorithm can be of any form as long as the input image can be distinguished from the high-quality image generated by the processing. Anything is fine.

In addition, the output unit 405 provides a display indicating what kind of teacher data the machine learning algorithm has used for the display indicating that the image is a high-quality image generated by processing using a machine learning algorithm. may be displayed on the display unit 20. The display may include a description of the types of input data and output data of the teacher data, and any display related to the teacher data such as imaging regions included in the input data and output data.

In the image quality enhancement engine according to the present embodiment, a superimposed image is used as the output data of the teacher data, but the teacher data is not limited to this. As the output data of the teacher data, a high-quality image obtained by performing at least one of superimposition processing, a processing group described later, and a photographing method described later, which are means for obtaining a high-quality image, may be used. .

For example, high-quality images obtained by performing maximum a posteriori probability estimation processing (MAP estimation processing) on the original image group may be used as the output data of the training data. In the MAP estimation process, a likelihood function is obtained from the probability density of each pixel value in a plurality of low-quality images, and a true signal value (pixel value) is estimated using the obtained likelihood function.

A high-quality image obtained by the MAP estimation process is a high-contrast image based on pixel values close to true signal values. In addition, since the estimated signal value is obtained based on the probability density, randomly generated noise is reduced in the high-quality image obtained by the MAP estimation process. For this reason, by using high-quality images obtained by MAP estimation processing as training data, the image-quality improvement engine converts input images into high-quality images with reduced noise and high contrast suitable for image diagnosis. A quality image can be generated. The method of generating pairs of input data and output data of teacher data may be the same as in the case of using superimposed images as teacher data.

Also, a high-quality image obtained by applying a smoothing filter process to the original image may be used as the output data of the teacher data. In this case, the image quality enhancement engine can generate a high quality image with reduced random noise from the input image. Furthermore, an image obtained by applying gradation conversion processing to the original image may be used as the output data of the teacher data. In this case, the image quality enhancement engine can generate a contrast-enhanced high quality image from the input image. The method of generating pairs of input data and output data of teacher data may be the same as in the case of using superimposed images as teacher data.

Note that the input data of the teacher data may be an image acquired from an imaging device having the same image quality tendency as that of the imaging device 10 . The output data of the teacher data may be a high-quality image obtained by expensive processing such as the iterative approximation method. It may be a high-quality image obtained by photographing with a photographing device. Furthermore, the output data may be a high quality image obtained by performing rule-based noise reduction processing. Here, the noise reduction processing can include, for example, processing such as replacing a single high-brightness pixel that appears in a low-brightness region, which is clearly noise, with an average value of nearby low-brightness pixel values. . For this reason, the image quality enhancement engine uses an image captured by a photographing device with higher performance than the photographing device used to photograph the input image, or an image obtained in a photographing process that requires more man-hours than the photographing process of the input image. It may be used as learning data.

Although omitted in the description of the present embodiment, a high-quality image generated from a plurality of images and used as output data for teacher data can be generated from a plurality of aligned images. As the alignment process, for example, one of a plurality of images is selected as a template, the degree of similarity with other images is obtained while changing the position and angle of the template, the amount of positional deviation from the template is obtained, Each image may be corrected based on the amount of misregistration. Also, any other existing alignment process may be performed.

When aligning three-dimensional images, the three-dimensional images are aligned by decomposing the three-dimensional images into a plurality of two-dimensional images and integrating the aligned images for each two-dimensional image. good too. Alternatively, the two-dimensional images may be aligned by decomposing the two-dimensional image into one-dimensional images and integrating the aligned images for each one-dimensional image. Note that these alignments may be performed not on the image but on the data for generating the image.

Further, in the present embodiment, when the image quality enhancement determination unit 403 determines that the input image can be handled by the image quality enhancement unit 404, the process proceeds to step S540, and the image quality enhancement unit 404 performs image quality enhancement. Processing started. On the other hand, the output unit 405 may cause the display unit 20 to display the determination result by the image quality enhancement determination unit 403, and the image quality enhancement unit 404 may start image quality enhancement processing in accordance with an instruction from the examiner. . At this time, the output unit 405 can cause the display unit 20 to display the input image and the imaging conditions such as the imaging region acquired for the input image together with the determination result. In this case, the examiner determines whether or not the determination result is correct, and then the image quality enhancement process is performed, so it is possible to reduce the image quality enhancement process based on the erroneous determination.

In addition, without performing determination by the image quality enhancement determination unit 403, the output unit 405 causes the display unit 20 to display the input image and the imaging conditions such as the imaging site acquired for the input image, and the image quality enhancement unit 404 outputs the The image quality improvement process may be started according to the instruction of .

<Second embodiment>
Next, an image processing apparatus according to a second embodiment will be described with reference to FIGS. 4 and 7. FIG. In the first embodiment, the image quality enhancement unit 404 has one image quality enhancement engine. In contrast, in the present embodiment, the image quality enhancing unit includes a plurality of image quality enhancing engines that have performed machine learning using different teacher data, and generates a plurality of high quality images for an input image.

Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 400 according to the first embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below, focusing on the differences from the image processing apparatus according to the first embodiment. Note that the configuration of the image processing apparatus according to this embodiment is the same as that of the image processing apparatus according to the first embodiment, so the same reference numerals are used for the configuration shown in FIG. do.

The image quality enhancement unit 404 according to the present embodiment includes two or more image quality enhancement engines for which machine learning is performed using different teacher data. Here, a method of creating a teacher data group according to this embodiment will be described. Specifically, first, a pair group of an original image as input data and a superimposed image as output data, in which various imaging regions are imaged, is prepared. Next, a teacher data group is created by grouping the pair groups for each imaging part. For example, first teacher data composed of a pair group obtained by imaging a first imaging part, and second teacher data composed of a pair group obtained by imaging a second imaging part. Create a training data group as follows.

After that, each training data is used to cause a separate image quality enhancement engine to perform machine learning. For example, a first high image quality engine corresponding to the machine learning model trained with the first teacher data, a second high image quality engine corresponding to the machine learning model trained with the second teacher data, and so on. Prepare a group of high image quality engines.

Since each image quality enhancement engine uses different training data for training the corresponding machine learning model, the degree to which the input image can be enhanced differs depending on the shooting conditions of the image input to the image quality enhancement engine. . Specifically, the first image quality improvement engine increases the degree of image quality improvement with respect to the input image acquired by imaging the first imaging region, and acquires the second imaging region by imaging. The degree of improvement in image quality is low for images that have been processed. Similarly, the second image quality improvement engine has a high degree of image quality improvement with respect to the input image acquired by imaging the second imaging region, and the image quality enhancement engine acquired by imaging the first imaging region The degree of high image quality is low for images.

Since each of the teacher data is composed of a pair group grouped according to an imaging part, the image quality tendencies of the image groups forming the pair group are similar. For this reason, the image quality improvement engine can improve the image quality more effectively than the image quality improvement engine according to the first embodiment if it is a corresponding imaging region. The imaging condition for grouping pairs of training data is not limited to the imaging site, but may be the imaging angle of view, the resolution of the image, or a combination of two or more of these. good.

A series of image processing according to this embodiment will be described below with reference to FIG. FIG. 7 is a flowchart of a series of image processing according to this embodiment. Note that the processing of steps S710 and S720 is the same as that of steps S510 and S520 according to the first embodiment, so description thereof will be omitted. Note that if the image quality of the input image is unconditionally increased, the process of step S730 may be omitted after the process of step S720, and the process may proceed to step S740.

When the shooting conditions for the input image are obtained in step S720, the process proceeds to step S730. In step S730, the image quality enhancement determination unit 403 uses the imaging condition group acquired in step S720 to determine whether any of the image quality enhancement engines included in the image quality enhancement unit 404 can handle the input image. Determine whether or not

If the image quality enhancement possibility determination unit 403 determines that none of the image quality enhancement engines can handle the input image, the process proceeds to step S760. On the other hand, if the image quality improvement possibility determination unit 403 determines that any one of the image quality improvement engine group can process the input image, the process proceeds to step S740. It should be noted that, depending on the settings and implementation of the image processing apparatus 400, as in the first embodiment, even if the image quality enhancement engine determines that some shooting conditions cannot be dealt with, step S740 is executed. You may

In step S740, the image quality enhancement unit 404 performs image quality enhancement processing from the image quality enhancement engine group based on the shooting conditions of the input image acquired in step S720 and the information of the training data of the image quality enhancement engine group. Select a rendering engine. Specifically, for example, for an imaging region in the imaging condition group acquired in step S720, there is teacher data information about the imaging region or surrounding imaging regions, and high image quality with a high degree of image quality improvement. select a conversion engine. In the above example, when the imaging region is the first imaging region, the image quality enhancement unit 404 selects the first image quality enhancement engine.

In step S750, the image quality enhancement unit 404 uses the image quality enhancement engine selected in step S740 to generate a high quality image by enhancing the image quality of the input image. After that, in step S760, the output unit 405 outputs the high-quality image and causes the display unit 20 to display the high-quality image if the high-quality image is generated in step S750. On the other hand, if it is determined in step S730 that the image quality enhancement process is impossible, the input image is output and displayed on the display unit 20. FIG. When displaying a high-quality image on the display unit 20, the output unit 405 may display that the high-quality image is generated using the image quality enhancement engine selected by the image quality enhancement unit 404. .

As described above, the image quality enhancement unit 404 according to the present embodiment includes a plurality of image quality enhancement engines that perform learning using different learning data. Here, each of the plurality of image quality enhancement engines performs learning using different learning data for at least one of the imaging region, imaging angle of view, and image resolution. The image quality enhancement unit 404 generates a high quality image using an image quality enhancement engine that corresponds to at least one of the imaging site, imaging angle of view, and image resolution of the input image.

With such a configuration, the image processing apparatus 400 according to this embodiment can generate a more effective high-quality image.

In the present embodiment, the image quality enhancement unit 404 selects an image quality enhancement engine to be used for image quality enhancement processing based on the shooting conditions of the input image, but the selection processing of the image quality enhancement engine is not limited to this. For example, the output unit 405 causes the user interface of the display unit 20 to display the imaging conditions of the acquired input image and the image quality enhancement engine group, and the image quality enhancement unit 404 performs image quality enhancement processing in accordance with instructions from the examiner. You may select the image quality improvement engine used for. The output unit 405 may cause the display unit 20 to display the information of the training data used for the learning of each image quality improvement engine together with the image quality improvement engine group. The display mode of the information of the teacher data used for learning the image quality improvement engine may be arbitrary. For example, the image quality improvement engine group may be displayed using a name related to the teacher data used for learning. good.

Further, the output unit 405 may display the image quality enhancement engine selected by the image quality enhancement unit 404 on the user interface of the display unit 20 and receive an instruction from the examiner. In this case, the image quality enhancement unit 404 may determine whether or not to finally select the image quality enhancement engine as the image quality enhancement engine to be used for image quality enhancement processing in accordance with an instruction from the examiner. .

Note that the output unit 405 may output the generated high-quality image to another device connected to the imaging device 10 or the image processing device 400, as in the first embodiment. Also, the output data of the training data from the image quality improvement engine is not limited to the high quality image that has undergone the superimposition process, as in the first embodiment. That is, at least one of a processing group and a photographing method such as superimposition processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, photographing using a high-performance photographing device, high-cost processing, and noise reduction processing. You may use the high quality image obtained by performing.

<Third Embodiment>
Next, an image processing apparatus according to a third embodiment will be described with reference to FIGS. 4 and 7. FIG. In the first and second embodiments, the imaging condition acquisition unit 402 acquires the imaging condition group from the data structure of the input image. On the other hand, in the present embodiment, the imaging condition acquisition unit uses the imaging location estimation engine to estimate the imaging part or imaging region of the input image based on the input image.

Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 400 according to the second embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below, focusing on the differences from the image processing apparatus according to the second embodiment. Since the configuration of the image processing apparatus according to this embodiment is the same as that of the image processing apparatuses according to the first and second embodiments, the same reference numerals are used for the configuration shown in FIG. omitted.

The imaging condition acquisition unit 402 according to this embodiment includes an imaging location estimation engine for estimating the imaging region or imaging region drawn in the input image acquired by the acquisition unit 401 . In the method of estimating the shooting location provided by the shooting location estimation engine according to the present embodiment, an estimation process using a machine learning algorithm is performed.

In the present embodiment, training of a machine learning model related to an imaging location estimation technique using a machine learning algorithm involves input data that is an image and output data that is an imaging region label or an imaging region label corresponding to the input data. Teacher data composed of pairs are used. Here, the input data is an image having specific shooting conditions assumed as a processing target (input image). The input data is preferably an image acquired from an image capturing apparatus having the same image quality tendency as the image capturing apparatus 10 , and more preferably the same model with the same settings as the image capturing apparatus 10 . The types of imaging region labels and imaging region labels that are output data may be imaging regions and imaging regions that are at least partially included in the input data. For example, in the case of OCT, the types of imaging site labels that are output data may be "macula", "optic papilla", "macula and optic papilla", and "others".

The imaged location estimation engine according to the present embodiment can output where the imaged part and imaged area drawn in the input image are by performing learning using such teacher data. . The imaging location estimation engine can also output the probability of being the imaging region or imaging region for each imaging region label or imaging region label of a required level of detail. By using the imaging location estimation engine, the imaging condition acquisition unit 402 can estimate the imaging region and imaging region of the input image based on the input image, and acquire them as the imaging conditions for the input image. Note that when the imaging site estimation engine outputs the probability of being the imaging site or imaging area for each imaging site label or imaging area label, the imaging condition acquisition unit 402 selects the imaging site or imaging area with the highest probability. Acquired as the shooting conditions of the input image.

Next, as in the second embodiment, a series of image processing according to this embodiment will be described with reference to the flowchart of FIG. Note that the processes of step S710 and steps S730 to S760 according to the present embodiment are the same as those processes in the second embodiment, so description thereof will be omitted. Note that if the image quality of the input image is unconditionally increased, the process of step S730 may be omitted after the process of step S720, and the process may proceed to step S740.

After the input image is acquired in step S710, the process proceeds to step S720. In step S720, the imaging condition acquisition unit 402 acquires the imaging condition group of the input image acquired in step S710.

Specifically, according to the data format of the input image, the imaging condition group stored in the data structure forming the input image is obtained. If the imaging condition group does not include information about the imaging region or the imaging region, the imaging condition acquisition unit 402 inputs the input image to the imaging region estimation engine, and determines which imaging region the input image was acquired by imaging. Guess what it is. Specifically, the imaging condition acquisition unit 402 inputs an input image to the imaging part estimation engine, evaluates the probability output for each imaging part label group, and selects an imaging part with the highest probability from the input image. Set/acquire as shooting conditions.

Note that if the input image does not store imaging conditions other than the imaging site and imaging region, the imaging condition acquisition unit 402 acquires the imaging information group including the imaging condition group from the imaging apparatus 10 or an image management system (not shown). can be obtained.

Since subsequent processing is the same as the series of image processing according to the second embodiment, description thereof is omitted.

As described above, the imaging condition acquiring unit 402 according to this embodiment functions as an estimating unit that estimates at least one of the imaging part and imaging area of the input image. The imaging condition acquisition unit 402 includes an imaging location estimation engine that uses images labeled with imaging regions and imaging regions as learning data. Estimate the shooting area.

As a result, the image processing apparatus 400 according to the present embodiment can acquire imaging conditions for the imaging part and imaging region of the input image based on the input image.

Note that in the present embodiment, the imaging condition acquisition unit 402 estimates the imaging region and imaging region of the input image using the imaging region estimation engine when the imaging condition group does not include information about the imaging region and imaging region. gone. However, the situation in which the imaging site estimation engine is used to estimate the imaging site and imaging area is not limited to this. The imaging condition acquisition unit 402 uses the imaging part estimation engine to obtain the imaging part and the imaging area even when the information about the imaging part and the imaging area included in the data structure of the input image is insufficient as information of the required level of detail. An estimation may be made for the imaging area.

In addition, regardless of whether or not the data structure of the input image contains information about the imaging region and imaging region, the imaging condition acquisition unit 402 estimates the imaging region and imaging region of the input image using the imaging region estimation engine. You may In this case, the output unit 405 causes the display unit 20 to display the estimation result output from the imaging location estimation engine and the information about the imaging region and imaging region included in the data structure of the input image, and the imaging condition acquisition unit 402 performs detection. These imaging conditions may be determined according to the instructions of the operator.

Note that the output unit 405 may output the generated high-quality image to another device connected to the imaging device 10 or the image processing device 400, as in the first embodiment. Also, the output data of the training data from the image quality improvement engine is not limited to the high quality image that has undergone the superimposition process, as in the first embodiment. That is, at least one of a processing group and a photographing method such as superimposition processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, photographing using a high-performance photographing device, high-cost processing, and noise reduction processing. You may use the high quality image obtained by performing.

<Fourth Embodiment>
Next, an image processing apparatus according to a fourth embodiment will be described with reference to FIGS. In this embodiment, the image quality enhancement unit enlarges or reduces the input image so that the input image has an image size that can be handled by the image quality enhancement engine. Also, the image quality enhancement unit reduces or enlarges the output image from the image quality enhancement engine so that the image size of the output image is the same as the image size of the input image to generate a high quality image.

Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 400 according to the first embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below, focusing on the differences from the image processing apparatus according to the first embodiment. Note that the configuration of the image processing apparatus according to this embodiment is the same as that of the image processing apparatus according to the first embodiment, so the same reference numerals are used for the configuration shown in FIG. do.

The image quality enhancement unit 404 according to this embodiment includes an image quality enhancement engine similar to the image quality enhancement engine according to the first embodiment. However, in the present embodiment, input data and a group of images obtained by enlarging or reducing an image of input data and an image of output data to have a certain image size are used as teacher data used for learning of the image quality improvement engine. It uses pairs of output data.

Now, with reference to FIG. 8, the teacher data of the image quality enhancement engine according to this embodiment will be described. As shown in FIG. 8, for example, consider a case where there are a low-quality image Im810 and a high-quality image Im820 smaller than a certain image size set for the teacher data. In this case, each of the low-quality image Im810 and the high-quality image Im820 is enlarged so as to have a constant image size set for the teacher data. Then, the enlarged low-quality image Im811 and the enlarged high-quality image Im821 are paired, and the pair is used as one of the teacher data.

Note that, as in the first embodiment, an image having specific shooting conditions assumed to be a processing target (input image) is used as the input data of the teacher data, but the specific shooting conditions are determined in advance. The imaging part, the imaging method, and the imaging angle of view. That is, unlike the first embodiment, the specific imaging conditions according to this embodiment do not include the image size.

The image quality enhancement unit 404 according to the present embodiment uses the image quality enhancement engine trained with such teacher data to enhance the image quality of the input image to generate a high quality image. At this time, the image quality enhancing unit 404 generates a deformed image by enlarging or reducing the input image so as to have a certain image size set for the teacher data, and inputs the deformed image to the image quality improving engine. Further, the image quality enhancing unit 404 reduces or enlarges the output image from the image quality enhancing engine so that it has the image size of the input image, and generates a high quality image. For this reason, the image quality enhancement unit 404 according to the present embodiment generates a high quality image by enhancing the image quality using the image quality enhancement engine even for an input image having an image size that could not be handled in the first embodiment. be able to.

Next, a series of image processing according to this embodiment will be described with reference to FIGS. FIG. 9 is a flowchart of image quality enhancement processing according to the present embodiment. Note that the processes of steps S510, S520, and S550 according to the present embodiment are the same as those processes in the first embodiment, and thus description thereof is omitted. Note that if the image quality of the input image is to be improved unconditionally with regard to shooting conditions other than the image size, the process of step S530 may be omitted after the process of step S520, and the process may proceed to step S540.

In step S520, similarly to the first embodiment, when the imaging condition acquisition unit 402 acquires the imaging condition group of the input image, the process proceeds to step S530. In step S530, the image quality enhancement determination unit 403 determines whether or not the image quality enhancement engine provided in the image quality enhancement unit 404 can handle the input image using the acquired imaging condition group. Specifically, the image quality enhancement determination unit 403 determines whether or not the imaging conditions of the input image are the imaging region, the imaging method, and the imaging angle of view that can be handled by the image quality enhancement engine. Unlike the first embodiment, the image quality enhancement determination unit 403 does not determine the image size.

The image quality enhancement determination unit 403 determines the imaging region, imaging method, and imaging angle of view, and if it is determined that the input image can be handled, the process proceeds to step S540. On the other hand, if the image quality enhancement determination unit 403 determines that the image quality enhancement engine cannot process the input image based on these shooting conditions, the process proceeds to step S550. Note that even if it is determined that the input image cannot be processed based on some of the imaging region, imaging method, and imaging angle of view, depending on the settings and implementation of the image processing apparatus 400, in step S540 Image quality improvement processing may be performed.

When the process proceeds to step S540, the image quality enhancement process according to this embodiment shown in FIG. 9 is started. In the image quality enhancement process according to this embodiment, first, in step S910, the image quality enhancement unit 404 enlarges or reduces the input image to a certain image size set for the teacher data to generate a deformed image.

Next, in step S920, the image quality enhancement unit 404 inputs the generated deformed image to the image quality enhancement engine to acquire a high quality deformed image.

Thereafter, in step S930, the image quality enhancing unit 404 reduces or enlarges the high image quality deformed image to the image size of the input image to generate a high image quality image. After the image quality improving unit 404 generates the high quality image in step S930, the image quality improving process according to the present embodiment ends, and the process proceeds to step S550. Since the process of step S550 is the same as that of step S550 of the first embodiment, the description thereof is omitted.

As described above, the image quality enhancement unit 404 according to the present embodiment adjusts the image size of the input image to an image size that can be handled by the image quality enhancement engine, and inputs the image to the image quality enhancement engine. Further, the image quality improving unit 404 generates a high quality image by adjusting the output image from the image quality improving engine to the original image size of the input image. As a result, the image processing apparatus 400 of the present embodiment uses the image quality improvement engine to improve the image quality of an input image having an image size that could not be handled in the first embodiment, thereby achieving a high image quality suitable for image diagnosis. Images can be generated.

Note that the output unit 405 may output the generated high-quality image to another device connected to the imaging device 10 or the image processing device 400, as in the first embodiment. Also, the output data of the training data from the image quality improvement engine is not limited to the high quality image that has undergone the superimposition process, as in the first embodiment. That is, at least one of a processing group and a photographing method such as superimposition processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, photographing using a high-performance photographing device, high-cost processing, and noise reduction processing. You may use the high quality image obtained by performing.

<Fifth Embodiment>
Next, an image processing apparatus according to a fifth embodiment will be described with reference to FIGS. In the present embodiment, the image quality enhancement unit generates a high quality image through image quality enhancement processing based on a fixed resolution by the image quality enhancement engine.

Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 400 according to the first embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below, focusing on the differences from the image processing apparatus according to the first embodiment. Note that the configuration of the image processing apparatus according to this embodiment is the same as that of the image processing apparatus according to the first embodiment, so the same reference numerals are used for the configuration shown in FIG. do.

The image quality enhancement unit 404 according to the present embodiment includes an image quality enhancement engine similar to that of the first embodiment. However, in this embodiment, the teacher data used for learning of the image quality enhancement engine is different from the teacher data in the first embodiment. Specifically, after enlarging or reducing the image group to an image size such that the resolution of the image group constituting the pair group of the input data and output data of the teacher data becomes a certain resolution, the image group is enlarged or reduced to a sufficiently large certain size. It is padded to the image size. Here, the resolution of an image group means, for example, the spatial resolution of an imaging device or the resolution of an imaging area.

Now, with reference to FIG. 10, teacher data for the image quality enhancement engine according to the present embodiment will be described. As shown in FIG. 10, for example, consider a case where there are a low-quality image Im1010 and a high-quality image Im1020 having a resolution lower than the fixed resolution set for the teacher data. In this case, each of the low-quality image Im1010 and the high-quality image Im1020 is enlarged so as to have a constant resolution set for the teacher data. Furthermore, each of the enlarged low-quality image Im1010 and high-quality image Im1020 is padded so as to have a constant image size set for the teacher data. Then, the low-quality image Im1011 and the high-quality image Im1021 that have been enlarged and padded are paired, and the pair is used as one of the teacher data.

Note that the constant image size set for the training data is the maximum possible image size when an image assumed to be a processing target (input image) is enlarged or reduced so as to have a constant resolution. If the fixed image size is not large enough, there is a possibility that when the image input to the image quality enhancement engine is enlarged, the image size becomes unmanageable by the machine learning model.

In addition, the area to be padded is filled with a constant pixel value, filled with neighboring pixel values, or mirror-padded in accordance with the characteristics of the machine learning model so as to effectively improve the image quality. As in the first embodiment, an image having specific imaging conditions assumed to be processed is used as the input data. This is the shooting angle of view. That is, unlike the first embodiment, the specific imaging conditions according to this embodiment do not include the image size.

The image quality enhancement unit 404 according to the present embodiment uses the image quality enhancement engine trained with such teacher data to enhance the image quality of the input image to generate a high quality image. At this time, the image quality enhancing unit 404 generates a deformed image by enlarging or reducing the input image so as to have a constant resolution set for the teacher data. Further, the image quality enhancing unit 404 generates a padding image by padding the deformed image so that it has a constant image size set for the teacher data, and inputs the padding image to the image quality enhancing engine.

Further, the image quality enhancing unit 404 trims the high image quality padded image output from the image quality enhancing engine by the padded area to generate a high image quality deformed image. Thereafter, the image quality enhancing unit 404 reduces or enlarges the generated high image quality deformed image to the image size of the input image to generate a high image quality image.

For this reason, the image quality enhancement unit 404 according to the present embodiment generates a high quality image by enhancing the image quality using the image quality enhancement engine even for an input image having an image size that could not be handled in the first embodiment. be able to.

Next, a series of image processing according to this embodiment will be described with reference to FIGS. FIG. 11 is a flowchart of image quality enhancement processing according to this embodiment. Note that the processes of steps S510, S520, and S550 according to the present embodiment are the same as those processes in the first embodiment, and thus description thereof is omitted. Note that if the image quality of the input image is to be improved unconditionally with regard to shooting conditions other than the image size, the process of step S530 may be omitted after the process of step S520, and the process may proceed to step S540.

In step S520, as in the first embodiment, after the imaging condition acquisition unit 402 acquires the imaging condition group of the input image, the process proceeds to step S530. In step S530, the image quality enhancement determination unit 403 determines whether or not the image quality enhancement engine provided in the image quality enhancement unit 404 can handle the input image using the acquired imaging condition group. Specifically, the image quality enhancement determination unit 403 determines whether or not the imaging conditions of the input image are the imaging region, the imaging method, and the imaging angle of view that can be handled by the image quality enhancement engine. Unlike the first embodiment, the image quality enhancement determination unit 403 does not determine the image size.

The image quality enhancement determination unit 403 determines the imaging region, imaging method, and imaging angle of view, and if it is determined that the input image can be handled, the process proceeds to step S540. On the other hand, if the image quality enhancement determination unit 403 determines that the image quality enhancement engine cannot process the input image based on these shooting conditions, the process proceeds to step S550. Note that even if it is determined that the input image cannot be processed based on some of the imaging region, imaging method, and imaging angle of view, depending on the settings and implementation of the image processing apparatus 400, in step S540 Image quality improvement processing may be performed.

When the process moves to step S540, the image quality enhancement process according to this embodiment shown in FIG. 11 is started. In the image quality enhancement process according to the present embodiment, first, in step S1110, the image quality enhancement unit 404 enlarges or reduces the input image to a certain resolution set for the teacher data, and generates a deformed image. .

Next, in step S1120, the image quality enhancing unit 404 generates a padding image by padding the generated deformed image so as to have the image size set for the teacher data. At this time, the image quality enhancement unit 404 fills the area to be padded with a constant pixel value, with neighboring pixel values, or with a mirror pixel value in accordance with the characteristics of the machine learning model so as to effectively improve the image quality. padding.

In step S1130, the image quality enhancement unit 404 inputs the padding image to the image quality enhancement engine to obtain a high image quality padding image.

Next, in step S1140, the image quality enhancing unit 404 trims the high image quality padded image by the area padded in step S1120 to generate a high image quality deformed image.

After that, in step S1150, the image quality enhancing unit 404 reduces or enlarges the high quality deformed image to the image size of the input image to generate a high quality image. After the image quality improving unit 404 generates the high quality image in step S1130, the image quality improving process according to the present embodiment ends, and the process proceeds to step S550. Since the process of step S550 is the same as that of step S550 of the first embodiment, the description thereof is omitted.

As described above, the image quality enhancing unit 404 according to the present embodiment adjusts the image size of the input image so that the resolution of the input image becomes a predetermined resolution. Further, the image quality enhancement unit 404 generates a padded image by padding the input image whose image size has been adjusted so that the adjusted image size becomes an image size that can be handled by the image quality enhancement engine, Input the padding image to the image quality enhancement engine. After that, the image quality enhancement unit 404 trims the output image from the image quality enhancement engine by the padding area. Then, the image quality enhancing unit 404 generates a high quality image by adjusting the image size of the trimmed image to the original image size of the input image.

As a result, the image quality enhancing unit 404 of the present embodiment can generate a high quality image by enhancing the image quality using the image quality enhancing engine even for an input image having an image size that could not be handled in the first embodiment. can be done. In addition, by using an image quality improvement engine that has learned from training data based on resolution, input images can be obtained more efficiently than the image quality improvement engine according to the fourth embodiment, which simply processes images of the same image size. may be able to improve the image quality.

Note that the output unit 405 may output the generated high-quality image to another device connected to the imaging device 10 or the image processing device 400, as in the first embodiment. Also, the output data of the training data from the image quality improvement engine is not limited to the high quality image that has undergone the superimposition process, as in the first embodiment. That is, at least one of a processing group and a photographing method such as superimposition processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, photographing using a high-performance photographing device, high-cost processing, and noise reduction processing. You may use the high quality image obtained by performing.

<Sixth embodiment>
Next, an image processing apparatus according to a sixth embodiment will be described with reference to FIGS. In the present embodiment, the image quality enhancing unit generates a high quality image by enhancing the image quality of the input image for each area of a certain image size.

Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 400 according to the first embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below, focusing on the differences from the image processing apparatus according to the first embodiment. Note that the configuration of the image processing apparatus according to this embodiment is the same as that of the image processing apparatus according to the first embodiment, so the same reference numerals are used for the configuration shown in FIG. do.

The image quality enhancement unit 404 according to the present embodiment includes an image quality enhancement engine similar to that of the first embodiment. However, in this embodiment, the teacher data used for learning of the image quality enhancement engine is different from the teacher data in the first embodiment. Specifically, a group of pairs of low-quality image input data and high-quality image output data, which constitutes training data, is set to a fixed image size corresponding to the positional relationship between the low-quality image and the high-quality image. It consists of a rectangular area image of

Now, with reference to FIG. 12, the teacher data of the image quality enhancement engine according to this embodiment will be described. As shown in FIG. 12, let us consider a case where, for example, one pair group forming the teacher data includes an original image Im1210, which is a low-quality image, and a superimposed image Im1220, which is a high-quality image. In this case, in the first embodiment, the input data of the teacher data is Im1210, and the output data is Im1220.

In contrast, in this embodiment, the rectangular area image R1211 of the original image Im1210 is used as input data, and the rectangular area image R1221, which is the same imaging area as the rectangular area image R1211 in the superimposed image Im1220, is used as output data. Then, a rectangular area image R1211 as input data and a rectangular area image R1221 as output data form a teacher data pair (hereinafter referred to as a first rectangular area image pair). Here, the rectangular area image R1211 and the rectangular area image R1221 are images of a fixed image size. Note that the original image Im1210 and the superimposed image Im1220 may be aligned by any method. Further, the corresponding positional relationship between the rectangular area image R1211 and the rectangular area image R1221 may be identified by any method such as template matching. Note that the image size and the number of dimensions of the input data and the output data may be different depending on the design of the image quality enhancement engine. For example, when the processing target is an OCT image, and the input data is part of a B-scan image (two-dimensional image), the output data may be part of an A-scan image (one-dimensional image). .

A fixed image size for the rectangular area images R1211 and R1221 can be determined from a common divisor of the number of pixels in each dimension corresponding to a group of image sizes of images assumed to be processed (input images), for example. In this case, it is possible to prevent the positional relationship of the rectangular area image groups output by the image quality enhancement engine from overlapping. Specifically, for example, the image assumed to be processed is a two-dimensional image, the first image size in the image size group is 500 pixels wide and 500 pixels high, and the second image size is 500 pixels wide and 500 pixels high. Consider the case of 100 pixels and 100 pixels high. Here, a certain image size is selected for the rectangular area images R1211 and R1221 from the common divisor of each side. In this case, for example, a fixed image size is selected from 100 pixels wide and 100 pixels high, 50 pixels wide and 50 pixels high, and 25 pixels wide and 25 pixels high.

When the image assumed to be processed is three-dimensional, the number of pixels is determined with respect to width, height, and depth. A plurality of rectangular areas can be set for one pair of a low-quality image corresponding to input data and a high-quality image corresponding to output data. Therefore, for example, the rectangular area image R1212 in the original image Im1210 is used as input data, and the rectangular area image R1222, which is the same imaging area as the rectangular area image R1212 in the superimposed image Im1220, is used as output data. The rectangular area image R1212 as input data and the rectangular area image R1222 as output data form a pair of teacher data. Thereby, a rectangular area image pair different from the first rectangular area image pair can be created.

By creating a large number of pairs of rectangular area images while changing the image of the rectangular area to an image with different coordinates, the pair group that constitutes the teacher data can be enriched, and learning was performed using the teacher pair. Efficient image quality improvement can be expected with the image quality improvement engine. However, pairs that do not contribute to improving the image quality of the machine learning model can be prevented from being added to the training data. For example, if the image quality of a rectangular area image created from high-quality images that constitute a pair of output data is not suitable for diagnosis, an image quality improvement engine that has learned using such teacher data will output The image may also be of unsuitable quality for diagnostic imaging. Therefore, pairs containing such high-quality images can be removed from the training data.

Also, for example, even if the pair of rectangular area images created from a low-quality image and a rectangular area image created from a high-quality image differ greatly in average luminance or luminance distribution, such a pair can be removed from the training data. can. If learning is performed using such teacher data, there is a possibility that the image quality enhancement engine will output an image that is not suitable for image diagnosis and has a brightness distribution that is significantly different from that of the input image.

Further, for example, consider a case where the structures and positions of the imaging targets drawn in the pair of rectangular area images created from the low-quality image and the rectangular area image created from the high-quality image are significantly different. In this case, there is a possibility that the image quality improvement engine that has learned using such training data will output an image that is not suitable for image diagnosis, in which the imaging target is drawn in a structure or position that is significantly different from the input image. be. Therefore, such pairs can be removed from the training data.

As in the first embodiment, an image having specific imaging conditions assumed to be processed is used as the input data of the teacher data. The photographing method and the photographing angle of view. That is, unlike the first embodiment, the specific imaging conditions according to this embodiment do not include the image size.

The image quality enhancement unit 404 according to the present embodiment uses the image quality enhancement engine trained with such teacher data to enhance the image quality of the input image to generate a high quality image. At this time, the image quality enhancing unit 404 divides the input image into a group of rectangular area images which are continuous without gaps and have a constant image size set for the teacher data. The image quality enhancing unit 404 enhances the image quality of each divided rectangular area image group by an image quality enhancing engine to generate a high quality rectangular area image group. Thereafter, the image quality enhancing unit 404 arranges and combines the generated high image quality rectangular area images according to the positional relationship of the input image to generate a high image quality image.

As described above, the image quality enhancing unit 404 of the present embodiment enhances the image quality of the input image in units of rectangular areas, and combines the images having the enhanced image quality. A high-quality image can be generated by increasing the image quality of an image of the same size.

Next, a series of image processing according to this embodiment will be described with reference to FIGS. FIG. 13 is a flowchart of image quality enhancement processing according to this embodiment. Note that the processes of steps S510, S520, and S550 according to the present embodiment are the same as those processes in the first embodiment, and thus description thereof is omitted. Note that if the image quality of the input image is to be improved unconditionally with regard to shooting conditions other than the image size, the process of step S530 may be omitted after the process of step S520, and the process may proceed to step S540.

In step S520, as in the first embodiment, after the imaging condition acquisition unit 402 acquires the imaging condition group of the input image, the process proceeds to step S530. In step S530, the image quality enhancement determination unit 403 determines whether or not the image quality enhancement engine provided in the image quality enhancement unit 404 can handle the input image using the acquired imaging condition group. Specifically, the image quality enhancement determination unit 403 determines whether or not the imaging conditions of the input image are the imaging region, the imaging method, and the imaging angle of view that can be handled by the image quality enhancement engine. Unlike the first embodiment, the image quality enhancement determination unit 403 does not determine the image size.

The image quality enhancement determination unit 403 determines the imaging region, imaging method, and imaging angle of view, and if it is determined that the input image can be handled, the process proceeds to step S540. On the other hand, if the image quality enhancement determination unit 403 determines that the image quality enhancement engine cannot process the input image based on these shooting conditions, the process proceeds to step S550. Note that even if it is determined that the input image cannot be processed based on some of the imaging region, imaging method, and imaging angle of view, depending on the settings and implementation of the image processing apparatus 400, in step S540 Image quality improvement processing may be performed.

When the process proceeds to step S540, the image quality enhancement process according to this embodiment shown in FIG. 13 is started. In the image quality enhancement process according to the present embodiment, first, in step S1310, as shown in FIG. 14, the input image is divided into a group of rectangular area images of a constant image size set for the teacher data, which are continuous without gaps. . Here, FIG. 14 shows an example of dividing the input image Im1410 into a group of rectangular area images R1411 to R1426 each having a fixed image size. As described above, depending on the design of the image quality enhancement engine, the image size and the number of dimensions of the input image and the output image of the image quality enhancement engine may be different. In this case, the division positions of the input image can be overlapped or separated to adjust so that the combined high-quality image generated in step S1320 has no defect.

Next, in step S1320, the image quality enhancing unit 404 enhances the image quality of each of the rectangular area images R1411 to R1426 by the image quality enhancing engine to generate a group of high image quality rectangular area images.

Then, in step S1330, the image quality enhancing unit 404 arranges and joins each of the generated high-quality rectangular area images R1411 to R1426 in the same positional relationship as each of the rectangular area images R1411 to R1426 obtained by dividing the input image. and produce high-quality images.

As described above, the image quality enhancing unit 404 according to the present embodiment divides an input image into a plurality of rectangular area images (third images) R1411 to R1426 each having a predetermined image size. After that, the image quality enhancement unit 404 inputs the plurality of divided rectangular area images R1411 to R1426 to the image quality enhancement engine to generate a plurality of fourth images, and integrates the plurality of fourth images, Produce high quality images. Note that if the positional relationship between the rectangular area groups overlaps at the time of integration, the pixel value groups of the rectangular area groups can be integrated or overwritten.

As a result, the image quality enhancing unit 404 of the present embodiment can generate a high quality image by enhancing the image quality using the image quality enhancing engine even for an input image having an image size that could not be handled in the first embodiment. can be done. Also, if the teacher data is created from a plurality of images obtained by dividing the low-quality image and the high-quality image into predetermined image sizes, a large amount of teacher data can be created from a small number of images. Therefore, in this case, the number of low-quality images and high-quality images for creating teacher data can be reduced.

Note that the output unit 405 may output the generated high-quality image to another device connected to the imaging device 10 or the image processing device 400, as in the first embodiment. Also, the output data of the training data from the image quality improvement engine is not limited to the high quality image that has undergone the superimposition process, as in the first embodiment. That is, at least one of a processing group and a photographing method such as superimposition processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, photographing using a high-performance photographing device, high-cost processing, and noise reduction processing. You may use the high quality image obtained by performing.

<Seventh embodiment>
Next, an image processing apparatus according to a seventh embodiment will be described with reference to FIGS. 15 to 17. FIG. In this embodiment, the image quality evaluation unit selects the highest image quality image among the plurality of high image quality images output from the plurality of image quality enhancement engines according to the examiner's instruction.

Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 400 according to the first embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below, focusing on the differences from the image processing apparatus according to the first embodiment.

FIG. 15 shows a schematic configuration of an image processing apparatus 1500 according to this embodiment. The image processing apparatus 1500 according to the present embodiment includes an image quality evaluation unit 1506 in addition to the acquisition unit 401, the imaging condition acquisition unit 402, the image quality enhancement determination unit 403, the image quality enhancement unit 404, and the output unit 405. It is Note that the image processing apparatus 1500 may be composed of a plurality of apparatuses provided with some of these components. Here, the acquisition unit 401, the imaging condition acquisition unit 402, the image quality enhancement determination unit 403, the image quality enhancement unit 404, and the output unit 405 are the same as those of the image processing apparatus according to the first embodiment. , are denoted by the same reference numerals as those shown in FIG. 4, and the description thereof is omitted.

Further, the image processing device 1500 may be connected to the imaging device 10, the display unit 20, and other devices (not shown) via arbitrary circuits and networks, similarly to the image processing device 400 according to the first embodiment. . Also, these devices may be connected to any other device via a circuit or network, or may be integrally configured with any other device. Although these devices are separate devices in this embodiment, some or all of these devices may be integrally configured.

The image quality enhancement unit 404 according to the present embodiment includes two or more image quality enhancement engines for which machine learning is performed using different teacher data. Here, a method of creating a teacher data group according to this embodiment will be described. Specifically, first, a group of pairs of input data, which is a low-quality image, and output data, which is a high-quality image, shot under various shooting conditions is prepared. Next, a teacher data group is created by grouping the pair groups according to an arbitrary combination of imaging conditions. For example, first teacher data composed of a pair group acquired by a combination of first imaging conditions, second teacher data composed of a pair group acquired by a combination of second imaging conditions, and so on. , is created as a training data group.

After that, each training data is used to cause a separate image quality enhancement engine to perform machine learning. For example, a first high image quality engine corresponding to the machine learning model trained with the first teacher data, a second high image quality engine corresponding to the machine learning model trained with the second teacher data, and so on. Prepare a group of high image quality engines.

Since each image quality enhancement engine uses different training data for training the corresponding machine learning model, the degree to which the input image can be enhanced differs depending on the shooting conditions of the image input to the image quality enhancement engine. . Specifically, the first image quality improvement engine increases the degree of image quality improvement with respect to the input image acquired by photographing under the first combination of photographing conditions, and increases the degree of image quality improvement with the second combination of photographing conditions. The degree of improvement in image quality is low for an image obtained by photographing. Similarly, the second image quality improvement engine has a high degree of image quality improvement with respect to the input image acquired by shooting under the second shooting conditions, and the image quality enhancement engine acquired by shooting under the first shooting conditions. The degree of high image quality is low for images.

Each of the teacher data is composed of a pair group grouped according to a combination of shooting conditions, so that the image quality tendencies of the image groups forming the pair group are similar. For this reason, the image quality enhancement engine can improve image quality more effectively than the image quality enhancement engine according to the first embodiment, provided that the corresponding shooting conditions are combined. Note that the combination of imaging conditions for grouping pairs of teacher data may be arbitrary, and may be, for example, a combination of two or more of the imaging region, imaging angle of view, and image resolution. Also, grouping of teacher data may be performed based on one imaging condition, as in the second embodiment.

The image quality evaluation unit 1506 selects a high quality image having the highest image quality from among the plurality of high quality images generated by the image quality enhancement unit 404 using a plurality of image quality enhancement engines, according to an instruction from the examiner.

The output unit 405 can display the high-quality image selected by the image quality evaluation unit 1506 on the display unit 20 or output it to another device. Note that the output unit 405 can display a plurality of high-quality images generated by the image quality enhancement unit 404 on the display unit 20, and the image quality evaluation unit 1506 performs A high-quality image with the highest image quality can be selected by

Accordingly, the image processing apparatus 1500 can output the highest quality image according to the examiner's instruction, among the plurality of high quality images generated using the plurality of image quality enhancement engines.

A series of image processing according to this embodiment will be described below with reference to FIGS. FIG. 16 is a flowchart of a series of image processing according to this embodiment. Note that the processing of steps S1610 and S1620 according to the present embodiment is the same as the processing of steps S510 and S520 according to the first embodiment, and thus description thereof is omitted. Note that if the image quality of the input image is to be improved unconditionally with respect to the shooting conditions, the process of step S1630 may be omitted after the process of step S1620, and the process may proceed to step S1640.

In step S1620, similarly to the first embodiment, after the imaging condition acquisition unit 402 acquires the imaging condition group of the input image, the process proceeds to step S1630. In step S<b>1630 , the image quality enhancement determination unit 403 uses the acquired imaging condition group to determine whether any of the image quality enhancement engines provided in the image quality enhancement unit 404 selects an input image as in the second embodiment. Determine whether or not the problem can be dealt with.

If the image quality enhancement possibility determination unit 403 determines that none of the image quality enhancement engines can handle the input image, the process proceeds to step S1660. On the other hand, if the image quality improvement possibility determination unit 403 determines that any one of the image quality improvement engine group can handle the input image, the process proceeds to step S1640. It should be noted that, depending on the settings and implementation of the image processing apparatus 1500, as in the first embodiment, even if the image quality enhancement engine determines that some shooting conditions cannot be dealt with, step S1640 is executed. You may

In step S1640, the image quality enhancement unit 404 inputs the input image acquired in step S1610 to each of the image quality enhancement engines to generate a high quality image group.

In step S1650, the image quality evaluation unit 1506 selects the highest image quality image from the high image quality image group generated in step S1640. Specifically, first, the output unit 405 causes the user interface of the display unit 20 to display the high-quality image group generated in step S1640.

Here, FIG. 17 shows an example of the interface. The interface displays an input image Im1710 and high-quality images Im1720, Im1730, Im1740, and Im1750 respectively output by the image quality improvement engine group. The examiner operates an arbitrary input device (not shown) to designate the image of the highest image quality, that is, the image most suitable for image diagnosis, out of the group of images (high-quality images Im1720 to Im1750). Since an input image that has not been enhanced in image quality by the image enhancement engine may be more suitable for image diagnosis, the input image may be added to the image group to be instructed by the examiner.

After that, the image quality evaluation unit 1506 selects the high quality image designated by the examiner as the highest quality image.

In step S1660, the output unit 405 displays the image selected in step S1650 on the display unit 20 or outputs it to another device. However, if it is determined in step S1630 that the input image cannot be processed, the output unit 405 outputs the input image as the output image. When the examiner instructs the input image or when the input image cannot be processed, the output unit 405 causes the display unit 20 to display that the output image is the same as the input image. good too.

As described above, the image quality enhancement unit 404 according to the present embodiment uses a plurality of image quality enhancement engines to generate a plurality of high quality images from the input image, and the output unit 405 of the image processing apparatus 1500 outputs the detected image. At least one of a plurality of high-quality images is output according to an instruction from the user. In particular, in this embodiment, the output unit 405 outputs the highest quality image according to the examiner's instruction. Accordingly, the image processing apparatus 1500 can output a high quality image according to the examiner's instruction, among a plurality of high quality images generated using a plurality of image quality enhancement engines.

Note that the output unit 405 may output the generated high-quality image to another device connected to the imaging device 10 or the image processing device 1500, as in the first embodiment. Also, the output data of the training data from the image quality improvement engine is not limited to the high quality image that has undergone the superimposition process, as in the first embodiment. That is, at least one of a processing group and a photographing method such as superimposition processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, photographing using a high-performance photographing device, high-cost processing, and noise reduction processing. You may use the high quality image obtained by performing.

<Eighth embodiment>
Next, an image processing apparatus according to an eighth embodiment will be described with reference to FIGS. 15 and 16. FIG. In this embodiment, the image quality evaluation unit uses the image quality evaluation engine to select the image with the highest image quality from among the plurality of high quality images output from the plurality of image quality enhancement engines.

Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 1500 according to the seventh embodiment. Therefore, in the following description, the image processing apparatus according to this embodiment will be described with a focus on the differences from the image processing apparatus according to the seventh embodiment. Note that the configuration of the image processing apparatus according to this embodiment is the same as the configuration of the image processing apparatus according to the seventh embodiment, so the same reference numerals are used for the configuration shown in FIG. 15, and description thereof is omitted. do.

The image quality evaluation unit 1506 according to this embodiment includes an image quality evaluation engine that evaluates the image quality of the input image. The image quality evaluation engine outputs an image quality evaluation index for the input image. The image quality evaluation processing method for calculating the image quality evaluation index in the image quality evaluation engine according to this embodiment uses a machine learning model constructed using a machine learning algorithm. A pair of input data constituting teacher data for training a machine learning model is an image group composed of a low-quality image group and a high-quality image group captured in advance under various shooting conditions. A pair of output data constituting teacher data for training a machine learning model is, for example, an image quality evaluation index group set for each image group of input data by an examiner who performs image diagnosis.

Next, a series of image processing according to this embodiment will be described with reference to FIG. Note that the processes of steps S1610, S1620, S1630, and S1660 according to the present embodiment are the same as those processes in the seventh embodiment, so descriptions thereof will be omitted. Note that if the image quality of the input image is to be improved unconditionally with respect to the shooting conditions, the process of step S1630 may be omitted after the process of step S1620, and the process may proceed to step S1640.

In step S1630, as in the seventh embodiment, if the image quality improvement possibility determination unit 403 determines that any one of the image quality improvement engine group can handle the input image, the process proceeds to step S1640. Transition. It should be noted that, depending on the settings and implementation of the image processing apparatus 1500, as in the first embodiment, even if the image quality enhancement engine determines that some shooting conditions cannot be dealt with, step S1640 is executed. You may

In step S1640, the image quality enhancement unit 404 inputs the input image acquired in step S1610 to each of the image quality enhancement engines to generate a high quality image group.

In step S1650, the image quality evaluation unit 1506 selects the highest image quality image from the high image quality image group generated in step S1640. Specifically, first, the image quality evaluation unit 1506 inputs the high-quality image group generated in step S1640 to the image quality evaluation engine. The image quality evaluation engine calculates an image quality evaluation index based on learning for each input high quality image. The image quality evaluation unit 1506 selects the high-quality image for which the highest image quality evaluation index is calculated among the calculated image quality evaluation indexes. Note that an input image that has not been enhanced in image quality by the image quality enhancement engine may be more suitable for image diagnosis. A rating index may also be added to the selection. Since step S1660 is the same as step S1660 in the seventh embodiment, the description is omitted.

As described above, the image processing apparatus 1500 according to this embodiment further includes the image quality evaluation unit 1506 that evaluates the image quality of high-quality images. The image quality enhancement unit 404 uses a plurality of image quality enhancement engines to generate a plurality of high quality images from the input image. At least one of the multiple high-quality images is output. In particular, the image quality evaluation unit 1506 according to this embodiment includes an image quality evaluation engine that uses evaluation values obtained by a predetermined evaluation method as learning data. The image quality evaluation unit 1506 selects, from among the plurality of high quality images, the high quality image with the highest evaluation result using the image quality evaluation engine by the image quality evaluation unit 1506 . The output unit 405 outputs the high quality image with the highest evaluation value selected by the image quality evaluation unit 1506 .

As a result, the image processing apparatus 1500 according to this embodiment can easily output a high-quality image most suitable for image diagnosis from a plurality of high-quality images based on the output of the image quality evaluation engine.

In this embodiment, the image quality evaluation unit 1506 selects a high-quality image with the highest image quality evaluation index among the image quality evaluation indexes output by the image quality evaluation engine, and the output unit 405 displays the selected high-quality image on the display unit. 20 was displayed. However, the configuration of image quality evaluation section 1506 is not limited to this. For example, the image quality evaluation unit 1506 selects high-quality images having several high image quality evaluation indexes among the image quality evaluation indexes output by the image quality evaluation engine, and the output unit 405 displays the selected high-quality images on the display unit 20. You may let Further, the output unit 405 causes the display unit 20 to display the image quality evaluation index output by the image quality evaluation engine together with the corresponding high quality image, and the image quality evaluation unit 1506 selects the highest quality image according to the examiner's instruction. You may choose.

Note that the output unit 405 may output the generated high-quality image to another device connected to the imaging device 10 or the image processing device 1500, as in the first embodiment. Also, the output data of the training data from the image quality improvement engine is not limited to the high quality image that has undergone the superimposition process, as in the first embodiment. That is, at least one of a processing group and a photographing method such as superimposition processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, photographing using a high-performance photographing device, high-cost processing, and noise reduction processing. You may use the high quality image obtained by performing.

<Ninth Embodiment>
Next, an image processing apparatus according to the ninth embodiment will be described with reference to FIGS. 18 and 19. FIG. In this embodiment, the authenticity evaluation unit uses the authentication evaluation engine to evaluate whether the high quality image generated by the image quality enhancement unit 404 is sufficiently high in quality.

Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 400 according to the first embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below, focusing on the differences from the image processing apparatus according to the first embodiment.

FIG. 18 shows a schematic configuration of an image processing apparatus 1800 according to this embodiment. The image processing apparatus 1800 according to the present embodiment includes an acquisition unit 401, an imaging condition acquisition unit 402, an image quality enhancement determination unit 403, an image quality enhancement unit 404, and an output unit 405, as well as an authenticity evaluation unit 1807. It is Note that the image processing apparatus 1800 may be composed of a plurality of apparatuses provided with some of these components. Here, the acquisition unit 401, the imaging condition acquisition unit 402, the image quality enhancement determination unit 403, the image quality enhancement unit 404, and the output unit 405 are the same as those of the image processing apparatus according to the first embodiment. , are denoted by the same reference numerals as those shown in FIG. 4, and the description thereof is omitted.

Further, the image processing device 1800 may be connected to the imaging device 10, the display unit 20, and other devices (not shown) via arbitrary circuits and networks, similarly to the image processing device 400 according to the first embodiment. . Also, these devices may be connected to any other device via a circuit or network, or may be integrally configured with any other device. Although these devices are separate devices in this embodiment, some or all of these devices may be integrally configured.

The authenticity evaluation unit 1807 is equipped with an authentication evaluation engine. The authenticity evaluation unit 1807 uses an authenticity evaluation engine to evaluate whether the high image quality image generated by the image quality improvement engine is sufficiently high in quality. The authenticity evaluation processing method in the authenticity evaluation engine according to this embodiment uses a machine learning model constructed using a machine learning algorithm.

The training data used to train the machine learning model consists of a group of high-quality images taken in advance under various shooting conditions and a pair of labels indicating that they were taken and acquired by the target camera (hereinafter referred to as the genuine label). groups are included. In addition, the training data includes a group of high-quality images generated by inputting low-quality images into a high-quality image enhancement engine with poor image quality accuracy, and labels indicating that they were not acquired by the target camera. , counterfeit labels).

The authenticity evaluation engine, which has been trained using such training data, cannot evaluate whether or not an input image has been captured and acquired with certainty by the imaging device. It is possible to evaluate whether or not the image has the resemblance to the captured and acquired image. Using this characteristic, the authenticity evaluation unit 1807 inputs the high-quality image generated by the image-quality improving unit 404 to the authenticity evaluation engine, so that the high-quality image generated by the image-quality improving unit 404 has sufficiently high image quality. It is possible to evaluate whether or not the

When the authenticity evaluation unit 1807 determines that the high quality image generated by the image quality enhancement unit 404 is sufficiently high in quality, the output unit 405 causes the display unit 20 to display the high quality image. On the other hand, the output unit 405 causes the display unit 20 to display the input image when the authenticity evaluation unit 1807 determines that the high image quality image generated by the image quality enhancement unit 404 is not sufficiently enhanced. When displaying the input image, the output unit 405 notifies that the high-quality image generated by the image-quality improving unit 404 is not sufficiently high-quality and that the displayed image is the input image. It can be displayed on the display unit 20 .

A series of image processing according to this embodiment will be described below with reference to FIG. FIG. 19 is a flowchart of a series of image processing according to this embodiment. Note that the processing of steps S1910 to S1940 according to the present embodiment is the same as the processing of steps S510 to S540 in the first embodiment, so description thereof will be omitted. Note that if the image quality of the input image is to be improved unconditionally with respect to the shooting conditions, the process of step S1930 may be omitted after the process of step S1920, and the process may proceed to step S1940.

In step S1940, after the image quality enhancement unit 404 generates the high quality image group, the process proceeds to step S1950. In step S1950, the authentication evaluation unit 1807 inputs the high-quality image generated in step S1940 to the authentication evaluation engine, and performs authentication based on the output of the authentication evaluation engine. Specifically, the authenticity evaluation unit 1807 evaluates that the generated high-quality image is sufficiently high-quality when the authenticity evaluation engine outputs the label (true). On the other hand, when the authenticity evaluation engine outputs a counterfeit label (false), the authenticity evaluation unit 1807 evaluates that the generated high-quality image is not sufficiently high in quality.

In step S1960, when the output unit 405 determines that the high-quality image generated by the image-quality improving unit 404 is sufficiently high-quality by the authenticity evaluating unit 1807, the high-quality image is displayed on the display unit 20. Let On the other hand, the output unit 405 causes the display unit 20 to display the input image when the authenticity evaluation unit 1807 determines that the high image quality image generated by the image quality enhancement unit 404 is not sufficiently enhanced.

As described above, the image processing apparatus 1800 according to this embodiment further includes the authentication evaluation unit 1807 that evaluates the image quality of the high-quality image, and the authentication evaluation unit 1807 includes an authentication evaluation engine that evaluates the authenticity of the image. The authenticity evaluation engine includes a machine learning engine that uses, as learning data, an image generated by an image quality enhancement engine whose image quality enhancement processing accuracy is lower (worse) than that of the image quality enhancement engine of the image quality enhancement unit 404 . The output unit 405 of the image processing device 1800 outputs a high quality image when the output from the authenticity evaluation engine of the authenticity evaluation unit is true.

Accordingly, with the image processing apparatus 1800 according to the present embodiment, the examiner can efficiently check a high-quality image with sufficiently high image quality.

Also, the machine learning model of the image quality enhancement engine and the machine learning model of the authenticity evaluation engine may be trained in cooperation to improve the efficiency and accuracy of both engines.

In this embodiment, the image quality enhancement unit 404 generates one high-quality image, and the authentication evaluation unit 1807 evaluates the generated high-quality image. is not limited to this. For example, as in the second embodiment, when the image quality enhancement unit 404 generates a plurality of high quality images using a plurality of image quality enhancement engines, the authenticity evaluation unit 1807 generates a plurality of high quality images. At least one of the images may be evaluated. In this case, for example, the authenticity evaluation unit 1807 may evaluate all of the plurality of generated high-quality images, or may evaluate only the image designated by the examiner among the plurality of high-quality images. good.

Furthermore, the output unit 405 causes the display unit 20 to display the determination result of whether or not the high-quality image is sufficiently enhanced by the authenticity evaluation unit 1807, and outputs the high-quality image in accordance with the examiner's instruction. You may

Note that the output unit 405 may output the generated high-quality image to another device connected to the imaging device 10 or the image processing device 1800, as in the first embodiment. Also, the output data of the training data from the image quality improvement engine is not limited to the high quality image that has undergone the superimposition process, as in the first embodiment. That is, at least one of a processing group and a photographing method such as superimposition processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, photographing using a high-performance photographing device, high-cost processing, and noise reduction processing. You may use the high quality image obtained by performing.

<Tenth Embodiment>
Next, an image processing apparatus according to a tenth embodiment will be described with reference to FIGS. 4 and 5. FIG. In this embodiment, the image quality enhancement unit divides a three-dimensional input image into a plurality of two-dimensional images, inputs them to the image quality enhancement engine, and combines the output images from the image quality enhancement engine to create a three-dimensional image. Generate quality images.

Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 400 according to the first embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below, focusing on the differences from the image processing apparatus according to the first embodiment. Note that the configuration of the image processing apparatus according to this embodiment is the same as that of the image processing apparatus according to the first embodiment, so the same reference numerals are used for the configuration shown in FIG. do.

The acquisition unit 401 according to this embodiment acquires a three-dimensional image composed of a group of structurally continuous two-dimensional images. Specifically, the three-dimensional image is, for example, a three-dimensional OCT volume image composed of a group of OCT B-scan images (tomographic images). Also, for example, it is a three-dimensional CT volume image composed of a group of axial tomographic images.

The image quality enhancement unit 404 includes an image quality enhancement engine, as in the first embodiment. A group of pairs of input data and output data, which is training data for the image quality enhancement engine, is composed of a group of two-dimensional images. The image quality enhancement unit 404 divides the acquired three-dimensional image into a plurality of two-dimensional images, and inputs each two-dimensional image to the image quality enhancement engine. Thereby, the image quality improving unit 404 can generate a plurality of two-dimensional high quality images.

The output unit 405 combines a plurality of two-dimensional high-quality images generated for each two-dimensional image of the three-dimensional image by the image quality enhancement unit 404, and outputs a three-dimensional high-quality image.

Next, a series of image processing according to this embodiment will be described with reference to FIG. Note that the processes of steps S510 to S530 and step S550 according to the present embodiment are the same as those processes in the first embodiment, so description thereof will be omitted. However, in step S510, the acquisition unit 401 acquires a three-dimensional image. Note that if the image quality of the input image is to be improved unconditionally with respect to the shooting conditions, the process of step S530 may be omitted after the process of step S520, and the process may proceed to step S540.

In step S530, if the image quality enhancement determination unit 403 determines that the input image can be processed by the image quality enhancement engine, the process proceeds to step S540. Note that the image quality improvement possibility determination unit 403 may perform the determination based on the imaging conditions of the three-dimensional image, or may perform the determination based on the imaging conditions of a plurality of two-dimensional images forming the three-dimensional image. may In step S540, the image quality enhancing unit 404 divides the acquired three-dimensional image into a plurality of two-dimensional images. The image quality enhancement unit 404 inputs each of the divided two-dimensional images to the image quality enhancement engine to generate a plurality of two-dimensional high-quality images. The image quality enhancing unit 404 combines a plurality of generated two-dimensional high-quality images based on the acquired three-dimensional image to generate a three-dimensional high-quality image.

In step S550, the output unit 405 causes the display unit 20 to display the generated three-dimensional high-quality image. Note that the display mode of the three-dimensional high-quality image may be arbitrary.

As described above, the image quality enhancement unit 404 according to the present embodiment divides a three-dimensional input image into a plurality of two-dimensional images and inputs them to the image quality enhancement engine. The image quality enhancement unit 404 combines a plurality of two-dimensional high-quality images output from the image quality enhancement engine to generate a three-dimensional high-quality image.

As a result, the image quality enhancing unit 404 according to the present embodiment can enhance the image quality of the 3D image using the image quality enhancing engine trained using the teacher data of the 2D image.

Note that the output unit 405 may output the generated high-quality image to another device connected to the imaging device 10 or the image processing device 400, as in the first embodiment. Also, the output data of the training data from the image quality improvement engine is not limited to the high quality image that has undergone the superimposition process, as in the first embodiment. That is, at least one of a processing group and a photographing method such as superimposition processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, photographing using a high-performance photographing device, high-cost processing, and noise reduction processing. You may use the high quality image obtained by performing.

<Eleventh Embodiment>
Next, an image processing apparatus according to an eleventh embodiment will be described with reference to FIGS. 4 and 5. FIG. In this embodiment, the image quality enhancement unit divides a three-dimensional input image into a plurality of two-dimensional images, and enhances the image quality of the plurality of two-dimensional images in parallel by a plurality of image quality enhancement engines. A three-dimensional high-quality image is generated by combining the output images.

Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 400 according to the tenth embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below, focusing on the differences from the image processing apparatus according to the tenth embodiment. Since the configuration of the image processing apparatus according to this embodiment is the same as that of the image processing apparatuses according to the first and tenth embodiments, the same reference numerals are used for the configuration shown in FIG. omitted.

The image quality enhancement unit 404 according to this embodiment includes a plurality of image quality enhancement engines similar to those of the tenth embodiment. Note that the plurality of image quality enhancement engine groups provided in the image quality enhancement unit 404 may be implemented so as to be able to perform distributed processing in two or more device groups via a circuit or network, or may be implemented in a single image quality enhancement engine group. device.

The image quality enhancing unit 404 divides the acquired three-dimensional image into a plurality of two-dimensional images, as in the tenth embodiment. The image quality enhancement unit 404 uses a plurality of image quality enhancement engines to share (in parallel) image quality enhancement of a plurality of two-dimensional images, thereby generating a plurality of two-dimensional high-quality images. The image quality enhancing unit 404 combines a plurality of two-dimensional high-quality images output from a plurality of image-quality enhancing engines based on a three-dimensional image to be processed to generate a three-dimensional high-quality image.

Next, a series of image processing according to this embodiment will be described with reference to FIG. Note that the processes of steps S510 to S530 and step S550 according to the present embodiment are the same as those processes in the tenth embodiment, so description thereof will be omitted. Note that if the image quality of the input image is to be improved unconditionally with respect to the shooting conditions, the process of step S530 may be omitted after the process of step S520, and the process may proceed to step S540.

In step S530, if the image quality enhancement determination unit 403 determines that the input image can be processed by the image quality enhancement engine, the process proceeds to step S540. Note that the image quality improvement possibility determination unit 403 may perform the determination based on the imaging conditions of the three-dimensional image, or may perform the determination based on the imaging conditions of a plurality of two-dimensional images forming the three-dimensional image. may

In step S540, the image quality enhancing unit 404 divides the acquired three-dimensional image into a plurality of two-dimensional images. The image quality enhancement unit 404 inputs each of the plurality of divided two-dimensional images to a plurality of image quality enhancement engines, performs image quality enhancement processing in parallel, and generates a plurality of two-dimensional high-quality images. The image quality enhancing unit 404 combines a plurality of generated two-dimensional high-quality images based on the acquired three-dimensional image to generate a three-dimensional high-quality image.

In step S550, the output unit 405 causes the display unit 20 to display the generated three-dimensional high-quality image. Note that the display mode of the three-dimensional high-quality image may be arbitrary.

As described above, the image quality enhancement unit 404 according to this embodiment includes a plurality of image quality enhancement engines. The image quality enhancement unit 404 divides a three-dimensional input image into a plurality of two-dimensional images, and uses a plurality of image quality enhancement engines in parallel to generate a plurality of two-dimensional high-quality images. The image quality enhancing unit 404 integrates a plurality of two-dimensional high-quality images to generate a three-dimensional high-quality image.

As a result, the image quality enhancing unit 404 according to the present embodiment can enhance the image quality of the 3D image using the image quality enhancing engine trained using the teacher data of the 2D image. Also, compared to the tenth embodiment, it is possible to improve the image quality of the three-dimensional image more efficiently.

Note that the teacher data for the plurality of image quality enhancement engines may be different teacher data according to the processing target to be processed by each image quality enhancement engine. For example, the first image quality improvement engine may learn using teacher data for the first imaging area, and the second image quality improvement engine may learn using teacher data for the second imaging area. In this case, each image quality improvement engine can improve the image quality of the two-dimensional image with higher accuracy.

Further, the output unit 405 may output the generated high-quality image to another device connected to the imaging device 10 or the image processing device 400, as in the first embodiment. Also, the output data of the training data from the image quality improvement engine is not limited to the high quality image that has undergone the superimposition process, as in the first embodiment. That is, at least one of a processing group and a photographing method such as superimposition processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, photographing using a high-performance photographing device, high-cost processing, and noise reduction processing. You may use the high quality image obtained by performing.

<Twelfth Embodiment>
Next, an image processing apparatus according to a twelfth embodiment will be described with reference to FIGS. 5 and 20. FIG. In this embodiment, the acquisition unit 401 acquires the input image from the image management system 2000 instead of the imaging device.

Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 400 according to the first embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below, focusing on the differences from the image processing apparatus according to the first embodiment. Note that the configuration of the image processing apparatus according to the present embodiment is the same as the configuration of the image processing apparatus 400 according to the first embodiment, so the same reference numerals are used for the configuration shown in FIG. 4 and the description thereof is omitted.

FIG. 20 shows a schematic configuration of an image processing apparatus 400 according to this embodiment. The image processing apparatus 400 according to this embodiment is connected to the image management system 2000 and the display unit 20 via arbitrary circuits and networks. The image management system 2000 is a device and system for receiving and storing images captured by an arbitrary image capturing device and images subjected to image processing. In addition, the image management system 2000 can transmit images in response to requests from connected devices, perform image processing on stored images, and request other devices to perform image processing. can be done. The image management system can include, for example, a picture archival communication system (PACS).

The acquisition unit 401 according to this embodiment can acquire an input image from the image management system 2000 connected to the image processing apparatus 400 . Also, the output unit 405 can output the high quality image generated by the image quality enhancement unit 404 to the image management system 2000 .

Next, a series of image processing according to this embodiment will be described with reference to FIG. Note that the processing of steps S520 to S540 according to the present embodiment is the same as those processing in the first embodiment, so description thereof will be omitted. Note that if the image quality of the input image is to be improved unconditionally with respect to the shooting conditions, the process of step S530 may be omitted after the process of step S520, and the process may proceed to step S540.

In step S510, the acquisition unit 401 acquires an image stored in the image management system 2000 as an input image from the image management system 2000 connected via a circuit or network. Note that the acquisition unit 401 may acquire an input image in response to a request from the image management system 2000 . Such a request may be issued, for example, when the image management system 2000 saves an image, displays the saved image on the display unit 20 before transmitting the saved image to another device. Further, the request is issued when, for example, the user operates the image management system 2000 to request high image quality processing, or when a high image quality image is used in the image analysis function of the image management system 2000. may be issued.

The processing from step S520 to step S540 is the same as the processing in the first embodiment. After the image quality improving unit 404 generates the high quality image in step S540, the process proceeds to step S550. In step S550, the output unit 405 outputs the high-quality image to the image management system 2000 as an output image if a high-quality image has been generated in step S540. If a high-quality image has not been generated in step S540, the input image is output to the image management system 2000 as an output image. Note that the output unit 405 may process the output image so that it can be used by the image management system 2000 or convert the data format of the output image, depending on the settings and implementation of the image processing apparatus 400 .

As described above, the acquisition unit 401 according to this embodiment acquires an input image from the image management system 2000 . For this reason, the image processing apparatus 400 of the present embodiment, based on the images stored in the image management system 2000, generates high-quality images suitable for image diagnosis, increases invasiveness for the photographer and the subject, It can be output without increasing effort. Also, the output high-quality image can be saved in the image management system 2000 or displayed on a user interface provided in the image management system 2000 . Also, the output high-quality image can be used for the image analysis function provided in the image management system 2000, or can be transmitted to another device connected to the image management system 2000 via the image management system 2000. .

Note that the image processing device 400, the image management system 2000, and the display unit 20 may be connected to other devices (not shown) via circuits or networks. Moreover, although these devices are separate devices in this embodiment, some or all of these devices may be integrally configured.

Also, the output unit 405 may output the generated high-quality image to the image management system 2000 or another device connected to the image processing apparatus 400, as in the first embodiment.

<Thirteenth Embodiment>
Next, an image processing apparatus according to a thirteenth embodiment will be described with reference to FIGS. 4, 5, 21A, and 21B. In this embodiment, the image quality enhancement unit uses a plurality of images as input images and generates one high quality image.

Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 400 according to the first embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below, focusing on the differences from the image processing apparatus according to the first embodiment. Note that the configuration of the image processing apparatus according to this embodiment is the same as that of the image processing apparatus according to the first embodiment, so the same reference numerals are used for the configuration shown in FIG. do.

The acquisition unit 401 according to the present embodiment acquires a plurality of images as input data to be processed from the imaging device 10 or another device.

The image quality enhancement unit 404 according to the present embodiment includes an image quality enhancement engine similar to that of the first embodiment. Also, the teacher data may be the same as in the first embodiment. The image quality enhancement unit 404 inputs each of the plurality of images acquired by the acquisition unit 401 to the image quality enhancement engine, superimposes the plurality of output high quality images, and produces a final high quality image. Generate. Note that the image quality enhancing unit 404 may position the plurality of high quality images by any method before superimposing the plurality of high quality images.

The output unit 405 causes the display unit 20 to display the final high-quality image generated by the image quality enhancement unit 404 . Note that the output unit 405 may cause the display unit 20 to display a plurality of input images together with the final high-quality image. Further, the output unit 405 may display a plurality of generated high-quality images on the display unit 20 together with the final high-quality image and the input image.

Next, a series of image processing according to this embodiment will be described with reference to FIGS. 5 and 21A. FIG. 21A is a flowchart of image quality enhancement processing according to this embodiment. Note that the processing of steps S510 to S530 according to the present embodiment is the same as those processing in the first embodiment, so description thereof will be omitted.

However, in step S510, the acquisition unit 401 acquires a plurality of images, and in steps S520 and S530, the shooting conditions are acquired for each of the plurality of images, and it is determined whether or not the image quality enhancement engine can handle the conditions. be done. Note that if the image quality of the input image is to be improved unconditionally with respect to the shooting conditions, the process of step S530 may be omitted after the process of step S520, and the process may proceed to step S540. Further, when it is determined that some of the plurality of images cannot be dealt with by the image quality enhancement engine, the image can be excluded from subsequent processing.

In step S530, if the image quality enhancement determination unit 403 determines that the plurality of input images can be handled by the image quality enhancement engine, the process proceeds to step S540. When the process moves to step S540, the image quality enhancement process according to this embodiment shown in FIG. 21A is started. In the image quality enhancement process according to this embodiment, first, in step S2110, the image quality enhancement unit 404 inputs each of a plurality of input images to the image quality enhancement engine to generate a high quality image group.

Next, in step S2120, the image quality enhancing unit 404 superimposes the generated high quality image groups to generate a final high quality image. Note that the superposition processing may be performed by averaging processing such as addition averaging or any other existing processing. Also, when superimposing, the image quality enhancing unit 404 may align a plurality of high quality images by any method and then superimpose them. After the image quality enhancing unit 404 has generated the final high quality image, the process proceeds to step S550.

In step S550, the final high-quality image generated by the output unit 405 is displayed on the display unit 20. FIG.

As described above, the image quality enhancement unit 404 according to this embodiment generates one final high quality image from a plurality of input images. Since the image quality improvement by the image quality improvement engine is based on the input image, for example, if a lesion is not properly displayed in an input image, a high-quality image obtained by improving the image quality of the input image will have a low pixel value. turn into. On the other hand, other input images obtained by photographing the same location may appropriately display lesions and the like, and high-quality images obtained by enhancing the image quality of the other input images may have high pixel values. Therefore, by superimposing these high-quality images, it becomes possible to appropriately display the portions with low or high pixel values, and it is possible to generate a high-contrast high-quality image. By setting the number of input images to be smaller than the number of images required for conventional superimposition, it is possible to further reduce costs such as longer photographing time as in the conventional art.

It should be noted that this effect becomes remarkable when using an input image using motion contrast data such as an OCTA image, for example. Here, the motion contrast data is data used in OCTA or the like, obtained by repeatedly photographing the same part of an object to be photographed and detecting a temporal change in the object to be photographed between the photographings. In OCTA, an OCTA image is generated by generating a front image using data of a desired range in the depth direction of the imaging target among the calculated motion contrast data.

Motion contrast data is obtained by detecting temporal changes in the object to be photographed over time intervals in which the same part of the object to be photographed is repeatedly photographed. Sometimes. On the other hand, if another time-interval photographing is performed, it may be possible to detect the movement of the photographed object as a larger movement. Therefore, by superimposing the motion contrast images obtained by increasing the quality of the motion contrast images in each case, it is possible to interpolate the motion contrast that was not generated at a specific timing or was only slightly detected. Therefore, according to such processing, it is possible to generate a motion contrast image in which contrast enhancement is performed for more movements of the object to be photographed, and the examiner can grasp the state of the object to be photographed more accurately. can.

Therefore, when an image such as an OCTA image that depicts a portion that changes with time is used as an input image, by superimposing high-quality images acquired at different times, the predetermined portion of the subject can be more clearly identified. It can be imaged in detail.

In this embodiment, high-quality images are generated from a plurality of input images, and the high-quality images are superimposed to generate a final high-quality image. The method for generating a high-quality image of is not limited to this. For example, in another example of the image quality improvement process of this embodiment shown in FIG. Generate a single superimposed input image.

After that, in step S2140, the image quality enhancement unit 404 inputs the one superimposed input image to the image quality enhancement engine to generate one high quality image. Even with this kind of image quality enhancement processing, like the image quality enhancement processing described above, portions of a plurality of input images with low or high pixel values can be displayed appropriately, resulting in a high-contrast image. High quality images can be generated. This processing can also produce a remarkable effect when a motion contrast image such as the OCTA image is used as an input image.

When the image quality processing is performed, a superimposed image of the same number of input images as the plurality of input images to be processed is used as the input data for the training data of the image quality enhancement engine. As a result, the image quality enhancement engine can perform appropriate image quality enhancement processing.

Further, in the image quality improvement processing according to the present embodiment and the other image quality improvement processing described above, the processing of combining a high quality image group or an input image group is not limited to superimposition. For example, one image may be generated by applying MAP estimation processing to these image groups. Alternatively, a single image may be generated by synthesizing a group of high-quality images or a group of input images.

When synthesizing a group of high-quality images or a group of input images to generate a single image, for example, an image having a wide gradation in a high luminance area and an image having a wide gradation in a low luminance area are used as an input image. may be used. In this case, for example, an image obtained by improving the image quality of an image having wide gradation in a high luminance area and an image obtained by improving the image quality of an image having a wide gradation in a low luminance area are synthesized. This makes it possible to generate an image that can express a wider range of brightness (dynamic range). In this case, the input data of the training data for the image quality enhancement engine is an image with wide gradation in the high luminance area or a low image quality image with wide gradation in the low luminance area, which is to be processed. be able to. Also, the output data of the teacher data from the image quality enhancement engine can be a high quality image corresponding to the input data.

Alternatively, an image having a wide gradation in a high luminance area and an image having a wide gradation in a low luminance area may be synthesized, and the image quality of the synthesized image may be improved by an image quality improvement engine. In this case also, an image that can express a wider range of brightness can be generated. In this case, the input data for the training data of the image quality enhancement engine is a low image quality image having wide gradation in the high luminance area and a low image quality image having wide gradation in the low luminance area. can be a composite image. Also, the output data of the teacher data from the image quality enhancement engine can be a high quality image corresponding to the input data.

In these cases, an image quality enhancement engine can be used to improve the image quality of an image that can express a wider range of brightness, making it possible to process images with a smaller number of images than before. , can provide images suitable for image analysis at a lower cost.

Any method, such as shortening or lengthening the exposure time of the photographing device, can be used as a method of photographing an image having a wide gradation in the high luminance region and an image having a wide gradation in the low luminance region. You can Moreover, the method of dividing the width of the gradation is not limited to the low-luminance region and the high-luminance region, and may be arbitrary.

Further, in the image quality enhancement processing according to the present embodiment, a plurality of image quality enhancement engines may be used to process a plurality of input images in parallel. Note that the output unit 405 may output the generated high-quality image to another device connected to the imaging device 10 or the image processing device 400, as in the first embodiment. Also, the output data of the training data from the image quality improvement engine is not limited to the high quality image that has undergone the superimposition process, as in the first embodiment. That is, at least one of a processing group and a photographing method such as superimposition processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, photographing using a high-performance photographing device, high-cost processing, and noise reduction processing. You may use the high quality image obtained by performing.

<Fourteenth Embodiment>
Next, an image processing apparatus according to a fourteenth embodiment will be described with reference to FIGS. 4 and 5. FIG. In this embodiment, the image quality enhancing unit uses, as an input image, a medium image quality image generated from a plurality of low image quality images, and generates a high image quality image.

Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 400 according to the first embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below, focusing on the differences from the image processing apparatus according to the first embodiment. Note that the configuration of the image processing apparatus according to this embodiment is the same as that of the image processing apparatus according to the first embodiment, so the same reference numerals are used for the configuration shown in FIG. do.

The acquisition unit 401 according to the present embodiment acquires a medium image quality image obtained by superimposing a plurality of low image quality images as input data to be processed from the imaging device 10 or another device. Arbitrary alignment processing may be performed when superimposing the low-quality images.

The image quality enhancement unit 404 according to the present embodiment includes an image quality enhancement engine similar to that of the first embodiment. However, the image quality enhancement engine of the present embodiment is designed to input a medium image quality image of medium image quality and to output a high image quality image. A medium-quality image is a superimposed image generated by superimposing a plurality of low-quality image groups. Also, the high quality image is an image of higher quality than the medium quality image. Also, for the pair groups that make up the training data used for training the image quality enhancement engine, the input data that makes up each pair is a medium quality image generated in the same way as the medium quality image, and the output data is a high quality image. It is a quality image.

The output unit 405 causes the display unit 20 to display the high quality image generated by the image quality enhancement unit 404 . Note that the output unit 405 may display the input image on the display unit 20 together with the high-quality image. may be displayed on the display unit 20.

Next, a series of image processing according to this embodiment will be described with reference to FIG. Note that the processing of steps S520 to S550 according to the present embodiment is the same as those processing in the first embodiment, so description thereof will be omitted.

In step S510, the acquisition unit 401 acquires a medium quality image as an input image from the imaging device 10 or another device. Note that the acquiring unit 401 may acquire the medium quality image generated by the imaging device 10 as the input image in response to a request from the imaging device 10 . For example, when the image capturing device 10 generates an image, such a request is made before or after saving the image generated by the image capturing device 10 in a recording device included in the image capturing device 10, and displaying the saved image on the display unit 20. may be issued when displaying a high quality image, when using a high quality image for an image analysis process, and so on.

Since the subsequent processing is the same as the processing in the first embodiment, the description is omitted.

As described above, the acquiring unit 401 according to the present embodiment acquires, as an input image, a medium-quality image that is an image generated using a plurality of images of a predetermined region of the subject. In this case, since the input image becomes a clearer image, the image quality enhancement engine can generate a high quality image with higher accuracy. Note that the number of low-quality images used to generate a medium-quality image may be less than the number of images used to generate a conventional superimposed image.

Note that the medium-quality image is not limited to an image obtained by superimposing a plurality of low-quality images. There may be. When synthesizing a plurality of low-quality images, images having different gradations may be synthesized.

Further, the output unit 405 may output the generated high-quality image to another device connected to the imaging device 10 or the image processing device 400, as in the first embodiment. Also, the output data of the training data from the image quality improvement engine is not limited to the high quality image that has undergone the superimposition process, as in the first embodiment. That is, at least one of a processing group and a photographing method such as superimposition processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, photographing using a high-performance photographing device, high-cost processing, and noise reduction processing. You may use the high quality image obtained by performing.

<Fifteenth Embodiment>
Next, an image processing apparatus according to a fifteenth embodiment will be described with reference to FIGS. 4 and 5. FIG. In the present embodiment, the image quality improving unit performs the image quality improvement according to the first embodiment and the like, and also increases the image size of the input image (increase in size).

Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 400 according to the first embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below, focusing on the differences from the image processing apparatus according to the first embodiment. Note that the configuration of the image processing apparatus according to this embodiment is the same as that of the image processing apparatus according to the first embodiment, so the same reference numerals are used for the configuration shown in FIG. do.

The acquisition unit 401 according to the present embodiment acquires a low image size image (low size image) as an input image. Note that the low-size image is an image having a smaller number of pixels than a high-image-size image (high-size image) output by an image quality enhancement engine, which will be described later. Specifically, for example, when the image size of the high-size image is 1024 pixels wide, 1024 pixels high, and 1024 pixels deep, the image size of the low-size image is 512 pixels, 512 pixels high, and 512 pixels deep. and so on. In relation to this, in this specification, increasing the image size refers to processing for increasing the number of pixels per image and enlarging the image size.

The image quality enhancement unit 404 according to the present embodiment includes an image quality enhancement engine, as in the first embodiment. However, the image quality enhancement engine of this embodiment is configured to reduce noise and enhance the contrast of the input image and increase the image size of the input image. Therefore, the image quality enhancement engine of this embodiment is configured to receive a low-size image and output a high-size image.

In relation to this, for a group of pairs forming teacher data for the image quality enhancement engine, the input data forming each pair is a low-size image, and the output data is a large-size image. Note that the high-size image used as output data can be acquired from a device with higher performance than the imaging device that acquired the low-size image, or by changing the settings of the imaging device. Further, when a group of high-size images already exists, the group of low-size images to be used as input data is acquired by reducing the group of high-size images to the image size of the image assumed to be acquired from the imaging device 10. may As for the high-size image, a superimposed low-size image is used as in the first embodiment.

Note that the enlargement of the image size of the input image by the image quality enhancement unit 404 according to the present embodiment is performed by acquiring training data from a device having higher performance than the imaging device 10 or by changing the settings of the imaging device 10 . It is different from simple image enlargement because it is obtained by Specifically, the processing for enlarging the image size of the input image by the image quality enhancing unit 404 according to the present embodiment can reduce degradation in resolution compared to simply enlarging the image.

With such a configuration, the image quality enhancing unit 404 according to the present embodiment can generate a high image quality image in which noise reduction and contrast enhancement are performed on the input image and the image size is increased.

Next, a series of image processing according to this embodiment will be described with reference to FIG. Note that the processes of steps S520, S530, and S550 according to the present embodiment are the same as those processes in the first embodiment, and therefore description thereof is omitted. Note that if the image quality of the input image is to be improved unconditionally with respect to the shooting conditions, the process of step S530 may be omitted after the process of step S520, and the process may proceed to step S540.

In step S510, the acquisition unit 401 acquires a low-size image as input data to be processed from the imaging device 10 or another device. Note that the acquisition unit 401 may acquire a low-size image generated by the imaging device 10 as an input image in response to a request from the imaging device 10 . For example, when the image capturing device 10 generates an image, such a request is made before or after saving the image generated by the image capturing device 10 in a recording device included in the image capturing device 10, and displaying the saved image on the display unit 20. may be issued when displaying a high quality image, when using a high quality image for an image analysis process, and so on.

Since the processing in steps S520 and S530 is the same as the processing in the first embodiment, description thereof is omitted. In step S540, the image quality enhancing unit 404 inputs the input image to the image quality enhancing engine, and generates an image of high image quality with noise reduction and contrast enhancement and with a large image size. Since subsequent processing is the same as that of the first embodiment, the description is omitted.

As described above, the image quality enhancement unit 404 according to the present embodiment generates a high quality image in which at least one of noise reduction and contrast enhancement is performed and the image size is enlarged as compared with the input image. . As a result, the image processing apparatus 400 according to the present embodiment can output high-quality images suitable for image diagnosis without increasing the invasiveness or labor of the photographer or subject.

In this embodiment, a single image quality enhancement engine generates a high quality image by performing the image quality enhancement process and the image size enhancement process according to the first embodiment and the like. is not limited to this. For example, the image quality enhancement unit may include an image quality enhancement engine that performs image quality enhancement processing according to the first embodiment or the like and another image quality enhancement engine that performs image size enhancement processing.

In this case, the image quality enhancement engine that performs the image quality enhancement process according to the first embodiment and the like can use a machine learning model that has undergone learning in the same manner as the image quality enhancement engine according to the first embodiment and the like. can. Also, as the input data for the training data of the image quality enhancement engine that performs the image size enhancement process, the high quality image generated by the image quality enhancement engine according to the first embodiment and the like is used. Also, as the output data of the training data of the image quality improvement engine, a high quality image generated by the image quality improvement engine according to the first embodiment and the like is used for the image acquired by the high-performance imaging device. As a result, the image quality enhancement engine that performs the image size enhancement process generates a final high quality image by increasing the image size of the high quality image that has undergone the image quality enhancement process according to the first embodiment. can be done.

Further, the image size increasing processing by the image quality enhancing engine may be performed before the image quality increasing processing by the image enhancing processing engine according to the first embodiment and the like. In this case, the training data for the image quality enhancement engine that performs the image size enhancement processing is composed of pairs of input data as low-size images and output data as high-size images acquired by the imaging device. In addition, as training data for the image quality enhancement engine that performs image quality enhancement processing according to the first embodiment, etc., a large size image is input data, and an image obtained by superimposing the high size image is composed of a pair group of output data. do.

Even with such a configuration, the image processing apparatus 400 can generate, as a high-quality image, an image in which at least one of noise reduction and contrast enhancement has been performed as compared with the input image and the image size has been enlarged. can.

In addition, in the present embodiment, regarding the image quality improvement processing according to the first embodiment, etc., a configuration in which a superimposed image is used as output data of teacher data has been described. It is not limited to this. That is, at least one of a processing group and a photographing method such as superimposition processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, photographing using a high-performance photographing device, high-cost processing, and noise reduction processing. You may use the high quality image obtained by performing.

Note that the output unit 405 may output the generated high-quality image to another device connected to the imaging device 10 or the image processing device 400, as in the first embodiment.

<Sixteenth Embodiment>
Next, an image processing apparatus according to a sixteenth embodiment will be described with reference to FIGS. 4 and 5. FIG. In this embodiment, the image quality enhancement unit performs the image quality enhancement according to the first embodiment and the like, as well as the spatial resolution enhancement.

Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 400 according to the first embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below, focusing on the differences from the image processing apparatus according to the first embodiment. Note that the configuration of the image processing apparatus according to this embodiment is the same as that of the image processing apparatus according to the first embodiment, so the same reference numerals are used for the configuration shown in FIG. do.

The acquisition unit 401 according to this embodiment acquires a low spatial resolution image as an input image. Note that the low spatial resolution image is an image having a spatial resolution lower than that of the high spatial resolution image output by the image quality enhancing unit 404 .

The image quality enhancement unit 404 includes an image quality enhancement engine, as in the first embodiment. However, the image quality enhancement engine of the present embodiment is configured to reduce noise and enhance the contrast of the input image and to increase the spatial resolution of the input image. Therefore, the image quality enhancement engine according to this embodiment is configured to receive a low spatial resolution image and output a high spatial resolution image.

In relation to this, the input data constituting each pair is a low spatial resolution image, and the output data is a high spatial resolution image, also for a group of pairs constituting teacher data for the image quality enhancement engine. Note that the high spatial resolution image can be acquired from a device with higher performance than the imaging device that acquired the low spatial resolution image, or by changing the settings of the imaging device. As for the high spatial resolution image, a superimposed low spatial resolution image is used as in the first embodiment.

With such a configuration, the image quality improving unit 404 according to the present embodiment can generate a high quality image in which noise reduction and contrast enhancement are performed on the input image and also high spatial resolution is achieved.

Next, a series of image processing according to this embodiment will be described with reference to FIG. Note that the processes of steps S520, S530, and S550 according to the present embodiment are the same as those processes in the first embodiment, and therefore description thereof is omitted. Note that if the image quality of the input image is to be improved unconditionally with respect to the shooting conditions, the process of step S530 may be omitted after the process of step S520, and the process may proceed to step S540.

In step S510, the acquisition unit 401 acquires a low spatial resolution image as input data to be processed from the imaging device 10 or another device. Note that the acquisition unit 401 may acquire the low spatial resolution image generated by the imaging device 10 as the input image in response to a request from the imaging device 10 . For example, when the image capturing device 10 generates an image, such a request is made before or after saving the image generated by the image capturing device 10 in a recording device included in the image capturing device 10, and displaying the saved image on the display unit 20. may be issued when displaying a high quality image, when using a high quality image for an image analysis process, and so on.

Since the processing in steps S520 and S530 is the same as the processing in the first embodiment, description thereof is omitted. In step S540, the image quality enhancement unit 404 inputs the input image to the image quality enhancement engine, and generates an image with high spatial resolution as well as noise reduction and contrast enhancement as a high quality image. Since subsequent processing is the same as that of the first embodiment, the description is omitted.

As described above, the image quality enhancing unit 404 according to the present embodiment generates an image having at least one of noise reduction and contrast enhancement and improved spatial resolution as a high quality image compared to the input image. do. As a result, the image processing apparatus 400 according to the present embodiment can output high-quality images suitable for image diagnosis without increasing the invasiveness or labor of the photographer or subject.

In the present embodiment, one high-quality image engine generates a high-quality image by performing the high-quality image processing and the high-resolution processing according to the first embodiment and the like. It is not limited to this. For example, the image quality enhancement unit may include an image quality enhancement engine that performs image quality enhancement processing according to the first embodiment or the like and another image quality enhancement engine that performs high spatial resolution enhancement processing.

In this case, the image quality enhancement engine that performs the image quality enhancement process according to the first embodiment and the like can use a machine learning model that has undergone learning in the same manner as the image quality enhancement engine according to the first embodiment and the like. can. Also, as the input data for the training data of the image quality enhancement engine that performs the spatial resolution enhancement process, the high quality image generated by the image quality enhancement engine according to the first embodiment and the like is used. Also, as the output data of the training data of the image quality improvement engine, a high quality image generated by the image quality improvement engine according to the first embodiment and the like is used for the image acquired by the high-performance imaging device. As a result, the image quality enhancement engine that performs the spatial resolution enhancement process generates a final high quality image by increasing the spatial resolution of the high quality image that has been subjected to the image quality enhancement process according to the first embodiment. can be done.

Also, the high spatial resolution processing by the image quality enhancement engine may be performed before the image quality enhancement processing by the image enhancement processing engine according to the first embodiment and the like. In this case, the training data for the high image quality engine that performs high spatial resolution processing is composed of pairs of input data as low spatial resolution images and output data as high spatial resolution images acquired by the imaging device. . Also, as training data for the image quality enhancement engine that performs image quality enhancement processing according to the first embodiment, a pair group of input data of a high spatial resolution image and output data of an image obtained by superimposing the high spatial resolution images is used. Consists of

With such a configuration as well, the image processing apparatus 400 can generate, as a high-quality image, an image in which at least one of noise reduction and contrast enhancement has been performed as compared with the input image, and spatial resolution has been improved. .

In addition, in the present embodiment, regarding the image quality improvement processing according to the first embodiment, etc., a configuration in which a superimposed image is used as output data of teacher data has been described. It is not limited to this. That is, at least one of a processing group and a photographing method such as superimposition processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, photographing using a high-performance photographing device, high-cost processing, and noise reduction processing. You may use the high quality image obtained by performing.

Further, the image quality enhancement unit 404 may use the image quality enhancement engine to perform the image quality enhancement processing according to the fifteenth embodiment in addition to the high spatial resolution enhancement processing. In this case, at least one of noise reduction and contrast enhancement compared to the input image is performed, and an image having a higher image size and higher spatial resolution than the input image is generated as a high-quality image. can be done. As a result, the image processing apparatus 400 according to the present embodiment can output high-quality images suitable for image diagnosis without increasing the invasiveness or labor of the photographer or subject.

Note that the output unit 405 may output the generated high-quality image to another device connected to the imaging device 10 or the image processing device 400, as in the first embodiment.

<Seventeenth Embodiment>
Next, an image processing apparatus according to the seventeenth embodiment will be described with reference to FIGS. 22 and 23. FIG. In this embodiment, the analysis unit analyzes the high quality image generated by the image quality enhancement unit.

Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 400 according to the first embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below, focusing on the differences from the image processing apparatus according to the first embodiment.

FIG. 22 shows a schematic configuration of an image processing apparatus 2200 according to this embodiment. The image processing apparatus 2200 according to the present embodiment is provided with an analysis unit 2208 in addition to the acquisition unit 401, the imaging condition acquisition unit 402, the image quality enhancement determination unit 403, the image quality enhancement unit 404, and the output unit 405. ing. Note that the image processing device 2200 may be composed of a plurality of devices provided with some of these components. Here, the acquisition unit 401, the imaging condition acquisition unit 402, the image quality enhancement determination unit 403, the image quality enhancement unit 404, and the output unit 405 are the same as those of the image processing apparatus according to the first embodiment. , are denoted by the same reference numerals as those shown in FIG. 4, and the description thereof is omitted.

The analysis unit 2208 applies predetermined image analysis processing to the high quality image generated by the image quality enhancement unit 404 . For example, in the field of ophthalmology, image analysis processing includes segmentation of retinal layers, layer thickness measurement, papilla three-dimensional shape analysis, cribriform plate analysis, blood vessel density measurement of OCTA images, and corneal shape analysis for images acquired by OCT. Including any existing image analysis processing such as. Further, image analysis processing is not limited to analysis processing in the field of ophthalmology, and includes arbitrary existing analysis processing in the field of radiology such as diffusion tensor analysis and VBL (Voxel-based Morphometry) analysis.

The output unit 405 can display the high-quality image generated by the image-quality improving unit 404 on the display unit 20 and display the analysis result of the image analysis processing by the analysis unit 2208 . Note that the output unit 405 may display only the image analysis result by the analysis unit 2208 on the display unit 20, or may output the image analysis result to the imaging device 10, an image management system, other devices, or the like. Note that the display form of the analysis result may be arbitrary according to the image analysis processing performed by the analysis unit 2208, and may be displayed as an image, numerical values, or characters, for example.

A series of image processing according to this embodiment will be described below with reference to FIG. FIG. 23 is a flowchart of a series of image processing according to this embodiment. Note that the processing of steps S2310 to S2340 according to the present embodiment is the same as the processing of steps S510 to S540 in the first embodiment, so description thereof will be omitted. Note that if the image quality of the input image is to be improved unconditionally with respect to the shooting conditions, the process of step S2330 may be omitted after the process of step S2320, and the process may proceed to step S2340.

In step S2340, after the image quality enhancement unit 404 generates the high quality image group, the process proceeds to step S2350. In step S2350, the analysis unit 2208 analyzes the high-quality image generated in step S2340 and outputs the analysis result. Any existing image analysis processing and processing method may be used for the image analysis processing content and processing method. Note that the analysis unit 2208 may perform a plurality of image analysis processes. In this regard, the analysis unit 2208 may perform some or all of the plurality of image analysis processes according to instructions from the examiner via any input device.

In step S<b>2360 , the output unit 405 causes the display unit 20 to display the high-quality image generated by the image quality enhancement unit 404 and the analysis result by the analysis unit 2208 . Note that the output unit 405 may output the high-quality image and the analysis result to separate display units or devices. Alternatively, the output unit 405 may cause the display unit 20 to display only the analysis result. Furthermore, when the analysis unit 2208 outputs a plurality of analysis results, the output unit 405 may output some or all of the plurality of analysis results to the display unit 20 or other device.

As described above, the image processing apparatus 2200 according to this embodiment further includes the analysis unit 2208 that analyzes the high-quality image, and the output unit 405 causes the display unit 20 to display the analysis result of the analysis unit 2208 . As described above, in the image processing apparatus 2200 according to the present embodiment, high-quality images are used for image analysis, so the accuracy of analysis can be improved.

Also, the output unit 405 may output the generated high-quality image to another device connected to the imaging device 10 or the image processing device 2200, as in the first embodiment. Also, the output data of the training data from the image quality improvement engine is not limited to the high quality image that has undergone the superimposition process, as in the first embodiment. That is, at least one of a processing group and a photographing method such as superimposition processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, photographing using a high-performance photographing device, high-cost processing, and noise reduction processing. You may use the high quality image obtained by performing.

In the first to seventeenth embodiments, the display of a high-quality image on the display unit 20 by the output unit 405 is basically the generation of a high-quality image by the image quality enhancement unit 404 and the output of the analysis result by the analysis unit 2208. It will be done automatically accordingly. However, the high-quality image may be displayed according to an instruction from the examiner. For example, the output unit 405 may cause the display unit 20 to display an image selected from the high-quality image generated by the image quality enhancement unit 404 and the input image in accordance with an instruction from the examiner. In addition, the output unit 405 may switch the display on the display unit 20 from a captured image (input image) to a high-quality image in accordance with an instruction from the examiner.

Furthermore, the image quality enhancement unit 404 starts image quality enhancement processing by the image quality enhancement engine (input of an image to the image quality enhancement engine) in accordance with an instruction from the examiner. The display unit 20 may display the high-quality image generated by the conversion unit 404 . On the other hand, when an input image is captured by the imaging device 10, the image quality enhancement engine automatically generates a high quality image based on the input image, and the output unit 405 outputs a high quality image according to an instruction from the examiner. A quality image may be displayed on the display unit 20 . Note that these processes can be similarly performed for the output of analysis results. Furthermore, the image processing apparatus may be configured so that the processing by the photographing location estimation engine, the image quality evaluation engine, the authenticity evaluation engine, and the evaluation unit is started according to an instruction from the examiner.

Regarding the first to seventeenth embodiments, the display mode in which the output unit 405 displays the high-quality image on the display unit 20 may be arbitrary. For example, the output unit 405 may display the input image and the high-quality image side by side, or may switch between them. Further, the output unit 405 may display the input images and the high-quality images in order according to the imaged part, the imaged date and time, the facility where the image was taken, and the like. Similarly, the output unit 405 may display the image analysis results using the high-quality images in order according to arbitrary shooting conditions of the high-quality images and the input images corresponding to the high-quality images. Furthermore, the output unit 405 may display the image analysis results using high-quality images in order for each analysis item.

(Other examples)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

Although the present invention has been described with reference to the embodiments, the present invention is not limited to the above embodiments. Inventions modified within the scope of the present invention and inventions equivalent to the present invention are also included in the present invention. Moreover, each of the above-described embodiments can be appropriately combined within the scope of the present invention.

100: Image processing apparatus (medical image processing apparatus), 104: Image quality enhancement unit, 105: Output unit

A medical image processing apparatus according to an embodiment of the present invention includes an acquisition unit that acquires a first image that is an image of a predetermined region of a subject, and an image quality enhancement engine that includes a machine learning engine. by inputting a generation unit that generates a second image having a higher image quality than the first image, and when the second image is generated, the first image and the second and a display control unit that controls the display unit to display at least one of the images of the image quality improvement engine, superimposition processing, maximum posterior probability estimation processing, smoothing filter processing, gradation conversion processing and noise an image obtained by at least one reduction process, an image captured by an image capturing apparatus having higher performance than the image capturing apparatus used to capture the first image, and the image capturing step of the first image. and a machine learning engine that uses at least one of images acquired in a photographing process that requires more man-hours as learning data.

Further, a medical image processing method according to another embodiment of the present invention acquires a first image, which is an image of a predetermined region of a subject, and causes an image quality enhancement engine including a machine learning engine to perform the first image processing. generating a second image having a higher image quality than the first image by inputting the image of; and when the second image is generated, the first image and the first and controlling a display unit to display at least one of the two images , wherein the image quality improvement engine performs superimposition processing, maximum a posteriori probability estimation processing, smoothing filter processing, tone conversion processing, and noise an image obtained by at least one reduction process, an image captured by an image capturing apparatus having higher performance than the image capturing apparatus used to capture the first image, and the image capturing step of the first image. and a machine learning engine that uses at least one of images acquired in a photographing process that requires more man-hours as learning data .

Claims (37)

an acquisition unit that acquires a first image that is an image of a predetermined region of the subject;
high image quality generating, from the first image, a second image having at least one of noise reduction and contrast enhancement relative to the first image using an image enhancement engine including a machine learning engine; chemical department and
A medical image processing apparatus comprising: The image quality improvement engine is a machine learning engine that uses an image obtained by at least one of superposition processing, maximum posterior probability estimation processing, smoothing filter processing, tone conversion processing, and noise reduction processing as learning data. The medical image processing apparatus according to claim 1, comprising: The image quality enhancement engine acquires an image captured by an image capturing device having higher performance than the image capturing device used to capture the first image, or an image capturing process involving more man-hours than the image capturing process of the first image. 2. The medical image processing apparatus according to claim 1, further comprising a machine learning engine using the processed image as learning data. The medical device according to any one of claims 1 to 3, further comprising a determination unit that determines whether or not the second image can be generated from the first image using the image quality enhancement engine. Image processing device. 5. The medical image processing apparatus according to claim 4, wherein the determination unit makes the determination based on at least one of the imaging region, imaging method, imaging angle of view, and image size of the first image. 6. The medical image processing apparatus according to claim 4, wherein said image quality enhancing section generates said second image in accordance with an input from an examiner regarding a determination result by said determining section. 7. The medical image processing apparatus according to any one of claims 1 to 6, wherein said image quality enhancing unit comprises a plurality of image quality enhancing engines each trained using different learning data. each of the plurality of image quality enhancement engines performs learning using different learning data for at least one of the imaging region, imaging angle of view, and image resolution;
The image quality enhancement unit generates the second image using the image quality enhancement engine corresponding to at least one of a photographed region, a photographing angle of view, and an image resolution of the first image. 8. The medical image processing apparatus according to claim 7. 8. The medical image processing according to claim 7, wherein said image quality enhancing unit generates said second image using said image quality enhancing engine among said plurality of image quality enhancing engines according to an instruction from an examiner. Device. The image quality enhancement unit uses the plurality of image quality enhancement engines to generate a plurality of the second images from the first image,
8. The medical image processing apparatus according to claim 7, wherein said medical image processing apparatus outputs at least one of said plurality of second images according to an instruction from an examiner. Further comprising an evaluation unit for evaluating the image quality of the second image,
The image quality enhancement unit uses the plurality of image quality enhancement engines to generate a plurality of the second images from the first image,
8. The medical image processing apparatus according to claim 7, wherein said medical image processing apparatus outputs at least one of said plurality of second images according to the evaluation result by said evaluation unit. The evaluation unit includes a machine learning engine whose learning data is an evaluation value obtained by a predetermined evaluation method,
12. The medical image processing apparatus according to claim 11, wherein said medical image processing apparatus outputs said second image having the highest evaluation value by said evaluation unit among said plurality of second images. Further comprising an evaluation unit for evaluating the image quality of the second image,
The evaluation unit includes an authenticity evaluation engine that evaluates the authenticity of the image,
The medical image processing apparatus according to any one of claims 1 to 10, wherein the medical image processing apparatus outputs the second image when the output from the authenticity evaluation engine of the evaluation unit is true. . 14. The medical image processing according to claim 13, wherein said authenticity evaluation engine includes a machine learning engine using, as learning data, an image generated by an image quality enhancement engine whose image quality enhancement processing accuracy is lower than that of said image quality enhancement engine. Device. The medical image processing apparatus according to any one of claims 1 to 14, further comprising an estimating unit that estimates at least one of an imaging region and an imaging area of the first image. 16. The medical image processing apparatus according to claim 15, wherein the estimating unit includes a machine learning engine using images labeled with at least one of an imaging region and an imaging region as learning data. The image quality enhancement unit
adjusting the image size of the first image to an image size that can be handled by the image quality improvement engine and inputting it to the image quality improvement engine;
17. The medical image processing according to any one of claims 1 to 16, wherein the second image is generated by adjusting the output image from the image quality enhancement engine to the original image size of the first image. Device. The image quality enhancement unit
adjusting the image size of the first image so that the resolution of the first image is a predetermined resolution;
The first image whose image size has been adjusted is input to an image quality enhancement engine by performing padding such that the adjusted image size becomes an image size that can be dealt with by the image quality enhancement engine. ,
Trimming the output image from the image quality enhancement engine by the padding area,
18. The medical device according to any one of claims 1 to 17, wherein the second image is generated by adjusting the image size of the trimmed image to the original image size of the first image. Image processing device. The image quality enhancement unit
dividing the first image into a plurality of third images of a predetermined image size;
inputting the plurality of third images into the image quality enhancement engine to generate a plurality of fourth images;
The medical image processing apparatus according to any one of claims 1 to 18, wherein the second image is generated by integrating the plurality of fourth images. The image quality enhancement unit
dividing the first three-dimensional image into a plurality of two-dimensional images or a plurality of one-dimensional images and inputting them to the image quality enhancement engine;
The medical image processing apparatus according to any one of claims 1 to 19, wherein the second image is generated by integrating a plurality of output images from the image quality enhancement engine. The image quality enhancement unit
including a plurality of the image quality enhancement engines,
dividing the three-dimensional first image into a plurality of two-dimensional images or a plurality of one-dimensional images;
generating a plurality of second images from the plurality of two-dimensional images or the plurality of one-dimensional images using the plurality of image quality enhancement engines in parallel;
The medical image processing apparatus according to any one of claims 1 to 20, wherein the three-dimensional second image is generated by integrating the plurality of second images. The image quality enhancement unit produces an image that has been subjected to at least one of noise reduction and contrast enhancement, and at least one of image size enlargement and spatial resolution enhancement, as compared with the first image. 22. The medical image processing apparatus according to any one of claims 1 to 21, which is generated as a second image. The medical image processing apparatus according to any one of claims 1 to 22, further comprising a display control section that displays the second image on a display section. 24. The medical image processing apparatus according to claim 23, wherein said display control section causes said display section to display said second image in accordance with an instruction from an examiner. 25. The display control unit according to claim 23, wherein the display control unit causes the display unit to display, together with the second image, a display indicating that the second image is the image generated by the image quality enhancement engine. medical image processing equipment. The display control unit causes the display unit to display the second image,
26. The medical image processing apparatus according to any one of claims 23 to 25, wherein said medical image processing apparatus outputs said second image according to an input from an examiner for said displayed second image. . Further comprising an analysis unit for image analysis of the second image,
The medical image processing apparatus according to any one of claims 23 to 26, wherein said display control unit causes said display unit to display an analysis result by said analysis unit. 28. The medical image processing apparatus according to any one of claims 1 to 27, wherein said image quality enhancing unit generates one said second image from a plurality of said first images. The medical image processing apparatus according to any one of claims 1 to 28, wherein said first image is an image generated using a plurality of images of a predetermined region of said subject. The medical image processing apparatus according to any one of claims 1 to 29, wherein said image quality enhancing unit generates a plurality of said second images and averages said plurality of second images. The medical image processing apparatus according to any one of claims 1 to 30, wherein said predetermined portion is an eye to be examined. The medical image processing apparatus according to any one of claims 1 to 31, wherein said acquisition unit acquires said first image from an imaging device. The medical image according to any one of claims 1 to 31, wherein said acquisition unit acquires data of said predetermined part of said subject from an imaging device and acquires said first image based on said data. processing equipment. The medical image processing apparatus according to any one of claims 1 to 31, wherein said acquisition unit acquires said first image from an image management system. The medical image processing apparatus according to any one of claims 1 to 34, wherein said image quality enhancement engine does not use said second image as learning data. Acquiring a first image that is an image of a predetermined part of the subject;
generating, from the first image, a second image having at least one of noise reduction and contrast enhancement compared to the first image using an image enhancement engine including a machine learning engine; ,
A medical image processing method, comprising: 37. A program which, when executed by a processor, causes the processor to perform the steps of the medical image processing method of claim 36.

Images