JP7362403B2 - Image processing device and image processing method - Google Patents

Description

The present invention relates to an image processing apparatus and an image processing method that process a tomographic image of a subject obtained by optical coherence tomography (OCT).

Medical tomographic imaging devices such as OCT are capable of three-dimensionally observing the internal state of the retinal layer, and are more useful in diagnosing ophthalmological and retinal diseases such as AMD, for example. In recent years, OCT that has been used in clinical practice includes SD-OCT (Spectral domain OCT), which uses a broadband light source to obtain interferograms with a spectrometer, as a method of acquiring images at high speed, and SS-OCT (Swept Source OCT) is broadly divided into two methods, which use a high-speed wavelength swept light source to measure spectral interference with a single-channel photodetector. OCT angiography (OCTA), which images blood vessels without using agents, has been attracting attention. OCTA generates motion contrast data from OCT images acquired by OCT. Here, motion contrast data is data obtained by repeatedly photographing the same cross-section of a measurement object using OCT and detecting temporal changes in the measurement object between the images, such as the phase, vector, and intensity of a complex OCT signal. It is calculated from the difference, ratio, correlation, etc. over time.

In addition, when displaying an OCTA image, it is becoming common to display it as a two-dimensional OCTA frontal image by projecting three-dimensional motion contrast data calculated from the acquired three-dimensional OCT image onto a two-dimensional plane. However, at this time, Patent Document 1 discloses a technique for generating a two-dimensional front image by specifying a range in the depth direction of motion contrast data to be projected.

JP 2017-6179 Publication

However, there may be room for improvement in various aspects regarding analysis processing of OCT data or motion contrast data. For example, when setting a region of interest to be subjected to analysis processing of OCT data or motion contrast data, it may be difficult to appropriately set it using only each frontal image.

One of the objects of the present invention, which has been made in view of the above-mentioned problems, is to provide a system that can suitably set a region of interest that is a target of analysis processing and the like.

One of the image processing devices according to the present invention is
acquisition means for acquiring an OCT front image and an OCTA front image in mutually corresponding regions and mutually corresponding depth ranges in the subject using an optical coherence tomography;
A trained model for high image quality obtained by learning an OCTA frontal image of a subject, the high-quality OCTA frontal image obtained by inputting the acquired OCTA frontal image as input data to the trained model. Displaying a blended image obtained by performing a blending process using the image and the acquired OCT frontal image using a transmittance that can be changed according to instructions from the examiner , and changing the transmittance. display control means for controlling the display means to display information that accepts instructions from the examiner regarding the
The display control means is configured to display a region of interest set in the displayed blended image according to an instruction from the examiner , and to display a region of interest corresponding to the set region of interest in the high-quality OCTA frontal image . Displays the results of the type of analysis processing selected in response to instructions from the examiner among the types of analysis processing performed, and displays the results of the type of analysis processing selected according to instructions from the examiner, and receives information from the examiner regarding changing from the high-quality OCTA frontal image to the acquired OCTA frontal image. If the instruction is received, the display of the result of the analysis process is changed to the display of the result of the selected type of analysis process for the set region of interest in the acquired OCTA frontal image. , controlling the display means .

According to one aspect of the present invention, it is possible to configure a region of interest that is a target of analysis processing, etc. to be suitably settable.

FIG. 1 is a block diagram showing the configuration of an image processing device according to a first embodiment. FIG. 1 is a diagram illustrating a tomographic imaging apparatus according to a first embodiment. FIG. 3 is a diagram illustrating a display screen that displays a front image of an optic disc and a display screen that displays a blended image obtained by transmission processing. FIG. 1 is a block diagram showing the configuration of an image processing device according to a first embodiment. 7 is a flowchart showing analysis processing according to the first embodiment. An example of the configuration of a neural network related to high image quality processing according to the fourth embodiment is shown. FIG. 7 is a flow diagram showing an example of the flow of image processing according to the fourth embodiment. An example of the configuration of a neural network related to high image quality processing according to the fourth embodiment is shown. An example of the configuration of a neural network used as a machine learning engine according to modification 6 is shown. An example of the configuration of a neural network used as a machine learning engine according to modification 6 is shown. An example of a user interface according to a fifth embodiment is shown. An example of a teacher image related to image quality enhancement processing is shown. An example of an input image related to high image quality processing is shown. An example of a user interface according to a fifth embodiment is shown. An example of the configuration of a neural network related to high image quality processing according to the fourth embodiment is shown.

[First embodiment]
A case will be described in which the image processing apparatus according to the present embodiment performs analysis processing while setting an analysis position and an analysis region while referring to a front image of OCT data for analysis of OCTA data. DESCRIPTION OF THE PREFERRED EMBODIMENTS An image processing system including an image processing apparatus according to a first embodiment of the present invention will be described below with reference to the drawings.

(Configuration of image processing device)
The configuration of the image processing apparatus 101 of this embodiment and the connection with other devices will be explained using FIG. 1. The image processing device 101 is a personal computer (PC) connected to the tomographic imaging device 100, and includes functional blocks such as an image acquisition section 101-01, an imaging control section 101-03, an image processing section 101-04, and a display. Each function of the control unit 101-05 is realized by a not-shown arithmetic processing unit CPU executing software modules stored in the storage unit 101-02. Of course, the present invention is not limited to such a PC configuration; for example, the image processing unit 101-04 may be implemented using dedicated hardware such as an ASIC, or the display control unit 101-05 may be implemented using a GPU different from the CPU. It may be realized using a dedicated processor such as. Furthermore, the connection between the tomographic imaging apparatus 100 and the image processing apparatus 101 may be configured via a network, and the external storage unit 102 may also be placed on the network so that data can be shared by multiple image processing apparatuses. may be configured.

The image acquisition unit 101-01 is a functional block that acquires the signal data of the SLO fundus image and tomographic image taken of the subject by the tomographic image capturing apparatus 100 and generates an image. It has a generation section 101-12. The tomographic image generation unit 101-11 acquires the signal data (interference signal) of the tomographic image taken by the tomographic imaging apparatus 100, generates a tomographic image by signal processing, and stores the generated tomographic image in the storage unit 101-02. Store. The motion contrast data generation unit 101-12 generates motion contrast data based on the plurality of tomographic images of the same region (mutually corresponding regions of the subject) generated by the tomographic image generation unit 101-11.

First, the tomographic image generation unit 101-11 performs wave number transformation, fast Fourier transform (FFT), and absolute value conversion (acquisition of amplitude) on the interference signal acquired by the image acquisition unit 101-01. Generate tomographic images for clusters.

Next, the alignment unit 101-41 aligns the tomographic images belonging to the same cluster and performs superimposition processing. An image feature acquisition unit 101-44 acquires layer boundary data from the superimposed tomographic image. In this embodiment, a variable shape model is used as a layer boundary acquisition method, but any known layer boundary acquisition method may be used. Note that the layer boundary acquisition process is not essential; for example, if a motion contrast image is generated only in three dimensions and a two-dimensional motion contrast image projected in the depth direction is not generated, the layer boundary acquisition process can be omitted. A motion contrast data generation unit 101-12 calculates motion contrast between adjacent tomographic images within the same cluster. In this embodiment, a decorrelation value Mxy is determined as the motion contrast based on the following equation (1).

Here, Axy indicates the amplitude (of complex number data after FFT processing) at the position (x, y) of the tomographic image data A, and Bxy indicates the amplitude at the same position (x, y) of the tomographic data B. 0≦Mxy≦1, and the larger the difference between both amplitude values, the closer the value is to 1. Perform the decorrelation calculation process as shown in Equation (1) between arbitrary adjacent tomographic images (belonging to the same cluster), and calculate the average of the obtained (number of tomographic images per cluster - 1) motion contrast values. An image having pixel values is generated as a final motion contrast image.

Note that although the motion contrast is calculated here based on the amplitude of the complex number data after FFT processing, the method of calculating the motion contrast is not limited to the above. For example, motion contrast may be calculated based on phase information of complex number data, or may be calculated based on both amplitude and phase information. Alternatively, motion contrast may be calculated based on the real part or imaginary part of complex number data.

Further, in this embodiment, a decorrelation value is calculated as the motion contrast, but the method of calculating the motion contrast is not limited to this. For example, the motion contrast may be calculated based on the difference between two values, or the motion contrast may be calculated based on the ratio of two values.

Further, in the above, the final motion contrast image is obtained by calculating the average value of the plurality of acquired decorrelation values, but the present invention is not limited to this. For example, an image having the median value or maximum value of the plurality of acquired decorrelation values as a pixel value may be generated as the final motion contrast image.

The imaging control unit 101-03 is a functional block that performs imaging control for the tomographic imaging apparatus 100, and instructs the tomographic imaging apparatus 100 regarding the setting of imaging parameters and the start or end of imaging. Also included. The image processing section 101-04 is a functional block including a positioning section 101-41, a composition section 101-42, a correction section 101-43, an image feature acquisition section 101-44, a projection section 101-45, and an analysis section 101-46. It is. The synthesis section 101-42 includes, for example, a synthesis method specification section 101-421, a same modality image synthesis section 101-422, and a different type of modality image synthesis section 101-423. The composition unit 101-42 composes one image from a plurality of two-dimensional images. Specifically, the synthesis method specification unit 101-421 specifies the type of images to be synthesized (tomographic image/motion contrast image/tomographic image and motion contrast image) and the synthesis processing method (superimposition/stitching/juxtaposed display). do. The same modality image synthesis unit 101-422 performs synthesis processing between tomographic images or between motion contrast images. The multiple modality image synthesis unit 101-423 performs synthesis processing between a tomographic image and a motion contrast image. Here, the synthesizing unit 101-42 is an example of an image quality improving unit that improves the image quality of motion contrast data. In addition, in this embodiment as well, in addition to the processing by the compositing unit 101-42, as the processing by the image quality improvement means, for example, the image quality improvement processing by machine learning in the fourth embodiment described later can be applied. It is. Further, the correction unit 101-43 performs processing to suppress projection artifacts occurring within the motion contrast image. Here, projection artifact is a phenomenon in which the motion contrast within the superficial retinal blood vessels is reflected in the deep layers (deep retina, outer retina, and choroid), resulting in high decorrelation values in deep regions where no blood vessels actually exist. Point. For example, the correction unit performs processing to reduce projection artifacts in the combined motion contrast data. That is, the correction unit 101-43 corresponds to an example of a processing unit that performs processing to reduce projection artifacts on the synthesized motion contrast data. Furthermore, the projection unit 101-45 projects a tomographic image or a motion contrast image in a depth range based on the boundary position acquired by the image feature acquisition unit 101-44, and generates a brightness front image (brightness tomographic image) or a motion contrast front image. generate. At this time, projection may be performed in any depth range, but in this embodiment, two types of frontal composite motion contrast images are generated in the depth range of the retinal surface layer and the retinal outer layer. Further, as the projection method, either maximum intensity projection (MIP) or average intensity projection (AIP) can be selected. Here, the projection range for generating the motion contrast front image can be changed by the operator selecting from a predetermined depth range set displayed in a selection list (not shown) or the like. In addition, the projection range can be changed by changing the type and offset position of the layer boundary used for specifying the projection range from the user interface, or by moving the layer boundary data superimposed on the tomographic image by operating from the input unit 103. can be changed. Note that the motion contrast image displayed on the display unit 104 is not limited to a motion contrast front image, and a three-dimensional motion contrast image rendered three-dimensionally may be displayed. Furthermore, the above-described projection method and the presence or absence of projection artifact suppression processing may be changed by selecting from a user interface such as a context menu, for example. For example, the motion contrast image after projection artifact suppression processing may be displayed on the display unit 104 as a three-dimensional image. Furthermore, the analysis section 101-46 is a functional block that includes an emphasis section 101-461, an extraction section 101-462, a measurement section 101-463, and a comparison section 101-464. Here, the extraction unit 101-462 acquires the layer boundaries of the retina and choroid, the boundaries of the anterior and posterior surfaces of the lamina cribrosa, and the positions of the fovea and the center of the optic disc from the tomographic image. Furthermore, the extraction unit 101-462 extracts a blood vessel region from the motion contrast frontal image. The measurement unit 101-463 calculates measured values such as blood vessel density using the extracted blood vessel region and blood vessel centerline data obtained by thinning the blood vessel region.

The image processing device 101 is configured by being connected to a tomographic imaging device 100, an external storage section 102, an input section 103, and a display section 104 via an interface. The image processing device 101 controls the stage section 100-2 and the alignment operation. The external storage unit 102 stores programs for tomographic imaging, information on the eye to be examined (patient's name, age, gender, etc.), captured images (tomographic images, SLO images, OCTA images), composite images, imaging parameters, and past examinations. Image data, measurement data, parameters set by the operator, etc. are stored in association with each other.

The input unit 103 is, for example, a mouse, a keyboard, a touch operation screen, etc., for giving instructions to the computer. conduct. The display unit 104 is, for example, a monitor, and may have a touch UI.

(Configuration of tomographic imaging device)
The tomographic image capturing apparatus 100 is a device that captures tomographic images of the eye. The configuration of the measurement optical system and spectrometer in the tomographic imaging apparatus 100 of this embodiment will be described using FIG. 2.

In this embodiment, SD-OCT is used as the tomographic imaging apparatus 100. The configuration is not limited to this, and may be configured using SS-OCT, for example.

The measurement optical system 100-1 is an optical system for acquiring an anterior segment image, an SLO fundus image of the eye to be examined, and a tomographic image. The stage section 100-2 allows the measurement optical system 100-1 to move back and forth and left and right. The base portion 100-3 incorporates a spectrometer to be described later.

First, the inside of the measurement optical system 100-1 will be explained. An objective lens 201 is placed facing the eye 200 to be examined, and a first dichroic mirror 202 and a second dichroic mirror 203 are placed on its optical axis. These dichroic mirrors branch the optical path into an optical path 250 for the OCT optical system, an optical path 251 for the SLO optical system and fixation lamp, and an optical path 252 for anterior eye observation for each wavelength band. The optical path 251 for the SLO optical system and fixation lamp includes an SLO scanning means 204, lenses 205 and 206, a mirror 207, a third dichroic mirror 208, an APD (Avalanche Photodiode) 209, an SLO light source 210, and a fixation lamp 211. ing. The mirror 207 is a prism in which a perforated mirror or a hollow mirror is vapor-deposited, and separates the illumination light from the SLO light source 210 and the return light from the eye to be examined. The third dichroic mirror 208 separates the optical path of the SLO light source 210 and the optical path of the fixation lamp 211 for each wavelength band. The SLO scanning means 204 scans the eye 200 with light emitted from the SLO light source 210, and is composed of an X scanner that scans in the X direction and a Y scanner that scans in the Y direction. In this embodiment, since it is necessary to perform high-speed scanning, the X scanner is configured with a polygon mirror, and the Y scanner is configured with a galvanometer mirror. The lens 205 is driven by a motor (not shown) to focus the SLO optical system and the fixation lamp 211. SLO light source 210 generates light at a wavelength around 780 nm. APD 209 detects return light from the eye to be examined. The fixation lamp 211 generates visible light to encourage the subject to fixate. The light emitted from the SLO light source 210 is reflected by the third dichroic mirror 208, passes through the mirror 207, passes through the lenses 206 and 205, and is scanned onto the eye 200 by the SLO scanning means 204. After the return light from the eye 200 to be examined returns along the same path as the illumination light, it is reflected by the mirror 207 and guided to the APD 209, whereby an SLO fundus image is obtained. The light emitted from the fixation lamp 211 passes through the third dichroic mirror 208 and the mirror 207, passes through the lenses 206 and 205, and forms a predetermined shape at an arbitrary position on the eye 200 by the SLO scanning means 204. Encourage the subject to fixate. Lenses 212 and 213, a split prism 214, and a CCD 215 for anterior eye observation that detects infrared light are arranged in the optical path 252 for anterior eye observation. This CCD 215 is sensitive to the wavelength of irradiation light for anterior ocular segment observation (not shown), specifically around 970 nm. The split prism 214 is arranged at a position conjugate with the pupil of the eye to be examined 200, and measures the distance in the Z-axis direction (optical axis direction) of the measurement optical system 100-1 to the eye to be examined 200 as a split image of the anterior segment of the eye. Can be detected. The optical path 250 of the OCT optical system constitutes the OCT optical system as described above, and is for capturing a tomographic image of the eye 200 to be examined. More specifically, the purpose is to obtain an interference signal for forming a tomographic image. The XY scanner 216 is for scanning light on the eye 200 to be examined, and although it is shown as a single mirror in FIG. 2(b), it is actually a galvanometer mirror that scans in two directions of the X and Y axes. Of the lenses 217 and 218, the lens 217 is driven by a motor (not shown) in order to focus the light from the OCT light source 220 emitted from the fiber 224 connected to the optical coupler 219 onto the eye 200 to be examined. By this focusing, the return light from the eye 200 to be examined is simultaneously formed into a spot image and incident on the tip of the fiber 224. Next, the configuration of the optical path from the OCT light source 220, the reference optical system, and the spectrometer will be explained. 220 is an OCT light source, 221 is a reference mirror, 222 is a dispersion compensating glass, 223 is a lens, 219 is an optical coupler, 224 to 227 are single mode optical fibers connected and integrated with the optical coupler, and 230 is a spectrometer. . These configurations constitute a Michelson interferometer. The light emitted from the OCT light source 220 passes through the optical fiber 225 and is split via the optical coupler 219 into measurement light on the optical fiber 224 side and reference light on the optical fiber 226 side. The measurement light is irradiated onto the eye 200 to be observed, which is the object of observation, through the optical path of the aforementioned OCT optical system, and reaches the optical coupler 219 through the same optical path due to reflection and scattering by the eye 200 to be examined. On the other hand, the reference light reaches the reference mirror 221 and is reflected through the optical fiber 226, the lens 223, and the dispersion compensating glass 222 inserted to match the wavelength dispersion of the measurement light and the reference light. Then, it returns along the same optical path and reaches the optical coupler 219. The measurement light and the reference light are combined by the optical coupler 219 to become interference light. Here, interference occurs when the optical path length of the measurement light and the optical path length of the reference light become almost the same. The reference mirror 221 is held adjustable in the optical axis direction by a motor and a drive mechanism (not shown), and can match the optical path length of the reference light to the optical path length of the measurement light. The interference light is guided to a spectrometer 230 via an optical fiber 227. Further, polarization adjustment units 228 and 229 are provided in the optical fibers 224 and 226, respectively, and perform polarization adjustment. These polarization adjustment sections have several sections in which optical fibers are routed in loops. By rotating this loop-shaped portion around the longitudinal direction of the fiber, it is possible to twist the fiber and adjust the polarization states of the measurement light and the reference light to match each other. The spectrometer 230 includes lenses 232 and 234, a diffraction grating 233, and a line sensor 231.

The interference light emitted from the optical fiber 227 becomes parallel light through the lens 234 , is separated by the diffraction grating 233 , and is imaged onto the line sensor 231 by the lens 232 . Next, the surroundings of the OCT light source 220 will be described. The OCT light source 220 is an SLD (Super Luminescent Diode), which is a typical low coherent light source. The center wavelength is 855 nm, and the wavelength bandwidth is approximately 100 nm. Here, the bandwidth is an important parameter because it affects the resolution of the obtained tomographic image in the optical axis direction. As for the type of light source, SLD is selected here, but it is sufficient that it can emit low coherent light, and ASE (Amplified Spontaneous Emission) or the like may be used. Considering that the eye is to be measured, near-infrared light is suitable as the center wavelength. Furthermore, since the center wavelength affects the lateral resolution of the obtained tomographic image, it is desirable that the center wavelength be as short as possible. For both reasons, the center wavelength was set to 855 nm. In this embodiment, a Michelson interferometer is used as the interferometer, but a Mach-Zehnder interferometer may also be used. Depending on the difference in light intensity between the measurement light and the reference light, it is desirable to use a Mach-Zehnder interferometer when the difference in light intensity is large, and to use a Michelson interferometer when the difference in light intensity is relatively small.

(OCTA data analysis processing)
Hereinafter, analysis processing targeting OCT motion contrast data will be specifically described, but the terms used in the description of the embodiments will be briefly defined. First, information on three-dimensional volume data is expressed as OCT data or OCTA data. Next, two-dimensional information that can be extracted from the volume data is referred to as an OCT image or an OCTA image, and in particular, an image created by projecting volume data in a specified depth direction range is referred to as an OCT front image or an OCTA front image. Further, two-dimensional information including data in the depth direction is referred to as a tomographic image.

FIG. 3A shows an OCTA frontal image 301 of the optic nerve head (ONH). The slide bar 302 indicates a transmittance of 0% as an initial value, which is the transmittance of an OCT front image to be described later. Here, the transmittance of the slide bar 302 may be stored as the previously set transmittance, or may be returned to the initial value of 0% when switching to another OCTA front image.

It is known that ONH vascular function is closely related to the progress of glaucoma, and quantitative analysis of vascular dysfunction is said to be of great clinical value. However, it is somewhat difficult to set the boundary of the neural canal opening (NCO) on an OCTA frontal image. If it is an OCT frontal image, the visibility of the NCO will increase, making it easier to set the analysis area. In glaucoma, it is important to obtain reliable information regarding microcirculation in order to evaluate the role of ONH circulation disorders.

FIG. 3B shows, as an example, a case where the operator sets the slide bar 302 to 60%. An image 303 generated from the OCTA front image and a second OCT front image different from the OCTA front image based on the set transmittance is displayed. That is, a blended image is generated that is subjected to blending processing based on changeable transmittance using an OCT image and an OCT image of the same location of the subject acquired by an optical coherence tomography. Then, analysis processing for the set analysis area 304 is executed. Specifically, the image processing system of the image processing apparatus 101 of this embodiment will be described with reference to FIGS. 1 and 4. First, when the operator specifies an OCTA front image in the specified depth direction range as the target image, the OCTA front image stored in the storage unit 101-02 and the OCT front image that becomes the second medical image are acquired. Ru. The ranges of the OCTA front image and the OCT front image in the depth direction do not necessarily have to match. It is also possible for the operator to specify different ranges in the depth direction. The transmittance setting unit 402 sets the transmittance based on the position set by the operator with the slide bar 302, and sets the transmittance coefficient α (0≦α≦1) of the second medical image (here, the OCT front image). decide. Here, the transparency process uses a general alpha blending process to perform a weighted average of two images for each pixel. For example, the blending process is performed by weighted averaging processing of pixel values at mutually corresponding positions in the OCT image and the OCTA image.

(Transparent image) = (first medical image) x (1 - α) + (second medical image) x α... (Formula 2)
The blend processing unit 403 generates a blend image (hereinafter referred to as a transparent image) based on the above (Equation 2), and the display control unit 101-05 displays the generated blend image on the display unit 104. The operator may change the transmittance while checking the transparent image displayed on the display unit 104 until a desired transparent image is obtained. Further, the operator may change the range of the image in the depth direction while checking the visibility.

Next, the ROI setting unit 404 sets a region of interest (ROI) to be analyzed on the transparent image. The ROI information may be set as parameters such as a center position and size, or may be set as a general shape (for example, a circle, an ellipse, a rectangle, etc.). Further, the ROI information may be set as a free region using a spline curve or the like with a plurality of control points. The ROI information may be displayed in a superimposed manner on the transparent image. Furthermore, in order to confirm whether the set ROI is a desired region, the operator can update the transparent image by changing the transmittance while the ROI information is displayed in a superimposed manner. In this way, by appropriately changing the transmittance of the OCT front image, the state of microcirculation or the visibility of the NCO boundary can be adjusted.

Finally, an analysis unit 101-46, which is an example of an execution unit that executes processing on a region of interest in an image, executes various image analyzes. The type of analysis may be specified by the operator or may be a preset analysis. An extraction unit 101-462 extracts image features according to the type of analysis, and a measurement unit 101-463 performs various measurements. The analysis results are displayed on the display unit 104. For example, the operator instructs blood vessel extraction processing based on the set ROI information. The extraction unit 101-462 uses the OCTA frontal image to perform blood vessel extraction by discriminating between blood vessel regions and non-vascular regions. As an example of the determination process, pixels satisfying a predetermined threshold may be extracted as blood vessel regions using threshold processing. The threshold value may be a fixed value set in advance, or may be arbitrarily set by the subject. Further, the threshold value may be adaptively set based on a predetermined algorithm (for example, histogram analysis) depending on the OCTA frontal image. The blood vessel extraction process may be expressed as binary information of blood vessel/non-blood vessel, or may be expressed as a continuous value as a measure of blood vessel-likeness (for example, distance from a threshold value). Regarding the blood vessel region, specific color information may be added, or in the case of continuous values, color information may be added with a predetermined gradation. The color and gradation indicating blood vessel information are not limited to reds, and may be freely selected by the operator.

Furthermore, colors may be added depending on the depth of blood vessels based on OCTA data. By coloring the blood vessels in this way, the image when the operator sets the ROI becomes easier to understand. Of course, blood vessels may be extracted from OCTA data. If blood vessels are extracted as three-dimensional information, it is also possible to add color information based on the position and thickness of the blood vessels.

The display control unit 101-05 performs blood vessel measurement based on the blood vessel information extracted by the extraction unit 101-462, and displays it on the display unit 104. Blood vessel measurement may be performed using, for example, blood vessel density or blood vessel area. As the density of the blood vessel region, for example, the area of the blood vessel per unit area is determined by calculating the ratio of the blood vessel region to the entire region of the ROI. Blood vessel measurement is not limited to this, but may also be the total volume of blood vessels, blood vessel tortuosity, etc.

Furthermore, the ROI may be divided into a plurality of regions and the difference or ratio between the measured values for each region may be calculated. In this way, for example, the symmetry of blood vessels can be evaluated. The blood vessel density may be displayed as an analysis result as a color map image by associating the density of each predetermined area with color data. The color map image and the OCTA front image may be blended and displayed at a predetermined transmittance (for example, 50%). Further, a blended image of the OCTA frontal image and the OCT frontal image and a color map image may be displayed in a blended manner. The transmittance for the color map image may be fixed or may be specified by the operator.

Note that as a follow-up for the subject, periodic examinations may be performed and a progress observation display screen may be displayed in which the respective analysis results are arranged in chronological order. In that case, the comparison section 101-464 compares the analysis results, and the emphasis section 101-461 may further highlight and display the changes.

(Optic disc analysis processing procedure)
Next, the processing procedure of the image processing apparatus 101 of this embodiment will be described with reference to FIG. In S501, the transmittance α of the OCT front image with respect to the OCTA front image of the optic nerve head (ONH) is changed based on the setting value of the GUI. Here, α is a real number between 0 and 1. However, it may be expressed as a percentage on the GUI. In S502, transparency processing is performed on the two images based on the changed transmittance, and the transparent images are displayed on the screen. In S503, the operator determines a transmittance that makes it easy to set the ROI while checking the transparent image. Subsequently, in S504, an ROI, which is an analysis position or an analysis region, is set. In S505, a blood vessel extraction process based on the set ROI information is instructed. Finally, in S506, blood vessels of the optic nerve head (ONH) are measured, and the results are displayed on the screen.

The above explanation has been made using the analysis of the optic nerve head (ONH) as an example, but analysis of the macular region of the eye to be examined, detection of the foveal avascular region, etc. may also be used. For example, when analyzing new blood vessels in a deep layer of the macular region, ROI setting for the macular region is facilitated by performing transparency processing on the OCT front image of the superficial layer instead of the layer corresponding to the OCTA front image. That is, the layers of the OCTA front image and the OCT front image do not necessarily have to match, and transparency processing may be performed between images of different layers.

(Analysis of foveal avascular region)
Detection of the foveal avascular zone (FAZ) will be described below. Since the FAZ is an avascular region and has low brightness, the FAZ is extracted by, for example, determining the connectivity of brightness in the periphery using the center point of the FAZ analysis region as a reference. The extraction method may be performed using any known method, such as extraction using a region expansion method or extraction using a dynamic contour model such as Snake. Note that the above analysis process is not limited to blood vessels, but can also be applied to blood vessel analysis (for example, lymph vessels) as a blood vessel analysis area. Further, in this embodiment, an OCTA front image has been described as an example, but the order of motion contrast data is not limited. For example, a three-dimensional ROI may be set by generating weighted average three-dimensional information from OCTA data and OCT data. Of course, the motion contrast data may be one-dimensional motion contrast data or two-dimensional motion contrast data.

Furthermore, it is also possible to set an ROI for a tomographic image. When the check button 305 in FIG. 3(a) is turned on, a line 306 is displayed on the OCTA front image. The operator can move the position of line 306 up or down by dragging this line with the mouse. In conjunction with this operation, the tomographic image 307 is updated to a corresponding tomographic image. The intersection of the set ROI 304 and the line 306 may be displayed as a vertical line 308 on the tomographic image 307. The operator may adjust the ROI while checking the tomographic image by moving the line 308 left and right while dragging it with the mouse. According to the adjustment on line 308, the shape of ROI 304 is changed so that it is smoothly connected. Further, the range of motion of the line 308 may be suppressed within a range where ROI adjustment in the tomographic image does not destroy the shape of the ROI 304.

Further, although not illustrated, the OCTA tomographic image and the OCT tomographic image may be displayed as a blend in the tomographic image 307 as well. In this case, the slide bar 302 may be used in common, or slide bars may be added individually. In particular, for data from non-healthy eyes, by increasing the transmittance of the OCT tomographic image, it may be possible to confirm to some extent whether the blood vessels extracted on the OCTA tomographic image are appropriate. That is, a more detailed ROI may be set while performing transparency processing between the tomographic images of OCTA data and OCT data.

Further, the second medical image used for transmission processing may be an image from an SLO or a fundus camera. Alternatively, it may be an OCTA en-face image of another layer. In this case, it is desirable that positional registration be performed between the images to be subjected to transparency processing.

Furthermore, the number of images used for transparency processing is not limited to two. Depending on the case, it may be possible to add a third medical image by weighted addition. For example, a first blend image obtained by blending a first medical image and a second medical image using a first transmittance and a third medical image are blended using a second transmittance. The second blended image may be obtained by:

[Second embodiment]
A case will be described in which the image processing apparatus according to the present embodiment performs analysis processing on OCT data while setting an analysis position and an analysis region while referring to an OCTA front image.

From OCT data, it is possible to analyze the thickness of the optic nerve fiber layer, the degree of concavity of the optic disc, and the curvature of the eyeball shape. In this way, various disease states can be known from layer thickness information and eyeball curvature information. In addition, the layer thickness information and curvature information may be displayed as an image by converting the thickness and curvature into a color map as a color gradation, or by dividing the ROI into multiple regions and displaying the average value of each region. Good too.

Alternatively, analyzing the state of the lamina cribrosa is also considered useful for diagnosing glaucoma and the like. Specifically, the thickness of the lamina cribrosa can also be measured by performing appropriate segmentation processing on a tomographic image of OCT data.

Depending on the eye to be examined, effective analysis may be possible by setting the ROI while comparing with more precise blood vessel information. For example, in the case of severe myopia, the shape of the eyeball is distorted, so by transmitting the OCT front image and the OCTA front image at a specified transmittance, it is possible to check the ROI while simultaneously checking blood vessel information. Settings are now possible. Based on the set ROI, layer thickness, curvature, etc. can be analyzed.

Alternatively, if the OCTA front image is subjected to transparency processing on the OCT front image or the analysis result image, the present invention can be used for making complex judgments other than ROI setting.

Specifically, by performing transparency processing on the OCTA front image with respect to the color map image of the layer thickness, the operator can visually recognize, for example, the state of the blood vessel in the region where the layer thickness is low. The same applies to color map images such as curvature. Alternatively, when checking the analysis results of the lamina cribrosa, by adding blood vessel information at a specified transmittance, it is possible to simultaneously check the thickness of the lamina cribrosa and the state of blood flow entering the lamina cribrosa.

In the case of motion contrast data, it is relatively difficult to visually recognize blood vessels in areas with leaky blood vessels or low blood flow. Therefore, if it is desired to perform transmission processing of blood flow information more strictly, an image obtained by a fluorescein ophthalmography examination using fluorescein, indocyanine green, or the like may be used as the second medical image.

Although the analysis of OCT data has been described above in this embodiment, it goes without saying that the analysis method is not limited to this. Furthermore, the first medical image to be subjected to transmission processing is not limited to an OCT frontal image, but may be an image that visualizes the analysis results. As in the first embodiment, tomographic images may be used. Further, the second medical image to be subjected to transparency processing is not limited to the OCTA frontal image, and may be any type of image different from the first medical image. At this time, the first medical image and the second medical image may be images of mutually corresponding regions of the subject.

[Third embodiment]
Regarding the image processing apparatus according to the present embodiment, a case will be described in which the transmittance is adaptively changed for each pixel regarding the transparency processing in the various embodiments described above. For example, when performing transmission processing on an OCTA frontal image and an OCT frontal image, information on blood vessels is important.

Therefore, when performing transparency processing, the method of transparency processing is switched by giving each pixel a classification as a blood vessel region or a non-vascular region in advance as an attribute. Extraction of the blood vessel region is as described in the first embodiment. In the simplest case, pixels in blood vessel regions are given attributes that do not undergo transparency processing, and pixels in non-vascular regions are given attributes that undergo transparency processing. The operator may be able to specify the threshold value for determining the blood vessel attribute on the screen. Attribute information is changed according to the changed threshold, and transparency processing is updated based on the changed attribute. Note that a plurality of threshold values for attribute determination may be specified, and, for example, attributes may be assigned by separating blood vessels and non-blood vessels within the range between the specified threshold values.

By doing so, it becomes possible to perform transparency processing on only pixels in the non-vascular region of the second medical image without performing transparency processing on the blood vessel information. Alternatively, for pixels having the attribute of a blood vessel region, transparency processing may be performed in a manner that suppresses the transmittance of the transparency processing. For example, if the operator sets the transmittance of the second medical image to α, it may be possible to perform transparency processing on non-vascular areas at α, and to suppress vascular areas to α/2. It will be done. This suppression method may be a predetermined ratio, or a function based on the transmittance α may be prepared.

Further, the vascular region or the non-vascular region may be separately held as a continuous value with respect to the measure of vascularity. In this case, for example, the maximum transmittance of the blood vessel attribute is set in advance relative to the maximum transmittance specified by the operator, and the transmittance is determined based on the value of blood vessel-likeness relative to the transmittance specified by the operator. You may. The transparency processing method is not limited to this, and various modifications are possible as long as it is based on attribute information for each pixel.

Furthermore, multiple attributes may be held. In the case of an OCTA frontal image, blood vessel attributes are managed by dividing the OCTA data into at least two ranges in the depth direction, for example, a shallow part and a deep part. The operator may be able to instantly switch which attribute to use using instructions on the GUI.

Note that in the above description, it has been shown that attributes are given to pixels based on blood vessel regions and non-vascular regions, but the present invention is not limited to this, and various attributes can be applied. For example, pixels having a specific signal value (for example, 0) in the second medical image may be given an attribute that does not undergo transparency processing. Alternatively, the attribute may be given to a preset partial area. For example, by manually specifying areas where bleeding is recognized on the GUI, attributes based on bleeding areas and non-bleeding areas may be given to each pixel.

Furthermore, since blood vessel depth information can be obtained from the OCTA data, attribute values may be set based on the blood vessel depth information. If blood vessels overlap when the motion contrast front image is obtained, it may be determined in advance that the maximum value, minimum value, or average value may be used. Further, since the OCT data includes layer thickness information, attributes based on the layer thickness may be set.

Note that the attribute information may be provided to each of the first medical image and the second medical image, or may be provided to only one of them. Further, the transparent processing is not limited to the method described above, and those skilled in the art can make various modifications as processing based on attribute information set on at least one side.

In each of the above embodiments, medical image processing of the eye has been explained as an example, but medical image data obtained by optical coherence tomography (for example, motion contrast data of skin tissue) can also be processed. Applicable.

[Fourth embodiment]
A medical image processing apparatus according to a fourth embodiment will be described below with reference to FIGS. 6, 7, and 8. The image processing device 101 according to the present embodiment is configured to apply image quality improvement processing based on machine learning, for example, instead of the above-mentioned synthesis unit 101-42, as an image quality improvement means for improving the image quality of motion contrast data. It includes a high image quality unit (not shown). At this time, the image quality improvement unit in the image processing device 101 (or image processing unit 101-04) is equipped with an image quality improvement engine. The image quality improvement method included in the image quality improvement engine according to this embodiment performs processing using a machine learning algorithm.

In this embodiment, for training a machine learning model related to a machine learning algorithm, input data is a low-quality image with specific shooting conditions assumed to be processed, and output data is a high-quality image corresponding to the input data. We use training data consisting of a group of pairs. Note that the specific imaging conditions specifically include a predetermined imaging site, imaging method, imaging angle of view, image size, and the like.

Here, the machine learning model is a model in which training (learning) is performed on an arbitrary machine learning algorithm using appropriate teacher data (learning data) in advance. The teaching data consists of one or more pairs of input data and output data (correct data). Note that the formats and combinations of the input data and output data of the pair groups that make up the training data may be such that one is an image and the other is a numerical value, one is composed of multiple image groups and the other is a character string, or both. may be an image, etc., which is suitable for the desired configuration.

Specifically, for example, teacher data (hereinafter referred to as first teacher data) configured by a pair group of an image acquired by OCT and a radiographed region label corresponding to the image can be mentioned. Note that the imaging site label is a unique numerical value or character string that represents the site. In addition, as an example of other training data, it is made up of a pair group of a noisy low-quality image obtained by normal OCT imaging and a high-quality image obtained by capturing multiple times using OCT and processing it to improve the image quality. Examples include teacher data (hereinafter referred to as second teacher data).

At this time, when input data is input to the machine learning model, output data according to the design of the machine learning model is output. A machine learning model outputs output data that is likely to correspond to input data, for example, according to trends trained using teacher data. Further, the machine learning model can, for example, output the possibility of corresponding to input data as a numerical value for each type of output data, according to trends trained using teacher data. Specifically, for example, when an image acquired by OCT is input to a machine learning model trained with the first teacher data, the machine learning model outputs the imaged region label of the imaged region captured in the image. or output the probability for each imaging site label. Also, for example, if a noisy, low-quality image obtained by normal OCT imaging is input to a machine learning model trained with the second teacher data, the machine learning model will take multiple images using OCT to improve the image quality. Outputs a high-quality image equivalent to the processed image. Note that, from the viewpoint of maintaining quality, the machine learning model can be configured not to use its own output data as training data.

Furthermore, machine learning algorithms include techniques related to deep learning such as convolutional neural networks (CNN). In methods related to deep learning, if the parameter settings for the layers and nodes that make up the neural network differ, the extent to which trends trained using teacher data can be reproduced in output data may vary. For example, in a deep learning machine learning model using first training data, if more appropriate parameters are set, the probability of outputting a correct imaged region label may become higher. Further, for example, in a deep learning machine learning model using the second teacher data, if more appropriate parameters are set, a higher quality image may be output in some cases.

Specifically, parameters in CNN include, for example, the kernel size of the filter, the number of filters, the stride value, and the dilation value set for the convolutional layer, as well as the number of output nodes of the fully connected layer. etc. can be included. Note that the parameter group and the number of training epochs can be set to values preferable for the usage pattern of the machine learning model based on the teacher data. For example, based on the training data, it is possible to set a parameter group and the number of epochs that can output correct imaging site labels with a higher probability or output higher quality images.

An example of a method for determining such a parameter group and the number of epochs will be described. First, 70% of the pairs forming the teacher data are used for training, and the remaining 30% are randomly set for evaluation. Next, a machine learning model is trained using the training pair group, and at the end of each epoch of training, a training evaluation value is calculated using the evaluation pair group. The training evaluation value is, for example, the average value of a group of values obtained by evaluating the output when the input data forming each pair is input to the machine learning model being trained and the output data corresponding to the input data using a loss function. be. Finally, the parameter group and epoch number when the training evaluation value becomes the smallest are determined as the parameter group and epoch number of the machine learning model. Note that by determining the number of epochs by dividing the pairs that make up the teaching data into those for training and those for evaluation, the machine learning model may overfit the pairs for training. can be prevented.

Further, the image quality improvement engine (trained model for image quality improvement) is a module that outputs a high quality image obtained by enhancing the image quality of an input low quality image. Here, high image quality in this specification refers to converting an input image into an image with a quality more suitable for image diagnosis. refers to an image that In addition, low-quality images include, for example, two-dimensional images or three-dimensional images obtained by X-ray photography, CT, MRI, OCT, PET, or SPECT, or three-dimensional moving images of continuously captured CT, etc. This photo was taken without any high-quality settings. Specifically, low-quality images include, for example, images obtained by low-dose imaging using an X-ray imaging device or CT, imaging by MRI that does not use a contrast agent, short-time imaging with OCT, etc. Contains OCTA images etc. acquired several times.

Further, the content of image quality suitable for image diagnosis depends on what is desired to be diagnosed by various image diagnoses. Therefore, it is difficult to make a general statement, but for example, image quality suitable for image diagnosis is one that has little noise, high contrast, shows the photographed subject in colors and gradations that are easy to observe, and has a large image size. Including image quality such as high resolution. Furthermore, the image quality can include images in which non-existent objects and gradations that were drawn during the image generation process have been removed from the image.

In addition, when high-quality images with low noise or high contrast are used for image analysis such as blood vessel analysis processing of images such as OCTA, and region segmentation processing of images such as CT and OCT, low-quality images can be used. In many cases, analysis can be performed more accurately than when using Therefore, high-quality images output by the high-quality engine may be useful not only for image diagnosis but also for image analysis.

The image processing method that constitutes the image quality improvement method in this embodiment performs processing using various machine learning algorithms such as deep learning. In addition to processing using machine learning algorithms, this image processing method uses existing arbitrary methods such as various image filter processing, matching processing using a database of high-quality images corresponding to similar images, and knowledge-based image processing. You may also perform the following processing.

In particular, an example of the configuration of a CNN that improves the quality of a two-dimensional image is the configuration shown in FIG. The configuration of the CNN includes a plurality of convolution processing blocks 100 groups. The convolution processing block 100 includes a convolution layer 101, a batch normalization layer 102, and an activation layer 103 using a normalized linear function (Rectifier Linear Unit). Further, the configuration of the CNN includes a merging layer 104 and a final convolution layer 105. The composition layer 104 combines the output value group of the convolution processing block 100 and the pixel value group constituting the image by connecting or adding them together. The final convolution layer 105 outputs a group of pixel values that are synthesized in the synthesis layer 104 and constitute the high-quality image Im120. In such a configuration, the pixel value group forming the input image Im110 is outputted through the convolution processing block 100 group, and the pixel value group forming the input image Im110 is combined in the synthesis layer 104. be synthesized. Thereafter, the combined pixel value group is formed into a high-quality image Im120 in the final convolution layer 105.
For example, by setting the number of convolution processing blocks 100 to 16, setting the kernel size of the filter to 3 pixels in width and 3 pixels in height, and the number of filters to 64 as parameters of the convolution layer 101 group, a certain high image quality can be achieved. You can get the effect of However, in reality, as described in the explanation of the machine learning model above, a better parameter group can be set using training data that corresponds to the usage pattern of the machine learning model. Note that if it is necessary to process a three-dimensional image or a four-dimensional image, the kernel size of the filter may be expanded to three-dimensional or four-dimensional.

Further, another example of the configuration of the CNN in the image quality improvement unit according to the present embodiment will be described using FIG. 15. FIG. 15 shows an example of a machine learning model configuration in the image quality improvement section. The configuration shown in FIG. 15 is composed of a plurality of layer groups that perform processing to process and output a group of input values. As shown in FIG. 15, the types of layers included in the configuration include a convolution layer, a downsampling layer, an upsampling layer, and a merge layer. The convolution layer is a layer that performs convolution processing on a group of input values according to parameters such as the set filter kernel size, the number of filters, the stride value, and the dilation value. Note that the number of dimensions of the kernel size of the filter may also be changed depending on the number of dimensions of the input image. The downsampling layer is a process of thinning out or combining input value groups to make the number of output value groups smaller than the number of input value groups. Specifically, for example, there is Max Pooling processing. The upsampling layer is a process of making the number of output value groups larger than the number of input value groups by duplicating the input value group or adding a value interpolated from the input value group. Specifically, for example, there is linear interpolation processing. The compositing layer is a layer that receives a group of values, such as a group of output values of a certain layer or a group of pixel values constituting an image, from a plurality of sources, and performs a process of combining them by concatenating or adding them. In such a configuration, the group of pixel values forming the input image Im2410 is outputted through the convolution processing block, and the group of pixel values forming the input image Im2410 are combined in the composition layer. Thereafter, the combined pixel value group is formed into a high-quality image Im2420 in the final convolution layer. Although not shown, examples of changes to the CNN configuration include incorporating a batch normalization layer or an activation layer using a rectifier linear unit after the convolution layer. You may.

Here, the GPU can perform efficient calculations by processing more data in parallel. For this reason, when learning is performed multiple times using a learning model such as deep learning, it is effective to perform processing on the GPU. Therefore, in this embodiment, a GPU is used in addition to a CPU for processing by the image processing unit 101-04, which is an example of a learning unit (not shown). Specifically, when a learning program including a learning model is executed, learning is performed by the CPU and GPU working together to perform calculations. Note that the processing of the learning section may be performed by only the CPU or GPU. Further, the image quality improvement section may also use a GPU like the learning section. Further, the learning section may include an error detection section and an updating section (not shown). The error detection unit obtains an error between output data output from the output layer of the neural network and correct data according to input data input to the input layer. The error detection unit may use a loss function to calculate the error between the output data from the neural network and the correct data. Furthermore, the updating section updates the connection weighting coefficients between the nodes of the neural network, etc., based on the error obtained by the error detection section, so that the error becomes smaller. This updating unit updates the connection weighting coefficients and the like using, for example, an error backpropagation method. The error backpropagation method is a method of adjusting connection weighting coefficients between nodes of each neural network so that the above-mentioned error is reduced.

Note that when using some image processing techniques, such as image processing using CNN, it is necessary to be careful about the image size. Specifically, in order to prevent problems such as the peripheral areas of high-quality images not being sufficiently high-quality, it should be noted that different image sizes may be required for input low-quality images and output high-quality images. Should.

Although not specified in this embodiment for the sake of clear explanation, if a high image quality engine is adopted that requires different image sizes for the input image to the high image quality engine and the output image, the image size may be changed as appropriate. It is assumed that the Specifically, we perform padding on input images, such as images used as teacher data for training machine learning models and images input to high-quality engines, and image capturing areas around the input images. Adjust the image size by merging. Note that the area to be padded is filled with a constant pixel value, filled with neighboring pixel values, or mirror padded, depending on the characteristics of the image quality improvement method, so that the image quality can be effectively improved.

Further, the image quality improvement method may be implemented using only one image processing method, or may be implemented by combining two or more image processing methods. Furthermore, after performing a plurality of image quality improvement method groups in parallel to generate a plurality of high-quality image groups, the highest-quality image may be finally selected as the high-quality image. Note that the selection of the highest quality image may be automatically performed using an image quality evaluation index, or a plurality of groups of high quality images may be displayed on a user interface provided on an arbitrary display unit, etc. It may also be performed according to instructions from the examiner (user).

Note that input images that have not been improved in quality may be more suitable for image diagnosis, and therefore may be added to the final image selection target. Furthermore, parameters may be input to the high-quality image together with the low-quality image. For example, a parameter specifying the degree of image quality enhancement or a parameter specifying the image filter size used in the image processing method may be input to the image quality enhancement engine together with the input image.

Here, in this embodiment, the input data of the teacher data is a low-quality image acquired with the same model as the tomographic imaging apparatus 100 and the same settings as the tomographic imaging apparatus 100. Further, the output data of the teacher data is a high-quality image obtained by the settings and image processing of the same model as the tomographic image. Specifically, the output data is, for example, a high-quality image (superimposed image) obtained by performing superimposition processing such as averaging on a group of images (original images) obtained by shooting multiple times. be. Here, high-quality images and low-quality images will be explained using OCTA motion contrast data as an example. Here, the motion contrast data is data used in OCTA and the like, which is obtained by repeatedly photographing the same part of the photographic subject and detecting temporal changes in the photographic subject between the photographs. At this time, from among the calculated motion contrast data (an example of three-dimensional medical image data), a front image is generated using data in a desired range in the depth direction of the object to be imaged, thereby generating an OCTA En-Face image. (motion contrast frontal image) can be generated. Note that, hereinafter, repeatedly capturing OCT data at the same location will be referred to as NOR (Number of Repeats).

In this embodiment, two different methods will be described as examples of generating a high-quality image and a low-quality image by overlapping processing with reference to FIG. 12.

The first method will be described with reference to FIG. 12A regarding motion contrast data generated from OCT data obtained by repeatedly photographing the same location of a photographic target as an example of a high-quality image. In FIG. 12A, Im2810 indicates three-dimensional motion contrast data, and Im2811 indicates two-dimensional motion contrast data constituting the three-dimensional motion contrast data. Im2811-1 to Im2811-3 indicate OCT tomographic images (B scan) for generating Im2811. Here, NOR refers to the number of OCT tomographic images in Im2811-1 to Im2811-3 in FIG. 12(a), and in the example shown in the figure, NOR is 3. Im2811-1 to Im2811-3 are photographed at predetermined time intervals (Δt). Note that the same location refers to one line in the front direction (XY) of the subject's eye, and corresponds to the location Im2811 in FIG. 12(a). Note that the front direction is an example of a direction intersecting the depth direction. Since motion contrast data is data that detects temporal changes, NOR must be performed at least twice in order to generate this data. For example, if NOR is 2, one motion contrast data is generated. When NOR is 3, two pieces of data are generated when motion contrast data is generated only by OCT in adjacent time intervals (first and second, second and third). When motion contrast data is generated using OCT data obtained at separate time intervals (first and third times), a total of three pieces of data are generated. That is, as the NOR is increased 3 times, 4 times, etc., the number of motion contrast data at the same location also increases. High-quality motion contrast data can be generated by aligning a plurality of pieces of motion contrast data obtained by repeatedly photographing the same location and performing superimposition processing such as averaging. Therefore, NOR is performed at least 3 times or more, preferably 5 times or more. On the other hand, an example of a low-quality image corresponding to this is motion contrast data before superimposition processing such as averaging is performed. In this case, it is desirable to use the low-quality image as a reference image when performing overlay processing such as averaging. When performing overlay processing, if the position and shape of the target image are transformed and aligned with respect to the reference image, there will be almost no spatial misalignment between the reference image and the image after overlay processing. . Therefore, it is possible to easily create a pair of a low-quality image and a high-quality image. Note that the target image subjected to image deformation processing for alignment may be used as a low-quality image instead of the reference image. By using each of the original image groups (reference image and target image) as input data and the corresponding superimposed image as output data, a plurality of pair groups can be generated. For example, when obtaining one superimposed image from a group of 15 original images, a pair of the first original image of the original image group and the superimposed image, a pair of the second original image of the original image group, Pairs with superimposed images can be generated. In this way, when one superimposed image is obtained from a group of 15 original images, it is possible to generate 15 pairs of one image from the original image group and the superimposed image. Note that three-dimensional high-quality image data can be generated by repeatedly photographing the same location in the main scanning (X) direction and scanning while shifting it in the sub-scanning (Y) direction.

Further, the second method will be described with reference to FIG. 12(b) regarding a process of generating a high-quality image by superimposing motion contrast data obtained by photographing the same area of the photographing target multiple times. Note that the same area refers to an area such as 3 x 3 mm or 10 x 10 mm in the front direction (XY) of the subject's eye, and three-dimensional motion contrast data including the depth direction of the tomographic image is collected. means to obtain. When performing superimposition processing by photographing the same region multiple times, it is desirable to set NOR to two or three times in order to shorten the number of times each photograph is taken. Furthermore, in order to generate high-quality three-dimensional motion contrast data, at least two pieces of three-dimensional data of the same area are acquired. FIG. 12(b) shows an example of a plurality of three-dimensional motion contrast data. Im2820 to Im2840 are three-dimensional motion contrast data as described in FIG. 12(a). Using these two or more three-dimensional motion contrast data, alignment processing is performed in the front direction (X-Y) and depth direction (Z), and after removing data that would be artifacts from each data, averaging processing is performed. I do. As a result, it is possible to generate one piece of high-quality three-dimensional motion contrast data from which artifacts are excluded. A high-quality image can be created by generating an arbitrary plane from 3D motion contrast data. On the other hand, it is preferable that the corresponding low-quality image be an arbitrary plane generated from reference data when superimposing processing such as averaging is performed. As explained in the first method, since there is almost no spatial positional shift between the reference image and the image after averaging, they can easily be paired as a low-quality image and a high-quality image. Note that an arbitrary plane generated from target data subjected to image deformation processing for alignment instead of reference data may be used as a low-quality image.

In the first method, the burden on the subject is low because the imaging itself is completed in one time. However, as the number of NOR operations increases, the time required for one imaging session becomes longer. Furthermore, if artifacts such as cloudy eyes or eyelashes appear during the shooting process, a good image may not necessarily be obtained. In the second method, the burden on the subject increases slightly because the images are taken multiple times. However, it takes less time to take one shot, and even if an artifact appears in one shot, as long as the artifact does not appear in another shot, you can ultimately obtain a clear image with fewer artifacts. Considering these characteristics, when collecting data, select any method that suits the subject's situation.

Although the present embodiment has been described using motion contrast data as an example, the present invention is not limited to this. Since OCT data is captured in order to generate motion contrast data, the above method can also be used for OCT data. Furthermore, although a description of the tracking process has been omitted in this embodiment, since the same location or area of the eye to be examined is photographed, it is desirable to perform imaging while tracking the eye to be examined.

In this embodiment, since a pair of three-dimensional high image quality data and low image quality data is created, any pair of two-dimensional images can be generated from this pair. This will be explained using FIG. 13. For example, when the target image is an OCTA En-Face image, the OCTA En-Face image is generated in a desired depth range from the three-dimensional data. The desired depth range refers to the range in the Z direction in FIG. 12. An example of the OCTA En-Face image generated here is shown in FIG. 13(a). Learning is performed using OCTA En-Face images generated in different depth ranges, such as superficial layer (Im2910), deep layer (Im2920), outer layer (Im2930), and choroidal vascular network (Im2940). . Note that the types of OCTA En-Face images are not limited to these, and the number of types may be increased by generating OCTA En-Face images in which different depth ranges are set by changing the reference layer and offset values. When performing learning, you can train each OCTA En-Face image at different depths separately, or you can train by combining multiple images with different depth ranges (for example, separating them into surface layer and deep layer). Alternatively, OCTA En-Face images of all depth ranges may be trained together. In the case of luminance En-Face images generated from OCT data, learning is performed using a plurality of En-Face images generated from arbitrary depth ranges, similar to OCTA En-Face. For example, consider a case where the image quality improvement engine includes a machine learning engine obtained using learning data including a plurality of motion contrast frontal images corresponding to different depth ranges of the eye to be examined. At this time, the acquisition unit can acquire, as the first image, a motion contrast frontal image corresponding to a part of the long depth range including different depth ranges. That is, a motion contrast front image corresponding to a depth range different from the plurality of depth ranges corresponding to the plurality of motion contrast front images included in the learning data can be used as an input image at the time of image quality improvement. Of course, a motion contrast front image with the same depth range as that during learning may be used as an input image during image quality enhancement. Further, some of the depth ranges may be set in response to the examiner pressing an arbitrary button on the user interface, or may be set automatically. Note that the above-mentioned content is not limited to motion contrast front images, but can also be applied to, for example, brightness En-Face images.

Note that when the image to be processed is a tomographic image, learning is performed using an OCT tomographic image that is a B scan or a tomographic image that is motion contrast data. This will be explained using FIG. 13(b). In FIG. 13(b), Im2951 to Im2953 are OCT tomographic images. The images in FIG. 13(b) are different because they show tomographic images at different positions in the sub-scanning (Y) direction. For tomographic images, learning may be performed together without worrying about the difference in position in the sub-scanning direction. However, in the case of images taken at different locations (for example, the center of the macula and the center of the optic disc), it is possible to perform learning for each region separately, or to do so without worrying about the location of the image. You may also choose to study together. Note that since the image feature amounts are significantly different between an OCT tomographic image and a tomographic image of motion contrast data, it is better to perform learning separately.

In the superimposed image that has been subjected to the superimposition process, pixels that are commonly depicted in the original image group are emphasized, resulting in a high-quality image suitable for image diagnosis. In this case, the generated high-quality image is a high-contrast image with clear differences between low-brightness areas and high-brightness areas, as a result of highlighting commonly drawn pixels. Furthermore, for example, in the superimposed images, random noise that occurs every time a photograph is taken can be reduced, and an area that was not well depicted in the original image at a certain point in time can be interpolated with another group of original images.

Furthermore, when input data for a machine learning model needs to be composed of a plurality of images, a necessary number of original image groups can be selected from the original image group and used as input data. For example, when obtaining one superimposed image from a group of 15 original images, if two images are required as input data for a machine learning model, it is possible to generate 105 (15C2=105) pair groups.

Note that among the pairs that constitute the teacher data, pairs that do not contribute to higher image quality can be removed from the teacher data. For example, if a high-quality image that is the output data that makes up a pair of training data is of an image quality that is not suitable for image diagnosis, the image that is output by the high-quality image trained using the training data is also not suitable for image diagnosis. This may result in poor image quality. Therefore, by removing pairs whose output data has an image quality unsuitable for image diagnosis from the teacher data, it is possible to reduce the possibility that the image quality improvement engine generates an image with an image quality unsuitable for image diagnosis.

In addition, if the average brightness or brightness distribution of a group of images in a pair is significantly different, the image quality improvement engine trained using the training data will be able to detect images that are not suitable for image diagnosis and have brightness distributions that are significantly different from those of low-quality images. may be output. Therefore, pairs of input data and output data with significantly different average brightness or brightness distribution can be removed from the teacher data.

Furthermore, if the structure or position of the object drawn in a pair of images is significantly different, the high-quality engine trained using the training data will draw the object in a structure or position that is significantly different from that of the low-quality image. There is a possibility of outputting an image that is not suitable for image diagnosis. Therefore, pairs of input data and output data in which the structure or position of the photographic object to be drawn differs greatly can be removed from the teacher data. Furthermore, from the viewpoint of quality maintenance, the high-quality image engine can be configured not to use the high-quality images that it outputs as training data.

By using the high image quality engine that performs machine learning in this way, the high image quality unit in the image processing device 101 (or the image processing unit 101-04) can input medical images acquired in one shooting. In this case, it is possible to output a high-quality image in which contrast has been increased, noise has been reduced, etc. through superimposition processing. Therefore, the image quality improvement unit can generate a high quality image suitable for image diagnosis based on the low quality image that is the input image.

Next, a series of image processing according to this embodiment will be described with reference to the flowchart in FIG. FIG. 7 is a flowchart of a series of image processing according to this embodiment. First, when a series of image processing according to this embodiment is started, the process moves to step S510.

In step S510, the image acquisition unit 101-01 acquires an image photographed by the tomographic imaging apparatus 100 as an input image from the tomographic imaging apparatus 100 connected via a circuit or network. Note that the image acquisition unit 101-01 may acquire input images in response to a request from the tomographic imaging apparatus 100. Such a request may be made, for example, when the tomographic imaging apparatus 100 generates an image, before or after the image generated by the tomographic imaging apparatus 100 is saved in a storage device included in the tomographic imaging apparatus 100. It may be issued when displaying a captured image on the display unit 104, when using a high-quality image for image analysis processing, etc.

Note that the image acquisition unit 101-01 may acquire data for generating an image from the tomographic imaging apparatus 100, and acquire an image generated by the image processing apparatus 101 based on the data as an input image. In this case, any existing image generation method may be employed as an image generation method for the image processing apparatus 101 to generate various images.

In step S520, the imaging condition acquisition unit (not shown) in the image processing apparatus 101 acquires a group of imaging conditions for the input image. Specifically, a group of imaging conditions stored in a data structure constituting the input image is acquired according to the data format of the input image. Note that, as described above, if the imaging conditions are not saved in the input image, the imaging condition acquisition unit acquires the imaging information group including the imaging conditions group from the tomographic imaging apparatus 100 or the image management system (not shown). can do.

In step S530, the high image quality determination unit (not shown) in the image processing device 101 uses the acquired imaging condition group to It is determined whether the image quality of the input image can be increased using the image quality improvement engine provided in the image quality improvement engine. Specifically, the image quality improvement determination unit determines whether or not the imaging site, imaging method, imaging angle of view, and image size of the input image match conditions that can be handled by the image quality improvement engine.

The high image quality determination unit determines all of the photographing conditions, and if it is determined that they can be handled, the process moves to step S540. On the other hand, if the image quality improvement determination unit determines that the image quality improvement engine cannot handle the input image based on these imaging conditions, the process moves to step S550.

Note that depending on the settings and implementation form of the image processing apparatus 101, even if it is determined that the input image cannot be processed based on part of the imaging site, imaging method, imaging angle of view, and image size, Image quality enhancement processing in step S540 may be implemented. For example, it is assumed that the high-quality image engine can comprehensively handle any imaged body part of the patient, and even if the input data includes an unknown body part. Such processing may be performed if the system is implemented in In addition, the high image quality determination unit determines, depending on the desired configuration, that at least one of the imaging site, imaging method, imaging angle of view, and image size of the input image matches a condition that can be handled by the high image quality engine. It may be determined whether or not to do so.

In step S540, the image quality improvement unit uses the image quality improvement engine to improve the image quality of the input image and generates a high quality image that is more suitable for image diagnosis than the input image. Specifically, the image quality improvement unit inputs the input image to an image quality improvement engine, and generates an improved high quality image. The high image quality engine generates a high quality image that looks like it has undergone superposition processing using input images, based on a machine learning model that has been subjected to machine learning using teacher data. Therefore, the high-quality image can generate a high-quality image with reduced noise or enhanced contrast compared to the input image.

Note that depending on the settings and implementation form of the image processing device 101, the image quality improvement unit inputs parameters along with the input image to the image quality improvement engine to adjust the degree of image quality improvement, etc., depending on the shooting condition group. It's okay. Further, the image quality improvement unit may adjust the degree of image quality improvement, etc. by inputting parameters according to input by the examiner to the image quality improvement engine together with the input image.

In step S550, if a high-quality image has been generated in step S540, the display control unit 101-05 outputs the high-quality image and displays it on the display unit 104. On the other hand, if it is determined in step S530 that high image quality processing is not possible, the input image is output and displayed on the display unit 104. Note that instead of displaying the output image on the display unit 104, the display control unit 101-05 may cause the tomographic imaging apparatus 100 or another device to display or store the output image. Furthermore, depending on the settings and implementation form of the image processing device 101, the display control unit 101-05 may process the output image so that it can be used by the tomographic imaging device 100 or other devices, or send it to an image management system or the like. The data format may be converted as necessary.

As described above, the image processing device 101 according to this embodiment includes an image acquisition section 101-01 and an image quality improvement section. The image acquisition unit 101-01 acquires an input image (first image) that is an image of a predetermined region of the subject. The image quality improvement unit uses a high image quality engine including a machine learning engine to generate a high quality image (second image) that has been subjected to at least one of noise reduction and contrast enhancement from the input image compared to the input image. generate. The image quality improvement engine includes a machine learning engine that uses images obtained through superimposition processing as learning data.

With this configuration, the image processing apparatus 101 according to the present embodiment can output a high-quality image with reduced noise and enhanced contrast from an input image. For this reason, the image processing device 101 can produce images suitable for image diagnosis, such as clearer images or images in which the site or lesion to be observed is emphasized, with less invasiveness for the photographer or patient than in the past. It can be obtained at a lower cost without increasing the price or increasing the effort.

The image processing apparatus 101 further includes a high-quality image determination unit that determines whether or not a high-quality image can be generated using the high-quality engine for the input image. The image quality improvement determination unit makes the determination based on at least one of the imaging site, imaging method, imaging angle of view, and image size of the input image.

With this configuration, the image processing device 101 according to the present embodiment can omit input images that cannot be processed by the image quality enhancement unit from the image quality enhancement process, thereby reducing the processing load on the image processing device 101 and the occurrence of errors. I can do it.

Note that in this embodiment, the display control unit 101-05 is configured to display the generated high-quality image on the display unit 104, but the operation of the display control unit 101-05 is not limited to this. For example, the display control unit 101-05 can also output high-quality images to other devices connected to the tomographic imaging apparatus 100 and the image processing apparatus 101. High quality images can thus be displayed on the user interface of these devices, stored in any storage device, used for any image analysis, or sent to an image management system.

In this embodiment, the image quality improvement possibility determination unit determines whether or not the input image can be improved in image quality by the image quality improvement engine, and if the input image can be improved in image quality, the image quality improvement unit has improved image quality. On the other hand, if the tomographic imaging apparatus 100 performs imaging only under imaging conditions that allow for high image quality, the images acquired from the tomographic imaging apparatus 100 may be unconditionally improved in image quality. In this case, as shown in FIG. 8, steps S520 and S530 can be omitted, and step S540 can be performed after step S510.

Note that in this embodiment, the display control unit 101-05 is configured to cause the display unit 104 to display a high-quality image. However, the display control unit 101-05 may display a high-quality image on the display unit 104 in response to an instruction from the examiner. For example, the display control unit 101-05 may display a high-quality image on the display unit 104 in response to the examiner pressing an arbitrary button on the user interface of the display unit 104. In this case, the display control unit 101-05 may display the high quality image by switching with the input image, or may display the high quality image alongside the input image.

Furthermore, when displaying a high-quality image on the display unit 104, the display control unit 101-05 displays a display indicating that the displayed image is a high-quality image generated by processing using a machine learning algorithm. It may also be displayed together with a high-quality image. In this case, the display allows the user to easily identify that the displayed high-quality image is not the image obtained by photography, which reduces misdiagnosis and improves diagnostic efficiency. I can do it. Note that the display indicating that the image is a high-quality image generated by processing using a machine learning algorithm may take any form as long as it is possible to distinguish between the input image and the high-quality image generated by the processing. It can be anything.

In addition, the display control unit 101-05 determines what kind of training data the machine learning algorithm used to perform learning regarding the display indicating that the image is a high-quality image generated by processing using the machine learning algorithm. The display unit 104 may display a display indicating the following. The display may include an explanation of the types of input data and output data of the teacher data, and any display related to the teacher data, such as an imaged body part included in the input data and output data.

In the image quality improvement engine according to the present embodiment, a superimposed image is used as the output data of the teacher data, but the teacher data is not limited to this. As the output data of the teacher data, a high-quality image obtained by performing at least one of the overlay processing, the processing group described below, and the photographing method described below, which are means for obtaining a high-quality image, may be used. .

For example, a high-quality image obtained by performing maximum a posteriori probability estimation processing (MAP estimation processing) on a group of original images may be used as the output data of the teacher data. In the MAP estimation process, a likelihood function is determined from the probability density of each pixel value in a plurality of low-quality images, and the true signal value (pixel value) is estimated using the determined likelihood function.

A high-quality image obtained by the MAP estimation process is a high-contrast image based on pixel values close to true signal values. Further, since the estimated signal value is obtained based on the probability density, randomly generated noise is reduced in a high-quality image obtained by the MAP estimation process. Therefore, by using the high-quality images obtained through MAP estimation processing as training data, the high-quality image processing engine can generate high-quality images suitable for image diagnosis, such as reduced noise and high contrast, from the input images. It is possible to generate high-quality images. Note that the method for generating pairs of input data and output data of the teaching data may be performed in the same manner as in the case where superimposed images are used as the teaching data.

Furthermore, a high-quality image obtained by applying smoothing filter processing to the original image may be used as the output data of the teacher data. In this case, the high-quality image can generate a high-quality image with reduced random noise from the input image. Furthermore, an image obtained by applying gradation conversion processing to the original image may be used as the output data of the teacher data. In this case, the high-quality image can generate a high-quality image with enhanced contrast from the input image. Note that the method for generating pairs of input data and output data of the teaching data may be performed in the same manner as in the case where superimposed images are used as the teaching data.

Note that the input data of the teacher data may be an image acquired from an image capturing apparatus having the same image quality tendency as the tomographic image capturing apparatus 100. Further, the output data of the teaching data may be a high-quality image obtained by expensive processing such as a successive approximation method, or the subject corresponding to the input data may be The image may be a high-quality image obtained by photographing with a high-performance photographing device. Furthermore, the output data may be a high-quality image obtained by performing rule-based noise reduction processing. Here, the noise reduction process can include, for example, processing such as replacing only one high-brightness pixel that is clearly noise that appears in a low-brightness area with the average value of nearby low-brightness pixel values. . For this reason, the high image quality engine uses images captured by a capturing device with higher performance than the capturing device used to capture the input image, or images acquired in a capturing process that requires more man-hours than the capturing process of the input image. It may also be used as learning data. For example, when a motion contrast frontal image is used as an input image, the high image quality engine uses an image obtained by OCTA imaging using an OCT imaging device with higher performance than the OCT imaging device used for OCTA imaging of the input image, or an input image. The learning data may be an image obtained by an OCTA imaging process that requires more man-hours than the OCTA imaging process.

Although omitted in the description of this embodiment, a high-quality image generated from a plurality of images, which is used as output data of teacher data, can be generated from a plurality of aligned images. The alignment process includes, for example, selecting one of a plurality of images as a template, determining the degree of similarity with other images while changing the position and angle of the template, and determining the amount of positional deviation from the template. Each image may be corrected based on the amount of positional deviation. Further, any other existing alignment processing may be performed.

In addition, when aligning 3D images, the 3D images are aligned by dividing the 3D image into multiple 2D images and integrating the aligned images for each 2D image. Good too. Alternatively, the two-dimensional images may be aligned by decomposing the two-dimensional images into one-dimensional images and integrating the aligned images for each one-dimensional image. Note that these alignments may be performed on data for generating an image instead of the image.

Further, in this embodiment, if the image quality improvement possibility determination unit determines that the input image can be handled by the image quality improvement unit, the process moves to step S540, and the image quality improvement process by the image quality improvement unit starts. It was done. In contrast, the display control unit 101-05 displays the determination result by the high image quality determination unit on the display unit 104, and even if the high image quality unit starts high image quality processing in response to instructions from the examiner. good. At this time, the display control unit 101-05 can cause the display unit 104 to display the input image and the imaging conditions such as the imaging site acquired for the input image, along with the determination result. In this case, the image quality improvement process is performed after the examiner determines whether the determination result is correct or not, so it is possible to reduce the image quality improvement process based on erroneous determinations.

Alternatively, the display control unit 101-05 displays the input image and the imaging conditions such as the body part acquired for the input image on the display unit 104 without making a determination by the high image quality determination unit. The image quality improvement process may be started in response to an instruction from the user.

[Fifth embodiment]
Next, with reference to FIG. 14, an image processing apparatus according to a fifth embodiment will be described. In this embodiment, an example will be described in which the display control unit 101-05 displays the processing result of the image quality improvement unit in the image processing device 101 (or the image processing unit 101-04) on the display unit 104. Note that although the present embodiment will be described using FIG. 14, the display screen is not limited to this. The image quality improvement process can similarly be applied to a display screen that displays a plurality of images taken at different dates and times side by side, such as during follow-up observation. Furthermore, the image quality enhancement process can be similarly applied to a display screen, such as an imaging confirmation screen, on which the examiner confirms the success or failure of imaging immediately after imaging.

Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 101 according to the first embodiment and the like. Therefore, the image processing apparatus according to the present embodiment will be described below, focusing on the differences from the image processing apparatus according to the first embodiment and the like.

The display control unit 101-05 can cause the display unit 104 to display a plurality of high-quality images generated by the image-quality enhancement unit and a low-quality image that has not been enhanced. Thereby, it is possible to output a low-quality image and a high-quality image, respectively, according to the examiner's instructions.

An example of the interface 3400 will be shown below with reference to FIG. 3400 represents the entire screen, 3401 represents the patient tab, 3402 represents the imaging tab, 3403 represents the report tab, and 3404 represents the settings tab, and the diagonal line in the report tab 3403 represents the active state of the report screen. In this embodiment, an example of displaying a report screen will be described. Im3405 is an SLO image, and Im3406 is an OCTA En-Face image shown in Im3407, which is displayed superimposed on the SLO image Im3405. Here, the SLO image is a frontal image of the fundus obtained by an SLO (Scanning Laser Ophthalmoscope) optical system (not shown). Im3407 and Im3408 are OCTA En-Face images, Im3409 is a luminance En-Face image, and Im3411 and Im3412 are tomographic images. 3413 and 3414 display the upper and lower boundaries of the OCTA En-Face images shown in Im3407 and Im3408, respectively, superimposed on the tomographic image. The button 3420 is a button for specifying execution of image quality improvement processing. Of course, as described later, the button 3420 may be a button for instructing display of a high-quality image.

In this embodiment, the image quality improvement process is executed by specifying the button 3420, or whether or not to execute it is determined based on information stored (stored) in the database. First, an example will be described in which display of a high-quality image and display of a low-quality image are switched by specifying the button 3420 in response to an instruction from the examiner. Note that the image to be subjected to the image quality enhancement processing will be described as an OCTA En-Face image. When the examiner specifies the report tab 3403 and transitions to the report screen, low-quality OCTA En-Face images Im3407 and Im3408 are displayed. Thereafter, when the examiner specifies a button 3420, the image quality improvement unit executes image quality improvement processing on the images Im3407 and Im3408 displayed on the screen. After the image quality improvement process is completed, the display control unit 101-05 displays the high quality image generated by the image quality improvement unit on the report screen. Note that since Im3406 is a display in which Im3407 is superimposed on SLO image Im3405, Im3406 also displays an image that has been subjected to high-quality image processing. Then, the display of the button 3420 is changed to an active state, and a display is made so that it can be seen that the image quality enhancement process has been executed. Here, the execution of the process in the image quality improvement unit does not need to be limited to the timing at which the examiner designates the button 3420. Since the types of OCTA En-Face images Im3407 and Im3408 to be displayed when opening the report screen are known in advance, image quality enhancement processing may be performed when transitioning to the report screen. Then, the display control unit 101-05 may display a high-quality image on the report screen at the timing when the button 3420 is pressed. Furthermore, it is not necessary that there are two types of images that are subjected to image quality enhancement processing in response to an instruction from the examiner or when transitioning to a report screen. For images that are likely to be displayed, for example, multiple OCTA En-Face images such as the superficial layer (Im2910), deep layer (Im2920), outer layer (Im2930), and choroidal vascular network (Im2940) as shown in Figure 13. The process may also be performed by In this case, the image obtained by high-quality image processing may be temporarily stored in a memory or in a database.

Next, a case will be described in which image quality improvement processing is executed based on information saved (stored) in a database. If the database stores the state of executing the image quality enhancement process, the high quality image obtained by executing the image quality enhancement process is displayed by default when the report screen is displayed. By displaying the button 3420 in an active state by default, the examiner can be configured to know that a high-quality image obtained by performing high-quality image processing is being displayed. . If the examiner wishes to display the low-quality image before the high-quality image processing, the examiner can display the low-quality image by designating the button 3420 and canceling the active state. If the examiner wishes to return to a high-quality image, the examiner specifies button 3420. Whether or not to perform high-image quality processing on the database is specified in common for all data stored in the database, and for each layer, such as for each photographic data (for each examination). For example, if a state in which image quality enhancement processing is executed for the entire database is saved, but a state in which the examiner does not execute image quality enhancement processing for individual imaging data (individual examinations) is saved. In this case, the next time the photographed data is displayed, it will be displayed without performing the image quality enhancement process. An unillustrated user interface (for example, a save button) may be used to save the execution state of the image quality improvement process for each photographic data (for each examination). In addition, when transitioning to other imaging data (another examination) or other patient data (for example, changing to a display screen other than the report screen in response to an instruction from the examiner), the display state (for example, if the button 3420 is The state for executing the image quality improvement process may be saved based on the state). As a result, if the execution of high image quality processing is not specified for each imaging data unit (inspection unit), processing is performed based on the information specified for the entire database, and the processing is performed for each imaging data unit (examination unit). If specified, processing can be executed individually based on that information.

Although an example is shown in which Im3407 and Im3408 are displayed as OCTA En-Face images in this embodiment, the OCTA En-Face images to be displayed can be changed by the examiner's designation. Therefore, a description will be given of changing the image when execution of the image quality enhancement process is specified (button 3420 is in the active state).

The image is changed using a user interface (for example, a combo box) not shown. For example, when the examiner changes the type of image from the superficial layer to the choroidal vascular network, the image quality enhancement unit executes image quality enhancement processing on the choroidal vascular network image, and the display control unit 101-05 Displays high-quality images generated by on the report screen. That is, the display control unit 101-05, in response to an instruction from the examiner, displays a high-quality image in a first depth range, and displays a high-quality image in a second depth range that is at least partially different from the first depth range. You may change the display to a high-quality image. At this time, the display control unit 101-05 changes the first depth range to the second depth range in response to an instruction from the examiner, thereby displaying the high-quality image in the first depth range. The display may be changed to a high-quality image in the second depth range. Note that, as described above, if a high-quality image has already been generated for an image that is likely to be displayed at the time of report screen transition, the display control unit 101-05 does not display the generated high-quality image. Bye. Note that the method of changing the image type is not limited to the above method, and it is also possible to generate an OCTA En-Face image in which a different depth range is set by changing the reference layer and offset values. In that case, when the reference layer or offset value is changed, the image quality enhancement unit executes image quality enhancement processing on any En-Face image of OCTA, and the display control unit 101-05 Display images on the report screen. The reference layer and offset value can be changed using a user interface (for example, a combo box or text box) not shown. Furthermore, by dragging (moving the layer boundary) either of the boundary lines 3413 and 3414 superimposed on the tomographic images Im3411 and Im3412, the generation range of the OCTA En-Face image can be changed. When changing the border line by dragging, the execution command for image quality improvement processing is executed continuously. Therefore, the image quality improvement unit may always process the execution command, or may execute the process after changing the layer boundary by dragging. Alternatively, the execution of the image quality improvement processing may be sequentially commanded, but when the next command arrives, the previous command may be canceled and the latest command may be executed. Note that the image quality improvement process may take a relatively long time. Therefore, no matter what timing the command is executed as described above, it may take a relatively long time until a high-quality image is displayed. Therefore, after the depth range for generating an OCTA En-Face image is set according to instructions from the examiner, until the high-quality image is displayed, the OCTA En-Face image corresponding to the set depth range is An En-Face image (low quality image) may be displayed. That is, when the depth range is set, the OCTA En-Face image (low quality image) corresponding to the set depth range is displayed, and when the image quality enhancement process is completed, the OCTA En-Face image is displayed. The display of (the low-quality image) may be changed to the display of a high-quality image. Furthermore, information indicating that image quality improvement processing is being executed may be displayed from the time the depth range is set until the high quality image is displayed. Note that these are not only cases where the execution of high image quality processing has already been specified (button 3420 is in an active state), but also cases where, for example, execution of high image quality processing is performed in response to an instruction from the examiner. It can be applied even when an instruction is given and until a high-quality image is displayed.

In this embodiment, an example was shown in which different layers are displayed in Im3407 and Im3408 as OCTA En-Face images, and low-quality and high-quality images are switched and displayed, but the present invention is not limited to this. For example, a low quality OCTA En-Face image may be displayed on Im3407, and a high quality OCTA En-Face image may be displayed side by side on Im3408. When images are switched and displayed, the images are switched at the same location, so it is easy to compare the parts that have changed, and when they are displayed side by side, the images can be displayed at the same time, making it easy to compare the entire image.

Here, the analysis unit 101-46 may perform image analysis on the high-quality image generated by the high-quality image processing. As for image analysis of a high-quality OCTA En-Face image, by applying arbitrary binarization processing, a location corresponding to a blood vessel (blood vessel region) can be detected from the image. The area density can be analyzed by determining the proportion of the area corresponding to the detected blood vessel in the image. Further, by thinning the portion corresponding to the binarized blood vessel, it is possible to obtain an image with a line width of 1 pixel, and obtain the proportion occupied by blood vessels (also referred to as skeleton density) that does not depend on the thickness. These images may be used to analyze the area and shape (circularity, etc.) of the avascular area (FAZ). As a method of analysis, the above-mentioned numerical values may be calculated from the entire image, or a user interface (not shown) may be used to calculate a specified region of interest (ROI) based on instructions from the examiner (user). You may also calculate a numerical value for that. The ROI setting is not necessarily specified by the examiner, but may also be such that a predetermined region is automatically specified. Here, the various parameters described above are examples of analysis results related to blood vessels, and any parameter related to blood vessels may be used. Note that the analysis unit 101-46 may perform multiple image analysis processes. In other words, here we have shown an example of analyzing En-Face images from OCTA, but this is not the only way to analyze images acquired by OCT. You may also perform analysis. In this regard, the analysis unit 101-46 may perform some or all of the plurality of image analysis processes in response to instructions from the examiner via any input device.

At this time, the display control unit 101-05 causes the display unit 104 to display the high-quality image generated by the image quality improvement unit and the analysis result by the analysis unit 101-46. Note that the display control unit 101-05 may output the high-quality image and the analysis results to separate display units or devices. Further, the display control unit 101-05 may display only the analysis results on the display unit 104. Furthermore, when the analysis section 101-46 outputs a plurality of analysis results, the display control section 101-05 may output some or all of the plurality of analysis results to the display section 104 or other devices. . For example, the analysis results regarding blood vessels in the OCTA En-Face image may be displayed on the display unit 104 as a two-dimensional map. Further, a value indicating the analysis result regarding blood vessels in the OCTA En-Face image may be superimposed on the OCTA En-Face image and displayed on the display unit 104. In this manner, the image processing apparatus 101 according to the present embodiment uses high-quality images for image analysis, so that the accuracy of analysis can be improved.

Next, the execution of image quality improvement processing during screen transition will be explained using FIGS. 14(a) and 14(b). FIG. 14(a) is an example of a screen in which the OCTA image in FIG. 14(b) is enlarged and displayed. In FIG. 14(a), a button 3420 is also displayed as in FIG. 14(b). The screen transition from FIG. 14(b) to FIG. 14(a) is made, for example, by double-clicking the OCTA image, and from FIG. 14(a) to FIG. 14(b) is made by clicking the close button 3430. Note that the screen transition is not limited to the method shown here, and a user interface (not shown) may be used.

If execution of image quality enhancement processing is specified at the time of screen transition (button 3420 is active), that state is maintained even at the time of screen transition. That is, when transitioning to the screen of FIG. 14(a) while a high-quality image is being displayed on the screen of FIG. 14(b), the high-quality image is also displayed on the screen of FIG. 14(a). The button 3420 is then activated. The same holds true when transitioning from FIG. 14(a) to FIG. 14(b). In FIG. 14(a), the display can also be switched to a low-quality image by specifying a button 3420.

Regarding screen transitions, it is not limited to the screens shown here, but if the transition is to a screen that displays the same imaging data, such as a follow-up display screen or a panoramic display screen, the display state of high-quality images can be maintained. Perform the transition as it is. That is, on the display screen after the transition, an image corresponding to the state of the button 3420 on the display screen before the transition is displayed. For example, if the button 3420 on the display screen before transition is in an active state, a high-quality image is displayed on the display screen after transition. Further, for example, if the active state of the button 3420 on the display screen before transition is released, a low-quality image is displayed on the display screen after transition. Note that when the button 3420 on the follow-up observation display screen becomes active, multiple images obtained at different dates and times (different examination dates) displayed side by side on the follow-up observation display screen are switched to high-quality images. It's okay. That is, when the button 3420 on the display screen for follow-up observation becomes active, it may be configured to be reflected on a plurality of images obtained at different dates and times at once.

Here, an example of a display screen for follow-up observation is shown in FIG. When the tab 3801 is selected in response to an instruction from the examiner, a display screen for progress observation is displayed as shown in FIG. At this time, the examiner can change the depth range of the measurement target area by selecting from the default depth range set (3802 and 3803) displayed in the list box. For example, the superficial layer of the retina is selected in list box 3802, and the deep layer of the retina is selected in list box 3803. The upper display area displays the analysis result of the motion contrast image of the superficial layer of the retina, and the lower display area displays the analysis result of the motion contrast image of the deep layer of the retina. That is, when a depth range is selected, multiple images taken at different dates and times are collectively changed to parallel display of analysis results of multiple motion contrast images in the selected depth range.

At this time, if the display of the analysis results is set to a non-selected state, the display may be changed to parallel display of a plurality of motion contrast images at different dates and times. Then, when the button 3420 is designated in response to an instruction from the examiner, the display of the plurality of motion contrast images is changed to the display of the plurality of high-quality images at once.

In addition, when the display of analysis results is in the selected state, when the button 3420 is specified in response to an instruction from the examiner, the display of the analysis results of multiple motion contrast images changes to the display of the analysis results of multiple high-quality images. The display will be changed all at once. Here, the display of the analysis results may be such that the analysis results are superimposed on the image with arbitrary transparency. At this time, the display of the analysis results may be changed to, for example, a state in which the analysis results are superimposed on the displayed image with arbitrary transparency. Further, the change to the display of the analysis result may be, for example, a change to the display of an image (for example, a two-dimensional map) obtained by blending the analysis result and the image with arbitrary transparency.

Further, the type of layer boundary and the offset position used for specifying the depth range can be changed all at once from user interfaces 3805 and 3806, respectively. By displaying the tomographic images together and moving the layer boundary data superimposed on the tomographic images according to instructions from the examiner, the depth range of multiple motion contrast images at different times can be changed at once. may be done. At this time, when a plurality of tomographic images with different dates and times are displayed side by side and the above movement is performed on one tomographic image, the layer boundary data may be similarly moved on other tomographic images. Further, the image projection method and the presence or absence of projection artifact suppression processing may be changed by selecting from a user interface such as a context menu, for example. Alternatively, a selection button 3807 may be selected to display a selection screen, and an image selected from the image list displayed on the selection screen may be displayed. Note that the arrow 3804 displayed at the top of FIG. 11 is a mark indicating the currently selected test, and the baseline test (Baseline) is the test selected at the time of follow-up imaging (the one shown in FIG. 11). The leftmost image). Of course, a mark indicating a reference test may be displayed on the display unit 104.

Furthermore, when the "Show Difference" check box 3808 is designated, the measurement value distribution (map or sector map) for the reference image is displayed on the reference image. Furthermore, in this case, a difference measurement value map between the measurement value distribution calculated for the reference image and the measurement distribution calculated for the image displayed in the area is displayed in the area corresponding to the other inspection dates. do. As the measurement results, a trend graph (a graph of measurement values for images on each inspection day obtained by measuring changes over time) may be displayed on the report screen. That is, time-series data (for example, a time-series graph) of multiple analysis results corresponding to multiple images at different dates and times may be displayed. At this time, analysis results related to dates and times other than the multiple dates and times corresponding to the multiple images being displayed are also displayed in a state that allows them to be distinguished from the multiple analysis results corresponding to the multiple images being displayed (for example, in a chronological order). It may also be displayed as time series data (the color of each point on the graph differs depending on whether or not an image is displayed). Further, the regression line (curve) of the trend graph and the corresponding mathematical formula may be displayed on the report screen.

Although the present embodiment has been described regarding a motion contrast image, the present invention is not limited to this. Images related to processing such as display, high image quality, and image analysis according to this embodiment may be tomographic images. Furthermore, not only a tomographic image but also a different image such as an SLO image, a fundus photograph, or a fluorescent fundus photograph may be used. In that case, the user interface for executing image quality enhancement processing may be one that instructs execution of image quality enhancement processing on multiple images of different types, or one that instructs to execute image quality enhancement processing on multiple images of different types, or one that allows you to select any image from multiple images of different types. There may also be an instruction to execute high image quality processing.

With such a configuration, the display control unit 101-05 can display on the display unit 104 an image processed by the image quality improvement unit (not shown) according to the present embodiment. At this time, as described above, if at least one of the multiple conditions regarding display of high-quality images, display of analysis results, depth range of the displayed frontal image, etc. is selected, the display screen is Even if the transition is made, the selected state may be maintained.

Furthermore, as described above, when at least one of the plurality of conditions is selected, even if the other conditions are changed to the selected state, the selected state of at least one of the conditions is maintained. It's okay. For example, when the display of the analysis results is selected, the display control unit 101-05 displays the analysis results of the low-quality image in response to an instruction from the examiner (for example, when the button 3420 is specified). may be changed to displaying the analysis results of high-quality images. In addition, when the display of the analysis results is in the selected state, the display control unit 101-05 displays the analysis results of the high-quality image in response to an instruction from the examiner (for example, when the button 3420 is deselected). The display may be changed to display the analysis results of low-quality images.

In addition, when the display of high-quality images is not selected, the display control unit 101-05 controls the display of low-quality images in response to an instruction from the examiner (for example, when the designation of displaying analysis results is canceled). The display of the image analysis results may be changed to the display of low-quality images. In addition, when the display of high-quality images is in a non-selected state, the display control unit 101-05 controls the display of low-quality images in response to an instruction from the examiner (for example, when the display of analysis results is specified). The display may be changed to display the analysis results of low-quality images. In addition, when displaying a high-quality image is selected, the display control unit 101-05 displays a high-quality image in response to an instruction from the examiner (for example, when the designation for displaying analysis results is canceled). The display of the analysis results may be changed to the display of high-quality images. In addition, when displaying a high-quality image is selected, the display control unit 101-05 displays a high-quality image in response to an instruction from the examiner (for example, when displaying analysis results is specified). may be changed to displaying the analysis results of high-quality images.

Also, consider a case where a high-quality image is displayed in a non-selected state and a first type of analysis result is displayed in a selected state. In this case, the display control unit 101-05 performs the first type of analysis of the low-quality image in response to an instruction from the examiner (for example, when displaying the second type of analysis results is specified). The display of the results may be changed to display of the second type of analysis results of low-quality images. Also, consider a case where a high-quality image is displayed in a selected state and a first type of analysis result is displayed in a selected state. In this case, the display control unit 101-05 performs the first type of analysis of the high-quality image in response to an instruction from the examiner (for example, when displaying the second type of analysis results is specified). The display of the results may be changed to display of the second type of analysis results of high-quality images.

Note that, as described above, the display screen for follow-up observation may be configured so that these display changes are reflected at once on a plurality of images obtained at different dates and times. Here, the display of the analysis results may be such that the analysis results are superimposed on the image with arbitrary transparency. At this time, the display of the analysis results may be changed to, for example, a state in which the analysis results are superimposed on the displayed image with arbitrary transparency. Further, the change to the display of the analysis result may be, for example, a change to the display of an image (for example, a two-dimensional map) obtained by blending the analysis result and the image with arbitrary transparency.

(Modification 1)
In the embodiment described above, the display control unit 101-05 causes the display unit 104 to display an image selected from among the high-quality image generated by the image quality improvement unit and the input image according to an instruction from the examiner. be able to. Furthermore, the display control unit 101-05 may switch the display on the display unit 104 from a captured image (input image) to a high-quality image in response to an instruction from the examiner. That is, the display control unit 101-05 may change the display of a low-quality image to the display of a high-quality image in response to an instruction from the examiner. Furthermore, the display control unit 101-05 may change the display of a high-quality image to the display of a low-quality image in response to an instruction from the examiner.

Furthermore, the image quality improvement unit in the image processing device 101 (or image processing unit 101-04) starts image quality improvement processing by the image quality improvement engine (trained model for image quality improvement) (initiates image quality improvement processing to the image quality improvement engine). The display control section 101-05 may cause the display section 104 to display the high-quality image generated by the image quality improvement section. On the other hand, when an input image is captured by the imaging device (tomographic imaging device 100), the high-quality image automatically generates a high-quality image based on the input image, and the display control unit 101-05 A high-quality image may be displayed on the display unit 104 in response to instructions from the examiner. Here, the image quality improvement engine includes a trained model that performs the above-described image quality improvement processing (image quality improvement processing).

Note that these processes can be similarly performed for outputting the analysis results. That is, the display control unit 101-05 may change the display of the analysis results of low-quality images to the display of the analysis results of high-quality images in response to instructions from the examiner. Further, the display control unit 101-05 may change the display of the analysis results of the high-quality images to the display of the analysis results of the low-quality images in response to an instruction from the examiner. Of course, the display control unit 101-05 may change the display of the analysis result of the low-quality image to the display of the low-quality image in response to an instruction from the examiner. Further, the display control unit 101-05 may change the display of the low-quality image to the display of the analysis result of the low-quality image in response to an instruction from the examiner. Further, the display control unit 101-05 may change the display of the analysis result of the high-quality image to the display of the high-quality image in response to an instruction from the examiner. Further, the display control unit 101-05 may change the display of the high-quality image to the display of the analysis result of the high-quality image in response to an instruction from the examiner.

Further, the display control unit 101-05 may change the display of the analysis result of the low-quality image to the display of another type of analysis result of the low-quality image in response to an instruction from the examiner. Further, the display control unit 101-05 may change the display of the analysis result of the high-quality image to the display of another type of analysis result of the high-quality image in response to an instruction from the examiner.

Here, the display of the analysis result of the high-quality image may be one in which the analysis result of the high-quality image is displayed superimposed on the high-quality image with arbitrary transparency. Moreover, the display of the analysis result of the low-quality image may be one in which the analysis result of the low-quality image is displayed superimposed on the low-quality image with arbitrary transparency. At this time, the display of the analysis results may be changed to, for example, a state in which the analysis results are superimposed on the displayed image with arbitrary transparency. Further, the change to the display of the analysis result may be, for example, a change to the display of an image (for example, a two-dimensional map) obtained by blending the analysis result and the image with arbitrary transparency.

Furthermore, in the various embodiments described above, the processing performed on the set region of interest is not limited to analysis processing, and may be, for example, image processing. Any image processing may be used as long as it is, for example, contrast processing, gradation conversion processing, super resolution processing, smoothing processing, etc. Furthermore, even after transitioning to another display screen, a blended image obtained by performing blending processing using the transmittance set before transition may be displayed. For example, after transitioning to the progress observation display screen, a plurality of blended images obtained by blending using the transmittance set before the transition may be displayed side by side as a plurality of images obtained at different dates and times. A similar slide bar is also displayed on the follow-up observation display screen, and when the transmittance is set (changed) according to instructions from the examiner, multiple images obtained at different dates and times with the set transmittance will be displayed. may be reflected all at once. That is, when the transmittance is set (changed), a plurality of blended images obtained by blending using the set transmittance may be displayed. Note that the blending process is not limited to these display screens, but at least one display screen such as the shooting confirmation screen, report screen, and preview screen for various adjustments before shooting (display screen on which various live video images are displayed). It is sufficient if it is executable.

(Modification 2)
In the various embodiments and modifications described above, the transmittance (transmittance coefficient) used in the blending process is not limited to being set only by instructions from the examiner, and may be set automatically or semi-automatically. may be set. For example, by using learning data in which medical images such as at least one of an OCT image and an OCTA image of mutually corresponding regions are input data, and the transmittance set according to instructions from the examiner is correct data (teacher data). A learned model obtained by machine learning may be used. That is, the transmittance setting unit may be configured to generate a new transmittance from a medical image such as at least one of an OCT image and an OCTA image of mutually corresponding regions, using the learned model. . At this time, the learned model may be, for example, a learned model obtained by additional learning using learning data in which the transmittance determined (changed) according to an instruction from the examiner is used as correct data. In addition, the trained model may be added by learning data that uses, as correct data, a transmittance that is changed from the new transmittance (transmittance obtained using the trained model) in response to an instruction from the examiner, for example. It may be a learned model obtained through learning. Thereby, for example, it is possible to obtain a new transmittance that takes into account the tendency of the examiner's preferred transmittance for medical images. That is, the transmittance setting section customized by the examiner can be configured with high precision. Therefore, the diagnostic efficiency of the examiner can be improved. Note that the OCT image and OCTA image of mutually corresponding regions may be images obtained using at least a portion of a common interference signal, for example.

Here, the learned model can be obtained by machine learning using learning data. Machine learning includes, for example, deep learning that consists of multilayer neural networks. Further, for example, a convolutional neural network (CNN) can be used as a machine learning model for at least a portion of the multi-layer neural network. Further, a technique related to an autoencoder (self-encoder) may be used for at least a portion of the multi-layer neural network. Furthermore, a technique related to backpropagation (error backpropagation method) may be used for learning. However, machine learning is not limited to deep learning, and may be any type of learning that uses a model that can self-extract (express) features of learning data such as images through learning. Further, a trained model is a model that has been trained (performed learning) using appropriate learning data in advance for a machine learning model based on an arbitrary machine learning algorithm. However, the trained model does not undergo any further learning, and additional learning can be performed on the trained model. Further, the learning data is composed of a pair of input data and output data (correct data). Here, the learning data may be referred to as teacher data, or the correct answer data may be referred to as teacher data. Further, the learned model may be updated through additional learning, so that it can be customized as a model suitable for the operator, for example. Of course, the trained model in this modification is not limited to a trained model obtained by additional learning, but may be a trained model obtained by learning using learning data including medical images and information regarding transmittance. Anything is fine.

Further, the trained model described above may be a trained model obtained by learning using learning data including input data that is a set of a plurality of different types of medical images of a predetermined region of a subject. At this time, the input data included in the learning data is, for example, input data that is a set of a motion contrast frontal image and a brightness frontal image (or a brightness tomographic image) of the fundus, a tomographic image (B scan image) of the fundus, and a color fundus Input data such as a set of images (or fluorescent fundus images) can be considered. Further, the plurality of medical images of different types may be any image as long as they are obtained using different modalities, different optical systems, different principles, etc. Further, the trained model described above may be a trained model obtained by learning using learning data including input data that is a set of a plurality of medical images of different parts of the subject. At this time, the input data included in the learning data is, for example, input data that is a set of a tomographic image of the fundus of the eye (B-scan image) and a tomographic image of the anterior segment of the eye (B-scan image), or a three-dimensional image of the macula of the fundus of the eye. Input data such as a set of an OCT image and a circle scan (or raster scan) tomographic image of the optic disc of the fundus can be considered. Note that the input data included in the learning data may be a plurality of medical images of different parts and different types of the subject. At this time, the input data included in the learning data may be, for example, input data that includes a tomographic image of the anterior segment of the eye and a color fundus image. Further, the above-mentioned trained model may be a trained model obtained by learning using learning data including input data that is a set of a plurality of medical images of a predetermined region of a subject at different photographing angles of view. Further, the input data included in the learning data may be a combination of a plurality of medical images obtained by time-divisionally dividing a predetermined region into a plurality of regions, such as a panoramic image. Furthermore, the input data included in the learning data may be input data that is a set of a plurality of medical images of a predetermined region of the subject at different times.

Note that the new transmittance obtained using the learned model described above may be set as an initial setting, and the transmittance may be changed in accordance with instructions from the examiner. Further, it may be configured such that it is possible to select whether or not to use the changed transmittance as learning data for additional learning in accordance with an instruction from the examiner. Furthermore, by setting the ROI on the blended image, the transmittance set (changed) when the ROI is set is also selected to be used as learning data for additional learning. Good too.

(Modification 3)
The display control unit 101-05 in the various embodiments and modifications described above may display analysis results such as the thickness of a desired layer and various blood vessel densities on a report screen of the display screen. In addition, the optic disc, macular area, blood vessel area, nerve fiber bundle, vitreous area, macular area, choroid area, sclera area, lamina cribrosa area, retinal layer boundary, retinal layer boundary edge, photoreceptor cells, blood cells, Values (distribution) of parameters related to a region of interest including at least one of a blood vessel wall, a blood vessel inner wall boundary, a blood vessel outer wall boundary, a ganglion cell, a corneal region, an angle region, Schlemm's canal, etc. may be displayed as an analysis result. At this time, for example, by analyzing a medical image to which various artifact reduction processes have been applied, highly accurate analysis results can be displayed. Note that artifacts include, for example, false image areas caused by light absorption by blood vessel areas, projection artifacts, and band-shaped artifacts in the front image that occur in the main scanning direction of the measurement light due to the condition of the eye being examined (movement, blinking, etc.). There may be. Furthermore, the artifact may be of any type, as long as it is a defective area that randomly appears on a medical image of a predetermined region of a subject each time the image is taken. Further, the values (distribution) of parameters regarding a region including at least one of the various artifacts (impossible regions) as described above may be displayed as an analysis result. Further, the values (distribution) of parameters related to a region including at least one of an abnormal site such as drusen, new blood vessels, vitiligo (hard vitiligo), and pseudodrusen may be displayed as an analysis result.
Further, the analysis results may be displayed as an analysis map, sectors showing statistical values corresponding to each divided area, or the like. Note that the analysis results may be generated using a trained model (analysis result generation engine, trained model for analysis result generation) obtained by learning the analysis results of medical images as learning data. . At this time, the trained model is trained using learning data that includes a medical image and an analysis result of the medical image, or learning data that includes a medical image and the analysis result of a medical image of a different type than the medical image. It may be obtained by In addition, the trained model is obtained by learning using learning data that includes input data that is a set of multiple medical images of different types of predetermined parts, such as brightness frontal images and motion contrast frontal images. Good too. Here, the brightness front image corresponds to the En-Face image of the tomographic image, and the motion contrast front image corresponds to the En-Face image of OCTA. Furthermore, it may be configured to display analysis results obtained using high-quality images generated by trained models for high-quality images. Note that the trained model for high image quality is obtained by learning training data in which the first image is used as input data and the second image, which has higher image quality than the first image, is used as correct data. It's okay. At this time, the second image is processed to increase contrast or reduce noise by, for example, superimposing a plurality of first images (for example, averaging processing of a plurality of first images obtained by positioning). It may be a high-quality image that has been subjected to the above process.

Further, the input data included in the learning data may be a high-quality image generated by a trained model for high-quality images, or may be a set of a low-quality image and a high-quality image. In addition, the learning data includes, for example, at least analysis values obtained by analyzing the analysis area (e.g., average value, median value, etc.), a table including the analysis values, an analysis map, the position of the analysis area such as a sector in an image, etc. The input data may be labeled (annotated) with information including one as the correct data (for supervised learning). Note that the analysis result obtained by the learned model for generating the analysis result may be displayed in response to an instruction from the examiner. For example, the image processing unit 101-04 uses a trained model for generating analysis results (different from a trained model for high image quality) to process at least one medical image among the plurality of medical images to be blended. An image analysis result related to the at least one medical image can be generated from the at least one medical image. Further, for example, the display control unit 101-05 can cause the display unit 104 to display an image analysis result obtained from the at least one medical image using a trained model for generating an analysis result.

Further, the display control unit 101-05 in the various embodiments and modifications described above may display various diagnostic results such as glaucoma and age-related macular degeneration on the report screen of the display screen. At this time, for example, by analyzing a medical image to which the various artifact reduction processes described above have been applied, highly accurate diagnostic results can be displayed. Further, as a diagnosis result, the position of the identified abnormal region, etc. may be displayed on the image, or the state of the abnormal region, etc. may be displayed in characters or the like. Furthermore, the classification results of abnormal regions and the like (for example, Curtin classification) may be displayed as the diagnosis results. Further, as the classification result, for example, information indicating the probability of each abnormal site (for example, a numerical value indicating a ratio) may be displayed. Further, information necessary for a doctor to confirm a diagnosis may be displayed as a diagnosis result. The necessary information may include, for example, advice on additional photography. For example, if an abnormal site is detected in a blood vessel region in an OCTA image, a message may be displayed to the effect that fluorescence imaging using a contrast agent that allows the blood vessels to be observed in more detail than in OCTA will be performed.

Note that the diagnostic results may be generated using a trained model (a diagnostic result generation engine, a trained model for generating diagnostic results) obtained by learning the diagnostic results of medical images as learning data. . In addition, the trained model can be trained using training data that includes a medical image and the diagnosis result of the medical image, or training data that includes the medical image and the diagnosis result of a different type of medical image than the medical image. It may be something you have obtained. Furthermore, the system may be configured to display diagnostic results obtained using high-quality images generated by trained models for high-quality images. For example, the image processing unit 101-04 uses a trained model for generating diagnostic results (different from a trained model for improving image quality) to process at least one medical image among the plurality of medical images to be blended. A diagnostic result related to the at least one medical image can be generated from the at least one medical image. Further, for example, the display control unit 101-05 can cause the display unit 104 to display a diagnosis result obtained from the at least one medical image using a trained model for generating a diagnosis result.

Further, the input data included in the learning data may be a high-quality image generated by a trained model for high-quality images, or may be a set of a low-quality image and a high-quality image. In addition, the learning data includes, for example, the diagnosis name, the type and condition (degree) of the lesion (abnormal site), the position of the lesion in the image, the position of the lesion relative to the region of interest, findings (image interpretation findings, etc.), and the basis for the diagnosis name (positive The input data is labeled (annotated) as the correct data (for supervised learning), including at least one of the following: negative medical support information, etc.), grounds for denying the diagnosis (negative medical support information), etc. It may be data. Note that the configuration may be such that the diagnostic results obtained by the trained model for generating diagnostic results are displayed in response to instructions from the examiner.

In addition, the display control unit 101-05 in the various embodiments and modifications described above also displays object recognition results (object detection results) such as the above-mentioned regions of interest, artifacts, and abnormal regions, and segmentation on the report screen of the display screen. The results may also be displayed. At this time, for example, a rectangular frame or the like may be superimposed and displayed around the object on the image. Furthermore, for example, a color or the like may be superimposed and displayed on an object in the image. Note that object recognition results and segmentation results are generated using a trained model (object recognition engine, object recognition It may be generated using a trained model, a segmentation engine, or a trained model for segmentation. Note that the analysis result generation and diagnosis result generation described above may be obtained by using the object recognition results and segmentation results described above. For example, analysis result generation or diagnostic result generation processing may be performed on a region of interest obtained through object recognition or segmentation processing.

Furthermore, when detecting an abnormal region, the image processing unit 101-04 may use a generative adversarial network (GAN) or a variational auto-encoder (VAE). For example, a DCGAN (Deep Convolutional GAN) consists of a generator obtained by learning to generate tomographic images, and a discriminator obtained by learning to discriminate between a new tomographic image generated by the generator and a real fundus front image. ) can be used as a machine learning model.

When using DCGAN, for example, a discriminator encodes an input tomographic image into a latent variable, and a generator generates a new tomographic image based on the latent variable. Thereafter, the difference between the input tomographic image and the generated new tomographic image can be extracted as an abnormal site. Furthermore, when using VAE, for example, an input tomographic image is encoded by an encoder to become a latent variable, and a new tomographic image is generated by decoding the latent variable by a decoder. Thereafter, the difference between the input tomographic image and the generated new tomographic image can be extracted as an abnormal site. Although the explanation has been given using a tomographic image as an example of input data, a fundus image, a frontal image of the anterior eye, etc. may also be used.

Furthermore, the image processing unit 101-04 may detect an abnormal region using a convolutional auto-encoder (CAE). When using CAE, the same image is trained as input data and output data during learning. As a result, when an image with an abnormal part is input to CAE at the time of estimation, an image without an abnormal part is outputted according to the learning tendency. Thereafter, the difference between the image input to CAE and the image output from CAE can be extracted as an abnormal site. Note that in this case as well, not only a tomographic image but also a fundus image, a frontal image of the anterior eye, etc. may be used as input data.

In these cases, the image processing unit 101-04 uses a medical image obtained using a generative adversarial network or an autoencoder for each of the different regions identified by segmentation processing, etc., and a medical image that is input to the generative adversarial network or autoencoder. Information regarding the difference from the medical image can be generated as information regarding the abnormal site. Thereby, the image processing unit 101-04 can be expected to detect an abnormal region at high speed and with high accuracy. Here, the autoencoder includes VAE, CAE, and the like. For example, the image processing unit 101-04 combines a medical image obtained from at least one medical image out of a plurality of medical images to be blended using an adversarial generative network or an autoencoder, and the at least one medical image. Information regarding the difference can be generated as information regarding the abnormal site. Further, for example, the display control unit 101-05 may display information regarding the difference between the at least one medical image and the at least one medical image obtained from the at least one medical image using the adversarial generative network or the autoencoder. This information can be displayed on the display unit 104 as information related to the above.

Furthermore, in a diseased eye, image characteristics differ depending on the type of disease. Therefore, the trained models used in the various embodiments and modifications described above may be generated and prepared for each type of disease or each abnormal site. In this case, for example, the image processing apparatus 101 can select a trained model to be used for processing in accordance with an input (instruction) from the operator regarding the type of disease, abnormal region, etc. of the eye to be examined. Note that the trained models prepared for each type of disease or abnormal site are not limited to trained models used for detecting retinal layers or generating area label images, etc., but are also used for image evaluation engines and analysis. The model may be a trained model used in an engine, etc. At this time, the image processing apparatus 101 may identify the type of disease or abnormal site of the eye to be examined from the image using a separately prepared trained model. In this case, the image processing device 101 may automatically select a trained model to be used for the above processing based on the type of disease or abnormal region identified using the separately prepared trained model. can. The trained model for identifying the disease type and abnormal area of the subject's eye uses tomographic images, fundus images, etc. as input data, and the training data uses the disease type and abnormal area in these images as output data. Learning may be performed using pairs. Here, as the input data of the learning data, a tomographic image, a fundus image, etc. may be used alone as input data, or a combination of these may be used as input data.

In addition, in particular, the trained model for generating diagnostic results may be a trained model obtained by training with training data that includes input data that is a set of multiple medical images of different types of predetermined parts of the subject. good. At this time, as input data included in the learning data, for example, input data including a set of a motion contrast frontal image and a brightness frontal image (or a brightness tomographic image) of the fundus can be considered. Further, as the input data included in the learning data, for example, input data including a set of a tomographic image of the fundus (B scan image) and a color fundus image (or a fluorescent fundus image) can be considered. Further, the plurality of medical images of different types may be any image as long as they are obtained using different modalities, different optical systems, or different principles.

In particular, the trained model for generating diagnostic results may be a trained model obtained by learning using learning data that includes input data that is a set of a plurality of medical images of different parts of the subject. At this time, as input data included in the learning data, for example, input data including a set of a tomographic image of the fundus (B-scan image) and a tomographic image of the anterior segment (B-scan image) can be considered. Furthermore, as input data included in the learning data, for example, input data that is a set of a three-dimensional OCT image (three-dimensional tomographic image) of the macula in the fundus and a circle scan (or raster scan) tomographic image of the optic disc in the fundus, etc. can also be considered.

Note that the input data included in the learning data may be a plurality of medical images of different parts and different types of the subject. At this time, the input data included in the learning data may be, for example, input data that includes a tomographic image of the anterior segment of the eye and a color fundus image. Further, the above-mentioned trained model may be a trained model obtained by learning using learning data including input data that is a set of a plurality of medical images of a predetermined region of a subject at different photographing angles of view. Further, the input data included in the learning data may be a combination of a plurality of medical images obtained by time-divisionally dividing a predetermined region into a plurality of regions, such as a panoramic image. At this time, by using wide-angle images such as panoramic images as learning data, it is possible to obtain image features with higher accuracy because they contain more information than narrow-angle images. Processing results can be improved. For example, at the time of estimation (at the time of prediction), when abnormal parts are detected at a plurality of positions in a wide-angle image, enlarged images of each abnormal part can be sequentially displayed. This makes it possible to efficiently confirm abnormal areas at multiple positions, thereby improving convenience for the examiner, for example. At this time, for example, the examiner may be configured to be able to select each position on the wide-angle image where an abnormal region is detected, and an enlarged image of the abnormal region at the selected position may be displayed. . Furthermore, the input data included in the learning data may be input data that is a set of a plurality of medical images of a predetermined region of the subject at different times.

Furthermore, the display screen on which at least one of the above-mentioned analysis results, diagnosis results, object recognition results, and segmentation results is displayed is not limited to the report screen. Such display screens include, for example, at least one display screen such as a shooting confirmation screen, a display screen for progress observation, and a preview screen for various adjustments before shooting (a display screen on which various live video images are displayed). may be displayed. For example, by displaying the at least one result obtained using the trained model described above on the imaging confirmation screen, the examiner can confirm highly accurate results even immediately after imaging. Further, the above-described display change between a low-quality image and a high-quality image may be, for example, a change in display between an analysis result of a low-quality image and an analysis result of a high-quality image.

Here, the various learned models described above can be obtained by machine learning using learning data. Machine learning includes, for example, deep learning that consists of multilayer neural networks. Further, for example, a convolutional neural network (CNN) can be used as a machine learning model for at least a portion of the multi-layer neural network. Further, a technique related to an autoencoder (self-encoder) may be used for at least a portion of the multi-layer neural network. Furthermore, a technique related to backpropagation (error backpropagation method) may be used for learning. However, machine learning is not limited to deep learning, and may be any type of learning that uses a model that can self-extract (express) features of learning data such as images through learning. Here, the machine learning model refers to a learning model using a machine learning algorithm such as deep learning. Further, a trained model is a model that has been trained (performed learning) using appropriate learning data in advance for a machine learning model based on an arbitrary machine learning algorithm. However, the trained model does not undergo any further learning, and additional learning can be performed on the trained model. Further, the learning data is composed of a pair of input data and output data (correct data). Here, the learning data may be referred to as teacher data, or the correct answer data may be referred to as teacher data.

Note that the GPU can perform efficient calculations by processing more data in parallel. For this reason, when learning is performed multiple times using a learning model such as deep learning, it is effective to perform processing on the GPU. Therefore, in this modification, a GPU is used in addition to the CPU for processing by the image processing unit 101-04, which is an example of a learning unit (not shown). Specifically, when a learning program including a learning model is executed, learning is performed by the CPU and GPU working together to perform calculations. Note that the processing of the learning section may be performed by only the CPU or GPU. Furthermore, the processing unit (estimation unit) that executes processing using the various learned models described above may also use a GPU, similar to the learning unit. Further, the learning section may include an error detection section and an updating section (not shown). The error detection unit obtains an error between output data output from the output layer of the neural network and correct data according to input data input to the input layer. The error detection unit may use a loss function to calculate the error between the output data from the neural network and the correct data. Furthermore, the updating section updates the connection weighting coefficients between the nodes of the neural network, etc., based on the error obtained by the error detection section, so that the error becomes smaller. This updating unit updates the connection weighting coefficients and the like using, for example, an error backpropagation method. The error backpropagation method is a method of adjusting connection weighting coefficients between nodes of each neural network so that the above-mentioned error is reduced.

In addition, machine learning models used for high image quality, segmentation, etc. have two functions: an encoder function consisting of multiple layers including multiple downsampling layers, and a decoder function consisting of multiple layers including multiple upsampling layers. A U-net type machine learning model having the following is applicable. In the U-net type machine learning model, position information (spatial information) that is made ambiguous in multiple layers configured as encoders is transferred to layers of the same dimension (layers that correspond to each other) in multiple layers configured as decoders. ) (e.g., using skip connections).

Further, as a machine learning model used for high image quality, segmentation, etc., for example, FCN (Fully Convolutional Network), SegNet, etc. can also be used. Alternatively, a machine learning model that performs object recognition in area units may be used depending on a desired configuration. As a machine learning model for object recognition, for example, RCNN (Region CNN), fastRCNN, or fasterRCNN can be used. Furthermore, YOLO (You Only Look Once) or SSD (Single Shot Detector or Single Shot MultiBox Detector) can also be used as a machine learning model that performs object recognition in area units.

Further, the machine learning model may be, for example, a capsule network (CapsNet). Here, in a general neural network, each unit (each neuron) is configured to output a scalar value, so that, for example, spatial information regarding the spatial positional relationship (relative position) between features in an image can be obtained. is configured to be reduced. Thereby, for example, learning can be performed such that the influence of local distortion or parallel movement of the image is reduced. On the other hand, in a capsule network, each unit (each capsule) is configured to output spatial information as a vector, so that, for example, spatial information is held. Thereby, for example, learning can be performed that takes into account the spatial positional relationship between features in an image.

In addition, the high image quality engine (trained model for high image quality) may be a trained model obtained by additionally learning training data that includes at least one high quality image generated by the high image quality engine. good. At this time, it may be possible to select whether or not to use the high-quality image as learning data for additional learning based on an instruction from the examiner. Note that these configurations are applicable not only to the trained model for high image quality but also to the various trained models described above. Moreover, a trained model for generating correct data for generating correct data such as labeling (annotation) may be used to generate correct data used for learning the various trained models described above. At this time, the trained model for generating correct answer data may be obtained by (sequentially) additionally learning correct answer data obtained by labeling (annotation) by the examiner. That is, the learned model for generating correct data may be obtained by additionally learning learning data in which unlabeled data is input data and labeled data is output data. In addition, in multiple consecutive frames such as a moving image, the system is configured to take into account the results of object recognition, segmentation, etc. of the previous and subsequent frames and correct the results of frames determined to have low accuracy. Good too. At this time, the corrected results may be additionally learned as correct data in response to instructions from the examiner.

In addition, in the various embodiments and modified examples described above, when detecting a region of the eye to be examined using a trained model for object recognition or a trained model for segmentation, predetermined image processing is performed for each detected region. You can also apply For example, consider a case where at least two of the vitreous region, retinal region, and choroid region are detected. In this case, when performing image processing such as contrast adjustment on at least two detected regions, by using different image processing parameters, it is possible to perform adjustment suitable for each region. By displaying images that have been adjusted appropriately for each region, the operator can more appropriately diagnose diseases and the like for each region. Note that the configuration in which different image processing parameters are used for each detected region may be similarly applied, for example, to regions of the eye to be examined that are detected without using a trained model.

(Modification 4)
In the preview screen in the various embodiments and modifications described above, the learned model described above may be used for each at least one frame of a live video image. At this time, if a plurality of live moving images of different parts or types are displayed on the preview screen, a learned model corresponding to each live moving image may be used. With this, for example, even for live moving images, the processing time can be shortened, so the examiner can obtain highly accurate information before starting imaging. Therefore, for example, it is possible to reduce failures in re-imaging, etc., and thus it is possible to improve the accuracy and efficiency of diagnosis.

Note that the plurality of live moving images may be, for example, a moving image of the anterior segment of the eye for alignment in the XYZ directions, or a frontal moving image of the fundus for focus adjustment of the fundus observation optical system or OCT focus adjustment. Further, the plurality of live moving images may be, for example, tomographic moving images of the fundus for OCT coherence gate adjustment (adjustment of the optical path length difference between the measurement optical path length and the reference optical path length). At this time, the above-mentioned various adjustments may be performed so that the region detected using the above-described trained model for object recognition or the trained model for segmentation satisfies predetermined conditions. For example, a value (for example, a contrast value or an intensity value) related to a predetermined retinal layer such as the vitreous region or RPE detected using a trained model for object recognition or a trained model for segmentation exceeds a threshold value ( Alternatively, the configuration may be such that various adjustments such as OCT focus adjustment are performed so that the peak value is reached. In addition, for example, OCT may be applied so that a predetermined retinal layer such as the vitreous region or RPE detected using a trained model for object recognition or a trained model for segmentation is at a predetermined position in the depth direction. It may be configured such that coherence gate adjustment is performed.

In these cases, the image quality improvement unit (not shown) in the image processing device 101 (or image processing unit 101-04) performs image quality improvement processing on the moving image using the trained model to achieve high image quality. It is possible to generate moving images. The imaging control unit 101-03 also controls the reference mirror 221 or the like so that one of the different areas identified by segmentation processing or the like is at a predetermined position in the display area while a high-quality moving image is displayed. It is possible to drive and control the optical member that changes the photographing range. In such a case, the imaging control unit 101-03 can automatically perform alignment processing based on highly accurate information so that the desired area is located at a predetermined position in the display area. Note that the optical member that changes the imaging range may be, for example, an optical member that adjusts the coherence gate position, and specifically may be the reference mirror 221 or the like. Further, the coherence gate position can be adjusted by an optical member that changes the optical path length difference between the measurement optical path length and the reference optical path length. It may also be a mirror, etc. Note that the optical member that changes the photographing range may be, for example, the stage section 100-2.

Further, the moving image to which the above-described trained model can be applied is not limited to a live moving image, but may be a moving image stored (saved) in the storage unit 101-02, for example. At this time, for example, a moving image obtained by aligning at least one frame of the tomographic moving image of the fundus oculi stored (saved) in the storage unit 101-02 may be displayed on the display screen. For example, if it is desired to observe the vitreous region in a suitable manner, a reference frame may be selected based on a condition such that as much of the vitreous region as possible exists on the frame. At this time, each frame is a tomographic image (B scan image) in the XZ direction. Then, a moving image in which other frames are aligned in the XZ directions with respect to the selected reference frame may be displayed on the display screen. At this time, for example, it may be configured to continuously display high-quality images (high-quality frames) that are sequentially generated by a learned model for improving image quality for each at least one frame of a moving image.

Note that as the above-mentioned method for positioning between frames, the same method may be applied for the positioning method in the X direction and the method for positioning in the Z direction (depth direction), or different methods may be applied. may be applied. Furthermore, alignment in the same direction may be performed multiple times using different techniques; for example, rough alignment may be followed by precise alignment. In addition, as a method for positioning, for example, positioning using retinal layer boundaries obtained by segmenting a tomographic image (B-scan image) (coarse in the Z direction), and positioning using multiple retinal layer boundaries obtained by dividing a tomographic image Positioning (accurate in the X and Z directions) using correlation information (similarity) between the region and the reference image, and one-dimensional projection images generated for each tomographic image (B scan image) (in the X direction) ) alignment, alignment (in the X direction) using a two-dimensional front image, etc. Alternatively, the configuration may be such that rough alignment is performed pixel by pixel, and then fine alignment is performed subpixel by subpixel.

Here, during various adjustments, there is a possibility that the object to be photographed, such as the retina of the eye to be examined, has not yet been successfully imaged. For this reason, since there is a large difference between the medical images input to the trained model and the medical images used as learning data, there is a possibility that high-quality images cannot be obtained with high accuracy. Therefore, when an evaluation value such as image quality evaluation of a tomographic image (B scan) exceeds a threshold value, display of high-quality moving images (continuous display of high-quality frames) may be automatically started. Further, when an evaluation value such as image quality evaluation of a tomographic image (B scan) exceeds a threshold value, the image quality improvement button may be changed to a state (active state) that can be specified by the examiner. Note that the image quality improvement button is a button for specifying execution of image quality improvement processing. Of course, the high-quality image button may be a button for instructing display of a high-quality image.

In addition, a different trained model for high image quality is prepared for each shooting mode with a different scanning pattern, etc., and the trained model for high image quality corresponding to the selected shooting mode is selected. Good too. Alternatively, one trained model for improving image quality obtained by learning training data including various medical images obtained in different imaging modes may be used.

(Modification 5)
In the various embodiments and modifications described above, when a trained model is undergoing additional learning, it may be difficult to output (inference/predict) using the trained model itself that is undergoing additional learning. For this reason, it is preferable to prohibit the input of medical images to a learned model that is undergoing additional learning. Further, another trained model that is the same as the trained model that is currently being additionally trained may be prepared as a preliminary trained model. At this time, during additional learning, it is preferable to allow medical images to be input to the preliminary trained model. Then, after the additional learning is completed, the trained model after the additional learning is evaluated, and if there is no problem, the preliminary trained model may be replaced with the trained model after the additional learning. Furthermore, if there is a problem, a preliminary trained model may be used. Note that for the evaluation of the trained model, for example, a trained model for classification for classifying high-quality images obtained by the trained model for high image quality from other types of images may be used. For example, a trained model for classification uses as input data multiple images including high-quality images and low-quality images obtained by a trained model for high image quality, and the types of these images are labeled (annotated). The model may be a trained model obtained by learning training data that includes the correct data as correct data. At this time, the type of image of the input data at the time of estimation (during prediction) is displayed together with information indicating the probability of each type of image included in the correct data at the time of learning (for example, a numerical value indicating a percentage). Good too. In addition to the above-mentioned images, the input data for the trained model for classification may also include superposition processing of multiple low-quality images (for example, averaging processing of multiple low-quality images obtained by positioning), etc. may include high-quality images that have undergone high contrast, noise reduction, and the like.

Further, it may be possible to selectively use a learned model obtained by learning for each part to be imaged. Specifically, a first trained model obtained using learning data including a first imaging region (lung, eye to be examined, etc.) and a learning model including a second imaging region different from the first imaging region. The second trained model obtained using the data may include a selection means (not shown) for selecting one of a plurality of trained models including the second trained model obtained using the data. At this time, the image processing unit 101-04 may include a control unit (not shown) that performs additional learning on the selected trained model. The control means searches for data in which a photographed region corresponding to the selected trained model and a photographed image of the photographed region are paired, and uses the data obtained by the search as learning data. This learning can be performed as additional learning on the selected trained model. Note that the imaged region corresponding to the selected learned model may be obtained from information in the header of the data, or may be manually input by the examiner. Further, the data search may be performed via a network, for example, from a server in an external facility such as a hospital or research institute. Thereby, additional learning can be efficiently performed for each imaged body part using the photographed images of the body part corresponding to the trained model.

Note that the selection means and the control means may be constituted by a software module executed by a processor such as a CPU or an MPU of the image processing section 101-04. Furthermore, the selection means and the control means may be constituted by a circuit that performs a specific function, such as an ASIC, or an independent device.

Furthermore, when acquiring learning data for additional learning via a network from a server in an external facility such as a hospital or research institute, it is desirable to reduce reliability degradation due to tampering or system trouble during additional learning. Therefore, the validity of the learning data for additional learning may be detected by checking the consistency using a digital signature or hashing. Thereby, learning data for additional learning can be protected. At this time, if the validity of the learning data for additional learning cannot be detected as a result of checking the consistency using digital signatures and hashing, a warning to that effect will be issued and additional learning will be performed using that learning data. do not have. Note that the server may take any form, such as a cloud server, fog server, or edge server, regardless of its installation location.

(Modification 6)
In the various embodiments and modified examples described above, the instructions from the examiner may be not only manual instructions (for example, instructions using a user interface or the like) but also instructions by voice or the like. At this time, for example, a machine learning engine including a speech recognition engine (speech recognition model, trained model for speech recognition) obtained by machine learning may be used. Further, the manual instruction may be an instruction by inputting characters using a keyboard, touch panel, or the like. At this time, for example, a machine learning engine including a character recognition engine (character recognition model, trained model for character recognition) obtained by machine learning may be used. Further, the instruction from the examiner may be an instruction using a gesture or the like. At this time, a machine learning engine including a gesture recognition engine (gesture recognition model, trained model for gesture recognition) obtained by machine learning may be used.

Further, the instruction from the examiner may be the result of detecting the examiner's line of sight on the display screen of the display unit 104 or the like. The line of sight detection result may be, for example, a pupil detection result using a moving image of the examiner taken from the periphery of the display screen on the display unit 104. At this time, the pupil detection from the moving image may be performed using the object recognition engine as described above. Further, the instructions from the examiner may be instructions based on brain waves, weak electrical signals flowing through the body, or the like.

In such a case, for example, the learning data may be text data or audio data (waveform data) indicating instructions for displaying the results of processing the various trained models as described above, and The learning data may be an execution command for actually displaying the results of model processing on the display unit 104 as correct data. In addition, as input data, for example, character data or voice data indicating instructions for displaying a high-quality image obtained by a trained model for high-quality images are used as input data, and an execution command for displaying a high-quality image and a high-quality image are used as input data. The learning data may be the correct data that is an execution command for changing the image quality button to the active state. Of course, any learning data may be used as long as the instruction content and the execution command content indicated by character data or audio data correspond to each other, for example. Furthermore, audio data may be converted into character data using an acoustic model, a language model, or the like. Furthermore, processing may be performed to reduce noise data superimposed on audio data using waveform data obtained by a plurality of microphones. Further, instructions using text or voice, etc., and instructions using a mouse, touch panel, etc. may be selectable in accordance with instructions from the examiner. Further, the device may be configured to be able to select whether to turn on or off instructions using text, voice, etc. in accordance with instructions from the examiner.

Here, machine learning includes deep learning as described above, and for example, a recurrent neural network (RNN) can be used for at least one layer of a multilayer neural network. Here, as an example of a machine learning model according to this modification, an RNN, which is a neural network that handles time-series information, will be described with reference to FIGS. 9(a) and (b). Further, a long short-term memory (hereinafter referred to as LSTM), which is a type of RNN, will be explained with reference to FIGS. 10(a) and (b).

FIG. 9(a) shows the structure of RNN, which is a machine learning model. The RNN 3520 has a loop structure in its network, inputs data x ^t 3510 at time t, and outputs data h ^t 3530. Since the RNN 3520 has a loop function in the network, it is possible to take over the state at the current time to the next state, so it can handle time-series information. FIG. 9(b) shows an example of input/output of the parameter vector at time t. The data x ^t 3510 includes N pieces of data (Params1 to ParamsN). Furthermore, the data h ^t 3530 output from the RNN 3520 includes N pieces of data (Params1 to ParamsN) corresponding to the input data.

However, since RNN cannot handle long-term information during error backpropagation, LSTM is sometimes used. LSTM can learn long-term information by providing a forgetting gate, an input gate, and an output gate. Here, the structure of LSTM is shown in FIG. 10(a). In the LSTM 3540, the information that the network takes over at the next time t is the internal state c ^t-1 of the network called a cell and the output data h ^t-1 . Note that lowercase letters (c, h, x) in the figure represent vectors.

Next, details of the LSTM3540 are shown in FIG. 10(b). In FIG. 10(b), FG is a forgetting gate network, IG is an input gate network, and OG is an output gate network, each of which is a sigmoid layer. Therefore, a vector in which each element has a value between 0 and 1 is output. The forgetting gate network FG determines how much past information to retain, and the input gate network IG determines which value to update. CU is a cell update candidate network and an activation function tanh layer. This creates a vector of new candidate values that is added to the cell. The output gate network OG selects the elements of the cell candidates and selects how much information to convey at the next time.

Note that since the LSTM model described above is a basic form, it is not limited to the network shown here. For example, the coupling between networks may be changed. Furthermore, instead of LSTM, QRNN (Quasi Recurrent Neural Network) may be used. Furthermore, the machine learning model is not limited to neural networks, and boosting, support vector machines, etc. may also be used. Furthermore, if the examiner's instructions are input by text or voice, techniques related to natural language processing (for example, sequence to sequence) may be applied. Further, a dialogue engine (a dialogue model, a trained model for dialogue) that responds to the examiner by outputting text or voice may be applied.

(Modification 7)
In the various embodiments and modifications described above, high-quality images and the like may be stored in the storage unit 101-02 according to instructions from the examiner. At this time, after receiving instructions from the examiner to save high-quality images, etc., when registering the file name, select any part of the file name (for example, the first part, the last part, etc.) as the recommended file name. If the file name contains information (e.g., characters) indicating that the image was generated by processing using a trained model for high image quality (high image quality processing) in The information may be displayed in an editable state as required.

In addition, when displaying a high-quality image on the display unit 104 on various display screens such as a report screen, the displayed image may be a high-quality image generated by processing using a trained model for improving image quality. A display indicating that this is the case may be displayed together with the high-quality image. In this case, the display allows the examiner to easily identify that the displayed high-quality image is not the image obtained by photography, which reduces misdiagnosis and improves diagnostic efficiency. be able to. Note that the display indicating that the image is a high-quality image generated by processing using a trained model for high-quality image is a display that allows the input image to be distinguished from the high-quality image generated by the processing. It may take any form. Furthermore, in addition to processing using trained models for high image quality, processing using various trained models such as those described above can also be performed using the results generated by processing using that type of trained model. An indication may be displayed along with the result.

At this time, the display screen such as the report screen may be stored in the storage unit 101-02 as image data in accordance with instructions from the examiner. For example, a report screen may be stored as a single image in which high-quality images are lined up with a display indicating that these images are high-quality images generated by processing using a trained model for high-quality images. It may be stored in the section 101-02.

In addition, regarding the display indicating that the image is a high-quality image generated by processing using a trained model for high image quality, what kind of training data was used to train the trained model for high image quality? An indication indicating whether there is a file may be displayed on the display unit 104. The display may include an explanation of the types of the input data and correct data of the learning data, and any display related to the correct data such as the imaged body part included in the input data and the correct data. Note that not only processing using a trained model for high image quality, but also processing using various trained models such as those mentioned above, depends on what kind of training data that type of trained model is used for learning. An indication may be displayed on the display unit 104 to indicate whether or not the information is correct.

Additionally, information indicating that the image is generated by processing using a trained model for high image quality (for example, text) is displayed or saved in a state superimposed on the high-quality image, etc. It's okay. At this time, the location to be superimposed on the image may be any region (for example, the edge of the image) that does not overlap with the region in which the region of interest to be photographed is displayed. Alternatively, a non-overlapping area may be determined and the area may be overlapped with the determined area.

In addition, if the default setting is such that the high-quality button is activated (high-quality processing is turned on) as the initial display screen of the report screen, high-quality images will be displayed in response to instructions from the examiner. The report image corresponding to the report screen including the above may be configured to be transmitted to a server such as the external storage unit 102. In addition, if the default setting is such that the high image quality button is activated, it will be possible to change the image quality at the end of the examination (for example, when the imaging confirmation screen or preview screen is changed to the report screen in response to instructions from the examiner). case), the report image corresponding to the report screen including a high-quality image etc. may be configured to be (automatically) transmitted to the server. At this time, various settings in the default settings (for example, the depth range for generating the En-Face image on the initial display screen of the report screen, whether or not the analysis map is superimposed, whether or not it is a high-quality image, the display screen for follow-up observation) The report image generated based on at least one setting such as whether or not the report image is sent to the server may be configured.

(Modification 8)
In the various embodiments and modifications described above, images obtained with the first type of trained model among the various trained models described above (e.g., high-quality images, analysis maps, etc. images, images showing object recognition results, images showing segmentation results) may be input to a second type of trained model that is different from the first type. At this time, the configuration may be such that results (for example, analysis results, diagnosis results, object recognition results, and segmentation results) are generated by processing the second type of learned model.

Furthermore, among the various trained models described above, the results of the processing of the first type of trained model (for example, analysis results, diagnosis results, object recognition results, segmentation results) are used to generate the first type of trained model. An image to be input to a second type of trained model different from the first type may be generated from an image input to the trained model. At this time, the generated image is highly likely to be an image suitable for processing by the second type of trained model. For this reason, images obtained by inputting the generated images to the second type of trained model (for example, high-quality images, images showing analysis results such as analysis maps, images showing object recognition results, and segmentation results) The accuracy of images shown) can be improved.

Further, the various trained models described above may be trained models obtained by learning training data including two-dimensional medical images of the subject, or may be trained models obtained by learning training data including two-dimensional medical images of the subject. The model may be a trained model obtained by learning training data including images.

Further, a similar case image search may be performed using an external database stored in a server or the like using the analysis results, diagnosis results, etc. obtained by processing the learned model as described above as a search key. In addition, in cases where multiple images stored in a database have already been managed with the feature values of each of the multiple images attached as additional information through machine learning, etc., the image itself can be used as the search key. A similar case image search engine (similar case image search model, trained model for similar case image search) may be used. For example, the image processing unit 101-04 uses a trained model for similar case image retrieval (different from a trained model for high image quality) to process at least one medical image among the plurality of medical images to be blended. Similar case images related to the at least one medical image can be searched from the images. Further, for example, the display control unit 101-05 can cause the display unit 104 to display a similar case image obtained from the at least one medical image using a learned model for similar case image search.

(Modification 9)
The motion contrast data generation processing in the various embodiments and modifications described above is not limited to a configuration in which it is performed based on the brightness value of a tomographic image. The various processes described above are performed on tomographic data including an interference signal acquired by the tomographic imaging apparatus 100, a signal obtained by performing Fourier transform on the interference signal, a signal obtained by performing arbitrary processing on the signal, and a tomographic image based on these. may be applied to In these cases as well, the same effects as the above configuration can be achieved. For example, although a fiber optic system using an optical coupler is used as the splitting means, a spatial optical system using a collimator and a beam splitter may also be used. Further, the configuration of the tomographic image capturing apparatus 100 is not limited to the above configuration, and a part of the configuration included in the tomographic image capturing apparatus 100 may be configured separately from the tomographic image capturing apparatus 100. Further, in the above configuration, a Michelson interferometer configuration is used as the interference optical system of the tomographic imaging apparatus 100, but the configuration of the interference optical system is not limited to this. For example, the interference optical system of the tomographic imaging apparatus 100 may have a Mach-Zehnder interferometer configuration. Furthermore, although a spectral domain OCT (SD-OCT) device using an SLD as a light source is described as an OCT device, the configuration of the OCT device is not limited to this. For example, the present invention can be applied to any other type of OCT device, such as a wavelength swept OCT (SS-OCT) device that uses a wavelength swept light source that can sweep the wavelength of emitted light. Further, the present invention can also be applied to a Line-OCT device (or SS-Line-OCT device) using line light. Further, the present invention can also be applied to a Full Field-OCT device (or SS-Full Field-OCT device) using area light. Further, the image processing unit 101-04 acquires interference signals acquired by the tomographic imaging apparatus 100, three-dimensional tomographic images generated by the image processing unit 101-04, etc. However, the configuration for acquiring these signals and images is not limited to this. For example, the image processing unit 101-04 may acquire these signals from a server or imaging device connected via a LAN, WAN, or the Internet.

Note that the trained model can be provided in the image processing unit 101-04. The trained model can be configured of, for example, a software module executed by a processor such as a CPU. Further, the trained model may be provided in another server or the like connected to the image processing unit 101-04. In this case, the image processing unit 101-04 can perform high image quality processing using the trained model by connecting to a server provided with the trained model via any network such as the Internet.

Further, in the process of generating motion contrast data, a high image quality engine can be applied as appropriate. For example, the image quality of the tomographic image before determining the decorrelation value may be increased in advance by using a high-quality engine prepared for tomographic images. Further, when the NOR is 3 or more, at least two pieces of motion contrast data can be generated, and it is also possible to improve the image quality by averaging a plurality of pieces of motion contrast data. At this time, the image quality of each motion contrast data before the averaging process may be enhanced by an image quality enhancement engine in advance. Alternatively, a high image quality engine may be applied to the averaged motion contrast data. Furthermore, by converting the motion contrast data into volume data (three-dimensional motion contrast data), the volume data can be improved in image quality using a high-quality engine for three-dimensional data configured in advance using the well-known 3D-UNet. good. In addition, when the NOR is 3 or more, at least two three-dimensional motion contrast data can be generated, and the final volume data may be obtained by averaging them. At this time, a high image quality engine may be applied to at least one of the volume data before averaging and the volume data after averaging processing. Furthermore, after each OCTA front image is generated from a plurality of volume data, it is also possible to average the OCTA front images. Similarly, the image quality improvement engine can be applied to at least one of the OCTA front image before averaging and the OCTA front image after averaging processing. In this way, when generating an OCTA frontal image from motion contrast data, various transformations are possible, especially when the NOR is 3 or more, and the high image quality engine can use any It may be applied to such data.

(Modification 10)
Images processed by the image processing apparatus 101 or the image processing method according to the various embodiments and modifications described above include medical images acquired using any modality (imaging device, imaging method). The medical images to be processed can include medical images acquired by any imaging device or the like, and images created by the image processing apparatus 101 or the image processing method according to the embodiments and modifications described above.

Furthermore, the medical image to be processed is an image of a predetermined region of a subject (subject), and the image of the predetermined region includes at least a part of the predetermined region of the subject. Further, the medical image may include other parts of the subject. Further, the medical image may be a still image or a moving image, and may be a black and white image or a color image. Furthermore, the medical image may be an image representing the structure (morphology) of a predetermined region, or an image representing its function. Images representing functions include, for example, images representing blood flow dynamics (blood flow volume, blood flow velocity, etc.) such as OCTA images, Doppler OCT images, fMRI images, and ultrasound Doppler images. The predetermined parts of the subject may be determined depending on the object to be imaged, and may include human eyes (eyes to be examined), organs such as the brain, lungs, intestines, heart, pancreas, kidneys, and liver, head, chest, Includes arbitrary parts such as legs and arms.

Furthermore, the medical image may be a tomographic image of the subject or a frontal image. The frontal image is, for example, a frontal image of the fundus, a frontal image of the anterior segment of the eye, a fluorescently photographed fundus image, or at least a partial range in the depth direction of the object to be photographed regarding data acquired by OCT (three-dimensional OCT data). Contains an En-Face image generated using the data. The En-Face image is an OCTA En-Face image (motion contrast frontal image) generated using data of at least a partial range in the depth direction of the imaging target for three-dimensional OCTA data (three-dimensional motion contrast data). ) is also fine. Furthermore, three-dimensional OCT data and three-dimensional motion contrast data are examples of three-dimensional medical image data.

Here, the motion contrast data is data indicating changes among a plurality of volume data obtained by controlling the measurement light to scan the same area (same position) of the eye to be examined multiple times. At this time, the volume data is composed of a plurality of tomographic images obtained at different positions. Then, by obtaining data indicating changes between a plurality of tomographic images obtained at substantially the same position at each different position, motion contrast data can be obtained as volume data. Note that the motion contrast front image is also called an OCTA front image (OCTA En-Face image) related to OCT angiography (OCTA) that measures the movement of blood flow, and the motion contrast data is also called OCTA data. Motion contrast data can be obtained, for example, as a decorrelation value, a variance value, or a value obtained by dividing the maximum value by the minimum value (maximum value/minimum value) between two tomographic images or their corresponding interference signals. , may be determined by any known method. At this time, the two tomographic images can be obtained, for example, by controlling the measuring light to scan the same area (same position) of the eye to be examined multiple times.

Further, the En-Face image is, for example, a front image generated by projecting data in the range between two layer boundaries in the XY directions. At this time, the frontal image is data that corresponds to at least a partial depth range of volume data (three-dimensional tomographic image) obtained using optical interference, and that corresponds to a depth range determined based on two reference planes. It is generated by projecting or integrating onto a two-dimensional plane. The En-Face image is a frontal image generated by projecting data corresponding to a depth range determined based on the detected retinal layer out of the volume data onto a two-dimensional plane. Note that as a method of projecting data corresponding to a depth range determined based on two reference planes onto a two-dimensional plane, for example, a representative value of data within the depth range is set as a pixel value on a two-dimensional plane. method can be used. Here, the representative value can include a value such as an average value, a median value, or a maximum value of pixel values within a range in the depth direction of a region surrounded by two reference planes. Further, the depth range related to the En-Face image is, for example, a range that includes a predetermined number of pixels in a deeper direction or a shallower direction with one of the two layer boundaries regarding the detected retinal layer as a reference. Good too. Furthermore, the depth range related to the En-Face image may be, for example, a range changed (offset) from the range between the two layer boundaries regarding the detected retinal layers in accordance with the operator's instructions. good.

Further, the photographing device is a device for photographing images used for diagnosis. The imaging device is, for example, a device that obtains an image of a predetermined region of a subject by irradiating the subject with radiation such as light, X-rays, electromagnetic waves, or ultrasound, or a device that detects radiation emitted from the subject. includes a device for obtaining an image of a predetermined region. More specifically, the imaging devices according to the various embodiments and modifications described above include at least an X-ray imaging device, a CT device, an MRI device, a PET device, a SPECT device, an SLO device, an OCT device, an OCTA device, and a fundus. Includes cameras, endoscopes, etc.

Note that the OCT device may include a time domain OCT (TD-OCT) device and a Fourier domain OCT (FD-OCT) device. Further, the Fourier domain OCT device may include a spectral domain OCT (SD-OCT) device or a wavelength swept OCT (SS-OCT) device. Furthermore, the SLO device and OCT device may include a wavefront compensated SLO (AO-SLO) device using a wavefront adaptive optical system, a wavefront compensated OCT (AO-OCT) device, and the like. Furthermore, the SLO device and OCT device may include a polarization SLO (PS-SLO) device, a polarization OCT (PS-OCT) device, etc. for visualizing information regarding polarization phase difference and depolarization.

[Other embodiments]
Further, the present invention can also be realized by executing the following processing. That is, software (programs) that implement one or more functions of the various embodiments and modifications described above is supplied to a system or device via a network or various storage media, and the computer (or CPU or (MPU, etc.) reads and executes a program.

Furthermore, the present invention provides software (programs) for realizing one or more functions of the various embodiments and modifications described above to a system or device via a network or a storage medium, and a computer of the system or device This can also be achieved by reading and executing a program. A computer has one or more processors or circuits and may include separate computers or a network of separate processors or circuits for reading and executing computer-executable instructions.

The processor or circuit may include a central processing unit (CPU), microprocessing unit (MPU), graphics processing unit (GPU), application specific integrated circuit (ASIC), or field programmable gateway (FPGA). The processor or circuit may also include a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU).

Claims (18)

acquisition means for acquiring an OCT front image and an OCTA front image in mutually corresponding regions and mutually corresponding depth ranges in the subject using an optical coherence tomography;
A trained model for high image quality obtained by learning an OCTA frontal image of a subject, the high-quality OCTA frontal image obtained by inputting the acquired OCTA frontal image as input data to the trained model. a blend processing unit that generates a blend image by performing blend processing using the image and the obtained OCT frontal image using a transmittance that can be changed according to an instruction from an examiner ;
display control means for controlling a display means to display the generated blended image and display information for accepting instructions from an examiner regarding changes in the transmittance;
a selection means for selecting the type of analysis processing for the high-quality OCTA frontal image in accordance with an instruction from an examiner ;
The display control means is configured to display a region of interest set in the displayed blended image in accordance with an instruction from the examiner , and to display a region of interest that is set in the displayed blended image in accordance with an instruction from the examiner. The results of the selected type of analysis processing are displayed, and when an instruction from the examiner regarding changing from the high-quality OCTA frontal image to the acquired OCTA frontal image is received, the results of the analysis processing are displayed. An image processing device that controls the display means to change the display to a display of the result of the selected type of analysis processing for the set region of interest in the acquired OCTA frontal image. At least one of the OCT frontal image and the OCTA frontal image has an attribute indicating two or more classifications for each pixel,
The blend processing means generates a blend image based on the attribute and the transmittance ,
The image having the attribute has an attribute that is classified based on the pixel value of the image, or an attribute that is set corresponding to a preset partial area of the image,
The image processing device according to claim 1, wherein the attribute is set based on whether the pixel value exceeds a threshold value. The image processing apparatus according to claim 2, wherein when the image having the attribute is an OCTA frontal image, the attribute is an attribute based on the probability that a pixel is a blood vessel. The image processing according to claim 2 or 3, wherein the blend processing means fixes the transmittance corresponding to the pixel of the image having the attribute to 0 or 1 when the attribute is an attribute indicating a predetermined classification. Device. acquisition means for acquiring an OCT front image and an OCTA front image in mutually corresponding regions and mutually corresponding depth ranges in the subject using an optical coherence tomography;
A trained model for high image quality obtained by learning an OCTA frontal image of a subject, the high-quality OCTA frontal image obtained by inputting the acquired OCTA frontal image as input data to the trained model. Displaying a blended image obtained by performing a blending process using the image and the acquired OCT frontal image using a transmittance that can be changed according to instructions from the examiner , and changing the transmittance. display control means for controlling the display means to display information that accepts instructions from the examiner regarding the
The display control means is configured to display a region of interest set in the displayed blended image according to an instruction from the examiner , and to display a region of interest corresponding to the set region of interest in the high-quality OCTA frontal image . Displays the results of the type of analysis processing selected in response to instructions from the examiner among the types of analysis processing performed, and displays the results of the type of analysis processing selected according to instructions from the examiner, and receives information from the examiner regarding changing from the high-quality OCTA frontal image to the acquired OCTA frontal image. If the instruction is received, the display of the result of the analysis process is changed to the display of the result of the selected type of analysis process for the set region of interest in the acquired OCTA frontal image. , an image processing device that controls the display means . 2. The display control means controls the display means so as to change the display of the result of the analysis process to the display of another type of analysis result selected in response to an instruction from the examiner. 6. The image processing device according to any one of 5 to 5. The display control means receives a request from the examiner regarding a change from one display screen to the other of the display screen of the blended image and the display screen of a plurality of OCTA images including the acquired front OCTA image. If the instruction is received, the display state of the high quality OCTA front image is maintained if the high quality OCTA front image is displayed, and the display state of the high quality OCTA front image is maintained if the acquired OCTA front image is displayed. The image processing apparatus according to any one of claims 1 to 6, wherein the display means is controlled to maintain a display state of the OCTA front image. A trained model different from the trained model for high image quality, which uses at least one of an OCT frontal image and an OCTA frontal image of the subject as input data, and calculates the transmittance used in the blending process. A new transmittance is set from at least one of the acquired OCT frontal image and the acquired OCTA frontal image using a trained model obtained by learning with learning data as correct data. The image processing device according to any one of claims 1 to 7 , configured to 9. The image processing apparatus according to claim 8 , wherein the learned model is a learned model obtained by additional learning using learning data whose correct data is a transmittance set in accordance with an instruction from an examiner. According to claim 8 or 9 , the learned model is a learned model obtained by additional learning using learning data in which the correct data is the transmittance changed from the new transmittance in response to an instruction from the examiner. The image processing device described. 11. The blending process is performed by weighted averaging processing of pixel values at mutually corresponding positions of the acquired OCT frontal image and the acquired OCTA frontal image. The image processing device described. The display control means is configured to display an analysis result generated using a trained model different from the trained model for high image quality, the analysis result regarding the high quality OCTA front image, and the above trained model for high image quality. A diagnosis result generated using a trained model different from the trained model for the high-quality OCTA frontal image, and a trained model different from the trained model for high image quality. The object detection results are the object detection results generated using the high quality OCTA frontal image, and the segmentation results generated using a trained model different from the trained model for high image quality. A segmentation result related to the high-quality OCTA frontal image and a similar case image generated using a trained model different from the trained model for high image quality, which is related to the high-quality OCTA frontal image. 12. The image processing apparatus according to claim 1, wherein at least one of the similar case images is displayed on the display means. The display control means provides information regarding the difference between the high-quality OCTA frontal image and an image generated using an adversarial generative network or an autoencoder to which the high-quality OCTA frontal image is input as input data. 13. The image processing apparatus according to claim 1, wherein the image processing apparatus displays information on the abnormal region on the display means. The instruction is a trained model different from the trained model for image quality improvement, and is one of a trained model for character recognition, a trained model for speech recognition, and a trained model for gesture recognition. The image processing device according to any one of claims 1 to 13 , wherein the information is obtained using at least one trained model. When the display control means receives an instruction from the examiner regarding changing from the high-quality OCTA frontal image to the acquired OCTA frontal image, the display control means further controls display of the blended image from the acquired OCTA frontal image. 15. The method according to claim 1, wherein the display means is controlled to change the display to a blended image obtained by performing a blending process based on the transmittance using an OCTA frontal image and the obtained OCT frontal image. The image processing device according to any one of the items. The display control means controls the display means to display information that accepts an instruction from an examiner regarding changing the depth range of each of the acquired OCT frontal image and the acquired OCTA frontal image. The image processing device according to any one of claims 1 to 15. Obtaining an OCT front image and an OCTA front image in mutually corresponding regions and mutually corresponding depth ranges in the subject using an optical coherence tomography;
A trained model for high image quality obtained by learning an OCTA frontal image of a subject, the high-quality OCTA frontal image obtained by inputting the acquired OCTA frontal image as input data to the trained model. Displaying a blended image obtained by performing a blending process using the image and the acquired OCT frontal image using a transmittance that can be changed according to instructions from the examiner , and changing the transmittance. controlling the display means to display information that accepts instructions from the examiner regarding the
A region of interest that is set in the displayed blended image according to instructions from the examiner , and the type of analysis processing for the set region of interest in the high-quality OCTA frontal image . Displaying the results of the type of analysis processing selected in response to instructions from the examiner, and receiving instructions from the examiner regarding changing from the high-quality OCTA frontal image to the acquired OCTA frontal image; controlling the display means to change the display of the result of the analysis process to the display of the result of the selected type of analysis process for the set region of interest in the acquired OCTA frontal image; The process of
image processing methods including; A program that causes a computer to execute each step of the image processing method according to claim 17 .

Images