JP7374615B2 - Information processing device, information processing method and program - Google Patents

Description

The disclosure of this specification relates to an information processing device, an information processing method, and a program.

OCT angiography (hereinafter referred to as OCTA), which non-invasively depicts fundus blood vessels using optical coherence tomography (OCT), is known. In OCTA, the same position is scanned multiple times with measurement light (cluster scanning) to obtain multiple OCT tomographic images. Based on the plurality of OCT tomographic images (clusters), motion contrast data obtained from the interaction between the displacement of the red blood cells and the measurement light is visualized as an OCTA image.

Patent Document 1 discloses that three-dimensional blood vessel measurement results are obtained by processing three-dimensional motion contrast data.

JP2017-77413A

However, for example, noise contained in the motion contrast data makes it difficult to obtain a three-dimensional blood vessel region with high accuracy, which may reduce the accuracy of blood vessel measurement results.

One of the objectives of the disclosure of this specification is to accurately obtain a desired region such as a three-dimensional blood vessel region.

It should be noted that the present disclosure is not limited to the above-mentioned objects, and the disclosure of this specification also provides effects derived from each configuration shown in the detailed description of the invention described below, which cannot be obtained by conventional techniques. It can be positioned as one of the other purposes.

One of the information processing devices disclosed in this specification is
a high image quality means for increasing the image quality of three-dimensional motion contrast data of the fundus;
acquisition means for acquiring information regarding a three-dimensional blood vessel region in at least a portion of the high-quality three-dimensional motion contrast data;
A blood vessel measurement value in a three-dimensional space is calculated using the information regarding the three-dimensional blood vessel region , and a blood vessel measurement value in the two-dimensional region is calculated from two-dimensional motion contrast data obtained by projecting the three-dimensional motion contrast data in the depth direction. Calculation means for calculating blood vessel measurement values;
and means for associating the calculated blood vessel measurement value with identification information that can identify whether it is a blood vessel measurement value in the three-dimensional space or a blood vessel measurement value in the two-dimensional area.

According to one of the disclosures of this specification, a desired region such as a three-dimensional blood vessel region can be obtained with high accuracy.

FIG. 1 is a block diagram showing an example of the configuration of an image processing device according to a first embodiment. FIG. 1 is a diagram illustrating an example of an image processing system according to a first embodiment and a second embodiment. 3 is a flowchart illustrating an example of a processing procedure of the image processing system according to the first embodiment. FIG. 3 is a diagram illustrating an example of a scanning method for OCTA imaging in the first embodiment and the second embodiment. FIG. 7 is a diagram illustrating an example of processing executed in S307 of the first embodiment. FIG. 6 is a diagram illustrating an example of processing executed in S308 of the first embodiment. FIG. 3 is a diagram illustrating an example of a photographing screen displayed on a display means in the first embodiment. FIG. 6 is a diagram illustrating an example of image processing contents in S304 and an example of a report screen displayed on a display means in S305 of the first embodiment. FIG. 6 is a diagram illustrating an example of a measurement operation screen displayed on a display means in the first embodiment and the second embodiment, and an example of a measurement report screen displayed in S1210 of the second embodiment. FIG. 6 is a diagram illustrating an example of an operation procedure and an example of image processing contents to be executed when a user manually corrects a blood vessel region specified in the first embodiment and the second embodiment. FIG. 2 is a block diagram illustrating an example of the configuration of an image processing device according to a second embodiment. 7 is a flowchart illustrating an example of a processing procedure of an image processing system according to a second embodiment. FIG. 7 is a diagram illustrating an example of image quality improvement processing executed in S1205 of the second embodiment. FIG. 7 is a diagram illustrating an example of measurement processing executed in S1208 of the second embodiment. An example of the configuration of a neural network used as a machine learning model according to modification 16 is shown. An example of the configuration of a neural network used as a machine learning model according to modification 16 is shown. An example of a user interface according to modification 11 is shown.

[First embodiment]
The image processing device according to the present embodiment acquires a three-dimensional composite motion contrast image from the subject's eye including choroidal neovascularization (CNV) in the outer retinal layer, and binarizes it by applying a three-dimensional blood vessel enhancement filter. For example, a blood vessel region containing CNV is specified in three dimensions. Furthermore, a case will be described in which the volume of the specified CNV region is calculated and accurately measured.

An image processing system including an image processing apparatus according to this embodiment will be described below with reference to the drawings.

FIG. 2 is a diagram showing the configuration of an image processing system 10 including an image processing apparatus 101 according to this embodiment. As shown in FIG. 2, in the image processing system 10, an image processing device 101 is connected to a tomographic imaging device 100 (also referred to as OCT), an external storage section 102, an input section 103, and a display section 104 via an interface. It is composed of:

The tomographic image capturing apparatus 100 is a device that captures tomographic images of the eye. In this embodiment, SD-OCT is used as the tomographic imaging apparatus 100. The configuration is not limited to this, and may be configured using SS-OCT, for example.

In FIG. 2(a), a measurement optical system 100-1 is an optical system for acquiring an anterior segment image, an SLO fundus image of the eye to be examined, and a tomographic image. The stage section 100-2 allows the measurement optical system 100-1 to move back and forth and left and right. The base portion 100-3 incorporates a spectrometer to be described later.

The image processing device 101 is a computer that controls the stage section 100-2, controls alignment operations, reconstructs tomographic images, and the like. The external storage unit 102 stores tomographic imaging programs, patient information, imaging data, image data of past examinations, measurement data, and the like.

The input unit 103 gives instructions to the computer, and specifically includes a keyboard and a mouse. The display unit 104 includes, for example, a monitor.

(Configuration of tomographic imaging device)
The configuration of the measurement optical system and spectrometer in the tomographic imaging apparatus 100 of this embodiment will be explained using FIG. 2(b).

First, the inside of the measurement optical system 100-1 will be explained. An objective lens 201 is placed facing the eye 200 to be examined, and a first dichroic mirror 202 and a second dichroic mirror 203 are placed on its optical axis. These dichroic mirrors branch the optical path into an optical path 250 for the OCT optical system, an optical path 251 for the SLO optical system and fixation lamp, and an optical path 252 for anterior eye observation for each wavelength band.

The optical path 251 for the SLO optical system and fixation lamp includes an SLO scanning means 204, lenses 205 and 206, a mirror 207, a third dichroic mirror 208, an APD (Avalanche Photodiode) 209, an SLO light source 210, and a fixation lamp 211. ing.

The mirror 207 is a prism in which a perforated mirror or a hollow mirror is vapor-deposited, and separates the illumination light from the SLO light source 210 and the return light from the eye to be examined. The third dichroic mirror 208 separates the optical path of the SLO light source 210 and the optical path of the fixation lamp 211 for each wavelength band.

The SLO scanning means 204 scans the eye 200 with light emitted from the SLO light source 210, and is composed of an X scanner that scans in the X direction and a Y scanner that scans in the Y direction. In this embodiment, since it is necessary to perform high-speed scanning, the X scanner is configured with a polygon mirror, and the Y scanner is configured with a galvanometer mirror.

The lens 205 is driven by a motor (not shown) to focus the SLO optical system and the fixation lamp 211. SLO light source 210 generates light at a wavelength around 780 nm. APD 209 detects return light from the eye to be examined. The fixation lamp 211 generates visible light to encourage the subject to fixate.

The light emitted from the SLO light source 210 is reflected by the third dichroic mirror 208, passes through the mirror 207, passes through the lenses 206 and 205, and is scanned onto the eye 200 by the SLO scanning means 204. After the return light from the eye 200 to be examined returns along the same path as the illumination light, it is reflected by the mirror 207 and guided to the APD 209, whereby an SLO fundus image is obtained.

The light emitted from the fixation lamp 211 passes through the third dichroic mirror 208 and the mirror 207, passes through the lenses 206 and 205, and forms a predetermined shape at an arbitrary position on the eye 200 by the SLO scanning means 204. Encourage the subject to fixate.

Lenses 212 and 213, a split prism 214, and a CCD 215 for anterior eye observation that detects infrared light are arranged in the optical path 252 for anterior eye observation. This CCD 215 is sensitive to the wavelength of irradiation light for anterior ocular segment observation (not shown), specifically around 970 nm. The split prism 214 is arranged at a position conjugate with the pupil of the eye to be examined 200, and measures the distance in the Z-axis direction (optical axis direction) of the measurement optical system 100-1 to the eye to be examined 200 as a split image of the anterior segment of the eye. Can be detected.

The optical path 250 of the OCT optical system constitutes the OCT optical system as described above, and is for capturing a tomographic image of the eye 200 to be examined. More specifically, the purpose is to obtain an interference signal for forming a tomographic image. The XY scanner 216 is for scanning light on the eye 200 to be examined, and although it is shown as a single mirror in FIG. 2(b), it is actually a galvanometer mirror that scans in two directions of the X and Y axes.

Of the lenses 217 and 218, the lens 217 is driven by a motor (not shown) in order to focus the light from the OCT light source 220 emitted from the fiber 224 connected to the optical coupler 219 onto the eye 200 to be examined. By this focusing, the return light from the eye 200 to be examined is simultaneously formed into a spot image and incident on the tip of the fiber 224. Next, the configuration of the optical path from the OCT light source 220, the reference optical system, and the spectrometer will be explained. 220 is an OCT light source, 221 is a reference mirror, 222 is a dispersion compensating glass, 223 is a lens, 219 is an optical coupler, 224 to 227 are single mode optical fibers connected and integrated with the optical coupler, and 230 is a spectrometer. .

These configurations constitute a Michelson interferometer. The light emitted from the OCT light source 220 passes through the optical fiber 225 and is split via the optical coupler 219 into measurement light on the optical fiber 224 side and reference light on the optical fiber 226 side. The measurement light is irradiated onto the eye 200 to be observed, which is the object of observation, through the optical path of the aforementioned OCT optical system, and reaches the optical coupler 219 through the same optical path due to reflection and scattering by the eye 200 to be examined.

On the other hand, the reference light reaches the reference mirror 221 and is reflected through the optical fiber 226, the lens 223, and the dispersion compensating glass 222 inserted to match the wavelength dispersion of the measurement light and the reference light. Then, it returns along the same optical path and reaches the optical coupler 219.
The measurement light and the reference light are combined by the optical coupler 219 to become interference light.

Here, interference occurs when the optical path length of the measurement light and the optical path length of the reference light become almost the same. The reference mirror 221 is held adjustable in the optical axis direction by a motor and a drive mechanism (not shown), and can match the optical path length of the reference light to the optical path length of the measurement light. The interference light is guided to a spectrometer 230 via an optical fiber 227.

Further, polarization adjustment units 228 and 229 are provided in the optical fibers 224 and 226, respectively, and perform polarization adjustment. These polarization adjustment sections have several sections in which optical fibers are routed in loops. By rotating this loop-shaped portion around the longitudinal direction of the fiber, it is possible to twist the fiber and adjust the polarization states of the measurement light and the reference light to match each other.

The spectrometer 230 includes lenses 232 and 234, a diffraction grating 233, and a line sensor 231. The interference light emitted from the optical fiber 227 becomes parallel light through the lens 234 , is separated by the diffraction grating 233 , and is imaged onto the line sensor 231 by the lens 232 .

Next, the surroundings of the OCT light source 220 will be described. The OCT light source 220 is an SLD (Super Luminescent Diode), which is a typical low coherent light source. The center wavelength is 855 nm, and the wavelength bandwidth is approximately 100 nm. Here, the bandwidth is an important parameter because it affects the resolution of the obtained tomographic image in the optical axis direction.

As for the type of light source, SLD is selected here, but it is sufficient that it can emit low coherent light, and ASE (Amplified Spontaneous Emission) or the like may be used. Considering that the eye is to be measured, near-infrared light is suitable as the center wavelength. Furthermore, since the center wavelength affects the lateral resolution of the obtained tomographic image, it is desirable that the center wavelength be as short as possible. For both reasons, the center wavelength was set to 855 nm.

In this embodiment, a Michelson interferometer is used as the interferometer, but a Mach-Zehnder interferometer may also be used. Depending on the difference in light intensity between the measurement light and the reference light, it is desirable to use a Mach-Zehnder interferometer when the difference in light intensity is large, and to use a Michelson interferometer when the difference in light intensity is relatively small.

(Configuration of image processing device)
The configuration of an image processing device (information processing device) 101 of this embodiment will be explained using FIG. 1.

The image processing device 101 is a personal computer (PC) connected to the tomographic imaging device 100, and includes an image acquisition section 101-01, a storage section 101-02, an imaging control section 101-03, an image processing section 101-04, and a display. It includes a control section 101-05. The image processing device 101 also functions as the arithmetic processing unit CPU executes software modules that implement an image acquisition unit 101-01, a shooting control unit 101-03, an image processing unit 101-04, and a display control unit 101-05. Realize. The software module is stored in the storage unit 101-02, for example. The present invention is not limited to this, and for example, the image processing unit 101-04 may be implemented using dedicated hardware such as an ASIC, and the display control unit 101-05 may be implemented using a dedicated processor such as a GPU that is different from the CPU. It may be realized by Further, the connection between the tomographic imaging apparatus 100 and the image processing apparatus 101 may be configured via a network. Note that there may be a plurality of processors such as a CPU, and there may be one or more memories that store programs executed by the processors.

The image acquisition unit 101-01 acquires signal data of the SLO fundus image and tomographic image photographed by the tomographic imaging apparatus 100. The image acquisition unit 101-01 also includes a tomographic image generation unit 101-11 and a motion contrast data generation unit 101-12. The tomographic image generation unit 101-11 acquires the signal data (interference signal) of the tomographic image taken by the tomographic imaging apparatus 100, generates a tomographic image by signal processing, and stores the generated tomographic image in the storage unit 101-02. Store.

The imaging control unit 101-03 performs imaging control for the tomographic imaging apparatus 100. Imaging control also includes instructing the tomographic imaging apparatus 100 regarding the setting of imaging parameters and instructing the start or end of imaging.

The image processing section 101-04 includes a positioning section 101-41, a composition section 101-42, a correction section 101-43, an image feature acquisition section 101-44, a projection section 101-45, and an analysis section 101-46. The image acquisition unit 101-01 described above is an example of an acquisition means according to the present invention. Furthermore, the synthesizing section 101-42 synthesizes the plurality of motion contrast data generated by the motion contrast data generating section 101-12 based on the alignment parameters obtained by the alignment section 101-41, and generates a synthesized motion contrast image. generate. Here, the synthesizing unit 101-42 is an example of an image quality improvement unit that improves the image quality of three-dimensional motion contrast data. Also in this embodiment, in addition to the processing by the compositing unit 101-42, it is possible to apply, for example, the image quality improvement processing by machine learning in the second embodiment described later as the processing by the image quality improvement means. It is. Further, the correction unit 101-43 performs processing to two-dimensionally or three-dimensionally suppress projection artifacts occurring within the motion contrast image (projection artifacts will be explained in S304). For example, the correction unit 101-43 performs processing to reduce projection artifacts in the synthesized three-dimensional motion contrast data. That is, the correction unit 101-43 corresponds to an example of a processing unit that performs processing to reduce projection artifacts on the synthesized three-dimensional motion contrast data. The image feature acquisition unit 101-44 acquires the layer boundaries of the retina and choroid, the positions of the fovea and the center of the optic disc from the tomographic image. The projection unit 101-45 projects a motion contrast image in a depth range based on the position of the layer boundary acquired by the image feature acquisition unit 101-44, and generates a front motion contrast image. The analysis unit 101-46 includes an emphasis unit 101-461, an extraction unit 101-462, and a measurement unit 101-463, and performs extraction and measurement processing of a blood vessel region from a three-dimensional or frontal motion contrast image. The emphasizing unit 101-461 executes blood vessel emphasizing processing. Further, the extraction unit 101-462 extracts a blood vessel region based on the blood vessel enhanced image. Furthermore, the measurement unit 101-463 calculates measured values such as blood vessel density using the extracted blood vessel region and blood vessel centerline data obtained by thinning the blood vessel region. That is, the image processing device according to the present embodiment can extract a three-dimensional blood vessel region in at least a portion of the three-dimensional motion contrast data with high image quality. Note that the extraction unit 101-462 is an example of an extraction unit that extracts a three-dimensional blood vessel region. Further, the image processing device according to the present embodiment can calculate a blood vessel measurement value using information regarding a three-dimensional blood vessel region in at least a portion of the high-quality three-dimensional motion contrast data. Note that the information regarding the three-dimensional blood vessel region is, for example, position information of the three-dimensional blood vessel region in high-quality three-dimensional motion contrast data. However, the information regarding the three-dimensional blood vessel region may be any information as long as it is possible to calculate blood vessel measurement values from high-quality three-dimensional motion contrast data.

The external storage unit 102 stores information about the eye to be examined (patient's name, age, gender, etc.), photographed images (tomographic images, SLO images, OCTA images), composite images, photographing parameters, and positions of blood vessel regions and blood vessel center lines. Data, measured values, and parameters set by the operator are stored in association with each other. The input unit 103 is, for example, a mouse, a keyboard, a touch operation screen, etc., and the operator issues instructions to the image processing apparatus 101 and the tomographic imaging apparatus 100 via the input unit 103.

Next, the processing procedure of the image processing apparatus 101 of this embodiment will be described with reference to FIG. FIG. 3 is a flowchart showing the flow of operation processing of the entire system in this embodiment.

<Step 301>
By operating the input unit 103, the operator sets the imaging conditions for the OCTA image to be instructed to the tomographic imaging apparatus 100. Note that there are setting items as shown in 3) below as imaging conditions for individual OCTA imaging, and after setting these setting items to the same values as the standard examination, in S302, the same imaging conditions as the standard examination are set. OCTA imaging is repeatedly performed a predetermined number of times. FIG. 4 is a diagram showing an example of a scanning pattern. Figure 4 shows an example of OCTA imaging in which the main scanning direction is the horizontal (x-axis) direction and B scans are performed r times consecutively at each position (yi; 1≦i≦n) in the sub-scanning (y-axis) direction. There is.

Specifically, the procedure consists of 1) selection or registration of an examination set, 2) selection or addition of a scan mode in the selected examination set, and 3) setting of imaging parameters corresponding to the scan mode. In this embodiment, the settings are as follows. Then, in S302, OCTA imaging (under the same imaging conditions) is repeatedly performed a predetermined number of times with appropriate breaks in between.

1) Register Macular Disease test set 2) Select OCTA scan mode 3) Set the following imaging parameters 3-1) Scanning pattern: Small Square
3-2) Scan area size: 3x3mm
3-3) Main scanning direction: horizontal direction 3-4) Scanning interval: 0.01mm
3-5) Fixation light position: fovea 3-6) Number of B scans per cluster: 4
3-7) Coherence gate position: Vitreous body side 3-8) Default display report type: Single examination report Note that an examination set refers to the imaging procedure (including scan mode) set for each examination purpose and the images acquired in each scan mode. Refers to the default display method for OCT images and OCTA images. Further, the above numerical values are just examples, and other numerical values may be used. Here, scanning the same position multiple times in OCTA is called cluster scanning. Note that decorrelation is not calculated using tomographic images between different clusters. Further, the unit in which rescanning is performed by blinking or the like is a cluster unit.

A test set including an OCTA scan mode configured for eyes with macular disease is registered under the name "Macular Disease." The registered test set is stored in the external storage unit 102.

In this embodiment, as shown in the photographing screen 710 of FIG. 7, "Maucular Disease" (711) is selected as the test set and "OCTA" mode 712 is selected as the scan mode.

<Step 302>
The operator operates the input unit 103 and presses a capture start button 713 on the capture screen 710 shown in FIG. 7 to start repeated OCTA capture under the capture conditions specified in S301.

The imaging control unit 101-03 instructs the tomographic imaging apparatus 100 to repeatedly perform OCTA imaging based on the settings instructed by the operator in S301, and the tomographic imaging apparatus 100 acquires the corresponding OCT tomographic image. do.

In addition, in this embodiment, the number of times of repeated imaging in this step is three times. The number of times of repeated imaging is not limited to this, and may be set to an arbitrary number of times. Further, the present invention is not limited to the case where the imaging time interval between repeated imaging is longer than the imaging time interval of the tomographic images in each repeated imaging, but the present invention also includes a case where both are substantially the same.

The tomographic imaging apparatus 100 also acquires SLO images and executes tracking processing based on the SLO moving images. In this embodiment, the reference SLO image used for tracking processing in repeated OCTA imaging is the reference SLO image set in the first repeated OCTA imaging, and a common reference SLO image is used in all repeated OCTA imaging.

Furthermore, during repeated OCTA imaging, in addition to the imaging conditions set in S301, the same setting values are used (not changed) for the selection of left and right eyes and whether or not to execute tracking processing.

<Step 303>
The image acquisition unit 101-01 and the image processing unit 101-04 generate a motion contrast image (motion contrast data) based on the OCT tomographic image acquired in S302. Specifically, the image processing unit 101-04 acquires a plurality of three-dimensional motion contrast data based on repeated OCTA imaging. That is, the image processing unit 101-04 corresponds to an example of an acquisition unit that acquires a plurality of three-dimensional motion contrast data of the fundus.

First, the tomographic image generation unit 101-11 performs wave number conversion, fast Fourier transform (FFT), and absolute value conversion (acquisition of amplitude) on the interference signal acquired by the image acquisition unit 101-01 to generate a tomographic image for one cluster. Generate an image.

Next, the alignment unit 101-41 aligns the tomographic images belonging to the same cluster and performs superimposition processing. An image feature acquisition unit 101-44 acquires layer boundary data from the superimposed tomographic image. In this embodiment, a variable shape model is used as a layer boundary acquisition method, but any known layer boundary acquisition method may be used. Note that the layer boundary acquisition process is not essential; for example, if a motion contrast image is generated only in three dimensions and a two-dimensional motion contrast image projected in the depth direction is not generated, the layer boundary acquisition process can be omitted. A motion contrast data generation unit 101-12 calculates motion contrast between adjacent tomographic images within the same cluster. In this embodiment, a decorrelation value Mxy is determined as the motion contrast based on the following equation (1).

Here, Axy indicates the amplitude (of complex number data after FFT processing) at the position (x, y) of the tomographic image data A, and Bxy indicates the amplitude at the same position (x, y) of the tomographic data B. 0≦Mxy≦1, and the larger the difference between both amplitude values, the closer the value is to 1. Perform the decorrelation calculation process as shown in Equation (1) between arbitrary adjacent tomographic images (belonging to the same cluster), and calculate the average of the obtained (number of tomographic images per cluster - 1) motion contrast values. An image having pixel values is generated as a final motion contrast image.

Note that although the motion contrast is calculated here based on the amplitude of the complex number data after FFT processing, the method of calculating the motion contrast is not limited to the above. For example, motion contrast may be calculated based on phase information of complex number data, or may be calculated based on both amplitude and phase information. Alternatively, motion contrast may be calculated based on the real part or imaginary part of complex number data.

Further, in this embodiment, a decorrelation value is calculated as the motion contrast, but the method of calculating the motion contrast is not limited to this. For example, the motion contrast may be calculated based on the difference between two values, or the motion contrast may be calculated based on the ratio of two values.

Further, in the above, the final motion contrast image is obtained by calculating the average value of the plurality of acquired decorrelation values, but the present invention is not limited to this. For example, an image whose pixel value is the median value or maximum value of the plurality of acquired decorrelation values may be generated as the final motion contrast image.

<Step 304>
The image processing unit 101-04 three-dimensionally aligns a plurality of three-dimensional motion contrast image groups (FIG. 8(a)) obtained through repeated OCTA imaging and averages them to produce the image shown in FIG. 8(b). As shown, a composite motion contrast image, which is an example of a high-quality (for example, high contrast) motion contrast image, is generated. In other words, the image processing unit 101-04 corresponds to an example of an image quality improvement unit that improves the image quality of three-dimensional motion contrast by combining a plurality of three-dimensional motion contrast data. Here, also in this embodiment, in addition to the processing by the image processing unit 101-04, as the processing by the image quality improvement means, for example, the image quality improvement processing by machine learning in the second embodiment described later is applied. is possible.

Note that the compositing process is not limited to simple arithmetic averaging. For example, the brightness value of each motion contrast image may be arbitrarily weighted and then averaged, or an arbitrary statistical value such as a median value may be calculated. Furthermore, the present invention also includes a case where the alignment process is performed two-dimensionally.

Furthermore, the composition unit 101-42 may be configured to determine whether or not a motion contrast image inappropriate for the composition process is included, and then perform the composition process excluding the motion contrast image determined to be inappropriate. For example, if the evaluation value (for example, the average value of decorrelation values or fSNR) for each motion contrast image is outside a predetermined range, it may be determined that the motion contrast image is unsuitable for the compositing process.

In this embodiment, after the synthesis unit 101-42 three-dimensionally synthesizes motion contrast images, the correction unit 101-43 performs processing to three-dimensionally suppress projection artifacts occurring in the motion contrast images.

Here, projection artifact refers to a phenomenon in which motion contrast within the superficial retinal blood vessels is reflected in the deep layers (deep retina, outer retina, and choroid), resulting in high decorrelation values in deep regions where no blood vessels actually exist. . FIG. 8(c) shows an example in which three-dimensional motion contrast data is displayed superimposed on a three-dimensional OCT tomographic image. A region 802 having a high decorrelation value is generated on the deep layer side (photoreceptor cell layer) of a region 801 having a high decorrelation value corresponding to the retinal superficial blood vessel region. Even though there are no blood vessels in the photoreceptor layer, the flickering of blood vessels occurring in the surface layer of the retina is reflected in the photoreceptor layer as a flickering blood vessel shadow, and the luminance value of the photoreceptor layer changes, causing artifact 802. occurs.

The correction unit 101-43 executes processing to suppress the projection artifact 802 generated on the three-dimensional composite motion contrast image. Although any known projection artifact suppression technique may be used, this embodiment uses Step-down Exponential Filtering. In Step-down Exponential Filtering, projection artifacts are suppressed by executing the process expressed by equation (2) on each A-scan data on the three-dimensional motion contrast image.

Here, γ is an attenuation coefficient with a negative value, D (x, y, z) is a decorrelation value before projection artifact suppression processing, and _DE (x, y, z) is a decorrelation value after the suppression processing. represents.

FIG. 8D shows an example in which three-dimensional composite motion contrast data (gray) after projection artifact suppression processing is displayed superimposed on a tomographic image. It can be seen that the artifacts observed on the photoreceptor layer before the projection artifact suppression process (FIG. 8(c)) were removed by the suppression process.

Next, the projection unit 101-45 projects a motion contrast image in a depth range based on the position of the layer boundary acquired by the image feature acquisition unit 101-44 in S303, and generates a front motion contrast image. Although projection may be performed in any depth range, in this embodiment, two types of frontal composite motion contrast images are generated in the depth range of the surface layer of the retina and the outer layer of the retina. Further, as a projection method, either maximum intensity projection (MIP) or average intensity projection (AIP) can be selected, and in this embodiment, projection is performed using maximum intensity projection.

Finally, the image processing device 101 inputs the acquired image group (SLO images and tomographic images), the imaging condition data of the image group, the generated three-dimensional and frontal motion contrast images, and the accompanying generation condition data, including the examination date and time, and the subject's eye. is stored in the external storage unit 102 in association with identifying information.

<Step 305>
The display control unit 101-05 causes the display unit 104 to display the tomographic image generated in S303, the three-dimensional and frontal motion contrast images synthesized in S304, and information regarding the imaging conditions and compositing conditions.

FIG. 8(e) shows an example of the report screen 803. In this embodiment, the SLO image and the tomographic image, the frontal motion contrast images of different depth ranges generated by combining and projecting in S304, and the corresponding frontal OCT image are displayed. In FIG. 8(e), the frontal motion contrast image generated with the retinal surface layer in the upper row and the outer retinal layer in the lower row as the projection depth range is displayed, and the frontal motion contrast image 808 of the outer retinal layer shows the choriocapillaris (CNV). It is depicted.

Note that the motion contrast image displayed on the display unit 104 is not limited to a front motion contrast image, and a three-dimensional motion contrast image rendered three-dimensionally may be displayed.

The projection range of the frontal motion contrast image can be changed by the operator selecting from the default depth range set (805 and 809) displayed in the list box. In addition, the type and offset position of the layer boundary used to specify the projection range can be changed from a user interface such as 806 or 810, or the layer boundary data (807 and 811) superimposed on the tomographic image can be operated from the input unit 103. You can change the projection range by moving it.

Furthermore, the image projection method and the presence or absence of projection artifact suppression processing may be changed by selecting from a user interface such as a context menu. For example, the motion contrast image after projection artifact suppression processing may be displayed on the display unit 104 as a three-dimensional image.

<Step 306>
The operator uses the input unit 103 to instruct the start of OCTA measurement processing.

In this embodiment, by double-clicking on the motion contrast image on the report screen 803 in FIG. 8(e), the screen shifts to an OCTA measurement screen as shown in FIG. 9(a). The motion contrast image is enlarged and displayed, and the analysis unit 101-46 starts measurement processing.

Any measurement process may be performed as the type of measurement process, and in this embodiment, the type of analysis shown in the item 905 displayed by selecting the Density Analysis 903 or Tools button 904 in FIG. By selecting the item 912 regarding the number of dimensions of analysis, the desired type of measurement is specified.

For example, when the user selects Area/Volume from the displayed item 905 after selecting the Tools button 904 and selects one point from the vascular region or avascular region using the input unit 103, the type of analysis can be changed. The area of blood vessels or avascular regions can be specified as Here, if the 3D Analysis item is also selected, the volume of blood vessels or avascular regions can be specified as the type of analysis. Alternatively, by selecting Area Density (or Skeleton Density) in Density Analysis 903, two-dimensional VAD (or VLD) can be specified as the type of analysis. Here, if the 3D Analysis item is also selected, three-dimensional VAD (or VLD) can be specified as the type of analysis. Note that Area Density and Skeleton Density are synonymous with VAD and VLD, respectively, which will be described later. Moreover, three-dimensional Area Density (VAD) may be rephrased as Volume Density (VVD).

Here, measurement by three-dimensional image processing can be broadly classified as follows.
1) Two-dimensional measurement of blood vessel area or blood vessel center line data that is emphasized in three dimensions and specified in two dimensions (measurement based on the two-dimensional position on the binarized projection enhanced image)
2) Two-dimensional measurement of the vascular region or blood vessel center line data highlighted and specified in three dimensions (measurement based on the two-dimensional position on the projected binary image, measurement for A-scan groups that meet predetermined conditions)
3) 3D measurement of 3D highlighted and specified vascular region or vascular centerline data (measurement based on 3D position on 3D binary image)

As an example of 1) and 2), the area and blood vessel density of the above-mentioned avascular region, the area, diameter, length, and curvature of the blood vessel region may be measured on the projection image, or the predetermined conditions may be measured without generating a projection image. For example, a two-dimensional measurement value may be calculated using the number of satisfying A-scan groups, horizontal distance, horizontal area, etc. Although the measurement content is the same as the measurement for the frontal motion contrast image, the blood vessel extraction ability is improved compared to when measuring by emphasizing and specifying the frontal motion contrast image, so the measurement accuracy is improved.

In addition, as an example of 3), 3-1) Volume measurement of blood vessels 3-2) Measurement on cross-sectional images or curved cross-sectional images in arbitrary directions (including measurement of blood vessel diameter or cross-sectional area)
3-3) Measurement of blood vessel length and curvature.

In this embodiment, the measurement shown in 3) above is performed. That is, in this embodiment, after performing three-dimensional blood vessel enhancement and blood vessel region identification processing, three-dimensional blood vessel density (VAD) is calculated from the three-dimensional motion contrast data in the depth range of the retinal surface layer, and three-dimensional blood vessel density (VAD) is calculated in the depth range of the outer retinal layer from the three-dimensional motion contrast data. The volume of each blood vessel region is measured. In this way, blood vessel density etc. are calculated directly from 3D motion contrast data rather than from 2D motion contrast data projected in the depth direction, so blood vessels overlap in the depth direction. It also becomes possible to accurately evaluate parts.

Here, VAD is an abbreviation for Vessel Area Density, and is the blood vessel density (unit: %) defined by the proportion of the blood vessel area included in the measurement target. In addition, VLD is an abbreviation for Vessel Length Density, and in the case of two dimensions, it is the sum of the lengths of blood vessels included per unit area (unit: mm ^-1 ), and in the case of three dimensions, it is the sum of the lengths of blood vessels included per unit volume. Blood vessel density is defined as the sum of blood vessel lengths (unit: mm ⁻² ). That is, the blood vessel density includes VAD and VLD.

Blood vessel density is an index for quantifying the extent of occlusion of blood vessels and the degree of sparseness of blood vessel networks, and VAD is most commonly used. However, in VAD, the contribution of the macrovascular region to the measurement value is large, so when you want to measure by focusing on the pathology of capillaries such as diabetic retinopathy (as an indicator that is more sensitive to capillary occlusion) VLD is used.

The measurement is not limited to this, and for example, Fractal Dimension, which quantifies the complexity of the blood vessel structure, or Vessel DiameterIndex, which represents the distribution of blood vessel diameter (distribution of blood vessel aneurysm or stenosis) may be measured.

Next, the analysis unit 101-46 performs preprocessing for measurement processing. Although any known image processing can be applied as preprocessing, in this embodiment, image enlargement and morphological calculation (top hat filter processing) are performed as preprocessing. By applying a top hat filter, it is possible to reduce uneven brightness of the background component. Specifically, it is assumed that the image is enlarged using three-dimensional Bicubic interpolation so that the pixel size of the composite motion contrast image becomes approximately 3 μm, and that top hat filter processing is performed using a spherical structural element.

<Step 307>
The analysis unit 101-46 performs blood vessel region identification processing. In this embodiment, the enhancement unit 101-461 performs blood vessel enhancement processing based on a three-dimensional Hessian filter and three-dimensional edge selective sharpening. Next, the extraction unit 101-462 performs binarization processing using the two types of blood vessel emphasis images, and performs shaping processing to specify the blood vessel region.

Details of the blood vessel region specifying process will be explained in S510 to S560.

<Step 308>
The measurement unit 101-463 measures the blood vessel density for a single examination image based on information regarding the measurement target area specified by the operator. Subsequently, the display control section 101-05 displays the measurement results on the display section 104. More specifically, the measurement unit 101-463 calculates the blood vessel density, etc. in the three-dimensional space using information regarding the three-dimensional blood vessel region in at least part of the plurality of averaged three-dimensional motion contrast data. . That is, the measurement unit 101-463 corresponds to an example of a calculation unit that calculates blood vessel density in a three-dimensional space using information regarding a three-dimensional blood vessel region in at least a portion of the synthesized three-dimensional motion contrast data. More specifically, the measurement unit 101-463 calculates the blood vessel density after the process of reducing projection artifacts is executed. That is, the measurement unit 101-463, which is an example of a calculation unit, uses information regarding a three-dimensional blood vessel region in at least a portion of the synthesized three-dimensional motion contrast data after projection artifacts have been reduced by the processing unit. Calculate blood vessel density in three-dimensional space.

There are two types of indicators for blood vessel density: VAD and VLD, and in this embodiment, the procedure for calculating VAD will be explained as an example. Note that the procedure for calculating VLD will also be described later.

When the operator inputs an instruction to manually correct the blood vessel region or blood vessel center line data from the input unit 103, the analysis unit 101-46 uses the position information specified by the operator via the input unit 103 to Manually correct the vessel area or vessel centerline data and recalculate the measured values.

VAD measurement in the retinal surface layer and volume measurement of choroidal neovascularization in the outer retinal layer will be explained in S810 to S820, and VLD measurement in the retinal superficial layer and total blood vessel length measurement of choroidal neovascularization in the outer retinal layer will be explained in S830 to S850.

<Step 309>
The analysis unit 101-46 acquires from the outside an instruction as to whether or not to manually correct the data on the blood vessel region and blood vessel center line identified in S307. This instruction is input by the operator via the input unit 103, for example. If manual correction processing is instructed, the process proceeds to S308, and if manual correction processing is not instructed, the process proceeds to S310.

<Step 310>
The display control unit 101-05 displays a report regarding the measurement results performed in S308 on the display unit 104.

In this embodiment, the VAD map and VAD sector map measured in the superficial layer of the retina are superimposed on the upper part of the single measurement report, and the lower part shows a binary image of the choroidal neovascular region identified in the deep outer retinal layer and the measured value. Display volume value. This makes it possible to view and understand the vascular pathology at different depth positions. Furthermore, it is possible to specify the three-dimensional position of CNV in the outer retinal layer and accurately quantify the amount present.

Furthermore, regarding each measurement target image, the display unit 104 may display information regarding the number of tomographic images at approximately the same position, the conditions for performing OCTA overlay processing, and the evaluation value (image quality index) of the OCT tomographic image or motion contrast image. good.

In this embodiment, images and measurement values of the retinal surface layer and the retinal outer layer are displayed as different depth ranges, but the present invention is not limited to this, and images and measurements of four types of depth ranges, for example, the retinal surface layer, the retinal deep layer, the retinal outer layer, and the choroid, may be displayed. Values may be displayed.

Furthermore, the motion contrast image and the binary image of the specified blood vessel region or blood vessel center line are not limited to being projected and displayed as a frontal image, but may be rendered three-dimensionally and displayed as a three-dimensional image. .

Alternatively, measurement results of different indicators may be displayed side by side. For example, the VAD map may be displayed in chronological order in the upper row, and the VLD map (or the size or shape value of the avascular region) may be displayed in the lower row.

Similarly, the projection method (MIP/AIP) and projection artifact suppression processing may also be changed by, for example, selecting from a context menu.

<Step 311>
The image processing apparatus 101 obtains an instruction from the outside as to whether or not to end the series of processes from S301 to S311. This instruction is input by the operator via the input unit 103. If an instruction to end the process is obtained, the process ends. On the other hand, if an instruction to continue the process is obtained, the process returns to S302 and the process for the next eye to be examined (or reprocessing for the same eye to be examined) is performed.

Furthermore, details of the process executed in S307 will be described with reference to the flowchart shown in FIG. 5(a).

<Step 510>
The enhancement unit 101-461 performs blood vessel enhancement filter processing based on the eigenvalues of the Hessian matrix on the motion contrast image that has undergone the preprocessing in step 306. Such enhancement filters are collectively referred to as Hessian filters, and include, for example, a Vesselness filter and a Multi-scale line filter. Although a multi-scale line filter is used in this embodiment, any known blood vessel enhancement filter may be used.

The Hessian filter smoothes an image to a size appropriate to the diameter of the blood vessel you want to emphasize, then calculates a Hessian matrix whose elements are the second-order differential values of the luminance values at each pixel of the smoothed image, and calculates the eigenvalue of the matrix. The local structure is emphasized based on the magnitude relationship of . The Hessian matrix is a square matrix as given by equation (3), and each element of the matrix is, for example, the second derivative of the brightness value Is of an image obtained by smoothing the brightness value I of the image as shown in equation (4). expressed as a value. In the Hessian filter, when "one of the eigenvalues (λ1, λ2, λ3) of the Hessian matrix is close to 0 and the others are negative and have large absolute values", it is regarded as a linear structure and emphasized. This is done by treating pixels that have the characteristics of blood vessel regions on motion contrast images, that is, ``changes in brightness are small in the direction of travel, and the brightness value decreases greatly in the direction orthogonal to the direction of travel'' as linear structures, and are emphasized. corresponds to

When a three-dimensional Hessian filter is used, even for blood vessels that are curved in the depth direction, the property that ``changes in brightness in the blood vessel running direction is small, and brightness in two directions orthogonal to the blood vessel running direction is greatly reduced'' holds true. It has the advantage of being able to be emphasized. Among the fundus blood vessels, there are blood vessels bent in the depth direction, such as choroidal neovascularization (CNV) that invades the retina from the choroid side.
・Vessels in the optic nerve head ・Connections between superficial retinal capillaries and deep retinal capillaries.

When a two-dimensional Hessian filter is applied to the above-mentioned blood vessel on a frontal motion contrast image, it is found that within the two-dimensional plane, the brightness change in the blood vessel running direction is small, and the brightness in the direction perpendicular to the blood vessel running direction is small. There is a problem in that because the property of ``significantly decreasing'' does not hold true, it is not emphasized enough and cannot be identified as a vascular region. By using a three-dimensional Hessian filter, the blood vessels described above can be emphasized well, and the blood vessel detection ability is improved.

Furthermore, since a motion contrast image includes blood vessels of various diameters from capillaries to arterioles and veins, a line-enhanced image is generated using a Hessian matrix for an image smoothed by a Gaussian filter at multiple scales. Next, as shown in equation (5), the images are multiplied by the square of the smoothing parameter σ of the Gaussian filter as a correction coefficient, and then synthesized by maximum value calculation, and the synthesized image Ihessian is used as the output of the Hessian filter.

The Hessian filter has the advantage of being resistant to noise and improving the continuity of blood vessels. On the other hand, in reality, the maximum diameter of the blood vessels included in the image is often unknown in advance, so if the smoothing parameter is too large for the maximum diameter of the blood vessels in the image, the emphasized blood vessel area may become thicker. It has the disadvantage of being easy.

Therefore, in this embodiment, the blood vessel region is prevented from becoming too thick by performing calculations on an image in which the blood vessel region is emphasized using another blood vessel enhancement method described in S530.

<Step 520>
The extraction unit 101-462 binarizes the blood vessel enhanced image (hereinafter referred to as a three-dimensional Hessian enhanced image) generated by the three-dimensional Hessian filter in S510.

In this embodiment, the luminance statistical value (average value, median value, etc.) of a three-dimensional Hessian-enhanced image is binarized using a threshold value. In order to avoid insufficient extraction of capillaries due to a high threshold due to the influence of high-intensity regions of large blood vessels, or erroneous detection of avascular regions as blood vessels due to a threshold that is too low, binarization is performed. An upper limit value or a lower limit value may be set for the threshold value when Alternatively, the threshold value may be determined based on an index value calculated such that the contribution of brightness values outside a predetermined brightness range is reduced when calculating the brightness statistical value, as in the case of a robust estimation method, for example. Also, any known threshold determination method may be used.

Alternatively, the present invention is not limited to threshold processing, and binarization may be performed using any known segmentation method.

In this embodiment, since the composite motion contrast image is enhanced using a Hessian filter, the continuity of the binarized blood vessel region is further improved compared to the case where a single motion contrast image is enhanced using a Hessian filter.

Furthermore, the present invention is not limited to binarizing a three-dimensional blood vessel enhanced image as three-dimensional data. For example, the present invention also includes the case where a frontal blood vessel emphasized image obtained by projecting a three-dimensional blood vessel emphasized image in a predetermined projection depth range is binarized. Any known threshold value determination method may be used when binarizing the front blood vessel enhanced image. Further, an upper limit value or a lower limit value may be set as a threshold value when binarizing the front blood vessel enhanced image. Alternatively, the binarization of the front blood vessel enhanced image is not limited to threshold processing, and may be binarized using any known segmentation method.

<Step 530>
The enhancement unit 101-461 performs three-dimensional edge selection sharpening processing on the top-hat filtered synthesized motion contrast image generated in S306.

Here, the edge selective sharpening processing refers to performing weighted sharpening processing after setting a large weight to the edge portion of the image. In this embodiment, edge selective sharpening processing is performed by performing three-dimensional unsharp mask processing using an image obtained by applying a three-dimensional Sobel filter to the synthetic motion contrast image as a weight.

When the sharpening process is performed with a small filter size, the edges of thin blood vessels are emphasized and the blood vessel region can be specified more accurately when binarized (the phenomenon of blood vessel regions becoming thicker can be prevented). On the other hand, especially in the case of a motion contrast image in which the number of tomographic images at the same imaging position is small, there is a lot of noise, so there is a risk that noise in blood vessels may also be emphasized. Therefore, noise enhancement is suppressed by performing edge selective sharpening.

<Step 540>
The extraction unit 101-462 binarizes the sharpened image generated in S530 to which the edge selective sharpening process has been applied. Any known binarization method may be used, but in this embodiment, the brightness statistical value (average value or median value) calculated within each three-dimensional local area on the three-dimensional sharpened image is used as a threshold, and a binary value is used. become

However, in the large blood vessel region of the optic disc, the threshold value set is too high, resulting in many holes in the blood vessel region on the binary image. Prevent the threshold from becoming too high.

Further, as in the case of S520, if the three-dimensional local area is in an avascular area, the threshold value may be too low and a part of the avascular area may be erroneously detected as a blood vessel. Therefore, false detection is suppressed by setting the lower limit of the threshold value.

Note that, as in the case of S520, edge-selective sharpening is performed on the composite motion contrast image, so noise-like erroneous detection areas when binarized are more reduced than when edge-selectively sharpening a single motion contrast image. can be reduced.

Furthermore, the present invention is not limited to binarizing a three-dimensional sharpened image as three-dimensional data. For example, the present invention also includes the case where a front sharpened image obtained by projecting a three-dimensional sharpened image in a predetermined projection depth range is binarized. Any known threshold value determination method may be used when the front sharpened image is binarized. Further, an upper limit value or a lower limit value may be set as a threshold value when the front sharpened image is binarized. Alternatively, the binarization of the front sharpened image is not limited to threshold processing, and may be binarized using any known segmentation method.

<Step 550>
The extraction unit 101-462 determines whether the brightness value of the binary image of the three-dimensional Hessian enhanced image generated in S520 and the brightness value of the binary image of the three-dimensional edge selective sharpening image generated in S540 are both greater than 0. The area is extracted as a blood vessel candidate area. This calculation process suppresses both the area where the vessel diameter is overestimated in the Hessian-enhanced image and the noise area seen in the edge-selective sharpened image, making it possible to accurately locate blood vessel boundaries and ensure blood vessel continuity. Good binary images can be obtained.

In addition, since both binary images are binary images based on a composite motion contrast image, the noise-like false detection area when binarized is reduced compared to a binary image based on a single motion contrast image. , especially the continuity of the capillary region is improved.

<Step 560>
The extraction unit 101-462 performs three-dimensional opening processing (performing expansion processing after contraction processing) and three-dimensional closing processing (performing contraction processing after expansion processing) of the binary image as shaping processing for the blood vessel region. Note that the shaping process is not limited to this, and for example, small area removal may be performed based on the volume of each label when a binary image is labeled.

Note that the method of adaptively determining the scale for enhancing blood vessels in a motion contrast image that includes blood vessels of various diameters is not limited to the methods described in S510 to S560. For example, as shown in S610 to S650 in FIG. 5B, the brightness statistical values ( For example, the blood vessel region may be identified by binarizing using the average value) as a threshold value. A lower limit value and an upper limit value can be set for the threshold value.

Alternatively, as shown in S710 to S740 in FIG. 5(c), the range of the smoothing parameter σ when applying the Hessian filter is adaptively changed according to the three-dimensional position of each pixel based on the fixation position, depth range, etc. Blood vessels may be emphasized by applying a Hessian filter and binarizing the image. For example, it is possible to set σ=1 to 10 for the papillary retinal surface layer, σ=1 to 8 for the macular retinal surface layer, and σ=1 to 6 for the macular retinal deep layer.

Furthermore, the binarization process is not limited to threshold processing, and any known segmentation method may be used.

Furthermore, details of the process executed in S308 will be described with reference to the flowchart shown in FIG. 6(a).

<Step 810>
The operator sets a region of interest in the measurement process via the input unit 103. That is, the three-dimensional space in which the blood vessel density is calculated is designated by the user. In this embodiment, the measurement contents are as follows: 1) VAD map and VAD sector map in the retinal surface layer 2) Volume of choroidal neovascularization in the outer retinal layer is calculated.

Therefore, in the retinal surface layer, the areas of interest are (i) the entire image, (ii) sector areas centered on the fixation lamp position (inside the annular area defined by the inner circle with a diameter of 1 mm and the outer circle with a diameter of 3 mm), Superior and Inferior.・Select the area divided into four fan shapes (Nasal and Temporal) and the inner circle area). However, the entire image in this embodiment is limited to a finite range defined using layer boundaries such as those set at 910 in FIG. 9(a) in the depth direction. In addition, the sector area in this embodiment is defined by "a fan-shaped divided area and an inner circular area as described above in the horizontal direction, and a layer boundary as set at 910 in FIG. 9(a) in the depth direction. refers to the dimensional area. The former is done by selecting the 3D Analysis check box 912 for Map and the number of dimensions from the DensityMap/Sector 902 items in FIG. Automatically set by selecting the checkbox. Furthermore, for the outer retinal layer, a layer boundary corresponding to the outer retinal layer (an area surrounded by the OPL/ONL boundary and a position where the Bruch's membrane boundary is moved 20 μm toward the deep layer side of the boundary) is specified. The region of interest can be specified in the two-dimensional direction of the fundus surface by moving sectors, and the depth direction of the fundus can be specified by specifying the depth range of the displayed En-Face image. Note that the region of interest may be directly specified with respect to the three-dimensional motion contrast data while the three-dimensional motion contrast data is displayed.

Note that the present invention is not limited to this, and for example, the operator may manually set the region of interest by specifying a button as shown at 904 in FIG. 9(a). Select the measurement type (Density in this case) from the setting screen 905 that is displayed when the button 904 is pressed, and select the region of interest as shown in the gray line section 1001 in FIG. 10(b) via the input unit 103. Make the settings and press the OK button.

Regarding the depth direction, in this embodiment, the depth range for the retinal surface layer is defined using layer boundaries as shown at 910 in FIG. 9(a). Regarding the outer retinal layer, the depth range is defined by the outer plexiform layer (OPL)-outer nuclear layer (ONL) boundary and Bruch's membrane (BM). The numerical value shown within the area indicates the value measured within the area (in this case, the VAD value because Area Density is selected in the Density Analysis item 903 of FIG. 9(a)). Note that the region of interest is not limited to a three-dimensional region, and a two-dimensional region may be set on the projection image and measurement values may be calculated within the two-dimensional region.

<Step 820>
The measurement unit 101-463 performs measurement processing based on the binary image of the blood vessel region obtained in S307. In this embodiment, the binary image of the blood vessel region identified in S307 is not projected within the range of the retinal surface layer, and instead of projecting the binary image of the blood vessel region specified in S307 within the range of the retinal surface layer, at each pixel position of the three-dimensional data, a non-zero The ratio of pixels (white pixels) is calculated as the blood vessel density (VAD) at the pixel. Furthermore, an image (VAD map) having a blood vessel density (VAD) value calculated for each pixel is generated.

Note that the ratio of non-zero pixels (white pixels) in each sector area (set in S810) on the projected binary image may be calculated as the blood vessel density (VAD) in the sector. Furthermore, a map (VAD sector map) having blood vessel density (VAD) values calculated in each sector region may be generated.

Furthermore, for the outer retinal layer, the volume of non-zero pixels (white pixels) within the region of interest corresponding to the outer retinal layer set in S810 is calculated. Note that the present invention is not limited to volume calculation; for example, an image in which the choroidal neovascular region is emphasized or binarized is projected within a region of interest corresponding to the outer layer of the retina, and the projected enhanced image is binarized. The area of the choroidal neovascular region may be calculated on the image or the projected binary image.

Above, we explained the procedure for measuring 3D VAD in the retinal surface layer based on the identified 3D blood vessel area and volume in the outer retinal layer as an example. When measuring the total blood vessel length in the outer retinal layer using three-dimensional VLD, steps S830 to S850 shown in FIG. 6(b) are executed instead of S810 to S820.

<Step 830>
The measurement unit 101-463 performs three-dimensional thinning processing on the binary image of the blood vessel region generated in S307, thereby creating a binary image with a line width of 1 pixel corresponding to the center line of the blood vessel (hereinafter referred to as a skeleton image). generate.

<Step 840>
The operator sets the region of interest via the input unit 103 in the same manner as in step S810. In this embodiment, it is assumed that a VLD map and a VLD sector map are calculated as measurement contents. Note that if you do not want to display the VLD or VLD sector map superimposed on the motion contrast image, you can set the check box of the Map or Sector item in the Density Map/Sector 902 in FIG. 9A to non-select.

<Step 850>
The measurement unit 101-463 performs measurement processing based on the skeleton image obtained in S830. That is, measurement is performed from the three-dimensional skeleton specified in the retinal surface layer. In the present embodiment, at each pixel position of the skeleton image, the total length [mm ⁻² ] of non-zero pixels (white pixels) per unit volume in a neighboring region centered on the pixel is calculated as the blood vessel density ( VLD). Furthermore, an image (VLD map) having a blood vessel density (VLD) value calculated for each pixel is generated.

Furthermore, the sum [mm ⁻² ] of the lengths of non-zero pixels (white pixels) per unit volume in each sector region (set in S840) on the skeleton image is calculated as the blood vessel density (VLD) in the sector. . Furthermore, a map (VLD sector map) having blood vessel density (VLD) values calculated in each sector area is generated.

Furthermore, in the outer retinal layer, the total length of non-zero pixels (white pixels) within the region of interest corresponding to the outer retinal layer set in S810 is calculated. Note that the present invention is not limited to calculating the length of a three-dimensional blood vessel; for example, a skeleton image is projected within a region of interest corresponding to the outer layer of the retina, and the length of the blood vessel is calculated on the projected binary image. You may calculate the total sum.

According to the configuration described above, the image processing device 101 acquires a three-dimensional composite motion contrast image from the subject's eye including choroidal neovascularization (CNV) in the outer retinal layer, applies a three-dimensional blood vessel enhancement filter, and obtains a binary image using a three-dimensional blood vessel enhancement filter. The blood vessel region containing the CNV is identified in three dimensions. Furthermore, the volume of the specified CNV region is calculated and accurately measured.

Thereby, a desired region such as a three-dimensional blood vessel region can be emphasized or specified and accurately measured while suppressing the influence of the signal intensity and image quality of the OCT tomographic image.

Furthermore, since the data for which the blood vessel density is calculated is synthesized three-dimensional motion contrast data, noise is reduced compared to one piece of three-dimensional motion contrast data. Therefore, according to the present embodiment, it is possible to calculate accurate indicators such as blood vessel density using information regarding the three-dimensional blood vessel region in at least a portion of the synthesized three-dimensional motion contrast data. In addition, since the blood vessel density and the like are calculated after projection artifacts that cause a decrease in accuracy when calculating the blood vessel density and the like are removed, the blood vessel density and the like can be calculated with high accuracy.

(Modification 1)
In the embodiment described above, the blood vessel measurement value may be calculated from two-dimensional motion contrast data obtained by projecting high-quality three-dimensional motion contrast data in a predetermined depth range.

For example, the user selects whether to perform blood vessel measurement in two dimensions or in three dimensions via the user interface, and the image processing device 101 (measuring unit 101-463) accepts the user's selection. Then, the measurement unit 101-463 performs blood vessel measurement from either the two-dimensional motion contrast data or the three-dimensional motion contrast data according to the user's selection. That is, the measurement unit 101-463 uses information regarding the three-dimensional blood vessel region in at least a portion of the two-dimensional motion contrast data obtained by projecting the high-quality three-dimensional motion contrast data in the depth direction. It is also possible to calculate blood vessel measurement values in a two-dimensional region.

Note that the measurement unit 101-463 may associate the calculated blood vessel measurement value with identification information that can identify whether it is calculated from two-dimensional motion contrast data or three-dimensional motion contrast data. That is, the measurement unit 101-463 associates the calculated blood vessel measurement value with information indicating that the blood vessel measurement value is calculated in a three-dimensional space. Specifically, the measurement unit 101-463 determines whether the calculated blood vessel measurement values (for example, blood vessel density, diameter, length, area, and volume of the blood vessel region) are three-dimensional measurement values or two-dimensional measurement values. Associate identification information that can identify whether it is a value or not.

(Modification 2)
In the embodiments and modifications described above, the display control unit 101-05 may display a plurality of calculated blood vessel measurement values and the like on the display unit 104 in chronological order. In this way, the user can view a list of blood vessel densities calculated on different days, for example, and can easily understand changes in the condition of the subject.

When displaying multiple blood vessel measurement values in chronological order, if blood vessel measurement values calculated from 2D motion contrast data and blood vessel measurement values calculated from 3D motion contrast data are mixed, the patient may It becomes impossible to accurately grasp the progress of the process. Therefore, the display control unit 101-05 prevents the blood vessel measurement values calculated from the two-dimensional motion contrast data and the blood vessel measurement values calculated from the three-dimensional motion contrast data based on the identification information described above in modification example 1 from being mixed. A plurality of blood vessel measurement values in time series are displayed on the display unit 104. The display control unit 101-05 selects a blood vessel measurement value calculated from the two-dimensional motion contrast data and a blood vessel measurement value calculated from the three-dimensional motion contrast data based on the identification information, for example, from the three-dimensional motion contrast data. Only the blood vessel measurement values obtained are displayed on the display unit 104 in chronological order. That is, when displaying a plurality of blood vessel measurement values calculated by the calculation means in chronological order, the display control means determines whether the blood vessel measurement values in the three-dimensional space and the blood vessel measurement values in the two-dimensional area are different from each other based on the identification information. Display them on the display so that they do not mix.

(Modification 3)
In the above-described embodiments and each modification example, choroidal neovascularization is a target case in which measurement is performed by three-dimensionally emphasizing or segmenting a blood vessel bent in the depth direction using a three-dimensional motion contrast image. Although the case has been described, the present invention is not limited to this.

For example, the arteriovenous and capillary vessels of the optic disc are highlighted and segmented by the procedure shown in S307, and the analysis unit 101-46 calculates the diameter, cross-sectional area, length, curvature, blood vessel density, etc. of the identified blood vessel region. may be measured.

Alternatively, the superficial capillaries and deep capillaries of the retina may be highlighted and segmented using the procedure shown in S307, and the capillary regions running in the depth direction (at junctions) may be displayed with different colors, or the analysis unit 101-46 may The number of capillaries at the junction may also be measured.

(Modification 4)
In the embodiment and each modification described above, the case where measurement is performed on an image obtained by performing three-dimensional enhancement processing and specific processing on a three-dimensional motion contrast image has been described, but the present invention is not limited to this. but not limited to.

For example, perform binarization processing using any known segmentation method on a front-enhanced image obtained by projecting the three-dimensional enhanced image in a predetermined depth range, and measure the two-dimensional binary area. (Area, circularity, blood vessel density, etc.) are also included in the present invention. As explained in S520, S540, and S820, measurements (area, circularity, two-dimensional blood vessel density, etc. ) is also included in the present invention.

(Modification 5)
In the above-described embodiments and each modification, a frontal image is generated by projecting an image obtained by performing three-dimensional enhancement processing and specific processing on a three-dimensional motion contrast image in a predetermined depth range. Although the case of measurement has been described, the present invention is not limited to this.

For example, a cross section in an arbitrary direction (MPR; Multi Planar Reconstruction) or a curved cross section of an arbitrary shape is set for the identified three-dimensional blood vessel region, and the area, diameter, number of regions, etc. of the blood vessel region in the cross section or the curved cross section are determined. may be measured.

(Modification 6)
In the above-described embodiments and modifications, the blood vessel density was measured in sector units in the frontal image with respect to the macular region, but the present invention is not limited thereto. For example, the present invention also includes a case where a sector-shaped region of interest is set for the optic disc and a measurement value is calculated within the region of interest.

Furthermore, in the present invention, a case has been described in which blood vessel volume is measured in units of slabs (three-dimensional regions surrounded by two layer boundary curved surfaces) as three-dimensional regions of interest. The setting method is not limited to this. For example, a three-dimensional grid (cubic area or quadrangular prism area) area or a three-dimensional sector area (an area obtained by extending a two-dimensional sector area in the depth direction) is set as a three-dimensional region of interest, and the It may be measured in units of dimensional regions of interest. Alternatively, the present invention also includes a case where measurement is performed in units of a three-dimensional region of interest obtained by combining the slab region and the three-dimensional grid or three-dimensional sector.

Note that the measured values measured in units of the three-dimensional region of interest are calculated for each predetermined depth range (if there is one measured value in the depth direction, the measured value is used as is; if there are multiple measured values in the depth direction, the measured value is projected as an average value or maximum value projected). The map may be displayed as a two-dimensional map or as a three-dimensional map.

(Modification 7)
In the embodiment and each modification described above, examples of measurement values that the analysis unit 101-46 calculates for the three-dimensional blood vessel region are not limited to the blood vessel volume, blood vessel length, blood vessel curvature, cross-sectional area, and blood vessel diameter. For example: ・Three-dimensional blood vessel density (ratio of pixels belonging to the blood vessel region within the three-dimensional region of interest, or average within a sector of "total blood vessel length per unit volume")
・Vessel Cross Section Index (vascular cross-sectional area per unit vessel length)
・3D Fractal Dimension
may be calculated.

(Modification 8)
In the above-described embodiment and each modification example, a case has been described in which a measurement map is superimposed and displayed on a frontal motion contrast image as a method for specifying blood vessel regions and displaying measurement results in a single examination, but the present invention is not limited to this. It's not something you can do. For example, at least one of a blood vessel enhanced image, a binary image of the specified blood vessel region, and a skeleton image may be displayed in 906 or 908 in FIG. 9(b). Alternatively, a motion contrast image is displayed in 906 or 908, and at least one of a blood vessel enhancement image, a binary image of a specified blood vessel region, and a skeleton image is displayed on top of the motion contrast image after adjusting the color or transparency as appropriate. The present invention also includes a configuration in which this is the case. Alternatively, at least one of a blood vessel emphasis image, a binary image of a specified blood vessel region, and a skeleton image may be superimposed and displayed on the frontal tomographic image or the B-scan tomographic image after adjusting the color or transparency as appropriate. . A line indicating the position of the tomographic image is displayed at 906 or 908 as appropriate, and when the operator moves the line using the input unit 103, an enhanced image on the corresponding tomographic image or a binary image of the blood vessel region is displayed. , at least one of the skeletons may be displayed.

(Modification 9)
In the embodiment and each modification described above, when the operator inputs an instruction to manually correct the blood vessel region or blood vessel center line data from the input unit 103 in S309, the manual correction may be performed using the following procedure. .

For example, when a binary image including an over-extracted area as shown in FIG. 10(c) is obtained from a composite motion contrast image as shown in FIG. 10(a), the operator The analysis unit 101-46 deletes the white pixel at the specified position. Addition/movement/deletion procedures include, for example, holding down the "d" key and deleting a white pixel at a position specified with the mouse, holding down the "a" key and adding a white pixel at a position specified with the mouse, or "s". There is a method of moving the specified white pixel by dragging it with the mouse while holding down the key. Alternatively, as shown in Figure (d), the color and transparency of the binary image (vascular region or blood vessel center line) to be manually corrected may be adjusted and superimposed on the image based on the motion contrast image to over-extract the image. Or, make it easy to identify areas where extraction is insufficient. An enlarged image of the inside of the rectangular area 1002 in FIG. 3(d) is shown in FIG. 2(e). Gray indicates the overextracted region, and white indicates the decorrelation value of the original motion contrast image. The system is configured such that the operator specifies the over-extracted (or under-extracted) area using the input unit 103 to manually correct the blood vessel or blood vessel center line area on the binary image accurately and efficiently. Good too. Note that manual correction processing for binary images is not limited to front images. For example, motion contrast data, binary data of a blood vessel region, or blood vessel center line region is superimposed on a B-scan tomographic image at an arbitrary slice position as shown on the right side of FIG. 9(a) after adjusting color and transparency. After making it easy to identify areas that are overextracted or underextracted in this way, the three-dimensional position (x, y, z coordinates) of the binary data that the operator manually corrects (adds, moves, deletes), It may be specified using the input unit 103 and manually corrected. Furthermore, manually corrected two-dimensional or three-dimensional binary data may be displayed on the display unit 104 or stored in the external storage unit 102.

Furthermore, information indicating that the binary image (binary image or skeleton image of the blood vessel region) has been manually corrected or information regarding the manual correction position is stored in the external storage unit 102 in association with the binary image, When displaying the blood vessel identification results and measurement results on the display unit 104 in S870, information indicating that the manual correction has been completed or information regarding the manual correction position may be displayed on the display unit 104.

(Modification 10)
In the embodiment and each modification described above, the image processing device 101 may identify a blood vessel region in a motion contrast image in which the number of tomographic images acquired at substantially the same scanning position is less than a predetermined value or in a composite motion contrast image equivalent to less than a predetermined value. It may be configured to display a warning on the display unit 104 when an instruction regarding measurement is received.

Further, in the above-described embodiment and each modification example, a case has been described in which the synthesizing unit 101-42 generates a synthesized motion contrast image at the end of repeated OCTA imaging, but the procedure for generating a synthesized motion contrast image is not limited to this. For example, a synthetic motion contrast image generation instruction button 812 is placed on the report screen 803 in FIG. 8(e). The image processing device 101 is configured so that the compositing unit 101-42 generates a composite motion contrast image when the operator explicitly presses the generation instruction button 812 after the OCTA imaging is completed (may be on a day after the imaging date). may be configured. When the operator explicitly presses the composite image generation instruction button 812 to generate a composite image, a composite motion contrast image 804, composite condition data, and inspection image list are displayed on the report screen 803 as shown in FIG. 8(e). Display items related to the composite image at the top.

Further, when the operator explicitly presses the generation instruction button 812, the display control unit 101-05 performs the following processing. That is, when the compositing target image selection screen is displayed, the operator operates the input unit 103 to specify a group of compositing target images, and presses the accept button, the compositing unit 101-42 generates a composite motion contrast image, It is displayed on the display unit 104. Note that the present invention also includes a case where a generated synthetic motion contrast image is selected and synthesized.

Further, when the operator presses the composite image generation instruction button 812, a 2D composite image may be generated by combining 2D images obtained by projecting 3D motion contrast images, or a 3D composite image may be generated. A two-dimensional composite image may be generated by later projecting.

In addition, in the various embodiments and modifications described above, it is assumed that three-dimensional motion contrast data is synthesized, but processing for reducing projection artifacts and blood vessel density etc. for one three-dimensional motion contrast data. A process of calculating a blood vessel measurement value (or a process of extracting a blood vessel region) may be performed.

Furthermore, in the various embodiments and modifications described above, the correction unit 101-43 corresponds to an example of a processing unit that performs processing to reduce projection artifacts on high-quality three-dimensional motion contrast data. However, the image quality improvement process is not essential for executing the process by the processing means. That is, any processing means may be used as long as it performs processing to reduce projection artifacts on the three-dimensional motion contrast data of the fundus. The extraction means may extract a three-dimensional blood vessel region in at least a portion of the three-dimensional motion contrast data after the projection artifact has been reduced by the processing means. Thereby, a three-dimensional blood vessel region can be obtained with high accuracy. Further, the calculation means may calculate the blood vessel measurement value using information regarding a three-dimensional blood vessel region in at least a portion of the three-dimensional motion contrast data after projection artifacts have been reduced by the processing means.

In addition, in the embodiment and each modification described above, the volume of choroidal neovascularization in the outer retinal layer was measured based on the three-dimensional blood vessel area obtained from the three-dimensional motion contrast data with high image quality. Three-dimensional blood vessel density (VAD or VLD) may be calculated in the outer layer. Since new blood vessels tend to travel not only in the in-plane direction but also in the depth direction, it is thought that there is an advantage in calculating the blood vessel density in three dimensions. In the case of three-dimensional VAD, the three-dimensional blood vessel density can be easily calculated without the need for thinning processing, and in the case of three-dimensional VLD, it can be made less susceptible to the quality of blood vessel segmentation.

[Second embodiment]
The image processing device according to the present embodiment applies image quality improvement processing using machine learning to the acquired three-dimensional motion contrast image of the subject's eye, and specifies a three-dimensional blood vessel region. Regarding the case where two-dimensional blood vessel density in the surface layer of the retina, area of choroidal neovascularization (CNV) in the outer layer of the retina, etc. are calculated as two-dimensional blood vessel measurement values based on the three-dimensional blood vessel area without generating a frontal image. explain. That is, the image processing device according to the present embodiment can extract a three-dimensional blood vessel region in at least a portion of the three-dimensional motion contrast data with high image quality. Note that the extraction unit 101-462 is an example of an extraction unit that extracts a three-dimensional blood vessel region. Further, the image processing device according to the present embodiment can calculate a blood vessel measurement value using information regarding a three-dimensional blood vessel region in at least a portion of the high-quality three-dimensional motion contrast data. Note that the information regarding the three-dimensional blood vessel region is, for example, position information of the three-dimensional blood vessel region in high-quality three-dimensional motion contrast data. However, the information regarding the three-dimensional blood vessel region may be any information as long as it is possible to calculate blood vessel measurement values from high-quality three-dimensional motion contrast data.

An image processing system including an image processing apparatus according to this embodiment will be described below with reference to the drawings. FIG. 11 shows the configuration of an image processing system 10 including an image processing device 101 according to this embodiment. This embodiment differs from the first embodiment in that the image processing section 101-04 includes an image quality improvement section 101-47. Here, the image quality improvement unit 101-47 is an example of an image quality improvement unit that improves the image quality of three-dimensional motion contrast data. Note that in this embodiment as well, in addition to the image quality improvement section 101-47, for example, the composition section 101-42 in the first embodiment described later can be applied as the image quality improvement means. Next, FIG. 12 shows an image processing flow in this embodiment. Note that in the image processing flow of this embodiment, S1207, S1209, and S1211 are the same as in the first embodiment, and therefore the description thereof will be omitted.

<Step 1201>
By operating the input unit 103, the operator sets the imaging conditions for the OCTA image to be instructed to the tomographic imaging apparatus 100.

Most of the imaging conditions are the same as those in the first embodiment, and the second embodiment differs from the first embodiment in that repeated OCTA imaging is not performed, that is, the number of clusters acquired in this step is one. That is,
1) Register Macular Disease test set 2) Select OCTA scan mode 3) Set the following imaging parameters 3-1) Scanning pattern: Small Square
3-2) Scan area size: 3x3mm
3-3) Main scanning direction: horizontal direction 3-4) Scanning interval: 0.01mm
3-5) Fixation light position: fovea 3-6) Number of B scans per cluster: 4
3-7) Coherence gate position: Vitreous body side 3-8) Default display report type: Set a single examination report as the imaging condition.

<Step 1202>
The operator operates the input unit 103 and presses a capture start button 713 on the capture screen 710 shown in FIG. 7 to start OCTA capture under the capture conditions specified in S1201. The imaging control unit 101-03 instructs the tomographic imaging apparatus 100 to perform OCTA imaging based on the settings instructed by the operator in S1201, and the tomographic imaging apparatus 100 acquires a corresponding OCT tomographic image. . The tomographic imaging apparatus 100 also acquires SLO images and executes tracking processing based on the SLO moving images.

<Step 1203>
The image acquisition unit 101-01 and the image processing unit 101-04 generate a motion contrast image (motion contrast data) based on the OCT tomographic image acquired in S1202. After generating a motion contrast image in the same procedure as S303 of the first embodiment, the correction unit 101-43 corrects the projection artifact 802 that has occurred on the motion contrast image in the same procedure as S304 of the first embodiment. Execute the process to suppress.

<Step 1204>
The display control unit 101-05 causes the display unit 104 to display the tomographic image generated in S1203, the three-dimensional and frontal motion contrast images, and information regarding the imaging conditions.

<Step 1205>
The image quality improvement unit 101-47 inputs a low-quality motion contrast image generated from a small number of tomographic images into a machine learning model, thereby producing a high-quality (low-quality) image equivalent to that generated from a large number of tomographic images. Generate motion-contrast (noisy, high-contrast) images. Here, the machine learning model is composed of a group of pairs of input data, which is a low-quality image acquired under predetermined shooting conditions assumed to be the processing target, and output data, which is a high-quality image corresponding to the input data. Refers to a function generated by performing machine learning using trained training data.

Note that the predetermined imaging conditions include the imaging site, imaging method, imaging angle of view, image size, and the like.

Furthermore, in this embodiment, when the user presses the button 911 (Denoise button) shown in the upper right corner of the report screen in FIG. It shall be.

In this embodiment, the input data used as training data is a low-quality motion contrast image generated from a single cluster with a small number of tomographic images, and the output data (used as training data) is the sum of multiple aligned motion contrast data. A high-quality motion contrast image obtained on average is used. Note that the output data used as the teacher data is not limited to this, and may be, for example, a high-quality motion contrast image generated from a single cluster composed of a large number of tomographic images. Further, the output data used as the teacher data may be a high-quality motion contrast image obtained by making a motion contrast image having a higher resolution (higher magnification) than the input image the same resolution (same magnification) as the input image. Note that the pair of input images and output images used for training the machine learning model is not limited to the above, and any known combination of images may be used. For example, an image obtained by adding a first noise component to a motion contrast image acquired by the photographing device 10 or another device is used as an input image, and the motion contrast image (obtained by the photographing device 10 or other device) is added to the (first noise component). An image to which a second noise component (different from the noise component) is added may be used as an output image for training a machine learning model. In other words, the image quality improvement unit 101-47 uses a trained model for image quality improvement obtained by learning learning data including three-dimensional motion contrast data of the fundus, to improve the three-dimensional motion input as the input image. Anything that can improve the image quality of the contrast data may be used.

FIG. 13 shows a configuration example of a machine learning model in the image quality improvement unit 101-47 according to this embodiment. The machine learning model is a convolutional neural network (CNN), and is composed of a plurality of layers responsible for processing and outputting a group of input values. The types of layers included in the configuration include a convolution layer, a downsampling layer, an upsampling layer, and a merge layer. The convolution layer is a layer that performs convolution processing on a group of input values according to parameters such as the set filter kernel size, the number of filters, the stride value, and the dilation value. Note that the number of dimensions of the kernel size of the filter may also be changed depending on the number of dimensions of the input image. The downsampling layer is a process of thinning out or combining input value groups to make the number of output value groups smaller than the number of input value groups. Specifically, for example, there is Max Pooling processing. The upsampling layer is a process of making the number of output value groups larger than the number of input value groups by duplicating the input value group or adding a value interpolated from the input value group. Specifically, for example, there is linear interpolation processing. The compositing layer is a layer that receives a group of values, such as a group of output values of a certain layer or a group of pixel values constituting an image, from a plurality of sources, and performs a process of combining them by concatenating or adding them. In such a configuration, the group of pixel values that make up the input image 1301 is outputted through the convolution processing block and the group of pixel values that make up the input image 1301 are combined in the synthesis layer. Thereafter, the combined pixel values are formed into a high-quality image 1302 in the final convolution layer. Although not shown, examples of changes to the CNN configuration include incorporating a batch normalization layer or an activation layer using a rectifier linear unit after the convolution layer. You may do so.

Note that although the image to be processed is described as a two-dimensional image in FIG. 13 to simplify the explanation, the present invention is not limited to this. For example, the present invention also includes a case where a three-dimensional low-quality motion contrast image is input to the high-quality image improvement unit 101-47 and a three-dimensional high-quality motion contrast image is output.

<Step 1206>
The operator uses the input unit 103 to instruct the start of OCTA measurement processing.

In this embodiment, double-clicking on the motion contrast image on the report screen transitions to the OCTA measurement screen. The motion contrast image is enlarged and displayed, and the analysis unit 101-46 starts measurement processing.

Any measurement process may be performed as the type of measurement process, and in this embodiment, the type of analysis shown in the item 905 displayed by selecting Density Analysis 903 or Tools button 904 in FIG. By selecting an item 912 regarding the number of dimensions of analysis, the desired type of measurement is specified.

In this embodiment, a case will be described in which a two-dimensional measurement value is calculated for three-dimensionally emphasized and specified blood vessel region or blood vessel center line data. Specifically, two-dimensional blood vessel density (VAD) is measured in the surface layer of the retina, and blood vessel area is measured in the outer layer of the retina.

Blood vessels in the retina and choroid (unlike in other organs) run along a layered structure, so they are often calculated as two-dimensional measured values. However, when image processing that emphasizes blood vessels is applied using a two-dimensional motion contrast image, especially a frontal motion contrast image, linear structures other than blood vessels are also emphasized, resulting in erroneous detection as blood vessels. There are cases. For example, the boundary (edge) of a cyst region is curved on a two-dimensional image, so it is likely to be mistakenly detected as a blood vessel region, but on a three-dimensional image, it is curved and therefore less likely to be mistakenly detected. As in this embodiment, by detecting the blood vessel region three-dimensionally and then calculating it as a two-dimensional measurement value, it is possible to reduce false detection of other structures and provide a two-dimensional measurement value that is easy for the user to understand. It can be calculated.

Next, the analysis unit 101-46 performs preprocessing for measurement processing. Although any known image processing can be applied as preprocessing, in this embodiment, image enlargement and morphological calculation (top hat filter processing) are performed as preprocessing, similar to the first embodiment.

<Step 1208>
The measurement unit 101-463 measures the blood vessel density for a single examination image based on information regarding the measurement target area designated by the operator. Subsequently, the display control section 101-05 displays the measurement results on the display section 104. More specifically, the measurement unit 101-463 measures at least a portion of the three-dimensional motion contrast data after projection artifacts have been reduced by the correction unit 101-43 and image quality enhancement processing has been performed by the image quality enhancement unit 101-47. A two-dimensional blood vessel measurement value is calculated for the three-dimensionally acquired blood vessel region using information regarding the three-dimensional blood vessel region. Specifically, the two-dimensional blood vessel density in the superficial layer of the retina and the area of the new blood vessel region in the outer layer of the retina are calculated. That is, the measurement unit 101-463 is an example of a calculation means that calculates a blood vessel measurement value in a two-dimensional space using information regarding a three-dimensional blood vessel region in at least a part of the high-quality three-dimensional motion contrast data. Equivalent to.

There are two types of indexes for two-dimensional blood vessel density: VAD and VLD, and in this embodiment, the procedure for calculating VAD will be explained as an example. Note that the procedure for calculating VLD will also be described later.

When the operator inputs an instruction to manually correct the blood vessel region or blood vessel center line data from the input unit 103, the analysis unit 101-46 uses the position information specified by the operator via the input unit 103 to Manually correct the vessel area or vessel centerline data and recalculate the measured values.

S810 to S820 for two-dimensional VAD measurement in the retinal surface layer and area measurement of choroidal neovascularization in the outer retinal layer; S830 to S850 for two-dimensional VLD measurement in the retinal superficial layer and total vessel length measurement in the horizontal direction of choroidal neovascularization in the outer retinal layer. Each will be explained below.

<Step 1210>
The display control unit 101-05 displays a report regarding the measurement results performed in S1208 on the display unit 104.

In this embodiment, the VAD map and VAD sector map measured in the retinal surface layer are superimposed on the upper row of the single measurement report shown in FIG. 9(b), and the choroidal neovascular region identified in the outer retinal layer is displayed on the lower row. Displays binary images and measured area values. This makes it possible to view and understand the vascular pathology at different depth positions. Furthermore, the position of CNV in the outer retinal layer can be identified and accurately quantified.

Furthermore, details of the process executed in S1208 will be described with reference to the flowchart shown in FIG. 6(a).

<Step 810>
The operator sets a region of interest in the measurement process via the input unit 103. That is, the target region for calculating the blood vessel density is specified by the user. In this embodiment, the measurement contents are as follows: 1) A two-dimensional VAD map and a two-dimensional VAD sector map in the superficial layer of the retina. 2) The area of choroidal neovascularization (CNV) in the outer layer of the retina is calculated.

Therefore, in terms of the in-plane direction, the region of interest in the retinal surface layer is (i) the entire image, (ii) a sector region centered on the fixation lamp position (within an annular region defined by an inner circle with a diameter of 1 mm and an outer circle with a diameter of 3 mm). is an area divided into four fan shapes of Superior, Inferior, Nasal, and Temporal, and the inner circular area). Furthermore, in the depth direction, the depth range is the same as in S810 of the first embodiment. Since the depth range in the depth direction is specified by the layer boundary 910 in FIG. 9(a), Map and Sector are selected from the Density Map/Sector 902 items in FIG. 9(a), and Area Density is selected from the Density Analysis 903 items. Automatically set by selecting each. The two-dimensional direction of the fundus surface can be specified by moving the sector, and the depth direction of the fundus can be specified by specifying the depth range of the displayed En-Face image. Furthermore, for the outer retinal layer, a layer boundary corresponding to the outer retinal layer (an area surrounded by the OPL/ONL boundary and a position where the Bruch's membrane boundary is moved 20 μm toward the deep layer side of the boundary) is specified. The region of interest in the in-plane direction may be specified for the entire image, or the region of interest may be manually set by the operator. For example, after selecting the measurement type (Area/Volume in this case) from the setting screen 905 that is displayed when a button 904 shown in FIG. may be set. Note that the region of interest may be directly specified with respect to the three-dimensional motion contrast data while the three-dimensional motion contrast data is displayed.

<Step 820>
The measurement unit 101-463 performs measurement processing based on the binary image of the blood vessel region obtained in S1207. In this embodiment, without generating a projection image for the binary image of the three-dimensional blood vessel region identified in S1207, calculation processing is performed on the A-scan group that satisfies a predetermined condition, or in a direction perpendicular to the A-scan direction. A two-dimensional measurement value is calculated by performing the measurement.

For example, when calculating a two-dimensional VAD, the measurement unit 101-463 performs the following processing on a binary image of a three-dimensional blood vessel region. That is, first, consider a unit A-scan group Ag that includes each A-scan at its center and neighboring A-scans. (The unit A-scan group Ag can be defined by any number of A-scans, but in this embodiment, it is nine.) The three-dimensional region of interest specified in S810 in each A-scan included in the unit A-scan group. At least one pixel (non-zero pixel, that is, a white pixel in FIG. 14(a)) belonging to the blood vessel region V1 is included in the depth range defined by (superficial curved surface Bu and deep curved surface Bl in FIG. 14(a)). Count the number of A scans that occur. By dividing the number of A scans as described above (including non-zero pixels within a predetermined depth range) by the number of unit A scan groups (9 in this embodiment) and multiplying by 100, the number of A scans that the A scan has The value (unit: %) of two-dimensional VAD in XY coordinates can be calculated. For example, if there is only one A-scan that satisfies the above conditions in a unit A-scan group, the two-dimensional VAD value calculated at the position of the A-scan is (1/9) x 100% ≒ 11.11% It is calculated as follows. In this embodiment, the two-dimensional VAD is calculated in the retinal surface layer, so in the measurement in the retinal surface layer, the superficial side curved surface Bu is a curved surface defined by the inner limiting membrane boundary, and the deep layer side curved surface Bl is the curved surface defined by the ganglion cell layer (GCL)-inner limiting membrane boundary. Specify the curved surface defined by the reticular layer (IPL) boundary. Note that the present invention is not limited to this, and any depth range, for example, the deep layer of the retina, may be designated as the depth range to be measured. Furthermore, an image (two-dimensional VAD map) having two-dimensional blood vessel density (VAD) values calculated at each XY position is generated.

In addition, in each three-dimensional sector region (cylindrical region C and partial circular columnar region (S/I/N/T) in FIG. 14(c)) set in S810 on the binary image of the three-dimensional blood vessel region, the surface layer The number of A-scans Aw is calculated such that at least one pixel belonging to the blood vessel region (white pixel in FIG. 14C) is included within the depth range of the side curved surface Bu and the deep side curved surface Bl. The calculated number of Aw is divided by the number of A scans belonging to the three-dimensional sector area, and the value multiplied by 100 is calculated as the two-dimensional blood vessel density (VAD) in the sector. Furthermore, a map (two-dimensional VAD sector map) having two-dimensional blood vessel density (VAD) values calculated in each sector region is generated.

In addition, in the outer retinal layer, the depth range defined by the region of interest specified in S810 (on the superficial side of FIG. 14(a) The number of A scans in which at least one pixel (white pixel in FIG. 14A) belonging to the blood vessel region is included in the curved surface Bu and the deep curved surface Bl is counted. By multiplying the number of A-scans that include non-zero pixels within the predetermined depth range by the in-plane area (unit: mm ² ) occupied by one A-scan, the (predetermined depth range) of the three-dimensional blood vessel region is calculated. The area occupied in the in-plane direction (in ) can be calculated. Note that the method for calculating two-dimensional blood vessel measurement values of the present invention is not limited to the above. For example, a frontal binary image generated by projecting and binarizing an image that emphasizes the choroidal neovascular region within a region of interest corresponding to the outer layer of the retina, or a frontal binary image generated by projecting a three-dimensional binarized image. The area of the choroidal neovascular region may be calculated on the frontal binary image.

Above, we explained the procedure for measuring 2D VAD in the retinal surface layer based on the identified 3D blood vessel area and blood vessel area in the outer retinal layer as an example. When measuring the two-dimensional VLD and the two-dimensional blood vessel length in the outer retinal layer, steps S830 to S850 shown in FIG. 6(b) are executed instead of S810 to S820 described above.

<Step 830>
The measurement unit 101-463 performs three-dimensional thinning processing on the binary image of the blood vessel region generated in S1207, thereby creating a binary image with a line width of 1 pixel corresponding to the center line of the blood vessel (hereinafter referred to as a skeleton image). generate.

<Step 840>
The operator sets the region of interest via the input unit 103 in the same manner as in step S820. In this embodiment, a two-dimensional VLD map and a two-dimensional VLD sector map are calculated as measurement contents. Note that if you do not want to display the VLD map or VLD sector map in a superimposed manner on the motion contrast image, you may set the check box of the Map or Sector item in the Density Map/Sector 902 in FIG. 9(a) to unselected.

<Step 850>
The measurement unit 101-463 performs measurement processing based on the skeleton image obtained in S830. That is, measurement is performed from the three-dimensional skeleton specified in the retinal surface layer. In this embodiment, for each A-scan position of the three-dimensional skeleton image, a unit A-scan group Ag including neighboring A-scans with the relevant A-scan as the center is considered, as in S820. (The unit A-scan group Ag can be defined by any number of A-scans, but in this embodiment, it is nine.) The three-dimensional region of interest specified in S840 in each A-scan included in the unit A-scan group. An A scan in which at least one pixel belonging to the blood vessel region (the white pixel in FIG. 14(b)) is included in the depth range defined by (superficial curved surface Bu and deep curved surface Bl in FIG. 14(b)) Calculate the number. The calculated number of A-scans is multiplied by the size (unit: mm) occupied by one A-scan (in the direction orthogonal to the A-scan), and the unit A-scan group is orthogonal to the A-scan group. The two-dimensional VLD is calculated by dividing by the area occupied in the direction (unit: mm ² ). (Unit: mm ⁻¹ ) Furthermore, an image (two-dimensional VLD map) having the value of blood vessel density (two-dimensional VLD) calculated at each XY position is generated.

In addition, in each three-dimensional sector area set in S840 on the three-dimensional skeleton image (cylindrical area C and partial circular columnar area (S/I/N/T) in FIG. 14(c)), the surface side curved surface Bu and The number of A scans Aw is calculated so that at least one pixel belonging to the three-dimensional skeleton region V2 is included within the depth range of the deep side curved surface Bl. The calculated number of Aw is multiplied by the size (unit: mm) occupied by one A scan (in the direction orthogonal to the A scan), and the size (unit: mm) occupied by the A scan group belonging to the three-dimensional sector area is calculated. The value divided by the area (in the direction perpendicular to the A-scan group) (unit: mm ² ) is calculated as the two-dimensional VLD (unit: mm ⁻¹ ) in the sector. Furthermore, a map (two-dimensional VLD sector map) having two-dimensional VLD values calculated in each sector area is generated.

In addition, in the outer retinal layer, the depth range defined by the region of interest specified in S840 (on the surface layer side in FIG. 14(b) The number of A scans in which at least one pixel (white pixel in FIG. 14(b)) belonging to the three-dimensional skeleton region is included in the curved surface Bu and the deep side curved surface Bl is counted. By multiplying the number of A-scans that include non-zero pixels within the predetermined depth range by the in-plane length (unit: mm) occupied by one A-scan, the blood vessel region (in the predetermined depth range) is calculated. The sum of lengths in the in-plane direction can be calculated. Note that the method for calculating two-dimensional blood vessel measurement values of the present invention is not limited to the above. For example, a frontal binary image generated by projecting an image that emphasizes the choroidal neovascular region within the region of interest corresponding to the outer layer of the retina, binarizing it, and thinning it, or an image that is binarized in three dimensions and thinning it. The length of the choroidal neovascular region may be calculated on the frontal binary image generated by projecting the choroidal neovascular region.

According to the configuration described above, the image processing device 101 applies image quality improvement processing using machine learning to the acquired three-dimensional motion contrast image of the eye to be examined, and identifies a three-dimensional blood vessel region. Based on the three-dimensional blood vessel region, the two-dimensional blood vessel density in the surface layer of the retina and the area of choroidal neovascularization (CNV) in the outer layer of the retina are calculated as two-dimensional blood vessel measurement values without generating a frontal image. Thereby, a desired region such as a three-dimensional blood vessel region can be emphasized or specified and accurately measured while suppressing the influence of the signal intensity and image quality of the OCT tomographic image.

(Modification 11)
Execution of image quality enhancement processing in screen transitions in the various embodiments and modifications described above will be explained using FIGS. 9(a) and 9(b). FIG. 9(a) is an example of a screen in which the OCTA image in FIG. 9(b) is enlarged and displayed. In FIG. 9(a), a button 911 is also displayed as in FIG. 9(b). The screen transition from FIG. 9(b) to FIG. 9(a) can be made, for example, by double-clicking the OCTA image, and from FIG. 9(a) to FIG. 9(b) can be done by clicking the close button (not shown). Transition. Note that the screen transition is not limited to the method shown here, and a user interface (not shown) may be used.

If execution of image quality improvement processing is specified at the time of screen transition (button 911 is active), that state is maintained even at the time of screen transition. That is, when transitioning to the screen of FIG. 9(a) while a high-quality image is being displayed on the screen of FIG. 9(b), the high-quality image is also displayed on the screen of FIG. 9(a). Then, the button 911 is activated. The same holds true when transitioning from FIG. 9(a) to FIG. 9(b). In FIG. 9A, it is also possible to switch the display to a low-quality image by specifying a button 911.

Regarding screen transitions, it is not limited to the screens shown here, but if the transition is to a screen that displays the same imaging data, such as a follow-up display screen or a panoramic display screen, the display state of high-quality images can be maintained. Perform the transition as it is. That is, on the display screen after the transition, an image corresponding to the state of the button 911 on the display screen before the transition is displayed. For example, if the button 911 on the display screen before transition is in an active state, a high-quality image is displayed on the display screen after transition. Further, for example, if the active state of the button 911 on the display screen before the transition is released, a low-quality image is displayed on the display screen after the transition. Note that when the button 911 on the follow-up observation display screen becomes active, the multiple images obtained at different dates and times (different examination dates) displayed side by side on the follow-up observation display screen are switched to high-quality images. It's okay. That is, when the button 911 on the display screen for follow-up observation becomes active, the configuration may be such that the effect is reflected on a plurality of images obtained at different dates and times at once.

Note that an example of a display screen for follow-up observation is shown in FIG. When the tab 3801 is selected in response to an instruction from the examiner, a display screen for progress observation is displayed as shown in FIG. At this time, the examiner can change the depth range of the measurement target area by selecting from the default depth range set (3802 and 3803) displayed in the list box. For example, the superficial layer of the retina is selected in list box 3802, and the deep layer of the retina is selected in list box 3803. The upper display area displays the analysis result of the motion contrast image of the superficial layer of the retina, and the lower display area displays the analysis result of the motion contrast image of the deep layer of the retina. That is, when a depth range is selected, multiple images taken at different dates and times are collectively changed to parallel display of analysis results of multiple motion contrast images in the selected depth range.

At this time, if the display of the analysis results is set to a non-selected state, the display may be changed to parallel display of a plurality of motion contrast images at different dates and times. Then, when the button 911 is designated in response to an instruction from the examiner, the display of the plurality of motion contrast images is changed to the display of the plurality of high-quality images at once.

In addition, when the display of analysis results is in the selected state, when the button 911 is specified in response to an instruction from the examiner, the display of the analysis results of multiple motion contrast images changes to the display of the analysis results of multiple high-quality images. The display will be changed all at once. Here, the display of the analysis results may be such that the analysis results are superimposed on the image with arbitrary transparency. At this time, the display of the analysis results may be changed to, for example, a state in which the analysis results are superimposed on the displayed image with arbitrary transparency. Further, the change to the display of the analysis result may be, for example, a change to the display of an image (for example, a two-dimensional map) obtained by blending the analysis result and the image with arbitrary transparency.

Further, the type of layer boundary and the offset position used for specifying the depth range can be changed all at once from user interfaces 3805 and 3806, respectively. By displaying the tomographic images together and moving the layer boundary data superimposed on the tomographic images according to instructions from the examiner, the depth range of multiple motion contrast images at different times can be changed at once. may be done. At this time, when a plurality of tomographic images with different dates and times are displayed side by side and the above movement is performed on one tomographic image, the layer boundary data may be similarly moved on other tomographic images.

Further, the image projection method and the presence or absence of projection artifact suppression processing may be changed by selecting from a user interface such as a context menu, for example.

Alternatively, a selection button 3807 may be selected to display a selection screen, and an image selected from the image list displayed on the selection screen may be displayed. Note that the arrow 3804 displayed at the top of FIG. 17 is a mark indicating the currently selected test, and the reference test (Baseline) is the test selected at the time of follow-up imaging (the one shown in FIG. 17). The leftmost image). Of course, a mark indicating the reference test may be displayed on the display section.

Furthermore, when the "Show Difference" check box 3808 is designated, the measurement value distribution (map or sector map) for the reference image is displayed on the reference image. Furthermore, in this case, a difference measurement value map between the measurement value distribution calculated for the reference image and the measurement distribution calculated for the image displayed in the area is displayed in the area corresponding to the other inspection dates. do. As the measurement results, a trend graph (a graph of measurement values for images on each inspection day obtained by measuring changes over time) may be displayed on the report screen. That is, time-series data (for example, a time-series graph) of multiple analysis results corresponding to multiple images at different dates and times may be displayed. At this time, analysis results related to dates and times other than the multiple dates and times corresponding to the multiple images being displayed are also displayed in a state that allows them to be distinguished from the multiple analysis results corresponding to the multiple images being displayed (for example, in a chronological order). It may also be displayed as time series data (the color of each point on the graph differs depending on whether or not an image is displayed). Further, the regression line (curve) of the trend graph and the corresponding mathematical formula may be displayed on the report screen.

In this modified example, the motion contrast image has been described, but the present invention is not limited to this. Images related to processing such as display, high image quality, and image analysis according to this modification may be tomographic images. Furthermore, not only a tomographic image but also a different image such as an SLO image, a fundus photograph, or a fluorescent fundus photograph may be used. In that case, the user interface for executing image quality enhancement processing may be one that instructs execution of image quality enhancement processing on multiple images of different types, or one that instructs to execute image quality enhancement processing on multiple images of different types, or one that allows you to select any image from multiple images of different types. There may also be an instruction to execute high image quality processing.

With this configuration, the display control unit 101-05 can display the image processed by the image quality improvement unit 101-47 according to this modification on the display unit 104. At this time, as described above, if at least one of the multiple conditions regarding display of high-quality images, display of analysis results, depth range of the displayed frontal image, etc. is selected, the display screen is Even if the transition is made, the selected state may be maintained.

Furthermore, as described above, when at least one of the plurality of conditions is selected, even if the other conditions are changed to the selected state, the selected state of at least one of the conditions is maintained. It's okay. For example, when the display of the analysis results is in the selected state, the display control unit 101-05 displays the analysis results of the low-quality image in response to an instruction from the examiner (for example, when the button 911 is specified). may be changed to displaying the analysis results of high-quality images. In addition, when the display of the analysis results is in the selected state, the display control unit 101-05 displays the analysis results of the high-quality image in response to an instruction from the examiner (for example, when the designation of the button 911 is canceled). The display may be changed to display the analysis results of low-quality images.

In addition, when the display of high-quality images is not selected, the display control unit 101-05 controls the display of low-quality images in response to an instruction from the examiner (for example, when the designation of displaying analysis results is canceled). The display of the image analysis results may be changed to the display of low-quality images. In addition, when the display of high-quality images is in a non-selected state, the display control unit 101-05 controls the display of low-quality images in response to an instruction from the examiner (for example, when the display of analysis results is specified). The display may be changed to display the analysis results of low-quality images. In addition, when displaying a high-quality image is selected, the display control unit 101-05 displays a high-quality image in response to an instruction from the examiner (for example, when the designation for displaying analysis results is canceled). The display of the analysis results may be changed to the display of high-quality images. In addition, when displaying a high-quality image is selected, the display control unit 101-05 displays a high-quality image in response to an instruction from the examiner (for example, when displaying analysis results is specified). may be changed to displaying the analysis results of high-quality images.

Also, consider a case where a high-quality image is displayed in a non-selected state and a first type of analysis result is displayed in a selected state. In this case, the display control unit 101-05 performs the first type of analysis of the low-quality image in response to an instruction from the examiner (for example, when displaying the second type of analysis results is specified). The display of the results may be changed to display of the second type of analysis results of low-quality images. Also, consider a case where a high-quality image is displayed in a selected state and a first type of analysis result is displayed in a selected state. In this case, the display control unit 101-05 performs the first type of analysis of the high-quality image in response to an instruction from the examiner (for example, when displaying the second type of analysis results is specified). The display of the results may be changed to display of the second type of analysis results of high-quality images.

Note that, as described above, the display screen for follow-up observation may be configured so that these display changes are reflected at once on a plurality of images obtained at different dates and times. Here, the display of the analysis results may be such that the analysis results are superimposed on the image with arbitrary transparency. At this time, the display of the analysis results may be changed to, for example, a state in which the analysis results are superimposed on the displayed image with arbitrary transparency. Further, the change to the display of the analysis result may be, for example, a change to the display of an image (for example, a two-dimensional map) obtained by blending the analysis result and the image with arbitrary transparency.

(Modification 12)
In the various embodiments and modifications described above, the display control unit 101-05 selects a high-quality image generated by the high-quality image improvement unit 101-47 and an input image according to an instruction from the examiner. The image can be displayed on the display unit 104. Furthermore, the display control unit 101-05 may switch the display on the display unit 104 from a captured image (input image) to a high-quality image in response to an instruction from the examiner. That is, the display control unit 101-05 may change the display of a low-quality image to the display of a high-quality image in response to an instruction from the examiner. Furthermore, the display control unit 101-05 may change the display of a high-quality image to the display of a low-quality image in response to an instruction from the examiner.

Furthermore, the image quality improvement unit 101-47 instructs the image quality improvement engine (trained model for image quality improvement) to start image quality improvement processing (inputting images to the image quality improvement engine) in response to an instruction from the examiner. The display control unit 101-05 may cause the display unit 104 to display the high-quality image generated by the image quality improvement unit 101-47. On the other hand, when an input image is captured by the imaging device (tomographic imaging device 100), the high-quality image automatically generates a high-quality image based on the input image, and the display control unit 101-05 A high-quality image may be displayed on the display unit 104 in response to instructions from the examiner. Here, the image quality improvement engine includes a trained model that performs the above-described image quality improvement processing (image quality improvement processing).

Note that these processes can be similarly performed for outputting the analysis results. That is, the display control unit 101-05 may change the display of the analysis results of low-quality images to the display of the analysis results of high-quality images in response to instructions from the examiner. Further, the display control unit 101-05 may change the display of the analysis results of the high-quality images to the display of the analysis results of the low-quality images in response to an instruction from the examiner. Of course, the display control unit 101-05 may change the display of the analysis result of the low-quality image to the display of the low-quality image in response to an instruction from the examiner. Further, the display control unit 101-05 may change the display of the low-quality image to the display of the analysis result of the low-quality image in response to an instruction from the examiner. Further, the display control unit 101-05 may change the display of the analysis result of the high-quality image to the display of the high-quality image in response to an instruction from the examiner. Further, the display control unit 101-05 may change the display of the high-quality image to the display of the analysis result of the high-quality image in response to an instruction from the examiner.

Further, the display control unit 101-05 may change the display of the analysis result of the low-quality image to the display of another type of analysis result of the low-quality image in response to an instruction from the examiner. Further, the display control unit 101-05 may change the display of the analysis result of the high-quality image to the display of another type of analysis result of the high-quality image in response to an instruction from the examiner.

Here, the display of the analysis result of the high-quality image may be one in which the analysis result of the high-quality image is displayed superimposed on the high-quality image with arbitrary transparency. Moreover, the display of the analysis result of the low-quality image may be one in which the analysis result of the low-quality image is displayed superimposed on the low-quality image with arbitrary transparency. At this time, the display of the analysis results may be changed to, for example, a state in which the analysis results are superimposed on the displayed image with arbitrary transparency. Further, the change to the display of the analysis result may be, for example, a change to the display of an image (for example, a two-dimensional map) obtained by blending the analysis result and the image with arbitrary transparency.

(Modification 13)
On the report screen in the various embodiments and modifications described above, analysis results such as the thickness of a desired layer and various blood vessel densities may be displayed. In addition, the optic disc, macular area, blood vessel area, nerve fiber bundle, vitreous area, macular area, choroid area, sclera area, lamina cribrosa area, retinal layer boundary, retinal layer boundary edge, photoreceptor cells, blood cells, Values (distribution) of parameters related to a region of interest including at least one of a blood vessel wall, a blood vessel inner wall boundary, a blood vessel outer wall boundary, a ganglion cell, a corneal region, an angle region, Schlemm's canal, etc. may be displayed as an analysis result. At this time, for example, by analyzing a medical image to which various artifact reduction processes have been applied, highly accurate analysis results can be displayed. Note that artifacts include, for example, false image areas caused by light absorption by blood vessel areas, projection artifacts, and band-shaped artifacts in the front image that occur in the main scanning direction of the measurement light due to the condition of the eye being examined (movement, blinking, etc.). There may be. Furthermore, the artifact may be of any type, as long as it is a defective area that randomly appears on a medical image of a predetermined region of a subject each time the image is taken. Further, the values (distribution) of parameters regarding a region including at least one of the various artifacts (impossible regions) as described above may be displayed as an analysis result. Further, the values (distribution) of parameters related to a region including at least one of an abnormal site such as drusen, new blood vessels, vitiligo (hard vitiligo), and pseudodrusen may be displayed as an analysis result.

Further, the analysis results may be displayed as an analysis map, sectors showing statistical values corresponding to each divided area, or the like. Note that the analysis results may be generated using a trained model (analysis result generation engine, trained model for analysis result generation) obtained by learning the analysis results of medical images as learning data. . At this time, the trained model is trained using learning data that includes a medical image and an analysis result of the medical image, or learning data that includes a medical image and the analysis result of a medical image of a different type than the medical image. It may be obtained by In addition, the trained model is obtained by learning using learning data that includes input data that is a set of multiple medical images of different types of predetermined parts, such as frontal tomographic images and frontal motion contrast images. Good too. Here, the frontal tomographic image corresponds to the En-Face image of the tomographic image, and the frontal motion contrast image corresponds to the En-Face image of OCTA. Furthermore, it may be configured to display analysis results obtained using high-quality images generated by trained models for high-quality images. Note that the trained model for high image quality is obtained by learning training data in which the first image is used as input data and the second image, which has higher image quality than the first image, is used as correct data. It's okay. At this time, the second image is processed to increase contrast or reduce noise by, for example, superimposing a plurality of first images (for example, averaging processing of a plurality of first images obtained by positioning). It may be a high-quality image that has been subjected to the above process.

Further, the input data included in the learning data may be a high-quality image generated by a trained model for high-quality images, or may be a set of a low-quality image and a high-quality image. In addition, the learning data includes, for example, at least analysis values obtained by analyzing the analysis area (e.g., average value, median value, etc.), a table including the analysis values, an analysis map, the position of the analysis area such as a sector in an image, etc. The input data may be labeled (annotated) with information including one as the correct data (for supervised learning). Note that the analysis result obtained by the learned model for generating the analysis result may be displayed in response to an instruction from the examiner.

Further, various diagnostic results such as glaucoma and age-related macular degeneration may be displayed on the report screen in the various embodiments and modified examples described above. At this time, for example, by analyzing a medical image to which the various artifact reduction processes described above have been applied, highly accurate diagnostic results can be displayed. Further, the diagnosis result may be displayed by displaying the position of the identified abnormal region on the image, or by displaying the state of the abnormal region by text or the like. Furthermore, the classification results (for example, Curtin classification) of abnormal regions and the like may be displayed as the diagnosis results. Further, as the classification result, for example, information indicating the probability of each abnormal site (for example, a numerical value indicating a ratio) may be displayed.

Note that the diagnostic results may be generated using a trained model (a diagnostic result generation engine, a trained model for generating diagnostic results) obtained by learning the diagnostic results of medical images as learning data. . In addition, the trained model can be trained using training data that includes a medical image and the diagnosis result of the medical image, or training data that includes the medical image and the diagnosis result of a different type of medical image than the medical image. It may be something you have obtained. Furthermore, the system may be configured to display diagnostic results obtained using high-quality images generated by trained models for high-quality images.

Further, the input data included in the learning data may be a high-quality image generated by a trained model for high-quality images, or may be a set of a low-quality image and a high-quality image. In addition, the learning data includes, for example, the diagnosis name, the type and condition (degree) of the lesion (abnormal site), the position of the lesion in the image, the position of the lesion relative to the region of interest, findings (image interpretation findings, etc.), and the basis for the diagnosis name (positive The input data is labeled (annotated) as the correct data (for supervised learning), including at least one of the following: negative medical support information, etc.), grounds for denying the diagnosis (negative medical support information), etc. It may be data. Note that the configuration may be such that the diagnostic results obtained by the trained model for generating diagnostic results are displayed in response to instructions from the examiner.

Further, on the report screen in the various embodiments and modifications described above, object recognition results (object detection results) and segmentation results such as the above-mentioned regions of interest, artifacts, and abnormal regions may be displayed. At this time, for example, a rectangular frame or the like may be superimposed and displayed around the object on the image. Furthermore, for example, a color or the like may be superimposed and displayed on an object in the image. Note that object recognition results and segmentation results are generated using a trained model (object recognition engine, object recognition It may be generated using a trained model segmentation engine (a trained model for segmentation). Note that the analysis result generation and diagnosis result generation described above may be obtained by using the object recognition results and segmentation results described above. For example, analysis result generation or diagnostic result generation processing may be performed on a region of interest obtained through object recognition or segmentation processing.

Further, the trained model described above may be a trained model obtained by learning using learning data including input data that is a set of a plurality of different types of medical images of a predetermined region of a subject. At this time, as input data included in the learning data, for example, input data including a set of a motion contrast frontal image and a brightness frontal image (or a brightness tomographic image) of the fundus can be considered. Further, as the input data included in the learning data, for example, input data including a set of a tomographic image of the fundus (B scan image) and a color fundus image (or a fluorescent fundus image) can be considered. Further, the plurality of medical images of different types may be any image as long as they are obtained using different modalities, different optical systems, or different principles.

Further, the trained model described above may be a trained model obtained by learning using learning data including input data that is a set of a plurality of medical images of different parts of the subject. At this time, as input data included in the learning data, for example, input data including a set of a tomographic image of the fundus (B-scan image) and a tomographic image of the anterior segment (B-scan image) can be considered. Furthermore, as input data included in the learning data, for example, input data that is a set of a three-dimensional OCT image (three-dimensional tomographic image) of the macula in the fundus and a circle scan (or raster scan) tomographic image of the optic disc in the fundus, etc. can also be considered.

Note that the input data included in the learning data may be a plurality of medical images of different parts and different types of the subject. At this time, the input data included in the learning data may be, for example, input data that includes a tomographic image of the anterior segment of the eye and a color fundus image. Further, the above-mentioned trained model may be a trained model obtained by learning using learning data including input data that is a set of a plurality of medical images of a predetermined region of a subject at different photographing angles of view. Further, the input data included in the learning data may be a combination of a plurality of medical images obtained by time-divisionally dividing a predetermined region into a plurality of regions, such as a panoramic image. At this time, by using wide-angle images such as panoramic images as learning data, it is possible to obtain image features with higher accuracy because they contain more information than narrow-angle images. Processing results can be improved. Furthermore, the input data included in the learning data may be input data that is a set of a plurality of medical images of a predetermined region of the subject at different times.

Furthermore, the display screen on which at least one of the above-mentioned analysis results, diagnosis results, object recognition results, and segmentation results is displayed is not limited to the report screen. Such display screens include, for example, at least one display screen such as a shooting confirmation screen, a display screen for progress observation, and a preview screen for various adjustments before shooting (a display screen on which various live video images are displayed). may be displayed. For example, by displaying the at least one result obtained using the trained model described above on the imaging confirmation screen, the examiner can confirm highly accurate results even immediately after imaging. Further, the above-described display change between a low-quality image and a high-quality image may be, for example, a change in display between an analysis result of a low-quality image and an analysis result of a high-quality image.

Here, the various learned models described above can be obtained by machine learning using learning data. Machine learning includes, for example, deep learning that consists of multilayer neural networks. Further, for example, a convolutional neural network (CNN) can be used for at least a portion of the multi-layer neural network. Further, a technique related to an autoencoder (self-encoder) may be used for at least a portion of the multi-layer neural network. Furthermore, a technique related to backpropagation (error backpropagation method) may be used for learning. However, machine learning is not limited to deep learning, and any model may be used as long as it is capable of extracting (expressing) features of learning data such as images by itself through learning.

In addition, the high image quality engine (trained model for high image quality) may be a trained model obtained by additionally learning training data that includes at least one high quality image generated by the high image quality engine. good. At this time, it may be possible to select whether or not to use the high-quality image as learning data for additional learning based on an instruction from the examiner. Moreover, a learned model for generating correct data for generating correct data such as labeling (annotation) may be used to generate the correct data. At this time, the trained model for generating correct answer data may be obtained by (sequentially) additionally learning correct answer data obtained by labeling (annotation) by the examiner.

(Modification 14)
In the preview screen in the various embodiments and modifications described above, the learned model described above may be used for each at least one frame of a live video image. At this time, if a plurality of live moving images of different parts or types are displayed on the preview screen, a learned model corresponding to each live moving image may be used. With this, for example, even for live moving images, the processing time can be shortened, so the examiner can obtain highly accurate information before starting imaging. Therefore, for example, it is possible to reduce failures in re-imaging, etc., and thus it is possible to improve the accuracy and efficiency of diagnosis. Note that the plurality of live moving images may be, for example, a moving image of the anterior segment of the eye for alignment in the XYZ directions, and a frontal moving image of the fundus for focus adjustment of the fundus observation optical system or OCT focus adjustment. Further, the plurality of live moving images may be, for example, tomographic moving images of the fundus for OCT coherence gate adjustment (adjustment of the optical path length difference between the measurement optical path length and the reference optical path length).

Further, the moving image to which the above-described learned model can be applied is not limited to a live moving image, but may be a moving image stored (saved) in a storage unit, for example. At this time, for example, a moving image obtained by aligning at least one frame of the tomographic moving image of the fundus oculi stored (saved) in the storage unit may be displayed on the display screen. For example, if you want to observe the vitreous body in a suitable manner, you may first select a reference frame based on conditions such as as much vitreous body as possible on the frame. At this time, each frame is a tomographic image (B scan image) in the XZ direction. Then, a moving image in which other frames are aligned in the XZ directions with respect to the selected reference frame may be displayed on the display screen. At this time, for example, high-quality images (high-quality frames) sequentially generated by a trained model for improving image quality may be continuously displayed for each at least one frame of the moving image.

Note that as the above-mentioned method for positioning between frames, the same method may be applied for the positioning method in the X direction and the method for positioning in the Z direction (depth direction), or different methods may be applied. may be applied. Furthermore, alignment in the same direction may be performed multiple times using different techniques; for example, rough alignment may be followed by precise alignment. In addition, as a method for positioning, for example, positioning using retinal layer boundaries obtained by segmenting a tomographic image (B-scan image) (coarse in the Z direction), and positioning using multiple retinal layer boundaries obtained by dividing a tomographic image Positioning (accurate in the X and Z directions) using correlation information (similarity) between the region and the reference image, and one-dimensional projection images generated for each tomographic image (B scan image) (in the X direction) ) alignment, alignment (in the X direction) using a two-dimensional front image, etc. Alternatively, the configuration may be such that rough alignment is performed pixel by pixel, and then fine alignment is performed subpixel by subpixel.

Here, during various adjustments, there is a possibility that the object to be photographed, such as the retina of the eye to be examined, has not yet been successfully imaged. For this reason, since there is a large difference between the medical images input to the trained model and the medical images used as learning data, there is a possibility that high-quality images cannot be obtained with high accuracy. Therefore, when an evaluation value such as image quality evaluation of a tomographic image (B scan) exceeds a threshold value, display of high-quality moving images (continuous display of high-quality frames) may be automatically started. Further, when an evaluation value such as image quality evaluation of a tomographic image (B scan) exceeds a threshold value, the image quality improvement button may be changed to a state (active state) that can be specified by the examiner. Note that the image quality improvement button is a button for specifying execution of image quality improvement processing. Of course, the high-quality image button may be a button for instructing display of a high-quality image.

In addition, a different trained model for high image quality is prepared for each shooting mode with a different scanning pattern, etc., and the trained model for high image quality corresponding to the selected shooting mode is selected. Good too. Alternatively, one trained model for improving image quality obtained by learning training data including various medical images obtained in different imaging modes may be used.

(Modification 15)
In the various embodiments and modifications described above, when a trained model is undergoing additional learning, it may be difficult to output (inference/predict) using the trained model itself that is undergoing additional learning. For this reason, it is preferable to prohibit the input of medical images to a learned model that is undergoing additional learning. Further, another trained model that is the same as the trained model that is currently being additionally trained may be prepared as a preliminary trained model. At this time, during additional learning, it is preferable to allow medical images to be input to the preliminary trained model. Then, after the additional learning is completed, the trained model after the additional learning is evaluated, and if there is no problem, the preliminary trained model may be replaced with the trained model after the additional learning. Furthermore, if there is a problem, a preliminary trained model may be used. Note that for the evaluation of the trained model, for example, a trained model for classification for classifying high-quality images obtained by the trained model for high image quality from other types of images may be used. For example, a trained model for classification uses as input data multiple images including high-quality images and low-quality images obtained by a trained model for high image quality, and the types of these images are labeled (annotated). The model may be a trained model obtained by learning training data that includes the correct data as correct data. At this time, the type of image of the input data at the time of estimation (during prediction) is displayed together with information indicating the probability of each type of image included in the correct data at the time of learning (for example, a numerical value indicating a percentage). Good too. In addition to the above-mentioned images, the input data for the trained model for classification may also include superposition processing of multiple low-quality images (for example, averaging processing of multiple low-quality images obtained by positioning), etc. may include high-quality images that have undergone high contrast, noise reduction, and the like.

Further, it may be possible to selectively use a learned model obtained by learning for each part to be imaged. Specifically, a first trained model obtained using learning data including a first imaging region (lung, eye to be examined, etc.) and a learning model including a second imaging region different from the first imaging region. A plurality of trained models including a second trained model obtained using data can be prepared. The image processing unit 101-04 may have a selection means for selecting one of the plurality of trained models. At this time, the image processing unit 101-04 may have a control unit that performs additional learning on the selected trained model. The control means searches for data in which a photographed region corresponding to the selected trained model and a photographed image of the photographed region are paired, and uses the data obtained through the search as learning data. This learning can be performed as additional learning on the selected trained model. Note that the imaged region corresponding to the selected learned model may be obtained from information in the header of the data, or may be manually input by the examiner. Further, the data search may be performed via a network, for example, from a server in an external facility such as a hospital or research institute. Thereby, additional learning can be efficiently performed for each imaged body part using the photographed images of the body part corresponding to the learned model.

Note that the selection means and the control means may be constituted by a software module executed by a processor such as a CPU or an MPU of the image processing section 101-04. Furthermore, the selection means and the control means may be constituted by a circuit that performs a specific function, such as an ASIC, or an independent device.

Furthermore, when acquiring learning data for additional learning via a network from a server in an external facility such as a hospital or research institute, it is desirable to reduce reliability degradation due to tampering or system trouble during additional learning. Therefore, the validity of the learning data for additional learning may be detected by checking the consistency using a digital signature or hashing. Thereby, learning data for additional learning can be protected. At this time, if the validity of the learning data for additional learning cannot be detected as a result of checking the consistency using digital signatures and hashing, a warning to that effect will be issued and additional learning will be performed using that learning data. do not have. Note that the server may take any form, such as a cloud server, fog server, or edge server, regardless of its installation location.

(Modification 16)
In the various embodiments and modified examples described above, the instructions from the examiner may be not only manual instructions (for example, instructions using a user interface or the like) but also instructions by voice or the like. At this time, for example, a machine learning model including a speech recognition model (speech recognition engine, trained model for speech recognition) obtained by machine learning may be used. Further, the manual instruction may be an instruction by inputting characters using a keyboard, a touch panel, or the like. At this time, for example, a machine learning model including a character recognition model (character recognition engine, trained model for character recognition) obtained by machine learning may be used. Further, the instruction from the examiner may be an instruction using a gesture or the like. At this time, a machine learning model including a gesture recognition model (gesture recognition engine, trained model for gesture recognition) obtained by machine learning may be used.

Further, the instruction from the examiner may be the result of detecting the examiner's line of sight on the monitor. The line of sight detection result may be, for example, a pupil detection result using a moving image of the examiner taken from around the monitor. At this time, the pupil detection from the moving image may be performed using the object recognition engine as described above. Further, the instructions from the examiner may be instructions based on brain waves, weak electrical signals flowing through the body, or the like.

In such a case, for example, the learning data may be text data or audio data (waveform data) indicating instructions for displaying the results of processing the various trained models as described above, and The learning data may be the correct data that is an execution command for actually displaying the results of model processing on the display unit. In addition, as input data, for example, character data or voice data indicating instructions for displaying a high-quality image obtained by a trained model for high-quality images are used as input data, and an execution command for displaying a high-quality image and a high-quality image are used as input data. The learning data may be the correct data that is an execution command for changing the image quality button to the active state. Of course, any learning data may be used as long as the instruction content and the execution command content indicated by character data or audio data correspond to each other, for example. Furthermore, audio data may be converted into character data using an acoustic model, a language model, or the like. Furthermore, processing may be performed to reduce noise data superimposed on audio data using waveform data obtained by a plurality of microphones. Further, instructions using text or voice, etc., and instructions using a mouse, touch panel, etc. may be selectable in accordance with instructions from the examiner. Further, the device may be configured to be able to select whether to turn on or off instructions using text, voice, etc. in accordance with instructions from the examiner.

Here, machine learning includes deep learning as described above, and for example, a recurrent neural network (RNN) can be used for at least a part of the multilayer neural network. Here, as an example of a machine learning model according to this modification, an RNN, which is a neural network that handles time-series information, will be described with reference to FIGS. 15(a) and 15(b). Further, a long short-term memory (hereinafter referred to as LSTM), which is a type of RNN, will be explained with reference to FIGS. 16(a) and (b).

FIG. 15(a) shows the structure of RNN, which is a machine learning model. The RNN 3520 has a loop structure in its network, inputs data x ^t 3510 at time t, and outputs data h ^t 3530. Since the RNN 3520 has a loop function in the network, it is possible to take over the state at the current time to the next state, so it can handle time-series information. FIG. 15(b) shows an example of input/output of the parameter vector at time t. The data x ^t 3510 includes N pieces of data (Params1 to ParamsN). Furthermore, the data h ^t 3530 output from the RNN 3520 includes N pieces of data (Params1 to ParamsN) corresponding to the input data.

However, since RNN cannot handle long-term information during error backpropagation, LSTM is sometimes used. LSTM can learn long-term information by providing a forgetting gate, an input gate, and an output gate. Here, the structure of LSTM is shown in FIG. 16(a). In the LSTM 3540, the information that the network takes over at the next time t is the internal state c ^t-1 of the network called a cell and the output data h ^t-1 . Note that lowercase letters (c, h, x) in the figure represent vectors.

Next, details of the LSTM3540 are shown in FIG. 16(b). In FIG. 16(b), FG represents a forgetting gate network, IG represents an input gate network, and OG represents an output gate network, each of which is a sigmoid layer. Therefore, a vector in which each element has a value between 0 and 1 is output. The forgetting gate network FG determines how much past information to retain, and the input gate network IG determines which value to update. CU is a cell update candidate network and an activation function tanh layer. This creates a vector of new candidate values that is added to the cell. The output gate network OG selects the elements of the cell candidates and selects how much information to convey at the next time.

Note that since the LSTM model described above is a basic form, it is not limited to the network shown here. Coupling between networks may be changed. Instead of LSTM, QRNN (Quasi Recurrent Neural Network) may be used. Furthermore, the machine learning model is not limited to neural networks, and boosting, support vector machines, etc. may also be used. Furthermore, if the examiner's instructions are input by text or voice, techniques related to natural language processing (for example, sequence to sequence) may be applied. Further, a dialogue engine (a dialogue model, a trained model for dialogue) that responds to the examiner by outputting text or voice may be applied.

(Modification 17)
In the various embodiments and modifications described above, high-quality images and the like may be stored in the storage unit according to instructions from the examiner. At this time, after receiving instructions from the examiner to save high-quality images, etc., when registering the file name, select any part of the file name (for example, the first part, the last part, etc.) as the recommended file name. If the file name contains information (e.g., characters) indicating that the image was generated by processing using a trained model for high image quality (high image quality processing) in The information may be displayed in an editable state as required.

In addition, when displaying high-quality images on the display on various display screens such as report screens, the displayed image is a high-quality image generated by processing using a trained model for high image quality. An indication that there is such a thing may be displayed together with the high-quality image. In this case, the display allows the examiner to easily identify that the displayed high-quality image is not the image obtained by photography, which reduces misdiagnosis and improves diagnostic efficiency. be able to. Note that the display indicating that the image is a high-quality image generated by processing using a trained model for high-quality image is a display that allows the input image to be distinguished from the high-quality image generated by the processing. It may take any form. Furthermore, in addition to processing using trained models for high image quality, processing using various trained models such as those described above can also be performed using the results generated by processing using that type of trained model. An indication may be displayed along with the result.

At this time, the display screen such as the report screen may be saved in the storage unit according to instructions from the examiner. For example, a report screen may be stored as a single image in which high-quality images are lined up with a display indicating that these images are high-quality images generated by processing using a trained model for high-quality images. It may be stored in the section.

In addition, regarding the display indicating that the image is a high-quality image generated by processing using a trained model for high image quality, what kind of training data was used to train the trained model for high image quality? An indication indicating whether or not there is one may be displayed on the display unit. The display may include an explanation of the types of the input data and correct data of the learning data, and any display related to the correct data such as the imaged body part included in the input data and the correct data. Note that not only processing using a trained model for high image quality, but also processing using various trained models such as those mentioned above, depends on what kind of training data that type of trained model is used for learning. An indication may be displayed on the display unit to indicate whether the information is correct or not.

Additionally, information indicating that the image is generated by processing using a trained model for high image quality (for example, text) is displayed or saved in a state superimposed on the high-quality image, etc. It's okay. At this time, the location to be superimposed on the image may be any region (for example, the edge of the image) that does not overlap with the region in which the region of interest to be photographed is displayed. Alternatively, a non-overlapping area may be determined and the area may be overlapped with the determined area.

In addition, if the default setting is such that the high-quality button is activated (high-quality processing is turned on) as the initial display screen of the report screen, high-quality images will be displayed in response to instructions from the examiner. The report image corresponding to the report screen including the above may be configured to be transmitted to the server. In addition, if the default setting is such that the high image quality button is activated, it will be possible to change the image quality at the end of the examination (for example, when the imaging confirmation screen or preview screen is changed to the report screen in response to instructions from the examiner). case), the report image corresponding to the report screen including a high-quality image etc. may be configured to be (automatically) transmitted to the server. At this time, various settings in the default settings (for example, the depth range for generating the En-Face image on the initial display screen of the report screen, whether or not the analysis map is superimposed, whether or not it is a high-quality image, the display screen for follow-up observation) The report image generated based on at least one setting such as whether or not the report image is sent to the server may be configured.

(Modification 18)
In the various embodiments and modifications described above, images obtained with the first type of trained model among the various trained models described above (e.g., high-quality images, analysis maps, etc. images, images showing object recognition results, images showing segmentation results) may be input to a second type of trained model that is different from the first type. At this time, the configuration may be such that results (for example, analysis results, diagnosis results, object recognition results, and segmentation results) are generated by processing the second type of learned model.

Furthermore, among the various trained models described above, the results of the processing of the first type of trained model (for example, analysis results, diagnosis results, object recognition results, segmentation results) are used to generate the first type of trained model. An image to be input to a second type of trained model different from the first type may be generated from an image input to the trained model. At this time, the generated image is highly likely to be an image suitable for processing by the second type of trained model. For this reason, images obtained by inputting the generated images to the second type of trained model (for example, high-quality images, images showing analysis results such as analysis maps, images showing object recognition results, and segmentation results) The accuracy of images shown) can be improved.

Further, the various trained models described above may be trained models obtained by learning training data including two-dimensional medical images of the subject, or may be trained models obtained by learning training data including two-dimensional medical images of the subject. The model may be a trained model obtained by learning training data including images.

Further, a similar image search may be performed using an external database stored in a server or the like using the analysis results, diagnosis results, etc. obtained by processing the learned model as described above as a search key. In addition, in cases where multiple images stored in a database have already been managed with the feature values of each of the multiple images attached as additional information through machine learning, etc., the image itself can be used as the search key. A similar image search engine (similar image inspection model, trained model for similar image search) may be used.

(Modification 19)
Note that the motion contrast data generation process in the above embodiments and modified examples is not limited to a configuration in which it is performed based on the brightness value of a tomographic image. The various processes described above are performed on tomographic data including an interference signal acquired by the tomographic imaging apparatus 100, a signal obtained by performing Fourier transform on the interference signal, a signal obtained by performing arbitrary processing on the signal, and a tomographic image based on these. may be applied to In these cases as well, the same effects as the above configuration can be achieved.

Although a fiber optical system using a coupler is used as the dividing means, a spatial optical system using a collimator and a beam splitter may also be used. Further, the configuration of the tomographic image capturing apparatus 100 is not limited to the above configuration, and a part of the configuration included in the tomographic image capturing apparatus 100 may be configured separately from the tomographic image capturing apparatus 100.

Furthermore, in the above embodiments and modifications, a Michelson interferometer configuration is used as the interference optical system of the tomographic imaging apparatus 100, but the configuration of the interference optical system is not limited to this. For example, the interference optical system of the tomographic imaging apparatus 100 may have a Mach-Zehnder interferometer configuration.

Further, in the above embodiments and modifications, a spectral domain OCT (SD-OCT) device using an SLD as a light source has been described as an OCT device, but the configuration of the OCT device according to the present invention is not limited to this. For example, the present invention can be applied to any other type of OCT device, such as a wavelength swept OCT (SS-OCT) device that uses a wavelength swept light source that can sweep the wavelength of emitted light. Further, the present invention can also be applied to a Line-OCT device using line light.

Furthermore, in the above embodiment and modification, the image processing unit 101-04 acquired the interference signal acquired by the tomographic imaging apparatus 100, the three-dimensional tomographic image generated by the image processing unit, and the like. However, the configuration in which the image processing unit 101-04 acquires these signals and images is not limited to this. For example, the image processing unit 101-04 may acquire these signals from a server or imaging device connected via a LAN, WAN, or the Internet.

Note that the trained model can be provided in the image processing unit 101-04. The trained model can be configured of, for example, a software module executed by a processor such as a CPU. Further, the trained model may be provided in another server or the like connected to the image processing unit 101-04. In this case, the image processing unit 101-04 can perform high image quality processing using the trained model by connecting to a server provided with the trained model via any network such as the Internet.

(Modification 20)
Furthermore, images processed by the image processing apparatus or image processing method according to the various embodiments and modifications described above include medical images acquired using any modality (imaging device, imaging method). The medical images to be processed can include medical images acquired by any imaging device or the like, and images created by the image processing apparatus or image processing method according to the embodiments and modifications described above.

Furthermore, the medical image to be processed is an image of a predetermined region of a subject (subject), and the image of the predetermined region includes at least a part of the predetermined region of the subject. Further, the medical image may include other parts of the subject. Further, the medical image may be a still image or a moving image, and may be a black and white image or a color image. Furthermore, the medical image may be an image representing the structure (morphology) of a predetermined region, or an image representing its function. Images representing functions include, for example, images representing blood flow dynamics (blood flow volume, blood flow velocity, etc.) such as OCTA images, Doppler OCT images, fMRI images, and ultrasound Doppler images. The predetermined parts of the subject may be determined depending on the object to be photographed, and may include human eyes (eyes to be examined), organs such as the brain, lungs, intestines, heart, pancreas, kidneys, and liver, head, chest, Includes arbitrary parts such as legs and arms.

Furthermore, the medical image may be a tomographic image of the subject or a frontal image. The frontal image is, for example, a frontal image of the fundus, a frontal image of the anterior segment of the eye, a fluorescently photographed fundus image, or at least a partial range in the depth direction of the object to be photographed regarding data acquired by OCT (three-dimensional OCT data). Contains an En-Face image generated using the data. The En-Face image is an OCTA En-Face image (motion contrast frontal image) generated using data of at least a partial range in the depth direction of the imaging target for three-dimensional OCTA data (three-dimensional motion contrast data). ) is also fine. Furthermore, three-dimensional OCT data and three-dimensional motion contrast data are examples of three-dimensional medical image data.

Further, the photographing device is a device for photographing images used for diagnosis. The imaging device is, for example, a device that obtains an image of a predetermined region of a subject by irradiating the subject with radiation such as light, X-rays, electromagnetic waves, or ultrasound, or a device that detects radiation emitted from the subject. The apparatus includes a device for obtaining an image of a predetermined region. More specifically, the imaging devices according to the various embodiments and modifications described above include at least an X-ray imaging device, a CT device, an MRI device, a PET device, a SPECT device, an SLO device, an OCT device, an OCTA device, and a fundus. Includes cameras, endoscopes, etc.

Note that the OCT device may include a time domain OCT (TD-OCT) device and a Fourier domain OCT (FD-OCT) device. Further, the Fourier domain OCT device may include a spectral domain OCT (SD-OCT) device or a wavelength swept OCT (SS-OCT) device. Furthermore, the SLO device and OCT device may include a wavefront compensated SLO (AO-SLO) device using a wavefront adaptive optical system, a wavefront compensated OCT (AO-OCT) device, and the like. Furthermore, the SLO device and OCT device may include a polarization SLO (PS-SLO) device, a polarization OCT (PS-OCT) device, etc. for visualizing information regarding polarization phase difference and depolarization.

[Other embodiments]
Various embodiments and modifications have been described above in detail, but the disclosed technology can be implemented as, for example, a system, an apparatus, a method, a program, a recording medium (storage medium), or the like. Specifically, it may be applied to a system consisting of multiple devices (for example, a host computer, an interface device, an imaging device, a web application, etc.), or it may be applied to a device consisting of a single device. good.

Moreover, it goes without saying that the object of the present invention is achieved by the following steps. That is, a recording medium (or storage medium) recording software program code (computer program) that implements the functions of the various embodiments and modifications described above is supplied to the system or device. Such a storage medium is, of course, a computer-readable storage medium. Then, the computer (or CPU or MPU) of the system or device reads and executes the program code stored in the recording medium. In this case, the program code read from the recording medium itself realizes the functions of the various embodiments and modifications described above, and the recording medium on which the program code is recorded constitutes the present invention.

Claims (16)

a high image quality means for increasing the image quality of three-dimensional motion contrast data of the fundus;
acquisition means for acquiring information regarding a three-dimensional blood vessel region in at least a portion of the high-quality three-dimensional motion contrast data;
A blood vessel measurement value in a three-dimensional space is calculated using the information regarding the three-dimensional blood vessel region , and a blood vessel measurement value in the two-dimensional region is calculated from two-dimensional motion contrast data obtained by projecting the three-dimensional motion contrast data in the depth direction. Calculation means for calculating blood vessel measurement values;
means for associating the calculated blood vessel measurement value with identification information that can identify whether it is a blood vessel measurement value in the three-dimensional space or a blood vessel measurement value in the two-dimensional area;
An information processing device comprising: 2. The information processing apparatus according to claim 1, wherein the image quality improving means improves the image quality of the three-dimensional motion contrast data by composing a plurality of the three-dimensional motion contrast data. The image quality improving means inputs the three-dimensional motion contrast data of the fundus into a trained model for high image quality obtained by learning training data including three-dimensional motion contrast data of the fundus, thereby improving the high image quality. The information processing device according to claim 1, which outputs converted three-dimensional motion contrast data. Further comprising processing means for performing processing to reduce projection artifacts on the high-quality three-dimensional motion contrast data,
The calculation means calculates the blood vessel measurement value using information regarding a three-dimensional blood vessel region in at least a portion of the high-quality three-dimensional motion contrast data after projection artifacts have been reduced by the processing means. The information processing device according to any one of claims 1 to 3. processing means for performing processing to reduce projection artifacts on three-dimensional motion contrast data of the fundus;
acquisition means for acquiring information regarding a three-dimensional blood vessel region in at least a portion of the three-dimensional motion contrast data after projection artifacts have been reduced by the processing means;
A blood vessel measurement value in a three-dimensional space is calculated using the information regarding the three-dimensional blood vessel region , and a blood vessel measurement value in the two-dimensional region is calculated from two-dimensional motion contrast data obtained by projecting the three-dimensional motion contrast data in the depth direction. Calculation means for calculating blood vessel measurement values;
means for associating the calculated blood vessel measurement value with identification information that can identify whether it is a blood vessel measurement value in the three-dimensional space or a blood vessel measurement value in the two-dimensional area;
An information processing device comprising: The blood vessel measurement value is a blood vessel measurement value in the three -dimensional space, and includes at least one of the volume of the blood vessel, the three-dimensional blood vessel density, and the diameter, length, and curvature of the blood vessel measured in the three-dimensional space. at least one of the following:
A blood vessel measurement value in the two-dimensional region, which is a measurement value based on at least one of the area of a blood vessel region or an avascular region, the two-dimensional blood vessel density, and the diameter, length, and curvature of the blood vessel measured in the two -dimensional region. The information processing device according to any one of claims 1 to 5, which is at least one of the following. The information processing apparatus according to claim 6, wherein the three-dimensional space is specified by a user. further comprising display control means for displaying the calculated blood vessel measurement value on a display unit,
When displaying a plurality of the calculated blood vessel measurement values in chronological order, the display control means controls, based on the identification information, blood vessel measurement values in a three-dimensional space and blood vessel measurement values in a two-dimensional area do not coexist. The information processing apparatus according to any one of claims 1 to 7 , wherein the information processing apparatus displays on the display unit. A blood vessel measurement value in a three-dimensional space is obtained using information regarding a three-dimensional blood vessel region in at least a part of the three-dimensional motion contrast data of the fundus, and the three-dimensional motion contrast data is obtained by projecting the three-dimensional motion contrast data in the depth direction. acquisition means for acquiring blood vessel measurement values in a two-dimensional region using the obtained two-dimensional motion contrast data;
means for associating the blood vessel measurement value obtained by the acquisition means with identification information that can identify whether the blood vessel measurement value is in the three-dimensional space or the blood vessel measurement value in the two- dimensional area;
An information processing device comprising : further comprising a display control means for displaying the blood vessel measurement value obtained by the acquisition means on a display unit,
When displaying a plurality of blood vessel measurement values obtained by the acquisition means in chronological order, the display control means may display blood vessel measurement values in the three-dimensional space and blood vessel measurements in the two-dimensional area based on the identification information. The information processing apparatus according to claim 9, wherein the information processing apparatus is configured to display the values on the display unit so that the values are not mixed. The acquisition means is a trained model for object recognition obtained by learning learning data including three-dimensional motion contrast data of the fundus, or a trained model obtained by learning training data including three-dimensional motion contrast data of the fundus. The information processing apparatus according to any one of claims 1 to 10 , wherein information regarding the three-dimensional blood vessel region is acquired by inputting three-dimensional motion contrast data of the fundus into a trained model for segmentation. By inputting the three-dimensional motion contrast data of the fundus into a trained model for generating analysis results obtained by learning learning data including three-dimensional motion contrast data of the fundus, the analysis results regarding the three-dimensional blood vessel region can be generated. By inputting the three-dimensional motion contrast data of the fundus into a trained model for generating diagnostic results obtained by learning learning data including three-dimensional motion contrast data of the fundus, the three-dimensional blood vessel region The information processing apparatus according to any one of claims 1 to 11 , further comprising a generating means for generating a diagnostic result regarding. A high-quality image improvement process that improves the image quality of three-dimensional motion contrast data of the fundus;
an acquisition step of acquiring information regarding a three-dimensional blood vessel region in at least a portion of the high-quality three-dimensional motion contrast data;
A blood vessel measurement value in a three-dimensional space is calculated using the information regarding the three-dimensional blood vessel region , and a blood vessel measurement value in the two-dimensional region is calculated from two-dimensional motion contrast data obtained by projecting the three-dimensional motion contrast data in the depth direction. a calculation step of calculating blood vessel measurement values;
a step of associating the calculated blood vessel measurement value with identification information that can identify whether it is a blood vessel measurement value in the three-dimensional space or a blood vessel measurement value in the two-dimensional area;
information processing methods, including a processing step of performing processing to reduce projection artifacts on three-dimensional motion contrast data of the fundus;
an acquisition step of acquiring information regarding a three-dimensional blood vessel region in at least a portion of the three-dimensional motion contrast data after projection artifacts have been reduced in the processing step ;
A blood vessel measurement value in a three-dimensional space is calculated using the information regarding the three-dimensional blood vessel region , and a blood vessel measurement value in the two-dimensional region is calculated from two-dimensional motion contrast data obtained by projecting the three-dimensional motion contrast data in the depth direction. a calculation step of calculating blood vessel measurement values;
a step of associating the calculated blood vessel measurement value with identification information that can identify whether it is a blood vessel measurement value in the three-dimensional space or a blood vessel measurement value in the two-dimensional area;
information processing methods, including A blood vessel measurement value in a three-dimensional space is obtained using information regarding a three-dimensional blood vessel region in at least a part of the three-dimensional motion contrast data of the fundus, and the three-dimensional motion contrast data is obtained by projecting the three-dimensional motion contrast data in the depth direction. an acquisition step of acquiring blood vessel measurement values in the two-dimensional region using the obtained two-dimensional motion contrast data;
a step of associating the blood vessel measurement value obtained in the acquisition step with identification information that can identify whether it is a blood vessel measurement value in the three-dimensional space or a blood vessel measurement value in the two-dimensional region;
information processing methods, including A program that causes a computer to execute the information processing method according to any one of claims 13 to 15 .

Images