JP7446730B2 - Image processing device, image processing method and program - Google Patents

Description

The present invention relates to an image processing device, an image processing method, and a program.

When a tomographic imaging device such as an optical coherence tomography (OCT) is used, the internal state of the eye to be examined can be observed three-dimensionally. This tomographic imaging apparatus is widely used in ophthalmological treatment because it is useful for more accurately diagnosing diseases. One form of OCT is, for example, TD-OCT (Time domain OCT), which combines a broadband light source and a Michelson interferometer. This is configured to move the position of the reference mirror at a constant speed and measure the interference light with the backscattered light acquired by the signal arm to obtain the reflected light intensity distribution in the depth direction. However, such TD-OCT requires mechanical scanning, making it difficult to obtain images at high speed. Therefore, as faster image acquisition methods, SD-OCT (Spectral domain OCT) uses a broadband light source and acquires interference signals with a spectrometer, and SS-OCT (Swept Source OCT) has been developed, making it possible to obtain tomographic images with a wider field of view.

In order to understand the pathology related to the blood vessels of the subject's eye, an OCT angiography (hereinafter referred to as OCTA) technique is used to non-invasively visualize the fundus blood vessels in three dimensions using OCT. In OCTA, the same position is scanned multiple times with measurement light, and the motion contrast obtained by the interaction between the displacement of red blood cells and the measurement light is imaged. Figure 4(a) shows OCTA imaging in which the fast axis direction is horizontal (x axis) and B scans are performed r times consecutively at each position (yi; 1≦i≦n) in the slow axis direction (y axis direction). An example is shown. Note that in OCTA imaging, scanning the same position multiple times is called cluster scanning, and a plurality of tomographic images obtained at the same position is called a cluster, and a motion contrast image is generated in cluster units.

Patent Document 1 discloses that in an OCTA image (an en-face image in which a blood vessel region is emphasized), a luminance value is A tomographic image is obtained based on the ratio or difference between a one-dimensional brightness profile along the Y direction (slow axis direction) and a smoothed one-dimensional brightness profile obtained by smoothing the one-dimensional brightness profile. A method for correcting brightness is disclosed. That is, Patent Document 1 discloses a technique for suppressing band-like artifacts that exist in the entire region along the fast axis direction in an OCTA image.

JP 2018-15189 Publication

However, conventional methods cannot reduce band-like artifacts that partially exist along the fast axis direction. Furthermore, there is a possibility that the brightness values of blood vessel regions and the like existing along the fast axis direction may be overcorrected or erroneously suppressed.

The present invention has been made in view of the above problems, and one of its objects is to reduce artifacts in images of an eye to be examined.

In addition, other objectives of the present invention are not limited to the above-mentioned objectives, but also include the effects derived from each configuration shown in the detailed description of the invention described later, which cannot be obtained by conventional techniques. It can be positioned as one.

In order to achieve the object of the present invention, one of the image processing devices of the present invention, for example,
an acquisition means for acquiring a plurality of frontal images corresponding to different depth ranges of the eye to be examined;
Generating means for generating a plurality of frontal images in which artifacts in the plurality of acquired frontal images are reduced;
a determination unit that uses a learned model obtained by learning a plurality of frontal images to determine a state of an artifact in at least one frontal image among the plurality of acquired frontal images;
Displaying at least one front image of the plurality of front images acquired and a determination result regarding the state of the artifact on a shooting confirmation screen , and accepting an instruction from an operator as a designation regarding whether or not reduction of the artifact is necessary. , the at least one frontal image displayed on the photographing confirmation screen is switched to at least one frontal image of the plurality of generated frontal images, and the display screen on the display means changes from the photographing confirmation screen to the report . A display control means is provided for controlling the display means so as to display the plurality of generated frontal images side by side on the report screen after the screen is switched.

According to one aspect of the present invention, artifacts in an image of an eye to be examined can be reduced.

FIG. 1 is a block diagram showing the configuration of an image processing device according to a first embodiment of the present invention. 1 is a diagram illustrating a measurement optical system included in an image processing system according to an embodiment of the present invention and a tomographic imaging apparatus that constitutes the image processing system. 3 is a flowchart of processing that can be executed by the image processing system according to the first embodiment of the present invention. FIG. 2 is a diagram illustrating a scanning method for OCTA imaging and brightness step artifacts that occur on an OCT tomographic image and an OCTA image in an embodiment of the present invention. It is a figure explaining the processing performed in S303 and S304 of the first embodiment of this invention. It is a figure explaining the process performed in S303 of the first embodiment of this invention. FIG. 3 is a diagram illustrating the content of image processing executed in the first embodiment of the present invention. It is a figure explaining the effect of the image processing performed in the first embodiment of the present invention. FIG. 2 is a block diagram showing the configuration of an image processing device according to a second embodiment of the present invention. It is a flowchart of processing that can be executed by the image processing system according to the second embodiment of the present invention. It is a figure explaining the image processing content performed in the second embodiment of this invention. It is a figure explaining the effect of the image processing performed in the second embodiment of the present invention. It is a figure explaining the report screen displayed on a display means in S1007 of the second embodiment of this invention. It is a figure explaining the confirmation screen in the third embodiment of this invention. It is a figure explaining the front motion contrast image displayed on the confirmation screen in the third embodiment of the present invention. It is a figure explaining the process performed in S1003 of the fourth embodiment of this invention. It is a figure explaining the content of image processing performed in the fourth embodiment of the present invention. It is a figure explaining the effect of the image processing performed in the fourth embodiment of this invention.

[First embodiment]
The image processing apparatus according to the present embodiment performs the following image correction processing in order to robustly suppress various luminance step artifacts that occur in the slow axis direction of a tomographic image of a subject's eye captured using OCT. That is, distribution information of blood vessel candidate regions is generated based on the brightness attenuation rate between the retinal surface layer and the retinal outer layer of the tomographic image. Next, by dividing the brightness value of the high-dimensional smoothed tomographic image by the brightness value of the low-dimensional (fast-axis direction only) smoothed tomographic image weighted for the blood vessel candidate region, the brightness correction coefficient value is calculated. Generate a distribution. Furthermore, a case will be described in which brightness step artifacts occurring in the slow axis direction of a tomographic image are robustly suppressed by multiplying each pixel of the tomographic image by a brightness correction coefficient value. Here, the luminance step artifact that occurs in the slow axis direction is, for example, a band-shaped artifact that extends in the X direction (fast axis direction) due to fixation shift. Note that the fast axis direction is, for example, the axial direction of the main scanning of the measurement light used when acquiring a three-dimensional tomographic image.

Furthermore, an example of a brightness level difference that occurs in the slow axis direction of a tomographic image of a subject's eye, which is a correction target in this embodiment, will be described. If a fixation shift of the subject's eye occurs during imaging of an OCT tomographic image, rescanning is performed. For example, during long-term imaging, the positions of the eyelashes and pupils of the subject's eye may differ between the first scan and the rescan, resulting in lower brightness in the rescanned area, as shown by the white arrow in Figure 4(b). A band-like low-luminance level difference as shown in is likely to occur. Note that in FIG. 4(b), the horizontal direction is the fast axis direction, and the vertical direction is the slow axis direction. In addition, the band-shaped brightness step artifact does not necessarily occur over the entire area of the image in the fast-axis direction, but in the following cases, a band-like brightness step artifact localized to a part of the fast-axis direction occurs. In other words, if an area containing vitreous opacity is scanned during rescanning and a shadow area occurs, a band-like low-intensity step localized in a part of the fast axis direction may occur within the rescanned area on the tomographic image. (low-luminance step indicated by the white arrow in FIG. 4(e)).

DESCRIPTION OF THE PREFERRED EMBODIMENTS An image processing system including an image processing apparatus according to a first embodiment of the present invention will be described below with reference to the drawings.

FIG. 2 is a diagram showing the configuration of an image processing system 10 including an image processing apparatus 101 according to this embodiment. As shown in FIG. 2, in the image processing system 10, an image processing device 101 is connected to a tomographic imaging device 100 (also referred to as OCT), an external storage section 102, an input section 103, and a display section 104 via an interface. It is composed of:

The tomographic image capturing apparatus 100 is an apparatus that captures tomographic images of an eye to be examined. In this embodiment, SD-OCT is used as the tomographic imaging apparatus 100. The configuration is not limited to this, and may be configured using SS-OCT, for example.

In FIG. 2(a), a measurement optical system 100-1 is an optical system for acquiring an anterior segment image, an SLO fundus image of the eye to be examined, and a tomographic image. The stage section 100-2 allows the measurement optical system 100-1 to move back and forth and left and right. The base portion 100-3 incorporates a spectrometer to be described later.

The image processing device 101 is a computer that controls the stage section 100-2, controls alignment operations, reconstructs tomographic images, and the like. The external storage unit 102 stores tomographic imaging programs, patient information, imaging data, image data of past examinations, measurement data, and the like.

The input unit 103 gives instructions to the computer, and specifically includes a keyboard and a mouse. The display unit 104 includes, for example, a monitor.

(Configuration of tomographic imaging device)
The configuration of the measurement optical system and spectrometer in the tomographic imaging apparatus 100 of this embodiment will be explained using FIG. 2(b).

First, the inside of the measurement optical system 100-1 will be explained. An objective lens 201 is placed facing the eye 200 to be examined, and a first dichroic mirror 202 and a second dichroic mirror 203 are placed on its optical axis. These dichroic mirrors branch the optical path into an optical path 250 for the OCT optical system, an optical path 251 for the SLO optical system and fixation lamp, and an optical path 252 for anterior eye observation for each wavelength band.

The optical path 251 for the SLO optical system and fixation lamp includes an SLO scanning means 204, lenses 205 and 206, a mirror 207, a third dichroic mirror 208, an APD (Avalanche Photodiode) 209, an SLO light source 210, and a fixation lamp 211. ing.

The mirror 207 is a prism in which a perforated mirror or a hollow mirror is vapor-deposited, and separates the illumination light from the SLO light source 210 and the return light from the eye to be examined. The third dichroic mirror 208 separates the optical path of the SLO light source 210 and the optical path of the fixation lamp 211 for each wavelength band.

The SLO scanning means 204 scans the eye 200 with light emitted from the SLO light source 210, and is composed of an X scanner that scans in the X direction and a Y scanner that scans in the Y direction. In this embodiment, since it is necessary to perform high-speed scanning, the X scanner is configured with a polygon mirror, and the Y scanner is configured with a galvanometer mirror.

The lens 205 is driven by a motor (not shown) to focus the SLO optical system and the fixation lamp 211. SLO light source 210 generates light at a wavelength around 780 nm. APD 209 detects return light from the eye to be examined. The fixation lamp 211 generates visible light to encourage the subject to fixate.

The light emitted from the SLO light source 210 is reflected by the third dichroic mirror 208, passes through the mirror 207, passes through the lenses 206 and 205, and is scanned onto the eye 200 by the SLO scanning means 204. After the return light from the eye 200 to be examined returns along the same path as the illumination light, it is reflected by the mirror 207 and guided to the APD 209, whereby an SLO fundus image is obtained.

The light emitted from the fixation lamp 211 passes through the third dichroic mirror 208 and the mirror 207, passes through the lenses 206 and 205, and forms a predetermined shape at an arbitrary position on the eye 200 by the SLO scanning means 204. Encourage the subject to fixate.

Lenses 212 and 213, a split prism 214, and a CCD 215 for anterior eye observation that detects infrared light are arranged in the optical path 252 for anterior eye observation. This CCD 215 is sensitive to the wavelength of irradiation light for anterior ocular segment observation (not shown), specifically around 970 nm. The split prism 214 is arranged at a position conjugate with the pupil of the eye to be examined 200, and measures the distance in the Z-axis direction (optical axis direction) of the measurement optical system 100-1 to the eye to be examined 200 as a split image of the anterior segment of the eye. Can be detected.

The optical path 250 of the OCT optical system constitutes the OCT optical system as described above, and is for capturing a tomographic image of the eye 200 to be examined. More specifically, the purpose is to obtain an interference signal for forming a tomographic image. The XY scanner 216 is for scanning light on the eye 200 to be examined, and although it is shown as a single mirror in FIG. 2(b), it is actually a galvanometer mirror that scans in two directions of the X and Y axes. In this embodiment, the X direction (fast axis direction) is the direction in which the X scanner scans the measurement light on the fundus of the eye to be examined, and the Y direction (slow axis direction) is the direction in which the Y scanner scans the measurement light on the fundus of the eye to be examined. This is the direction in which the measurement light is scanned over the fundus. However, this is not the case when the scanning is not a raster scan as in this embodiment (for example, a circle scan or a radial scan).

Of the lenses 217 and 218, the lens 217 is driven by a motor (not shown) in order to focus the light from the OCT light source 220 emitted from the fiber 224 connected to the optical coupler 219 onto the eye 200 to be examined. By this focusing, the return light from the eye 200 to be examined is simultaneously formed into a spot image and incident on the tip of the fiber 224. Next, the configuration of the optical path from the OCT light source 220, the reference optical system, and the spectrometer will be explained. 220 is an OCT light source, 221 is a reference mirror, 222 is a dispersion compensating glass, 223 is a lens, 219 is an optical coupler, 224 to 227 are single mode optical fibers connected and integrated with the optical coupler, and 230 is a spectrometer. .

These configurations constitute a Michelson interferometer. The light emitted from the OCT light source 220 passes through the optical fiber 225 and is split via the optical coupler 219 into measurement light on the optical fiber 224 side and reference light on the optical fiber 226 side. The measurement light is irradiated onto the eye 200 to be observed, which is the object of observation, through the optical path of the aforementioned OCT optical system, and reaches the optical coupler 219 through the same optical path due to reflection and scattering by the eye 200 to be examined.

On the other hand, the reference light reaches the reference mirror 221 and is reflected through the optical fiber 226, the lens 223, and the dispersion compensating glass 222 inserted to match the wavelength dispersion of the measurement light and the reference light. Then, it returns along the same optical path and reaches the optical coupler 219.

The measurement light and the reference light are combined by the optical coupler 219 to become interference light.

Here, interference occurs when the optical path length of the measurement light and the optical path length of the reference light become approximately the same. The reference mirror 221 is held adjustable in the optical axis direction by a motor and a drive mechanism (not shown), and can match the optical path length of the reference light to the optical path length of the measurement light. The interference light is guided to a spectrometer 230 via an optical fiber 227.

Further, polarization adjustment units 228 and 229 are provided in the optical fibers 224 and 226, respectively, and perform polarization adjustment. These polarization adjustment sections have several sections in which optical fibers are routed in loops. By rotating this loop-shaped portion around the longitudinal direction of the fiber, it is possible to twist the fiber and adjust the polarization states of the measurement light and the reference light to match each other.

The spectrometer 230 includes lenses 232 and 234, a diffraction grating 233, and a line sensor 231. The interference light emitted from the optical fiber 227 becomes parallel light through the lens 234 , is separated by the diffraction grating 233 , and is imaged onto the line sensor 231 by the lens 232 .

Next, the surroundings of the OCT light source 220 will be described. The OCT light source 220 is an SLD (Super Luminescent Diode), which is a typical low coherent light source. The center wavelength is 855 nm, and the wavelength bandwidth is approximately 100 nm. Here, the bandwidth is an important parameter because it affects the resolution of the obtained tomographic image in the optical axis direction.

As for the type of light source, SLD is selected here, but it is sufficient that it can emit low coherent light, and ASE (Amplified Spontaneous Emission) or the like may be used. Considering that the eye is to be measured, near-infrared light is suitable as the center wavelength. Furthermore, since the center wavelength affects the lateral resolution of the obtained tomographic image, it is desirable that the center wavelength be as short as possible. For both reasons, the center wavelength was set to 855 nm.

In this embodiment, a Michelson interferometer is used as the interferometer, but a Mach-Zehnder interferometer may also be used. Depending on the difference in light intensity between the measurement light and the reference light, it is desirable to use a Mach-Zehnder interferometer when the difference in light intensity is large, and to use a Michelson interferometer when the difference in light intensity is relatively small.

(Configuration of image processing device)
The configuration of the image processing apparatus 101 of this embodiment will be explained using FIG. 1.

The image processing device 101 is a personal computer (PC) connected to the tomographic imaging device 100, and includes an image acquisition section 101-01, a storage section 101-02, an imaging control section 101-03, an image processing section 101-04, and a display. It includes a control section 101-05. The image processing device 101 also functions as the arithmetic processing unit CPU executes software modules that implement an image acquisition unit 101-01, a shooting control unit 101-03, an image processing unit 101-04, and a display control unit 101-05. Realize. The present invention is not limited to this, and for example, the image processing unit 101-04 may be implemented using dedicated hardware such as an ASIC, and the display control unit 101-05 may be implemented using a dedicated processor such as a GPU that is different from the CPU. It may be realized by Further, the connection between the tomographic imaging apparatus 100 and the image processing apparatus 101 may be configured via a network.

The image acquisition unit 101-01 acquires signal data of the SLO fundus image and tomographic image photographed by the tomographic imaging apparatus 100. The image acquisition unit 101-01 also includes a tomographic image generation unit 101-11. The tomographic image generation unit 101-11 acquires the signal data (interference signal) of the tomographic image taken by the tomographic imaging apparatus 100, generates a tomographic image by signal processing, and stores the generated tomographic image in the storage unit 101-02. Store.

The imaging control unit 101-03 performs imaging control for the tomographic imaging apparatus 100. Imaging control also includes instructing the tomographic imaging apparatus 100 regarding the setting of imaging parameters and instructing the start or end of imaging.

The image processing section 101-04 includes a positioning section 101-41, an image feature acquisition section 101-42, a projection section 101-43, and a correction section 101-44. The image acquisition unit 101-01 described above is an example of the first acquisition means according to the present invention. The image feature acquisition unit 101-42 acquires the layer boundaries of the retina and choroid, blood vessel candidate regions, and the positions of the fovea and the center of the optic disc from the tomographic image. The projection unit 101-43 projects an image in a depth range based on the position of the layer boundary acquired by the image feature acquisition unit 101-42, and generates a front image. The correction unit 101-44 includes a conversion unit 101-441, a weighting unit 101-442, and a calculation unit 101-443. The correction unit 101-44 uses a brightness correction coefficient calculated by arithmetic processing of a high-dimensional smoothed tomographic image and a low-dimensional smoothed image obtained by smoothing a tomographic image in which the brightness of the blood vessel candidate region is weighted in the fast axis direction. Processing is performed to suppress the brightness level difference that occurs in the slow axis direction of the tomographic image. The conversion unit 101-441 includes a high-dimensional conversion unit 101-4411 that generates a high-dimensional approximate luminance value distribution, and a low-dimensional conversion unit 101-4412 that generates a low-dimensional approximate luminance value distribution. The weighting unit 101-442 weights the brightness value of the tomographic image based on the distribution information of the blood vessel candidate region acquired by the blood vessel acquisition unit 101-421. The calculation unit 101-443 calculates the brightness correction coefficient value distribution by calculating the high-dimensional smoothed image generated by the high-dimensional conversion unit 101-4411 and the low-dimensional smoothed image generated by the low-dimensional conversion unit 101-4412. Calculate.

The external storage unit 102 stores information about the eye to be examined (patient's name, age, gender, etc.), photographed tomographic images and SLO images, photographing parameters, images obtained by processing the images, and distribution data of blood vessel candidate regions. , brightness correction coefficient value distribution, and parameters set by the operator are stored in association with each other. The input unit 103 is, for example, a mouse, a keyboard, a touch operation screen, etc., and the operator issues instructions to the image processing apparatus 101 and the tomographic imaging apparatus 100 via the input unit 103.

Next, the processing procedure of the image processing apparatus 101 of this embodiment will be described with reference to FIG. FIG. 3 is a flowchart showing the flow of operation processing of the entire system in this embodiment.

<Step 301>
By operating the input unit 103, the operator sets imaging conditions for an OCT image (three-dimensional tomographic image) to be instructed to the tomographic imaging apparatus 100.

Specifically, the procedure includes 1) selection of a scan mode, and 2) setting of imaging parameters corresponding to the scan mode. In this embodiment, OCT imaging is executed with the following settings.
1) Select Macula 3D scan mode 2) Set the following shooting parameters 2-1) Scan area size: 10x10mm
2-2) Main scanning direction: horizontal direction 2-3) Scanning interval: 0.01mm
2-4) Fixation lamp position: midway between the fovea and optic disc 2-5) Number of B scans at the same imaging position: 1
2-6) Coherence gate position: Vitreous body side Next, the operator operates the input unit 103 and presses the shooting start button (hidden) on the shooting screen to obtain an OCT tomographic image under the shooting conditions set above. Start shooting.

The imaging control unit 101-03 instructs the tomographic imaging apparatus 100 to perform OCT imaging based on the above settings, and the tomographic imaging apparatus 100 acquires a corresponding OCT tomographic image.

The tomographic imaging apparatus 100 also acquires SLO images and executes tracking processing based on the SLO moving images. Note that in this embodiment, the number of times the tomographic image is captured at the same scanning position is once (not repeated). The number of times of imaging at the same scanning position is not limited to this, and may be set to an arbitrary number of times.

<Step 302>
The image acquisition unit 101-01 and the image processing unit 101-04 reconstruct the tomographic image acquired in S301.

First, the tomographic image generation unit 101-11 generates a tomographic image by performing wave number transformation, fast Fourier transformation (FFT), and absolute value conversion (amplitude acquisition) on the interference signal acquired by the image acquisition unit 101-01. . Next, the positioning unit 101-41 performs positioning between the B-scan tomographic images.

Furthermore, the image feature acquisition unit 101-42 acquires the layer boundaries of the retina and choroid, and the front and back boundaries (not shown) of the lamina cribrosa from the tomographic image. In this embodiment, as shown in FIG. 6(a), the layer boundaries include an internal limiting membrane 1, a nerve fiber layer-ganglion cell layer boundary 2, a ganglion cell layer-inner plexiform layer boundary 3, and an inner photoreceptor cell outer segment. The junction 4, retinal pigment epithelium 5, Bruch's membrane 6, and choroid-sclera border 7 are obtained. Furthermore, the detected end of Bruch's membrane 6 (opening end of Bruch's membrane) is specified as the Disc boundary of the optic disc. In this embodiment, a variable shape model is used as a method for obtaining the layer boundaries of the retina and choroid and the anterior and posterior boundaries of the lamina cribrosa, but any known segmentation method may be used. Furthermore, the layer boundaries to be acquired are not limited to those described above. For example, the inner plexiform layer-inner nuclear layer boundary, the inner nuclear layer-outer nuclear layer boundary, the outer plexiform layer-outer nuclear layer boundary, the outer limiting membrane, and the photoreceptor outer segment tip (COST) of the retina can be determined using any known segmentation method. You may obtain it. Alternatively, the present invention also includes the case where the choroidal capillary plate-Sattler layer boundary and the Sattler layer-Haller layer boundary of the choroid are obtained by any known segmentation method. Further, the front and rear boundaries of the lamina cribriform portion may be set manually. For example, it can be set manually by moving the position of a particular layer boundary (eg, internal limiting membrane 1) by a predetermined amount.

Note that the process for acquiring the layer boundaries and the anterior and posterior boundaries of the lamina cribrosa may be performed not in this step but at the time of acquiring the blood vessel candidate region in S303.

<Step 303>
The blood vessel acquisition unit 101-421 generates information regarding the distribution of blood vessel candidate regions based on the results of comparing brightness statistics between different predetermined depth ranges.

As shown in 601 in FIG. 6(a), the blood vessel region has a roughly fixed depth range (layer type), and is characterized in that a shadow 602 is likely to occur under the region. On the other hand, in the case of brightness step artifacts or vitreous opacity that occur in the rescanning region of a tomographic image, the brightness tends to be low over most of the depth range, as shown at 603 in FIG. 6(a).

Therefore, in this embodiment, the degree of difference (difference or ratio) in brightness between "the depth range where blood vessels are likely to exist (surface layer of the retina)" and "the depth range where the decrease in brightness due to shadows is most noticeable (outer layer of the retina)" A blood vessel candidate region is identified based on.

Details of the blood vessel candidate region map generation process will be explained in S501 to S506. Note that the map is an example of distribution information in an in-plane direction intersecting the depth direction of the eye to be examined.

<Step 304>
The weighting unit 101-442 generates a weighted tomographic image by weighting the brightness values in the blood vessel candidate area of the tomographic image using the information regarding the distribution of the blood vessel candidate area generated by the blood vessel acquisition unit 101-421 in S303. Next, the high-dimensional transformation unit 101-4411 generates a high-dimensional smoothed tomographic image, and the low-dimensional transformation unit 101-4412 performs a smoothing process on the weighted tomographic image in the fast axis direction. generate a tomographic image. Further, the calculation unit 101-443 generates a brightness correction coefficient map for the tomographic image by performing calculation processing on the high-dimensional smoothed tomographic image and the low-dimensional smoothed tomographic image.

Details of the brightness correction coefficient map generation process will be explained in S511 to S516.

<Step 305>
The correction unit 101-44 multiplies each pixel of the tomographic image by the brightness correction coefficient value calculated in S304 to generate a tomographic image with brightness level difference corrected. Note that the method of applying the brightness correction coefficient is not limited to multiplication, and any known calculation method may be applied. For example, addition, subtraction, or division may be applied. Further, at least a portion of the three-dimensional tomographic image may be corrected using the brightness correction coefficient value. At this time, at least a portion of the three-dimensional tomographic image includes a C-scan image and the like.

<Step 306>
The display control unit 101-05 displays the brightness-corrected tomographic image generated in S305 on the display unit 104. In addition, when the operator selects a button (not shown) or a shortcut menu displayed on the display unit 104 using the input unit 103, the brightness-corrected tomographic image is stored in the storage unit 101-02 or the external storage unit 102. save. Note that it is preferable that the image processing unit 101-04, which is an example of an image generation unit, generates at least one frontal image (frontal tomographic image) based on at least some of the corrected three-dimensional tomographic images. At this time, it is preferable that the display control unit 101-05 causes the display unit 104 to display at least one generated front image.

Next, details of the process executed in S303 will be described with reference to the flowchart shown in FIG. 5(a).

<Step 501>
The blood vessel acquisition unit 101-421 acquires the tomographic image generated by the tomographic image generation unit 101-11 in S302.

<Step 502>
The blood vessel acquisition unit 101-421 acquires boundary data of the layer boundaries of the retina and choroid, and the anterior and posterior surfaces of the lamina cribrosa, identified by the image feature acquisition unit 101-42 in S302.

<Step 503>
The blood vessel acquisition unit 101-421 instructs the correction unit 101-44 to perform correction processing (hereinafter referred to as roll-off correction) to compensate for signal attenuation in the depth direction caused by the roll-off characteristics of the tomographic imaging apparatus 100. , the correction unit 101-44 performs the roll-off correction process.

The correction coefficient H(z) in the depth direction for performing roll-off correction can be expressed, for example, as in equation (1), where RoF(z) is the normalized roll-off characteristic function with depth position z as an argument. , roll-off correction is performed by multiplying each pixel value of the tomographic image by the correction coefficient H(z).
H(z)={(BGa+2σ)/(BGa(z)+2σ(z))}/(1+RoF(z)-RoF(z0))...(1)

BGa and σ each indicate the average value and standard deviation of the brightness distribution BG of the entire B-scan data acquired without an object to be inspected. In addition, BGa(z) and σ(z) are the average value and standard deviation of the brightness distribution in the direction perpendicular to the z-axis, calculated at each depth position (z) in B-scan data acquired without an object to be inspected. shows. z0 indicates a reference depth position included in the B-scan range. Although z0 may be set to any constant, it is assumed here that it is set to a value that is 1/4 of the maximum value of z. Note that the roll-off correction formula is not limited to the above, and any known correction process may be executed as long as it has the effect of compensating for signal attenuation in the depth direction caused by the roll-off characteristics of the tomographic imaging apparatus 100. good.

<Step 504>
The blood vessel acquisition unit 101-421 instructs the projection unit 101-43 to generate a frontal tomographic image of the surface layer of the retina and a frontal tomographic image of the outer layer of the retina in preparation for comparing luminance statistics values in different depth ranges, and -43 generates the front tomographic image. Any known projection method may be used as the projection method, but in this embodiment, average value projection is performed. FIG. 6(b) shows an example of a front tomographic image of the surface layer of the retina, and FIG. 6(c) shows an example of a front tomographic image of the outer layer of the retina. In a frontal tomographic image of the surface layer of the retina, the brightness value in the blood vessel area is high (due to the interaction between the measurement light and red blood cells in the blood vessel area), and in a frontal tomographic image of the outer layer of the retina, the brightness value in the blood vessel area is low due to shadows. I understand that.

<Step 505>
The blood vessel acquisition unit 101-421, which is an example of information generation means, calculates the distribution of the brightness attenuation rate Ar based on the brightness values of the two types of frontal tomographic images calculated in S504, in order to compare the brightness statistical values in different depth ranges. Calculate. In this embodiment, as an index for comparing brightness statistical values in different depth ranges, (brightness of the superficial tomographic image of the retina) ÷ (brightness of the front tomographic image of the outer retinal layer) is calculated for each pixel (x, y), and the brightness A map (FIG. 6(d)) of the attenuation rate Ar(x,y) is generated.

<Step 506>
The blood vessel acquisition unit 101-421 normalizes the brightness attenuation rate map Ar(x,y) generated in S505 to generate a blood vessel candidate region map V(x,y) representing the likelihood of a blood vessel region.

In this embodiment, by normalizing the brightness attenuation rate map Ar(x,y) calculated in S505 using predetermined values WL and WW, the blood vessel candidate region map V(x,y) is transformed into V( x,y)=(Ar(x,y)-WL)/WW
It is calculated as follows, and 0≦V(x,y)≦1 is satisfied. FIG. 6(e) shows an example of the blood vessel candidate region map V(x,y). It can be seen that the blood vessel candidate region is highlighted. Note that the normalization process is not limited to the above method, and any known normalization method may be used.

When a projection image in which the luminance values of all depth ranges are summed or a low luminance region in a frontal tomographic image generated in the depth range of the outer retinal layer is considered to be a blood vessel region, shadows due to vitreous opacity (603 in Fig. 6(a) ) and shadows due to vitiligo (605 in FIG. 6(a)) are also included. In contrast, the method shown in S303 of the present embodiment can generate distribution information only for blood vessel candidate regions (and bleeding regions that have arisen from the blood vessels). Further, a cluster scan like OCTA is not necessary, and distribution information regarding blood vessel candidate regions can be generated even with a single scan of a tomographic image.

Note that when generating distribution information for occluding objects other than blood vessels (objects that cause shadows), for example, high-intensity lesions such as vitiligo (604 in Fig. 6(a)), for example (deep retinal layer What is necessary is to calculate a brightness attenuation rate such as (average brightness value in the outer layer of the retina)÷(average brightness value in the outer retinal layer). Here, the blood vessel region, bleeding region, high-intensity lesion, etc. are regions included in the eye to be examined, and are examples of regions that cause shadows that occur along the depth direction of the eye to be examined.

Furthermore, in calculating the brightness attenuation rate, it is not essential to generate a frontal tomographic image, and the calculation may be performed in A-scan units as is from a three-dimensional tomographic image. Furthermore, the brightness attenuation rate is not limited to the ratio of brightness statistical values in different depth ranges, but may be calculated based on the amount of difference (between brightness statistical values in different depth ranges), for example.

Furthermore, details of the process executed in S304 will be explained with reference to the flowchart shown in FIG. 5(b).

<Step 511>
The correction unit 101-44 acquires the tomographic image generated by the tomographic image generation unit 101-11 in S302. Next, the operator instructs generation of a desired projection depth range and a front tomographic image corresponding to the projection depth range via the user interface displayed on the display unit 104. The projection unit 101-43 projects the three-dimensional tomographic image after applying the roll-off correction in the designated depth range to generate a frontal tomographic image (FIG. 7(a)).

In this embodiment, in the projection process of the three-dimensional tomographic image, the average value of the tomographic data in the depth direction corresponding to each pixel in the plane corresponding to the front surface of the fundus is used as the pixel value of the pixel. However, the projection process is not limited to such average value projection, and any known projection method may be used. For example, the pixel value may be the median value, maximum value, mode value, etc. of the tomographic data in the depth direction corresponding to each pixel. Furthermore, by projecting only the luminance values of pixels that exceed the noise threshold corresponding to the background luminance value, it is possible to obtain pixel values (derived from the luminance value of the fundus tissue) excluding the influence of the background region.

<Step 512>
The high-dimensional conversion unit 101-4411 calculates a high-dimensional approximate value distribution, which is an example of the first approximate value distribution, by two-dimensionally smoothing the brightness values of the frontal tomographic image. Here, the two-dimensional smoothing process is an example of a process of two-dimensionally converting the front image (two-dimensional conversion process) when obtaining the first approximate value distribution. In this embodiment, the high-dimensional conversion unit 101-4411 two-dimensionally smoothes the brightness value of each pixel in the front tomographic image generated in S511, so that the brightness of the tomographic image as shown in FIG. Compute a high-dimensional approximate value distribution for values.

Note that in this embodiment, smoothing processing is performed as an example of processing for calculating the approximate value distribution, but morphological operations such as closing processing and opening processing may also be performed as described later. Further, the smoothing process may be performed using an arbitrary spatial filter, or may be performed by frequency-converting the tomographic data using fast Fourier transform (FFT) or the like, and then suppressing high-frequency components. When FFT is used, the smoothing process can be executed at high speed because convolution calculation is not necessary. When smoothing tomographic data by frequency converting and suppressing high frequency components, a predetermined window function (Hamming window or Hanning window) is applied in the frequency domain or a Butterworth filter is applied to suppress ringing. The high frequency components may be suppressed by doing the following.

<Step 513>
The weighting unit 101-442 acquires the blood vessel candidate region map V(x,y) (FIG. 7(d)) from the blood vessel acquiring unit 101-421.

<Step 514>
The weighting unit 101-442 weights the luminance value in the blood vessel candidate region of the tomographic image using the value of the blood vessel candidate region map V(x,y). Note that this weighting exists along the fast axis direction of the measurement light used when acquiring the three-dimensional tomographic image when acquiring the low-dimensional approximate value distribution, which is an example of the second approximate value distribution. This is an example of different calculation processes executed for a predetermined tissue that is a blood vessel or bleeding area and for other areas. Further, this weighting is not essential in the present invention.

In the present embodiment, the higher the value of the blood vessel candidate region map V (x, y) (likelihood of a blood vessel) is, the brightness value of the region corresponding to the higher region in the front tomographic image acquired in S511 is calculated in S512. The brightness value I(x,y) of the frontal tomographic image acquired in S511 is maintained as much as possible so that it approaches the high-dimensional approximate value I_2ds(x,y), and the lower the value of V(x,y) is, the lower the value of V(x,y) is. A weighted frontal image I_w(x,y) is generated by weighting the frontal tomographic image so that the frontal tomographic image is weighted. Specifically, it may be calculated as follows.
I_w(x,y)=(1.0-V(x,y))*I(x,y)+V(x,y)*I_2ds(x,y)

FIG. 7(e) shows an example of the weighted front tomographic image I_w(x,y).

Note that the weighting method for the brightness value of the blood vessel candidate region shown here is just an example, and the process of increasing the brightness value of the blood vessel candidate region running in the fast axis direction or the process of bringing it closer to the brightness value in the vicinity of the blood vessel candidate region. Any weighting may be applied if necessary.

<Step 515>
The low-dimensional conversion unit 101-4412 calculates a low-dimensional approximate value distribution regarding the brightness values of the tomographic image. Specifically, a process (smoothing process or morphological calculation) is performed to calculate an approximate value distribution in the fast axis direction for the brightness value of each pixel of the front tomographic image weighted with the brightness value of the blood vessel candidate region. FIG. 7(f) shows an example of the low-dimensional approximate value distribution calculated in this step. In order to perform low-dimensional transformation (smoothing in the fast axis direction) on an image in which the luminance value of the blood vessel region running in the fast axis direction has been raised, the "blood vessel region running in the fast axis direction" is Problems remaining as brightness areas can be avoided. Here, the low-dimensional smoothing process is an example of the process of converting the front image in one dimension (one-dimensional conversion process) when acquiring the second approximate value distribution. Note that when smoothing processing is performed in the frequency domain as in S512, a predetermined window function (Hamming window, Hanning window, etc.) or a Butterworth filter or the like is applied in the frequency domain to suppress ringing. By doing so, high frequency components may be suppressed and smoothed.

<Step 516>
The calculation unit 101-443 calculates the brightness correction coefficient distribution for the tomographic image by calculating the high-dimensional approximate value distribution and the low-dimensional approximate value distribution of the tomographic image.

In this embodiment, the brightness correction coefficient map for tomographic images ( FIG. 7(g)) is generated.

Next, the effect of luminance weighting on blood vessel candidate regions (facilitating selective suppression of only luminance step artifacts) will be explained using FIGS. 8(a) to 8(g). FIG. 8A is an example of a tomographic image that includes both a band-shaped brightness step localized in a certain range in the fast axis direction and a blood vessel region running in the fast axis direction. It is necessary to selectively suppress only the band-shaped brightness level difference without overcorrecting the brightness value of the blood vessel region running in the fast axis direction.

FIG. 8B shows an example of the processing results when the low-dimensional conversion process in S515, the brightness correction coefficient map calculation process in S516, and the brightness level difference correction process in S305 are executed without performing brightness weighting on the blood vessel candidate region. This figure is shown in (d) and (f) of the same figure. In the blood vessel region (running in the fast axis direction) indicated by the white arrow in FIG. 8(b), a band-shaped low luminance region remains and is similar to a luminance step. Therefore, a high correction coefficient value is calculated in the blood vessel region shown by the white arrow in FIG. Overcorrection of the brightness value of the area occurs (the area indicated by the white arrow in FIG. 8(f)).

On the other hand, FIG. 8C shows the processing results when the low-dimensional conversion process in S515, the brightness correction coefficient map calculation process in S516, and the brightness level difference correction process in S305 are performed after performing brightness weighting on the blood vessel candidate region. This figure is shown in (e) and (g). In FIG. 8(c), there is no band-shaped low-luminance region corresponding to the blood vessel region running in the fast axis direction. Therefore, an appropriate correction coefficient value is calculated for the blood vessel region in FIG. 8(e), and no overcorrection of the brightness value of the blood vessel running in the fast axis direction and its neighboring region is observed in the brightness level difference correction process in S305. No (Figure 8(g)).

Note that although the present embodiment has described a method of suppressing band-shaped brightness steps that occur on a front tomographic image (generating a front tomographic image that has been corrected for brightness steps), the present invention is not limited to this. The band-shaped brightness level difference that occurs on the three-dimensional tomographic image may be suppressed by the following procedure, and a three-dimensional tomographic image with the brightness level difference corrected may be generated.

That is, front tomographic images are generated in a large number of different projection depth ranges, and a brightness level difference correction coefficient map is generated for each front tomographic image. Next, for the brightness value of each pixel on the three-dimensional tomographic image, the value of the brightness step correction coefficient map (correction coefficient value) corresponding to the projection depth range to which each pixel belongs is calculated. Can generate 3D tomographic images. The different projection depth ranges include, for example, four types: the retinal surface layer, the deep retinal layer, the outer retinal layer, and the choroid. Alternatively, the type of each layer belonging to the retina and choroid may be specified.

Note that when suppressing the luminance step artifact that occurs in a C-scan image, since the image contains multiple types of layers, it is necessary to selectively suppress only the luminance step artifact (without adversely affecting pixel values near the layer boundaries). It can be difficult to do so. A brightness level difference corrected C-scan image can be obtained by generating a three-dimensional tomographic image with brightness level difference correction in advance, and generating a C-scan image from the brightness level difference-corrected three-dimensional tomographic image.

Furthermore, the present invention is not limited to correction processing for band-shaped brightness steps that occur when tomographic images are taken using so-called 3-D scanning, but also for correction processing in the slow axis direction when tomographic images are taken using various scan patterns. It can also be applied to correct brightness differences that occur in For example, the present invention also includes the case of correcting a brightness level difference that occurs in the slow axis direction when photographing is performed using a large number of circle scans with different radii or a radial scan. In the case of circle scanning, for example, the circumferential direction is considered to be the fast axis direction, and the direction orthogonal to the circumferential direction is considered to be the slow axis direction. In the case of radial scanning, for example, each radial scan direction passing through a predetermined point is considered to be a fast axis direction, and the circumferential direction around the predetermined point is considered to be a slow axis direction.

According to the configuration described above, the image processing apparatus 101 performs the following image correction processing in order to robustly suppress various brightness step artifacts that occur in the slow axis direction of a tomographic image of the subject's eye taken using OCT. conduct. That is, the image processing device compares a plurality of pieces of distribution information in the in-plane direction intersecting the depth direction of the subject's eye, which corresponds to a plurality of depth ranges in the three-dimensional tomographic image. Distribution information regarding a predetermined region (blood vessel candidate region) in the eye to be examined that is the cause of shadows occurring along the eye is generated. For example, the image processing device generates distribution information of blood vessel candidate regions based on the brightness attenuation rate between the retinal surface layer and the outer retinal layer of the tomographic image. Next, a brightness correction coefficient map is generated by weighting the brightness value of the blood vessel candidate region with respect to the brightness value of the high-dimensional smoothed tomographic image, and then dividing the brightness value of the tomographic image smoothed in the fast axis direction. do. Furthermore, by multiplying the tomographic image by a brightness correction coefficient, brightness step artifacts occurring in the slow axis direction of the tomographic image are robustly suppressed. Note that it is sufficient that the distribution information is finally generated, and there is no need to generate, for example, images (maps) as a plurality of distribution information during the generation.

Thereby, it is possible to robustly suppress the brightness level difference that occurs in the slow axis direction of the tomographic image of the eye to be examined.

[Second embodiment]
The image processing device according to the present embodiment is designed to robustly suppress various luminance step artifacts that occur in the slow axis direction of a motion contrast image generated from a tomographic image of a subject's eye obtained by cluster imaging using OCT. , performs the following image processing. That is, the brightness value of the motion contrast image is weighted with the brightness value of the blood vessel candidate region obtained in the same manner as in the first embodiment with respect to the brightness value of the high-dimensional smoothed motion contrast image, and then smoothed in the fast axis direction. A brightness correction coefficient map is generated by dividing . Furthermore, a case will be described in which a brightness step artifact occurring in the slow axis direction of a motion contrast image is robustly suppressed by multiplying a tomographic image by a brightness correction coefficient.

Here, an example of a brightness level difference that occurs in the slow axis direction of a motion contrast image of a subject's eye to be corrected in this embodiment will be described. When a motion contrast image is generated using a tomographic image in which the rescanning region shown in FIG. 4(b) has low brightness, the corresponding (white arrow) A band-shaped, low-luminance step appears in the area (shown). Furthermore, when a motion contrast image is generated using a tomographic image that includes a short, low-intensity step region in the rescanning region as shown in FIG. There is a problem of creating a level difference.

Furthermore, if there is a positional shift between tomographic images scanned multiple times at the same position, a high motion contrast value will be calculated even for areas where no displacement of red blood cells has actually occurred. A band-like high-intensity step as shown by the white arrow in 4(d) is generated. Note that the same motion contrast image may include both a low-luminance step and a high-luminance step. In addition, when ocular tissues such as the retina and choroid are included only in a part of the fast axis direction on the image, or when there are areas where layer boundaries are poorly detected, band-like high-intensity steps may appear in the middle. There is a problem that the brightness step may be interrupted or the thickness or height of the brightness step may change midway.

FIG. 9 shows the configuration of an image processing system 10 including an image processing device 101 according to this embodiment. This embodiment differs from the first embodiment in that the image acquisition section 101-01 includes a motion contrast data generation section 101-12, and the image processing section 101-04 includes a composition section 101-45. Further, an image processing flow in this embodiment is shown in FIG. Note that in FIG. 10, S1002 and S1003 are the same as in the first embodiment, so their explanation will be omitted.

<Step 1001>
By operating the input unit 103, the operator sets OCT image imaging conditions to be instructed to the tomographic imaging apparatus 100.

Specifically, the procedure includes 1) selection of a scan mode, and 2) setting of imaging parameters corresponding to the scan mode. In this embodiment, OCT imaging is performed with the following settings.
1) Select OCTA scan mode 2) Set the following imaging parameters 2-1) Scan area size: 10x10mm
2-2) Main scanning direction: horizontal direction 2-3) Scanning interval: 0.01mm
2-4) Fixation lamp position: midway between the fovea and optic disc 2-5) Number of B scans at the same imaging position: 4
2-6) Coherence gate position: Vitreous body side Next, the operator operates the input unit 103 and presses the imaging start button (hidden) on the imaging screen to repeatedly perform OCTA imaging under the imaging conditions set above. Start.

The imaging control unit 101-03 instructs the tomographic imaging apparatus 100 to repeatedly perform OCTA imaging based on the above settings, and the tomographic imaging apparatus 100 acquires corresponding OCT tomographic images.

In this embodiment, the number of times of repeated imaging (the number of clusters) in this step is five times. The number of times of repeated imaging (the number of clusters) is not limited to this, and may be set to an arbitrary number of times.

The tomographic imaging apparatus 100 also acquires SLO images and executes tracking processing based on the SLO moving images.

Note that when the number of clusters is two or more, the reference SLO image used for tracking processing in repeated OCTA imaging is the reference SLO image set during the first cluster imaging, and a common reference SLO image is used in all cluster imaging. Also, during the second and subsequent cluster shooting,
- The same setting values as in the case of the first cluster imaging are used (not changed) for selection of left and right eyes and whether to execute tracking processing.

<Step 1004>
In the image acquisition unit 101-01 and the image processing unit 101-04, the alignment unit 101-41 aligns tomographic images belonging to the same cluster and aligns tomographic images between clusters, and the aligned tomographic images Generate a motion contrast image using

A motion contrast data generation unit 101-12 calculates motion contrast between adjacent tomographic images within the same cluster. In this embodiment, a decorrelation value Mxy is determined as the motion contrast based on the following equation (2).

Here, Axy indicates the amplitude (of complex number data after FFT processing) at the position (x, y) of the tomographic image data A, and Bxy indicates the amplitude at the same position (x, y) of the tomographic data B. 0≦Mxy≦1, and the larger the difference between both amplitude values, the closer the value is to 1. Perform the decorrelation calculation process as shown in Equation (2) between arbitrary adjacent tomographic images (belonging to the same cluster), and calculate the average of the obtained (number of tomographic images per cluster - 1) motion contrast values. An image having pixel values is generated as a final motion contrast image.

Note that although the motion contrast is calculated here based on the amplitude of the complex number data after FFT processing, the method of calculating the motion contrast is not limited to the above. For example, motion contrast may be calculated based on phase information of complex number data, or may be calculated based on both amplitude and phase information. Alternatively, motion contrast may be calculated based on the real part or imaginary part of complex number data.

Further, in this embodiment, a decorrelation value is calculated as the motion contrast, but the method of calculating the motion contrast is not limited to this. For example, the motion contrast may be calculated based on the difference between two values, or the motion contrast may be calculated based on the ratio of two values.

Further, in the above, the final motion contrast image is obtained by calculating the average value of the plurality of acquired decorrelation values, but the present invention is not limited to this. For example, an image whose pixel value is the median value or maximum value of the plurality of acquired decorrelation values may be generated as the final motion contrast image.

The image processing unit 101-04 three-dimensionally aligns a group of motion contrast images obtained through repeated OCTA imaging and averages them to generate a high-contrast composite motion contrast image. Note that the compositing process is not limited to simple arithmetic averaging. For example, the brightness value of each motion contrast image may be arbitrarily weighted and then averaged, or an arbitrary statistical value such as a median value may be calculated. The present invention also includes a case where the alignment process is performed two-dimensionally.

Note that the compositing unit 101-45 may be configured to determine whether or not a motion contrast image inappropriate for the compositing process is included, and then perform the compositing process excluding the motion contrast image determined to be inappropriate. For example, if the evaluation value (for example, the average value or median value of decorrelation values) for each motion contrast image is outside a predetermined range, it may be determined that the motion contrast image is unsuitable for compositing processing.

In this embodiment, after the synthesis unit 101-45 three-dimensionally synthesizes motion contrast images, the correction unit 101-44 performs processing to three-dimensionally suppress projection artifacts occurring in the motion contrast images.

Here, projection artifact refers to a phenomenon in which motion contrast within the superficial retinal blood vessels is reflected in the deep layers (deep retina, outer retina, and choroid), resulting in high decorrelation values in deep regions where no blood vessels actually exist. . The correction unit 101-44 executes processing to suppress projection artifacts generated on the three-dimensional composite motion contrast image. Although any known projection artifact suppression technique may be used, this embodiment uses Step-down Exponential Filtering. In Step-down Exponential Filtering, projection artifacts are suppressed by executing the process expressed by equation (3) on each A-scan data on a three-dimensional motion contrast image.

Here, γ is an attenuation coefficient with a negative value, D (x, y, z) is a decorrelation value before projection artifact suppression processing, and _DE (x, y, z) is a decorrelation value after the suppression processing. represents.

Finally, the image processing device 101 converts the acquired image group (SLO image or tomographic image), the imaging condition data of the image group, the generated motion contrast image and the accompanying generation condition data into information such as the examination date and time, and the identification of the eye to be examined. The information is stored in the external storage unit 102 in association with the .

<Step 1005>
The weighting unit 101-442 generates a weighted motion contrast image in which the luminance values in the blood vessel candidate region of the motion contrast image are weighted using the information regarding the distribution of the blood vessel candidate region generated by the blood vessel acquisition unit 101-421 in S1003. Next, the high-dimensional transformation unit 101-4411 generates a high-dimensional smoothed motion contrast image, and the low-dimensional transformation unit 101-4412 performs smoothing processing on the weighted motion contrast image in the fast axis direction. Generate a dimensionally smoothed motion contrast image. Further, the calculation unit 101-443 generates a brightness correction coefficient map for the motion contrast image by performing calculation processing on the high-dimensional smoothed motion contrast image and the low-dimensional smoothed motion contrast image.

Note that details of the brightness correction coefficient map generation process will be explained in S511 to S516.

<Step 1006>
The correction unit 101-44 multiplies each pixel of the motion contrast image by the brightness correction coefficient value calculated in S1005, thereby generating a motion contrast image with the brightness level difference corrected. Note that the method of applying the brightness correction coefficient is not limited to multiplication, and any known calculation method may be applied. Further, at least a portion of the three-dimensional motion contrast image may be corrected using the brightness correction coefficient value. At this time, at least a portion of the three-dimensional motion contrast image includes a C-scan motion contrast image and the like.

<Step 1007>
The display control unit 101-05 displays the brightness-corrected motion contrast image generated in S1006 on the display unit 104. In addition, when the operator uses the input unit 103 to select a button or shortcut menu (not shown) displayed on the display unit 104, the brightness-corrected motion contrast image is saved in the storage unit 101-02 or the external storage unit 102. do. Note that it is preferable that the image processing unit 101-04, which is an example of an image generation unit, generates at least one frontal image (frontal motion contrast image) based on at least a portion of the corrected three-dimensional motion contrast image. At this time, it is preferable that the display control unit 101-05 causes the display unit 104 to display at least one generated front image.

In this embodiment, a report screen 1300 is displayed on the display unit 104 by pressing the Report button 1312 in FIG. A brightness level difference corrected front tomographic image 1309 is displayed at the lower left of the report screen 1300, and the projection range can be changed by the operator making a selection from a list displayed in a list box 1310. A front tomographic image or a front motion contrast image that has been corrected for brightness level difference is superimposed and displayed on the SLO image at the upper left of the report screen 1300. In addition, brightness step-corrected motion contrast images 1301 and 1305 having different projection depth ranges are displayed above and below the center of the report screen 1300. The projection range of the brightness step-corrected frontal motion contrast image can be changed by the operator selecting from the predetermined depth range set (1302 and 1306) displayed in the list box. In addition, the type and offset position of the layer boundary used to specify the projection range can be changed from the user interfaces 1303 and 1307, and the layer boundary data (1304 and 1308) superimposed on the tomographic image can be operated from the input unit 103. You can change the projection range by moving the screen.

Furthermore, the image projection method of the luminance step-corrected motion contrast image and the presence or absence of projection artifact suppression processing may be changed by selecting from a user interface such as a context menu.

Furthermore, the operator may press a combination instruction button 1311 for combining a plurality of motion contrast images, thereby generating a combined motion contrast image that has been corrected for brightness differences. Although the compositing instruction button in FIG. 13 shows an example for superimposing processing, the invention is not limited to this, and the present invention also includes a compositing instruction regarding pasting processing.

Note that by selecting the user interface (1313) that specifies whether or not to apply brightness level difference correction processing to a tomographic image or motion contrast image, a tomographic image or motion contrast image with changed applicability of brightness level difference correction processing can be displayed. good. For example, the application status (apply/not apply) of brightness level difference correction processing to the tomographic image or motion contrast image displayed on the report screen 1300 can be determined depending on the selection/unselection of the check box shown in 1313 in FIG. You can switch. The user interface is not limited to check boxes, and any known user interface may be used. Further, a user interface may be provided that can independently instruct whether or not to apply brightness level difference correction processing for tomographic images and motion contrast images. For example, separate instruction buttons may be provided for tomographic images and motion contrast images, or a single user interface may provide four options ((1) apply to both, (2) apply only to tomographic images). It may be configured to select from 3) applied only to motion contrast images, and 4) applied to neither. Alternatively, a tomographic image or motion contrast image to which the brightness level difference correction process has been applied and a tomographic image or motion contrast image to which the brightness level difference correction process has not been applied may be displayed side by side on the display unit 104.

Next, details of the process executed in S1005 will be described with reference to the flowchart shown in FIG. 5(b).

<Step 511>
The correction unit 101-44 obtains the motion contrast image and the composite motion contrast image generated by the motion contrast data generation unit 101-12 and the synthesis unit 101-45 in S1004. Next, the operator instructs generation of a desired projection depth range and a frontal motion contrast image corresponding to the projection depth range via the user interface displayed on the display unit 104. The projection unit 101-43 projects in the designated depth range and generates a frontal motion contrast image (FIG. 11(a)). In this embodiment, in the projection process of the three-dimensional motion contrast image, the maximum value of the motion contrast data in the depth direction corresponding to each pixel in the plane corresponding to the front surface of the fundus is set as the pixel value of the pixel. However, the projection process is not limited to such maximum intensity projection, and any known projection method may be used. For example, the pixel value may be the median value, maximum value, mode value, etc. of motion contrast data in the depth direction corresponding to each pixel.

<Step 512>
The high-dimensional conversion unit 101-4411 calculates a high-dimensional approximate value distribution by two-dimensionally smoothing the brightness values of the frontal motion contrast image. In this embodiment, the high-dimensional conversion unit 101-4411 creates a motion contrast image as shown in FIG. A high-dimensional approximate value distribution regarding the brightness values of is calculated.

Note that in this embodiment, smoothing processing is performed as an example of processing for calculating the approximate value distribution, but morphological operations such as closing processing and opening processing may also be performed as described later. Further, the smoothing process may be performed by using an arbitrary spatial filter, or by converting the frequency of the motion contrast data using fast Fourier transform (FFT), etc., and then suppressing high frequency components. . When FFT is used, the smoothing process can be executed at high speed because convolution calculation is not necessary.

<Step 513>
The weighting unit 101-442 acquires the blood vessel candidate region map V(x,y) (FIG. 7(d)) from the blood vessel acquiring unit 101-421.

<Step 514>
The weighting unit 101-442 weights the luminance value in the blood vessel candidate region of the motion contrast image using the value of the blood vessel candidate region map V(x,y). Note that this weighting exists along the fast axis direction of the measurement light used when acquiring the three-dimensional motion contrast image when acquiring the low-dimensional approximate value distribution, which is an example of the second approximate value distribution. This is an example of different calculation processes executed for a predetermined tissue that is a blood vessel or a bleeding area, and for other areas. Further, this weighting is not essential in the present invention.

In the present embodiment, the higher the value of the blood vessel candidate region map V (x, y) (likelihood of a blood vessel) is, the brightness value of the region corresponding to the higher region in the frontal motion contrast image acquired in S511 is calculated in S512. The brightness value M(x,y) of the frontal motion contrast image acquired in S511 is A weighted frontal motion contrast image M_w(x,y) is generated by weighting the frontal motion contrast image so as to maintain it as much as possible. Specifically, it may be calculated as follows.
M_w(x,y)=(1.0-V(x,y))*M(x,y)+V(x,y)*M_2ds(x,y)

FIG. 11(e) shows an example of the weighted frontal motion contrast image M_w(x,y).

Note that the weighting method for the brightness value of the blood vessel candidate region shown here is just an example, and the process of reducing the brightness value of the blood vessel candidate region running in the fast axis direction or the process of bringing it closer to the brightness value in the vicinity of the blood vessel candidate region. Any weighting may be applied if necessary.

<Step 515>
The low-dimensional conversion unit 101-4412 calculates a low-dimensional approximate value distribution regarding the brightness values of the motion contrast image. Specifically, processing (smoothing processing or morphological calculation) is performed to calculate the approximate value distribution in the fast axis direction for the luminance value of each pixel of the frontal motion contrast image weighted with the luminance value of the blood vessel candidate region. FIG. 11(f) shows an example of the low-dimensional approximate value distribution calculated in this step. In order to perform low-dimensional transformation (smoothing in the fast axis direction) on images in which the luminance values of blood vessel regions running in the fast axis direction are suppressed, the "vessel regions running in the fast axis direction" are transformed into band-shaped images. The problem of remaining high brightness areas can be avoided.

<Step 516>
The calculation unit 101-443 calculates the luminance correction coefficient distribution for the motion contrast image by calculating the high-dimensional approximate value distribution of the motion contrast image and the low-dimensional approximate value distribution of the motion contrast image.

In this embodiment, a brightness correction coefficient map for the motion contrast image is created by dividing the brightness value of the two-dimensional smoothed motion contrast image generated in S512 by the brightness value of the weighted fast-axis direction smoothed motion contrast image generated in S515. (Fig. 11(g)) is generated.

Next, the effect of luminance weighting on blood vessel candidate regions (facilitating selective suppression of only luminance level difference artifacts) will be explained using FIGS. 12(a) to 12(g). FIG. 12A is an example of a motion contrast image that includes both a band-shaped brightness step (white line) and a blood vessel region running in the fast axis direction. It is necessary to selectively suppress only the band-shaped brightness step without over-suppressing the brightness value of the blood vessel region running in the fast axis direction.

FIG. 12B shows an example of the processing results when the low-dimensional conversion process in S515, the brightness correction coefficient map calculation process in S516, and the brightness level difference correction process in S1006 are executed without performing brightness weighting on the blood vessel candidate region. It is shown in the same figure (d) and the same figure (f). In the blood vessel region (traversing in the fast axis direction) indicated by the white arrow in FIG. 12(b), a band-like high brightness region remains and is similar to a brightness step. Therefore, a low correction coefficient value is calculated for the blood vessel region indicated by the white arrow in FIG. Over-suppression of the luminance value occurs (region indicated by the white arrow in FIG. 12(f)).

On the other hand, FIG. 12C shows the processing results when the low-dimensional conversion process in S515, the brightness correction coefficient map calculation process in S516, and the brightness level difference correction process in S1006 are performed after performing brightness weighting on the blood vessel candidate region. It is shown in the same figure (e) and the same figure (g). In FIG. 12(c), there is no band-shaped high-intensity region corresponding to a blood vessel region running in the fast axis direction. Therefore, an appropriate correction coefficient value is calculated for the blood vessel region in FIG. 12(e), and no excessive suppression of the brightness value of the blood vessel running in the fast axis direction and its neighboring region is observed in the brightness level difference correction processing of S1006. No (Figure 12(g)).

Although the present embodiment has described a method of suppressing band-shaped brightness steps that occur on a frontal motion contrast image (generating a frontal motion contrast image that has been corrected for brightness steps), the present invention is not limited to this. The band-shaped brightness level difference that occurs on the three-dimensional motion contrast image may be suppressed by the following procedure, and a three-dimensional motion contrast image that has been corrected for the brightness level difference may be generated.

That is, frontal motion contrast images are generated in a large number of different projection depth ranges, and a brightness level difference correction coefficient map is generated for each frontal motion contrast image. Next, the brightness level difference is corrected by calculating the value of the brightness level difference correction coefficient map (correction coefficient value) corresponding to the projection depth range to which each pixel belongs to the brightness value of each pixel on the three-dimensional motion contrast image. 3D motion contrast images can be generated. The different projection depth ranges include, for example, four types: the retinal surface layer, the deep retinal layer, the outer retinal layer, and the choroid. Alternatively, the type of each layer belonging to the retina and choroid may be specified.

In addition, the timing for correcting the brightness level difference is such that a 3D tomographic image or a 3D motion contrast image that has been corrected for the brightness level difference is generated in advance using the above procedure, and then the operator gives an instruction to generate a front tomographic image or a frontal motion contrast image. The frontal image may be generated and displayed at the moment the front image is detected. Examples of the timing of generating a three-dimensional tomographic image or a three-dimensional motion contrast image with brightness level difference correction include, for example, immediately after taking a tomographic image, at the time of reconstruction, and at the time of storage. Alternatively, when an operator instructs the generation of a frontal tomographic image or a frontal motion contrast image, the brightness level difference correction is performed (for the projection depth range specified by the instruction), and the brightness level difference corrected frontal tomographic image is created. Alternatively, a front motion contrast image may be displayed. An example of a user interface for instructing generation of a frontal tomographic image is a list box 1310 shown in FIG. 13 . Further, as the front motion contrast image generation instruction user interface, for example, the projection depth range designation/change user interface (1302, 1303, 1304, 1306, 1307, 1308) shown in FIG. 13 can be mentioned. Note that luminance level difference correction may be performed on each tomographic image or motion contrast image in advance or based on a composition (superimposition or pasting) instruction from the operator, and then composition may be performed. Alternatively, the brightness level difference correction may be performed on the composite image (overlaid image or composite image) and the resultant image may be displayed on the display unit 104.

Furthermore, the present invention is not limited to correction of band-like brightness steps on motion contrast images that occur when photographing with so-called 3-D scanning, but also correction of band-like brightness steps in the slow axis direction on motion contrast images when photographing with various scan patterns. It can also be applied to correct brightness differences that occur in For example, the present invention also includes the case of correcting a brightness level difference that occurs in the slow axis direction when imaging is performed using a large number of circle scans with different radii or a radial scan.

According to the configuration described above, the image processing device 101 robustly suppresses various luminance step artifacts occurring in the slow axis direction of a motion contrast image generated from a tomographic image of the eye to be examined obtained by cluster imaging using OCT. In order to do this, the following image processing is performed. That is, the brightness value of the motion contrast image is weighted with the brightness value of the blood vessel candidate region obtained in the same manner as in the first embodiment, and then smoothed in the fast axis direction with respect to the brightness value of the high-dimensional smoothed motion contrast image. A brightness correction coefficient map is generated by dividing . Furthermore, by multiplying the tomographic image by a brightness correction coefficient, brightness step artifacts occurring in the slow axis direction of the motion contrast image are robustly suppressed. Thereby, it is possible to robustly suppress the brightness level difference that occurs in the slow axis direction of the motion contrast image of the eye to be examined.

[Third embodiment]
The image processing apparatus according to the present embodiment displays a medical image such as a frontal image (frontal tomographic image or frontal motion contrast image) to which various artifact reduction processes such as the luminance step artifact suppression process described above have not been applied on the imaging confirmation screen. On the other hand, the report screen displays a medical image to which artifact reduction processing has been applied. As a result, in order to easily understand the success or failure (or degree of failure) of a medical image on the post-imaging display screen (imaging confirmation screen), the operator can, for example, display a medical You can check the image. In addition, for example, on other display screens (report screens), in order to understand the analysis results, the operator can check the medical images in which various artifacts unnecessary for analysis have been reduced as much as possible. can. Therefore, the operator can confirm a medical image suitable for the purpose.

Note that the artifact in this embodiment is not limited to the luminance level difference artifact described above, and may be any artifact within the range of the purpose described above. For example, the artifact in this embodiment may be a projection artifact in OCTA (a blood vessel that does not originally exist in the lower layer is depicted by erroneously detecting fluctuations in the shadow of the blood vessel in the upper layer as motion contrast). Furthermore, the various artifact reduction processes themselves may be started immediately after shooting, or may be started after the shooting confirmation screen is transitioned to the report screen. Further, the image processing unit 101-04 is an example of a generation unit that generates a front image in which various artifacts are reduced. Further, the shooting confirmation screen is an example of the first front image, and the report screen is an example of the second display screen. Further, the report screen is one of the display screens after switching from the shooting confirmation screen, and is a display screen for the operator to check images after various processing, analysis results, and the like.

Further, the image processing apparatus according to the present embodiment may include a reception unit that receives an instruction regarding whether or not to apply brightness step artifact suppression processing that occurs in the slow axis direction of a tomographic image or a motion contrast image. As a result, the display control unit displays the tomographic image or motion contrast image with the luminance step artifact suppression process applied, in response to the instruction regarding whether or not to apply the suppression process, which the reception unit receives on each of the imaging confirmation screen and the report screen. Alternatively, it can be displayed on the display unit in a non-applied state. Specifically, a case will be described in which settings regarding application of brightness level difference artifact suppression processing when displaying a tomographic image and a motion contrast image are made different between the imaging confirmation screen and the report screen.

The image processing apparatus according to the present embodiment differs from the first embodiment or the second embodiment in that it includes a reception section (not shown). Further, the image processing flow in this embodiment is the same as that in the second embodiment (FIG. 10). Note that in FIG. 10, steps S1001 to S1004 are the same as in the second embodiment, so their explanation will be omitted.

<Step 1005>
The weighting unit 101-442 generates a brightness correction coefficient map for the motion contrast image by performing processing similar to S1005 of the second embodiment. Note that in this embodiment, the weighting unit 101-442 also generates a brightness correction coefficient map for tomographic images by performing the same process as S304 of the first embodiment.

<Step 1006>
The correction unit 101-44 multiplies each pixel of the motion contrast image by the brightness correction coefficient value for the motion contrast image calculated in S1005, thereby generating a motion contrast image with the brightness level difference corrected. Further, in this embodiment, the correction unit 101-44 multiplies each pixel of the tomographic image by the brightness correction coefficient value for the tomographic image calculated in S1005, thereby generating a tomographic image with brightness level difference corrected. .

<Step 1007>
Based on the tomographic image generated in S1002, the motion contrast image generated in S1004, and the motion contrast image and tomographic image with brightness step correction generated by the correction unit 101-44, the display control unit 101-05 displays the shooting confirmation on the display unit 104. Display the screen (Figure 14). In the main imaging confirmation screen, the fundus image 1401 is shown in the upper left, the front tomographic image 1402 is shown in the lower left, and the B-scan tomographic images (1406a, 1406b, 1406c) corresponding to each scanning position (1402a, 1402b, 1402c) of the front tomographic image are displayed. indicate. It also includes an image quality index value 1405 of the acquired tomographic image and an instruction button 1404 for automatically and continuously displaying each slice of the acquired three-dimensional tomographic image. Based on the tomographic image and motion contrast image displayed on the imaging confirmation screen, the operator issues instructions regarding whether to save the captured tomographic image (by pressing the OK button 1407 or NG button 1408) and instructions regarding the continuation of repeated imaging (Repeat). button 1409). Based on the instruction received by the reception unit, the image processing device performs corresponding data storage/imaging continuation processing. Furthermore, by pressing the Report button 1312 similarly to S1007 of the second embodiment, the display control unit 101-05 displays a report screen 1300 on the display unit 104.

In the imaging confirmation screen (FIG. 14), a front tomographic image 1402 is displayed at the lower left, and when the motion contrast image generation and brightness level difference correction processing are completed, the display is switched to the front motion contrast image (FIG. 15(b)). In this embodiment, a user interface 1403 is provided at the bottom left side of the imaging confirmation screen for instructing whether or not to apply brightness level difference correction processing to the tomographic image or motion contrast image displayed on the imaging confirmation screen. In FIG. 14, a check box labeled "Image Quality Enhancement" is displayed as an example of the user interface 1403, and the check box is in an unselected state (OFF state). In such a selection state, the display control unit 101-05 displays the tomographic image and the motion contrast image in a state in which brightness level difference correction processing is not applied (for example, in the case of a motion contrast image, the state is as shown in FIG. 15(b)). is displayed on the display unit 104. On the other hand, if the checkbox is selected (ON), the tomographic image or motion contrast image is displayed on the display unit 104 with the brightness level difference correction process applied (FIGS. 15(a) and 15(c)). do. Note that, as in the case of the second embodiment, a user interface may be provided that can independently instruct whether or not to apply the brightness level difference correction process on the actual imaging confirmation screen for the tomographic image and the motion contrast image. For example, separate instruction user interfaces may be provided for tomographic images and motion contrast images, or a single user interface may provide four options ((1) applicable to both, (2) only for tomographic images). It may be configured to select from among (3) application only to motion contrast images, and (4) application to neither. For example, by selecting (1), it is possible to instruct whether or not to save the tomographic image or continue repeated imaging after understanding the image quality of the tomographic image and motion contrast image used in actual medical treatment. In addition, by selecting (4), it is possible to determine the extent and location of positional deviations between tomographic images that cause white lines when generating poorly captured tomographic images (low-luminance areas) and motion contrast images. Once this information is understood, it is possible to instruct whether or not to save the tomographic image and continue repeated imaging. Alternatively, by selecting (3), it is possible to instruct whether or not to save the tomographic image or continue repeated imaging while understanding the defective portions of the tomographic image and the quality of the final motion contrast image. If you want to understand only small positional deviations between tomographic images, select (2).

Further, the user interface regarding whether or not the brightness level difference correction process is applicable to tomographic images and motion contrast images may be configured to be able to be set independently between the imaging confirmation screen and the report screen. For example, the user interface defaults to a state where an instruction is selected to display the luminance level difference correction processing for tomographic images and motion contrast images in a non-applied state on the imaging screen and in a state where it is applied on the report screen. Good as a setting. With this configuration, it is possible to easily identify areas where imaging has failed during imaging confirmation, and high-quality tomographic images and motion contrast images suitable for medical treatment can be observed on the report screen.

At this time, when the display screen changes, the instruction selected before the display screen changes may be configured to be reflected even after the display screen changes. For example, the instruction selected on the shooting confirmation screen may be configured to be reflected even after transition to the report screen. Further, for example, an instruction selected on a display screen that is a report screen and on which images obtained at a predetermined date and time are displayed is configured to be reflected even after transition to a display screen for follow-up observation. It's okay. Further, the instruction selected on the progress observation display screen may be configured to be reflected on a plurality of images at different times and times at once. These can improve convenience for the operator.

Further, the correction processing targeted by the user interface for specifying applicability on the shooting confirmation screen and the report screen is not limited to the luminance level difference correction processing, and any known image quality improvement processing may be selectable. For example, a user interface is provided that accepts an instruction as to whether to apply high image quality processing using machine learning on the imaging confirmation screen, and the high image quality processing is applied/disabled to a tomographic image or a motion contrast image depending on the selection state of the user interface. The application may be switched and displayed.

Note that on the imaging confirmation screen or report screen, a tomographic image or a motion contrast image may be displayed by switching between application and non-application of the brightness level difference correction process using a predetermined user interface or script. Alternatively, the image processing device 101 may be configured to display a tomographic image or motion contrast image to which the brightness level difference correction process has been applied and a tomographic image or motion contrast image to which the brightness level difference correction process has not been applied side by side.

According to the configuration described above, the image processing device 101 may include a reception unit that receives an instruction regarding whether or not to apply a luminance step artifact suppression process that occurs in the slow axis direction of a tomographic image or a motion contrast image. As a result, the display control unit displays the tomographic image or motion contrast image with the luminance step artifact suppression process applied, in response to the instruction regarding whether or not to apply the suppression process, which the reception unit receives on each of the imaging confirmation screen and the report screen. Alternatively, it can be displayed on the display unit in a non-applied state. Therefore, when confirming imaging, it is possible to easily confirm the location of failure in imaging, and when observing the report screen, it is possible to observe high-quality tomographic images and motion contrast images suitable for medical treatment.

(Modification 1)
In the imaging confirmation screen in the various embodiments described above, the display control unit displays the determination results (classification results) of the states of various artifacts (for example, brightness step artifacts) in the frontal image (frontal tomographic image or frontal motion contrast image). It may also be displayed together with the front image. Here, the state of the artifact is, for example, the presence or absence of an artifact. At this time, for example, labels are attached to the states of various artifacts (for example, presence or absence) to multiple frontal images of at least one subject's eye, and the learning obtained by machine learning using the multiple frontal images with the labels is applied. Using the completed model, the state of various artifacts (for example, the presence of various artifacts) in the input frontal image is displayed on the imaging confirmation screen. That is, the determination result of the state of the artifact obtained using the above-described trained model can be displayed on the photographing confirmation screen. Thereby, for example, by using a trained model, it is possible to make a determination with high accuracy and to shorten the processing time. Therefore, the examiner can confirm highly accurate determination results even immediately after imaging. Furthermore, for example, even immediately after imaging, it is possible to improve the efficiency of the examiner's decision as to whether or not re-imaging is necessary. Therefore, the accuracy and efficiency of diagnosis can be improved. Note that the labels for the states of the various artifacts may be manually input by the operator via the user interface, or may be the results of a rule-based analysis that automatically or semi-automatically determines the various artifacts. In addition, the display screen on which the judgment result of the state of the artifact is displayed is not limited to the shooting confirmation screen, but also includes, for example, a report screen, a display screen for progress observation, a preview screen for various adjustments before shooting (various live video may be displayed on at least one display screen, such as a display screen on which an image is displayed.

Here, machine learning includes, for example, deep learning consisting of a multilayer neural network. Further, for example, a convolutional neural network (CNN) can be used for at least one layer of the multi-layer neural network. However, machine learning is not limited to deep learning, and any model may be used as long as it is capable of extracting (expressing) features of learning data such as images by itself through learning. At this time, the trained model may be obtained, for example, by supervised learning using learning data in which information regarding the states of various artifacts is used as correct data (supervised data) and medical images such as frontal images are used as input data. Furthermore, the trained model is obtained by unsupervised learning using, for example, multiple medical images such as multiple frontal images of at least one eye to be examined as learning data, without using the above-mentioned correct data (supervised data). Also good. Further, the learned model may be updated through additional learning, so that it can be customized as a model suitable for the operator, for example.

Further, the image processing unit 101-04 is an example of a determination unit (classification unit) that determines the state of various artifacts in a medical image such as a frontal image. At this time, the determining means can, for example, determine the presence or absence of an artifact in the medical image as an artifact state (classify it as presence or absence). Further, the determining means may determine, for example, a stage corresponding to the degree of the artifact in the medical image as the state of the artifact (classifying the medical image into one of a plurality of stages). At this time, the plurality of stages may be, for example, the presence or absence of artifacts, or may be a plurality of levels depending on the number of artifacts, the size of the existence range, etc. Further, the determining means may, for example, evaluate the type of artifact in the medical image as the state of the artifact (classify the medical image into one of a plurality of types).

Further, the trained model described above may be obtained by learning using learning data that is a set of multiple types of medical images that correspond to each other. At this time, the trained model uses, for example, learning data that is a set of a frontal tomographic image and a frontal motion contrast image of the same region of the same subject's eye (or obtained from an interference signal in which at least a part of a predetermined region is common). It can be obtained by learning using In this way, by learning using learning data that is a set of multiple medical images of different types, it is possible not only to determine the state of the artifact, but also to classify the medical image into a type that corresponds to the feature amount. The accuracy of this classification can be improved.

Furthermore, analysis results such as the thickness of a desired layer and various blood vessel densities may be displayed on the report screen in the various embodiments described above. At this time, for example, by analyzing a medical image to which various artifact reduction processes have been applied, highly accurate analysis results can be displayed. Further, the analysis results may be displayed as an analysis map, sectors showing statistical values corresponding to each divided area, or the like. Note that the trained model described above may be obtained by learning using the analysis results of medical images as learning data. In addition, the trained model can be trained using training data that includes a medical image and the analysis result of the medical image, or training data that includes the medical image and the analysis result of a different type of medical image than the medical image. It may be something you have obtained. In addition, the trained model is obtained by learning using learning data that includes input data that is a set of multiple medical images of different types of predetermined parts, such as frontal tomographic images and frontal motion contrast images. Also good.

Furthermore, various diagnostic results such as glaucoma and age-related macular degeneration may be displayed on the report screen in the various embodiments described above. At this time, for example, by analyzing medical images to which various artifact reduction processes have been applied, highly accurate diagnostic results can be displayed. In addition, the diagnosis result may be displayed by displaying the position of the identified abnormal region on the image, or by displaying the state of the abnormal region by text or the like. Note that the trained model described above may be obtained by learning using diagnostic results of medical images as learning data. In addition, the trained model can be trained using training data that includes a medical image and the diagnosis result of the medical image, or training data that includes the medical image and the diagnosis result of a different type of medical image than the medical image. It may be something you have obtained.

Further, the trained model described above may be a trained model obtained by learning using learning data including input data that is a set of a plurality of different types of medical images of a predetermined region of a subject. At this time, the input data included in the learning data is, for example, input data that is a set of a motion contrast frontal image and a brightness frontal image (or a brightness tomographic image) of the fundus, a tomographic image (B scan image) of the fundus, and a color fundus Input data such as a set of images (or fluorescent fundus images) can be considered. Further, the plurality of medical images of different types may be of any type as long as they are obtained using different modalities, different optical systems, different principles, etc. Further, the trained model described above may be a trained model obtained by learning using learning data including input data that is a set of a plurality of medical images of different parts of the subject. At this time, the input data included in the learning data is, for example, input data that is a set of a tomographic image of the fundus (B-scan image) and a tomographic image of the anterior segment (B-scan image), or a three-dimensional image of the macula of the fundus. Input data such as a set of an OCT image and a circle scan (or raster scan) tomographic image of the optic disc of the fundus can be considered. Note that the input data included in the learning data may be a plurality of medical images of different parts and different types of the subject. At this time, the input data included in the learning data may be, for example, input data that includes a tomographic image of the anterior segment of the eye and a color fundus image. Further, the above-mentioned trained model may be a trained model obtained by learning using learning data including input data that is a set of a plurality of medical images of a predetermined region of a subject at different photographing angles of view. Further, the input data included in the learning data may be a combination of a plurality of medical images obtained by time-divisionally dividing a predetermined region into a plurality of regions, such as a panoramic image. Further, the input data included in the learning data may be input data that is a set of a plurality of medical images of a predetermined region of the subject at different times and times.

In addition, the display screen on which at least one of the above-mentioned analysis results and diagnostic results is displayed is not limited to the report screen, but includes, for example, an imaging confirmation screen, a display screen for follow-up observation, and a display screen for various adjustments before imaging. It may be displayed on at least one display screen such as a preview screen (a display screen on which various live moving images are displayed). For example, by displaying at least one of the analysis results and diagnostic results obtained using the trained model described above on the imaging confirmation screen, the examiner can confirm accurate results even immediately after imaging. can do.

(Modification 2)
In the preview screen in the various embodiments and modifications described above, the learned model described above may be used for each at least one frame of a live video image. At this time, if a plurality of live moving images of different parts or types are displayed on the preview screen, a learned model corresponding to each live moving image may be used. With this, for example, even for live moving images, the processing time can be shortened, so the examiner can obtain highly accurate information before starting imaging. Therefore, for example, it is possible to reduce failures in re-imaging, etc., and thus it is possible to improve the accuracy and efficiency of diagnosis. Note that the plurality of live moving images include, for example, a moving image of the anterior segment of the eye for alignment in the XYZ directions, a frontal moving image of the fundus for focus adjustment of the fundus observation optical system and OCT focus adjustment, and a coherence gate adjustment of OCT. This is a tomographic moving image of the fundus of the eye for (adjustment of the optical path length difference between the measurement optical path length and the reference optical path length).

Further, the moving image to which the above-described trained model can be applied is not limited to a live moving image, but may be a moving image stored (saved) in a storage unit, for example. At this time, for example, a moving image obtained by aligning at least one frame of the tomographic moving image of the fundus oculi stored (saved) in the storage unit may be displayed on the display screen. For example, if you want to observe the vitreous body in a suitable manner, select a reference frame based on conditions such as as much vitreous body as possible on the frame, and align other frames with respect to the selected reference frame. A moving image may be displayed on the display screen.

(Modification 3)
In the various embodiments and modifications described above, when a trained model is undergoing additional learning, it may be difficult to output (inference/predict) using the trained model itself that is undergoing additional learning. For this reason, it is preferable to prohibit the input of medical images to the learned model that is undergoing additional learning. Further, another trained model that is the same as the trained model that is currently being additionally trained may be prepared as a preliminary trained model. At this time, during additional learning, it is preferable to allow medical images to be input to the preliminary trained model. Then, after the additional learning is completed, the trained model after the additional learning is evaluated, and if there is no problem, the preliminary trained model may be replaced with the trained model after the additional learning. Furthermore, if there is a problem, a preliminary trained model may be used.

Further, it may be possible to selectively use learned models obtained by learning for each part to be imaged. Specifically, a first trained model obtained using learning data including a first imaging region (lung, eye to be examined, etc.) and a learning model including a second imaging region different from the first imaging region. The second trained model obtained using the data may include selection means for selecting one of a plurality of trained models. At this time, in response to instructions from the operator, the imaged region corresponding to the selected learned model (header information or manually input by the operator) is paired with the imaged image of the region. Additional training is performed on the selected trained model by searching for data (for example, from a server in an external facility such as a hospital or research institute via a network), and using the data obtained through the search as learning data. and a control means that executes as a controller. Thereby, additional learning can be efficiently performed for each imaged body part using the photographed images of the body part corresponding to the trained model.

Additionally, when acquiring learning data for additional learning via a network from a server in an external facility such as a hospital or research institute, it is desirable to reduce reliability degradation due to tampering, system troubles, etc. during additional learning. Therefore, the validity of the learning data for additional learning may be detected by checking the consistency using a digital signature or hashing. Thereby, learning data for additional learning can be protected. At this time, if the validity of the learning data for additional learning cannot be detected as a result of checking the consistency using digital signatures and hashing, a warning to that effect will be issued and additional learning will be performed using that learning data. do not have.

(Modification 4)
In the various embodiments and modifications described above, the instructions from the examiner may be not only manual instructions (for example, instructions using a user interface or the like) but also voice instructions. At this time, for example, a machine learning engine including a speech recognition engine obtained by machine learning may be used. Further, the manual instruction may be an instruction by inputting characters using a keyboard, a touch panel, or the like. At this time, for example, a machine learning engine including a character recognition engine obtained by machine learning may be used. Further, the instructions from the examiner may be instructions by gesture. At this time, a machine learning engine including a gesture recognition engine obtained by machine learning may be used. Here, machine learning includes deep learning as described above, and for example, a recurrent neural network (RNN) can be used for at least one layer of a multilayer neural network.

(Modification 5)
In the various embodiments and modifications described above, the object to be examined is not limited to the eye to be examined, but may be any predetermined part of the examinee. Further, the frontal image of the predetermined region of the subject may be any medical image. At this time, the medical image to be processed is an image of a predetermined region of the subject, and the image of the predetermined region includes at least a part of the predetermined region of the subject. Further, the medical image may include other parts of the subject. Further, the medical image may be a still image or a moving image, and may be a black and white image or a color image. Furthermore, the medical image may be an image representing the structure (morphology) of a predetermined region, or an image representing its function. Images representing functions include, for example, images representing blood flow dynamics (blood flow volume, blood flow velocity, etc.) such as OCTA images, Doppler OCT images, fMRI images, and ultrasound Doppler images. The predetermined parts of the subject may be determined depending on the object to be photographed, and may include human eyes (eyes to be examined), organs such as the brain, lungs, intestines, heart, pancreas, kidneys, and liver, head, chest, Includes arbitrary parts such as legs and arms.

Furthermore, the medical image may be a tomographic image of the subject or a frontal image. The frontal image is, for example, a frontal image of the fundus, a frontal image of the anterior segment of the eye, a fluorescently photographed fundus image, or at least a partial range in the depth direction of the object to be photographed regarding data acquired by OCT (three-dimensional OCT data). Contains En-Face images generated using the data. Note that the En-Face image is an OCTA En-Face image (motion contrast frontal image) generated using data of at least a part of the depth direction of the object to be imaged with respect to three-dimensional OCTA data (three-dimensional motion contrast data). image). Furthermore, three-dimensional OCT data and three-dimensional motion contrast data are examples of three-dimensional medical image data.

Further, the photographing device is a device for photographing images used for diagnosis. The imaging device is, for example, a device that obtains an image of a predetermined region of a subject by irradiating the subject with radiation such as light, X-rays, electromagnetic waves, or ultrasound, or a device that detects radiation emitted from the subject. includes a device for obtaining an image of a predetermined region. More specifically, the imaging device according to the following embodiments includes at least an X-ray imaging device, a CT device, an MRI device, a PET device, a SPECT device, an SLO device, an OCT device, an OCTA device, a fundus camera, and an endoscopy device. Including mirrors, etc.

Note that the OCT device may include a time domain OCT (TD-OCT) device and a Fourier domain OCT (FD-OCT) device. Further, the Fourier domain OCT device may include a spectral domain OCT (SD-OCT) device or a wavelength swept OCT (SS-OCT) device. Furthermore, the SLO device and OCT device may include a wavefront compensated SLO (AO-SLO) device using a wavefront adaptive optical system, a wavefront compensated OCT (AO-OCT) device, and the like.

[Fourth embodiment]
In order to robustly suppress brightness step artifacts that occur in the slow axis direction of a wide-angle tomographic image or a motion contrast image, the image processing device according to the present embodiment uses brightness statistical values calculated in different depth ranges to Distribution information of a blood vessel candidate region is generated based on a value normalized by a local representative value regarding the in-plane distribution of brightness statistics. In other words, the image processing apparatus according to the present embodiment uses distribution information obtained by comparing a plurality of distribution information corresponding to a plurality of depth ranges in a three-dimensional tomographic image, and distribution information obtained by comparing a plurality of distribution information. By comparing the distribution information obtained by normalizing (for example, performing smoothing processing) with a local representative value, distribution information regarding a predetermined region (blood vessel candidate region) can be generated. Note that it is sufficient that the distribution information is finally generated, and there is no need to generate, for example, an image (map) as the distribution information during the generation. Next, the brightness value of the low-dimensional smoothed tomographic image or motion contrast image weighted for the blood vessel candidate region is divided by the brightness value of the high-dimensional smoothed tomographic image or motion contrast image. By doing so, a luminance correction coefficient value distribution is generated. Furthermore, by multiplying each pixel of a tomographic image or motion contrast image by a brightness correction coefficient value, the brightness step artifact that occurs in the slow axis direction of a wide-angle tomographic image or motion contrast image can be robustly suppressed. I will explain about it.

Here, an example of a brightness level difference that occurs in the slow axis direction of a wide-angle motion contrast image among the brightness level differences to be corrected in this embodiment will be described. If there is a positional shift between tomographic images scanned multiple times at the same location, a high motion contrast value will be calculated even for areas where red blood cells have not actually been displaced. A brightness level difference (white line) occurs. When acquiring a motion contrast image with a wide angle of view as in this embodiment, fixation tends to become poor as the imaging time elapses, so many brightness steps (white lines) may occur on the same motion contrast image. (Fig. 17(a) and Fig. 18(a)). In addition, poor fixation causes not only positional deviation between tomographic images but also blinking of the subject's eye and a drop in the position of the eyelashes. (black belt) may also be included.

On the other hand, information regarding the distribution of blood vessel candidate regions is generated for the wide-angle motion contrast image using the procedure described in S303 of the first embodiment, and brightness step correction is performed using the procedures described in S1004 to S1006 of the second embodiment. When generating a motion contrast image, there is a problem in that the brightness attenuation rate is easily influenced by the nerve fiber layer thickness. When the angle of view of a tomographic image or a motion contrast image is small, the difference in nerve fiber layer thickness from region to region is small, and the influence on the brightness attenuation rate is small. However, when a tomographic image or a motion contrast image has a wide angle of view, the difference in nerve fiber layer thickness from region to region is large, and the influence on the brightness attenuation rate cannot be ignored. For example, as shown in Figure 18(b), regardless of the presence or absence of blood vessels, the brightness attenuation rate tends to be large near the optic disc (white arrow) (where the nerve fiber layer thickness is large); ) The brightness attenuation rate tends to be small in the peripheral area (gray arrow area). Therefore, if brightness level difference correction is performed based on information regarding the distribution of blood vessel candidate regions, a high brightness level difference (white line) may remain near the optic disc (white arrow in Fig. 18(f)). be. Further, in the peripheral region, the motion contrast value of a blood vessel region running in the fast axis direction may be excessively suppressed (as shown by the gray arrow in FIG. 18(f)).

Therefore, in this embodiment, after calculating the brightness attenuation rate at each A-scan position (any value indicating the degree of difference may be used, in this embodiment, the difference value is used) at each A-scan position in steps S501 to S505. A representative value (for example, an average value or a median value) of the brightness attenuation rate in a nearby area in the in-plane direction centering on the position is calculated. In this embodiment, by normalizing using the representative value at each A-scan position, the brightness attenuation rate is prevented from being influenced by the nerve fiber layer thickness.

The configuration of an image processing system 10 including an image processing device 101 according to this embodiment and the image processing flow in this embodiment are the same as those in the second embodiment, so a description thereof will be omitted. Note that in FIG. 10, the steps other than S1003 and S1005 are the same as in the second embodiment, so the explanation will be omitted.

<Step 1003>
The blood vessel acquisition unit 101-421 generates information regarding the distribution of blood vessel candidate regions based on the results of comparing brightness statistics between different predetermined depth ranges. This embodiment is based on the degree of difference (difference or ratio) in brightness between "the depth range where blood vessels are likely to exist (surface layer of the retina)" and "the depth range where the decrease in brightness due to shadows is most noticeable (outer layer of the retina)". to identify blood vessel candidate regions.

In order to avoid the brightness attenuation rate being affected by the nerve fiber layer thickness, in this embodiment, after calculating the brightness attenuation rate at each A-scan position in steps S501 to S505, A representative value (local average value) of the brightness attenuation rate in a region near the direction is calculated. By dividing the brightness attenuation rate calculated at each A-scan position by the local average value and normalizing it, information regarding the distribution of blood vessel candidate regions can be robustly generated even for a wide-angle tomographic image. Note that the normalization of the brightness attenuation rate in the present invention is not limited to the method based on the local representative value of the brightness attenuation rate; for example, the brightness attenuation rate calculated at each A-scan position is calculated based on the nerve fiber layer thickness or nerve fiber layer It may be normalized by dividing by a local representative value of thickness (local average value, etc.). That is, the image processing device according to the present embodiment uses distribution information obtained by comparing a plurality of distribution information corresponding to a plurality of depth ranges in a three-dimensional tomographic image, and distribution information regarding the layer thickness of a predetermined layer of the eye to be examined. Distribution information regarding a predetermined region (blood vessel candidate region) may be generated by comparing.

Furthermore, details of the process executed in S1003 will be described with reference to the flowchart shown in FIG. 5(a). Note that steps S501 to S504 are the same as in the first embodiment and the second embodiment, and will therefore be omitted.

<Step 505>
The blood vessel acquisition unit 101-421, which is an example of an information generation unit, generates two types of frontal tomographic images (FIGS. 16(a) and 16(b) calculated in S504 in order to compare luminance statistical values in different depth ranges. ) is used to calculate the distribution of the brightness attenuation rate Ar. In this embodiment, as an index for comparing brightness statistical values in different depth ranges, (brightness of the superficial tomographic image of the retina) - (brightness of the front tomographic image of the outer retina) is calculated for each pixel (x, y), and the brightness A map (FIG. 16(c)) of the attenuation rate Ar(x,y) is generated.

<Step 506>
The blood vessel acquisition unit 101-421 normalizes the brightness attenuation rate map Ar(x,y) generated in S505 to generate a blood vessel candidate region map V(x,y) representing the likelihood of a blood vessel region.

In this embodiment, at each pixel position (x,y) of the brightness attenuation rate map Ar(x,y) calculated in S505, a representative value regarding the brightness attenuation rate is calculated in a neighboring area of a predetermined size. In this embodiment, a local average is calculated as the representative value. An example of the calculated local average value distribution is shown in FIG. 16(d). The representative value is not limited to this, and any known representative value (for example, median value) may be calculated. Next, normalization processing using the representative value is performed on the value of the brightness attenuation rate Ar (x, y) at each pixel position (x, y). In this embodiment, subtraction processing is performed as normalization processing. The present invention is not limited to this, and for example, when calculating (retina surface layer luminance representative value ÷ retinal outer layer luminance representative value) as the luminance attenuation rate, division processing is performed as normalization processing using the representative value in this step. You may go. Furthermore, as mentioned above, the normalization of the brightness attenuation rate in the present invention is not limited to the method based on the local representative value of the brightness attenuation rate, but for example, the brightness attenuation rate calculated at each A-scan position is used to determine the thickness of the nerve fiber layer. Alternatively, it may be normalized by dividing by a local representative value (local average value, etc.) of the nerve fiber layer thickness.

Furthermore, by normalizing the normalized brightness attenuation rate map Ar(x, y) using predetermined values WL and WW, the blood vessel candidate region map V(x, y) becomes V(x, y )=(Ar(x,y)-WL)/WW
It is calculated as follows, and 0≦V(x,y)≦1 is satisfied. FIG. 16(e) shows an example of the blood vessel candidate region map V(x,y). It can be seen that the blood vessel candidate region is stably depicted regardless of the location. Note that the normalization process is not limited to the above method, and any known normalization method may be used.

<Step 1005>
The weighting unit 101-442 uses the information regarding the distribution of blood vessel candidate regions (FIG. 17(d)) generated by the blood vessel acquiring unit 101-421 in S1003 to determine the blood vessels in the wide-angle motion contrast image (FIG. 17(a)). A weighted motion contrast image (FIG. 17(e)) is generated by weighting the brightness values in the candidate area. Next, the high-dimensional transformation unit 101-4411 generates a high-dimensional smoothed motion contrast image (FIG. 17(c)), and the low-dimensional transformation unit 101-4412 transforms the weighted motion contrast image in the fast axis direction. A low-dimensional smoothed motion contrast image (FIG. 17(f)) that has been subjected to smoothing processing is generated. Furthermore, the calculation unit 101-443 generates a brightness correction coefficient map for the motion contrast image (FIG. 17(g)) by performing calculation processing on the high-dimensional smoothed motion contrast image and the low-dimensional smoothed motion contrast image.

Furthermore, details of the process executed in S1005 will be described with reference to the flowchart shown in FIG. 5(b). Note that the steps other than S514 are the same as in the second embodiment, and will therefore be omitted.

<Step 514>
The weighting unit 101-442 weights the luminance value in the blood vessel candidate region of the motion contrast image using the value of the blood vessel candidate region map V(x,y). Note that this weighting exists along the fast axis direction of the measurement light used when acquiring the three-dimensional motion contrast image when acquiring the low-dimensional approximate value distribution, which is an example of the second approximate value distribution. This is an example of different calculation processes executed for a predetermined tissue that is a blood vessel or a bleeding area, and for other areas. Further, this weighting is not essential in the present invention.

In the present embodiment, the higher the value of the blood vessel candidate region map V (x, y) (likelihood of a blood vessel) is, the brightness value of the region corresponding to the higher region in the frontal motion contrast image acquired in S511 is calculated in S512. The brightness value M(x,y) of the frontal motion contrast image obtained in S511 is A weighted frontal motion contrast image M_w(x,y) is generated by weighting the frontal motion contrast image so as to maintain it as much as possible. Specifically, it may be calculated as follows.
M_w(x,y)=(1.0-V(x,y))*M(x,y)+V(x,y)*M_2ds(x,y)

FIG. 17(e) shows an example of the weighted frontal motion contrast image M_w(x,y).

The weighting method for the brightness value of the blood vessel candidate region shown here is just an example, and if the process is to reduce the brightness value of the blood vessel candidate region running in the fast axis direction or to bring it closer to the brightness value in the vicinity of the blood vessel candidate region, Any weighting may be applied.

Note that the brightness level difference that occurs in wide-angle tomographic images has a larger variation in depth (height) than the brightness level difference that occurs in wide-angle motion contrast images, so the brightness level difference that occurs in wide-angle tomographic images is suppressed. In this case, it is desirable to lower the weight on M_2ds (larger the weight on M(x, y)) (than when suppressing the brightness level difference that occurs in a wide-angle motion contrast image). Alternatively, a blood vessel candidate region map Vt (x, y) for a wide-angle tomographic image and a blood vessel candidate region map Vm (x, y) for a wide-angle motion contrast image are generated separately, and Vt (x, y) is generated separately. y)<Vm(x, y).

Next, the effect of luminance weighting on the blood vessel candidate region calculated by the blood vessel candidate region map to which the brightness attenuation rate normalization processing is applied using FIGS. 18(a) to 18(g) (in the wide-angle image, This section explains how to make it easier to selectively suppress only the luminance step artifact without being affected by the fiber layer thickness. FIG. 18A is an example of a motion contrast image that includes both a large number of band-shaped brightness steps (white lines) and a blood vessel region running in the fast axis direction. It is necessary to stably and selectively suppress only the brightness step, both in areas where the nerve fiber layer thickness is large (near the optic disc) and in small areas (periphery of a wide-angle image).

Blood vessel candidate region map generation processing in S1003, weighted image generation processing for the blood vessel region in S514, and luminance level difference correction in S1006 based on the blood vessel candidate region map generated without performing normalization processing using the local average value of brightness attenuation rate. Examples of processing results obtained when each process is executed are shown in FIGS. 18(b), 18(d), and 18(f). In the region with a large nerve fiber layer thickness (near the optic disc (white arrow)) in FIG. 18(b), the brightness attenuation rate is calculated to be high even in non-vascular regions, while in the region with a small nerve fiber layer thickness (vessel candidate region) In the periphery of the map (gray arrow)), the brightness attenuation rate is calculated to be too low even though the area is a blood vessel area. Therefore, in the vicinity of the optic nerve head (white arrow) in FIG. 18(d), the difference between blood vessel and non-vascular regions tends to be small, and the blood vessel region remains high brightness in the periphery of the image (gray arrow). As a result, the luminance level difference is insufficiently suppressed in the vicinity of the optic disc (white arrow) in FIG. 18(f), and the blood vessel region is oversuppressed in the image periphery (gray arrow).

On the other hand, in the case where the blood vessel candidate region map generation process in S1003, the weighted image generation process for the blood vessel area in S514, and the brightness level difference correction process in S1006 are executed after performing the normalization process using the local average value of the brightness attenuation rate, The processing results are shown in FIGS. 18(c), 18(e), and 18(g). In FIG. 18(c), blood vessel regions are stably depicted both in regions where the nerve fiber layer thickness is large (near the optic disc) and in regions where the nerve fiber layer thickness is small (periphery of the image). Therefore, appropriate weighting is performed on the blood vessel region in FIG. 18(e), and in the brightness level difference correction process in S1006, there is no insufficient suppression of the brightness level difference near the optic nerve head or excessive suppression of the brightness value of the blood vessel area in the peripheral area of the image. Not visible (Figure 18(g)).

Note that in this embodiment, a method for suppressing band-shaped brightness steps that occur on a wide-angle front motion contrast image (generating a normal wide-angle front motion contrast image with brightness steps corrected) has been described, but the present invention is not limited to this. The band-shaped brightness level difference that occurs on the wide-angle three-dimensional motion contrast image may be suppressed by the following procedure, and a wide-angle three-dimensional motion contrast image with the brightness level difference corrected may be generated.

That is, frontal motion contrast images are generated in a large number of different projection depth ranges, and a brightness level difference correction coefficient map is generated for each frontal motion contrast image. Next, the brightness level difference is corrected by calculating the value of the brightness level difference correction coefficient map (correction coefficient value) corresponding to the projection depth range to which each pixel belongs to the brightness value of each pixel on the three-dimensional motion contrast image. 3D motion contrast images can be generated. The different projection depth ranges include, for example, four types: the retinal surface layer, the deep retinal layer, the outer retinal layer, and the choroid. Alternatively, the type of each layer belonging to the retina and choroid may be specified.

In addition, the timing for correcting the brightness level difference is such that a wide-angle three-dimensional tomographic image or a motion contrast image that has been corrected for the brightness level difference is generated in advance using the procedure described above, and then the operator gives an instruction to generate a front tomographic image or a motion contrast image. The frontal image may be generated and displayed at the moment the front image is detected. Examples of the timing of generating a wide-angle three-dimensional tomographic image or a motion contrast image with brightness level difference correction include, for example, immediately after taking a tomographic image, at the time of reconstruction, and at the time of storage. Alternatively, when the operator instructs the generation of a wide-angle front tomographic image or a motion contrast image, the brightness level difference correction is performed (for the projection depth range specified by the instruction), and the brightness level difference corrected wide A field-of-view front tomographic image or a motion contrast image may be displayed.

Note that in this embodiment, the brightness level difference suppression process for a wide-angle motion contrast image has been described as an example of a wide-angle image, but the present invention is not limited thereto. That is, by performing the image processing described in S304 to S305 of the first embodiment using the blood vessel candidate region map generated in the procedure described in S1003 of this embodiment, the brightness generated in the wide-angle tomographic image is The present invention also includes robustly suppressing steps (both in areas where the nerve fiber layer thickness is high and in areas where the nerve fiber layer thickness is low).

According to the configuration described above, in order to robustly suppress the brightness step artifact that occurs in the slow axis direction of a wide-angle tomographic image or a motion contrast image, the brightness statistics calculated in different depth ranges are Distribution information of blood vessel candidates is generated based on values normalized by local representative values regarding the in-plane distribution of values. Next, the brightness value of the low-dimensional smoothed tomographic image or motion contrast image weighted for the blood vessel candidate region is divided by the brightness value of the high-dimensional smoothed tomographic image or motion contrast image. By doing so, a luminance correction coefficient value distribution is generated. Furthermore, by multiplying each pixel of a tomographic image or a motion contrast image by a brightness correction coefficient value, brightness step artifacts occurring in the slow axis direction of a wide-angle tomographic image or a motion contrast image are robustly suppressed.

Thereby, it is possible to robustly suppress the brightness level difference that occurs in the slow axis direction of a wide-angle tomographic image or a motion contrast image of the eye to be examined.

[Other embodiments]
In each of the embodiments described above, the present invention is implemented as the image processing device 101, but the embodiments of the present invention are not limited to the image processing device 101 only. For example, the present invention can be implemented as a system, device, method, program, storage medium, or the like. Further, the present invention can also be realized by executing the following processing. That is, software (programs) that realize the functions of the various embodiments and modifications described above are supplied to a system or device via a network or various storage media, and the computer (or CPU, MPU, etc.) of the system or device is is the process of reading and executing a program.

Claims (17)

an acquisition means for acquiring a plurality of frontal images corresponding to different depth ranges of the eye to be examined;
Generating means for generating a plurality of frontal images in which artifacts in the plurality of acquired frontal images are reduced;
a determination unit that uses a learned model obtained by learning a plurality of frontal images to determine a state of an artifact in at least one frontal image among the plurality of acquired frontal images;
Displaying at least one front image of the plurality of front images acquired and a determination result regarding the state of the artifact on a shooting confirmation screen, and accepting an instruction from an operator as a designation regarding whether or not reduction of the artifact is necessary. , the at least one frontal image displayed on the photographing confirmation screen is switched to at least one frontal image of the plurality of generated frontal images, and the display screen on the display means changes from the photographing confirmation screen to the report. Display control means for controlling the display means so as to display the plurality of generated front images side by side on the report screen after switching to the screen;
An image processing device comprising: an acquisition means for acquiring a plurality of frontal images corresponding to different depth ranges of the eye to be examined;
Generating means for generating a plurality of frontal images in which artifacts in the plurality of acquired frontal images are reduced;
a determination unit that uses a learned model obtained by learning a plurality of frontal images to determine a state of an artifact in at least one frontal image among the plurality of acquired frontal images;
Displaying at least one front image of the plurality of acquired front images and a determination result regarding the state of the artifact on a first display screen, and receiving an instruction from an operator as a designation regarding the necessity of artifact reduction. In response to the reception, at least one front image displayed on the first display screen is switched to at least one front image among the plurality of generated front images, and the display screen on the display means is switched to the first front image. Display control means for controlling the display means so as to display the plurality of generated front images side by side on the second display screen after switching from the first display screen to the second display screen;
An image processing device comprising: The display control means moves at least one front image of the plurality of front images displayed side by side on the second display screen to another depth range of the different depth ranges in response to an instruction from an operator. The image processing device according to claim 2, wherein the display means is controlled to switch to a corresponding front image. The display control means may display a plurality of front faces displayed side by side on the second display screen in response to an instruction from the operator as a designation regarding the necessity of artifact reduction . The image processing apparatus according to claim 2 or 3, wherein the display means is controlled to switch at least one front image of the images to at least one front image of the plurality of acquired front images. The determining means is configured to classify the acquired frontal image into one of a plurality of stages depending on the degree of the artifact, or by classifying the front image into one of a plurality of types of the artifact. The image processing device according to any one of claims 1 to 4 , which determines the state of an artifact. The learned model is obtained by learning using learning data that is a set of a plurality of frontal images of different types, or is obtained by learning using learning data that is a set of frontal images and analysis results. 6. The image processing device according to any one of 1 to 5 . A first approximate value distribution obtained by performing two-dimensional smoothing processing on at least one frontal image based on a three-dimensional tomographic image or a three-dimensional motion contrast image of the eye to be examined , At least a portion of the three-dimensional tomographic image or the three-dimensional motion contrast image is calculated using the distribution of correction coefficient values obtained by dividing by the second approximate value distribution obtained by performing dimensional smoothing processing. further comprising a correction means for correcting the
The image processing according to any one of claims 1 to 6 , wherein the generating means generates at least one frontal image based on at least a portion of the corrected three-dimensional tomographic image or the three-dimensional motion contrast image. Device. The generating means generates at least one frontal image based on at least a portion of the corrected three-dimensional tomographic image or the three-dimensional motion contrast image,
The image processing device according to claim 7 , wherein the front image is either a front tomographic image generated from the three-dimensional tomographic image or a front motion contrast image generated from the three-dimensional motion contrast image. The acquisition means acquires a distribution of correction coefficient values for each of a plurality of depth ranges specified based on a predetermined layer boundary of the eye to be examined;
8. The correction means corrects at least a portion of the three-dimensional tomographic image or the three-dimensional motion contrast image using a distribution of correction coefficient values obtained for each of the plurality of depth ranges. 8. The image processing device according to 8 . The generating means generates the three-dimensional information by comparing a plurality of distribution information in an in-plane direction intersecting the depth direction of the eye to be examined, which corresponds to a plurality of depth ranges in the three-dimensional tomographic image. Generating distribution information regarding a predetermined region that is a blood vessel-related region or a bleeding region existing along the fast axis direction of the measurement light used when acquiring the tomographic image or the three-dimensional motion contrast image,
Any one of claims 7 to 9 , wherein the acquisition means acquires the second approximate value distribution by different calculation processes for the predetermined area and other areas, based on the generated distribution information. The image processing device according to item 1. The generating means compares distribution information obtained by comparing the plurality of distribution information and distribution information obtained by normalizing the distribution information obtained by comparing the plurality of distribution information with a local representative value. or by comparing distribution information obtained by comparing the plurality of pieces of distribution information with distribution information regarding the layer thickness of a predetermined layer of the eye to be examined. The image processing device according to item 10 . The plurality of pieces of distribution information are two average values obtained by averaging brightness values calculated in two different depth ranges in the three-dimensional tomographic image in the depth direction,
The image processing apparatus according to claim 10 , wherein the generating means generates distribution information regarding the predetermined area by comparing the two average values. The image processing apparatus according to any one of claims 10 to 12 , wherein the distribution information regarding the predetermined region is distribution information of a region regarding blood vessels of the eye to be examined. acquiring a plurality of frontal images corresponding to different depth ranges of the eye to be examined ;
generating a plurality of frontal images in which artifacts in the plurality of acquired frontal images are reduced;
using a learned model obtained by learning a plurality of frontal images to determine the state of an artifact in at least one frontal image of the plurality of acquired frontal images;
Displaying at least one front image of the plurality of front images acquired and a determination result regarding the state of the artifact on a shooting confirmation screen, and accepting an instruction from an operator as a designation regarding whether or not reduction of the artifact is necessary. , the at least one frontal image displayed on the photographing confirmation screen is switched to at least one frontal image of the plurality of generated frontal images, and the display screen on the display means changes from the photographing confirmation screen to the report. controlling the display means so as to display the plurality of generated front images side by side on the report screen after switching to the screen;
image processing methods including; acquiring a plurality of frontal images corresponding to different depth ranges of the eye to be examined ;
generating a plurality of frontal images in which artifacts in the plurality of acquired frontal images are reduced;
using a learned model obtained by learning a plurality of frontal images to determine the state of an artifact in at least one frontal image of the plurality of acquired frontal images;
Displaying at least one front image of the plurality of acquired front images and a determination result regarding the state of the artifact on a first display screen, and receiving an instruction from an operator as a designation regarding the necessity of artifact reduction. In response to the reception, at least one front image displayed on the first display screen is switched to at least one front image among the plurality of generated front images, and the display screen on the display means is switched to the first front image. controlling the display means so as to display the plurality of generated front images side by side on the second display screen after switching from the first display screen to the second display screen;
image processing methods including; A program that causes a computer to execute the image processing method according to claim 14 or 15 . a photographing device for photographing the eye to be examined;
An image processing system comprising: the image processing device according to any one of claims 1 to 13, communicatively connected to the photographing device.

Images