WO2020050308A1

WO2020050308A1 - Image processing device, image processing method and program

Info

Publication number: WO2020050308A1
Application number: PCT/JP2019/034752
Authority: WO
Inventors: 裕之今村; 秀謙溝部
Original assignee: キヤノン株式会社
Priority date: 2018-09-06
Filing date: 2019-09-04
Publication date: 2020-03-12
Also published as: JP2020039430A

Abstract

This image processing device, for correcting a shadow region in a tomographic image regardless of the ailment or the site, comprises an image acquisition means for acquiring a tomographic image of an eye under examination, and a correction means using a trained model for correcting an image value for the shadow region in the tomographic image, which is a shadow region arising due to an object contained in the eye under examination.

Description

Image processing apparatus, image processing method, and program

The present invention relates to an image processing device, an image processing method, and a program.

With the use of a tomographic imaging apparatus for the eye such as an optical coherence tomograph (optical coherence tomography: hereinafter referred to as OCT), a state inside the retinal layer can be observed three-dimensionally. The tomographic imaging apparatus is widely used for ophthalmic medical treatment because it is useful for more accurately diagnosing a disease.

As a form of the OCT, for example, there is a TD-OCT (Time domain OCT) that combines a broadband light source and a Michelson interferometer. This is configured to move the position of the reference mirror at a constant speed, measure interference light with backscattered light acquired by the signal arm, and obtain a reflected light intensity distribution in the depth direction. However, such TD-OCT requires mechanical scanning, so that high-speed image acquisition is difficult. Therefore, SD-OCT (Spectral domain OCT) and SS-OCT (Swept source OCT) have been developed as faster image acquisition methods. In SD-OCT, a broadband light source is used, and an interference signal is acquired by a spectroscope. In the SS-OCT, the light is temporally separated by using a high-speed wavelength sweep light source. According to these OCT, a tomographic image having a wider angle of view and a higher depth can be obtained.

On the other hand, as shown in FIG. 4A, in the tomographic image 403, when the measurement light is strongly reflected or absorbed by an object such as the blood vessel 401, a shadow (shadow) region 402 where the signal is attenuated or lost occurs behind the object. Sometimes. FIG. 4A is a diagram exemplifying a tomographic image in which a shadow region has occurred. Here, the object is, for example, a tissue such as a blood vessel 401 or a lesion such as vitiligo or bleeding. Normally, the brightness becomes maximum near the photoreceptor inner / outer segment boundary (IS / OS) 4 or the retinal pigment epithelium 5 in the depth direction of the retina. However, when the shadow area 402 is generated below the blood vessel 401, the brightness near the IS / OS 4 or the retinal pigment epithelium 5 in the shadow area 402 is reduced or lost.

Ｏ Further, in order to grasp the pathological condition of the fundus blood vessels, an OCT Angiography (hereinafter referred to as OCTA) technique of non-invasively rendering the fundus blood vessels three-dimensionally using OCT is used. In OCTA, the same position is scanned a plurality of times with the measurement light, and the motion contrast obtained by the interaction between the displacement of the red blood cells and the measurement light is imaged.

走査 The scanning mode of the measuring light in this case will be described with reference to FIG. 4B. In the figure, the main scanning direction is the horizontal (x-axis) direction, and an interference signal is obtained at each position of x1, x2,..., Xm. The scanning of the measurement light in the horizontal direction is referred to as a B scan here. The figure shows an example of OCTA imaging in which B scanning is performed r times continuously at each position (yi; 1 ≦ i ≦ n) in the sub-scanning direction (y-axis direction).

In OCTA imaging, scanning the measurement light at the same position a plurality of times is called cluster scanning, and a plurality of tomographic images obtained at the same position is called a cluster. The motion contrast image is generated in cluster units. FIG. 4C shows an example in which a three-dimensional motion contrast image is superimposed and displayed on a three-dimensional OCT tomographic image. A high motion contrast, which is a projection artifact (hereinafter referred to as PA), occurs in a region 405 below the surface blood vessels of the retina. PA is a phenomenon in which the OCT signal in the shadow region 402 correspondingly increases or decreases as the OCT signal in the blood vessel 401 (retina superficial blood vessel) repeatedly increases and decreases, and high motion contrast occurs in the outer retina layer where blood vessels do not originally exist. Point. In FIG. 4C, a region 405 having a high motion contrast value is formed on the deep side of a region 404 having a high motion contrast value corresponding to a retinal surface blood vessel.

On the other hand, in the technology disclosed in Patent Document 1, a layer boundary is obtained from a tomographic image of an eye obtained by OCT, and a shadow region is detected based on image features (brightness values and layer shapes) at the layer boundary. ing. Then, the luminance correction or the layer detection parameter in the shadow area is changed by image processing.

JP 2010-279440 A

However, in the technique disclosed in Patent Document 1 that corrects a shadow area on an OCT tomographic image by image processing based on image features on a layer boundary, the propriety of shadow area correction depends on the propriety of layer detection. For this reason, depending on a part or disease, layer detection is difficult, and as a result, shadow area correction may be difficult.

The present invention has been made in view of the above problems, and has as its object to correct a shadow region in a tomographic image regardless of a disease or a site.

It is to be noted that the present invention is not limited to the above object, and it is another object of the present invention to provide an operation effect derived from each configuration shown in the embodiment for carrying out the invention described later, and to obtain an operation effect which cannot be obtained by the conventional technology. It can be positioned as one of the.

In order to solve the above problems, an image processing apparatus according to one embodiment of the present invention
Image acquisition means for acquiring a tomographic image of the eye to be examined,
And a correction unit configured to correct a pixel value of the shadow area in the tomographic image, which is a shadow area generated by an object included in the eye to be examined, using a learned model.

According to one aspect of the present invention, a shadow area in a tomographic image can be corrected regardless of a disease or a site.

FIG. 1 is a block diagram illustrating a configuration of an image processing device according to a first embodiment of the present invention. FIG. 1 is a schematic configuration diagram of an image processing system according to an embodiment of the present invention. FIG. 2B is a diagram illustrating a measurement optical system included in the tomographic image photographing apparatus included in the image processing system illustrated in FIG. 2A. 5 is a flowchart of a process that can be executed by the image processing system according to the first embodiment of the present invention. FIG. 3 is a diagram illustrating an OCT tomographic image including a shadow area. FIG. 4 is a diagram illustrating a method of scanning measurement light during OCTA imaging. FIG. 4 is a diagram for explaining PA occurring below a blood vessel in a motion contrast image. FIG. 9 is a diagram illustrating an example of a shadow area corrected OCT tomographic image. FIG. 4 is a diagram illustrating an example of a size of a tomographic image used when obtaining a learned model according to the embodiment of the present invention. FIG. 4 is a diagram illustrating an example of a size of a tomographic image used when obtaining a learned model according to the embodiment of the present invention. FIG. 4 is a diagram illustrating an example of a size of a tomographic image used when obtaining a learned model according to the embodiment of the present invention. FIG. 4 is a diagram illustrating an example of a size of a tomographic image used when obtaining a learned model according to the embodiment of the present invention. FIG. 4 is a diagram illustrating an example of a size of a tomographic image used when obtaining a learned model according to the embodiment of the present invention. FIG. 4 is a diagram illustrating an example of a size of a tomographic image used when obtaining a learned model according to the embodiment of the present invention. FIG. 4 is a diagram illustrating an example of a learning model according to the embodiment of the present invention. It is a figure explaining the report screen displayed on a display means in S305 in the process demonstrated as the first embodiment of the present invention. It is a block diagram showing the composition of the image processing device concerning a second embodiment of the present invention. 9 is a flowchart of a process that can be executed by the image processing system according to the second embodiment of the present invention. It is a figure explaining the OCT tomographic image before processing of S905 in processing explained as a second embodiment of the present invention. FIG. 9 is a diagram illustrating an image in which a motion contrast image generated in S905 is superimposed on a tomographic image. FIG. 9 is a diagram illustrating an example of a report screen displayed on a display unit in S906. It is a block diagram showing the composition of the image processing device concerning a third embodiment of the present invention. 13 is a flowchart of a process that can be executed by the image processing system according to the third embodiment of the present invention. It is a figure explaining the report screen displayed on a display means in S1206 in the processing explained as a third embodiment of the present invention. It is a block diagram showing the composition of the image processing device concerning a 4th embodiment of the present invention. 14 is a flowchart of a process that can be executed by the image processing system according to the fourth embodiment of the present invention. It is a figure explaining processing performed at S1508 in processing explained as a 4th embodiment of the present invention. FIG. 9 is a diagram for describing processing executed in S1509.

Hereinafter, exemplary embodiments for carrying out the present invention will be described in detail with reference to the drawings. However, dimensions, materials, shapes, relative positions of components and the like described in the following embodiments are arbitrary, and can be changed according to the configuration of the apparatus to which the present invention is applied or various conditions. Also, in the drawings, the same reference numerals are used between the drawings to indicate the same or functionally similar elements.

[First embodiment]
The image processing apparatus according to the present embodiment makes a machine learning model by deep learning learn pairs of tomographic images of various parts and diseases and tomographic images obtained by applying a shadow area correction process to the tomographic images. By inputting a tomographic image to the learned machine learning model, a shadow area on the tomographic image is robustly reduced.

In the following, a machine learning model refers to a learning model based on a machine learning algorithm such as deep learning. The learned model is a model obtained (trained) by previously training an appropriate machine learning model or the like using an arbitrary machine learning algorithm using appropriate teacher data. However, it is assumed that the learned model does not perform any further learning and can perform additional learning.

Hereinafter, an image processing system including an image processing apparatus according to the first embodiment of the present invention will be described with reference to the drawings.

FIG. 2A is a diagram illustrating a configuration of an image processing system 10 including the image processing apparatus 101 according to the present embodiment. As shown in FIG. 2A, in the image processing system 10, the image processing apparatus 101 is connected to a tomographic image capturing apparatus 100 (also referred to as OCT), an external storage unit 102, an input unit 103, and a display unit 104 via an interface. It is constituted by doing.

The tomographic image capturing apparatus 100 is an apparatus that captures a tomographic image of the eye. In the present embodiment, it is assumed that SD-OCT is used as the tomographic imaging apparatus 100. However, the mode of the tomographic imaging apparatus is not limited to this, and may be configured using, for example, SS-OCT.

The tomographic imaging apparatus 100 includes a measurement optical system 100-1, a stage unit 100-2, and a base unit 100-3. In FIG. 2A, a measurement optical system 100-1 is an optical system for acquiring an anterior eye image, an SLO fundus image of a subject's eye, and a tomographic image. The stage unit 100-2 supports the measurement optical system 100-1 so as to be movable in the front, rear, left, and right directions. The base unit 100-3 incorporates a spectroscope described later.

In the present embodiment, the image processing apparatus 101 is a computer that controls the stage unit 100-2, controls the alignment operation of the measurement optical system 100-1, and reconstructs a tomographic image. The external storage unit 102 stores a program for tomography, patient information, imaging data, measurement data, and the like. The input unit 103 issues an instruction to the computer, and specifically includes a keyboard and a mouse. The display unit 104 includes, for example, a monitor.

(Configuration of tomographic imaging device)
The configuration of the measuring optical system and the spectroscope in the tomographic imaging apparatus 100 of the present embodiment will be described with reference to FIG. 2B.

First, the inside of the measuring optical system 100-1 will be described. An objective lens 201 is installed to face the subject's eye 200, and a first dichroic mirror 202 and a second dichroic mirror 203 are arranged on the optical axis. The dichroic mirror divides the optical path to the eye 200 into the optical path 250 of the OCT optical system, the optical path 251 for the SLO optical system and the fixation lamp, and the optical path 252 for anterior eye observation for each wavelength band.

The SLO optical system and the optical path 251 for the fixation lamp include an SLO scanning unit 204 and

lenses

205 and 206. A mirror 207, a third dichroic mirror 208, an APD (Avalanche Photodiode) 209, an SLO light source 210, and a fixation lamp 211 are provided downstream of the lens 206. The mirror 207 is a prism on which a perforated mirror or a hollow mirror is deposited, and separates illumination light from the SLO light source 210 from light returned from the subject's eye. The third dichroic mirror 208 separates the optical path 251 into an optical path of the SLO light source 210 and an optical path of the fixation lamp 211 for each wavelength band.

The SLO scanning unit 204 scans the illumination light emitted from the SLO light source 210 on the subject's eye 200, and includes an X scanner that scans in the X direction and a Y scanner that scans in the Y direction. In the present embodiment, since the X scanner needs to perform high-speed scanning, the X scanner is constituted by a polygon mirror, and the Y scanner is constituted by a galvanometer mirror. However, these mirrors can be appropriately replaced by various known deflection mirrors according to the required specifications.

The lens 205 is driven by a motor (not shown) for focusing the SLO optical system and the fixation lamp 211. The SLO light source 210 generates light having a wavelength near 780 nm as illumination light. The APD 209 detects the return light from the subject's eye. The fixation lamp 211 generates visible light to urge the subject to fixate.

The illumination light emitted from the SLO light source 210 is reflected by the third dichroic mirror 208 and passes through the mirror 207. Thereafter, the light passes through the

lenses

206 and 205 and is scanned on the eye 200 by the SLO scanning means 204. The return light from the subject's eye 200 returns along the same path as the illumination light, is reflected by the mirror 207, and is guided to the APD 209, whereby an SLO fundus image is obtained.

光 The light emitted from the fixation lamp 211 passes through the third dichroic mirror 208 and the mirror 207. Thereafter, a predetermined shape is formed at an arbitrary position on the subject's eye 200 by the SLO scanning means 204 through the

lenses

206 and 205, and the subject's fixation is promoted.

In the optical path 252 for anterior eye observation,

lenses

212 and 213, a split prism 214, and an anterior eye part observation CCD 215 for detecting infrared light are arranged. The CCD 215 has sensitivity at the wavelength of irradiation light (not shown) for anterior ocular segment observation, specifically around 970 nm. The split prism 214 is arranged at a position conjugate with the pupil of the eye 200 to be inspected. The distance in the Z-axis direction (optical axis direction) of the measurement optical system 100-1 from the subject's eye 200 can be detected from the split image of the anterior segment obtained through the split prism 214.

The optical path 250 of the OCT optical system constitutes the OCT optical system as described above, and captures a tomographic image of the eye 200 to be inspected. More specifically, an interference signal for forming a tomographic image is obtained. The OCTXY scanner 216 scans the measurement light on the eye 200 to be inspected. Although shown as a single mirror in FIG. 2B, it is actually composed of two galvanometer mirrors that perform scanning in the XY biaxial directions. Note that the configuration of the OCTXY scanner 216 is not limited to this, and may be configured using any other deflecting mirror.

Among the

lenses

217 and 218, the lens 217 is used to focus light from the OCT light source 220 emitted from the fiber 224 connected to the optical coupler 219 to the eye 200. Specifically, it is driven in the optical axis direction indicated by an arrow in the figure by a motor (not shown). By this focusing, the return light from the eye 200 to be inspected is simultaneously focused on the tip of the fiber 224 in the form of a spot and is incident.

Next, the configuration of the optical path from the OCT light source 220, the reference optical system, and the spectroscope will be described. These components include an OCT light source 220, a reference mirror 221, a dispersion compensation glass 222, a lens 223, an optical coupler 219, optical fibers 224 to 227, and a spectroscope 230. The optical fibers 224 to 227 are single mode optical fibers connected to and integrated with an optical coupler.

では In the present embodiment, the Michelson interferometer is configured by these configurations. The light emitted from the OCT light source 220 is split through the optical fiber 225 into measurement light guided to the optical fiber 224 via the optical coupler 219 and reference light guided to the optical fiber 226. The measurement light is applied to the subject's eye 200 to be observed through the above-described OCT optical system optical path, and reaches the optical coupler 219 via the same optical path due to reflection and scattering by the subject's eye 200.

On the other hand, the reference light reaches the reference mirror 221 via the optical fiber 226, the lens 223, and the dispersion compensation glass 222 inserted for adjusting the wavelength dispersion of the measurement light and the reference light, and is reflected. Then, the light returns to the same optical path and reaches the optical coupler 219.

測定 The measuring light and the reference light are multiplexed by the optical coupler 219 to become interference light. Here, interference occurs when the optical path length of the measurement light and the optical path length of the reference light become substantially the same. The reference mirror 221 is held by a motor and a driving mechanism (not shown) so as to be adjustable in the optical axis direction indicated by the arrow in the figure, and can adjust the optical path length of the reference light to the optical path length of the measurement light. The obtained interference light is guided to the spectroscope 230 via the optical fiber 227.

In the present embodiment, the

polarization adjusting units

228 and 229 are provided in the

optical fibers

224 and 226, respectively, and perform polarization adjustment. These

polarization adjusting sections

228 and 229 have several portions where optical fibers are looped. By rotating the loop portion about the longitudinal direction of the optical fiber, the optical fiber is twisted, and the polarization states of the measurement light and the reference light can be adjusted and matched.

The spectroscope 230 includes

lenses

232 and 234, a diffraction grating 233, and a line sensor 231. The interference light emitted from the optical fiber 227 is converted into parallel light through the lens 234, is then separated by the diffraction grating 233, and is imaged on the line sensor 231 by the lens 232.

Next, the periphery of the OCT light source 220 will be described. The OCT light source 220 is an SLD (Super Luminescent Diode) that is a typical low coherent light source. The center wavelength is 855 nm and the wavelength bandwidth is about 100 nm. Here, the bandwidth is an important parameter because it affects the resolution of the obtained tomographic image in the optical axis direction.

In the present embodiment, the SLD is selected as the light source, but it is sufficient that low coherent light can be emitted, and ASE (Amplified Spontaneous Emission) or the like can be used. As the center wavelength of the light used, near-infrared light is suitable in view of measuring the eye. Since the center wavelength affects the resolution of the obtained tomographic image in the horizontal direction, it is desirable that the center wavelength be as short as possible. For both reasons, the center wavelength is 855 nm in this embodiment.

In the present embodiment, a Michelson interferometer is used as an interferometer, but a Mach-Zehnder interferometer may be used. According to the light amount difference between the measurement light and the reference light, it is desirable to use a Mach-Zehnder interferometer when the light amount difference is large and to use a Michelson interferometer when the light amount difference is relatively small.

(Configuration of image processing device)
Next, the configuration of the image processing apparatus 101 according to the present embodiment will be described with reference to FIG. In the present embodiment, the image processing apparatus 101 is configured as a personal computer (PC) connected to the tomographic image capturing apparatus 100. The image processing apparatus 101 includes an image acquisition unit 101-01, a storage unit 101-02, a shooting control unit 101-03, an image processing unit 101-04, and a display control unit 101-05.

In the image processing apparatus 101, the arithmetic processing unit CPU executes software modules that implement the image acquisition unit 101-01, the imaging control unit 101-03, the image processing unit 101-04, and the display control unit 101-05. Implement the function. Note that the present invention is not limited to this. For example, the image processing unit 101-04 may be realized by dedicated hardware such as an ASIC, or the display control unit 101-05 may be realized by a dedicated processor such as a GPU different from a CPU. May be implemented. Further, the storage unit 101-2 may be configured by an arbitrary storage medium such as an arbitrary memory or an optical disk. Further, the connection between the tomographic imaging apparatus 100 and the image processing apparatus 101 may be configured via a network.

The image acquisition unit 101-01 acquires signal data of an SLO fundus image or a tomographic image captured by the tomographic image capturing apparatus 100. The image acquisition unit 101-01 has a tomographic image generation unit 101-11. The tomographic image generation unit 101-11 acquires signal data (interference signal) of the tomographic image captured by the tomographic image capturing apparatus 100, generates a tomographic image by signal processing, and stores the generated tomographic image in the storage unit 101-02. To be stored.

(4) The imaging control unit 101-03 controls imaging of the tomographic imaging apparatus 100. The imaging control includes instructing the tomographic imaging apparatus 100 regarding setting of imaging parameters, and instructing the start or end of imaging.

The image processing unit 101-04 includes a photographing condition acquisition unit 101-41, a positioning unit 101-42, a correction unit 101-43, an image feature acquisition unit 101-44, and a projection unit 101-45. The above-described image acquisition unit 101-01 is an example of an acquisition unit according to the present invention. The photographing condition acquisition unit 101-41 acquires photographing condition data of an input image required when the image processing unit 101-04 performs image processing. The photographing condition data includes, for example, photographing date and time, part name, angle of view, scan mode, image resolution and number of gradations, pixel size, information on image data format, and the like.

The correction unit 101-43 performs a process of two-dimensionally or three-dimensionally suppressing a shadow region generated below an object such as a blood vessel in a tomographic image using the learned model. The image feature acquisition unit 101-44 acquires a layer boundary of the retina or choroid, a boundary of the lamina cribrosa region, the position of the fovea or the center of the optic disc, and the like from the tomographic image. The projection unit 101-45 projects a tomographic image in a depth range based on the boundary position acquired by the image feature acquiring unit 101-44, and generates a front tomographic image such as an Enface image.

The external storage unit 102 stores information on the subject's eye (eg, patient's name, age, and gender), captured images (tomographic images and SLO images), images obtained by processing the images, imaging parameters, and operators. Is stored in association with the parameter set by. The input unit 103 is, for example, a mouse, a keyboard, a touch operation screen, or the like. An operator issues an instruction to the image processing apparatus 101 or the tomographic image capturing apparatus 100 via the input unit 103.

Next, the processing procedure of the image processing apparatus 101 according to the present embodiment will be described with reference to FIG. FIG. 3 is a flowchart showing a flow of operation processing of the entire system in the present embodiment.

<Step 301 (S301)>
By operating the input unit 103, the operator sets imaging conditions when the OCT image is captured by the tomographic image capturing apparatus 100.
In particular,
The procedure includes 1) selection of a scan mode, and 2) setting of shooting parameters corresponding to the scan mode. In the present embodiment, the shooting conditions are set as follows.
1) Select the Macula 3D scan mode 2) Set the following shooting parameters 2-1) Scanning area size: 12x12mm
2-2) Main scanning direction: horizontal direction 2-3) Fixation lamp position: lighting position for macular imaging

<Step 302 (S302)>
After setting the shooting conditions, the operator operates the input unit 103 and presses a shooting start button (not displayed) in the shooting screen. Accordingly, the tomographic image capturing apparatus 100 starts capturing an OCT tomographic image under the capturing conditions specified in S301. Specifically, the imaging control unit 101-03 instructs the tomographic imaging apparatus 100 to perform OCT imaging based on the settings instructed by the operator in S301. Thereby, the tomographic imaging apparatus 100 acquires an interference signal for generating a corresponding OCT tomographic image.

Further, the tomographic imaging apparatus 100 also acquires an SLO image, and executes a tracking process based on the SLO moving image. In the present embodiment, the number of times of repetitive imaging at the same scanning position is one (not repeated). However, the number of repetitive imagings at the same scanning position is not limited to this, and may be set to an arbitrary number. When the number of times of repetitive imaging is two or more, the reference SLO image used for tracking processing during imaging is the standard SLO image set at the time of the first OCT imaging, and a common reference SLO image is used for all repeated OCT imaging. During repeated shooting, the same set values are used (not changed) for 1) selection of left and right eyes and 2) execution of tracking processing in addition to the shooting conditions set in S301. However, the conditions for the tracking processing are not limited to the above, and can be changed as appropriate according to the conditions for capturing the OCT tomographic image.

<Step 303 (S303)>
Next, the image acquisition unit 101-01 and the image processing unit 101-04 reconstruct a tomographic image using the interference signal acquired in S302. First, the tomographic image generation unit 101-11 performs wave number conversion, fast Fourier transform (hereinafter, referred to as FFT), and absolute value conversion (acquisition of amplitude) on the interference signal acquired by the image acquisition unit 101-01, thereby obtaining a tomographic image. Generate In this embodiment, a tomographic image is generated from the interference signal. However, the tomographic image described here can also be handled as various data obtained by the above-described various conversions that are obtained when the image is generated. Therefore, the tomographic image described here can also be grasped as tomographic data including these various data. Next, the positioning unit 101-42 performs positioning between tomographic images obtained from each scanning position.

In the case where the number of times of repetitive imaging at the same scanning position is 2 or more in S301, the positioning unit 101-42 also performs alignment in a tomographic image captured within the same scanning position. Then, the tomographic images after the alignment are superimposed to generate a superimposed tomographic image. The positioning unit 101-42 performs this operation in addition to the above-described positioning between tomographic images between scanning positions.

{Circle around (4)} The image feature acquiring unit 101-44 acquires a layer boundary between the retina and the choroid and a boundary (not shown) between the anterior surface and the posterior surface of the cribriform plate from the single or superimposed tomographic images. In the present embodiment, the inner boundary membrane 1, the nerve fiber layer-ganglion cell layer boundary 2, the ganglion cell layer-inner plexiform layer boundary 3, the photoreceptor inner and outer segment junction 4, the retinal pigment epithelium 5, The Bruch's membrane 6 and the choroid-sclera boundary 7 are acquired (see FIG. 4A). In addition, the detected end of the Bruch's membrane 6 (the end of the Bruch's membrane opening) is specified as a Disc boundary of the optic papilla. In the present embodiment, the variable shape model is used as a method for acquiring the layer boundaries of the retina and the choroid, and the front and rear boundaries of the cribriform plate, but any known segmentation method may be used. The layer boundary to be obtained is not limited to the above example. For example, any known segmentation method can be used to determine the inner plexus-inner plexus boundary, the inner plexus-outer plexus boundary, the outer plexus-outer plexus boundary, outer boundary membrane, photoreceptor outer segment tip (COST), etc. May be obtained by Alternatively, the present invention includes the case where the choroid capillary plate-Sattler layer boundary and the Sattler layer-Haller layer boundary of the choroid are obtained by any known segmentation method. The front and rear boundaries of the sieve plate may be manually set. For example, it is preferable that a layer boundary position can be set by manually moving a position of a specific layer boundary (for example, the inner limiting membrane 1) by a predetermined amount. Note that the process of acquiring the layer boundary and the front / rear surface boundary of the cribriform plate may be performed after the image correction process of S304 described later instead of this step.

<Step 304>
Next, the correction unit 101-43 corrects a shadow region generated below an object in the eye such as a blood vessel using the learned model. As described above, the learned model in the description of the present invention is a model obtained by training in advance using an appropriate teacher data for an arbitrary known machine learning algorithm. In the present embodiment, a tomographic image having a high possibility of being obtained when a shadow region does not appear is generated from an image having a shadow region or data for generating the image using the learned model. Here, the machine learning algorithm will be described with reference to FIGS. 4A to 4D, FIGS. 5A to 5F and FIG.

The teacher data is composed of one or more pairs of input data and output data. Specifically, teacher data composed of pairs of tomographic images of various parts and diseases including a shadow area obtained by OCT (FIG. 4A) and corresponding shadow area corrected tomographic images (FIG. 4D) (Hereinafter, first teacher data).

The first teacher data is composed of a group of pairs of a tomographic image in which a shadow area has occurred and a tomographic image in which the shadow area has been corrected by using the image processing technique disclosed in Patent Document 1, for example. In this case, one of the pairs obtains the tomographic image corrected using the image processing technique by correcting the brightness value of the pixel determined by the operator to be insufficiently corrected for the shadow region to a desired value. You may. Alternatively, by replacing the brightness statistic (for example, the average value or the median value) of a neighboring region calculated in a region having a size sufficiently including an object causing a shadow behind, such as a blood vessel, and calculated in the same layer region. May be acquired. Although FIGS. 4A to 4D, 5A to 5F, and 6 illustrate only a tomographic image including a macula region, the tomographic image actually includes a region of the optic papilla. Shadow areas also occur below the blood vessels in the area. A large number of thick shadow areas are formed in the optic disc due to the collection of large blood vessels in the retina, and this tends to hinder the depiction, identification, and measurement of the lamina cribrosa existing particularly in the deep portion of the optic disc. Of course, training may be performed on a tomographic image obtained by photographing only the macula or the optic papilla. Further, examples of other teacher data include teacher data (hereinafter, second teacher data) configured by a pair group of the tomographic image 403 before the shadow area correction and the correction value data illustrated in FIG. 4A. . In this case, the correction value data is used to calculate each pixel value of the tomographic image after the shadow region correction (the shadow region corrected tomographic image 407) from each pixel value in the tomographic image 403 for the shadow region correction. Includes data used.

Next, the images used during training will be described. An image group forming a pair group of the tomographic image 403 before the shadow area correction and the tomographic image corrected for the shadow area 407 constituting the first teacher data is defined by a rectangular area image having a fixed pixel size corresponding to the positional relationship. create. This will be described below with reference to FIGS. 5A to 5F. 5A to 5F are diagrams respectively showing a tomographic image before shadow area correction as input data and a tomographic image with shadow area corrected as output data.

{Suppose a case where one of the group of pairs forming the first teacher data includes a tomographic image 403 including a shadow area and a tomographic image 407 having a shadow area corrected. In this case, as shown in FIG. 5A and FIG. 5B, the input data forming the pair is a tomographic image 501 and the output data is a tomographic image 501 ′. 5A and 5B, the entire image is a pair image, but the configuration of the pair image is not limited to this. For example, as shown in FIG. 5C,

rectangular area images

5021 and 5022 of the tomographic image 403 including the shadow area may be used as input data. In this case, as shown in FIG. 5D, the output data is rectangular area images 5021 'and 5022' which are the same imaging area in the shadow area corrected tomographic image 407. That is, a pair of the input data and the output data may be constituted by these rectangular area images. Note that this rectangular area is based on the A-scan unit. The A scan unit may be one A scan or several A scan units. Alternatively, as shown in FIG. 5E,

rectangular area images

5031 and 5032 of the tomographic image 403 including the shadow area may be used as input data. In this case, as shown in FIG. 5F, rectangular area images 5031 'and 5032', which are the same imaging area in the shadow area corrected tomographic image 407, are set as output data and form a pair with the input data.

At the time of training, it is desirable to normalize the scan range (angle of view) and scan density (number of A-scans) to make the image size uniform, and to make the rectangular area size during learning uniform. The rectangular area images shown in FIGS. 5A to 5F are examples of the rectangular area sizes when training is performed separately. Here, the number of rectangular areas can be set to one in FIGS. 5A and 5B, and a plurality can be set as described above in FIGS. 5C to 5F. For example, the rectangular area image 5022 on the tomographic image 403 including the shadow area in FIG. 5C is input data, and the rectangular area image 5022 'at the same position on the shadow area corrected tomographic image 407 in FIG. 5D is output data. Thus, a rectangular area image pair different from the first rectangular area image pairs 5021 and 5021 'can be created. It should be noted that by creating many pairs of rectangular area images while changing to different coordinates, the group of pairs constituting the first teacher data is enhanced. 5C to 5F, rectangular regions are discretely shown. However, in practice, the image is divided into rectangular region images having a fixed pixel size and without gaps, and a pair group is generated. Good to do. In the original tomographic image and the tomographic image after the shadow correction, by creating many pairs of rectangular area images while changing the position of the area to different coordinates, it is possible to enrich the group of pairs forming the teacher data. Further, by selecting an image of a smaller area as a rectangular area as a pair of input data and output data, it is possible to generate a lot of pair data from the tomographic image forming the original pair and the tomographic image after shadow correction. Therefore, the time required for training the trained model can be reduced.

Next, as an example of the learned model according to the present embodiment, a convolutional neural network (hereinafter, referred to as CNN) that performs a shadow area correction process on an input tomographic image will be described with reference to FIG. FIG. 6 shows an example of the configuration of the learned model in the correction units 101-43. The configuration illustrated in FIG. 6 is configured by a plurality of layer groups that are responsible for processing the input value group and outputting the processed value group. Note that, as shown in FIG. 6, the types of layers included in this configuration include a convolution layer, a downsampling (Downsampling) layer, an upsampling (Upsampling) layer, and a synthesis (Merger) layer. The convolution layer is a layer that performs a convolution process on an input value group according to parameters such as a set filter kernel size, the number of filters, a stride value, and a dilation value. The number of dimensions of the kernel size of the above-described filter may be changed according to the number of dimensions of the input image. The downsampling layer is a process in which the number of output value groups is made smaller than the number of input value groups by thinning out or combining input value groups. Specifically, for example, there is a Max @ Pooling process. The upsampling layer is a process of increasing the number of output value groups beyond the number of input value groups by duplicating the input value group or adding a value interpolated from the input value group. Specifically, for example, there is a linear interpolation process. The synthesis layer is a layer that performs processing of inputting a value group such as an output value group of a certain layer or a pixel value group forming an image from a plurality of sources, and connecting or adding them to synthesize.

As parameters set in the convolutional layer group included in the configuration of FIG. 6, for example, by setting the kernel size of the filter to 3 pixels in width, 3 pixels in height, and 64 to the number of filters, shadows with a certain accuracy Area correction processing is possible. However, it should be noted that if the parameter setting for the layer group or the node group forming the neural network is different, the degree to which the tendency trained from the teacher data can be reproduced in the output data may be different. That is, in many cases, appropriate parameters are different depending on the mode of implementation, and it is preferable to change them as necessary. Further, by changing the configuration of the CNN as well as the method of changing the parameters as described above, there may be a case where the CNN can obtain better characteristics. The better characteristics include, for example, a high ability to correct the shadow area, a short time for the shadow area correction processing, and a short time for training when obtaining a learned model.

Although not shown, as a modification of the configuration of the CNN, for example, a batch normalization (Batch Normalization) layer may be incorporated after the convolutional layer. Alternatively, an activation layer using a normalized linear function (Rectifier @ Linear @ Unit) may be incorporated.

When data is input to such a trained model, data according to the design of the trained model is output. For example, output data having a high possibility of corresponding to the input data is output according to the tendency trained from the teacher data. Further, for example, the possibility is output as a numerical value for each of the types of output data trained from the teacher data. Specifically, for example, when a tomographic image 601 obtained by OCT is input to a trained model trained by the first teacher data, a shadow region corrected tomographic image 602 is output.

Also, for example, when the OCT tomographic image 601 is input to the learned model trained by the second teacher data, a group of correction values to be applied when correcting the shadow area is output. The format and combination of the input data and the output data of the pair group forming the teacher data may be such that one is an image and the other is a numerical value, one is a plurality of image groups and the other is a numerical value, or both are numerical values. It is performed in a combination suitable for the embodiment, such as an image. When learning is performed by dividing an area as shown in FIGS. 5C to 5F, the learned model sets the brightness value of the shadow area corrected tomographic image or the correction value for the shadow area correction in each rectangular area. Output. When the learned model outputs a correction value, the correction unit 101-43 calculates a correction value with respect to the input luminance value of the tomographic image 601 to output a shadow area corrected luminance value. The correction unit 101-43 arranges and combines the respective shadow area corrected rectangular area image groups in the same positional relationship as the input rectangular area image group, and corrects the shadow area. 602 is obtained.

In the present embodiment, the segmentation of the layer boundary and the front / rear boundary of the lamina cribrosa is performed using the layer boundary and the front / rear boundary positions of the lamina cribrosa as initial positions for the tomographic image in which the shadow region is corrected. Shall be implemented again. This makes it possible to obtain accurate layer boundaries and front / rear boundaries of the cribriform plate that are not easily affected by the shadow area. Further, the projection unit 101-45 projects the shadow area corrected superimposed tomographic image using the position of the layer boundary and the front / back boundary of the lamina cribrosa acquired by the image feature acquisition unit 101-44, and A superimposed front tomographic image is generated. As a projection method, any known projection is possible, and in this embodiment, average projection is performed. However, the projection method is not limited to this, and various known projection methods can be applied.

Finally, the image processing apparatus 101 associates the acquired image group (SLO image or tomographic image), the imaging condition data of the image group, and the data obtained in S304 with the examination date and time and the information for identifying the eye to be examined. Are stored in the external storage unit 102. The data obtained in S304 includes a tomographic image or a correction value for which the shadow region has been corrected, a layer boundary, and a front / rear boundary data of the lamina cribrosa.

<Step 305 (S305)>
The display control unit 101-05 causes the display unit 104 to display the information regarding the superimposed tomographic image (3D image / B-scan image / front image) and the imaging condition after the shadow area correction generated in S304. FIG. 7 shows an example of a report screen 700 displayed on the display unit 104.

The display control unit 101-05 displays the SLO image 702 on the upper left of the report screen 700 and the superimposed tomographic image (B-scan image 704) with the shadow area corrected on the upper right. The display control unit 101-05 superimposes and displays an arrow (gray) indicating the scanning position of the measurement light at the time of obtaining the B-scan image 704 (tomographic image) with the shadow area corrected on the SLO image 702. Further, the layer boundaries (the inner limiting membrane 1 and the retinal pigment epithelium 5 and the like) and the boundaries between the lamina cribrosa regions acquired in S303 and S304 are superimposed on the B scan image 704. On the lower left of the report screen 700, a superimposed front tomographic image 703 with the shadow area corrected is displayed. Since the B-scan and the front tomographic image in which the shadow area is suppressed are displayed, the front tomographic image can be observed with the original luminance of the subject's eye.

<Step 306 (S306)>
After observing the above-mentioned front tomographic image on the report screen 700, the operator instructs, via the input unit 103, whether to perform another image observation again or to end the observation. When the end instruction is given, the operation processing ends. When an instruction to continue image observation is input, the flow returns to S302, and imaging of a new OCT tomographic image is started.

As shown in FIG. 7, a user interface 705 for switching whether to apply the shadow area correction process may be displayed. Furthermore, a character string, a mark, or a correction amount value indicating the application state of the shadow area correction processing on the tomographic image (B scan / front / 3D image) displayed on the display unit 104 may be displayed. These may be displayed together with the user interface 705. The user interface 705 inputs an instruction regarding whether or not the shadow area correction processing can be applied. Then, based on this instruction, whether to apply the shadow area correction processing to the tomographic image (B-scan image 704, front

tomographic image

703, 3D image (not shown)) displayed on display unit 104 is switched. Further, a user interface (slider or the like) for adjusting the luminance correction amount of the shadow region may be displayed on the display unit 104 so that the user can manually adjust the correction amount of the shadow region.

では In the present embodiment, the correction unit 101-43 two-dimensionally or three-dimensionally reduces a shadow region generated under an object such as a blood vessel in a tomographic image using a learned model. However, the present invention is not limited to correction of blood vessel shadows. For example, a shadow region generated under vitiligo, opacity of an intermediate translucent body, bleeding, or the like may be reduced by using a learned model. That is, training of a learned model is performed using a pair of a tomographic image before correction including a shadow region due to vitiligo, vitreous opacity, and bleeding and a tomographic image corrected for the shadow region as teacher data. The present invention also includes a case where a tomographic image including a shadow region caused by vitiligo, vitreous opacity, or bleeding is input to the trained model obtained by this training, so that the shadow region on the tomographic image is robustly reduced. .

In the present embodiment, in S304, a shadow generated below an object in the eye such as a blood vessel in a tomographic image is obtained using a single learned model trained using teacher data of various parts and diseases. The area is being corrected. However, the present invention is not limited to this. For example, since the density and thickness of the included blood vessels differ depending on the site in the fundus, it is conceivable that the reduction of the shadow area may be insufficient if a single learned model is used for all sites. In this case, the correction unit 101-43 is provided with a plurality of learned models, and for each of the learned models, teacher data composed of a “pair of shadow region pre-correction / corrected tomographic images of only a specific part” is used. It is good to train. Similarly, training may be performed using teacher data composed of “a pair of a shadow region before correction and a corrected tomographic image of only a specific disease” according to a disease or the like. The present invention includes a case in which a shadow region generated below an object in the eye such as a blood vessel in a tomographic image is corrected based on outputs from a plurality of learned models.

画像 As described above, the image processing apparatus 101 according to the present embodiment includes the image acquisition unit and the correction unit. The image acquisition unit (image acquisition unit 101-1) acquires a tomographic image of the eye 200 to be inspected. The correction unit (correction unit 101-43) is a shadow region (402) generated by an object included in the subject's eye 200 using a learned model obtained by training, and is a shadow region generated in a tomographic image of the subject's eye 200. Correct the pixel value of the area. The trained model described here is a tomographic image including a shadow region acquired from the eye 200 to be examined, and a tomographic image obtained by performing image processing for correcting a pixel value of the shadow region of the tomographic image including the shadow region. Is obtained based on learning using a pair with. In addition, the present invention can be understood as an image processing method including an image acquisition step and a correction step performed by the above-described units or the image processing apparatus. Further, the image processing apparatus according to the present embodiment may be configured to include the above-described image acquisition unit and the generation unit. In this case, the generation unit corrects the pixel value of the shadow area (402) generated by the object included in the eye 200 and the shadow area generated in the tomographic image of the eye 200 using the learned model described above. Generated tomographic image.

態様 In the depth direction of the subject's eye 200, the configuration of the component that generates the shadow region is different. For example, in the tomographic image, the form of PA differs depending on which part of the retina, choroid, or cribriform plate is focused. Therefore, it is preferable to perform image correction of the shadow area according to these parts. In this case, the correction unit 101-43 converts a plurality of learned models obtained by training based on a pair provided for each of the retina, choroid, and lamina cribrosa in the tomographic image according to the region. It is preferable to use them. Further, when the learned model is not used, the image value of the shadow area may be corrected in a known image processing method in a manner (processing method) corresponding to a part.

According to the configuration described above, the image processing apparatus 101 converts a pair of a tomographic image of various parts and diseases and a tomographic image obtained by applying the shadow area correction processing to the tomographic image into a trained model by deep learning. Train. By inputting a tomographic image to the trained (learned) learned model, the shadow area on the tomographic image is robustly reduced. This makes it possible to correct a shadow region in a tomographic image regardless of a disease or a part.

[Second embodiment]
The image processing apparatus according to the present embodiment generates a motion contrast image using a tomographic image obtained by applying the shadow area correction processing described in the first embodiment to a tomographic image scanned a plurality of times at the same position. I do. The manner in which the PA is reduced in the motion contrast image will be described.

FIG. 8 shows the configuration of the image processing apparatus 801 according to the present embodiment. Note that components having substantially the same functions as those of the components of the image processing apparatus 101 according to the first embodiment will be denoted by the same reference numerals, and description thereof will be omitted. The image processing apparatus 801 according to the present embodiment is different from the first embodiment in that the image acquisition unit 101-01 includes the motion contrast data generation unit 101-12 and the image processing unit includes the synthesis unit 101-46. ing.

Next, a processing procedure of the image processing apparatus 801 according to the present embodiment will be described with reference to FIG. FIG. 9 is a flowchart showing a flow of operation processing of the entire system in the present embodiment. Note that among the operation processing flows in the present embodiment, except for S901, S902, S905, and S906 in FIG. 9, are the same as the processing performed in the corresponding steps in the first embodiment, and thus description thereof will be omitted. I do.

<Step 901>
By operating the input unit 103, the operator sets imaging conditions when an OCTA image is imaged by the tomographic image imaging apparatus 100. In this embodiment, the photographing conditions are set as follows, and OCTA photographing (under the same photographing conditions) is repeatedly executed a predetermined number of times while appropriately taking a break in S902.
1) Select the OCTA scan mode 2) Set the following imaging parameters 2-1) Scan pattern: Small Square
2-2) Scanning area size: 3 x 3 mm
2-3) Main scanning direction: horizontal direction 2-4) Scan interval: 0.01 mm
2-5) Fixation light position: macular or lighting position at the time of imaging of optic disc 2-6) Number of B scans per cluster: 4

<Step 902 (S902)>
After setting the shooting conditions, the operator operates the input unit 103 and presses a shooting start button (not displayed) in the shooting screen. Thus, repeated OCTA imaging by the tomographic imaging apparatus 100 under the imaging conditions specified in S901 is started. More specifically, the imaging control unit 101-03 instructs the tomographic imaging apparatus 100 to repeatedly perform OCTA imaging based on the setting instructed by the operator in S901. Thereby, the tomographic imaging apparatus 100 acquires a corresponding OCT tomographic image.

In the present embodiment, the number of times of repetitive imaging (the number of clusters) in this step is five. However, the number of times of repetitive imaging (the number of clusters) is not limited to this, and may be set to an arbitrary number.

Further, the tomographic imaging apparatus 100 also acquires an SLO image, and executes a tracking process based on the SLO moving image. When the number of clusters is two or more, the reference SLO image used in the tracking processing in the repeated OCTA imaging is the reference SLO image set in the first cluster imaging, and a common reference SLO image is used in all cluster imaging. During the second and subsequent cluster imaging, the same setting values are used (no change) as to 1) selection of the left and right eyes 2) whether or not to perform the tracking processing in addition to the imaging conditions set in S901. However, the conditions for the tracking processing are not limited to the above, and can be changed as appropriate according to the conditions for capturing the OCT tomographic image.

<Step 905 (S905)>
Next, the image acquisition unit 101-01 and the image processing unit 101-04 generate a motion contrast image using the OCT tomographic image to which the shadow area correction and alignment processing generated in S904 has been applied. In S905, the motion contrast data generation unit 101-12 calculates a motion contrast between adjacent tomographic images in the same cluster.

In the present embodiment, the decorrelation value Mxy is obtained as the motion contrast based on the following equation (1).

Here, Axy indicates the amplitude (of the complex number data after the FFT processing) at the position (x, y) of the tomographic image data A, and Bxy indicates the amplitude at the same position (x, y) of the tomographic data B. The tomographic image data A and B are tomographic image data in the same cluster, for example, obtained sequentially in time. 0 ≦ Mxy ≦ 1, and a value closer to 1 is taken as the difference between the two amplitude values is larger. The decorrelation calculation processing as in Expression (1) is performed between arbitrary adjacent tomographic images (belonging to the same cluster). Then, an image having an average of the obtained (number of tomographic images per cluster minus one) motion contrast values as pixel values is generated as a final motion contrast image.

In the present embodiment, the motion contrast is calculated based on the amplitude of the complex data after the FFT processing. However, the method of calculating the motion contrast is not limited to this. For example, the motion contrast may be calculated based on the phase information of the complex data, or the motion contrast may be calculated based on both the amplitude and the phase information. Alternatively, the motion contrast may be calculated based on the real part or the imaginary part of the complex data.

In the present embodiment, the decorrelation value is calculated as the motion contrast, but the motion contrast calculation method is not limited to this. For example, the motion contrast may be calculated based on the difference between the two values, or the motion contrast may be calculated based on the ratio of the two values. Furthermore, in the present embodiment, a final motion contrast image is obtained by calculating an average value of a plurality of acquired decorrelation values, but the present invention is not limited to this. For example, an image having the pixel value of the median value or the maximum value of the acquired plurality of decorrelation values may be generated as the final motion contrast image.

The image processing unit 101-04 three-dimensionally aligns a group of motion contrast images obtained through repeated OCTA imaging, and performs averaging to generate a high-contrast combined motion contrast image. The combining process is not limited to the simple averaging. For example, an average value may be used after arbitrarily weighting the luminance value of each motion contrast image. Alternatively, an arbitrary statistical value such as a median value may be calculated. The present invention also includes a case where the positioning process is performed two-dimensionally.

Note that the synthesizing unit 101-46 may be configured to determine whether a motion contrast image inappropriate for the synthesizing process is included, and then perform the synthesizing process excluding the motion contrast image determined to be inappropriate. For example, when the evaluation value (for example, the average value or median of decorrelation values) of each motion contrast image is out of a predetermined range, it may be determined that the motion contrast image is not suitable for the combination processing.

In the present embodiment, the motion contrast is calculated using the tomographic image in which the shadow area has been corrected. Therefore, the amplitude (of the complex number data after the FFT processing) at the position (x, y) under the blood vessel region between the adjacent tomographic images in the same cluster is similar, and as a result, the motion contrast is suppressed. FIG. 10A is a schematic diagram of a normal motion contrast image, and FIG. 10B is a schematic diagram of a motion contrast image generated using the shadow region corrected tomographic image. Normally, as shown in FIG. 10A, an area 405 corresponding to PA is generated below an area 404 corresponding to a blood vessel in the motion contrast image. On the other hand, as shown in FIG. 10B, in the present embodiment, the PA under the region 404 in the finally generated motion contrast image is suppressed.

The projection unit 101-45 projects a motion contrast image based on the layer boundaries and the positions of the front and back surfaces of the lamina acquired by the image feature acquisition unit 101-44. Then, these are superimposed to generate a front motion contrast image. As a projection method at this time, either maximum intensity projection (MIP; Maximum Intensity Projection) or average intensity projection (AIP; Average Average Intensity Projection) can be selected. In the present embodiment, it is assumed that a motion contrast image is projected by maximum intensity projection.

Finally, the image processing apparatus 101 associates the acquired image group (SLO image or tomographic image), the imaging condition data of the image group, and the data obtained in S905 with the examination date and time and the information for identifying the subject's eye. Are stored in the external storage unit 102. The data obtained in S905 includes the generated three-dimensional and frontal motion contrast images, and their accompanying generation condition data.

<Step 906 (S906)>
The display control unit 101-05 causes the display unit 104 to display the tomographic images generated and corrected in S903 and S904, the three-dimensional and frontal motion contrast images synthesized in S905, and information on imaging conditions and synthesis conditions. FIG. 10C shows an example of the report screen 1000 displayed on the display unit 104.

In the present embodiment, the SLO image, the tomographic image with the shadow area corrected, the frontal motion contrast images in different depth ranges generated by combining and projecting in S905, and the corresponding frontal OCT image are displayed. On the report screen 1000 shown in FIG. 10C, front

motion contrast images

1001 and 1005 generated with the retinal surface layer as the projection depth range in the upper row and the retinal deep layer in the lower row are displayed. PA is suppressed in the front motion contrast image 1005 of the deep retina shown in the lower part. The motion contrast image displayed on the display unit 104 is not limited to the front motion contrast image. For example, a three-dimensional (PA suppressed) motion contrast image may be displayed.

In the report screen 1000, the projection range of the front motion contrast image can be changed by the operator selecting from a predetermined

depth range set

1002, 1006 displayed in the list box. Further, the type and offset position of the layer boundary used to specify the projection range can be changed using the

user interfaces

1003 and 1007. Further, by moving the

layer boundary data

1004 and 1008 superimposed on the tomographic image by operating the input unit 103, the projection range of the motion contrast image can be changed. Further, the image processing apparatus 801 may be configured so that the user presses the button 1009 shown in FIG. 10C to perform the motion contrast combining processing performed in S905.

Further, a B-scan tomographic image or a frontal tomographic image in which the applicability of the shadow area correction processing is changed may be displayed by using the user interface 705 for designating the applicability of the shadow area correction processing in the tomographic image. In this case, in conjunction with the operation of the user interface 705, the presence or absence of the PA suppression processing in the motion contrast image or the motion contrast front image superimposed on the B-scan tomographic image is also changed. In FIG. 10C, a user interface 705 for specifying whether or not to apply the shadow area correction processing on the tomographic image is displayed. However, the display mode of the user interface for the instruction is not limited to this. For example, the present invention includes a case where a user interface for designating whether or not to apply the PA suppression processing is displayed on the display unit 104, and the image processing content of the image processing unit 101-04 is the same as the input for this display. The PA suppression processing application state on the motion contrast image displayed on the display unit 104 is changed according to an input to the user interface for designating whether or not the PA suppression processing can be applied. The shadow area on the tomographic image (B scan or front view) displayed on the display unit 104 may be corrected in conjunction with the input to the user interface for specifying whether or not to apply the PA suppression processing. The case without correction is also included in the present invention.

[Modification of Second Embodiment]
In the first embodiment, in S904, the shadow region generated under the object in the eye such as the blood vessel in the tomographic image is corrected using the learned model. However, the image correction of the shadow region is not limited to the image correction using the learned model described above. For example, the present invention includes a case in which PA is suppressed by generating a motion contrast image using a tomographic image in which image correction has been performed on a shadow region based on any known image processing method disclosed in Patent Document 1. It is. An example of such image processing will be described below.

In this case, in the acquired tomographic image, for example, points on the inner limiting membrane 1 and the retinal pigment epithelium layer (5) shown in FIG. 4A are detected as candidate points. Next, based on the connectivity of the detected candidate point sequence, it is determined whether or not the shadow area 402 is generated near each candidate point (whether or not the image indicates a shadow area). If it is determined that the area is a shadow area, a statistic related to a luminance value in the shadow area is obtained. After that, the luminance value (pixel value) in the shadow area is corrected based on the obtained statistics. Even with such image processing, a tomographic image in which the pixel value of the shadow region 402 has been corrected can be obtained.

Note that the shadow region 402 usually occurs behind a specific object included in the eye to be examined, such as a blood vessel, in the optical axis direction of the measurement light. For this reason, as a pre-processing, a main blood vessel which is the specific object may be identified, and an area where the image processing is performed may be limited to the surrounding area. By adopting such a method, it is expected that the processing speed of image processing of one tomographic image will be increased. In this case, the above-described configuration for identifying a shadow region may be used as an identification unit included in the correction units 101-43. Such identification of a blood vessel or a shadow region is also effective in correcting a pixel value using the learned model described above. For example, instead of teacher data consisting of a pair of one tomographic image, a case where a tomographic image is divided into a plurality of smaller rectangular areas and a pair with a corresponding image of an image-processed rectangular area is used as teacher data is also assumed. You. When the above-described identification unit is provided, it is possible to specify a blood vessel identified as having a possibility of generating a shadow area, or a rectangular area including a configuration identified as a shadow area. Accordingly, by performing image processing on the identified rectangular region including the blood vessel or the shadow region as a partial region in the tomographic image, a reduction in processing time required for image processing of one tomographic image can be expected.

As described above, the image processing device 801 according to the present embodiment includes a motion contrast image generation unit in addition to the above-described image acquisition unit and correction unit. Then, the motion contrast image generating means generates a motion contrast image using the plurality of tomographic images whose pixel values have been corrected by the image correcting means. In the present embodiment, the image acquiring unit (image acquiring unit 101-1) acquires a cluster including a plurality of tomographic images acquired by scanning the same position of the eye 200 a plurality of times. It is desirable that the tomographic images included in the cluster be originally acquired from the same position (on the scanning line) on the subject's eye, but are not actually acquired from the exact same position due to the so-called fixation fine movement of the subject's eye. Therefore, in this specification, a plurality of tomographic images in a cluster are defined as a plurality of tomographic images acquired with the intention of scanning the same position a plurality of times. Note that the plurality of tomographic images in the cluster can be said to be a plurality of tomographic images obtained by photographing the same portion of the eye to be inspected, and this is obtained by controlling the same portion of the eye to be inspected so that the measurement light is scanned. A plurality of tomographic images may be used. The motion contrast image generation unit (motion contrast data generation unit 101-12, image processing unit 101-04) generates a motion contrast image using a cluster or a plurality of tomographic images. At this time, the correction unit (correction unit 101-43) corrects the pixel value of the shadow area generated by the object included in the subject's eye 200 in the plurality of tomographic images. Then, the motion contrast image generation unit generates a motion contrast image using the tomographic image after the pixel value correction. In the present embodiment, a blood vessel is taken as an example of an object included in the eye to be inspected, but an object such as a vitiligo, opacity of an intermediate light-transmitting body, and a lesion such as bleeding is also included.

Note that the image processing device 801 according to the present embodiment can also be constructed as a mode including an image acquisition unit, a motion contrast generation unit, and a correction unit. In this case, the image acquiring unit (image acquiring unit 101-1) acquires a cluster including a plurality of tomographic images acquired with the intention of scanning the same position of the eye 200 a plurality of times. The motion contrast generation means (motion contrast data generation unit 101-12, image processing unit 101-04) can generate a motion contrast image using this cluster. As described above, the correction unit (correction unit 101-43) corrects the pixel value of the shadow area in the tomographic image, which is the shadow area generated by the object included in the eye 200 to be inspected. Then, the motion contrast image generation means generates a motion contrast image using a cluster including the tomographic image in which the pixel value of the shadow area has been corrected. The correction of the pixel value at that time can also be performed by various known image processing.

Note that the image correction unit performs learning using a pair of a tomographic image including a shadow region acquired from the subject's eye 200 and a tomographic image obtained by performing image processing on the shadow region of the tomographic image including the shadow region. It has a trained model obtained based on it. By using this learned model, a tomographic image after pixel value correction can be easily obtained. However, as described above, the image correction unit can correct the pixel value of the pixel corresponding to the shadow area by a known image processing method without using the learned model.

Note that, in order for the operator to observe the image of the eye to be inspected, the image processing apparatus 801 according to the present embodiment causes the display control unit 101-05 (display control unit) to display the display target image on the display unit 104 (display unit). ) May be further provided. In the present embodiment, the display control unit causes the display unit 104 to display at least one of a tomographic image and a motion contrast image whose pixel values have been corrected. In such an embodiment, when the tomographic image with the corrected pixel value is displayed on the display unit 104, it is preferable that the operator determines whether or not the pixel value is appropriately corrected in the tomographic image. For example, when the correction is incomplete or excessive, the motion contrast image generated using the correction is likely to include PA and the like.

In contrast, it is preferable to further include an input unit (the input unit 103 and the user interface 705) that receives a determination as to whether the tomographic image after the pixel value correction is acceptable. If the input means determines that the displayed tomographic image is unacceptable, it is not suitable for diagnosis, etc., if a tomographic image with improper pixel value correction is obtained. A motion contrast image may be generated. Therefore, in this case, the display control unit 101-05 causes the display unit 104 to display a motion contrast image generated using a plurality of tomographic images before pixel value correction. Therefore, an image more suitable for diagnosis or the like is always displayed on the display unit 104. Further, as described above, a case where the motion contrast image generated from the tomographic image after the pixel value correction is inappropriate for diagnosis due to PA or the like may be considered. For this purpose, the display control unit 101-05 outputs one of the motion contrast image generated using the tomographic image before the pixel value correction and the motion contrast image generated using the tomographic image after the pixel value correction. It is desirable to be able to display.

As described above, the display control unit 101-05 may cause the display unit 104 to display a user interface for switching whether to apply the correction of the image value of the shadow area to the acquired plurality of tomographic images. In this case, the display unit 104 may further display at least one of a character string or a mark indicating the application state of the correction of the image value in the shadow area, and the value of the correction amount of the pixel value. By displaying these, the operator can instantaneously determine whether or not the pixel value has been corrected, and whether or not the degree of correction is appropriate, and can switch the motion contrast image as necessary. Becomes Furthermore, when an instruction to apply the correction of the image value of the shadow area is input to the user interface, the display unit 104 may display the tomographic image with the corrected pixel value. With such a configuration, an image desired by the operator can be provided as appropriate.

According to the configuration described above, the image processing device 801 applies the shadow area correction processing to the tomographic image scanned a plurality of times at the same position, and generates a motion contrast image using the obtained tomographic image. By using the tomographic image in which the shadow area has been corrected, a motion contrast image with reduced PA can be obtained.

[Third embodiment]
The image processing apparatus according to the present embodiment describes a case in which a layer shape or a cribriform plate shape is measured using a tomographic image to which a shadow region correction process using a learned model described in the first embodiment is applied. .

FIG. 11 shows the configuration of the image processing apparatus 1101 according to the present embodiment. Note that components having substantially the same functions as those of the components of the image processing apparatus 101 according to the first embodiment will be denoted by the same reference numerals, and description thereof will be omitted. The image processing apparatus 1101 according to the present embodiment is different from the first embodiment in that an image processing unit 101-04 includes an analysis unit 101-47. The analyzing unit 101-47 includes an extracting unit 101-471 and a measuring unit 101-472.

Next, a processing procedure of the image processing apparatus 1101 according to the present embodiment will be described with reference to FIG. FIG. 12 is a flowchart showing a flow of operation processing of the entire system in the present embodiment. In the operation processing flow of the present embodiment, the processes other than S1205 and S1206 in FIG. 12 are the same as the processes performed in the corresponding steps in the first embodiment, and thus description thereof will be omitted.

<Step 1205 (S1205)>
The extraction unit 101-471 performs a predetermined layer region, a cristal plate region, and a cristal plate hole region based on the retinal and choroid layer boundaries and the anterior / posterior boundary of the lamina cribrosa acquired from the shadow region corrected tomographic image. To identify. In the present embodiment, a high-luminance region within the depth range surrounded by the end of the Bruch's membrane boundary and the front and back surfaces of the cribriform plate detected in S1203 is specified as the cribriform plate region. Then, in step S1203, the low-luminance mass region acquired from the superimposed tomographic front image is specified as the cribriform plate hole region. Note that the sieve plate hole region is not limited to being specified as a two-dimensional region. For example, a three-dimensional Hessian filter is applied to the three-dimensional tomographic image in which the shadow region has been corrected to emphasize the cribriform plate hole region, and then the depth surrounded by the edge of the Bruch's membrane boundary and the front and back surfaces of the cribriform plate. The low-luminance tubular region existing within the range may be specified as a three-dimensional cribrosa plate region.

In the present embodiment, the shadow region correction processing applied to the tomographic image in S1204 reduces the shadow region in the deep and outer layers of the retina, the layer regions belonging to the choroid, and the cribriform plate, in particular. Therefore, the deep and outer layers of the retina, the choroid, the lamina cribrosa, and the lamina cribrosa region can be more accurately specified.

The measuring unit 101-472 calculates a measured value related to the layer region belonging to the retina and the choroid and the shape of the lamina cribrosa. In the present embodiment, the retinal thickness and the choroid thickness are measured as the measurement values relating to the layer shape, and the thickness in the depth direction of the sieve plate and the diameter of the sieve plate hole are measured as the measurement values relating to the sieve plate shape.

The segmentation process and the measurement process performed in S1205 are not limited to being applied to the entire image. For example, the operator may use the input unit 103 to perform a segmentation process or a measurement process only on an area of an arbitrary shape set on a tomographic image or an enhanced image of the tomographic image. For example, the segmentation or measurement processing may be performed only in the ETDRS chart for the macula portion, and only in the (circular graph-shaped) divided circle region for the optic disc. In the present embodiment, the segmentation process is directly performed on the tomographic image. However, the present invention is not limited to the exemplified processing, and the segmentation processing may be performed after applying any known enhancement processing to the tomographic image.

<Step 1206>
The display control unit 101-05 causes the display unit 104 to display the information related to the measurement acquired in S1205. FIG. 13 shows an example of a report screen 1300 displayed on the display unit 104.

The display control unit 101-05 superimposes a translucent color map 1301 indicating the retinal thickness measured by the measurement unit 101-472 in S1205 on the SLO image 702 at the upper left of the report screen 1300. Note that the thickness map to be displayed is not limited to the retinal thickness, and an arbitrary layer thickness map may be displayed as long as the layer thickness can be measured in S1205. For example, a choroid thickness map may be displayed. As in the case of the first embodiment, a superimposed tomographic image (B-scan image) 704 with the shadow area corrected is displayed on the upper right of the report screen 1300, and a superimposed front tomographic image 703 with the shadow area corrected is displayed on the lower left. Is displayed. On the lower right of the report screen 1300, a layer thickness (retinal thickness in the present embodiment) graph 1302 measured on the currently displayed B-scan image is displayed.

According to the present embodiment, the measurement data calculated based on the layer boundary and the boundary of the lamina cribrosa acquired from the B-scan tomographic image in which the shadow area is suppressed is displayed. For this reason, it is possible to measure the shape of the robust layer or the cribrosa plate that is hardly affected by the shadow region.

Note that, similarly to the first embodiment, a user interface 705 for switching whether to apply the shadow area correction processing may be displayed. Furthermore, a character string, a mark, or a correction amount value indicating the application state of the shadow area correction processing on the tomographic image (B scan / front / 3D image) displayed on the display unit 104 may be displayed. These may be displayed together with the user interface 705. The user interface 705 inputs an instruction regarding whether or not the shadow area correction processing can be applied. Then, based on this instruction, whether to apply the shadow area correction processing to the tomographic image (B-scan image 704, front

tomographic image

703, 3D image (not shown)) displayed on display unit 104 is switched.

The measurement data may be switched and displayed on the display unit 104 in conjunction with the switching of the application of the shadow area correction process to the tomographic image (the B-scan image 704 or the front tomographic image 703, the 3D image). In this case, the measurement data includes measurement data relating to the layer shape and the cribriform plate shape for the tomographic image (with / without shadow area correction). Further, a user interface (slider or the like) for adjusting the luminance correction amount of the shadow area may be provided so that the user can manually adjust the correction amount in the shadow area. The extraction unit 101-471 and the measurement unit 101-472 calculate measurement data based on the adjusted tomographic image of the shadow correction amount, and the display control unit 101-05 causes the display unit 104 to display this.

As described above, in the image processing apparatus 1101 according to the present embodiment, the correction unit (correction unit 101-43) generates at least one of the retina, choroid, and cribriform plate in the tomographic image. The correction of the pixel value of the shadow region is performed. Then, the specifying unit (extracting units 101-471) specifies at least one boundary of the layer region, the cribrosa region, and the cribrosa hole region of the eye 200 from the tomographic image in which the pixel values have been corrected. Then, the measurement means (measurement units 101-472) calculates a measurement value for at least one of the layer region, the cribriform plate region, the cribriform plate hole region, the vascular region, and the avascular region specified by the specifying unit. The display control unit 101-05 in the image processing device 1101 according to the present embodiment is not described here. However, the display control unit 101-05 can control the display unit 104 to perform similar display, for example, as described in the second embodiment.

Alternatively, the image processing apparatus 1101 according to the present embodiment may be configured to include an image acquisition unit, a correction unit using the learned model, a specifying unit, and a measurement unit. In this case, the specifying unit and the measuring unit perform the specifying and measuring process using the tomographic image whose pixel value has been corrected by the correcting unit. Then, as described above, the image acquisition unit (image acquisition unit 101-01) acquires a tomographic image (intensity image) of the eye 200 to be inspected. The correction unit (correction unit 101-43) corrects a pixel value of a shadow region generated in a tomographic image by a structure included in the eye to be inspected. The specifying means (extracting units 101-471) specifies at least one boundary of the layer region, the lamina cribrosa region, the lamina cribrosa region, the blood vessel region, and the avascular region of the eye 200 to be examined in the tomographic image. The measurement unit (measurement unit 101-472) calculates a measurement value for at least one of the layer region, the lamina cribrosa region, the lamina cribrosa region, the blood vessel region, and the avascular region specified by the identification unit.

According to the configuration described above, the image processing apparatus 1101 measures the layer shape and the laminar shape on the tomographic image to which the shadow region correction processing using the learned model described in the first embodiment is applied. This makes it possible to measure the shape of a robust layer or cribriform plate that is less likely to be affected by the shadow area.

[Fourth embodiment]
The image processing apparatus according to the present embodiment uses a PA-suppressed motion contrast image generated using a tomographic image to which the shadow area correction processing described in the second embodiment is applied. The blood vessel region is specified from the motion contrast image, and the shape and distribution of the blood vessel are measured.

FIG. 14 shows the configuration of the image processing apparatus 101 according to the present embodiment. Note that components having substantially the same functions as those of the components of the

image processing apparatuses

101 and 801 in the first and second embodiments will be denoted by the same reference numerals, and description thereof will be omitted. An image processing apparatus 1401 according to the present embodiment is different from the second embodiment in that an image processing unit 101-04 includes an analysis unit 101-47. The analyzing unit 101-47 includes an emphasizing unit 101-473 for emphasizing a blood vessel region in addition to the extracting unit 101-471 and the measuring unit 101-472 described in the third embodiment.

Next, a processing procedure of the image processing apparatus 1401 according to the present embodiment will be described with reference to FIG. FIG. 15 is a flowchart showing the flow of the operation processing of the entire system in the present embodiment. In the operation processing flow of the present embodiment, except for S1507 to S1510 in FIG. 15, the processing is the same as the processing performed in each corresponding step in the second embodiment, and the description thereof will be omitted.

<Step 1507>
The operator operates the input unit 103 to instruct the start of the measurement process. In the present embodiment, it is assumed that the measurement screen is displayed by double-clicking the image on the report screen in FIG. 10C, and the analysis unit 101-47 starts the measurement processing. At this time, the operator uses the input unit 103 to select the type of measurement processing on a measurement start screen (not shown). In the present embodiment, as a type of measurement for a motion contrast image,
1) Vessel density (VAD) and 2) Vessel density (VLD)
The operator selects a desired measurement type from these.

The measurement performed using the motion contrast image is not limited to the type described above. For example, the present invention includes a case where the area or shape of a non-vascular area (NPA) is calculated for a motion contrast image. Here, VAD is an abbreviation for Vessel Area Density, and is a blood vessel density (unit:%) defined by the ratio of the blood vessel region included in the measurement target. VLD is an abbreviation of Vessel Length Density, and is a blood vessel density defined by the total length of blood vessels included in a unit area (unit: mm ^-1 ).

Vessel density is an index for quantifying the occlusion range of blood vessels and the degree of density of the vascular network, and VAD is most often used. However, in the VAD, the contribution of the large blood vessel region to the measured value is large. Therefore, when it is desired to perform the measurement by paying attention to the pathology of the capillary, VLD is used (as an index more sensitive to the occlusion of the capillary). However, the type of measurement to which the present invention can be applied to blood vessels is not limited to this. For example, Fractal @ Dimension for quantifying the complexity of the blood vessel structure or Vessel @ Diameter @ Index representing the distribution of the blood vessel diameter (the distribution of an aneurysm or stenosis of a blood vessel) may be measured.

Next, the analysis unit 101-047 performs pre-processing of the measurement processing. Any known image processing can be applied to the pre-processing. In the present embodiment, a top-hat filter process, which is a type of morphological operation, is performed on a motion contrast image. By applying the top hat filter, the luminance unevenness of the background component can be reduced.

<Step 1508>
The analysis unit 101-47 performs a process of specifying a blood vessel region on the motion contrast image. In the present embodiment, the enhancement units 101 to 473 perform a blood vessel enhancement process on the motion contrast image based on the Hessian filter. Next, the extraction unit 101-471 performs a segmentation process on the blood vessel enhanced image, and specifies a blood vessel region by performing a shaping process. Details of the blood vessel region identification processing will be described later with reference to S1610 to S1650 shown in the flowchart of FIG. 16A.

<Step 1509>
The measurement units 101-472 measure the blood vessel distribution with respect to the image of the single examination based on the information on the measurement target area specified by the operator. Subsequently, the display control unit 101-05 displays the measurement result on the display unit 104. Here, as the blood vessel density which is an index of the blood vessel distribution, there are two kinds of indices of VAD and VLD described above. In the present embodiment, an example of a procedure for calculating VLD which is an index more sensitive to capillary failure is described. Will be described. The measurement of the VLD for the motion contrast image will be described later with reference to S1660 to S1670 shown in the flowchart of FIG. 16B.

<Step 1510>
The display control unit 101-05 causes the display unit 104 to display a report on the measurement performed in step S1509. At this time, the display unit 104 also displays the left and right eyes, the shooting date and time, the angle of view and the number of pixels, the number of tomographic images at substantially the same position, and the OCTA superimposition processing execution conditions for each measurement target image. You may. Further, information regarding the evaluation value of the motion contrast image, the projection method, and whether or not the PA removal has been performed may be displayed on the display unit 104 together.

Alternatively, a motion contrast image or a binary image of a blood vessel region or a blood vessel center line may be superimposed on the tomographic front image on the display unit 104 by appropriately changing the color and transparency for each predetermined depth range. In this case, the motion contrast image or the binary image of the blood vessel region or the blood vessel center line is not limited to the projection display as the front image, and may be rendered three-dimensionally and displayed as a three-dimensional image. Further, the projection method (MIP / AIP) and the projection artifact suppression processing may be changed by a method such as selection from a context menu. Further, the binary image relating to the blood vessel region specified in S1508, the measured value or the measured value map calculated in S1509 may be output to the external storage unit 102 as a file and stored.

Furthermore, details of the blood vessel region identification processing executed in S1508 will be described with reference to the flowchart shown in FIG. 16A.

<Step 1610>
In the present embodiment, the emphasis scale adjustment processing is performed by setting an appropriate emphasis scale (parameter setting) according to the depth range and the region. In areas such as the macula, the retinal surface of the optic papilla, or the deep choroid, a large emphasis scale can be used to reduce the size of vessels with large diameters, such as retinal arteries and veins or deep choroidal vessels. Appropriately emphasized (without separation). Therefore, the blood vessel region can be accurately specified. On the other hand, in a region where only capillaries and new blood vessels are present, such as a deep layer or outer layer of the retina, or a region where only capillaries and new blood vessels are present, the edges of thin blood vessels are emphasized by performing the enhancement process on a small scale. Therefore, the blood vessel region can be specified more accurately when binarized (the phenomenon of overdetecting the blood vessel region can be prevented).

In the present embodiment, the emphasis scale is set for the front motion contrast image with respect to the emphasis scale adjustment processing for blood vessels having different thicknesses. However, the present invention is not limited to this. For example, an emphasis scale may be adaptively set for a three-dimensional motion contrast image.

<Step 1620>
The enhancement unit 101-473 performs a blood vessel enhancement filter process (tubular structure enhancement process) based on the eigenvalue of the Hessian matrix on the motion contrast image that has been subjected to the preprocessing of S1507. Such an emphasis filter is generically referred to as a Hessian filter, and includes, for example, a Vesselness filter and a Multi-scale line filter. In the present embodiment, a multi-scale line filter is used, but any known blood vessel enhancement filter may be used.

The Hessian filter smoothes the image with a size suitable for the diameter of the blood vessel to be emphasized. Then, a Hessian matrix having a second derivative of the luminance value as an element at each pixel of the smoothed image is calculated, and the local structure is emphasized based on the magnitude relation of the eigenvalues of the matrix. The Hessian matrix is a square matrix given by Expression (2), and each element of the matrix is, for example, a second derivative of the luminance value Is of the image obtained by smoothing the luminance value I of the image as shown in Expression (3). Expressed by value. The Hessian filter emphasizes such a Hessian matrix when "one of the eigenvalues (λ1, λ2) is close to 0 and the other is negative and has a large absolute value" as a linear structure. This is to emphasize pixels that satisfy the characteristics of the blood vessel region on the motion contrast image, that is, "the luminance change is small in the running direction and the brightness value is greatly reduced in the direction orthogonal to the running direction" as a linear structure. Is equivalent to

モーション In addition, the motion contrast image includes blood vessels of various diameters from capillaries to fibrillation veins. Therefore, a line-enhanced image is generated using an Hessian matrix for an image smoothed by a Gaussian filter at a plurality of scales. Next, as shown in Expression (4), after multiplying the square of the smoothing parameter σ of the Gaussian filter as a correction coefficient, the combined image is synthesized by a maximum value operation, and the synthesized image Ihesian is set as the output of the Hessian filter.

The present invention is not limited to applying a two-dimensional Hessian filter to a front motion contrast image. For example, a three-dimensional hessian filter may be applied to a three-dimensional motion contrast image to generate a three-dimensional enhanced image.

The Hessian filter has the advantage of being resistant to noise and improving the continuity of blood vessels. On the other hand, in practice, the maximum diameter of the blood vessel included in the image is often unknown in advance, and particularly when the smoothing parameter is too large relative to the maximum diameter of the blood vessel in the image, the emphasized blood vessel region is There is a disadvantage that it tends to be thick. In this embodiment, the defect is suppressed by performing the enhancement scale adjustment processing as described in S1610. The method for appropriately enhancing and binarizing the motion contrast image regardless of the blood vessel diameter is not limited to the method described in the present embodiment. For example, a common region of the binary image of the Hessian-enhanced image and the binary image of the blood vessel-enhanced image based on the edge selection sharpening may be specified as a blood vessel region.

<Step 1630>
The extraction unit 101-471 binarizes the blood vessel-enhanced image (hereinafter, referred to as a Hessian-enhanced image) using the Hessian filter generated in S1620. In the present embodiment, binarization is performed using the average value of the Hessian emphasized image as a threshold. However, by setting a predetermined lower limit as the threshold value for binarization, it is possible to prevent a region other than a blood vessel from being erroneously detected as an artifact. Note that the binarization processing described here is not limited to threshold processing, and may be binarized by any known segmentation method. Further, the segmentation processing by binarization is not limited to being applied to the entire image. For example, the operator may use the input unit 103 to perform the segmentation processing only on a motion contrast image or an arbitrary shape region set on the enhanced image of the motion contrast image.

<Step 1640>
The extraction unit 101-471 performs a thinning process on the binary image of the blood vessel region generated in S1630 to generate a binary image having a line width of one pixel corresponding to the center line of the blood vessel (hereinafter, referred to as a skeleton image). I do. As this processing, any thinning method or skeleton processing may be used, but in the present embodiment, Hilditch's thinning method is used as the thinning method.

<Step 1650>
The extraction units 101 to 471 perform morphological operation processing (opening processing (performing expansion processing after erosion processing) and closing processing (performing erosion processing after expansion processing)) as shaping processing of a blood vessel region. The shaping process is not limited to this, and for example, small regions may be removed based on the area of each label when a binary image is labeled.

{Details of the measurement processing executed in S1509 will be described with reference to the flowchart shown in FIG. 16B.

<Step 1660>
The analysis unit 101-47 sets a region of interest (measurement target image and measurement region) based on the content specified by the operator using the input unit 103. In the present embodiment, since “None” is selected as the OCTA map and “VLD” is selected as the OCTA sector map, the ETDRS sector region is set as the region of interest. The OCTA map refers to a color (or gray scale) map related to the blood vessel density measured in pixel units for the entire OCTA image. Further, the OCTA sector map refers to a statistical value (for example, an average) map of a blood vessel density distribution in the ETDRS sector set in the OCTA image. Note that the mode of the region of interest is not limited to this, and the region of interest may be divided into a pie chart-shaped segmented region, or a region of interest of any shape may be set.

<Step 1670>
The measurement unit 101-472 performs a measurement process based on the skeleton image obtained in S1640. In the present embodiment, at each pixel position of the skeleton image, the sum [mm ⁻¹ ] of the length of non-zero pixels (white pixels) per unit area in the vicinity area around the pixel is determined by the blood vessel density ( VLD). Further, an image (VLD map) having a value of the blood vessel density (VLD) calculated for each pixel is generated.

When a sector region is designated as the region of interest, the sum [mm ⁻¹ ] of the length of non-zero pixels (white pixels) per unit area in each sector region on the skeleton image is determined by the blood vessel in the sector. What is necessary is just to calculate as a density (VLD). When a VAD map is specified as a measurement for a motion contrast image, the following may be performed. That is, at each pixel position of the binary image relating to the blood vessel region acquired in S1630, the ratio of non-zero pixels (white pixels) occupying in the vicinity area around the pixel is calculated as the blood vessel density (VAD) in the pixel. An image (VAD map) having the VAD value calculated by the pixel is generated. Further, a VAD sector map can be generated by calculating the ratio of non-zero pixels (white pixels) in each sector area on the binary image relating to the blood vessel area as the blood vessel density (VAD) in the sector.

In the present embodiment, in S1504, the shadow region generated under the object in the eye such as the blood vessel in the tomographic image is corrected using the learned model. However, the method of correcting the shadow area is not limited to the image correction using the learned model. For example, the present invention includes a case in which PA is suppressed by generating a motion contrast image using a tomographic image in which image correction has been performed on a shadow region based on any known image processing method disclosed in Patent Document 1. It is.

The image processing apparatus 1401 according to the present embodiment further includes an area specifying unit in addition to the configuration described in the first embodiment and the second embodiment. The region specifying means (extracting units 101-471) specifies a blood vessel region or an avascular region by, for example, the above-described method from a motion contrast image generated using a plurality of tomographic images with corrected pixel values. The image processing apparatus 1401 further includes a calculation unit (measurement unit 101-472) that calculates a measurement value for at least one of the blood vessel region and the avascular region specified by the region specification unit.

According to the configuration described above, the image processing apparatus 1401 identifies a blood vessel region from a PA-suppressed motion contrast image generated using a tomographic image to which the shadow region correction processing described in the second embodiment has been applied, Measure and calculate the density. This makes it possible to accurately specify a blood vessel region and measure a blood vessel shape / distribution from a motion contrast image in which PA is suppressed based on an efficiently trained model in which learning of image features on an OCTA image is unnecessary. it can.

[Other Embodiments]
In each of the embodiments described above, the present invention is implemented as the

image processing apparatuses

101, 801, 1101, and 1401, but the embodiments of the present invention are not limited to the illustrated image processing apparatuses. For example, the present invention can take an embodiment as a system, an apparatus, a method, a program, a storage medium, or the like. That is, the present invention supplies a program for realizing one or more functions of the above-described embodiments to a system or an apparatus via a network or a storage medium. The present invention can also be realized by processing in which one or more processors in a computer of the system or the apparatus read and execute a program. Further, it can also be realized by a circuit (for example, an ASIC) that realizes one or more functions.

Although the present invention has been described with reference to the exemplary embodiments, the present invention is not limited to the above-described exemplary embodiments. The present invention includes inventions that have been modified within a scope not contrary to the gist of the present invention and inventions equivalent to the present invention. The above embodiments can be combined as appropriate without departing from the spirit of the present invention.

The present invention is not limited to the above embodiment, and various changes and modifications can be made without departing from the spirit and scope of the present invention. Therefore, the following claims are appended to make the scope of the present invention public.

This application claims the priority of Japanese Patent Application No. 2018-167195 filed on Sep. 6, 2018, the entire contents of which are incorporated herein by reference.

101-01: Image acquisition unit 101-04: Image processing unit 101-43: Correction unit

Claims

Image acquisition means for acquiring a tomographic image of the eye to be examined,
Correction means for correcting a pixel value of a shadow area in the tomographic image, which is a shadow area generated by an object included in the eye to be inspected, using a learned model;
An image processing apparatus comprising:
Image acquisition means for acquiring a tomographic image of the eye to be examined,
Generating means for generating, using a learned model, a tomographic image which is a shadow area generated by an object included in the eye to be inspected and in which the pixel value of the shadow area in the tomographic image is corrected,
An image processing apparatus comprising:
An identification unit for identifying at least one of a blood vessel and the shadow region generated by the blood vessel in the object,
The pixel value of the shadow area is corrected using a learned model corresponding to a partial area including at least one of the identified blood vessel and the identified shadow area. An image processing apparatus according to claim 1.
4. The pixel value correction is performed on a pixel value of a shadow region generated in at least one of a retina, a choroid, and a cribriform plate portion in the tomographic image. The image processing device according to any one of claims 1 to 4.
The image processing apparatus according to claim 4, wherein the image value of the shadow area is corrected using a learned model corresponding to each of a retina, a choroid, and a cribriform plate in the tomographic image. .
6. The image processing apparatus according to claim 1, further comprising: a specifying unit configured to specify at least one boundary between a layer region, a lamina cribrosa region, and a lamina cribrosa region of the subject's eye from the tomographic image in which the pixel values are corrected. The image processing device according to any one of claims 1 to 4.
The image according to claim 6, further comprising a measurement unit configured to calculate a measurement value for at least one of a layer region, a lamina cribrosa region, and a lamina cribrosa region of the eye to be examined identified by the identification unit. Processing equipment.
Motion contrast image generation means for generating a motion contrast image using a plurality of tomographic images acquired with the intention of scanning the same position of the subject's eye a plurality of times, wherein the plurality of tomographic images whose pixel values have been corrected are used. The image processing apparatus according to claim 1, further comprising:
Image acquisition means for acquiring a cluster including a plurality of tomographic images acquired with the intention of scanning the same position of the eye to be examined a plurality of times,
A motion contrast image for generating a motion contrast image using the cluster including a plurality of tomographic images, which are shadow regions generated by an object included in the eye to be inspected, and in which the pixel values of the shadow regions in the plurality of tomographic images are corrected. Generating means;
An image processing apparatus comprising:
10. The image processing apparatus according to claim 8, further comprising a display control unit that causes a display unit to display at least one of the tomographic image in which the pixel value is corrected and the motion contrast image.
When the tomographic image whose pixel value has been corrected is displayed on the display unit, the display unit further includes an input unit that receives a determination as to whether or not the displayed tomographic image is acceptable. Item 11. The image processing device according to Item 10.
When a determination that the displayed tomographic image is unacceptable is received by the input unit,
The image processing apparatus according to claim 11, wherein the display control unit causes the display unit to display a motion contrast image generated by the motion contrast image generation unit using the plurality of acquired tomographic images.
The motion contrast image generating means can generate a motion contrast image using the plurality of acquired tomographic images,
The display control means, a motion contrast image generated using the plurality of acquired tomographic images, and a motion contrast image generated using the plurality of tomographic images corrected pixel values, the display means The image processing apparatus according to claim 10, wherein the image processing apparatus can display the image data.
The display control unit includes a user interface for switching whether to apply the correction of the image value of the shadow area to the obtained plurality of tomographic images, and a character string or a character string indicating an application state of the correction of the image value of the shadow area. Mark, or at least one of the values of the correction amount of the pixel value, is displayed on the display means,
When an instruction to apply the correction of the image value of the shadow area is input to the user interface, the display control unit displays the tomographic image with the corrected pixel value on the display unit. The image processing device according to claim 10.
15. The image processing apparatus according to claim 8, further comprising a region specifying unit that specifies a blood vessel region or an avascular region from a motion contrast image generated using the plurality of tomographic images whose pixel values have been corrected. An image processing apparatus according to the item.
16. The image processing apparatus according to claim 15, further comprising a calculation unit configured to calculate a measurement value for at least one of the blood vessel region and the avascular region specified by the region specification unit.
The image processing apparatus further includes a display control unit that causes the display unit to display the tomographic image in which the pixel values are corrected,
The display control means, a user interface for switching whether to apply the correction of the image value of the shadow area to the acquired tomographic image, a character string or a mark indicating the application state of the correction of the image value of the shadow area, Or at least one of the values of the correction amount of the pixel value, and display on the display means,
When an instruction to apply the correction of the image value of the shadow area is input to the user interface, the display control unit displays the tomographic image with the corrected pixel value on the display unit. The image processing device according to claim 1.
The learned model is a teacher data in which a tomographic image of the subject's eye before correction is used as input data, and a corrected tomographic image obtained by correcting at least a part of the pixel values of the tomographic image before correction is output data. The image processing apparatus according to claim 1, wherein the image processing apparatus is obtained by using:
A step of acquiring a tomographic image of the subject's eye;
Correcting a pixel value of a shadow area in the tomographic image, which is a shadow area generated by an object included in the eye to be examined, using a learned model;
An image processing method comprising:
A step of acquiring a tomographic image of the subject's eye;
A step of generating a tomographic image in which a shadow area caused by an object included in the eye to be inspected and in which the pixel value of the shadow area in the tomographic image is corrected using a learned model,
An image processing method comprising:
A step of acquiring a cluster including a plurality of tomographic images acquired with the intention of scanning the same position of the subject's eye a plurality of times,
A shadow region generated by an object included in the eye to be inspected, a motion contrast image is generated using the cluster including the plurality of tomographic images in which the pixel values of the shadow region in the plurality of tomographic images are corrected,
An image processing method comprising:
A program which, when executed by a processor, causes the processor to execute each step of the image processing method according to any one of claims 19 to 21.