JP2022098699A - Image processing device, image processing method, and program - Google Patents

Abstract

To provide an image processing device capable of acquiring a front image that allows an object structure to be confirmed easily.SOLUTION: An image processing device includes: an acquisition unit for acquiring a plurality of front images corresponding to different depth ranges in three-dimensional data on an eye to be examined; an evaluation unit for acquiring an evaluation value that evaluates the presence of an object region by using at least two front images of the plurality of acquired front images as input data input to different channels of a learned model acquired by executing learning using learning data including the plurality of front images corresponding to the different depth ranges of the eye to be examined; and a determination unit for determining at least one of the plurality of front images as an output image using the evaluation value.SELECTED DRAWING: Figure 12

Description

The present invention relates to an image processing apparatus, an image processing method, and a program.

It is known that the morphology of blood vessels in the retina can be observed by optical coherence tomography (OCTA: Optical Coherence Tomography). Patent Document 1 relates to a general OCT (Optical Coherence Tomography) and an OCTA imaging device and its optical system, a method for generating motion contrast data, and a technique for projecting motion contrast data on a two-dimensional plane in a predetermined depth range. Have been described.

Japanese Unexamined Patent Publication No. 2018-89160

A 3D OCT image or a 3D OCTA frontal image (motion contrast image) is projected within a predetermined depth range to extract structural features (including blood vessels) of the eye including the retina, vitreous body, and choroid. However, it may generate a two-dimensional front image. In this case, it is common to define a depth range in which the target structure can be easily extracted with a specific layer of the retinal layer as a reference, and project an image in that depth range. In addition, multiple projection methods can be applied, and for volume angiography data, there are two types of projection: average image projection (AIP), in which the data is averaged, and maximum image projection (MIP), in which the maximum value is extracted from the data in a specific depth range. Two projection methods are commonly used. On the other hand, other projection methods can be used depending on the type of volume data. For example, projection may be performed by obtaining the total value, the minimum value, the median value, or the like of the data in the depth range.

However, depending on individual differences, the morphology of the structure, its condition (degree of lesion progression), etc., the characteristics of the structure to be confirmed may not be confirmed on the front image or may not be clearly seen. .. Therefore, in one embodiment of the present invention, there is provided an image processing device, an image processing method, and a program capable of acquiring a front image in which it is easy to confirm the target structure.

The image processing apparatus according to one embodiment of the present invention includes an acquisition unit that acquires a plurality of front images corresponding to different depth ranges of three-dimensional data of the eye to be inspected, and a plurality of front faces corresponding to different depth ranges of the eye to be inspected. By using at least two front images out of the plurality of acquired front images as input data input to different channels of the trained model obtained by training using the training data including images, the target area An image processing apparatus comprising an evaluation unit for acquiring an evaluation value for evaluating the existence of an image, and a determination unit for determining at least one of the plurality of front images as an output image using the evaluation value.

According to one embodiment of the present invention, it is possible to obtain a front image in which it is easy to confirm the target structure.

A schematic configuration example of the OCT apparatus according to the first embodiment is shown. A schematic functional configuration example of the image processing apparatus according to the first embodiment is shown. It is a figure for demonstrating the image generation part which concerns on Example 1. FIG. An example of the GUI according to the first embodiment is shown. An example of the GUI according to the first embodiment is shown. An example of the GUI according to the first embodiment is shown. A flowchart of a series of processes according to the first embodiment is shown. The flowchart of the front image generation processing which concerns on Example 1 is shown. It is a figure for demonstrating the image generation part which concerns on Example 2. FIG. The flowchart of the front image generation processing which concerns on Example 2 is shown. The flowchart of the front image determination process which concerns on Example 2 is shown. It is a figure for demonstrating the structure of the trained model which concerns on Example 2. FIG. The flowchart of the front image determination process which concerns on the modification of Example 2 is shown. The flowchart of the front image determination process which concerns on the modification of Example 2 is shown. It is a figure for demonstrating the structure of the trained model which concerns on Example 3. FIG. It is a figure for demonstrating the effect of the additional image of the trained model which concerns on Example 3. FIG. An example of the input image of the trained model according to the third embodiment is shown. It is a figure for demonstrating the structure of the trained model which concerns on Example 4. FIG. It is a figure for demonstrating the structure of the trained model which concerns on Example 5. FIG. The flowchart of the front image generation processing which concerns on Example 6 is shown.

Hereinafter, exemplary examples for carrying out the present invention will be described in detail with reference to the drawings. However, the dimensions, materials, shapes, relative positions of the components, etc. described in the following examples are arbitrary and can be changed according to the configuration of the device to which the present invention is applied or various conditions. Also, in the drawings, the same reference numerals are used between the drawings to indicate elements that are the same or functionally similar.

In the following, the machine learning model refers to a learning model based on a machine learning algorithm. Specific algorithms for machine learning include the nearest neighbor method, the naive Bayes method, the decision tree, and the support vector machine. In addition, deep learning (deep learning) in which features for learning and coupling weighting coefficients are generated by themselves using a neural network can also be mentioned. Further, as an algorithm using a decision tree, a method using gradient boosting such as LightGBM and XGBoost can be mentioned. As appropriate, any of the above algorithms that can be used can be applied to the following examples and modifications. The teacher data refers to learning data, and is composed of a pair of input data and output data (correct answer data).

The trained model is a model in which training (learning) is performed in advance using appropriate teacher data (learning data) for a machine learning model that follows an arbitrary machine learning algorithm such as deep learning. .. However, although the trained model is obtained by using appropriate training data in advance, it is not that no further training is performed, and additional training can be performed. Additional learning can be done even after the device has been installed at the site of use.

In Examples 1 to 6, an example of generating a frontal image for confirming a neovascularization (CNV) derived from exudative age-related macular degeneration (AMD) will be described. do. On the other hand, the present invention can also be applied to generate a frontal image for confirming, for example, a lamina cribrosa of the optic nerve head, a layer of choroid (Sattler layer, Haller layer), or a capillary aneurysm of a retinal blood vessel. be.

(Example 1)
In Example 1 of the present invention, by providing the operator with frontal images relating to a plurality of depth ranges and projection methods according to the extraction target (target area), it is easy to confirm the target structure. Is displayed. Hereinafter, with reference to FIGS. 1 to 8, the image processing device and the image processing method of the ophthalmic device according to the first embodiment of the present invention, particularly the optical coherence tomography device (OCT device) used in an ophthalmic clinic or the like will be described. .. Hereinafter, a method for displaying a new blood vessel (CNV) using OCTA according to this embodiment will be described.

(OCT optical system)
FIG. 1 shows a schematic configuration example of the OCT apparatus according to this embodiment. The OCT device according to this embodiment is provided with an optical interference unit 100, a scanning optical system 200, an image processing device 300, a display unit 310, a pointing device 320, and a keyboard 321. The optical interference unit 100 is provided with a low coherence light source 101 that emits near-infrared light, an optical branching unit 103, a collimating optical system 111, a dispersion compensation optical system 112, and a reference mirror 113. Further, the optical interference unit 100 is provided with a collimating optical system 122, a diffraction grating 123, an imaging lens 124, and a line sensor 125. The light emitted from the low coherence light source 101 propagates through the optical fiber 102a and is divided into the measurement light and the reference light by the optical branching portion 103. The measurement light divided by the optical branching portion 103 is incident on the optical fiber 102b and guided to the scanning optical system 200. On the other hand, the reference light branched by the optical branching portion 103 is incident on the optical fiber 102c and guided to the reference mirror 113. The optical branching portion 103 may be configured by using, for example, an optical fiber coupler or the like.

The reference light incident on the optical fiber 102c is emitted from the fiber end, enters the dispersion adaptive optics system 112 via the collimating optical system 111, and is guided to the reference mirror 113. The reference light reflected by the reference mirror 113 follows the optical path in the opposite direction and is incident on the optical fiber 102c again. The dispersion compensating optical system 112 corrects the dispersion of the optical system in the scanning optical system 200 and the eye E to be inspected, which is the object to be measured. The reference mirror 113 is configured to be driven in the optical axis direction by a drive unit including a motor (not shown) or the like, and changes the optical path length of the reference light relative to the optical path length of the measurement light. Can be done. On the other hand, the measurement light incident on the optical fiber 102b is emitted from the fiber end and incident on the scanning optical system 200. These low coherence light sources 101 and a drive unit (not shown) are controlled under the control of the image processing device 300.

Next, the scanning optical system 200 will be described. The scanning optical system 200 is an optical system configured to be relatively movable with respect to the eye E to be inspected. A scanning optical system 200, a collimating optical system 202, a scanning unit 203, and a lens 204 are provided. The scanning optical system 200 is configured to be able to be driven in the front-back, up-down, left-right directions with respect to the eye axis of the eye E to be inspected by a drive unit (not shown) controlled by the image processing device 300. The image processing device 300 can align the scanning optical system 200 with respect to the eye E to be inspected by controlling a drive unit (not shown).

The measurement light emitted from the fiber end of the optical fiber 102b is substantially parallelized by the collimating optical system 202 and is incident on the scanning unit 203. The scanning unit 203 has two galvano mirrors whose mirror surface can be rotated, one of which deflects light in the horizontal direction and the other of which deflects light in the vertical direction, and the light incident under the control of the image processing apparatus 300. Is deflected. As a result, the scanning unit 203 can scan the measurement light on the fundus Er of the eye E to be inspected in two directions, the main scanning direction in the paper surface and the sub-scanning direction in the direction perpendicular to the paper surface. The main scanning direction and the sub-scanning direction are not limited to this, and may be any direction that is orthogonal to the depth direction of the eye E to be inspected and intersects with each other. Further, the scanning unit 203 may be configured by using any changing means, and may be configured by using, for example, a MEMS mirror or the like that can deflect light in the biaxial direction with one sheet.

The measurement light scanned by the scanning unit 203 forms an illumination spot on the fundus Er of the eye E to be inspected via the lens 204. When the in-plane deflection is received by the scanning unit 203, each illumination spot moves (scans) on the fundus Er of the eye E to be inspected. The reflected light at the illumination spot position follows the optical path in the opposite direction, enters the optical fiber 102b, and returns to the optical branch portion 103.

As described above, the reference light reflected by the reference mirror 113 and the measurement light reflected by the fundus Er of the eye E to be inspected are returned to the optical branching portion 103 as return light and interfere with each other to generate interference light. The interference light passes through the optical fiber 102d, and the interference light emitted to the collimating optical system 122 is substantially parallelized and incident on the diffraction grating 123. The diffraction grating 123 has a periodic structure and disperses the input interference light. The dispersed interference light is imaged on the line sensor 125 by the imaging lens 124 whose focusing state can be changed. The line sensor 125 is connected to the image processing device 300, and outputs a signal corresponding to the intensity of the light applied to each sensor unit to the image processing device 300.

Further, the OCT apparatus may be provided with a fundus camera (not shown) for capturing a frontal image of the fundus of the eye to be inspected E, an optical system of a scanning laser Ophthalmoscopy (SLO), and the like. In this case, a part of the SLO optical system may have an optical path common to a part of the scanning optical system 200.

(Image processing device)
FIG. 2 shows a schematic functional configuration example of the image processing apparatus 300. As shown in FIG. 2, the image processing device 300 is provided with a reconstruction unit 301, a motion contrast image generation unit 302, a layer recognition unit 303, an image generation unit 304, a storage unit 305, and a display control unit 306. .. The image processing apparatus 300 according to the present embodiment is connected to the optical interference unit 100 using the spectral domain (SD) method, and can acquire the output data of the line sensor 125 of the optical interference unit 100. The image processing device 300 may be connected to an external device (not shown) to acquire an interference signal of the eye to be inspected, a tomographic image, or the like from the external device.

The reconstruction unit 301 generates the tomographic data of the eye to be inspected E by wave number transforming the acquired output data (interference signal) of the line sensor 125 and Fourier transforming it. Here, the tomographic data is data including information on the tomographic of the subject, and includes a signal obtained by subjecting an interference signal by OCT to Fourier transform, a signal obtained by subjecting the signal to an arbitrary process, and the like. In addition, the reconstruction unit 301 can also generate a tomographic image as tomographic data based on the interference signal. The reconstructing unit 301 may generate tomographic data based on the interference signal of the eye to be inspected acquired by the image processing device 300 from the external device. Although the OCT apparatus according to the present embodiment includes the SD type optical interference unit 100, it may also include a time domain (TD) type or wavelength sweep (SS) type optical interference unit.

The motion contrast image generation unit 302 generates motion contrast data from a plurality of tomographic data. The method of generating motion contrast data will be described later. The motion contrast image generation unit 302 can generate three-dimensional motion contrast data from a plurality of three-dimensional tomographic data. In the following, three-dimensional tomographic data and three-dimensional motion contrast data are collectively referred to as three-dimensional volume data.

The layer recognition unit 303 analyzes the generated tomographic data of the eye E to be inspected and performs segmentation for identifying an arbitrary layer structure in the retinal layer. The segmented result serves as a reference for the projection range when generating the OCTA front image as described later. For example, the layer boundary line shapes detected by the layer recognition unit 303 include ILM, NFL / GCL, GCL / IPL, IPL / INL, INL / OPL, OPL / ONL, IS / OS, OS / RPE, RPE / Choroid, and There are 10 types of BM. The object detected by the layer recognition unit 303 is not limited to this, and may be any structure included in the eye E to be inspected. As the segmentation method, any known method may be used.

The image generation unit 304 generates an image for display from the generated tomographic data and motion contrast data. The image generation unit 304, for example, is an OCTA front image (OCTA En-Face) in which a 3D tomographic data is projected onto a 2D plane or an integrated brightness En-Face image or a 3D motion contrast data is projected onto a 2D plane. Image) can be generated. Further, the image generation unit 304 can generate an OCTA front image or an En-Face image of brightness according to the depth range acquired from the storage unit 305. The method of generating the OCTA front image and the like will be described later. Further, the image generation unit 304 can also generate a fundus frontal image based on a signal acquired from a fundus camera (not shown) or an SLO optical system.

The display control unit 306 outputs the generated display image to the display unit 310. The storage unit 305 stores tomographic data and motion contrast data generated by the reconstruction unit 301, an image for display generated by the image generation unit 304, definitions of a plurality of depth ranges, definitions applied by default, and the like. be able to. Further, the storage unit 305 may include software or the like in order to realize each unit.

Further, a display unit 310, a pointing device 320, and a keyboard 321 are connected to the image processing device 300. The display unit 310 can be configured by using any monitor.

The pointing device 320 is a mouse provided with a rotary wheel and buttons, and can specify an arbitrary position on the display unit 310. Although the mouse is used as the pointing device in this embodiment, any pointing device such as a joystick, a touch pad, a trackball, a touch panel, or a stylus pen may be used.

As described above, the OCT device according to the present embodiment is configured by using the optical interference unit 100, the scanning optical system 200, the image processing device 300, the display unit 310, the pointing device 320, and the keyboard 321. In this embodiment, the optical interference unit 100, the scanning optical system 200, the image processing device 300, the display unit 310, the pointing device 320, and the keyboard 321 have separate configurations, but all or one of them is configured. The parts may be integrally configured. For example, the display unit 310 and the pointing device 320 may be integrally configured as a touch panel display. Similarly, a fundus camera (not shown) and an SLO optical system may be configured as separate devices.

The image processing device 300 may be configured using, for example, a general-purpose computer. The image processing device 300 may be configured by using a dedicated computer of the OCT device. The image processing device 300 includes a processor (not shown) and a storage medium including a memory such as an optical disk and a ROM (Read Only Memory). The processor may be a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like. The processor is not limited to the CPU and the MPU, and may be a GPU (Graphics Processing Unit), an FPGA (Field-Programmable Gate Array), or the like. Each component other than the storage unit 305 of the image processing device 300 may be configured by a software module executed by a processor such as a CPU, MPU, or GPU. Further, each component may be configured by a circuit that performs a specific function such as an ASIC, an independent device, or the like. The storage unit 305 may be configured by any storage medium such as an optical disk or a memory.

The image processing unit 300 includes a processor such as a CPU and a storage medium such as a ROM, which may be one or a plurality. Therefore, when each component of the image processing device 300 is connected to at least one processor and at least one storage medium, and at least one processor executes a program stored in at least one storage medium. It may be configured to work. Further, each component of the image processing device 300 may be realized by a separate device.

(Control method for tomographic imaging)
Next, a control method for taking a tomographic image of the eye E to be inspected will be described using the OCT apparatus according to the present embodiment.

First, the examiner seats the patient who is the subject in front of the scanning optical system 200, inputs alignment, patient information, and the like, and then starts OCT imaging. The light emitted from the low coherence light source 101 passes through the optical fiber 102a and is divided into measurement light toward the eye E to be inspected and reference light toward the reference mirror 113 at the optical branching portion 103.

The measurement light directed to the eye E to be inspected passes through the optical fiber 102b, is emitted from the fiber end, is substantially paralleled by the collimating optical system 202, and is incident on the scanning unit 203. The scanning unit 203 has a galvano mirror, and the measurement light deflected by the mirror irradiates the eye E to be inspected via the lens 204. Then, the reflected light reflected by the eye E to be inspected follows the path in the reverse direction and is returned to the optical branching portion 103.

On the other hand, the reference light directed to the reference mirror 113 passes through the optical fiber 102c, is emitted from the fiber end, and reaches the reference mirror 113 through the collimating optical system 111 and the dispersion-compensated optical system 112. The reference light reflected by the reference mirror 113 is returned to the optical branching portion 103 by following the path in the reverse direction.

The measurement light and the reference light that have returned to the optical branching portion 103 interfere with each other, become interference light and enter the optical fiber 102d, are substantially collimated by the collimated optical system 122, and enter the diffraction grating 123. The interference light input to the diffraction grating 123 is imaged on the line sensor 125 by the imaging lens 124. Thereby, the line sensor 125 can be used to obtain an interference signal at one point on the eye E to be inspected.

The interference signal acquired by the line sensor 125 is output to the image processing device 300. The interference signal output from the line sensor 125 is 12-bit integer format data. The reconstruction unit 301 performs wave number conversion, fast Fourier transform (FFT), and absolute value conversion (acquisition of amplitude) on the 12-bit integer format data, and performs a fault in the depth direction at one point on the eye E to be inspected. Generate data. The data format of the interference signal and the like may be arbitrarily set according to the desired configuration.

After acquiring the interference signal at one point on the eye E to be inspected, the scanning unit 203 drives the galvano mirror and scans the measurement light to one adjacent point on the eye E to be inspected. The line sensor 125 detects the interference light based on the measurement light and acquires the interference signal. The reconstruction unit 301 generates tomographic data in the depth direction at the adjacent point on the eye E to be inspected based on the interference signal of the adjacent point. By repeating this series of controls, it is possible to generate tomographic data (two-dimensional tomographic data) relating to one tomographic image in one transverse direction (main scanning direction) of the eye E to be inspected.

Further, the scanning unit 203 drives a galvano mirror and scans the same location (same scanning line) of the eye E to be examined a plurality of times to acquire a plurality of tomographic data (two-dimensional tomographic data) at the same location of the eye E to be inspected. .. Further, the scanning unit 203 drives the galvano mirror to move the measurement light minutely in the sub-scanning direction orthogonal to the main scanning direction, and a plurality of tomographic data (adjacent scanning lines) at another location (adjacent scanning line) of the eye E to be inspected (adjacent scanning line). Two-dimensional tomographic data) is acquired. By repeating this control, it is possible to acquire tomographic data (three-dimensional tomographic data) relating to a plurality of three-dimensional tomographic images in a predetermined range of the eye E to be inspected.

In the above, one tomographic data at one point of the eye E to be inspected is acquired by performing FFT processing on a set of interference signals obtained from the line sensor 125. However, it is also possible to divide the interference signal into a plurality of sets, perform FFT processing on each of the divided interference signals, and acquire a plurality of tomographic data from one interference signal. According to this method, it is possible to acquire more tomographic data than the number of times that the same portion of the eye E to be inspected is actually scanned.

(Motion contrast data generation)
Next, a method of generating motion contrast data from tomographic data in the image processing apparatus 300 will be described.

The complex number format tomographic data generated by the reconstruction unit 301 is output to the motion contrast image generation unit 302. First, the motion contrast image generation unit 302 corrects the positional deviation of a plurality of tomographic data (two-dimensional tomographic data) at the same location of the eye E to be inspected. As the method for correcting the misalignment, any known method may be used. For example, the reference tomographic data may be selected as a template and the amount of misalignment with the template may be acquired as the amount of misalignment related to each tomographic data. ..

ここで、Ａｘｚは断層データＡの位置（ｘ，ｚ）における振幅、Ｂｘｚは断層データＢの同一位置（ｘ，ｚ）における振幅を示している。結果として得られる脱相関値Ｍｘｚは０から１までの値を取り、二つの振幅値の差異が大きいほど１に近い値となる。 The motion contrast image generation unit 302 obtains the decorrelation value between the two two-dimensional tomographic data in which the positional deviation has been corrected by the following equation (1).

Here, Axz indicates the amplitude of the tomographic data A at the position (x, z), and Bxz indicates the amplitude of the tomographic data B at the same position (x, z). The resulting decorrelation value Mxz takes a value from 0 to 1, and the larger the difference between the two amplitude values, the closer to 1.

The motion contrast image generation unit 302 obtains a plurality of decorrelation values by repeating the above decorrelation calculation for the number of acquired tomographic data, and obtains the average value of the plurality of decorrelation values to obtain the final motion. Acquire contrast data. Here, the motion contrast data is obtained based on the amplitude of the complex number data after the FFT, but the method of obtaining the motion contrast data is not limited to the above method. The motion contrast data may be obtained based on the phase information of the complex number data, or the motion contrast data may be obtained based on both the amplitude and phase information. It is also possible to obtain motion contrast data based on the real part and the imaginary part of the complex number data. Further, the motion contrast image generation unit 302 may perform the same processing on each pixel value of the two-dimensional tomographic image to obtain motion contrast data.

Further, in the above method, the motion contrast data is acquired by calculating the decorrelation value of the two values, but the motion contrast data may be obtained based on the difference between the two values, or the ratio of the two values may be obtained. Motion contrast data may be obtained based on the above. Further, the motion contrast data may be obtained based on the dispersion value of the tomographic data. Further, in the above, the final motion contrast data is obtained by obtaining the average value of the acquired multiple decorrelation values, but the final values such as the multiple decorrelation values and differences, the maximum value and the median of the ratio are finally obtained. It may be used as motion contrast data. The two tomographic data used when acquiring the motion contrast data may be the data acquired at a predetermined time interval.

(Generation of OCTA front image)
Next, in the image processing apparatus 300, a procedure for defining a depth range for generating an OCTA front image will be described.

The OCTA front image is a front image obtained by projecting or integrating a three-dimensional motion contrast image (three-dimensional motion contrast data) onto a two-dimensional plane in an arbitrary depth range. The depth range can be set arbitrarily, but generally, from the retina to the choroid side, the superficial retinal layer (Superficial Capillary), the deep retinal layer (Deep Capillary), the outer layer of the retina (Outer Retina), and the radial peri-papillary capillaries ( RPCs: Radial Peri-Pillary Capillaries), Choroidal Capillaries, and Lamina Cribrosa are defined depth ranges.

Each definition is defined for the layer boundaries of the retina, for example the Superficial Capillary is defined as ILM + 0 μm to GCL / IPL + 50 μm. Here, GCL / IPL means the boundary between the GCL layer and the IPL layer. Further, in the following, an offset amount such as +50 μm or -100 μm means that a positive value shifts to the choroid side and a negative value shifts to the pupil side.

When identifying new blood vessels with exudative age-related macular degeneration, the outer layer of the retina and the choroidal capillary plate are often used as the depth range. The outer layer of the retina is often defined as OPL / ONL + 0 μm to RPE / Choroid + 0 μm, but as will be described later, the depth range can be adjusted depending on the size of CNV, the location of occurrence (depth position), and the like.

As a method of projecting the data corresponding to the depth range on the two-dimensional plane, for example, a method of using the representative value of the data in the depth range as the pixel value on the two-dimensional plane can be used. Here, the representative value may include a value such as an average value, a median value, or a maximum value of pixel values within the depth range.

The brightness En-Face image is a front image obtained by projecting or integrating a three-dimensional tomographic image onto a two-dimensional plane in an arbitrary depth range. The brightness En-Face image may be generated in the same manner as the OCTA front image by using a three-dimensional tomographic image instead of the three-dimensional motion contrast image. Further, the brightness En-Face image may be generated by using the three-dimensional tomographic data.

Further, the depth range for the OCTA front image and the En-Face image is determined based on the retinal layer detected by the segmentation process for the 2D tomographic data (or the 2D tomographic image) of the 3D volume data. Can be. Further, the depth range may be a range including a predetermined number of pixels in a deeper direction or a shallower direction with respect to one of the two layer boundaries relating to the retinal layer detected by these segmentation processes.

Further, the depth range may be configured so as to be changed according to a desired configuration. For example, the depth range can range from the range between the two layer boundaries with respect to the detected retinal layer to a range modified (offset) in response to instructions from the operator of the pointing device 320. At this time, the operator can change the depth range by operating the pointing device 320, for example, by moving an index indicating the upper limit or the lower limit of the depth range superimposed on the tomographic image.

(Image generator)
FIG. 3 is a diagram for explaining the image generation unit 304. The image generation unit 304 includes a projection range control unit 341 and a front image generation unit 342. The projection range control unit 341 is a front image based on the motion contrast image generated by the motion contrast image generation unit 302, the layer recognition result by the layer recognition unit 303, and the depth range stored in the storage unit 305. Specify the 3D motion contrast data used for generation. The front image generation unit 342 projects or integrates the motion contrast data specified by the projection range control unit 341 on a two-dimensional plane to generate an OCTA front image.

Similarly, the projection range control unit 341 uses a three-dimensional tomographic image (three-dimensional tomographic data), a layer recognition result, and a three-dimensional tomographic data used to generate an En-Face image of brightness based on the depth range. Can be identified. In this case, the front image generation unit 342 can project or integrate the tomographic data specified by the projection range control unit 341 on a two-dimensional plane to generate an En-Face image of luminance.

(Report screen)
FIG. 4 shows an example of the GUI 400 for displaying an image including an OCTA front image generated by the image generation unit 304. In the GUI 400, a tab 401 for screen selection is shown, and in the example shown in FIG. 4, a report screen (Report tab) is selected. In addition to the report tab, the GUI 400 may include a patient screen (Patient tab) for selecting a patient, an imaging screen (OCT Capture tab) for performing imaging, and the like.

An inspection selector 408 is provided on the left side of the report screen, and a display area is provided on the right side. The test selector 408 displays a list of tests performed so far for the currently selected patient, and when one of them is selected, the display control unit 306 displays the test results in the display area on the right side of the report screen. Let me.

In the display area, an SLO image 406 generated by using an SLO optical system (not shown) is shown, and an OCTA front image is superimposed and displayed on the SLO image 406. Further, in the display area, the first OCTA front image 402, the first tomographic image 403, the luminance En-Face image 407, the second OCTA front image 404, and the second tomographic image 405 are displayed. .. A pull-down is provided on the upper part of the En-Face image 407, and Enface Image 1 is selected. This means that the depth range of the En-Face image is the same as the depth range of OCTA Image 1 (first OCTA front image 402). By operating the pull-down of the pointing device 320 from the operator, the depth range of the En-Face image can be made the same as the depth range of the OCTIimage 2 (second OCTA front image 404) and the like.

In the first tomographic image 403, the depth range when the first OCTA frontal image 402 is generated is shown by a broken line on the tomographic image. In the example of GUI400, the depth range of the first OCTA front image 402 is the Superficial Capillary (SCP). Further, the second OCTA front image 404 is an image generated using data in a depth range different from that of the first OCTA front image 402. In the second tomographic image 405, the depth range when the second OCTA front image 404 is generated is shown by a broken line. Here, the depth range of the second OCTA front image 404 is CNV, and in this example, it is in the range of OPL / ONL + 50 μm to BM + 10 μm.

The depth range of the first OCTA front image 402 and the second OCTA front image 404 can be set according to the pull-down operation provided on the upper part of these images. Further, these depth ranges may be set in advance, or may be set according to an instruction from the operator of the pointing device 320. Here, the image generation unit 304 can function as a target designation unit for designating the extraction target to be the target of the following extraction process based on the setting. The pull-down for setting the depth range of the OCTA front image corresponds not only to the depth range for each layer such as the superficial retinal layer described above, but also to abnormal parts such as CNV that are desired to be extracted by the following processing. The range may be included.

In this embodiment, when the second OCTA front image 404 is double-clicked according to the instruction of the pointing device 320 from the operator, the display control unit 306 changes from the GUI 400 shown in FIG. 4 to the GUI 500 shown in FIG. Switch the screen to be displayed on. The GUI500 includes OCTA frontal images 501,505,509,513 and corresponding tomographic images 503,507,511,515 and depth ranges 502,506,510,514 for four different depth range settings. Is displayed.

In this regard, the image generation unit 304 designates the CNV corresponding to the depth range of the second OCTA front image 404 as the extraction target. The image generation unit 304, based on the four depth ranges 502,506,510,514 pre-stored in the storage unit 305 for the CNV designated as the extraction target, the corresponding OCTA front image 501,505,509, Generate 513. The display control unit 306 causes the display unit 310 to display the generated OCTA front image 501, 505, 509, 513, the corresponding tomographic image 503, 507, 511, 515, and the depth range 502, 506, 510, 514.

In this example, as the depth range 502 for the leftmost image (OCTA front image 501), a depth range assuming a Type 1 CNV is set, and is BM + 0 μm to BM + 20 μm. Here, the type 1 CNV means a CNV below the RPE / Choroid.

The depth range 506 for the second image from the left (OCTA front image 505) is a depth range assuming a very small CNV slightly above the BM, and is BM-20 μm to BM + 0 μm. The depth range 510 for the third image from the left (OCTA front image 509) is a depth range assuming a large CNV generated above the BM, and is BM-100 μm to BM + 0 μm. The depth range 514 for the fourth image from the left (OCTA front image 513) is a depth range that covers the entire outer layer of the retina, and is a depth range (OPL + 50 μm to BM + 10 μm) assuming a considerably large CNV.

Selection buttons 504, 508, 512, 516 are displayed at the bottom of the GUI 500. By selecting the selection button according to the instruction of the pointing device 320 from the operator, an OCTA front image (front image to be displayed) preferable for use in diagnosis or the like can be selected from the displayed OCTA front images.

In this embodiment, when the selection button 512 is pressed by the instruction of the pointing device 320 from the operator, the display control unit 306 switches the screen displayed on the display unit 310 to the GUI 600 of the report screen shown in FIG. .. The GUI 600 is similar to the GUI 400, but the OCTA front image 509 and the OCTA front based on the conditions selected by the second OCTA front image 404 and the second tomographic image 405 according to the instruction of the pointing device 320 from the operator. It has been switched to the tomographic image 511 showing the depth range for image 509.

As described above, in this embodiment, the OCTA front image regarding a plurality of depth ranges according to the extraction target is provided to the operator, and the operator selects a preferable image to display the OCTA front image for the optimum depth range. be able to. As a result, the risk of oversight of lesions can be reduced, and additional work such as image quality adjustment can be reduced for doctors and laboratory technicians.

Next, a series of processes according to the present embodiment will be described with reference to FIGS. 7 and 8. FIG. 7 is a flowchart of a series of processes according to the present embodiment, and FIG. 8 is a flowchart of the front image generation process according to the present embodiment. When a series of processing is started, first, in step S701, the image processing apparatus 300 acquires a three-dimensional interference signal relating to the eye to be inspected E from the line sensor 125 of the optical interference unit 100, and the reconstruction unit 301 obtains three-dimensional tomographic data. Is generated and acquired. At this time, the reconstruction unit 301 can also generate a three-dimensional tomographic image based on the three-dimensional tomographic data. The image processing device 300 may acquire a three-dimensional interference signal, three-dimensional interference data, a three-dimensional tomographic image, or the like related to the eye to be inspected E from a connected external device (not shown).

When the three-dimensional tomographic data is generated by the reconstruction unit 301, the motion contrast image generation unit 302 generates the three-dimensional motion contrast data (three-dimensional motion contrast image) based on the three-dimensional tomographic data.

Next, in step S702, the image generation unit 304 designates the extraction target (target area) according to the preset setting or the instruction of the pointing device 320 from the operator. At this time, the layer recognition unit 303 can perform segmentation on the three-dimensional tomographic data and acquire the layer recognition result. Further, the image generation unit 304 generates the first OCTA front image 402 and the like based on the three-dimensional volume data, the layer recognition result, the setting of a predetermined depth range, and the like, and the display control unit 306 displays the GUI 400 on the display unit 310. It may be displayed in. In this case, according to the instruction of the pointing device 320 from the operator, the pull-down or the like regarding the OCTA front image can be operated to input the instruction regarding the extraction target. When the extraction target is specified, the process proceeds to step S703.

In step S703, the image generation unit 304 starts the front image generation process according to this embodiment. In the front image generation process according to the present embodiment, first, in step S801, the image generation unit 304 specifies a plurality of depth ranges stored in the storage unit 305 corresponding to the designated extraction target. Further, the projection range control unit 341 of the image generation unit 304 specifies the 3D motion contrast data used for generating the OCTA front image based on the specified plurality of depth ranges, 3D motion contrast data, and the layer recognition result. .. The front image generation unit 342 generates a plurality of OCTA front images corresponding to a plurality of depth ranges based on the specified three-dimensional motion contrast data.

In step S802, the display control unit 306 causes the display unit 310 to display the generated plurality of OCTA front images. At this time, the display control unit 306 can display the information regarding the corresponding depth range on the display unit 310 together with the generated plurality of OCTA front images. Here, the information regarding the corresponding depth range may be numerical information indicating the depth range, a broken line indicating the depth range on the tomographic image, or both.

In step S803, the pointing device 320 receives an instruction from the operator and designates an OCTA front image preferable for diagnosis or the like from a plurality of OCTA front images displayed on the display unit 310. The image processing device 300 selects an OCTA front image to be displayed in response to an instruction from the operator of the pointing device 320. The instruction of the pointing device 320 from the operator may be given, for example, by selecting a selection button in the GUI 500 shown in FIG.

When the image processing device 300 selects the OCTA front image to be displayed, in step S704, the display control unit 306 causes the display unit 310 to display the selected OCTA front image. This makes it possible to display an OCTA front image that makes it easy to confirm the target structure. In this embodiment, the OCTA front image is generated and displayed, but the generated / displayed image may be an En-Face image having luminance. In this case, the same process as the above process may be performed by using the tomographic data instead of the motion contrast data.

As described above, the image processing apparatus 300 according to the present embodiment includes an image generation unit 304 and a display control unit 306. The image generation unit 304 also functions as an example of a target designation unit that designates an extraction target (target area) from the three-dimensional volume data of the eye E to be inspected. The front image generation unit 342 of the image generation unit 304 generates a plurality of front images corresponding to different depth ranges of the three-dimensional volume data based on the designated extraction target. The display control unit 306 controls the display of the display unit 310. The projection range control unit 341 of the image generation unit 304 determines the depth range for generating a plurality of front images based on the designation of the extraction target. The display control unit 306 displays the generated plurality of front images side by side on the display unit 310. In particular, in the image processing apparatus 300 according to the present embodiment, the extraction target is a neovascularization (CNV), and the three-dimensional volume data is three-dimensional motion contrast data. Further, each depth range for generating a plurality of front images is, for example, a depth range within a range of 0 to 50 μm from the outer layer of the retina or the Bruch membrane to the choroid side.

In this embodiment, the projection range control unit 341 of the image generation unit 304 determines the depth range for generating a plurality of front images based on the designation of the extraction target. Here, the projection range control unit 341 is, for example, the type of three-dimensional volume data, the layer or depth range to be extracted, the number of front images to be generated, the depth range to generate the front image, and the depth to generate the front image. At least one of the range intervals can be determined.

Further, in this embodiment, as in the GUI 500 shown in FIG. 5, an image corresponding to a plurality of depth ranges is displayed separately from the GUI 400 shown in FIG. 4, but the present invention is not limited to this. For example, images in a plurality of depth ranges may be displayed side by side on the GUI 400. Further, an image having a plurality of depth ranges may be displayed while being temporally switched to the display area of the second OCTA front image 404 of the GUI 400, or may be switched and displayed according to an instruction from the operator of the pointing device 320. You may. In this case, an image in a preferable depth range may be selected from the images displayed by switching by an operation on the GUI 400.

In this embodiment, the OCTA front image corresponding to a plurality of depth ranges is displayed as in the GUI 500 shown in FIG. 5, but the present invention is not limited to this. For example, the images whose projection method has been changed may be displayed side by side together with the depth range, and the operator may select the images. Here, the projection method may be any known method such as maximum value projection or average value projection. Even within the same depth range, the appearance of the front image changes depending on the projection method. Therefore, in such a case, the operator can be made to select the front image corresponding to the preferable projection method.

According to such a configuration, by providing the operator with front images relating to a plurality of depth ranges and projection methods according to the extraction target, it is possible to display a front image in which it is easy to confirm the target structure. Can be done. As a result, the risk of oversight of lesions can be reduced, and additional work such as image quality adjustment can be reduced for doctors and laboratory technicians.

(Variation example of Example 1)
In the first embodiment, an example is shown in which the front image of a plurality of depth ranges is displayed by double-clicking the front image according to the instruction of the pointing device 320 from the operator, but the front image of a plurality of depth ranges is displayed. The processing at the time of making is not limited to this. For example, at the stage of displaying the report screen of the GUI 400, a front image corresponding to a plurality of depth ranges set for the target disease may be displayed and the operator may select the front image.

In another example, it is determined whether or not there is an abnormality such as CNV in the eye E to be inspected when the examination is performed, and if it is determined that there is an abnormality, a front image for a plurality of depth ranges is displayed. The operator may be allowed to select the most suitable image. In addition, when an OCTA test is performed on a patient with a disease such as a patient with exudative age-related macular degeneration having CNV, a frontal image corresponding to a plurality of depth ranges set for the disease is displayed. , The operator may be allowed to select the most suitable image. It should be noted that the determination as to whether or not there is an abnormality may be performed by any known method.

(Example 2)
In the first embodiment, a plurality of OCTA front images corresponding to a plurality of depth ranges are displayed, and a preferable image among them is selected according to an instruction of the pointing device 320 from the operator, so that the OCTA front projected in a preferable depth range is projected. The image was provided. The image processing apparatus according to the second embodiment further provides a front image determination unit 343 in the image generation unit 304, and automatically outputs an OCTA front image having an optimum depth range without the intervention of an operator.

Hereinafter, the image processing apparatus according to the present embodiment will be described with reference to FIGS. 9 to 12. The configuration of the image processing apparatus according to the present embodiment is the same as the configuration of the image processing apparatus according to the first embodiment except that the front image determination unit 343 is added to the image generation unit 304. The description will be omitted by using reference numerals. Hereinafter, the image processing apparatus according to the present embodiment will be described focusing on the difference from the image processing apparatus 300 according to the first embodiment.

FIG. 9 is a diagram for explaining the image generation unit 304 according to this embodiment. As shown in the figure, the image generation unit 304 according to this embodiment is provided with a front image determination unit 343 in addition to the projection range control unit 341 and the front image generation unit 342. Here, the front image determination unit 343 may be configured by a software module executed by a processor such as a CPU, MPU, or GPU. Further, the front image determination unit 343 may be configured by a circuit that performs a specific function such as an ASIC, an independent device, or the like.

The frontal image determination unit 343 evaluates the OCTA frontal image corresponding to the plurality of depth ranges generated by the frontal image generation unit 342, and has the existence probability of the pathological unit corresponding to the highest (higher than others) evaluation value. The OCTA front image is determined. In this embodiment, the front image determination unit 343 may be configured by a software module executed by a processor such as a CPU, MPU, or GPU. Further, the front image determination unit 343 may be configured by a circuit that performs a specific function such as an ASIC, an independent device, or the like. Further, in this example, the pathological part is described as a new blood vessel (CNV), but the pathological part is not limited to CNV, and may be another pathological part such as a capillary aneurysm as in Example 1.

Next, with reference to FIG. 10, the front image generation process according to this embodiment will be described. FIG. 10 is a flowchart of the front image generation process according to the present embodiment. Since the flow of a series of processes other than the front image generation process is the same as the series of processes according to the first embodiment, the description thereof will be omitted. Further, since step S1001 is the same step as step S801 according to the first embodiment, the description thereof will be omitted. When the OCTA front image having a plurality of depth ranges is generated in step S1001, the process proceeds to step S1002.

In step S1002, the front image determination unit 343 compares the OCTA front images of a plurality of depth ranges generated by the front image generation unit 342 using the trained model, and the optimum depth range for observing the CNV. Determine and select the OCTA front image. Since the subsequent processing is the same as the processing according to the first embodiment, the description thereof will be omitted.

Here, with reference to FIG. 11, the front image determination process in step S1002 will be described in more detail. When the front image determination process is started in step S1002, the process proceeds to step S1110. It is assumed that the plurality of front images generated by the front image generation unit 342 are held as the sequence Img [].

In step S1110, the front image determination unit 343 initializes the variable AW to the first image Img [0] stored in the array. Here, the variable AW is a variable indicating a winner having a high probability of existence of CNV.

In step S1120, the front image determination unit 343 initializes the front image index t to 1. Further, the front image determination unit 343 initializes the index i to 0.

In step S1130, the front image determination unit 343 compares the current winner AW with the next image Img [t] in the image sequence using the trained model, and obtains an evaluation value for evaluating the presence of CNV in each image. .. Here, the trained model used by the front image determination unit 343 will be described with reference to FIG. In this embodiment, the trained model used by the front image determination unit 343 is provided in the image processing device 300, and may be configured by a software module executed by a processor such as a CPU, MPU, or GPU. Further, the trained model may be configured by a circuit that performs a specific function such as an ASIC, an independent device, or the like.

FIG. 12 shows an example of the configuration of a trained model for image comparison used by the front image determination unit 343. In this embodiment, the input data of the trained model 1210 for comparison is composed of a pair of OCTA front images 1201 and 1202. Further, the output data of the trained model 1210 is composed of weight vectors (referred to as evaluation values) w1, w2, and w3. The trained model can be configured to have a plurality of channels. In this embodiment, for a pair of OCTA front images 1201, 1202, the OCTA front image 1201 is input to the first input channel of the trained model, and the OCTA front image 1202 is input to the second input channel of the trained model. The configuration was set.

Here, the evaluation value is a value associated with the content of the input image. In this embodiment, the evaluation value w1 and the evaluation value w2 correspond to the image quality (probability) for evaluating the existence of the pathological structure according to the learning process described below. On the other hand, the evaluation value w3 corresponds to the probability of showing the absence of the pathological structure in both input images, and when both input images have no pathological structure, the evaluation value w3 is the highest. Further, in this embodiment, the total of the evaluation values is w1 + w2 + w3 = 1, and the evaluation values w1, w2, and w3 are the presence of CNV for the first input image and the presence of CNV for the second input image, respectively. And the probability that CNV is not included in either input image is shown.

In this embodiment, the trained model 1210 can be configured, for example, with an Xception convolutional neural network (XCNN) module available in the Keras framework (“Keras Documentation”, 2019). Hereinafter, CNN will be described as an example of the trained model according to this embodiment. The CNN according to this embodiment is composed of a plurality of layers responsible for processing and outputting an input value group. The types of layers included in the configuration of the trained model include a convolution layer, a Downsampling layer, an Upsampling layer, and a Merger layer.

The convolution layer is a layer that performs convolution processing on the input value group according to parameters such as the kernel size of the set filter, the number of filters, the stride value, and the dilation value. The input may have multiple channels. For example, when the input is a color image, there are three channels of R / G / B. In this case, the convolution process is performed using a 3-channel filter. In this way, the number of dimensions of the kernel size of the filter may be changed according to the number of dimensions of the input image.

The downsampling layer is a layer that performs processing to reduce the number of output value groups to be smaller than the number of input value groups by thinning out or synthesizing input value groups. Specifically, as such a process, for example, there is a Max Polling process.

The upsampling layer is a layer that performs processing to increase the number of output value groups to be larger than the number of input value groups by duplicating the input value group or adding interpolated values from the input value group. Specifically, as such a process, for example, there is a linear interpolation process.

The composite layer is a layer in which a value group such as an output value group of a certain layer or a pixel value group constituting an image is input from a plurality of sources, and the processing is performed by concatenating or adding them.

As parameters set in the convolutional layer group included in the CNN configuration, for example, the kernel size of the filter is 3 pixels in width, 3 pixels in height, and the number of filters is 64, so that the comparison process has a certain accuracy. Is possible. However, it should be noted that if the parameter settings for the layers and nodes that make up the neural network are different, the degree to which the tendency trained from the teacher data can be reproduced in the output data may differ. That is, in many cases, the appropriate parameters differ depending on the embodiment, and therefore, the values can be changed to preferable values as needed.

In addition to the method of changing the parameters as described above, there are cases where the CNN can obtain better characteristics by changing the configuration of the CNN. Better characteristics include, for example, high accuracy of comparison processing, short comparison processing time, short training time of machine learning model, and the like.

Here, the learning data according to this embodiment will be described. For the learning data according to this embodiment, two OCTA front images in the database were used as input data, and evaluation results selected by an expert such as a doctor for the two OCTA front images were used as correct answer data. Here, the database consisted of examination data for eyes with CNV and examination data for healthy eyes. The correct answer data was selected by a specialist such as a doctor as the OCTA front image (best image) having the optimum depth range for the eye having CNV among the two OCTA front images corresponding to the input data. The evaluation value w1 or the evaluation value w2 of the image was set to 1, and the other evaluation values were set to 0. Further, when it was determined by an expert such as a doctor that CNV did not appear in the two OCTA front images corresponding to the input data, the evaluation values w1 and w2 were set to 0, and the evaluation value w3 was set to 1. The evaluation value for the correct answer data is not limited to 1 or 0, and an evaluation value after the decimal point may be set by a specialist such as a doctor. However, even in this case, each evaluation value is set so that the total of the evaluation values is 1.

Table 1 shown below shows a combination of learning data according to this embodiment. "Best" in Table 1 means that the corresponding input data is the best image for CNV observation among OCTA frontal images in multiple depth ranges. Further, "other" means that the corresponding input data is an image in a depth range other than "best" among OCTA front images in a plurality of depth ranges. Further, "health" means that the two input data are images obtained from healthy eye data, assuming that the two input data are not the same image (the front image index t is different). The learning data according to this embodiment is only these three patterns, and the machine learning model is made to learn which image is superior to the observation of CNV or neither.

By using such training data, it is possible to perform training on comparison of OCTA front images suitable for observing CNV, and by using a trained model that has undergone such training, highly reliable comparison can be performed. be able to. Further, in the learning (training) of the neural network, so-called augmentation may be performed, such as rotating the image, inverting the image vertically and horizontally, and changing the range in which the image is cut. In this case, augmentation can increase the number of OCTA front images that are input data for training data.

As the OCTA front image used as the input data of the training data and the input data at the time of actual operation, an image from which projection artifacts have been removed can be used. Here, the projection artifact is a phenomenon in which blood vessels such as the surface layer of the retina are reflected in a layer below the surface layer. As an algorithm for removing projection artifacts, any known method may be used.

The machine learning model according to the presented invention is not limited to XCNN, and other types of CNN models such as InsertionV3, InjectionResnetV2, NASNet, or NASNetMobile can also be used in particular. Machine learning models can also use non-CNN neural network solutions such as multilayer perceptrons, decision trees, or support vector machines.

When data is input to the trained model of such a machine learning model, the data according to the design of the machine learning model is output. For example, output data that is likely to correspond to the input data is output according to the tendency of learning using the training data. In the trained model according to this embodiment, in the trained model 1210 of the front image determination unit 343, when a pair of OCTA front images is input, the presence of CNV with respect to the first input image in the input OCTA front image. Second, the existence of CNV for the second input image and the probability that CNV is not included in either of them are output. For example, the trained model can output evaluation values (w1, w2, w3) = (0.7, 0.2, 0.1) for two input OCTA front images.

In step S1140, the frontal image determination unit 343 determines whether the image Img [t] is superior to the current winner AW in observing CNV. Specifically, the front image determination unit 343 determines that the highest evaluation value among the evaluation values obtained by using the trained model 1210 is true. When the evaluation value w3 is the highest, the higher one of the evaluation values w1 and w2 is regarded as true. The frontal image determination unit 343 determines whether the image Img [t] is superior to the current winner AW in observing CNV based on the true rating.

For example, the evaluation value w1 was the highest in the output of the trained model when the image Img [1] was used as the first input data and the winner AW (image Img [0]) was used as the second input data. In that case, the evaluation value w1 is true. After that, the front image determination unit 343 determines in step S1140 that the image Img [1] corresponding to the first input data is superior to the current winner AW because the evaluation value w1 is true. True. On the other hand, when the evaluation value w2 is the highest, the front image determination unit 343 sets the evaluation value w2 to true. In this case, the front image determination unit 343 considers that the image Img [1] corresponding to the first input data is not superior to the current winner AW because the evaluation value w2 is true, and in step S1140. The judgment of is false.

However, when it is determined that the evaluation value w3 is the highest in all the series of comparisons in the front image determination process, the front image determination unit 343 determines that CNV does not exist in the image Img [i]. However, the winner AW can be determined without the winner AW in step S1180 described later. In this case, in step S704, the display control unit 306 can display that the CNV does not exist in the display unit 310.

If it is determined to be true in step S1140, the process proceeds to step S1150. In step S1150, the front image determination unit 343 sets (updates) the winner AW to the image Img [t]. Further, the front image determination unit 343 sets the index i to the value of the front image index t.

When the winner AW and the index i are set in step S1150, the process proceeds to step S1160. Further, even if it is determined to be false in step S1140, the process proceeds to step S1160. In step S1160, the front image determination unit 343 adds 1 to the front image index t so that the front image index t indicates the index of the next front image.

In step S1170, the front image determination unit 343 determines whether or not all OCTA front images have been compared. Specifically, the front image determination unit 343 determines whether or not the front image index t is lower than the number N of front images. If the front image determination unit 343 determines in step S1170 that the front image index t is lower (true) than the number N of front images, the process returns to step S1130. On the other hand, if the front image determination unit 343 determines in step S1170 that the front image index t is not lower than the number N of front images (false), the process proceeds to step S1180.

In step S1180, the front image determination unit 343 determines the winner AW as the image Img [i]. In such a front image determination process, when all OCTA front images are compared, the index i indicates the front image index in the optimum depth range. Therefore, in step S1180, the front image determination unit 343 determines and selects the image Img [i] as the OCTA front image having the optimum depth range for observing the CNV by determining the winner AW as the image Img [i]. be able to.

As described above, the image processing apparatus 300 according to the present embodiment includes an acquisition unit, an evaluation unit, and a determination unit. The acquisition unit acquires a plurality of front images corresponding to different depth ranges of the three-dimensional data of the eye to be inspected. Here, the front image generation unit 342 functions as an example of the acquisition unit. Further, the evaluation unit is used as input data to be input to different channels of the trained model obtained by learning using the training data including a plurality of front images corresponding to different depth ranges of the eyes to be examined. By using at least two front images out of the front images, an evaluation value for evaluating the existence of the target area is acquired. Further, the determination unit determines at least one of the plurality of front images as an output image by using the evaluation value. Here, the front image determination unit 343 functions as an example of the evaluation unit and the determination unit. The learning data according to this embodiment can include a plurality of front images corresponding to different depth ranges of the eye to be inspected and evaluation values corresponding to each front image.

According to such a configuration, it is possible to acquire and display a front image in a depth range where it is easy to confirm the target structure without the intervention of an operator, and it is possible to improve the processing efficiency. can. In this embodiment, the front image determination unit 343 functions as an example of the evaluation unit and the determination unit, but the evaluation unit and the determination unit may be provided separately. In this case, the evaluation unit and the determination unit may be included in the front image determination unit 343 as a part of the front image determination unit 343.

Further, in this embodiment, an example in which an OCTA front image is generated and displayed as a front image has been described, but as in the first embodiment, a brightness En-Face image may be generated and displayed as a front image. In this case as well, by the same front image determination process as described above, the front image determination unit 343 can determine an En-Face image having a brightness in a depth range suitable for observing the pathological unit. Further, also in the case of this embodiment, if the operator thinks that it is not preferable, the image processing device 300 may be configured so that the depth range of the generated front image can be manually adjusted.

In this example, the area of the new blood vessel (CNV) was set as the target area, and the OCTA front image was generated from the three-dimensional motion contrast data and evaluated. However, the target area is not limited to the CNV area. For example, the region of the sieve plate may be set as the target region, and an En-Face image of brightness may be generated from the three-dimensional tomographic data of the papilla and evaluated. Further, for example, the area of the capillary aneurysm may be set as the target area, and an OCTA frontal image may be generated from the three-dimensional motion contrast data of the macula and evaluated. The various modifications described above can be similarly applied to the following examples and modifications.

The front image determination process according to this embodiment is not limited to the above flow. The frontal image determination process can be considered for any other implementation of the above flow, depending on the desired configuration, according to any known programming method. For example, in this embodiment, the front image is compared so that the front image with the winner AW is compared with the next front image, but different front images are compared and the winners are compared sequentially. It may be held in a winning tournament system.

(Modified example of Example 2)
Hereinafter, a modified example of the second embodiment will be described. Since the configuration of the image processing apparatus according to the present modification is the same as the configuration of the image processing apparatus according to the second embodiment, the description thereof will be omitted using the same reference numerals. In this modification, the processing by the front image determination unit 343 is different from the processing by the front image determination unit 343 according to the second embodiment. In step S1140 of the second embodiment, the front image determination unit 343 is compared based on the evaluation value w1 or the evaluation value w2 even if the value of the evaluation value w3 obtained by using the trained model 1210 is the highest. It was determined which of the images was better. On the other hand, in this modification, when the value of the evaluation value w3 obtained by using the trained model 1210 is the highest, the front image determination unit 343 displays both the evaluation value w1 and the image corresponding to the evaluation value w2. Judging that it is inappropriate, it is not used for subsequent comparisons, so-called defeat of both parties is performed.

Hereinafter, with reference to FIGS. 13A and 13B, a portion of the front image determination process according to the present modification, which is different from the front image determination process according to the second embodiment, will be described. 13A and 13B show a flowchart of the front image determination process according to the present modification. The same reference numerals will be used for the same processing as that of the second embodiment, and the description thereof will be omitted.

In step S1130, when the front image determination unit 343 compares the winner AW with the image Img [t] and obtains an evaluation value, the process proceeds to step S1300. In step S1300, the front image determination unit 343 determines whether CNV is present in the current winner AW and the image Img [t]. More specifically, the front image determination unit 343 determines that CNV does not exist in either the current winner AW or the image Img [t] when the evaluation value w3 is the largest. However, the front image determination unit 343 determines, for example, that CNV does not exist in either the current winner AW or the image Img [t] when the evaluation value w3 exceeds a predetermined threshold value, or another determination method. May be adopted.

If the front image determination unit 343 determines in step S1300 that CNV is present (true) in the current winner AW or image Img [t], the process proceeds to step S1140. Since the processing after step S1140 is the same as the processing according to the second embodiment, the description thereof will be omitted.

On the other hand, if it is determined in step S1300 by the front image determination unit 343 that CNV does not exist (false) in the current winner AW and the image Img [t], the process proceeds to step S1310. In step S1310, the front image determination unit 343 adds 1 to the front image index t so that the front image index t indicates the index of the next front image.

In step S1320, the front image determination unit 343 determines whether or not all OCTA front images have been compared. Specifically, the front image determination unit 343 determines whether or not the front image index t is lower than the number N of front images.

If the front image determination unit 343 determines in step S1320 that the front image index t is not lower than the number N of front images (false), the process proceeds to step S1330. In step S1330, the front image determination unit 343 determines without the winner AW. In this case, in step S704, the display control unit 306 can display that the CNV does not exist in the display unit 310.

On the other hand, if the front image determination unit 343 determines in step S1320 that the front image index t is lower (true) than the number N of front images, the process proceeds to step S1340. In step S1340, the front image determination unit 343 updates the winner AW to a new image Img [t] and sets the index i to the value of the front image index t. Further, the front image determination unit 343 sets the temporary comparison image Tmp to the image Img [t-1].

Next, in step S1350, the front image determination unit 343 adds 1 to the front image index t so that the front image index t indicates the index of the next front image. Further, in step S1360, it is determined whether or not all the frontal images have been compared, in other words, whether or not the frontal image to be compared remains. Specifically, the front image determination unit 343 determines whether or not the front image index t is lower than the number N of front images.

If the front image determination unit 343 determines in step S1360 that the front image index t is lower (true) than the number N of front images, the process returns to step S1130. On the other hand, in step S1360, when the front image determination unit 343 determines that the front image index t is not lower than the number N of front images (false), in other words, there is no front image to be compared. The process proceeds to step S1370.

In step S1370, the front image determination unit 343 compares the current winner AW with the tentative comparison image Tmp using the trained model. The process may be the same as the process of step S1130 according to the first embodiment.

In step S1380, the front image determination unit 343 determines whether the tentative comparative image Tmp is superior to the current winner AW, as in step S1140. However, if it is determined in step S1380 that the evaluation value w3 is the highest, the front image determination unit 343 determines that CNV does not exist in the current winner AW and the provisional comparison image Tmp, and succeeds. In step S1180 of, the winner AW can be determined without. In this case, in step S704, the display control unit 306 can display that the CNV does not exist in the display unit 310.

If it is determined to be true in step S1380, the process proceeds to step S1390. In step S1390, the front image determination unit 343 determines without the winner AW. In this case, in step S704, the display control unit 306 can display that the CNV does not exist in the display unit 310.

On the other hand, if it is determined to be false in step S1380, the process proceeds to step S1180. The step S1180 may be the same process as the step S1180 according to the second embodiment, and the same reference numerals will be used and the description thereof will be omitted. By such processing, the index i indicates the front image index in the optimum depth range, and the image Img [i] is output after step S1180.

In this embodiment, the front image determination unit 343 uses at least two front images input to the trained model when calculating the evaluation value indicating that the extraction target does not exist to calculate the evaluation value for the other front image. Not used. According to this embodiment, when the front image determination unit 343 determines that CNV does not exist in either of the two front images using the trained model 1210, the front image determination unit 343 determines those two images in the subsequent processing. Since it is not used as a comparison target, the calculation load can be reduced. In this embodiment, after it is determined that the CNVs of the two front images do not exist, when the remaining one of the front images to be compared is one, the front surface is exceptionally determined to have no CNV. One of the images will be used for comparison. Even in this case, except in the above exceptional case, the front image in which CNV does not exist is not used as a comparison target in the subsequent processing, so that the calculation load can be reduced. In the above exceptional case, the front image determination unit 343 may perform comparison processing using a dummy front image in which a preset CNV does not exist.

The front image determination process according to this embodiment is not limited to the above flow. The frontal image determination process can be considered for any other implementation of the above flow, depending on the desired configuration, according to any known programming method. For example, in this embodiment, the front image is compared so that the front image with the winner AW is compared with the next front image, but different front images are compared and the winners are compared sequentially. It may be held in a winning tournament system.

(Example 3)
In Example 2, in order to determine the optimum depth range, the front image determination unit 343 input two front images into the trained model 1210 and obtained evaluation values. On the other hand, in the third embodiment, the trained model is configured so that an additional front image is input in addition to the two front images.

Hereinafter, the image processing apparatus according to this embodiment will be described with reference to FIGS. 14 to 16. Since the configuration of the image processing apparatus according to the present embodiment is the same as the configuration of the image processing apparatus according to the second embodiment, the description thereof will be omitted using the same reference numerals. Hereinafter, the image processing apparatus according to the present embodiment will be described focusing on the difference from the image processing apparatus 300 according to the second embodiment.

FIG. 14 is a diagram for explaining the configuration of the trained model according to the present embodiment. In this embodiment, as shown in FIG. 14, the trained model 1410 is configured to input an additional OCTA front image 1405 in addition to the two OCTA front images 1401 and 1402. Here, the two OCTA front images 1401, 1402 are OCTA front images to be compared, and the additional OCTA front image 1405 is an OCTA in another depth range of the eye to be inspected associated with the two front images 1401, 1402. It is a front image.

In one aspect according to this embodiment, the additional frontal image 1405 is an OCTA frontal image of the surface layer of the retina. By inputting the OCTA front image of the surface layer of the retina, it is considered that the projection artifacts generated in the OCTA front image can be taken into consideration for the processing in the trained model 1410.

Here, the projection artifact will be described with reference to FIG. FIG. 15 shows an example of the OCTA front image 1501, 1505 in which contrast correction is performed on the OCTA front images 1401, 1405. The structure of the lower part 1501b of the OCTA front image 1501 correlates well with the surface structure of the lower part 1505b of the OCTA front image 1505. On the other hand, the upper 1501a of the OCTA front image 1501 contains a structure mainly related to CNV and does not correlate well with the surface structure of the upper 1505a of the OCTA front image 1505. As described above, the projection artifact is a phenomenon in which blood vessels such as the surface layer of the retina are reflected in the layer below the surface layer. Therefore, it is considered that the structure of the lower part 1501b of the OCTA front image 1501 that correlates well with the surface layer structure of the lower part 1505b of the OCTA front image 1505 can be grasped separately from the structure related to CNV because of the influence of the projection artifact. Be done.

Therefore, by inputting an additional OCTA front image 1405 into the trained model 1410 in addition to the OCTA front images 1401 and 1402 to be compared, pathological parts and artifacts are considered based on the correlation with the OCTA front image 1405. It is believed that processing will take place. Therefore, by using the trained model according to this example, it is considered that a more reliable comparison can be made by more appropriately distinguishing the pathological part and the projection artifact.

As the learning data according to the present embodiment, in addition to the learning data according to the second embodiment, the OCTA front image of the retinal surface layer of the eye to be inspected, which is related to the two OCTA front images, is used as the input data. On the other hand, the correct answer data (output data) may be the same as the correct answer data according to the second embodiment.

Further, in this embodiment, when the OCTA front image is input to the trained model 1410, the OCTA front image 1401, 1402 and the additional front image 1405 can be input to different input channels. For example, as shown in FIG. 16, an RGB image 1601 having an OCTA front image 1401 as an R channel, an OCTA front image 1402 as a G channel, and an additional OCTA front image 1405 as a B channel may be input to the trained model. In this case, based on the RGB image 1601, the OCTA front image 1401 is on the R channel of the trained model, the OCTA front image 1402 is on the G channel of the trained model, and the additional OCTA front image 1405 is the B of the trained model. It will be input to the channel. In this case, the RGB image similarly generated may be used as the input data of the learning data.

As described above, the frontal image determination unit 343 according to the present embodiment has the OCTA corresponding to the retinal surface layer of the eye to be inspected as an image (additional image) different from the at least two frontal images in addition to the at least two frontal images. Enter the front image into the trained model. Here, the different image input to the trained model can be an OCTA front image of the surface layer of the retina of the eye to be inspected. The front image determination unit 343 can input each of the at least two front images and the different images to different channels of the trained model. According to this example, it is considered that a more reliable comparison can be made by more appropriately distinguishing the pathological part and the projection artifact.

(Example 4)
In Example 2, in order to determine the optimum depth range, the front image determination unit 343 inputs a pair of two front images out of different depth ranges into the trained model 1210 and makes a comparison. On the other hand, in the front image determination process according to the fourth embodiment, the front image determination unit inputs all of the plurality of front images corresponding to different depth ranges into the trained model at one time, and compares them. It is different from the front image determination process (step S1002) in the second embodiment. As an advantage of this embodiment, since the front image determination unit compares all the front images at the same time using the trained model, it is expected that more reliable analysis can be performed. Further, there is an advantage that the flow of the front image determination process in step S1002 can be simplified into a single step.

Hereinafter, the image processing apparatus according to this embodiment will be described with reference to FIG. Since the configuration of the image processing apparatus according to the present embodiment is the same as the configuration of the image processing apparatus according to the second embodiment, the description thereof will be omitted using the same reference numerals. Hereinafter, the image processing apparatus according to the present embodiment will be described focusing on the difference from the image processing apparatus 300 according to the second embodiment.

FIG. 17 is a diagram for explaining the configuration of the trained model according to the present embodiment. In this embodiment, as shown in FIG. 17, the trained model 1710 is configured to input a plurality of OCTA front images 1700 equal to the number N of OCTA front images in a plurality of depth ranges generated in step S1001. do. FIG. 17 shows an example of N = 4 for the sake of simplicity of explanation. The trained model outputs weight vectors (evaluation values w1 to w5) of a number of elements equal to N + 1. Here, the first element 1720 of the weight vector including the number of weight vectors equal to N (evaluation values w1 to w4) corresponds to the existence probability of CNV in each input OCTA front image, and the second weight vector. Element 1725 (evaluation value w5) corresponds to the probability that CNV does not exist in any of the input images.

In the learning data according to this embodiment, OCTA front images having a plurality of depth ranges are used as input data in a random order. Further, as the output data, a weight vector having a weight vector of 1 corresponding to the best image among the input data is used. As an example, when the OCTA front image with N = 4 is provided from the data of the eye having CNV and the best image is provided as the third input data, the weight vector (0,) is given as the correct answer data in the learning process. 0,1,0,0) is used. On the other hand, when the OCTA front image is provided from normal eye data, the weight vector (0,0,0,0,1) is always used as the correct answer data given in the learning process. The trained model 1710 that has performed such training can output evaluation values at once from all OCTA front images in a plurality of input depth ranges.

Further, in this embodiment, the index of the highest evaluation value among the indexes of the evaluation values (weight vectors) output from the trained model corresponds to the index of the front image in the optimum depth range. Therefore, the front image determination unit 343 can determine the optimum depth range based on the index of the highest evaluation value.

According to this embodiment, it is expected that the trained model 1710 can be used to compare all of the plurality of front images corresponding to different depth ranges at the same time, and more reliable comparison can be performed. Further, the flow of the front image determination process in step S1002 can be simplified into a single step.

When using the trained model according to this embodiment, it is necessary to input the same number of input data as the number of input data at the time of training. Therefore, for example, if the number of input data at the time of training is set relatively large and the number of front images to be compared during operation of the trained model is small, a preset CNV is added to the compared front images. A dummy image such as a front image that does not exist may be input. Further, as in the third embodiment, the OCTA front image of the surface layer of the retina may be input as an additional input image to the trained model 1710.

(Example 5)
In Example 3, the frontal image determination unit 343 input an OCTA frontal image of the surface layer of the retina as an additional input image to the trained model 1710. On the other hand, in the fifth embodiment, as an additional input image, an En-Face image having a brightness in the depth range corresponding to each of the input front images is input instead of the OCTA front image of the retinal surface layer. Construct a trained model.

Hereinafter, the image processing apparatus according to this embodiment will be described with reference to FIG. Since the configuration of the image processing apparatus according to the present embodiment is the same as the configuration of the image processing apparatus according to the third embodiment, the description thereof will be omitted using the same reference numerals. Hereinafter, the image processing apparatus according to the present embodiment will be described focusing on the difference from the image processing apparatus 300 according to the second embodiment.

FIG. 18 is a diagram for explaining the configuration of the trained model according to this embodiment. In this embodiment, as shown in FIG. 18, the trained model 1810 is subjected to En-Face of brightness in the depth range corresponding to each of the two OCTA front images 1801, 1802 and the OCTA front images 1801, 1802. Images 1805 and 1806 are configured to be input.

As the learning data according to the present embodiment, similarly to the learning data according to the third embodiment, as input data, an En-Face image having a brightness in the depth range corresponding to the two OCTA front images is used. On the other hand, the correct answer data (output data) may be the same as the correct answer data according to the second embodiment.

By using the trained model that has been trained in this way, it is possible to make a comparison in consideration of the characteristics of the En-Face image of luminance. Therefore, according to this embodiment, it is possible to make a more reliable comparison.

In this embodiment, as an additional image to be input to the trained model, an En-Face image having a brightness corresponding to the depth range of the input OCTA front image was used. However, the additional images input to the trained model are not limited to this, for example, fundus camera images, scanning laser eye optical images (SLO images), ocular angiography images, ocular autofluorescence images, or indocyanine green. Images obtained from different imaging modalities, such as angiography images, may be used. In these cases, as the input data of the training data, the imaging modality corresponding to the additional image input to the trained model was used, and the front image as the input data was obtained from the subject to be inspected. Images may be used.

When the front image to be compared is an En-Face image of brightness, an OCTA front image corresponding to the depth range of the En-Face image of brightness to be compared can be used as an additional image. In this case, the input data of the training data also uses the brightness En-Face image as the front image to be compared, and the OCTA front image corresponding to the depth range of the brightness En-Face image to be compared as the additional image. You can use it.

As described above, the front image generation unit 342 according to the present embodiment inputs, in addition to at least two front images, an image (additional image) different from the at least two front images into the trained model. Here, the different images input to the trained model can be En-Face images or OCTA front images with at least two luminances corresponding to at least two front images input to the trained model. Further, the different images input to the trained model may be images of the eye to be inspected acquired by a modality other than the optical interference tomography apparatus. The front image determination unit 343 can input each of the at least two front images and different images to different channels of the trained model. According to this embodiment, it is possible to make a comparison in consideration of the features of additional images, and it is possible to make a more reliable comparison.

The additional image was acquired by the OCTA front image of the surface layer of the retina, the En-Face image or OCTA front image of the above brightness corresponding to at least two front images input to the trained model, and other modality. It may be a combination of images of the eye to be inspected. In this case, as the training data, an image corresponding to the front image and the additional image input to the trained model may be used. For example, consider the case where an OCTA frontal image of the surface layer of the retina and an En-Face image of brightness corresponding to at least two frontal images input to the trained model are used as additional images. In this case, as the input data of the training data, an OCTA front image of the surface layer of the retina and an En-Face image having a brightness corresponding to the front image to be evaluated may be used.

(Example 6)
Hereinafter, the image processing apparatus according to the sixth embodiment will be described with reference to FIG. In this embodiment, the plurality of depth ranges of the OCTA front image generated by the front image generation unit 342 are determined based on a predefined projection configuration list. Further, in this embodiment, as the OCTA front image input to the trained model, a thumbnail image with a reduced resolution of the OCTA front image is used.

Since the configuration of the image processing apparatus according to the present embodiment is the same as the configuration of the image processing apparatus according to the second embodiment, the description thereof will be omitted using the same reference numerals. In the series of processes according to the present embodiment, only the front image generation process is different from the process of the second embodiment, and the other processes are the same as the series of processes according to the second embodiment. Therefore, the series of processes according to the second embodiment. The difference from the processing of is mainly explained. FIG. 19 shows a flowchart of the front image generation process according to the present embodiment. When the front image generation process according to this embodiment is started, the process proceeds to step S1901.

In step S1901, the projection range control unit 341 uses the generated motion contrast image, the layer recognition result, and the projection configuration list stored in the storage unit 305 to generate the OCTA front image. Identify the contrast data. Further, the projection range control unit 341 specifies a projection method for each OCTA front image based on the projection configuration list. The front image generation unit 342 projects the 3D motion contrast data onto the 2D plane by the 3D motion contrast data and the projection method specified by the projection range control unit 341, and generates a low-resolution thumbnail image of the OCTA front image. do.

Here, the projection configuration list includes definitions of the upper and lower ends of the depth range, as well as projection methods. The projection method defined in the projection configuration list may be any known projection method, and may be, for example, average image projection (AIP) or maximum image projection (MIP). Table 2 below shows an example of a projection configuration list according to one aspect of this embodiment.

The projection configuration list according to one aspect of this example was determined as follows, based on 98 data of the eyes of patients with different forms of CNV. First, for each inspection, experts determined the depth range (top and bottom) and selected the projection method to determine the projection configuration to obtain an image representing complete CNV with optimum quality. Next, the distribution of projection configurations determined by the expert was analyzed, and the 98 projection configurations determined by the expert were grouped into the above 17 projection configurations.

Note that the number and definition of projection configurations is not limited to those listed above, and the final projection configuration list depends on the number and variation of cases analyzed, as well as grouping criteria and pathological types. It may be different. Also, if desired, the user may add a new projection configuration to the predefined projection configuration list if the user finds some additional projection settings that provide excellent image quality. If the training data of the trained model uses OCTA frontal images with sufficiently different depth ranges and projection methods, even if the projection configuration is added to the projection configuration list, the machine learning model can be redesigned. No retraining required. Further, the depth range and the projection method regarding the OCTA front image used as the training data of the trained model do not have to be the same as the depth range and the projection method defined in the projection configuration list.

Further, the thumbnail image of the OCTA front image means an OCTA front image having a small image size (low resolution). For example, if the scan size of the OCTA test consists of 464 x 464 scans, the resolution of the full size front image will be 464 x 464. On the other hand, the resolution of the thumbnail image may be, for example, 232 × 232. As a method of generating a thumbnail image, the front image generation unit 342 may generate a full-size OCTA front image and reduce the resolution of the OCTA front image to a target resolution, or another method may be used. It may be done. The resolution of the thumbnail image is not limited to this value, and may be a value lower than the resolution of the full-size front image, and may be smaller than, for example, 232 × 232. When the brightness En-Face image is used, the thumbnail image of the brightness En-Face image may be generated in the same manner as the thumbnail image of the OCTA front image and used for comparison.

Next, in step S1902, the front image determination unit 343 compares the pair of thumbnail images of the OCTA front image of the projection configuration (# 1 to # 17) defined in the projection configuration list using the trained model. Select the optimum depth range and projection method (projection configuration). The series of processes for determining the optimum depth range and projection method in step S1902 may be performed in the same manner as the front image determination process (S1002) according to the second embodiment. However, in step S1902, the thumbnail image that is the winner AW is determined, and the depth range and the projection method corresponding to the determined thumbnail image are selected, which is different from the front image determination process according to the second embodiment.

Further, in this embodiment, the thumbnail image is used as an input to the trained model, which is different from the front image determination process (step S1002) according to the second embodiment. In this embodiment, in order to perform such processing, the thumbnail image of the OCTA front image is also used as the input data of the training data. The thumbnail image generation method may be the above-mentioned method. Further, in step S1901, when the front image generation unit 342 generates a thumbnail image of the OCTA front image, it is possible to generate a thumbnail image having the same resolution as the thumbnail image used for the input data of the training data. .. Further, the method of generating the thumbnail image by the front image generation unit 342 can be the same as the method of generating the thumbnail image used for the input data.

By using the trained model trained using such training data, the front image determination unit 343 can obtain a projection configuration suitable for observation of CNV from the thumbnail image of each OCTA front image based on the projection configuration list. You can choose. In this embodiment, since the thumbnail image is used for comparing the images, the resources and processing time used for the comparison processing can be suppressed, and the comparison processing can be performed efficiently.

Next, in step S1903, the front image generation unit 342 generates a full-size OCTA front image for the projection configuration obtained in step S1902. When the full-size OCTA front image is generated when the thumbnail image is generated in step S1901, the front image generation unit 342 receives the full-size OCTA front image corresponding to the projection configuration obtained in step S1902. Should be specified. When the full-size OCTA front image is generated in step S1903, the process proceeds to step S704. Since the subsequent processing is the same as that of the second embodiment, the description thereof will be omitted.

In this embodiment, the front image generation unit 342 generates a plurality of front images corresponding to different depth ranges from the three-dimensional data of the eye to be inspected by using a projection configuration list including different depth ranges defined in advance. As a result, the front image generation unit 342 acquires a plurality of front images including at least two front images used as input data of the trained model. Further, the front image determination unit 343 determines the depth range using the evaluation value, and determines the image generated using the determined depth range as the output image. According to this embodiment, since the OCTA front image of a plurality of depth ranges and projection methods is generated based on a predefined projection configuration list, the operation related to the setting of the depth range and the projection method at the time of shooting is complicated. Can be reduced.

The projection configuration list includes a projection method for calculating the depth range defined by the upper and lower ends and the pixel value of the front image. In addition, the upper and lower ends of the depth range included in the projection configuration list can be defined based on the structure of the eye to be inspected. The structure of the eye to be inspected includes, for example, the boundaries of the layers of the eye. Also, the top and bottom edges of the depth range can include depth offset from the boundaries of the eye layer. Further, the projection configuration list can be stored as a search table in the storage unit 305 of the image processing device 300, and the image processing device 300 can refer to the projection configuration list to obtain a depth range corresponding to each front image. You can search for and identify the projection method.

Further, in this embodiment, the learning data includes thumbnail images of a plurality of front images as a plurality of front images corresponding to different depth ranges of the eye to be inspected. The front image generation unit 342 generates thumbnail images of a plurality of front images as a plurality of front images corresponding to different depth ranges of the eye to be inspected. Further, the front image generation unit 342 generates a full-size front image having a higher resolution than the thumbnail image as the front image corresponding to the depth range determined by the front image determination unit 343.

With such a configuration, thumbnail images with reduced resolution are used instead of full-size images for input of the trained model, so that resources and processing time used for comparison processing can be reduced, which is efficient. Can be processed. Further, since the image is evaluated using the trained model that has been appropriately trained and the optimum depth range is determined, highly reliable evaluation can be realized even if the thumbnail image is used as an input. In addition, since a new projection configuration can be registered in the projection configuration list, it is possible to easily support the projection of various lesions.

In this embodiment, the projection configuration includes the depth range and the projection method. However, the items included in the projection configuration are not limited to this. For example, the projection method may be fixed to any method, and only the depth range may be included in the projection configuration.

Further, the display mode of the GUI and the like described in the above-mentioned Examples and Modifications is not limited to the above-mentioned ones, and may be arbitrarily changed according to a desired configuration. For example, although it is described that the OCTA front image, the tomographic image, and the depth range are displayed for the GUI 500 and the like, the motion contrast data may be displayed on the tomographic image. In this case, it is possible to confirm at which depth the motion contrast value is distributed. Further, colors may be used for displaying an image or the like.

Further, in the trained model for evaluation of the extraction target according to the above embodiment and the modified example, the magnitude of the brightness value of the front image, the order and inclination of the bright part and the dark part, the position, the distribution, the continuity, etc. are one of the feature quantities. It can be considered that it is extracted as a part and used for the estimation process. Further, when two front images are input to the training model, an evaluation value indicating the existence of the extraction target in each front image is output. Therefore, in the trained model, the brightness value corresponding to the extraction target in each front image is output. It is considered that processing such as comparison is performed.

In the above embodiments and modifications, the spectral domain OCT (SD-OCT) device using the SLD as the light source has been described as the OCT device, but the configuration of the OCT device according to the present invention is not limited to this. For example, the present invention can be applied to any other type of OCT device such as a wavelength sweep type OCT (SS-OCT) device using a wavelength sweep light source capable of sweeping the wavelength of emitted light. Further, the present invention can also be applied to a Line-OCT device (or SS-Line-OCT device) using line light. The present invention can also be applied to a Full Field-OCT device (or SS-Full Field-OCT device) using area light. Further, the present invention may be applied to a wave surface adaptive optics (AO-OCT) device using an adaptive optics system or a polarized OCT (PS-OCT) device for visualizing information on polarization phase difference and polarization elimination. can.

In the above-described embodiment and modification, an optical fiber optical system using a coupler is used as the dividing means, but a spatial optical system using a collimator and a beam splitter may be used. Further, the configuration of the optical interference unit 100 and the scanning optical system 200 is not limited to the above configuration, and a part of the configuration included in the optical interference unit 100 and the scanning optical system 200 may be a separate configuration from these.

Further, in the above embodiment and the modified example, the image processing apparatus 300 has acquired the interference signal acquired by the optical interference unit 100, the tomographic data generated by the reconstruction unit 301, and the like. However, the configuration in which the image processing device 300 acquires these signals and images is not limited to this. For example, the image processing device 300 may acquire these signals and data from a server or a photographing device connected to the image processing device 300 via a LAN, WAN, the Internet, or the like.

The trained model according to the above embodiment and the modified example can be provided in the image processing apparatus 300. The trained model may be configured by, for example, a CPU, a software module executed by a processor such as an MPU, GPU, FPGA, or the like, or may be configured by a circuit or the like that performs a specific function such as an ASIC. Further, these trained models may be provided in a device of another server connected to the image processing device 300 or the like. In this case, the image processing device 300 can use the trained model by connecting to a server or the like having the trained model via an arbitrary network such as the Internet. Here, the server provided with the trained model may be, for example, a cloud server, a fog server, an edge server, or the like. Further, the training data of the trained model is not limited to the data obtained by using the ophthalmology device itself that actually performs the imaging, but also the data obtained by using the same type of ophthalmology device or the same type of ophthalmology according to the desired configuration. It may be data or the like obtained by using the device.

The GPU can perform efficient operations by processing more data in parallel. Therefore, when learning is performed a plurality of times using a learning model such as deep learning, it is effective to perform processing on the GPU. Therefore, a GPU may be used in addition to the CPU for the processing by the front image determination unit 343, which is an example of the learning unit (not shown). In this case, when the learning program including the learning model is executed, the CPU and the GPU cooperate to perform the learning to perform the learning. In addition, the processing of the learning unit may be performed only by the CPU or the GPU. Further, the processing unit (estimation unit) that executes the processing using the various trained models described above may also use the GPU in the same manner as the learning unit. Further, the learning unit may include an error detection unit and an update unit (not shown). The error detection unit obtains an error between the output data output from the output layer of the neural network and the correct answer data according to the input data input to the input layer. The error detection unit may use the loss function to calculate the error between the output data from the neural network and the correct answer data. Further, the update unit updates the coupling weighting coefficient between the nodes of the neural network based on the error obtained by the error detection unit so that the error becomes small. This updating unit updates the coupling weighting coefficient and the like by using, for example, the backpropagation method. The error back propagation method is a method of adjusting the coupling weighting coefficient and the like between the nodes of each neural network so that the above error becomes small.

(Other examples)
The present invention is also a process of supplying a program that realizes one or more functions of the above-described examples and modifications to a system or device via a network or a storage medium, and the computer of the system or device reads and executes the program. It is feasible. A computer may have one or more processors or circuits and may include multiple separate computers or a network of separate processors or circuits for reading and executing computer-executable instructions.

The processor or circuit may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), or a field programmable gateway (FPGA). Also, the processor or circuit may include a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU).

Although the present invention has been described above with reference to Examples and Modifications, the present invention is not limited to the above Examples and Modifications. The present invention also includes inventions modified to the extent not contrary to the gist of the present invention, and inventions equivalent to the present invention. In addition, the above-mentioned Examples and Modifications can be appropriately combined as long as they do not contradict the gist of the present invention.

300: Image processing device, 342: Front image generation unit (acquisition unit), 343: Front image determination unit (evaluation unit, determination unit)

Claims (18)

An acquisition unit that acquires multiple front images corresponding to different depth ranges of the three-dimensional data of the eye to be inspected, and an acquisition unit.
Of the plurality of acquired front images, as input data input to different channels of the trained model obtained by training using training data including a plurality of front images corresponding to different depth ranges of the eye to be inspected. An evaluation unit that acquires an evaluation value that evaluates the existence of a target area by using at least two front images, and an evaluation unit.
A determination unit that determines at least one of the plurality of front images as an output image using the evaluation value, and a determination unit.
An image processing device. An acquisition unit that acquires multiple front images corresponding to different depth ranges of the three-dimensional data of the eye to be inspected, and an acquisition unit.
Of the plurality of acquired front images, as input data input to different channels of the trained model obtained by training using training data including a plurality of front images corresponding to different depth ranges of the eye to be inspected. An evaluation unit that acquires an evaluation value that evaluates the existence of a target area by using at least two front images, and an evaluation unit.
A determination unit that determines the depth range using the evaluation value and determines an image generated using the determined depth range as an output image.
An image processing device. The training data includes thumbnail images of a plurality of front images as a plurality of front images corresponding to different depth ranges of the eye to be inspected.
The acquisition unit
Generate thumbnail images of multiple front images as multiple front images corresponding to different depth ranges of the eye to be inspected.
The image processing apparatus according to claim 2, wherein a front image having a higher resolution than a thumbnail image is generated as a front image corresponding to the determined depth range. The acquisition unit is at least used as the input data by generating a plurality of front images corresponding to different depth ranges from the three-dimensional data of the eye to be inspected using a list including different predefined depth ranges. The image processing apparatus according to any one of claims 1 to 3, wherein a plurality of front images including two front images are acquired. The list includes projection methods for calculating the depth range and frontal image pixel values defined by the top and bottom edges.
The image processing apparatus according to claim 4, wherein the upper end and the lower end of the depth range are defined based on the structure of the eye to be inspected. The structure of the eye to be inspected includes the boundaries of the layers of the eye, including the boundaries of the layers of the eye.
The image processing apparatus according to claim 5, wherein the upper end and the lower end of the depth range include a depth offset from the boundary. The image processing apparatus according to any one of claims 4 to 6, wherein the list is stored in the image processing apparatus as a search table. In addition to the at least two front images, the evaluation unit inputs an image different from the at least two front images into the trained model.
The different images are acquired by a front image corresponding to the surface layer of the retina of the eye to be inspected, an En-Face image or OCTA front image having at least two brightnesses corresponding to the at least two front images, and a modality other than the optical interference tomography device. The image processing apparatus according to any one of claims 1 to 7, which is at least one of the images of the subject to be inspected. The image processing apparatus according to claim 8, wherein the evaluation unit inputs each of the at least two front images and the different images to different channels of the trained model. The image processing apparatus according to any one of claims 1 to 9, wherein the learning data includes a plurality of front images corresponding to different depth ranges of the eye to be inspected and evaluation values corresponding to each front image. The evaluation unit does not use at least two front images input to the trained model when calculating the evaluation value indicating that the target area does not exist in the calculation of the evaluation value for the other front image. The image processing apparatus according to any one of 1 to 10. The image processing apparatus according to any one of claims 1 to 10, wherein the evaluation unit inputs all of the generated plurality of front images into the trained model. The target area is the area of new blood vessels.
The image processing apparatus according to any one of claims 1 to 12, wherein the three-dimensional data is three-dimensional motion contrast data. The target area is the area of the sieve plate, and is the area of the sieve plate.
The image processing apparatus according to any one of claims 1 to 12, wherein the three-dimensional data is three-dimensional tomographic data of the papilla. The target area is the area of the capillary aneurysm and
The image processing apparatus according to any one of claims 1 to 12, wherein the three-dimensional data is three-dimensional motion contrast data of the yellow spot portion. Acquiring multiple frontal images corresponding to different depth ranges of 3D data of the eye to be inspected,
Of the plurality of acquired front images, as input data input to different channels of the trained model obtained by training using training data including a plurality of front images corresponding to different depth ranges of the eye to be inspected. Obtaining an evaluation value that evaluates the existence of the target area by using at least two front images,
Using the evaluation value, at least one of the plurality of front images is determined as an output image.
Image processing methods, including. Acquiring multiple frontal images corresponding to different depth ranges of the 3D data of the eye to be inspected,
Of the plurality of acquired front images, as input data input to different channels of the trained model obtained by training using training data including a plurality of front images corresponding to different depth ranges of the eye to be inspected. By using at least two front images, it is possible to obtain an evaluation value that evaluates the existence of the target area.
The depth range is determined using the evaluation value, and the image generated using the determined depth range is determined as the output image.
Image processing methods, including. A program that, when executed by a computer, causes the computer to perform each step of the image processing method according to claim 16 or 17.

Images