JP2021153786A - Image processing device, image processing method and program - Google Patents

Abstract

To provide a system that can acquire an image having a wide-range vertical width relating to an eye to be examined, in a short time with a high-resolution.SOLUTION: An image processing method includes the steps of: acquiring a first image 220 having a vertical width exceeding a width of a focal depth of measuring beam relative to an eye to be examined; and generating a second image 230 having a vertical width of the first image 220 and having a resolution higher than resolution of second regions 221 and 223 except for a first region 222 relating to a width of a focal depth on the first image 220 using a learned model 300.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image processing device and an image processing method for processing an image related to an eye to be inspected, and a program for operating a computer as the image processing device. At this time, as the image processing device, for example, it is preferable to apply an ophthalmologic imaging device having an adaptive optics function of measuring and correcting an aberration in the return light of the measurement light from the eye to be inspected.

In recent years, as an ophthalmologic imaging device, an SLO (Scanning Laser Ophthalmoscope) device that two-dimensionally irradiates the fundus of the eye to be inspected with a laser beam and receives the return light to form an image, and a low coherence light device. Imaging devices that utilize interference have been developed. An imaging device that utilizes the interference of low coherence light is called an OCT (Optical Coherence Tomography) device, and is used especially for the purpose of obtaining a tomographic image of the fundus of the eye to be examined or its vicinity. Has been done. Various types of OCT have been developed, including TD-OCT (Time Domain OCT) and SD-OCT (Spectral Domain OCT).

In particular, in recent years, in such an ophthalmologic imaging apparatus, the resolution of captured images has been further increased by increasing the NA of the irradiation laser and the like. However, for example, when photographing the fundus of the eye to be inspected, it is necessary to take an image through an optical tissue such as the cornea or the crystalline lens of the eye to be inspected. Aberrations caused by the crystalline lens have come to have a great influence on the image quality of captured images.

Currently, research is underway on AO-SLO devices and AO-OCT devices that incorporate adaptive optics (AO) functions that measure aberrations caused by the eye to be inspected and correct the aberrations in the optical system. For example, Non-Patent Document 1 shows an example of an AO-OCT apparatus. In the AO-SLO device and the AO-OCT device, the wavefront of the return light is generally measured by the Shack-Hartmann wavefront sensor method. Here, the Shack-Hartmann wave surface sensor method measures the wave surface of the return light by incident the measurement light on the eye to be inspected and receiving the return light with a CCD camera through a microlens array. Then, in the AO-SLO device and the AO-OCT device, a variable shape mirror, a spatial phase modulator, or the like is driven so as to correct the wavefront of the measured return light, and the fundus of the eye to be inspected is photographed through them. High-resolution plane images and tomographic images can be taken (see, for example, Patent Document 1).

JP 2015-221091

ZHANG, et al. , "High-speed volumetric imaging of connection optical coherence with adaptive optics spectral-domain optics, optical coherence tomography, Optics4, Optics4, 14, No. 10

For example, when a tomographic image is taken using an AO-OCT device as an ophthalmologic imaging device, the depth of focus of the tomographic image obtained by increasing the NA is shallow, and a tomographic tomography with a limited (narrow) vertical width is taken in one shot. Only images can be acquired. For example, in order to obtain a tomographic image with a wide range of vertical width from the nerve fiber layer of the eye to be examined to the choroid with high resolution, the focusing position (focus position), which is the focusing position for concentrating the measurement light, is set in the vertical direction (focus position). This can be achieved by taking a number of tomographic images at each condensing position while gradually shifting (in the depth direction of the eye to be inspected), and finally adopting a technique of combining (arranging) these tomographic images. However, this technique requires the patient to open the eyelids for a long period of time, which places a heavy burden on the patient and tends to cause the patient's eye to be examined to become unstable, resulting in high accuracy (high resolution). There is a problem that the image cannot be obtained.

The present invention has been made in view of such problems, and an object of the present invention is to provide a mechanism capable of acquiring a wide range of vertical width images related to an eye to be inspected in a short time with high resolution.

The image processing apparatus of the present invention uses an acquisition means for acquiring a first image having a vertical width exceeding the width of the depth of focus of the measurement light with respect to the eye to be examined and a trained model, and the focus in the first image. It has a generation means for generating an image having a resolution higher than the resolution of the second region excluding the first region related to the width of depth and having the vertical width.
The present invention also includes an image processing method using the above-mentioned image processing device and a program for operating a computer as each means of the above-mentioned image processing device.

According to the present invention, an image having a wide vertical width related to an eye to be inspected can be acquired in a short time with high resolution.

It is a figure which shows the concept of generating the output data (teacher data) from the input data by using the trained model in the image processing apparatus which concerns on 1st Embodiment of this invention. It is a figure which shows the concept of the process at the time of creating the trained model shown in FIG. It is a figure which shows the concept of the process at the time of creating the trained model shown in FIG. It is a figure which shows an example of the schematic structure of the ophthalmologic imaging apparatus applied as the image processing apparatus which concerns on 1st Embodiment of this invention. It is a figure which shows an example of the schematic structure of the wave surface sensor shown in FIG. It is a figure for demonstrating the wavefront measured by the wavefront sensor shown in FIG. It is a flowchart which shows an example of the processing procedure in the image processing method by the ophthalmologic imaging apparatus applied as the image processing apparatus which concerns on 1st Embodiment of this invention. It is a figure which shows an example of the schematic structure of the image processing unit shown in FIG. It is a figure which shows an example of the schematic structure of the CNN processing part shown in FIG. It is a flowchart which shows an example of the processing procedure in the image processing method by the ophthalmologic imaging apparatus applied as the image processing apparatus which concerns on 1st Embodiment of this invention. It is a figure for demonstrating the image processing method by the ophthalmologic imaging apparatus applied as the image processing apparatus which concerns on 2nd Embodiment of this invention.

Hereinafter, embodiments (embodiments) for carrying out the present invention will be described with reference to the drawings.

(First Embodiment)
First, the first embodiment of the present invention will be described.

FIG. 1 is a diagram showing a concept of generating output data (teacher data) 230 from input data 220 by using the trained model 300 in the image processing apparatus according to the first embodiment of the present invention. FIG. 1 shows an example of the OCT image 200 and the AO-OCT image 210. Here, an example of taking a tomographic image using AO-OCT will be described.

In FIG. 1, the input data 220 is an image obtained when a tomographic image is taken using AO-OCT, and has a vertical width exceeding the width of the depth of focus of the measurement light with respect to the eye to be inspected. ". The input data 220 includes a condensing position region (first region) 222 related to the width of the depth of focus, and a blurred region (second region) 221 and 223 having a resolution lower than that of the first region 222. Has been done. At this time, the first region 222 indicating the region of the focusing position is the region of the focusing position that focuses the measurement light on the focusing position. Then, in the present embodiment, the trained model 300 is used to generate the output data (teacher data) 230 from the input data 220. Specifically, the output data (teacher data) 230 is an image having a resolution higher than the resolutions of the second regions 221 and 223 excluding the first region 222 in the input data 220, and is equivalent to the vertical width of the input data 220. It is a "second image" having a vertical width. That is, in the present embodiment, the trained model 300 is used to input a tomographic image having a shallow depth of focus (the vertical width of the condensing position (first region 222) is narrow) obtained by imaging by AO-OCT as input data 220. Therefore, a tomographic image with a deep depth of focus (wide vertical width of the focusing position) is generated as output data (teacher data) 230.

Next, the process for creating the trained model 300 shown in FIG. 1 will be described.
FIG. 2 is a diagram showing a concept of processing when creating the trained model 300 shown in FIG.
In the training model when the trained model 300 is created, as shown in FIG. 2, the image has a vertical width exceeding the width of the depth of focus of the measurement light with respect to the eye to be inspected, and the focus position is in the vertical width. A plurality of images having different condensing position regions are used as input data 320. In the example shown in FIG. 2, each image in the plurality of images serving as the input data 320 has a condensing position region (first region) 322 corresponding to the first region 222 in FIG. 1 and a second region 221 in FIG. 6 images [1] -1 to [6] -1 are shown, which include blurred regions (second regions) 321 and 323 corresponding to 223 and 223, and the first region 322, which is the region of the condensing position, is different from each other. Has been done. Further, in the example shown in FIG. 2, each first region 322 in the six images [1] -1 to [6] -1 is a partial image of a single-layer image different from each other. Further, the six images [1] -1 to [6] -1 are, for example, a group of images obtained by a series of continuous shooting. Then, in the training model when creating the trained model 300, for example, the vertical wide-range tomographic image 1 to be obtained as a result is each of the six images [1] -1 to [6] -1 which are the input data 320. It is generated by combining the partial images (tomographic images) of the first region 322 of the above. That is, the trained model 300 of FIG. 1 is generated by using a plurality of images having different positions of the first region 322 in the vertical width as input data 320 and using partial images of the first region 322 of each of the plurality of images. It is obtained by learning the teacher data using the image 1 as the output data.

FIG. 3 is a diagram showing a concept of processing when creating the trained model 300 shown in FIG. Specifically, FIG. 3 is a conceptual diagram for deriving output data (teacher data) 330 from the input data 320 of FIG. 2 in the learning model for creating the trained model 300. In FIG. 3, the vertical wide-range tomographic image 1 to be acquired is a region of each condensing position in a plurality of images [1] -1 to [6] -1 which are input data 320 (first region 322 in FIG. 2). It is generated by combining the partial images (tomographic images) of. Similarly, the vertical wide-range tomographic image 2 to be acquired is the region of each condensing position (first region 322 in FIG. 2) in the plurality of images [1] -2 to [6] -2 which are the input data 320. It is generated by combining partial images (tomographic images).

For example, consider the case where the image [1] -1 of FIG. 3 is obtained by photographing by AO-OCT. In this case, in the trained model 300 of FIG. 1, the image [1] -1 of FIG. It is something to derive.

The concept of the present invention has been described above, but the embodiments of the present invention will be specifically described below. First, the image processing apparatus according to the embodiment of the present invention will be described.

<Device configuration>
FIG. 4 is a diagram showing an example of a schematic configuration of an ophthalmologic imaging apparatus 100 applied as an image processing apparatus according to the first embodiment of the present invention. In the present embodiment, the object to be measured is the eye to be inspected E (more specifically, the fundus Ef of the eye to be inspected E), and the ophthalmologic imaging apparatus 100 includes the AO-SLO unit 101 and the AO-OCT unit 102 in the same apparatus. An example in which an adaptive optics OCT-SLO device having both of the above functions is applied will be described. Further, FIG. 4 shows XYZ coordinates showing the Z direction corresponding to the vertical direction in the vertical width of the images shown in FIGS. 1 to 3 and the X and Y directions orthogonal to and mutually orthogonal to the Z direction. The system is illustrated. As described above, the vertical direction in the vertical width of the images shown in FIGS. 1 to 3 corresponds to the Z direction indicating the depth direction in the fundus Ef of the eye E to be inspected.

First, the AO-SLO unit 101 will be described.
As shown in FIG. 4, the AO-SLO unit 101 includes a light source 141, a single-mode optical fiber 142, a collimator 143, an optical dividing unit 144, a condensing lens 145, and an optical sensor 146.

As the light source 141, for example, an SLD light source (Super Luminate Diode) having a wavelength of 760 nm can be used. At this time, the wavelength of the light source 141 is not particularly limited, but for imaging the fundus Ef, a wavelength of about 750 nm to 1500 nm is preferable in order to reduce the glare of the subject and maintain the resolution. Used. Further, in the present embodiment, the SLD light source is used as the light source 141, but a laser or the like can also be used. Further, in the present embodiment, the light sources for fundus imaging and wave surface measurement are shared, but each may be a separate light source and may be configured to combine waves in the middle of the optical path. Further, in the present embodiment, the AO-SLO optical system including the AO-SLO unit 101 shares a part of the optical system with the AO-OCT optical system including the AO-OCT unit 102. In order to branch off from the optical path of the above, a wavelength different from the wavelength of the light source 151 included in the AO-OCT unit 102 is selected, and the optical path is branched at the dichroic mirror island.

The light emitted from the light source 141 is emitted as parallel rays (measurement light 105) by the collimator 143 through the single-mode optical fiber 142. The polarization of the emitted measurement light 105 is adjusted by a polarization regulator (not shown) provided in the path of the single-mode optical fiber 142. As another configuration, there is a configuration in which an optical component for adjusting the polarization is arranged in the optical path after being emitted from the collimator 143.

The irradiated measurement light 105 passes through an optical splitting unit 144 composed of a beam splitter, and further passes through a beam splitter 104 for optical branching with the AO-OCT unit 102, and an optical system having an adaptive optics function (adaptive optics system). Guided to.

The adaptive optics system includes an optical dividing unit 106, reflection mirrors 107-1 to 107-4, a wave surface correction device 108, a relay optical system 113-1 to 113-2, an aperture 114, and a wave surface sensor 115. ing. Here, the reflection mirrors 107-1 to 107-4 are installed so that at least the pupil of the eye E to be inspected, the wavefront sensor 115, and the wavefront correction device 108 are optically conjugated. Further, in the present embodiment, a beam splitter is used as the optical splitting unit 106.

The measurement light 105 transmitted through the light dividing unit 106 is reflected by the reflection mirror 107-1 and the reflection mirror 107-2 and is incident on the wave surface correction device 108. Then, the measurement light 105 reflected by the wave surface correction device 108 is further reflected by the reflection mirror 107-3 and the reflection mirror 107-4, and is guided to the scanning optical system 109-1.

In this embodiment, a variable shape mirror is used as the wave surface correction device 108. At this time, the variable shape mirror is a mirror in which the reflecting surface is divided into a plurality of regions, and the wavefront of the reflected light can be changed by changing the angle of each region. Further, as the wave surface correction device 108, it is also possible to use a spatial phase modulator using a liquid crystal element instead of the variable shape mirror. In that case, two spatial phase modulators may be used to correct the wavefront of the bipolarized light from the eye E to be inspected.

The light reflected by the reflection mirror 107-3 and the reflection mirror 107-4 is scanned one-dimensionally or two-dimensionally by the scanning optical system 109-1. In the present embodiment, one resonance scanner and one galvano scanner are used in the scanning optical system 109-1 for the main scanning (horizontal direction of the fundus) and for the sub-scanning (vertical direction of the fundus). In another configuration, two galvano scanners may be used for the scanning optical system 109-1. In order to bring each scanner in the scanning optical system 109-1 into an optically conjugated state, a configuration in which an optical element such as a mirror or a lens is used between the scanners may be adopted.

In the present embodiment, in addition to the scanning optical system 109-1, a tracking mirror 109-2 is further provided. The tracking mirror 109-2 is composed of two galvano scanners, and can further move the imaging region in two directions. In another configuration, the scanning optical system 109-1 also functions as the tracking mirror 109-2, the tracking mirror 109-2 is configured only in the resonance scanner direction of the scanning optical system 109-1, and the tracking mirror 109-2 is There may be a configuration that is a two-dimensional mirror. Further, in order to optically couple the scanning optical system 109-1 and the tracking mirror 109-2, a relay optical system (not shown) is often used.

The measurement light 105 scanned by the scanning optical system 109-1 and the tracking mirror 109-2 is applied to the eye E to be inspected through the eyepiece 110-1 and the eyepiece 110-2. The measurement light 105 applied to the eye E to be inspected is reflected or scattered by the fundus Ef. By adjusting the positions of the eyepiece lens 110-1 and the eyepiece lens 110-2, it is possible to irradiate the optimum measurement light 105 according to the diopter of the eye E to be inspected. Here, the eyepiece lens 110 is arranged in the eyepiece portion of the ophthalmologic imaging apparatus 100, but it may be configured by a spherical mirror or the like.

The return light of the measurement light 105 reflected or scattered from the retina (fundus Ef) of the eye E to be inspected travels in the opposite direction at the time of incidence of the measurement light 105, and a part of the light is wave-planed by the light dividing portion 106. It is reflected to the side of the sensor 115 and is used to measure the wave front of the return light. The return light reflected by the light dividing unit 106 toward the wave surface sensor 115 passes through the relay optical system 113-1, the aperture 114, and the relay optical system 113-2, and is incident on the wave surface sensor 115. The aperture 114 arranged between the relay optical system 113-1 and the relay optical system 113-2 is an optical member for preventing unnecessary reflected scattered light from a lens or the like from incident on the wave surface sensor 115. In this embodiment, a Shack-Hartmann sensor is used as the wavefront sensor 115.

The wavefront sensor 115 is connected to the adaptive optics control unit 116 and transmits the wavefront of the received return light to the adaptive optics control unit 116. The wave surface correction device 108 is also connected to the adaptive optics control unit 116, and performs modulation (modulation to the in-focus position) instructed by the adaptive optics control unit 116. For example, the adaptive optics control unit 116 calculates the in-focus position from the number of spots sufficient for imaging derived from the measurement result of the wavefront sensor 115, and modulates the focus position to the focus value of the wavefront correction device 108. Command to do. After that, the adaptive optics control unit 116 corrects the modulation amount (aberration correction) for each pixel of the wavefront correction device 108 so as to correct the wavefront without aberration based on the wavefront of the return light acquired by the measurement result of the wavefront sensor 115. The quantity) is calculated and the wavefront correction device 108 is instructed to do so. Measurement of the wavefront of the return light by the wavefront sensor 115 and the instruction to the wavefront correction device 108 by the adaptive optics control unit 116 are repeatedly processed, and feedback control is performed so that the optimum focusing position and the wavefront are always obtained.

A part of the return light transmitted through the light dividing unit 106 is reflected by the light dividing unit 144, and is collected by the condensing lens 145 on the optical sensor 146 having a pinhole. Then, the optical sensor 146 converts it into an electric signal according to the intensity of the received light. The optical sensor 146 is connected to the control unit 117, and the control unit 117 constitutes a planar image (fundus image) of the fundus Ef of the eye E to be inspected based on the electric signal obtained by the optical sensor 146 and the position of the optical scan. .. After that, the plane image of the fundus Ef of the eye E to be inspected is subjected to image processing by the image processing unit 118 and then displayed on the display 119 as an AO-SLO image.

Next, the AO-OCT unit 102 will be described.
As shown in FIG. 4, the AO-OCT unit 102 includes a light source 151, a fiber coupler 152, a reference optical system 153, and a spectroscope 154.

As the light source 151, for example, an SLD light source having a wavelength of 840 nm is used. The light source 151 may be any as long as it can emit light having low coherence, and an SLD having a wavelength width of 30 nm or more is preferably used. Further, an ultrashort pulse laser such as a titanium sapphire laser can be used as the light source 151. In the present embodiment, the AO-OCT optical system including the AO-OCT unit 102 shares a part of the optical system with the AO-SLO optical system including the AO-SLO unit 101, so that the light source 151 is the AO-SLO unit. It is desirable that light having a wavelength different from that of the light source 141 of 101 is emitted, and that the light is branched by a dichroic mirror or the like.

The light emitted from the light source 151 is guided to the fiber coupler 152 through the single mode optical fiber. The fiber coupler 152 branches the light emitted from the light source 151 into the measurement optical path L1 and the reference optical path L2. In the present embodiment, the fiber coupler 152 has a branching ratio of 10:90, and 10% of the input light amount goes to the measurement light path L1.

The light that has passed through the measurement light path L1 is irradiated as parallel light rays by the collimator 103 as measurement light. The polarization of the emitted measurement light is adjusted by a polarization regulator (not shown) provided in the single-mode optical fiber as the measurement light path L1. As another configuration, there is also a configuration in which an optical component for adjusting polarization is arranged in the optical path after being emitted from the collimator 103. Further, the measurement optical path L1 may include an optical element for adjusting the dispersion characteristic of the measurement light and an optical element for adjusting the chromatic aberration characteristic.

The measurement light emitted from the AO-OCT unit 102 is combined with the measurement light emitted from the AO-SLO unit 101 by the beam splitter 104 for optical branching, and the measurement light 105 is the same as that of the AO-SLO optical system. Following the optical path, the eye to be inspected E (specifically, the fundus Ef of the eye to be inspected E) is irradiated. The return light scattered / reflected from the eye E to be examined travels in the same path as the outward path in the opposite direction as in the AO-SLO optical system, is reflected by the beam splitter 104 for optical branching, and passes through the optical fiber as the measurement optical path L1. Return to fiber coupler 152.

The wavefront of the return light of the measurement light emitted from the AO-OCT unit 102 is also measured by the wavefront sensor 115 and corrected by the wavefront correction device 108. The wavefront correction method is not limited to such a method, and only when measuring only the wavefront of the return light related to the AO-OCT optical system or only the wavefront of the return light related to the AO-SLO optical system. In the case of measuring, for example, an optical filter is added in front of the wavefront sensor 115. It is also possible to control the return light for measuring the wavefront by dynamically inserting and removing the optical filter and changing it.

On the other hand, in the AO-OCT unit 102, the reference light passing through the reference optical path L2 is emitted from the collimator 1531 of the reference optical system 153 and reflected by the mirror 1532-1 and the mirror 1532-2 to the optical path length variable portion 1533. To reach. Then, the reference light that has reached the optical path length variable unit 1533 is reflected by the optical path length variable unit 1533, and returns to the fiber coupler 152 again through the mirror 1532 and the collimator 1531.

The return light and the reference light of the measurement light that has reached the fiber coupler 152 are combined and guided to the spectroscope 154 as interference light through an optical fiber. The interfering light entering the spectroscope 154 is emitted by the collimator 1541 and separated for each wavelength by the grating 1542. The interference light separated by the grating 1542 is irradiated to the line sensor 1544 through the lens system 1543-1 and the lens system 1543-2. The line sensor 1544 may be composed of a CCD sensor or a CMOS sensor.

The line sensor 1544 of the spectroscope 154 is connected to the control unit 117, and the control unit 117 is a tomographic image of the fundus Ef of the eye E to be inspected based on an electric signal (interference signal) based on the interference light obtained by the line sensor 1544. To configure. At this time, the control unit 117 drives and controls the optical path length variable unit 1533 to acquire a tomographic image of a desired depth position (depth region) in the fundus E of the eye E to be inspected. Further, the control unit 117 also drives and controls the scanning optical system 109-1 and the tracking mirror 109-2 at the same time, and can acquire an interference signal at an arbitrary position (region). The control unit 117 can acquire three-dimensional volume data by generating a tomographic image from the obtained interference signal. After that, the tomographic image of the fundus Ef of the eye E to be inspected is subjected to image processing by the image processing unit 118 and then displayed on the display 119 as an AO-OCT image.

Since AO-OCT can obtain a very high-resolution and high-definition tomographic image, the cell structure such as the ganglion cell of the eye E to be inspected can be clearly visualized, which is very useful for cell-level analysis.

Next, the wave surface sensor 115 will be described in detail.
FIG. 5 is a diagram showing an example of a schematic configuration of the wave surface sensor 115 shown in FIG. Specifically, FIG. 5 shows an example in which a Shack-Hartmann sensor is used as the wavefront sensor 115. Specifically, as shown in FIG. 5A, the wave surface sensor 115 includes a microlens array 132 and a CCD sensor 133.

The microlens array 132 concentrates the return light 131 of the measurement light 105 on the focal plane 134 on the CCD sensor 133.

FIG. 5B is a diagram showing a state in which the wave surface sensor 115 shown in FIG. 5A is viewed from the position indicated by AA'in FIG. 5A. As shown in FIG. 5B, the microlens array 132 shown in FIG. 5A is composed of a plurality of microlenses 135. The return light 131 of the measurement light 105 is focused on the CCD sensor 133 through each of the microlenses 135 shown in FIG. 5 (b). Therefore, as shown in FIG. 5C, the return light 131 of the measurement light 105 is divided into spots 136 as many as the number of microlenses 135, and is focused on the CCD sensor 133.

The adaptive optics control unit 116 shown in FIG. 4 calculates the wave surface of the incident return light 131 from the position of each spot 136 shown in FIG. 5C based on the control of the control unit 117. For example, the adaptive optics control unit 116 calculates the inclination of the wavefront of the return light 131 at each aberration measurement point from the difference between the reference position when there is no aberration in the wavefront of each spot 136 and the position measured by the wavefront sensor 115. .. Further, for example, the adaptive optics control unit 116 can calculate the Zernike coefficient from the difference between the reference position of each spot 136 and the position measured by the wavefront sensor 115.

FIG. 6 is a diagram for explaining the wavefront measured by the wavefront sensor 115 shown in FIG. In FIG. 6, the same reference numerals are given to the configurations similar to those shown in FIG. 5, and detailed description thereof will be omitted.

FIG. 6A shows the wavefront 137 of the return light 131 having spherical aberration. In the case of the example shown in FIG. 6A, the return light 131 of the measurement light 105 is formed by the wave surface 137 shown in FIG. 6A. The return light 131 of the measurement light 105 is focused by the microlens array 132 at a position in the local perpendicular direction of the wavefront 137 on the CCD sensor 133. FIG. 6B is a diagram showing an example of a light condensing state in the CCD sensor 133 of the return light 131 in this case. Since the return light 131 has spherical aberration, the spot 136 is focused in a biased state toward the central portion of the CCD sensor 133 as shown in FIG. 6 (b). By calculating this position, the wave front of the return light 131 is detected. Here, as the wavefront sensor 115, a Shack-Hartmann sensor having 30 × 40 microlens arrays 132 was used.

When the eyepiece 110-1 and the eyepiece 110-2 shown in FIG. 4 have an optical system corresponding to the diopter of the eye E to be inspected, the eyepiece 110-1 and the eyepiece 110-2 are satisfactorily adjusted. It is important that it is done. By doing so, it is not necessary to correct the defocus component which occupies most of the aberration of the eye E to be inspected by the wave surface correction device 108 when the aberration correction process is executed. Further, the correct aberration can be measured by cutting unnecessary light by the aperture 114 arranged immediately before the wavefront sensor 115. However, if the defocus of the aberration of the eye E to be inspected is not corrected, the return light from the retina of the eye E to be inspected, which should be passed through, is also spread in the aperture 114, and most of the light is cut by the aperture 114. Will end up. Further, in order to operate the aberration correction processing as effectively as possible, it is important to place the eye E to be inspected at the correct position, and the subject is covered while observing the measurement results of the anterior eye monitor and the wave surface sensor 115 (not shown). It is necessary to adjust the position of optometry E. The measurement result of the wavefront sensor 115 displayed is, for example, an image called a Hartmann image shown in FIG. 6 (b).

FIG. 7 is a flowchart showing an example of a processing procedure in the image processing method by the ophthalmologic imaging apparatus 100 applied as the image processing apparatus according to the first embodiment of the present invention. Specifically, FIG. 7 is a flowchart showing an example of a processing procedure for acquiring an aberration-corrected image which is the teacher data for learning of the present embodiment.

First, in step S110, the control unit 117, which receives an instruction from the inspector, drives the light source 151 and the like to start light irradiation. It should be noted that the eye E to be inspected will be described from a state in which the eye E to be inspected is roughly aligned with the ophthalmologic imaging apparatus 100 by a known method. In the present embodiment, since the light for image capture and the light for wave surface measurement are the same, in step S110, the wave surface of the fundus Ef of the eye E to be inspected and the return light of the measurement light 105 can be measured. ..

Subsequently, in step S120, the control unit 117 finely adjusts the face position of the eye E to be inspected, adjusts the focus, and the like based on the instruction from the inspector using the input device. At this time, the focus value related to the focus adjustment may be configured to input a value measured in advance by another device.

Subsequently, in step S130, the wavefront sensor 115 acquires the Hartmann image shown in FIG. 6B described above, and outputs the Hartmann image to the adaptive optics control unit 116.

Subsequently, in step S140, the adaptive optics control unit 116 performs the focusing position correction process based on the Hartmann image obtained in step S130. The tomographic image of the eye E to be inspected at the corrected in-focus position obtained here is not in focus up to all tomographic images (shallow depth of focus), for example, as shown in image [3] -1 shown in FIG. It is a tomographic image.

Subsequently, in step S150, the adaptive optics control unit 116 performs aberration correction processing. Specifically, the adaptive optics control unit 116 measures the aberration of the return light with the wave surface sensor 115 and acquires the aberration information to calculate the aberration correction amount based on the measurement result, and the wave surface correction device based on this. Aberration correction processing is performed by driving 108.

Here, the aberration is measured by measuring the spot 136 of the Shack-Hartmann sensor used as the wave surface sensor 115 and calculating the amount of movement (deviation amount) of the spot position of each measurement point from the reference position. Generally, the amount of movement is represented by the amount of displacement in the X and Y directions. In this embodiment, since the Shack-Hartmann sensor of 30 × 40 microlens arrays 132 is used, when the measurement light is incident on all the microlens arrays 132, the measurement light is 1200 at 30 × 40. The amount of movement of the spot 136 at the measurement point is calculated. The adaptive optics control unit 116 calculates the aberration correction amount from the movement amount data of the spot 136. In addition to this method, it is also possible to calculate the Zernike coefficient for expressing the wavefront of the return light from the measured movement amount of the spot 136, and it is possible to control the wavefront correction device 108 based on this. Needless to say,

Subsequently, in step S160, the control unit 117 takes an image of the eye E to be inspected in a state where the in-focus position and the aberration are corrected. Here, for example, the tomographic image acquired by the photographing is referred to as the image [3] -1 shown in FIG.

In order to acquire a tomographic image with a deep depth of focus (wide in focus), the image is taken in step S160, and then the process returns to step S140. This is because the tomographic image acquired in step S160 described above is an image having a shallow depth of focus (the focused vertical width is narrow). Then, the focus position is corrected from the focus position corrected in step S140 (for example, the image [3] -1 shown in FIG. 2) to the focus position such as the image [4] -1 shown in FIG. do. After that, the aberration is corrected again and the photograph is taken. For example, the tomographic image acquired by the photographing is referred to as the image [4] -1 shown in FIG.

In this way, steps S140 to S160 are repeatedly executed until there is a request to end shooting. For example, a single-layer image of images [1] -1 to [6] -1 shown in FIG. 2 is acquired, and they are combined to form a fault with a deep depth of focus (wide in focus). Generate an image.

Next, the details of the in-focus position correction process in step S140 will be described.

In the focusing position correction process of step S140, first, in step S141, the adaptive optics control unit 116 recognizes the spot image output from the Hartmann image and calculates (detects) the number of recognized spots n. At this time, when the number of spots is sufficient for shooting, the process proceeds to step S142, which is the next step. On the other hand, if the number of spots is not sufficient, the process returns to step S130 after a certain period of time has elapsed.

Proceeding to step S142, the adaptive optics control unit 116 calculates the focus value F from the measured spot 136 by a known method. When the focus value is corrected to be sufficiently small, the process proceeds to step S150, which is the next step. If the focus value is not corrected to be sufficiently small, the process returns to step S130 after a certain period of time has elapsed.

Here, the single-layer image obtained in step SS160 (for example, the image [3] -1 shown in FIG. 2) and the total tomographic image obtained by combining the single-layer image (for example, obtained as a result) shown in FIG. 2 are combined. The total tomographic image 1 ”) is stored in a memory (not shown). Then, in the present embodiment, the entire tomographic image 1 is obtained by combining and combining a plurality of images having a vertical width exceeding the width of the depth of focus of the measurement light with respect to the eye E to be inspected and having different focusing positions in the vertical width. Used as teacher data.

<Learning>
Next, an example in which an all-tomographic image is learned from a single-layer image obtained by imaging with the ophthalmologic imaging apparatus 100 will be described. The single-layer image referred to here is, for example, the image [3] -1 shown in FIG. 2, and the total tomographic image referred to here is, for example, the “total tomographic image 1 desired to be obtained as a result” shown in FIG. ".

FIG. 8 is a diagram showing an example of a schematic configuration of the image processing unit 118 shown in FIG. Here, in FIG. 8, the control unit 117 and the display 119, which are connected to the image processing unit 118, are also shown.

As shown in FIG. 8, the image processing unit 118 includes an image acquisition unit 810, a CNN processing unit 820, and a learning unit 830.

The image acquisition unit 810 acquires a single-layer image of the fundus Ef of the eye E to be inspected via the control unit 117.

The CNN processing unit 820 performs image processing using a convolutional neural network, and generates an all-tomographic image from the single-layer image acquired by the image acquisition unit 810. In this embodiment, the model is trained by CNN (Convolutional Neural Network), which is a kind of deep learning.

The learning unit 830 performs learning processing of the convolutional neural network referred to by the CNN processing unit 820.

FIG. 9 is a diagram showing an example of a schematic configuration of the CNN processing unit 820 shown in FIG. As shown in FIG. 8, the CNN processing unit 820 includes an image encoder network 821 and a decoder network 822. The CNN processing unit 820 includes a convolution layer, an activation layer, a downsampling layer, an upsampling layer, and a merger layer. The convolution layer is a layer that performs a convolution process on an input value group according to parameters such as a kernel size, a number of filters, a stride value, and a dilation value of a set filter. The kernel size of the filter may be changed according to the number of dimensions of the input image. The activation layer determines the activation of the sum of the input signals and is composed of a step function, a sigmoid function and a ReLU (Rectifier Liner Unit). The downsampling (Pooling) layer is a process such as Max Pooling processing in which the number of output value groups is reduced from the number of input value groups by thinning out or synthesizing the input value groups. The UpSampling layer makes the number of output value groups larger than the number of input value groups by duplicating the input value group or adding the interpolated value from the input value group, such as linear interpolation processing. It is a process to be done a lot. The merger layer is a process in which a group of values such as an output value group of a certain layer and a group of pixel values constituting an image are input from a plurality of sources, and they are concatenated or added to be combined. It is a layer to do.

The image encoder network 821 performs a convolution operation by a plurality of encoding layers on the above-mentioned single-layer image which is an input image. Each encode layer has one or more convolution processing units and a pooling processing unit, and is configured so that the results of each layer can be internally retained. As the training data, a single-layer image is input, and as an output, all tomographic images are used as training data. Using this training data, a model is machine-learned, and the trained model is used as a trained model 300 to obtain network parameters for converting a single-layer image to an all-tomographic image.

<Shooting using a combination of single-layer images by CNN>
Next, a method of taking a tomographic image using the trained model 300 described above will be described. The apparatus configuration is as shown in FIGS. 4 to 6 and 8 to 9.

Specifically, as shown in FIG. 4, the wave surface correction device 108 is connected to the adaptive optics control unit 116, and modulates the focus position and aberration so as to be in-focus position and aberration instructed by the adaptive optics control unit 116. The image processing unit 118 is based on the obtained single-layer image data and all tomographic image data obtained by the above-mentioned learning (learned model 300), and the focal depth focused on the entire vertical width of all tomography. A tomographic image with a deep (wide vertical focus position) is generated and displayed on the display 119.

FIG. 10 is a flowchart showing an example of a processing procedure in the image processing method by the ophthalmologic imaging apparatus 100 applied as the image processing apparatus according to the first embodiment of the present invention. Specifically, FIG. 10 is a flowchart showing an example of a processing procedure when generating output data 230 using the trained model 300.

Steps S210 to S260 of FIG. 10 are the same as the processes of steps S110 to S160 of FIG. 7, respectively. At this time, the focusing position (condensing position) of the single-layer image obtained in step S260 is adjusted to the characteristic portion of the single-layer image. Here, it is assumed that the focus is on an arbitrary position (different for each patient) in the total vertical width of all faults having a high brightness value.

By inputting the single-layer image obtained in step S260 and the focusing position (focusing position) into the network of the trained model 300 obtained by the above-mentioned learning, it is possible to generate an all-tomographic image. Specifically, to explain with reference to FIG. 1, for example, the single-layer image obtained in step S260 has a focusing position (condensing position) and a shallow depth of focus (condensing position) as represented by the input data 220. It is a tomographic image (with a narrow vertical width). In the present embodiment, this (one) tomographic image is used as the input data 220, and the trained model 300 is used to combine the single-layer images by CNN to output data relating to the total vertical width of all the tomographic images shown in FIG. Generate 230.

Here, a description will be given with reference to FIGS. 3 and 8 as an example.
Here, it is assumed that the single-layer image obtained in step S260 is the image [2] -1. The input information for the trained model 300 is "image information with a shallow depth of focus (the vertical width of the focusing position (condensing position) is narrow)" and "high brightness value in the total vertical width of all faults". Arbitrary focusing position (focusing position) (varies for each patient) ". The input information is set in the image acquisition unit 810 of the image processing unit 118, the CNN processing unit 820 performs image processing using the convolutional neural network, and the learning unit 830 learns the convolutional neural network referred to by the CNN processing unit 820. Perform processing. For example, in the derivation engine [2] shown in FIG. 3, a wide-range vertical tomographic image is derived based on a feature amount (slope of brightness change, etc.) based on the luminance distribution in the image information of the input information.
The vertical wide-range tomographic image as the output data (teacher data) 230 derived as described above is displayed on the display 119 and clearly shown to the user. From the above, an arbitrary in-focus position and a single-layer image can be obtained by one shooting in step S260, and by using the trained model 300, the output data (teacher data) 230 which is a vertical wide-range tomographic image can be obtained. Can be derived and displayed.

In the present embodiment, as shown in FIG. 1, the output data (teacher data) 230 generated by using the trained model 300 has a resolution equivalent to that of the first region 222 than the input data 220. The image has a large range. Further, it is preferable that the output data (teacher data) 230 generated by using the trained model 300 is, for example, an image for screening in the eye E to be inspected.

As described above, the ophthalmologic imaging apparatus 100 according to the first embodiment acquires the input data 220 having a vertical width exceeding the width of the depth of focus of the measurement light with respect to the eye E to be inspected. Then, in the ophthalmologic imaging apparatus 100 according to the first embodiment, the resolution is higher than the resolutions of the second regions 221 and 223 excluding the first region 222 related to the width of the depth of focus in the input data 220 by using the trained model 300. Output data (teacher data) 230, which is an image having a resolution and has a vertical width of the input data 220, is generated.
According to such a configuration, an image having a wide vertical width related to the eye E to be inspected can be acquired in a short time with high resolution. More specifically, it is possible to acquire a wide range of high-resolution tomographic images in a short time without taking a plurality of tomographic images at each condensing position (each focusing position).

(Second Embodiment)
Next, a second embodiment of the present invention will be described. In the description of the second embodiment described below, the matters common to the above-mentioned first embodiment will be omitted, and the matters different from the above-mentioned first embodiment will be described.

In the first embodiment described above, for example, as shown in the input data 220 of FIG. 1, the region of the condensing position (first region 222) is a single layer, but there are a plurality of layers. It may be. Further, as a method of generating an image in which the region of the condensing position (first region 222) is a plurality of layers, it is artificially obtained by shifting the focusing position. Specifically, a plurality of single-layer images are intentionally acquired, and for example, when two single-layer images are obtained, a combination of single-layer images by CNN is used to generate a "total tomographic image to be obtained as a result". do.

The schematic configuration of the ophthalmologic imaging apparatus applied as the image processing apparatus according to the second embodiment is the same as the schematic configuration of the ophthalmologic imaging apparatus 100 applied as the image processing apparatus according to the first embodiment shown in FIG.

FIG. 11 is a diagram for explaining an image processing method by an ophthalmologic imaging apparatus 100 applied as an image processing apparatus according to a second embodiment of the present invention. For example, if there is an in-focus position in the fundus Ef of the eye E to be inspected in advance, that position can be selected. Here, the reason for making it possible to select is that there are various scenes for which there is a purpose to aim at the position in advance, such as when a finding is recognized at the position as prior information or when it is desired to observe a change over time of the position. be. That is, the doctor demands an unprocessed true image rather than an image derived by the learning function depending on the part.

In the present embodiment, the information obtained in the photographing in step S260 of FIG. 10 is images 1110 and 1120 including "selected single-layer images 1111 and 1121" and "arbitrary focusing position and single-layer image". The input information includes "selected single-layer image information", "image information with a shallow focal depth (the vertical width of the focusing position (condensing position) is narrow)", and "all tomographic images with high brightness value". Arbitrary focusing position (condensing position) in the entire vertical width (varies for each patient) ". The input information is set in the image acquisition unit 810 in the image processing unit 118, and the CNN processing unit 820 performs image processing in a range excluding "selected single-layer images 1111 and 1121" using a convolutional neural network. , The learning unit 830 performs the learning process of the convolutional neural network referred to by the CNN processing unit 820. For example, in the derivation engine [2] shown in FIG. 3, a vertical wide-range tomographic image is generated by the trained model 300 based on the feature amount (slope of brightness change, etc.) based on the brightness distribution in the image information of the input information. It is something to derive.

After that, the vertical wide-range tomographic image generated by the trained model 300 is combined with the "selected single-layer images 1111 and 1121", and the vertical wide-range tomographic image as output data (teacher data) 1130 is displayed on the display 119. And clearly indicate to the user. Here, in FIG. 11, the partial image of the single-layer image 1131 and the partial image of the single-layer image 1132 of the output data (teacher data) 1130 are the partial image of the selected single-layer image 1111 and the selected partial image, respectively. It corresponds to a partial image of the single-layer image 1121. That is, the output data (teacher data) 1130 is an image obtained by synthesizing the partial image of the single-layer image 1111 and the partial image of the single-layer image 1121 in the images 1110 and 1120 with the image obtained by using the trained model 300. ing.

The difference between the first embodiment and the present embodiment described above is that, in the present embodiment, the derivation process is performed in a range excluding "selected single-layer images 1111 and 1121", and the derived vertical width is wide. The point is that the tomographic image and the selected single-layer image (true image) are then combined. As shown in the conceptual diagram of FIG. 11, output data (teacher data) 1130 related to all tomographic images is obtained based on, for example, a partial image of a single-layer image 1111 with a lesion and a partial image of another single-layer image 1121. The generation will generate a tomographic image that also covers the lesion, and the processing time can be shortened.

(Third Embodiment)
Next, a third embodiment of the present invention will be described. In the description of the third embodiment described below, the matters common to the above-mentioned first and second embodiments are omitted, and the matters different from the above-mentioned first and second embodiments are omitted. Will be explained.

In the first and second embodiments described above, an example in which the vertical width of the single-layer image is divided into six layers, for example, images [1] -1 to [6] -1 shown in FIG. 2 has been described. In the present invention, the number of layers is not limited to six.

(Other embodiments)
In the above-described embodiment of the present invention, the embodiment in which the image processing unit 118 is configured inside the ophthalmologic imaging apparatus 100 and the ophthalmologic imaging apparatus 100 is applied as the image processing apparatus according to the present invention has been described. It is not limited to the form. For example, the present invention also includes a mode in which the image processing unit 118 is configured as an external device external to the ophthalmic imaging device 100, and the external device is applied as the image processing device according to the present invention.

The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.
This program and a computer-readable storage medium that stores the program are included in the present invention.

It should be noted that the above-described embodiments of the present invention merely show examples of embodiment in carrying out the present invention, and the technical scope of the present invention should not be construed in a limited manner by these. Is. That is, the present invention can be implemented in various forms without departing from the technical idea or its main features.

100: Ophthalmologic imaging device, 101: AO-SLO unit, 102: AO-OCT unit, 103: Collimator. 104: Beam splitter for optical branching, 105: Measurement light, 106: Optical divider, 107: Reflective mirror, 108: Wave surface correction device, 109-1: Scanning optical system, 109-2: Tracking mirror, 113: Relay optics System, 114: aperture, 115: wave surface sensor, 116: adaptive optics control unit, 117: control unit, 118: image processing unit, 119: display, E: eye to be examined, Ef: fundus, 200: OCT image, 210: AO -OCT image, 220: input data, 230: output data (teacher data), 300: trained model

Claims (7)

An acquisition means for acquiring a first image having a vertical width exceeding the width of the depth of focus of the measurement light with respect to the eye to be inspected.
Using the trained model, an image having a resolution higher than the resolution of the second region excluding the first region related to the width of the depth of focus in the first image and having the vertical width is obtained. The generation means to generate and
An image processing device characterized by having. The trained model uses a plurality of images having different positions of the first region in the vertical width as input data, and outputs images generated by using the partial images of the first region of each of the plurality of images as output data. The image processing apparatus according to claim 1, wherein the image processing apparatus is obtained by learning the teacher data. The image processing apparatus according to claim 1 or 2, wherein the second image has a larger range having a resolution equivalent to that of the first region than the first image. The image processing apparatus according to any one of claims 1 to 3, wherein the second image is an image for screening in the eye to be inspected. The second image is an image obtained by synthesizing a partial image of the first region in the first image with an image obtained by using the trained model, according to claims 1 to 4. The image processing apparatus according to any one item. The acquisition step of acquiring a first image having a vertical width exceeding the width of the depth of focus of the measurement light with respect to the eye to be inspected, and
Using the trained model, an image having a resolution higher than the resolution of the second region excluding the first region related to the width of the depth of focus in the first image and having the vertical width is obtained. Generation steps to generate and
An image processing method characterized by having. A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 5.

Images