JP2023009530A - Image processing method, image processing device, and program - Google Patents

Abstract

To accurately calculate an amount of movement of an eye to be examined.SOLUTION: An image processing method executed by a processor includes: a step for acquiring an ocular fundus image of an eye to be examined at least twice at different timings; a step for extracting a feature point of a choroid blood vessel from each of the ocular fundus images acquired at least twice; and a step for calculating an amount of movement of the eye to be examined using the feature points of the choroid blood vessel extracted from each of the ocular fundus images acquired at least twice.SELECTED DRAWING: Figure 5

Description

The technology of the present disclosure relates to an image processing method, an image processing apparatus, and a program.

Patent Literature 1 discloses a tracking method for moving an optical system in accordance with the movement of an eye to be examined. Conventionally, there has been a demand for capturing a fundus image without blurring.

U.S. Patent Application Publication No. 2019/0059723

A first aspect of the technology of the present disclosure is an image processing method performed by a processor, comprising: obtaining a fundus image of an eye to be examined at least twice at different timings; , extracting feature points of choroidal blood vessels; and calculating an amount of movement of the subject's eye using the feature points of the choroidal vessels extracted from each of the fundus images acquired at least twice. .

A second aspect of the technology of the present disclosure is an image processing apparatus comprising a processor, wherein the processor acquires a fundus image of an eye to be inspected at least twice at different timings; extracting feature points of choroidal blood vessels from each of the fundus images; and calculating the amount of movement of the subject's eye using the feature points of the choroidal blood vessels extracted from each of the fundus images acquired at least twice. and performing image processing.

A third aspect of the technology of the present disclosure includes a step of obtaining, in a computer, fundus images of an eye to be examined at least twice at different timings, and from each of the fundus images obtained at least twice, characteristic points of choroidal blood vessels. and calculating the movement amount of the subject's eye using the characteristic points of the choroidal blood vessels extracted from each of the fundus images acquired at least twice. be.

1 is a block diagram of an ophthalmic system 100; FIG. 1 is a schematic configuration diagram showing the overall configuration of an ophthalmologic apparatus 110; FIG. 3 is a functional block diagram of the CPU 16A of the control device 16 of the ophthalmologic apparatus 110. FIG. 4 is a flow chart showing a program executed by a CPU 16A of the ophthalmologic apparatus 110; FIG. 5 is a flowchart of a subroutine of eye tracking processing in step 306 of FIG. 4; FIG. FIG. 4 shows a UWF-SLO fundus image 400G. FIG. 4 shows a UWF-SLO fundus image 400G superimposed with a position 402 for acquiring OCT data. 4 is a diagram showing a UWF-SLO fundus image 400G in which a position 402 for acquiring OCT data and a region 400 for acquiring a rectangular SLO fundus image are superimposed. FIG. Fig. 5 shows a screen 500 of the display of viewer 150;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

The configuration of the ophthalmic system 100 will be described with reference to FIG. As shown in FIG. 1, an ophthalmic system 100 includes an ophthalmic device 110, an eye axial length measuring device 120, a management server device (hereinafter referred to as "server") 140, and an image display device (hereinafter referred to as "viewer"). 150 and . The ophthalmologic device 110 acquires a fundus image. The axial length measuring device 120 measures the axial length of the subject. The server 140 stores the fundus image obtained by photographing the fundus of the subject with the ophthalmologic apparatus 110 in association with the ID of the subject. The viewer 150 displays medical information such as fundus images acquired from the server 140 .

The ophthalmologic device 110 is an example of the “image processing device” of the technology of the present disclosure.

The ophthalmologic apparatus 110 , axial length measuring instrument 120 , server 140 and viewer 150 are interconnected via network 130 . Network 130 is any network such as LAN, WAN, the Internet, or a wide area Ethernet network. For example, if the ophthalmic system 100 is built in one hospital, a LAN can be used as the network 130 .

Note that other ophthalmologic equipment (inspection equipment for visual field measurement, intraocular pressure measurement, etc.) and a diagnosis support device that performs image analysis using artificial intelligence are connected via the network 130 to the ophthalmic equipment 110, the eye axial length measuring device 120, and the server. 140 and viewer 150 .

Next, the configuration of the ophthalmologic apparatus 110 will be described with reference to FIG.

For convenience of explanation, a scanning laser ophthalmoscope is referred to as "SLO". Also, optical coherence tomography is referred to as "OCT".

When the ophthalmologic apparatus 110 is installed on a horizontal plane, the horizontal direction is the "X direction", and the vertical direction to the horizontal plane is the "Y direction". Let the direction be “Z direction”. Therefore, the X, Y and Z directions are perpendicular to each other.

Ophthalmic device 110 includes imager 14 and controller 16 . The imaging device 14 includes an SLO unit 18, an OCT unit 20, and an imaging optical system 19, and acquires a fundus image of the eye 12 to be examined. The two-dimensional fundus image acquired by the SLO unit 18 is hereinafter referred to as an SLO fundus image. A tomographic image of the retina, a front image (en-face image), and the like created based on the OCT data acquired by the OCT unit 20 are referred to as OCT images.

The control device 16 comprises a computer having a CPU (Central Processing Unit) 16A, a RAM (Random Access Memory) 16B, a ROM (Read-Only Memory) 16C, and an input/output (I/O) port 16D. ing. The ROM 16C stores an image processing program, which will be described later. The control device 16 may further include an external storage device and store the image processing program in the external storage device.

The image processing program is an example of the "program" of the technology of the present disclosure. The ROM 16C (or external storage device) is an example of the "memory" and "computer-readable storage medium" of the technology of the present disclosure. The CPU 16A is an example of the "processor" of the technology of the present disclosure. The control device 16 is an example of the "computer program product" of the technology of the present disclosure.

Controller 16 includes an input/display device 16E connected to CPU 16A via I/O port 16D. The input/display device 16E has a graphic user interface that displays an image of the subject's eye 12 and receives various instructions from the user. Graphic user interfaces include touch panel displays.

The control device 16 has a communication interface (I/F) 16F connected to an I/O port 16D. The ophthalmologic apparatus 110 is connected to an axial length measuring instrument 120 , a server 140 and a viewer 150 via a communication interface (I/F) 16F and a network 130 .

As noted above, in FIG. 2, the controller 16 of the ophthalmic device 110 includes an input/display device 16E, although the techniques of this disclosure are not so limited. For example, the controller 16 of the ophthalmic device 110 may not have the input/display device 16E, but may have a separate input/display device physically separate from the ophthalmic device 110. FIG. In this case, the display device comprises an image processor unit operating under the control of the CPU 16A of the control device 16. FIG. The image processor unit may display an SLO fundus image, an OCT image, etc., based on the image signal output by the CPU 16A.

The imaging device 14 operates under the control of the CPU 16A of the control device 16. FIG. The imaging device 14 includes an SLO unit 18 , an imaging optical system 19 and an OCT unit 20 . The imaging optical system 19 includes a first optical scanner 22 , a second optical scanner 24 and a wide-angle optical system 30 .

The first optical scanner 22 two-dimensionally scans the light emitted from the SLO unit 18 in the X direction and the Y direction. The second optical scanner 24 two-dimensionally scans the light emitted from the OCT unit 20 in the X direction and the Y direction. The first optical scanner 22 and the second optical scanner 24 may be optical elements capable of deflecting light beams, such as polygon mirrors and galvanometer mirrors. Moreover, those combinations may be sufficient.

The wide-angle optical system 30 includes an objective optical system (not shown in FIG. 2) having a common optical system 28 and a combiner 26 that combines the light from the SLO unit 18 and the light from the OCT unit 20 .

The objective optical system of the common optical system 28 may be a reflective optical system using a concave mirror such as an elliptical mirror, a refractive optical system using a wide-angle lens, or a catadioptric system combining concave mirrors and lenses. good. By using a wide-angle optical system with an elliptical mirror and wide-angle lens, it is possible to image not only the central part of the fundus where the optic disc and macula exist, but also the equatorial part of the eyeball and the peripheral part of the fundus where vortex veins exist. It becomes possible.

The wide-angle optical system 30 realizes observation in a wide field of view (FOV: Field of View) 12A at the fundus. The FOV 12A indicates a range that can be photographed by the photographing device 14. FIG. FOV12A can be expressed as a viewing angle. A viewing angle can be defined by an internal illumination angle and an external illumination angle in this embodiment. The external irradiation angle is an irradiation angle defined by using the pupil 27 as a reference for the irradiation angle of the light beam irradiated from the ophthalmologic apparatus 110 to the eye 12 to be examined. Also, the internal illumination angle is an illumination angle defined with the center O of the eyeball as a reference for the illumination angle of the luminous flux that illuminates the fundus. The external illumination angle and the internal illumination angle are in correspondence. For example, an external illumination angle of 120 degrees corresponds to an internal illumination angle of approximately 160 degrees. In this embodiment, the internal illumination angle is 200 degrees.

Here, an SLO fundus image obtained by photographing at an internal illumination angle of 160 degrees or more is referred to as a UWF-SLO fundus image. Note that UWF is an abbreviation for UltraWide Field.

The SLO system is implemented by the controller 16, SLO unit 18, and imaging optical system 19 shown in FIG. Since the SLO system includes the wide-angle optical system 30, it enables fundus imaging with a wide FOV 12A.

The SLO unit 18 includes a plurality of light sources, for example, a B light (blue light) light source 40, a G light (green light) light source 42, an R light (red light) light source 44, and an IR light (infrared (for example, near infrared) light source). and optical systems 48, 50, 52, 54, and 56 that reflect or transmit the light from the light sources 40, 42, 44, and 46 and guide them to one optical path. Optical systems 48, 50, 56 are mirrors and optical systems 52, 54 are beam splitters. The B light is reflected by the optical system 48, transmitted through the optical system 50, and reflected by the optical system 54, the G light is reflected by the optical systems 50 and 54, and the R light is transmitted by the optical systems 52 and 54. , IR light is reflected by the optical systems 56, 52 and directed to one optical path, respectively.

The SLO unit 18 is configured to be switchable between light sources that emit laser light of different wavelengths, such as a mode that emits G light, R light, and B light, and a mode that emits infrared light, or a combination of light sources that emit laser light. Although the example shown in FIG. 2 includes four light sources, a B light (blue light) light source 40, a G light source 42, an R light source 44, and an IR light source 46, the technology of the present disclosure is not limited to For example, SLO unit 18 may further include a white light source to emit light in various modes, such as a mode that emits only white light.

Light incident on the imaging optical system 19 from the SLO unit 18 is scanned in the X and Y directions by the first optical scanner 22 . The scanning light passes through the wide-angle optical system 30 and the pupil 27 and irradiates the posterior segment of the eye 12 to be examined. Reflected light reflected by the fundus enters the SLO unit 18 via the wide-angle optical system 30 and the first optical scanner 22 .

The SLO unit 18 includes a beam splitter 64 that reflects B light and transmits light other than B light from the posterior segment (for example, fundus) of the eye 12 to be inspected, and G light that has passed through the beam splitter 64. A beam splitter 58 that reflects light and transmits light other than G light is provided. The SLO unit 18 has a beam splitter 60 that reflects the R light and transmits other than the R light out of the light transmitted through the beam splitter 58 . The SLO unit 18 has a beam splitter 62 that reflects IR light out of the light transmitted through the beam splitter 60 .

The SLO unit 18 has a plurality of photodetectors corresponding to a plurality of light sources. The SLO unit 18 includes a B light detection element 70 that detects B light reflected by the beam splitter 64 and a G light detection element 72 that detects G light reflected by the beam splitter 58 . The SLO unit 18 includes an R photodetector element 74 that detects R light reflected by the beam splitter 60 and an IR photodetector element 76 that detects IR light reflected by the beam splitter 62 .

Light (reflected light reflected by the fundus) that is incident on the SLO unit 18 via the wide-angle optical system 30 and the first optical scanner 22 is reflected by the beam splitter 64 in the case of B light, and is detected by the B light detection element 70 . G light is transmitted through the beam splitter 64 , reflected by the beam splitter 58 , and received by the G light detection element 72 . The incident light, in the case of R light, passes through the beam splitters 64 and 58 , is reflected by the beam splitter 60 , and is received by the R light detection element 74 . In the case of IR light, the incident light passes through beam splitters 64 , 58 and 60 , is reflected by beam splitter 62 , and is received by IR photodetector 76 . The CPU 16A uses the signals detected by the B photodetector 70, G photodetector 72, R photodetector 74, and IR photodetector 76 to generate a UWF-SLO fundus image.

A UWF-SLO fundus image (also referred to as a UWF fundus image or an original fundus image as described later) includes a UWF-SLO fundus image obtained by photographing the fundus in G color (G color fundus image) and a fundus image in R color. A UWF-SLO fundus image (R-color fundus image) obtained by photographing in color. The UWF-SLO fundus image includes a UWF-SLO fundus image obtained by photographing the fundus in B color (B color fundus image) and a UWF-SLO fundus image obtained by photographing the fundus in IR (IR fundus image). image).

Controller 16 also controls light sources 40, 42, 44 to emit light simultaneously. By simultaneously photographing the fundus of the subject's eye 12 with B light, G light, and R light, a G-color fundus image, an R-color fundus image, and a B-color fundus image whose respective positions correspond to each other are obtained. An RGB color fundus image is obtained from the G color fundus image, the R color fundus image, and the B color fundus image. The control device 16 controls the light sources 42 and 44 to emit light at the same time, and the fundus of the subject's eye 12 is photographed simultaneously with the G light and the R light, thereby obtaining a G-color fundus image and an R-color fundus image corresponding to each other at each position. A fundus image is obtained. An RG color fundus image is obtained from the G color fundus image and the R color fundus image.

Specifically, UWF-SLO fundus images include B-color fundus images, G-color fundus images, R-color fundus images, IR fundus images, RGB color fundus images, and RG color fundus images. Each image data of the UWF-SLO fundus image is transmitted from the ophthalmologic apparatus 110 to the server 140 via the communication interface (I/F) 16F together with the subject information input via the input/display device 16E. . Each image data of the UWF-SLO fundus image and information of the subject are stored in the storage device 254 correspondingly. The information of the subject includes, for example, subject ID, name, age, visual acuity, distinction between right eye/left eye, and the like. The subject information is entered by the operator through the input/display device 16E.

The OCT system is implemented by the control device 16, OCT unit 20, and imaging optical system 19 shown in FIG. Since the OCT system includes the wide-angle optical system 30, it enables fundus imaging with a wide FOV 12A, as in the above-described SLO fundus imaging. The OCT unit 20 includes a light source 20A, a sensor (detection element) 20B, a first optical coupler 20C, a reference optical system 20D, a collimating lens 20E, and a second optical coupler 20F.

Light emitted from the light source 20A is branched by the first optical coupler 20C. One of the split beams is collimated by the collimating lens 20E and then enters the imaging optical system 19 as measurement light. The measurement light is scanned in the X and Y directions by the second optical scanner 24 . The scanning light passes through the wide-angle optical system 30 and the pupil 27 and illuminates the fundus. The measurement light reflected by the fundus enters the OCT unit 20 via the wide-angle optical system 30 and the second optical scanner 24, passes through the collimating lens 20E and the first optical coupler 20C, and reaches the second optical coupler 20F. incident on

The other light emitted from the light source 20A and branched by the first optical coupler 20C enters the reference optical system 20D as reference light, passes through the reference optical system 20D, and enters the second optical coupler 20F. do.

These lights incident on the second optical coupler 20F, that is, the measurement light reflected by the fundus and the reference light are interfered by the second optical coupler 20F to generate interference light. Interference light is received by sensor 20B. The CPU 16A performs signal processing such as Fourier transform on the detection signal detected by the sensor 20B to generate OCT data. The CPU 16A generates OCT images such as tomographic images and en-face images based on the OCT data.

Here, the OCT system can acquire OCT data of the imaging area realized by the wide-angle optical system 30 .

The OCT data, the tomographic image, and the en-face image generated by the CPU 16A are transmitted from the ophthalmologic apparatus 110 to the server 140 via the communication interface (I/F) 16F together with information on the subject. Various OCT images such as OCT data, tomographic images and en-face images are associated with subject information and stored in the storage device 254 .

In this embodiment, the light source 20A exemplifies a wavelength sweep type SS-OCT (Swept-Source OCT), but SD-OCT (Spectral-Domain OCT), TD-OCT (Time-Domain OCT), etc. Various types of OCT systems may be used.

Next, the axial length measuring device 120 will be described. The axial length measuring device 120 measures the axial length of the subject's eye 12 in the axial direction.

Axial length measuring device 120 transmits the measured axial length to server 140 . The server 140 stores the subject's eye axial length in association with the subject ID.

Next, with reference to FIG. 3, various functions realized by the CPU 16A of the ophthalmologic apparatus 110 executing the control program for the ophthalmic equipment will be described. The control program for the ophthalmic equipment has an imaging control function, a display control function, an image processing function, and a processing function. When the CPU 16A executes the control program for the ophthalmologic equipment having these functions, the CPU 16A functions as an imaging control unit 202, a display control unit 204, an image processing unit 206, and a processing unit 208, as shown in FIG. .

Next, image processing of the ophthalmologic apparatus 110 will be described in detail with reference to FIG. The CPU 16A of the control device 16 of the ophthalmologic apparatus 110 executes the image processing program of the ophthalmologic apparatus, thereby realizing the control of the ophthalmologic apparatus shown in the flowchart of FIG.

The processing shown in the flowchart of FIG. 4 is an example of the “image processing method” of the technique of the present disclosure.

The operator of the ophthalmologic apparatus 110 adjusts the position of the subject's eye 12 by having the subject rest his/her chin on a support portion (not shown) of the ophthalmologic apparatus 110 .

The display control unit 204 of the ophthalmologic apparatus 110 displays a menu screen for inputting subject information and mode selection on the screen of the input/display device 16E. Modes include an SLO mode for acquiring an SLO fundus image and an OCT mode for acquiring an OCT fundus image. When the operator inputs the subject's information via the input/display device 16E and selects the OCT mode, the CPU 16A starts executing the ophthalmic equipment control program shown in FIG.

In step 302 , the imaging control unit 202 controls the SLO unit 18 and the imaging optical system 19 to obtain a first fundus image of the eye fundus of the subject's eye 12 , specifically, B light source 40 , G light source 42 and R light source 44 . to acquire UWF-SLO fundus images at three different wavelengths. As described above, the UWF-SLO fundus image includes a G-color fundus image, an R-color fundus image, a B-color fundus image, and an RGB color fundus image.
A UWF-SLO fundus image is an example of the "first fundus image" of the technology of the present disclosure.

At step 304, the display controller 204 displays the UWF-SLO fundus image 400G on the display of the input/display device 16E. FIG. 6A shows a UWF-SLO fundus image 400G displayed on the display. The UWF-SLO fundus image 400G corresponds to an image of an area that can be scanned by the SLO unit 18, and as shown in FIG. Contains 400gg.

A user (operator of the ophthalmologic apparatus 110) uses the displayed UWF-SLO fundus image 400G to select a region for acquiring OCT data (position for acquiring a tomographic image) using a touch panel or an input device (not shown). set. FIG. 6B shows a case where a position 402 for obtaining a tomographic image is set by a straight line in the X direction using the UWF-SLO fundus image 400G. When a position 402 for acquiring a tomographic image is set by a straight line, the position 402 for acquiring a tomographic image is represented by an arrow.

Note that the position 402 for acquiring a tomographic image is not limited to the straight line in the X direction as shown in FIG. A curve or the like connecting two points, or a surface such as a circular area or a rectangular area may be used.

When the tomographic image is acquired at one point, OCT data (referred to as "A scan data") is obtained by scanning one point of the fundus in the depth (optical axis) direction (this scanning is referred to as "A scan"). is obtained.
When the position for acquiring a tomographic image is a line, the A scan is performed a plurality of times while moving along the line (referred to as "B scan") to obtain OCT data ("B scan data”) is obtained.
When the position for acquiring the tomographic image is a plane, the B scan is repeated while moving along the plane (referred to as "C scan") to obtain OCT data (referred to as "C scan data"). ) is obtained. Three-dimensional OCT data is generated from the C-scan data, and a two-dimensional en-face image or the like is generated based on the three-dimensional OCT data.

Then, the processing unit 208 acquires position data (coordinate data, etc.) of a position 402 for acquiring a tomographic image, which is set using the UWF-SLO fundus image 400G. The data indicating the position where the tomographic image is acquired is not limited to coordinate data, and may be a number or the like that roughly indicates the position of the fundus image.

At step 306, eye tracking processing is performed. Eye tracking processing will be described with reference to FIG.

Note that, in the technology of the present disclosure, eye tracking processing is executed immediately after the position data of the position 402 for acquiring the tomographic image, which is set using the UWF-SLO fundus image 400G in step 304, is acquired. is not limited to For example, the operator confirms that the subject's eye 12 and the ophthalmic device 110 are properly aligned. After confirming that the alignment is in an appropriate state, the operator instructs to start OCT imaging by operating buttons or the like on the display of the input/display device 16E. When an operation to start OCT imaging is instructed in this way, eye tracking processing is executed.

At step 350 in FIG. 5, the image processing unit 206 extracts feature points of choroidal blood vessels using the R-color fundus image and the G-color fundus image of the UWF-SLO fundus image. Specifically, the image processing unit 206 first extracts choroidal blood vessels, and extracts characteristic points of the choroidal blood vessels.

First, a method for extracting choroidal blood vessels using an R-color fundus image and a G-color fundus image will be described.

The structure of the eye is such that the vitreous is covered by multiple layers of different structures. The multiple layers include, from innermost to outermost on the vitreous side, the retina, choroid, and sclera. The R light passes through the retina and reaches the choroid. Therefore, the R-color fundus image includes information on blood vessels existing in the retina (retinal vessels) and information on blood vessels existing in the choroid (choroidal vessels). On the other hand, the G light reaches only the retina. Therefore, the G-color fundus image contains only information on blood vessels existing in the retina (retinal blood vessels).

The image processing unit 206 extracts retinal blood vessels from the G-color fundus image by performing image processing such as black hat filter processing on the G-color fundus image. Thus, a retinal blood vessel image in which only retinal blood vessel pixels are extracted from the G-color fundus image is obtained. Black hat filtering is performed by performing N times (N is an integer of 1 or more) expansion processing and N times contraction processing on the image data of the G-color fundus image, which is the original image, and the closing processing performed on this original image. Refers to the process of taking the difference from the obtained image data.

Next, the image processing unit 206 removes retinal blood vessels from the R-color fundus image by inpainting processing or the like using the retinal blood vessel image extracted from the G-color fundus image. The inpainting process is a process of setting a pixel value at a predetermined position so that the difference from the average value of surrounding pixels is within a specific range (for example, 0). That is, the position information of the retinal blood vessels extracted from the G-color fundus image is used to fill in the pixels corresponding to the retinal blood vessels in the R-color fundus image with the same value as the surrounding pixels. As a result, a choroidal blood vessel image is obtained from the R-color fundus image in which only the pixels of the retinal blood vessels are extracted. Further, the image processing unit 206 may perform CLAHE (Contrast Limited Adaptive Histogram Equalization) processing on the R-color fundus image from which the retinal blood vessels have been removed, and perform enhancement processing for emphasizing the choroidal blood vessels. .

Next, the image processing unit 206 extracts a branch point or a confluence point of the choroidal blood vessel from the choroidal blood vessel image as a first characteristic point of the choroidal blood vessel. The first feature point of the choroidal blood vessel is stored in the RAM 16B by the processing unit 208. FIG.

In the example described above, the choroidal blood vessel image is generated from the R-color fundus image and the G-color fundus image, but the technique of the present disclosure is not limited to this. For example, the image processing unit 206 may generate a choroidal blood vessel image using an R color fundus image or an IR fundus image captured with IR light.
Specifically, the image processing unit 206 extracts retinal blood vessels from the R-color fundus image by applying black-hat filter processing to the R-color fundus image. As described above, the R-color fundus image includes information on blood vessels existing in the retina (retinal vessels) and information on blood vessels existing in the choroid (choroidal vessels). However, when black-hat filtering is applied to the R color fundus image, only information on retinal blood vessels is extracted. Since retinal blood vessels absorb not only G light but also R light, they appear darker than the periphery of the blood vessels in the fundus image. Therefore, retinal blood vessels can be extracted by applying black hat filter processing to the R-color fundus image. Then, the image processing unit 206 uses the position information of the retinal blood vessels extracted from the R-color fundus image to fill in the pixels corresponding to the retinal blood vessels in the R-color fundus image with the same value as the surrounding pixels. A choroidal blood vessel image is thus obtained. Furthermore, by applying adaptive histogram equalization processing (CLAHE) as described above, the choroidal blood vessels are emphasized in the R-color fundus image.

At step 352 , the imaging control unit 202 controls the SLO unit 18 and the imaging optical system 19 to obtain a rectangular SLO fundus image of the eye 12 to be examined.
Note that the rectangular SLO fundus image of the fundus of the subject's eye 12 is an example of the "second fundus image" of the technology of the present disclosure.

Specifically, first, the operator uses the UWF-SLO fundus image 400G to set a region for obtaining a rectangular SLO fundus image. FIG. 6C shows a UWF-SLO fundus image 400G on which a region 400 for acquiring a rectangular SLO fundus image is superimposed on a position 402 for acquiring a tomographic image. In the example shown in FIG. 6C, the area 400 for acquiring the rectangular SLO fundus image includes the entire area of the position 402 for acquiring the tomographic image.

The technology of the present disclosure is not limited to the case where the area 400 for acquiring the rectangular SLO fundus image includes the entire area of the position 402 for acquiring the tomographic image. For example, the area for acquiring the rectangular SLO fundus image may include only part of the position 402 for acquiring the tomographic image. In this way, the area for acquiring the rectangular SLO fundus image may include at least part of the position 402 for acquiring the tomographic image.

The area for acquiring the rectangular SLO fundus image may not include the position 402 for acquiring the tomographic image. Therefore, the area for acquiring the rectangular SLO fundus image may be set independently of the position 402 for acquiring the tomographic image.

It should be noted that when the region 400 for acquiring the rectangular SLO fundus image includes the entire region of the position 402 for acquiring the tomographic image, the region 400 does not include the position 402 at all or includes only a part of the position 402. It is possible to increase the probability that the search range for calculating is narrowed. Thereby, eye tracking processing can be performed smoothly, that is, processing time can be shortened.

The region for acquiring the rectangular SLO fundus image includes at least part of the fundus region 400gg of the subject's eye.

The size of the area for acquiring the rectangular SLO fundus image is smaller than the size of the UWF-SLO fundus image 400G. That is, the angle of view of the rectangular SLO fundus image is smaller than the angle of view of the UWF-SLO fundus image, and is set to, for example, 10 degrees to 50 degrees.

At step 352, the imaging control unit 202 acquires the position of the area 400 for acquiring the rectangular SLO fundus image set using the UWF-SLO fundus image 400G. The processing unit 208 stores and holds the position of the region for acquiring the rectangular SLO fundus image as the first position in the RAM 16B. Then, the imaging control unit 202 controls the SLO unit 18 and the imaging optical system 19 based on the acquired position of the region 400 to acquire an image of the fundus of the eye 12 to be inspected. Acquire a rectangular SLO fundus image of the fundus. When acquiring the rectangular SLO fundus image, the imaging control unit 202 causes the IR light source 46 to emit light to irradiate the fundus with IR light, thereby capturing an image of the fundus. This is because the IR light is not perceived by the visual cells of the retina of the subject's eye, so that the subject does not feel glare.

Next, in step 354, the image processing unit 206 extracts the feature points of the choroidal blood vessels (specifically, the bifurcation points or confluence points of the choroidal blood vessels) in the rectangular SLO fundus image of the second fundus image.

By the way, in a rectangular SLO fundus image captured with IR light, retinal vessels appear as thin black vessels, and choroidal vessels appear as thick white vessels. Therefore, the image processing unit 206 performs black hat processing on the rectangular SLO fundus image to specify the pixel positions of the retinal blood vessels.
Next, the image processing unit 206 removes the retinal blood vessels from the rectangular SLO fundus image by performing inpainting processing on the pixel positions of the retinal blood vessels. A second feature point is extracted. The processing unit 208 stores the second feature point of the choroidal blood vessel in the RAM 16B.

Next, at step 356, a registration process is performed.

Here, the registration process is a process of aligning a rectangular SLO fundus image and a UWF-SLO fundus image (in particular, an RGB color fundus image). Specifically, it is a process of specifying the position of the rectangular SLO fundus image on the RGB color fundus image by image matching.

In the image matching of the registration processing in step 356, the image processing unit 206 extracts, for example, three second feature points of the choroidal blood vessels extracted from the rectangular SLO fundus image. The image processing unit 206 searches for positions where the three second feature points match the first feature points of the choroidal blood vessels extracted from the UWF-SLO fundus image. The processing unit 208 stores and holds the position matched in the registration process in the RAM 16B as the second position.
In the registration processing, the UWF-SLO fundus image is not subjected to denoising processing, but it may be performed.

As described above, in the image matching of the registration process, three second feature points of the choroidal blood vessels are extracted, but the technique of the present disclosure is not limited to this. For example, the number of feature points may be 2, 4, 5, or the like.

At step 358, the image processing unit 206 calculates the amount of eye movement from the first position and the second position. Specifically, the image processing unit 206 calculates the magnitude and direction of the deviation between the first position and the second position. Then, the image processing unit 206 calculates the amount of movement of the subject's eye 12 from the calculated magnitude and direction of the displacement. The amount of motion is an amount having a magnitude and a direction of motion, that is, a vector amount. The time elapses from when the fundus of the subject's eye 12 is photographed to obtain a UWF-SLO fundus image in step 302 to when the fundus of the subject's eye 12 is photographed to obtain a rectangular SLO fundus image in step 352. Therefore, the same location of the subject's eye is not necessarily photographed. In addition, the subject's eye may move due to involuntary eye movement or the like during OCT imaging. Therefore, the image processing unit 206 calculates in what direction and how much the subject's eye 12 is displaced at the start of OCT imaging or during OCT imaging, that is, the amount of movement (vector amount) of the subject's eye 12. do.
The deviation amount and deviation direction calculated in step 358 after step 356 is executed are an example of "the deviation amount and deviation direction."

Since the subject's eye 12 is displaced in this way, if the subject's eye 12 is scanned based on the position for acquiring the tomographic image set based on the UWF-SLO fundus image, the movement amount (displacement amount) from the position originally desired to be acquired. and in the direction of displacement).

Therefore, in step 360, the imaging control unit 202, based on the movement amount (vector amount) of the subject's eye 12, controls the second optical system so that the tomographic image of the subject's eye 12 after movement can be acquired at the position to be acquired. The scanning range of the scanner 24 is adjusted.
The second optical scanner 24 is an example of the "scanning device" of the technology of the present disclosure.

When the adjustment of the scanning range of the subject's eye 12 is completed in this way, the eye tracking processing of step 306 in FIG.

In step 308, the imaging control unit 202 controls the OCT unit 20 and the second optical scanner 24 whose scanning range is adjusted, and scans the position of the subject's eye 12 for obtaining a tomographic image, thereby obtaining a tomographic image. get.

At step 310, the processing unit 208 outputs the acquired tomographic image, the RG-UWF-SLO fundus image, and the data of the position 402 where the tomographic image is acquired to the server 140 in correspondence with the subject's information. Thus, the image processing in FIG. 4 ends.

The server 140 associates the tomographic image, the RGB color fundus image, and the data of the position 402 where the tomographic image is acquired with the subject ID and stores them in a storage device (not shown).

The viewer 150 requests the server 140 to transmit data such as a tomographic image corresponding to the subject ID according to an instruction from a user such as an ophthalmologist.

When the above request is sent to the server 140, the server 140 transmits the tomographic image, the UWF-SLO fundus image, and the data of the position 402 for acquiring the tomographic image stored in association with the subject ID. It transmits to the viewer 150 corresponding to the information and the like. Thereby, the viewer 150 displays each received data on a display (not shown).

FIG. 7 shows screen 500 displayed on the display of viewer 150 . The screen 500 has a subject information display area 502 and an image display area 504 .

The subject information display area 502 includes a subject ID display field 512, subject name display field 514, age display field 516, visual acuity display field 518, right/left eye display field 520, and axial length display field. 522. The display control unit 204 receives from the server 140 the subject ID, the subject's name, the subject's age, the visual acuity, the right or left eye information, and the subject's eye axial length data, Displayed in corresponding display fields 512-522.

The image display area 504 includes an RGB color fundus image display field 508 , a tomographic image display field 506 and a text data display field 510 .

An RGB color fundus image display field 508 displays a UWF-SLO fundus image 400G superimposed with a position 402 for acquiring a tomographic image.

A tomographic image is displayed in the tomographic image display field 506 . The text data display field 510 displays comments and the like at the time of medical examination.

As described above, in the present embodiment, the motion amount of the subject's eye is calculated using the feature points of the choroidal blood vessels extracted from each of the fundus images acquired at least twice at different timings. By the way, in the fundus area of the same size, the number of characteristic points of the choroidal vessels is larger than that of the retinal vessels. Therefore, the present embodiment can more accurately calculate the amount of eye movement.

As described above, in the present embodiment, in step 302, the imaging control section 202 controls the SLO unit 18 and the imaging optical system 19 to acquire a UWF-SLO fundus image.
The technology of the present disclosure is not limited to this. For example, the UWF-SLO fundus image acquired in this manner is stored in a storage medium (not shown) of the server 140 . At step 302 , the processing unit 208 may acquire the UWF-SLO fundus image from the storage medium of the server 140 . Also, the UWF-SLO fundus image acquired as described above is stored in the RAM 16B of the ophthalmologic apparatus 110. FIG. At step 302 , the processing unit 208 may acquire the UWF-SLO fundus image from the storage medium of the RAM 16B of the ophthalmologic apparatus 110 .

In the above embodiment, the tomographic image is acquired after executing the eye tracking process. The technology of the present disclosure is not limited to this. For example, eye tracking processing may be performed while acquiring a tomographic image. In this case, steps 352 to 360 in FIG. 5 may be repeatedly executed while the tomographic image is being acquired. Then, every time steps 352 to 360 are repeatedly executed a predetermined number of times, the average value of the movement amounts of the eye 12 to be examined is obtained, and the scanning range of the second optical scanner 24 is adjusted while the tomographic image is being acquired. and a tomographic image may be obtained. Thereby, a plurality of tomographic images of the same position of the subject's eye 12 are acquired. A noise-reduced tomographic image may be obtained by averaging these multiple tomographic images.
In the above embodiment, the tomographic image is acquired after executing the eye tracking process. The technology of the present disclosure is not limited to this. For example, a plurality of tomographic images are obtained without tracking the optical scanner. Repeating steps 352 to 360 in FIG. 5 while acquiring a plurality of tomographic images, and deleting tomographic images taken when eye movement exceeds a predetermined value, An addition average may be performed using a plurality of remaining tomographic images.

In the present disclosure, as long as there is no contradiction, each component (device, etc.) may be present singly or in two or more.

In each example described above, a case where image processing is realized by a software configuration using a computer was illustrated, but the technique of the present disclosure is not limited to this. For example, image processing may be performed only by a hardware configuration such as FPGA (Field-Programmable Gate Array) or ASIC (Application Specific Integrated Circuit) instead of software configuration using a computer. A part of the image processing may be performed by a software configuration, and the rest of the image processing may be performed by a hardware configuration.

As described above, the technology of the present disclosure includes the following technology, since it includes the case where image processing is realized by software configuration using a computer and the case where it is not.

(First technology)
an acquisition unit that acquires a fundus image of an eye to be inspected at least twice at different timings;
an extraction unit that extracts feature points of choroidal blood vessels from each of the fundus images acquired at least twice;
a calculation unit that calculates the amount of movement of the subject's eye using the feature points of the choroidal blood vessels extracted from each of the fundus images acquired at least twice;
An image processing device comprising:

(Second technology)
a step in which the acquisition unit acquires the fundus image of the eye to be examined at least twice at different timings;
an extraction unit extracting feature points of choroidal blood vessels from each of the fundus images acquired at least twice;
a calculating unit calculating the movement amount of the subject eye using the feature points of the choroidal blood vessels extracted from each of the fundus images acquired at least twice;
An image processing method comprising:

The imaging control unit 202 is an example of the “acquisition unit” of the technology of the present disclosure. The image processing unit 206 is an example of the “extraction unit” and the “calculation unit” of the technology of the present disclosure.

The following technique is proposed from the above disclosure.
(Third technology)
1. A computer program product for image processing, said computer program product comprising a computer readable storage medium which is not itself a temporary signal, said computer readable storage medium having a program stored thereon, said The program is the image processing performed by the processor in the computer,
acquiring fundus images of the subject's eye at least twice at different timings;
extracting feature points of choroidal vessels from each of the at least two acquired fundus images;
calculating an amount of movement of the subject's eye using the feature points of the choroidal blood vessels extracted from each of the fundus images acquired at least twice;
A computer program product that performs image processing, including

Each image processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps added, and the order of processing may be changed without departing from the scope of the present invention.

All publications, patent applications and technical standards mentioned herein, as if each individual publication, patent application and technical standard were specifically and individually indicated to be incorporated by reference. incorporated herein by reference.

100 ophthalmic system 16 control device 16A CPU
16C ROM
202 shooting control unit 204 display control unit 206 image processing unit 208 processing unit

Claims (14)

An image processing method performed by a processor,
acquiring fundus images of the subject's eye at least twice at different timings;
extracting feature points of choroidal vessels from each of the at least two acquired fundus images;
calculating an amount of movement of the subject's eye using the feature points of the choroidal blood vessels extracted from each of the fundus images acquired at least twice;
An image processing method including The step of acquiring the fundus image includes:
obtaining a first fundus image of the subject eye;
acquiring a second fundus image of the subject eye after the first fundus image is acquired;
including
The step of extracting feature points of the choroidal blood vessels includes:
extracting a first feature point of a first choroidal vessel from the first fundus image;
extracting a second feature point of a second choroidal vessel from the second fundus image;
including
The step of calculating the amount of motion includes:
performing a registration process for aligning the first fundus image and the second fundus image using the first feature point and the second feature point, and calculating the amount of movement of the subject's eye;
including,
The image processing method according to claim 1. The first fundus image is obtained by photographing the subject's eye using a combination of at least two types of light of red (R) light, green (G) light, and blue (B) light,
The second fundus image is obtained by photographing the subject eye using infrared (IR),
3. The image processing method according to claim 2. 4. The image processing method according to claim 2, wherein the second fundus image is obtained by photographing at least a partial area of the first fundus image. 5. The image processing method according to claim 4, wherein the size of said region is smaller than the size of said first fundus image. 6. The image processing method according to claim 4, wherein said region includes at least part of a position where a tomographic image of the fundus of said eye to be examined is acquired. 7. The image processing method according to any one of claims 1 to 6, further comprising controlling a scanning device for acquiring a tomographic image of the fundus of the subject's eye based on the amount of motion. 8. The image processing method of claim 7, further comprising acquiring a tomographic image of the fundus of the subject eye using the controlled scanning device. 9. The image processing method according to any one of claims 1 to 8, wherein said motion amount is a vector amount consisting of magnitude and direction. An image processing device comprising a processor,
The processor
acquiring fundus images of the subject's eye at least twice at different timings;
extracting feature points of choroidal vessels from each of the at least two acquired fundus images;
calculating an amount of movement of the subject's eye using the feature points of the choroidal blood vessels extracted from each of the fundus images acquired at least twice;
An image processing device that performs image processing including to the computer,
acquiring fundus images of the subject's eye at least twice at different timings;
extracting feature points of choroidal vessels from each of the at least two acquired fundus images;
calculating an amount of movement of the subject's eye using the feature points of the choroidal blood vessels extracted from each of the fundus images acquired at least twice;
A program that executes image processing including An image processing method performed by a processor,
obtaining a first fundus image of an eye to be examined from a recording medium;
obtaining a second fundus image by imaging the subject eye with infrared (IR);
extracting a first feature point of a choroidal blood vessel from the first fundus image;
extracting a second feature point of the choroidal blood vessel from the second fundus image;
calculating an amount of movement of the subject's eye using the first feature point and the second feature point;
An image processing method including An image processing device comprising a processor,
The processor
obtaining a first fundus image of an eye to be examined from a recording medium;
obtaining a second fundus image by imaging the subject eye with infrared (IR);
extracting a first feature point of a choroidal blood vessel from the first fundus image;
extracting a second feature point of the choroidal blood vessel from the second fundus image;
calculating an amount of movement of the subject's eye using the first feature point and the second feature point;
An image processing device that performs image processing including to the computer,
obtaining a first fundus image of an eye to be examined from a recording medium;
obtaining a second fundus image by imaging the subject eye with infrared (IR);
extracting a first feature point of a choroidal blood vessel from the first fundus image;
extracting a second feature point of the choroidal blood vessel from the second fundus image;
calculating an amount of movement of the subject's eye using the first feature point and the second feature point;
A program that executes image processing including

Images