JP2013034658A - Fundus imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a fundus imaging apparatus capable of acquiring a clear fundus image.SOLUTION: The apparatus includes an imaging optical system for receiving a luminous flux reflected against the fundus of an eye to be examined, and has a fundus imaging means and an image processing section. The fundus imaging means acquires the fundus image data of a plurality of frames having a deviation less than one pixel, by using the fixation nystagmus of the eye of the subject to be examined which occurs while the fundus image data of each frame are acquired. The image processing section acquires a high resolution image which has higher resolution than that of the image data at the time of acquisition, by performing composite processing of a plurality of fundus image data acquired through the fundus imaging optical system.

Description

  The present invention relates to a fundus imaging apparatus that images a fundus image of a subject's eye.

  A fundus tomography device (for example, optical coherence tomography (OCT)) is used as a fundus imaging device that scans measurement light on the fundus using an optical scanning unit (for example, a galvanomirror) to obtain a fundus image. And a fundus front image capturing apparatus (for example, a scanning laser opthalmoscope (SLO)) are known (see Patent Document 1).

  These apparatuses observe the lesioned part by observing the acquired fundus tomographic image and fundus front image on a monitor. For example, in a fundus tomography apparatus, a front image is obtained by scanning a measurement light beam two-dimensionally and integrating the spectral intensity of an interference signal from a light receiving element for each XY point (see Patent Document 2).

JP 2008-29467 A US Patent Registration No. 7301644

  For example, when a front image is obtained based on a spectral interference signal, it takes time to form a front image. Therefore, the front image can be obtained by reducing the number of XY points (number of points) used to form the front image. Time is being reduced. However, since the resolution is reduced by reducing the number of points, the image quality of the formed front image is lowered, and it is difficult to obtain a clear fundus image. For this reason, it is difficult to observe the lesioned part by the acquired fundus image.

  In addition, when obtaining a tomographic image by OCT, in order to increase the resolution, it is only necessary to slow down the scanning speed of the measurement light and increase the number of samplings. As a result, the tomographic image may become a distorted image. When a front image is obtained by SLO, the resolution can be increased by slowing the scanning speed of the measuring light and increasing the number of samplings. However, the position shift occurs between the scanning lines due to the influence of fixation fixation. The tomographic image may become a distorted image. For this reason, it is preferable that the scanning speed on the fundus is high in consideration of the fixation movement.

  In view of the above-described problems of the prior art, it is an object of the present invention to provide a fundus imaging apparatus capable of acquiring a clear fundus image.

  In order to solve the above problems, the present invention is characterized by having the following configuration.

(1) Obtained by a fundus photographing means having a photographing optical system that receives a light beam reflected from the fundus of the subject's eye and obtaining a plurality of frames of fundus image data having a shift of less than one pixel, and a fundus photographing optical system. And an image processing unit that acquires a high-resolution image having a higher resolution than that at the time of acquiring the image data by subjecting the plurality of fundus image data to complex processing.
(2) The fundus imaging means obtains fundus image data for a plurality of frames having a shift of less than one pixel by using fixation eye movement of the subject's eye that occurs while obtaining fundus image data of each frame. The fundus imaging apparatus according to (1).
(3) The imaging optical system has an optical scanner that scans the fundus of the measurement light, and optical tomographic interference that receives the combined light of the light beam reflected from the fundus of the eye to be examined and the reference light to obtain a tomographic image of the fundus Any one of scanning optical fundus photographing optical systems that has an optical system and an optical scanner that scans the fundus with laser light and receives a light beam reflected by the fundus of the eye to be examined to obtain a front image of the fundus (1) to ( The fundus imaging apparatus according to any one of 2).
(4) The fundus photographing unit includes a driving unit that changes the scanning position on the fundus every time a fundus image of each frame is obtained in order to acquire fundus image data for a plurality of frames having a shift of less than one pixel (3 ) Any fundus imaging device.

  According to the present invention, a clear fundus image can be acquired.

  Embodiments according to the present invention will be described below with reference to the drawings. FIG. 1 is a schematic configuration diagram illustrating the configuration of the ophthalmologic photographing apparatus according to the present embodiment. In the present embodiment, the axial direction of the subject's eye (eye E) will be described as the Z direction, the horizontal direction as the X direction, and the vertical direction as the Y direction. The surface direction of the fundus may be considered as the XY direction.

  An outline of the apparatus configuration will be described. This apparatus is an optical coherence tomography device (OCT device) 10 for taking a tomographic image of the fundus oculi Ef of the subject's eye E. The OCT device 10 includes an optical tomographic interference optical system (OCT optical system) 100, a fixation target projection unit 300, and an arithmetic control unit (CPU) 70.

  The OCT optical system 100 irradiates the fundus with measurement light. The OCT optical system 100 detects the interference state between the measurement light reflected from the fundus and the reference light by the light receiving element (detector) 120. The OCT optical system 100 includes an irradiation position changing unit (for example, the optical scanner 108 and the fixation target projection unit 300) that changes the irradiation position of the measurement light on the fundus oculi Ef in order to change the imaging position on the fundus oculi Ef. The control unit 70 controls the operation of the irradiation position changing unit based on the set imaging position information, and processes the output signal (light reception signal) from the detector 120 to acquire tomographic image data (tomographic image).

<OCT optical system>
The OCT optical system 100 has an apparatus configuration of a so-called ophthalmic optical tomography (OCT: Optical coherence tomography) and takes a tomographic image of the eye E. The OCT optical system 100 splits the light emitted from the measurement light source 102 into measurement light (sample light) and reference light by a coupler (light splitter) 104. The OCT optical system 100 guides the measurement light to the fundus oculi Ef of the eye E by the measurement optical system 106 and guides the reference light to the reference optical system 110. Thereafter, the detector 120 receives interference light obtained by combining the measurement light reflected by the fundus oculi Ef and the reference light.

  The detector 120 detects an interference state between the measurement light and the reference light. In the case of Fourier domain OCT, the spectral intensity of the interference light is detected by the detector 120, and a depth profile (A scan signal) in a predetermined range is obtained by Fourier transform on the spectral intensity data. Examples include Spectral-domain OCT (SD-OCT) and Swept-source OCT (SS-OCT). Moreover, Time-domain OCT (TD-OCT) may be used.

  The optical scanner 108 scans the measurement light in the XY (transverse) direction on the fundus. The optical scanner 108 is arranged at a position substantially conjugate with the pupil. The optical scanner 108 is, for example, two galvanometer mirrors, and the reflection angle thereof is arbitrarily adjusted by the drive mechanism 50.

  As a result, the light beam emitted from the light source 102 changes its reflection (advance) direction and is scanned in an arbitrary direction on the fundus. Thereby, the imaging position on the fundus oculi Ef is changed. The optical scanner 108 may be configured to deflect light. For example, as the optical scanner 108, an acousto-optic device (AOM) that changes the traveling (deflection) direction of light is used in addition to a reflecting mirror (galvano mirror, polygon mirror, resonant scanner).

  The reference optical system 110 generates reference light that is combined with reflected light acquired by reflection of measurement light at the fundus oculi Ef. The reference optical system 110 may be a Michelson type or a Mach-Zehnder type. The reference optical system 110 is formed by, for example, a reflection optical system (for example, a reference mirror), and reflects light from the coupler 104 back to the coupler 104 by the reflection optical system, and guides it to the detector 120. As another example, the reference optical system 110 is formed by a transmission optical system (for example, an optical fiber), and guides the light from the coupler 104 to the detector 120 by transmitting the light without returning.

  The reference optical system 110 has a configuration in which the optical path length difference between the measurement light and the reference light is changed by moving an optical member in the reference optical path. For example, the reference mirror is moved in the optical axis direction. The configuration for changing the optical path length difference may be arranged in the measurement optical path of the measurement optical system 106.

<Fixation target projection unit>
The fixation target projecting unit 300 includes an optical system for guiding the line-of-sight direction of the eye E. The projection unit 300 has a fixation target presented to the eye E, and can guide the eye E in a plurality of directions.

  For example, the fixation target projection unit 300 has a visible light source that emits visible light, and changes the presentation position of the target two-dimensionally. Thereby, the line-of-sight direction is changed, and as a result, the imaging region is changed. For example, when the fixation target is presented from the same direction as the imaging optical axis, the center of the fundus is set as the imaging site. When the fixation target is presented upward with respect to the imaging optical axis, the upper part of the fundus is set as the imaging region. That is, the imaging region is changed according to the position of the target with respect to the imaging optical axis.

  As the fixation target projection unit 300, for example, a configuration in which the fixation position is adjusted by the lighting positions of LEDs arranged in a matrix, light from a light source is scanned using an optical scanner, and fixation is performed by lighting control of the light source. Various configurations such as a configuration for adjusting the position are conceivable. The projection unit 300 may be an internal fixation lamp type or an external fixation lamp type.

<Control unit>
The control unit 70 controls the entire apparatus such as each member of each configuration 100 to 300. The control unit 70 also serves as an image processing unit that processes the acquired image, an image analysis unit that analyzes the acquired image, and the like. The control unit 70 is realized by a general CPU (Central Processing Unit) or the like. As shown below, the control unit 70 analyzes the fundus oculi Ef based on the tomographic image.

  The control unit 70 acquires a tomographic image and a front image by image processing based on the light reception signal output from the detector 120 of the OCT optical system 100. Further, the control unit 70 controls the fixation target projection unit 300 to change the fixation position.

  The frame memory (for example, RAM) 77, the memory (for example, hard disk) 72 as an auxiliary storage device, the display monitor 75, and the control unit 74 are electrically connected to the control unit 70, respectively. The control unit 70 controls the display screen of the monitor 75. The acquired fundus image is output to the monitor 75 as a still image or a moving image and stored in the memory 72. The memory 72 records, for example, various types of information related to imaging such as a captured tomographic image, a front image, and imaging position information of each tomographic image. The control unit 70 controls each member of the OCT optical system 100 and the fixation target projection unit 300 based on the operation signal output from the control unit 74.

  The control operation of the apparatus having the above configuration will be described. The examiner instructs the subject to gaze at the fixation target of the fixation target projection unit 300, and then observes the anterior ocular segment observation image captured by the anterior ocular segment observation camera (not shown) on the monitor 75. The alignment operation is performed using a joystick (not shown) so that the measurement optical axis is at the center of the pupil of the eye to be examined.

  Then, the control unit 70 controls driving of the optical scanner 108, scans the measurement light on the fundus in a predetermined direction, and receives a light reception signal corresponding to a predetermined scanning area from an output signal output from the detector 120 during the scanning. To obtain a fundus image.

  Hereinafter, an example of a method for acquiring a fundus tomographic image (hereinafter referred to as a tomographic image) and a fundus frontal image (hereinafter referred to as a front image) according to the present embodiment will be described. The control unit 70 processes the spectral data detected by the detector 120 and forms a tomographic image and a front image by image processing. The tomographic image and the front image may be acquired at the same time, may be acquired alternately, or may be acquired sequentially. That is, the spectrum data is used for obtaining at least one of a tomographic image and a front image. Note that the acquired tomographic image and front image are displayed on the monitor 75.

  The control unit 70 controls the OCT optical system 100 and acquires a three-dimensional tomographic image corresponding to the set region. And the control part 70 acquires a three-dimensional tomogram at any time with the OCT optical system 100. FIG. The three-dimensional tomographic image includes image data in which A scan signals are arranged two-dimensionally in the XY directions, a three-dimensional graphic image, and the like.

  When obtaining a three-dimensional tomographic image, the control unit 70 controls the operation of the optical scanner 108 and acquires the three-dimensional tomographic image by scanning the measurement light in the XY directions two-dimensionally in the scanning range corresponding to the imaging region. To do. As the scanning pattern, for example, a raster scan and a plurality of line scans can be considered.

  The examiner observes the lesioned part and the like while observing the tomographic image and the front image displayed on the monitor 75. Means for obtaining a clear observation image will be described in order to suitably observe the lesion.

  The control unit 70 may include noise or the like in the observation image, or the observation image may not be clear due to low resolution. For this reason, in the present embodiment, processing using super-resolution technology (super-resolution processing) is performed to improve the resolution of the observation image, thereby improving the quality of the observation image.

  The super-resolution technique is a technique for generating an output signal by increasing the resolution of an input signal. The resolution of an input moving image or still image is increased to output a high-resolution image. In this embodiment, super-resolution processing is performed using a super-resolution technique that generates a single high-resolution image using a plurality of images.

  Hereinafter, processing of the super-resolution technique will be described with reference to the super-resolution conceptual diagram of FIG. In the super-resolution processing, alignment processing between a plurality of frames is performed, and image reconstruction processing is performed based on the alignment information and each pixel element information of the plurality of frames. The alignment process is performed precisely with sub-pixel unit precision that is finer than the pixel unit.

  An image A and an image B shown in FIG. 2 are one-dimensional images obtained by shifting the imaging position in units of subpixels with respect to the same object. The interval between the pixel positions S1 to S4 of the image A is 1 pixel (1 pixel), and the interval between the pixel positions S5 to S8 of the image B is 1 pixel (1 pixel). The image B is an image acquired at a position shifted from the image A by a half pixel (sub pixel). In this case, when the two images are aligned and the images are integrated, the amount of displacement is exactly half a pixel, so that the image B is positioned at the subpixel position between the pixel positions S1 to S4 of the image A. The luminance information of each of the pixel positions S5 to S8 is arranged. As described above, since the number of pixels can be increased by integrating the two images, an image C having a double resolution can be acquired.

  Hereinafter, a plurality of observation images acquired by the OCT device 10 and a super-resolution process using them will be described.

  For example, the control unit 70 acquires a high-resolution observation image by performing super-resolution processing on a plurality of observation images acquired in time series by the OCT device 10 and an observation image newly acquired by the OCT device 10. .

  The control unit 70 updates the high-resolution observation image displayed on the monitor 75 in response to the newly acquired high-resolution observation image, and displays the high-resolution observation image on the monitor 75 as a moving image. To do. Thereby, the image quality of the observed captured image is improved. Note that the observation image on which the super-resolution processing is performed may not be a moving image, but may be a still image acquired when a predetermined trigger signal is output. In the present embodiment, a front image is taken as an example of a high-resolution observation image acquired by the OCT device 10 and will be described below.

  FIG. 3 is a diagram illustrating a front image (low-resolution frame image) acquired by the OCT optical system 100 and a front image after super-resolution processing. In FIG. 3, front images P0 to P3 are front images acquired at different pixel positions. The front image P0 shows a reference image serving as a reference for super-resolution processing, and the front images P1 to P3 show images used for super-resolution processing. A region F surrounded by a dotted line in the front images P1 to P3 indicates a region of the front image P0 that is a reference image.

  The front image P1 is displaced in the X direction with respect to the front image P0, and the amount of positional deviation is (ΔX, 0). The front image P2 is displaced in the Y direction with respect to the front image P0, and the amount of positional deviation is (0, ΔY). The front image P3 is displaced in the X direction and the Y direction with respect to the front image P0, and the positional shift amount is (ΔX, ΔY). The correction amount of the positional deviation of the front images P1 to P3 with respect to the front image P0 is calculated based on the respective positional deviation amounts.

  In FIG. 3, the deviation of the front images P1 to P3 with respect to the front image P0 is schematically shown as one pixel unit. However, in the present embodiment, a plurality of front images acquired by the OCT optical system 100 have subpixel shifts between the front images. For this reason, it is possible to increase the resolution by using the subpixel shift (see FIG. 2).

  At this time, even if the luminance information of the pixels of the front images P0 to P3 is integrated, a pixel having no luminance information may be generated. In such pixels, integration is performed and resolution is increased by performing predetermined interpolation processing using luminance information of pixels existing around the pixels.

  For example, as the interpolation processing, interpolation processing by a bilinear method is performed. For example, the bilinear method determines four pixels surrounding a pixel on which interpolation processing is performed in a super-resolution image. By applying a predetermined weight to the luminance information of these four pixels and averaging the values, the luminance information of the pixels to be subjected to the interpolation processing is acquired and interpolation is performed.

  By repeating the interpolation processing as described above, a high-resolution front image can be acquired. In addition, although the bilinear method was used as an interpolation process, it is not limited to this. For example, an interpolation process such as a two-arrest neighbor method or a bicubic method may be used. The luminance information of the pixels of the reference image may be used as it is as the pixels of the high resolution image.

  Hereinafter, more specifically, the flow of the control operation when acquiring a plurality of observation images acquired by the OCT device 10 and generating a high-resolution observation image by super-resolution processing will be described using the flowchart shown in FIG. To do.

  In general, the control unit 70 acquires fundus image data for a plurality of frames having a shift of less than one pixel. Then, a high-resolution image having a higher resolution than that at the time of acquiring the image data is acquired by subjecting the plurality of acquired fundus image data to composite processing.

<Acquisition of front image used for super-resolution processing>
First, the control unit 70 processes the spectral data detected by the detector 120 using the OCT optical system 100, and forms a front image of one frame (one sheet) by image processing. Then, the front image is stored in the frame memory 77. And the control part 70 acquires a front image sequentially, in order to acquire the several front image (in this embodiment, for example, ten front images) for creating a high-resolution front image. The data is stored in the frame memory 77. It is determined whether or not the acquired front image is appropriate, and the image determined to be appropriate is stored in the frame memory 77 (details will be described later). Thereby, it is avoided that an error image due to blinking or the like is used for super-resolution processing.

  The amount of data that can be stored in the frame memory 77 is limited. In the present embodiment, a maximum of 10 captured images can be stored. Of course, depending on the capacity of the frame memory 77, the number of images that can be stored can be changed, and as the capacity of the frame memory 77 is reduced, the captured image need not be stored. The maximum number of images to be stored may be set arbitrarily.

<Determination in memory of front image>
When the acquisition of the front image is started by the OCT optical system 100, the control unit 70 starts acquiring the front image for use in the super-resolution processing. When the front image is acquired, the control unit 70 performs a determination process on the newly acquired front image and determines whether to store the front image in the frame memory 77. Since the frame memory 77 has a limited storable capacity, the number of images that can be used for super-resolution processing is increased by avoiding storage of images that are not suitable for super-resolution processing. For the determination, a predetermined evaluation value V is used.

  Here, the evaluation value V will be described. The evaluation value V is obtained from the equation V = ((average maximum luminance value of the image) − (average luminance value of the background region of the image)) / (standard deviation of the luminance value of the background region). That is, an image that is not good as an image has a large noise, and thus the evaluation value V is small. When obtaining the evaluation value V, for example, the signal intensity of the front image is used.

  In this case, the control unit 70 may acquire a tomographic image based on the spectrum information that forms the front image, and may determine whether the front image is appropriate using the acquired tomographic image. For example, it is calculated from a tomographic image corresponding to one scanning line in the front image. As shown in FIG. 5, the control unit 70 calculates the evaluation value V using a tomographic image formed by scanning lines at predetermined positions of the front image. As the predetermined position, for example, the scanning line B passing through the center position of the front image is used. Of course, although the center position is used in the present embodiment, the present invention is not limited to this. As shown in FIG. 6, the control unit 70 sets a plurality of scanning lines C to be scanned in the depth direction (A scanning direction) in the tomographic image, and obtains luminance distribution data on each scanning line C.

  The control unit 70 calculates the maximum luminance value (hereinafter abbreviated as the maximum luminance value) in the luminance distribution corresponding to each scanning line C. And the control part 70 calculates the average value of the maximum luminance value in each scanning line as an average maximum luminance value. Further, the control unit 70 calculates the average value of the luminance values of the background area in each scanning line as the average luminance value of the background area. Then, an evaluation value V is calculated.

  After calculating the evaluation value V, the control unit 70 determines whether it is necessary to store the captured image in the frame memory 77 based on whether the evaluation value V exceeds a predetermined threshold. For example, when the evaluation value V exceeds a threshold value, the front image is stored in the frame memory 77. The determination is performed as described above. As a result, it is possible to perform super-resolution processing on many photographed images, and the image quality of the images is improved.

  In the present embodiment, the evaluation value V is used to determine whether the captured image needs to be stored, but the present invention is not limited to this. For example, it is good also as a structure which determines from another observation image. For example, an anterior ocular segment observation system (not shown) may be provided to perform determination (blink determination or the like) from the anterior ocular segment image.

  Next, when the acquired front image of one frame is stored in the frame memory 77, the control unit 70 determines whether or not the number of front images stored in the frame memory 77 has reached a predetermined number. In the present embodiment, the predetermined number is set to 10 which is the maximum number that can be stored in the frame memory 77. Note that the predetermined number is not limited to this, and for example, the predetermined number may be set arbitrarily by the examiner. Alternatively, the number of images may be set based on the light amount of the acquired captured image. In this case, for example, it is conceivable to increase the predetermined number if the light amount of the captured image is low, and decrease the predetermined number if the light amount of the captured image is high.

  Then, when the number of acquired front images stored in the frame memory 77 reaches a predetermined number, the control unit 70 stores the front images acquired after the timing of reaching the number in the frame memory 77. Super-resolution processing of a plurality of front images is performed. If the number of acquired front images has not reached the predetermined number, the control unit 70 starts acquiring the next front image. As described above, a plurality of front images for creating a super-resolution process are acquired.

  Here, in order to perform the super-resolution processing, a plurality of front images having a shift of less than one pixel (sub-pixel) in the X direction and / or the Y direction are necessary. That is, a plurality of front images in which a sub-pixel shift is generated in the X direction and / or the Y direction are necessary for the front image serving as a reference for super-resolution processing.

  As a first technique, when a plurality of front images are acquired in a predetermined region, the OCT optical system 100 is moved to shift the image acquisition position and acquire a plurality of images having different pixel positions. For example, the control unit 70 uses the driving unit to acquire fundus image data for a plurality of frames having a shift of less than one pixel. Therefore, every time a fundus image of each frame is obtained, the imaging position (scanning position) on the fundus ). More specifically, when acquiring the image of each frame, the control unit 70 controls the driving of the optical scanner 108 and changes the scanning region on the fundus in units of subpixels every time the image is acquired. In addition to using the optical scanner 108, a drive mechanism that finely moves the entire OCT optical system 100 relative to the eye E every time an image is acquired may be provided.

  As a second method, for a predetermined region on the fundus, the control unit 70 uses a fixation micromotion of the subject's eye that occurs while acquiring fundus image data of each frame, and shifts less than one pixel. Fundus image data for a plurality of frames having. Since this apparatus is photographing the eye to be inspected, even when a plurality of front images are acquired in the same predetermined region, the pixel positions between the plurality of front images are obtained by fixation fine movement. Is slightly displaced and a front image is acquired.

For example, fixation fine movement is roughly divided into tremor, drift, and saccade components. Tremore has a fine movement of about 10 ″ to 20 ″ with a frequency of about 50 to 150 Hz. The drift moves at an average of about 5 'per second. Saccades occur on the order of 2 'to 50' after a draft lasts from 0.3 seconds to several seconds. The acquisition time of one frame of the front image acquired by the OCT optical system 100 is 1.0 s to 1.9 s, and any of these components performs fine movement in units of subpixels between the images of one frame. Born.
For this reason, after acquiring the front image of 1 frame, when acquiring the front image of the next 1 frame, at least fixation fine movement arises and the acquisition position of an image shifts. In the second method, unlike the first method, it is not necessary to drive the apparatus main body side in order to obtain a low-resolution image used for creating a super-resolution image. A plurality of images having different acquisition positions in units can be acquired.

<Super-resolution processing of multiple front images>
Hereinafter, super-resolution processing will be described using a plurality of front images stored in the frame memory 77.

  The control unit 70 sets a reference image as a reference for super-resolution processing in the front image (newly acquired front image) formed last by the OCT optical system 100, and performs super-resolution processing. The control unit 70 detects a shift between the reference image and the plurality of front images by image processing, determines whether or not the super-resolution processing is appropriate based on the shift detection result, and shifts between the reference image and the plurality of front images. Are corrected, and a plurality of front images are subjected to super-resolution processing with respect to the reference image. In the present embodiment, the reference image is set to the last-acquired front image (latest photographed image), but the present invention is not limited to this. For example, a reference image may be set as a reference for super-resolution processing among a plurality of front images.

  The control unit 70 sequentially performs super-resolution processing on each front image with respect to the reference image. Then, a deviation amount between each front image and the reference image is detected for each front image, and each front image is aligned with the reference image. That is, the reference image and each front image are compared, and the position shift direction and the position shift amount of each front image with respect to the reference image are detected for each front image by image processing.

  Various image processing methods (a method using template matching, a method using Fourier transform, and a method based on feature point matching) can be used as a method for detecting a deviation amount.

  For example, when a predetermined reference image (for example, the last acquired front image) or a target image (past front image) is displaced pixel by pixel, the reference image and the target image are compared, and the two data are most consistent ( It is conceivable to detect a position shift direction and a position shift amount between both data (when the correlation is highest). Further, a method is conceivable in which common feature points are extracted from a predetermined reference image and target image, and the positional deviation direction and the positional deviation amount of the extracted characteristic points are detected in units of pixels.

  Then, after detecting the displacement amount in units of pixels by various image processing methods, the position displacement direction and the amount of displacement in units of subpixels are calculated by subpixel estimation.

  In the present embodiment, the control unit 70 sequentially calculates a correlation value (a correlation between images increases as the value increases (maximum 1)) while shifting each front image with respect to the reference image by pixels. Further, the control unit 70 performs subpixel estimation based on the calculated correlation value in units of pixels, and obtains a correlation value in units of subpixels. Then, the control unit 70 calculates the displacement amount (number of displaced pixels) of the correlation value pixels calculated by the subpixel estimation as the displacement amount, and calculates the displaced direction as the displacement direction.

  As a determination method, determination is performed using a correlation value calculated at the time of detecting a deviation. For example, when the correlation value is smaller than a predetermined threshold (for example, 0.4), the correlation value is excluded from the target of the front image used for the super-resolution processing. That is, when the correlation value is small, there is a high possibility that the shooting area is greatly different between the reference image and the front image due to a shift between the apparatus and the eyes. If super-resolution processing is performed using images with greatly different shooting areas, the image quality and the like deteriorate when a high-resolution front image is created. For this reason, when the correlation value is smaller than a predetermined threshold, the image quality is prevented from being deteriorated when a high-resolution front image is created by excluding it from the target of the front image used for super-resolution processing. Note that the determination as to whether or not the image used for the super-resolution processing is an appropriate image is not limited to this. For example, a front image in which the detected positional deviation amount exceeds the allowable range may be excluded from the super resolution processing target.

  As described above, the displacement amount and the displacement direction are detected, and the suitability as an image used for the super-resolution processing is determined. Then, with respect to the image determined to be appropriate as the image for super-resolution processing, the control unit 70 uses each front image as a reference for the amount of positional deviation so that the positional deviation is corrected. Each image is displaced. Then, after correcting the positional deviation, the control unit 70 integrates the front image with the reference image.

  After the super-resolution processing of the front image, the control unit 70 determines whether or not processing (shift detection, appropriateness determination, super-resolution processing) of the front image of all the frames stored in the frame memory 77 has been completed. If the processing for all front images has not been completed, the control unit 70 starts processing for another front image stored in the frame memory 77. In the present embodiment, the control unit 70 determines that the super-resolution processing has been completed when the processing has been completed for all front images stored in the frame memory 77.

  As described above, the control unit 70 repeats the positional deviation detection and positional deviation correction processing in the front image determined to be appropriate as the image for super-resolution processing, so that the low-resolution image (FIG. 7A). A high-resolution image (see FIG. 7B) is created by performing super-resolution processing on (see). FIG. 7 is a diagram illustrating an experimental example in which super-resolution processing is performed on the fundus image, and is an image obtained by enlarging a part of the acquired fundus image. FIG. 7A is an enlarged view of a low resolution, and FIG. 7B is an enlarged view of a super resolution. When these figures are compared, the image in FIG. 7A has a low resolution, but the image in FIG. 7B can be obtained by a super-resolution process even if it is a zoomed image. ing.

  In the present embodiment, up to 10 front images stored in the frame memory 77 are processed for the reference image. Then, one high-resolution front image is created from a maximum of 11 front images (including the reference image). In the present embodiment, all the front images stored in the frame memory 77 are subjected to super-resolution processing as a target of super-resolution processing, but the present invention is not limited to this. It may be configured to create a high-resolution front image with a plurality of images. For example, five super-resolution processes may be performed in order from the newest one from the reference image. That is, it is not necessary to use the full front image stored in the frame memory 77. Of course, the super-resolution processing does not need to be performed in order from the newest one, and may be performed by selecting one having a high correlation value with the reference image. In addition, when doubling the resolution of the observation image, it is preferable to perform super-resolution processing using at least four images.

  The created high-resolution front image is displayed on the monitor 75. Then, when a new front image is acquired, the control unit 70 sequentially creates a new high-resolution front image and displays it on the monitor 75. As a result, the high-resolution front image displayed on the monitor 75 is updated, and the high-resolution front image of the front image is displayed as a moving image.

  When the examiner operates a photographing switch (not shown) at a desired position, the high-resolution front image displayed on the monitor 75 is stored in the memory 72 as a still image. Thereby, even after imaging, analysis can be performed with a high-resolution front image, and the accuracy of analysis can be improved. The acquired front image may be a high-resolution front image displayed on the monitor 75, or may be a latest front image that has not been subjected to super-resolution processing.

<Updating high-resolution front image>
Here, the update of the high-resolution front image will be described more specifically using the flowchart shown in FIG.

  After the front image is newly acquired for one frame by the OCT optical system 100, the acquired front image is determined to be appropriate, and when the front image is stored in the frame memory 77, the high-resolution front image is updated. . That is, when the plurality of front images stored in the frame memory 77 reaches a predetermined number, the control unit 70 deletes the observation image acquired in the past and replaces it with the newly acquired front image. .

  Hereinafter, the update of the front image in the frame memory 77 will be described. In the present embodiment, the maximum number of photographed images that can be stored in the memory 75 is 10, and when a new front image (front image stored in the eleventh image) is newly acquired, it is stored in the frame memory 77. I can't let you. For this reason, as shown in FIG. 9, when the front image F11 is newly acquired, the past front image (for example, the oldest front image F1) is deleted, and the new front image F11 next to the front image F10. Remember. Thereby, the past front image is deleted, and the capacity of the frame memory 77 is secured. In this way, the front image stored in the frame memory 77 is updated.

  When the front image stored in the frame memory 77 is updated as described above, the control unit 70 displays ten front images (for example, the front image F2 to the front image F11) stored in the frame memory 77. Is used to perform super-resolution processing on the reference image, and a new high-resolution front image is created.

  Creation of a high-resolution front image is performed in the same manner as described above. When the front image is newly updated, the latest front image F11 is set as the reference image for super-resolution processing. Then, processing of all front images (F2 to F10) stored in the frame memory 77 (shift detection, appropriateness determination, super resolution processing) is performed on the reference image.

  When the controller 70 newly creates a high-resolution front image, the controller 70 displays it on the monitor 75. That is, the high-resolution front image displayed on the monitor 75 is updated by replacing the past high-resolution front image with the newly created high-resolution front image. Each time a new front image is acquired, the high-resolution front image is updated, so that a moving image of the high-resolution front image can be observed on the monitor 75.

  As described above, a high-resolution front image of the front image is displayed on the monitor 75 as a moving image. And since the front image displayed on the monitor 75 is a high-resolution front image, it becomes a clear image. Thereby, the resolution of the front image is increased by the super-resolution processing, and a clear front image with improved image quality can be observed on the monitor 75. Furthermore, noise and the like can be removed. Further, by performing the super-resolution processing in real time, a clear front image with improved image quality is always displayed on the monitor 75 as a moving image. Therefore, the examiner can appropriately observe the lesioned part while observing the image on the monitor 75.

<Transformation example>
In the present embodiment, when the predetermined number of front images for use in super resolution processing stored in the frame memory 77 has reached a predetermined number, super resolution processing is started on the newly acquired front image. However, the present invention is not limited to this. For example, as shown in the flowchart of FIG. 10, the super-resolution processing may be started immediately after the front image acquisition is started by the OCT optical system 100.

  The control unit 70 determines whether or not the super-resolution processing of the newly acquired front image and the plurality of front images acquired in time series is appropriate along with the detection of the positional deviation. Then, when it is determined that the control unit 70 is appropriate, the control unit 70 causes the high-resolution front image acquired by the super-resolution processing including the newly acquired front image to be displayed on the monitor 75. The newly acquired front image is displayed on the monitor 75.

  In this case, when a new front image is acquired, this front image is set as a reference image for performing super-resolution processing. Then, super-resolution processing is performed on the reference image. At this time, it is determined whether or not the super-resolution processing between the newly acquired front image (reference image) and the front image stored in the frame memory 77 is appropriate. For example, as described above, suitability is determined based on the correlation value between each front image stored in the frame memory 77 and the newly acquired front image. In the present embodiment, the suitability of the super-resolution processing for the reference image is determined based on the correlation value, but the present invention is not limited to this. Suitability may be determined by a signal for forming a front image. For example, whether the signal intensity (for example, the evaluation value V) of the newly acquired front image, the luminance information of the A scan that forms the front image, or the tomographic image acquired by the interference signal that forms the front image is appropriate. It may be determined.

  In the case of a front image (first front image) acquired for the first time, it is displayed on the monitor 75 without being subjected to super-resolution processing. Next, when the second front image is acquired, it is first determined whether or not the first front image is appropriate (for example, the evaluation value V). If it is stored and it is determined that it is not appropriate, the front image is discarded.

  For example, when the first front image is stored in the frame memory 77, the control unit 70 determines the suitability of the super-resolution processing with the first front image using the second front image as a reference image. . When it is determined that the super-resolution processing is appropriate, the super-resolution processing is performed using the second image as a reference image. Then, a high-resolution front image including the second front image is displayed on the monitor 75. When it is determined that the super-resolution processing is not appropriate, the second front image is displayed on the monitor 75 in a state where the super-resolution processing is not performed.

  Next, when the third front image is acquired, it is first determined whether or not the second front image is appropriate. If it is determined to be appropriate, it is stored in the frame memory 77 and determined not to be appropriate. Then, the front image is discarded.

  Next, the control unit 70 determines the suitability of the super-resolution processing of the first and second front images using the third front image as a reference image. If it is determined to be appropriate, the third image is subjected to super-resolution processing using the reference image as a reference image. Then, a high-resolution front image including the third front image is displayed on the monitor 75. When it is determined that the super-resolution processing is not appropriate, the third front image is displayed on the monitor 75 in a state where the super-resolution processing is not performed.

  By repeating the processing as described above, the high-resolution front image based on the latest front image or the latest front image is displayed on the monitor 75 as it is. In this way, when the eye to be examined moves greatly or blinks, the latest front image is displayed without performing super-resolution processing, so that the state of the eye to be examined can be observed in real time.

  On the other hand, when the eye to be examined moves greatly or blinks, when a high-resolution front image based on a plurality of front images acquired in time series is displayed, a plurality of front images acquired in the past are centered. Therefore, there is a possibility that the actual state of the eye to be examined cannot be grasped in real time because the high-resolution front image is constructed. In this respect, the above processing is useful.

  Of course, even if the control is such that a high-resolution front image including the latest front image is acquired at any time and the super-resolution processing is determined to be inappropriate, the latest front image is directly displayed on the monitor 75. The effect is obtained.

  In addition, in the determination of whether or not the front image is appropriate, the front image determined to be inappropriate is not used for creating a high-resolution front image. In order to create the front image, super-resolution processing is performed between the front image stored in the frame memory 77 and the newly acquired front image. Therefore, the reference image is displayed on the monitor 75 if the front image for obtaining the high-resolution front image is not stored in the frame memory 77 even after the second image.

  In the above description, the fundus front image is described as an example. However, any configuration may be used as long as the imaging optical system receives the light beam reflected from the fundus of the subject's eye and acquires the fundus image. For example, even when a tomographic image is acquired as an observation image using the OCT optical system 100, the above method can be applied. In this case, the time for acquiring a tomographic image of one frame is about 0.03 s to 1.0 s. However, a shift in sub-pixel units occurs in each image due to the above-described fixation fine movement (especially tremor or drift). Therefore, super-resolution processing can be performed. When obtaining a super-resolution tomographic image based on a plurality of tomographic images related to the same part, a tracking unit for tracking the scanning position of the measurement light on the fundus is preferably provided. For example, a front image acquisition unit (for example, SLO, infrared fundus camera) that obtains a front image of the fundus is provided separately from the OCT optical system 100. The control unit 70 detects the shift of the scanning position between the images based on the image output from the front image acquisition unit. The controller 70 controls the driving of the optical scanner 108 based on the detection result, and corrects the scanning position of the measurement light so that the positional deviation is canceled. Even if the tracking is used, a shift in sub-pixel units occurs, so that a super-resolution image can be created. Of course, if the tracking unit is capable of tracking in units of subpixels, the control unit 70 may shift the scanning position by the optical scanner 108 in units of subpixels for each frame of the tomographic image.

  The apparatus of the present embodiment may acquire a plurality of low-resolution fundus three-dimensional tomographic images and create a super-resolution three-dimensional tomographic image based on the plurality of acquired three-dimensional tomographic images. For example, the control unit 90 creates an OCT two-dimensional image related to a direction orthogonal to the depth direction based on data (spectrum data, tomographic data) forming a three-dimensional tomographic image for each three-dimensional tomographic image.

  The control unit 90 corrects the shift between the three-dimensional tomographic images by matching the OCT two-dimensional images formed corresponding to the three-dimensional tomographic images. Furthermore, it is preferable that the control unit 90 processes each three-dimensional tomographic image to correct a shift in the depth direction.

  The control unit 90 acquires a super-resolution three-dimensional tomographic image based on each three-dimensional image whose deviation is corrected. For example, the control unit 90 obtains a super-resolution image by creating a super-resolution image for each B-scan image that forms a three-dimensional tomographic image with respect to the same part. As a result, super-resolution images having different scanning positions in the sub-scanning direction are acquired, and the control unit 70 obtains a super-resolution three-dimensional tomographic image by arranging them.

  As the OCT two-dimensional image, for example, the OCT two-dimensional image is acquired by integrating the spectrum intensity (or depth profile) of the interference signal for each XY point. The OCT two-dimensional image is acquired by converting the number of zero-cross points of the spectrum of the interference signal for each XY point into a luminance value.

  In the above embodiment, an OCT two-dimensional image is used, but it is also possible to perform matching between three-dimensional tomographic images.

  In the above description, the spectrum domain OCT using a spectrum meter has been described as an example, but the present invention is not limited to this. For example, SS-OCT (Swept source OCT) and TD-OCT (Time domain OCT) provided with a wavelength variable light source may be used.

  In the case of super resolution processing, if a part of the front image has a defect due to blinking or the like, the super resolution process may not be performed on the region of the defect part. In this case, for example, when performing super-resolution processing, the luminance distribution of each scanning line in the sub-scanning direction is obtained, and if there are a plurality of scanning lines having luminance values equal to or less than a predetermined value, the super-resolution processing is not performed. That's fine. Thereby, the image quality of the high-resolution front image is improved. Of course, the determination of whether or not to perform super-resolution processing may be performed when determining whether or not to store a front image in the frame memory 77, or when determining whether or not to perform super-resolution processing. You may do it. In the case of determining whether or not to store the front image in the frame memory 77 based on the evaluation value V, for example, the evaluation value V of all the scanning lines forming the front image is determined, and a predetermined number of scanning lines is determined. If the predetermined evaluation value V cannot be acquired, the front image may not be stored.

  In the present embodiment, the frontal fundus image acquired by the OCT optical system 100 has been described as an example, but the present invention is not limited to this. An observation optical system (shooting optical system) may be separately provided to acquire a fundus front image. For example, the observation optical system has an optical scanner that scans the fundus with laser light, and is acquired by an SLO equipped with a scanning fundus imaging optical system that receives a light beam reflected from the fundus of the subject's eye to obtain a front image of the fundus. This is also applied to the fundus image to be processed. The present invention is also applied to a fundus image acquired by a fundus camera or the like. In these cases, the time for acquiring the front image of one frame is about 0.03 s to 1.0 s, but each image is shifted by subpixel due to the above-mentioned fixation fine movement (especially tremor or drift). As a result, super-resolution processing can be performed.

  In the present embodiment, the super-resolution processing in units of 1/10 pixels has been described. However, it may be performed in sub-pixel units (for example, units of 1/100 pixels). Can be obtained.

  In the present embodiment, the super-resolution processing is configured to increase the resolution of the reference image, but is not limited thereto. For example, the super-resolution processing may form a high-resolution front image using a plurality of front images in the template. In this case, alignment between the front images is performed based on the reference image.

  In the present embodiment, when a front image is newly acquired, it is determined whether or not the front image is stored in the frame memory 77. However, the present invention is not limited to this. For example, a configuration in which the determination process is not performed may be used. In this case, when the suitability of the super-resolution process is determined, the correlation value is low with respect to the reference image, and thus the super-resolution process is removed. If the reference image is an error image due to blinking or the like, the correlation value becomes lower than the other front images. For this reason, when the determination process is performed and the number of front images having a low correlation value reaches a predetermined number, the reference image is reset to a different front image. By adopting such a configuration, it is possible to avoid using an error image due to blinking or the like for super-resolution processing without determining whether or not to store a front image in the frame memory 77.

  In the present embodiment, the super-resolution processing is performed on the entire observation image. However, the present invention is not limited to this. The observation image may be divided into predetermined regions, and super resolution processing may be performed for each divided region. Thereby, the processing speed of super-resolution processing can be shortened.

It is a schematic block diagram explaining the structure of the ophthalmologic imaging device which concerns on this embodiment. It is a figure which shows the concept of super-resolution. It is a figure explaining the front image acquired by the OCT optical system, and the front image after a super-resolution process. It is a flowchart explaining the flow of control operation | movement in the case of producing | generating a high-resolution observation image by super-resolution processing. It is a figure explaining the scanning line in the predetermined position of a front image. It is a figure explaining the some scanning line scanned in the depth direction (A scan direction) in a tomogram. It is a figure which shows the high-resolution image acquired by performing a super-resolution process to a low-resolution image and the low-resolution image. It is a flowchart explaining the flow of the update of a high-resolution front image. It is a figure explaining the update of the front image in a frame memory. It is a flowchart explaining the flow of the example of a change of a super-resolution process.

70 Control Unit 72 Memory 74 Control Unit 75 Monitor 77 Frame Memory 100 Optical Tomographic Interference Optical System (OCT Optical System)
108 Optical scanner 300 Fixation target projection unit

Claims (4)

A fundus photographing unit that has a photographing optical system that receives a light flux reflected from the fundus of the subject's eye and acquires fundus image data for a plurality of frames having a shift of less than one pixel;
An image processing unit that obtains a high-resolution image having a higher resolution than that at the time of obtaining the image data by performing composite processing on a plurality of fundus image data obtained by the fundus photographing optical system;
A fundus photographing apparatus comprising:   The fundus imaging means acquires fundus image data for a plurality of frames having a shift of less than one pixel, using fixation eye movement of a subject's eye that occurs while acquiring fundus image data of each frame. 1 fundus imaging apparatus;   The imaging optical system has an optical scanner that scans measurement light on the fundus, and an optical tomographic interference optical system that receives light in which the light beam reflected from the fundus of the eye to be examined and the reference light are combined in order to obtain a tomographic image of the fundus. The scanning fundus photographing optical system according to any one of claims 1 to 2, wherein the scanning fundus photographing optical system has an optical scanner that scans the fundus with laser light and receives a light beam reflected by the fundus of the eye to be examined in order to obtain a front image of the fundus. Fundus photographing device.   The fundus imaging means includes driving means for changing a scanning position on the fundus every time a fundus image of each frame is obtained in order to acquire fundus image data for a plurality of frames having a shift of less than one pixel. Fundus photography device.