JP7410481B2 - Image processing method, scanning imaging method, image processing device, control method thereof, scanning imaging device, control method thereof, program, and recording medium - Google Patents

Description

The present invention relates to an image processing method, a scanning imaging method, an image processing apparatus, a control method thereof, a scanning imaging apparatus, a control method thereof, a program, and a recording medium.

One of the imaging techniques is scanning imaging. Scanning imaging is a technique that collects data by sequentially irradiating multiple locations on a sample with a beam, and constructs an image of the sample from the collected data.

Optical coherence tomography (OCT) is known as a typical example of scanning imaging using light. OCT is a technology that can image a light scattering medium at a resolution of micrometer level or lower, and is applied to medical imaging, nondestructive testing, and the like. OCT is a technique based on low coherence interferometry and typically utilizes near-infrared light to ensure deep penetration of the light scattering medium into the sample.

For example, in ophthalmological image diagnosis, OCT devices are becoming more and more popular, and not only two-dimensional imaging but also three-dimensional imaging, structural analysis, and functional analysis have been put into practical use, and are widely used as powerful diagnostic tools. It has reached this point. Furthermore, in the field of ophthalmology, scanning imaging other than OCT, such as scanning laser ophthalmoscopy (SLO), is also used. Note that scanning imaging that uses light (electromagnetic waves) in a wavelength band other than near-infrared light or ultrasonic waves is also known.

There are various scanning modes used in OCT and SLO, but the so-called "Lissajous scan" for the purpose of motion artifact correction has attracted attention in recent years (for example, Patent Documents 1 to 4, (See Non-Patent Documents 1 and 2).

In a typical Lissajous scan, the measurement light is scanned at high speed in a loop (cycle) of a certain size, so the difference in data acquisition time from one cycle can be virtually ignored. Furthermore, since alignment between cycles can be performed by referring to the intersecting regions of different cycles, it is possible to correct artifacts caused by sample movement. Focusing on these characteristics of Lissajous scans, the ophthalmology field is trying to deal with artifacts caused by eye movements.

In the technique described in Non-Patent Document 1, data collected by a Lissajous scan is divided into a plurality of sub-volumes without relatively large movements, and an en face projection image of each sub-volume is constructed. . Such a front projection image is called a strip. By performing registration between these strips, an image with motion artifacts corrected can be obtained.

As a characteristic of the Lissajous scan, the two strips overlap (intersect) at four points. In the image processing described in Non-Patent Document 1, the largest strip is adopted as an initial reference strip, and strips are sequentially registered and combined in order of size with respect to the initial reference strip. Registration utilizes a cross-correlation function to determine the relative position between a reference strip and other strips. Note that since the strips have an arbitrary shape, a correlation calculation of the overlap area between the strips is performed using a mask to treat each strip as an image of a specific shape (for example, a square shape) (Appendix of Non-Patent Document 1). see A).

Here, the strip is an intensity image of a predetermined gradation, and the mask is binary data. Therefore, while the absolute value of the pixel value of the mask is always small (0 or 1), the absolute value of the pixel value of the strip may be large. Due to this influence, the effect of the mask may be ignored in correlation calculations using strips and masks, and correct correlation coefficients may not be obtained. In particular, when single-precision floating-point type (float type) arithmetic is employed to obtain the correlation coefficient between strips, this problem becomes conspicuous due to the influence of rounding errors caused by the small number of significant digits.

Japanese Patent Application Publication No. 2016-17915 JP2018-68578A JP 2018-140004 Publication JP 2018-140049 Publication

Yiwei Chen, Young-Joo Hong, Shuichi Makita, and Yoshiaki Yasuno, “Three-dimensional eye motion correction by Lissajous scan optical coherence tomography”, Biomedical Optics EXPRESS, Vol. 8, No. 3, 1 Mar 2017, PP. 1783- 1802 Yiwei Chen, Young-Joo Hong, Shuichi Makita, and Yoshiaki Yasuno, “Eye-motion-corrected optical coherence tomography angiography using Lissajous scanning”, Biomedical Optics EXPRESS, Vol. 9, No. 3, 1 Mar 2018, PP. 1111- 1129

One purpose of this invention is to improve the accuracy of image registration using a mask.

Some aspects provide an image processing method, comprising: preparing a first image and a second image of a sample; preparing a mask image; and determining a range of pixel values of the first image and a pixel value of the second image. relatively adjusting a range and a range of pixel values of the mask image, and composing the mask image with each of the first image and the second image to generate a first composite image and a second composite image, A plurality of cross-correlation functions are obtained based on the first composite image and the second composite image, a correlation coefficient is calculated based on the plurality of cross-correlation functions, and a correlation coefficient is calculated between the first image and the second composite image based on the correlation coefficient. Registration with the second image is performed.

In some aspects, an image processing method reduces a difference between a range of pixel values of the sample image and a range of pixel values of the mask image.

In some aspects, the image processing method matches a pixel value range of one of the sample image and the mask image to a pixel value range of the other.

In some aspects, the image processing method normalizes a pixel value range of the sample image according to a pixel value range of the mask image.

In some aspects, the range of pixel values of the mask image is included in a closed interval [0,1], and the image processing method includes dividing the value of each pixel of the sample image by the maximum pixel value in the image. do.

In some aspects, the range of pixel values of the mask image is included in a closed interval [0, 1], and the image processing method is configured to calculate the value of each pixel of the sample image within the range of pixel values of the image. Divide by the maximum value.

In some aspects, the mask image is a binary image with pixel values of 0 or 1.

In some aspects, the mask image is a binary image in which pixels in a region corresponding to the domain of the sample image have a value of 1 and other pixels have values of 0.

In some aspects, the mask image includes a first mask image and a second mask image, and the image processing method includes combining the first mask image with the first image to generate the first composite image. Then, the second mask image is combined with the second image to generate the second combined image.

In some aspects, the first mask image and the second mask image have the same dimensions and the same shape.

In some aspects, the first mask image is a binary image in which the value of a pixel in a region corresponding to the domain of the first image is 1, and the value of other pixels is 0, and the The second mask image is a binary image in which the values of pixels in a region corresponding to the domain of the second image are 1, and the values of other pixels are 0.

In some aspects, each of the first image and the second image is constructed based on data collected by applying a scan to the sample that follows a two-dimensional pattern that includes a series of cycles.

In some aspects, each of the first image and the second image is constructed based on data collected by applying a scan to the sample according to a two-dimensional pattern based on a preset scanning protocol based on a Lissajous function. be done.

Some aspects are scanning imaging methods, the method comprising: applying a light scan to a sample according to a two-dimensional pattern including a series of cycles to collect data; and constructing a first image and a second image based on the data. The pixel value range of the first image, the pixel value range of the second image, and the pixel value range of the mask image are relatively adjusted, and the pixel value range of the first image and the second image are each composing the mask images to generate a first composite image and a second composite image; determining a plurality of cross-correlation functions based on the first composite image and the second composite image; and determining a plurality of cross-correlation functions based on the plurality of cross-correlation functions. calculating a correlation coefficient, registering the first image and the second image based on the correlation coefficient, and merging the first image and the second image based on the result of the registration; Build the image.

In some aspects, the scanning imaging method constructs a third image based on the data, and includes a range of pixel values of the third image, a range of pixel values of the merged image, and a range of pixel values of the mask image. and combine the mask image with each of the third image and the merged image to generate a third composite image and a fourth composite image, and generate the third composite image and the fourth composite image. determine a plurality of cross-correlation functions based on the above, calculate a correlation coefficient based on the plurality of cross-correlation functions, and register the third image and the fourth image based on the correlation coefficient, The third image and the merged image are merged based on the registration result.

In some embodiments, any two cycles of the series of cycles intersect each other at at least one point, and the scanning imaging method collects with an optical scan according to a first group of cycles of the series of cycles. The first image is constructed based on the first data that has been generated, and the second image is constructed based on second data collected by optical scanning according to a second cycle group different from the first cycle group among the series of cycles. Build.

In some embodiments, the scanning imaging method performs optical scanning according to the two-dimensional pattern based on a preset scanning protocol based on a Lissajous function.

Some aspects of the image processing apparatus include a storage unit that stores a first image and a second image of a sample, a first mask image corresponding to the first image, and a second mask image corresponding to the second image. a mask image generation unit that generates a mask image, and a pixel value range of the first image and the second image based on the pixel value range of the first mask image and the pixel value range of the second mask image. a range adjustment unit that adjusts a range of pixel values; a range adjusting unit that combines the first mask image with the first image to generate a first composite image; and combines the second mask image with the second image; a composite image generation unit that generates a second composite image based on the first composite image and the second composite image; a first calculation unit that calculates a plurality of cross-correlation functions based on the plurality of cross-correlation functions; The image forming apparatus includes a second calculation section that calculates a correlation coefficient, and a registration section that performs registration of the first image and the second image based on the correlation coefficient.

In some aspects, the image processing device includes a data receiving unit that receives data collected from a sample by optical scanning according to a two-dimensional pattern including a series of cycles; and an image construction unit that constructs the image.

Some aspects are a method of controlling an image processing device including a storage unit that stores a first image and a second image of a sample, and a processor, the method of controlling the image processing device including a storage unit that stores a first image and a second image of a sample, A mask image and a second mask image corresponding to the second image are generated, and the pixel values of the first image are determined based on the pixel value range of the first mask image and the pixel value range of the second mask image. and a range of pixel values of the second image, and combine the first mask image with the first image to generate a first composite image, and combine the second mask image with the second image. to generate a second composite image, obtain a plurality of cross-correlation functions based on the first composite image and the second composite image, calculate a correlation coefficient based on the plurality of cross-correlation functions, Then, control is performed to perform registration between the first image and the second image based on the correlation coefficient.

Some embodiments are scanning imaging devices, including a scanning unit that collects data by applying optical scanning to a sample according to a two-dimensional pattern that includes a series of cycles; an image construction section that constructs an image; a mask image generation section that generates a first mask image according to the first image and a second mask image according to the second image; and a pixel value of the first mask image. a range adjustment unit that adjusts a pixel value range of the first image and a pixel value range of the second image based on a range of pixel values of the first image and a pixel value range of the second mask image; a composite image generation unit that generates a first composite image by combining two mask images, and generates a second composite image by combining the second mask image with the second image; a first calculation section that calculates a plurality of cross-correlation functions based on the second composite image; a second calculation section that calculates a correlation coefficient based on the plurality of cross-correlation functions; and a second calculation section that calculates a correlation coefficient based on the correlation coefficient. The image forming apparatus includes a registration section that performs registration of the first image and the second image, and a merge processing section that constructs a merged image of the first image and the second image based on the result of the registration.

In some embodiments, the scanning unit includes a deflector capable of deflecting light in a first direction and a second direction that are different from each other, and repeats a change in the deflection direction along the first direction in a first period, and Optical scanning according to the two-dimensional pattern is applied to the sample by repeating changes in the deflection direction along the second direction at a second period different from the first period.

Some aspects are methods of controlling a scanning imaging device including a scanning unit that collects data by applying optical scanning to a sample, and a processor, the scanning unit being a two-dimensional imaging device that includes a series of cycles. controlling the processor to collect data by applying light scanning according to a pattern to the sample; and controlling the processor to construct a first image and a second image based on the data, and to construct a first mask image responsive to the first image. and a second mask image corresponding to the second image, and generate a pixel value range of the first image based on the pixel value range of the first mask image and the pixel value range of the second mask image. and adjusting a pixel value range of the second image, composing the first mask image with the first image to generate a first composite image, and composing the second mask image with the second image. to generate a second composite image, obtain a plurality of cross-correlation functions based on the first composite image and the second composite image, calculate a correlation coefficient based on the plurality of cross-correlation functions, and calculate the correlation coefficient based on the plurality of cross-correlation functions. Control is performed to register the first image and the second image based on a relational coefficient, and to construct a merged image of the first image and the second image based on the result of the registration. do.

Some embodiments are programs that cause a computer to execute any of the methods described above.

Some embodiments are computer-readable non-transitory recording media on which a program for causing a computer to execute any of the methods described above is recorded.

It is possible to at least partially combine any two or more aspects. Furthermore, any aspect of the present disclosure can be at least partially combined with any aspect. Further, it is possible to at least partially combine any two or more aspects of the present disclosure with at least partially.

According to example aspects, accuracy of image registration using a mask can be improved.

FIG. 1 is a schematic diagram illustrating an example of the configuration of a scanning imaging device (ophthalmological device) according to an exemplary embodiment. FIG. 1 is a schematic diagram illustrating an example of the configuration of a scanning imaging device (ophthalmological device) according to an exemplary embodiment. FIG. 1 is a schematic diagram illustrating an example of the configuration of a scanning imaging device (ophthalmological device) according to an exemplary embodiment. FIG. 1 is a schematic diagram illustrating an example of the configuration of a scanning imaging device (ophthalmological device) according to an exemplary embodiment. FIG. 1 is a schematic diagram illustrating an example of the configuration of a scanning imaging device (ophthalmological device) according to an exemplary embodiment. FIG. 2 is a schematic diagram illustrating an example of a Lissajous scan pattern performed by a scanning imaging device (ophthalmological device) according to an exemplary embodiment. 1 is a flowchart illustrating an example of the operation of a scanning imaging device (ophthalmological device) according to an exemplary embodiment. FIG. 2 is a diagram for explaining an example of the operation of a scanning imaging device (ophthalmological device) according to an exemplary embodiment. FIG. 2 is a diagram for explaining an example of the operation of a scanning imaging device (ophthalmological device) according to an exemplary embodiment. FIG. 2 is a diagram for explaining an example of the operation of a scanning imaging device (ophthalmological device) according to an exemplary embodiment.

An image processing method, a scanning imaging method, an image processing device, a control method thereof, a scanning imaging device, a control method thereof, a program, and a recording medium according to some exemplary embodiments will be described with reference to the drawings. .

The techniques disclosed in the documents cited herein and any other known techniques may be incorporated into the exemplary embodiments. Furthermore, unless otherwise specified, there will be no distinction between "image data" and an "image" based on it, and no distinction will be made between a "part" of the eye to be examined and its "image."

A scanning imaging device according to an exemplary embodiment described below is an ophthalmological device capable of measuring the fundus of a living eye using Fourier domain OCT (eg, swept source OCT). The type of OCT that can be employed in the embodiments is not limited to swept source OCT, but may be, for example, spectral domain OCT or time domain OCT.

Scanning imaging devices according to some example embodiments may be capable of utilizing scanning modalities other than OCT. For example, some example embodiments may employ any optical scanning modality such as SLO.

Additionally, scanning modalities applicable to scanning imaging devices according to some example embodiments are not limited to optical scanning modalities. For example, some exemplary embodiments may employ scanning modalities that utilize electromagnetic waves other than light, scanning modalities that utilize ultrasound, and the like.

In addition to processing data collected with an OCT scan and/or processing data collected with an SLO, the ophthalmological device according to some example embodiments can process data acquired with other modalities. It's fine. Other modalities may be, for example, a fundus camera, a slit lamp microscope, and an ophthalmic surgical microscope. Ophthalmic devices according to some example embodiments may have the functionality of such modalities.

The target (sample) to which OCT is applied is not limited to the fundus of the eye, but may be any part of the eye such as the anterior segment or vitreous body, or may be a part or tissue of a living body other than the eye. It may be an object other than a living body.

Ophthalmic devices according to exemplary embodiments described below include some aspects of a scanning imaging device, some aspects of a method of controlling a scanning imaging device, some aspects of an image processing device, and some aspects of an image processing device. Some aspects of methods for controlling devices, some aspects of image processing methods, and some aspects of scanning imaging methods are provided.

<Configuration of scanning imaging device>
The ophthalmologic apparatus 1 shown in FIG. 1 includes a fundus camera unit 2, an OCT unit 100, and an arithmetic control unit 200. The fundus camera unit 2 is provided with a group of elements (optical elements, mechanisms, etc.) for photographing the eye E from the front. The OCT unit 100 is provided with a part of a group of elements (optical elements, mechanisms, etc.) for applying an OCT scan to the eye E to be examined. Another part of the element group for OCT scanning is provided in the fundus camera unit 2. Arithmetic control unit 200 includes one or more processors that perform various calculations and controls. In addition to these, the ophthalmologic apparatus 1 may include an element for supporting the subject's face and an element for switching the region to which an OCT scan is applied. Examples of the former element include a chin rest and a forehead rest. An example of the latter element is a lens unit used to switch the OCT scan application site from the fundus to the anterior segment of the eye.

The functionality of the elements disclosed herein is implemented using circuitry or processing circuitry. Circuitry or processing circuitry may include a general purpose processor, a special purpose processor, an integrated circuit, a central processing unit (CPU), a graphics processing unit (GPU), an ASIC ( Application Specific Integrated Circuit), programmable logic devices (for example, SPLD (Simple Programmable Logic Device), CPLD (Complex Programmable Logic Device), FPGA (Field Programmable Gate Array), conventional circuit configurations, and any combination thereof A processor is considered to be a processing circuitry or circuitry that includes transistors and/or other circuitry. In this disclosure, the term circuitry, unit, means, or similar terms may be used to perform the disclosed functionality. Hardware that executes or is programmed to perform the functions disclosed. The hardware may be the hardware disclosed herein or that is programmed to perform the functions described. It may be known hardware programmed and/or configured to is a combination of hardware and software that is used to configure the hardware and/or processor.

<Funus camera unit 2>
The fundus camera unit 2 is provided with an optical system for photographing the fundus Ef of the eye E to be examined. The acquired image of the fundus Ef (referred to as a fundus image, fundus photograph, etc.) is a frontal image such as an observation image or a photographed image. Observation images are obtained, for example, by video shooting using near-infrared light, and are used for alignment, focusing, tracking, and the like. The photographed image is, for example, a still image using flash light in the visible region or infrared region.

The fundus camera unit 2 includes an illumination optical system 10 and a photographing optical system 30. The illumination optical system 10 irradiates the eye E to be examined with illumination light. The photographing optical system 30 detects the return light of the illumination light from the eye E to be examined. The measurement light from the OCT unit 100 is guided to the eye E through the optical path within the fundus camera unit 2, and the return light is guided to the OCT unit 100 through the same optical path.

Light (observation illumination light) output from the observation light source 11 of the illumination optical system 10 is reflected by the concave mirror 12, passes through the condensing lens 13, and passes through the visible cut filter 14 to become near-infrared light. Further, the observation illumination light is once focused near the photographing light source 15, reflected by a mirror 16, and passes through a relay lens system 17, a relay lens 18, an aperture 19, and a relay lens system 20. The observation illumination light is reflected at the periphery of the apertured mirror 21 (the area around the aperture), passes through the dichroic mirror 46, and is refracted by the objective lens 22 to illuminate the eye E (fundus Ef). do. The return light of the observation illumination light from the subject's eye E is refracted by the objective lens 22, passes through the dichroic mirror 46, passes through the hole formed in the center area of the perforated mirror 21, and passes through the dichroic mirror 55. , passes through the photographic focusing lens 31 and is reflected by the mirror 32. Furthermore, this returned light passes through the half mirror 33A, is reflected by the dichroic mirror 33, and is imaged by the imaging lens 34 on the light receiving surface of the image sensor 35. The image sensor 35 detects the returned light at a predetermined frame rate. Note that the focus (focal position) of the photographing optical system 30 is adjusted to match the fundus Ef or the anterior segment of the eye.

The light output from the photographing light source 15 (photographing illumination light) passes through the same path as the observation illumination light and is irradiated onto the fundus Ef. The return light of the imaging illumination light from the eye E is guided to the dichroic mirror 33 through the same path as the return light of the observation illumination light, passes through the dichroic mirror 33, is reflected by the mirror 36, and is reflected by the imaging lens 37. An image is formed on the light receiving surface of the image sensor 38.

A liquid crystal display (LCD) 39 displays a fixation target (fixation target image). A portion of the light beam output from the LCD 39 is reflected by the half mirror 33A, then reflected by the mirror 32, passes through the photographic focusing lens 31 and the dichroic mirror 55, and then passes through the hole of the perforated mirror 21. The light beam that has passed through the hole of the perforated mirror 21 is transmitted through the dichroic mirror 46, refracted by the objective lens 22, and projected onto the fundus Ef. By changing the display position of the fixation target image on the LCD 39, the direction (fixation direction, fixation position) in which the line of sight of the eye E to be examined is guided is changed.

Instead of a display device such as an LCD, for example, a light emitting element array or a combination of a light emitting element and a mechanism for moving the light emitting element may be used.

The alignment optical system 50 generates an alignment index used for positioning (alignment) the optical system with respect to the eye E to be examined. The alignment light output from the light emitting diode (LED) 51 passes through the aperture 52 , the aperture 53 , and the relay lens 54 , is reflected by the dichroic mirror 55 , passes through the hole of the perforated mirror 21 , and passes through the dichroic mirror 46 . The light is transmitted through the objective lens 22 and projected onto the eye E to be examined. Return light of the alignment light from the eye E to be examined (corneal reflected light, etc.) is guided to the image sensor 35 through the same path as the return light of the observation illumination light. Manual alignment or auto alignment can be performed based on the received light image (alignment index image).

As in the prior art, the alignment index image of this example consists of two bright spot images whose positions change depending on the alignment state. When the relative position between the eye E and the optical system changes in the xy direction, the two bright spot images are integrally displaced in the xy direction. When the relative position between the eye E and the optical system changes in the z direction, the relative position (distance) between the two bright spot images changes. When the distance between the eye E and the optical system in the z direction matches the predetermined working distance, the two bright spot images overlap. When the position of the eye E and the position of the optical system match in the xy direction, two bright spot images are presented within or near a predetermined alignment target. When the distance between the eye E and the optical system in the z direction matches the working distance, and the position of the eye E and the optical system match in the xy direction, the two bright spot images overlap and alignment occurs. presented within the target.

In auto-alignment, the data processing unit 230 detects the positions of two bright spot images, and the main control unit 211 controls the movement mechanism 150, which will be described later, based on the positional relationship between the two bright spot images and the alignment target. . In manual alignment, the main control unit 211 displays two bright spot images on the display unit 241 together with the observation image of the eye E, and the user uses the operation unit 242 while referring to the two displayed bright spot images. to operate the moving mechanism 150.

The focus optical system 60 generates a split index used for focus adjustment for the eye E to be examined. In conjunction with the movement of the photographing focusing lens 31 along the optical path of the photographing optical system 30 (photographing optical path), the focusing optical system 60 is moved along the optical path of the illumination optical system 10 (illumination optical path). The reflection rod 67 is inserted into and removed from the illumination optical path. When performing focus adjustment, the reflective surface of the reflective rod 67 is arranged at an angle to the illumination optical path. The focus light output from the LED 61 passes through a relay lens 62 , is separated into two beams by a split indicator plate 63 , passes through a two-hole diaphragm 64 , is reflected by a mirror 65 , and is reflected by a condensing lens 66 into a reflecting rod 67 . Once the image is formed on the reflective surface of the image, it is reflected. Further, the focus light passes through the relay lens 20, is reflected by the perforated mirror 21, passes through the dichroic mirror 46, and is projected onto the eye E through the objective lens 22. Return light of the focus light from the eye E to be examined (fundus reflected light, etc.) is guided to the image sensor 35 through the same path as the return light of the alignment light. Manual focusing or autofocusing can be performed based on the received light image (split index image).

Diopter correction lenses 70 and 71 can be selectively inserted into the photographing optical path between the perforated mirror 21 and the dichroic mirror 55. The diopter correction lens 70 is a plus lens (convex lens) for correcting severe hyperopia. The diopter correction lens 71 is a minus lens (concave lens) for correcting severe myopia.

The dichroic mirror 46 combines the fundus photographing optical path and the OCT optical path (measuring arm). The dichroic mirror 46 reflects light in a wavelength band used for OCT and transmits light for fundus photography. The measurement arm is provided with a collimator lens unit 40, a retroreflector 41, a dispersion compensation member 42, an OCT focusing lens 43, an optical scanner 44, and a relay lens 45 in this order from the OCT unit 100 side.

The retroreflector 41 is movable in the direction of the arrow shown in FIG. 1, thereby changing the length of the measurement arm. Changing the measurement arm length is used, for example, to correct the optical path length according to the axial length or to adjust the interference state.

The dispersion compensation member 42 acts together with a dispersion compensation member 113 (described later) disposed on the reference arm to match the dispersion characteristics of the measurement light LS and the dispersion characteristics of the reference light LR.

The OCT focusing lens 43 is moved along the measurement arm to perform focus adjustment of the measurement arm. Movement of the imaging focusing lens 31, movement of the focusing optical system 60, and movement of the OCT focusing lens 43 can be controlled in a coordinated manner.

The optical scanner 44 is arranged at a position that is substantially optically conjugate with the pupil of the eye E to be examined. The optical scanner 44 deflects the measurement light LS guided by the measurement arm. The optical scanner 44 is a galvano scanner capable of two-dimensional scanning, including, for example, a galvano mirror for scanning in the x direction and a galvano mirror for scanning in the y direction.

<OCT unit 100>
As illustrated in FIG. 2, the OCT unit 100 is provided with an optical system for applying swept source OCT. This optical system includes an interference optical system. This interference optical system splits the light from the wavelength variable light source (wavelength swept light source) into measurement light and reference light, and overlaps the return light of the measurement light from the eye E with the reference light that has passed through the reference optical path. Together, they generate interference light, and this interference light is detected. The data (detection signal) obtained by the interference optical system is a signal representing the spectrum of interference light, and is sent to the arithmetic and control unit 200.

The light source unit 101 includes, for example, a near-infrared variable wavelength laser that changes the emission wavelength at high speed at least in the near-infrared wavelength band. Light L0 output from the light source unit 101 is guided to a polarization controller 103 by an optical fiber 102, and its polarization state is adjusted. Further, the light L0 is guided to a fiber coupler 105 by an optical fiber 104 and is split into a measurement light LS and a reference light LR. The optical path of the measurement light LS is called a measurement arm, and the optical path of the reference light LR is called a reference arm.

The reference light LR is guided to a collimator 111 by an optical fiber 110, converted into a parallel light beam, and guided to a retroreflector 114 via an optical path length correction member 112 and a dispersion compensation member 113. The optical path length correction member 112 acts to match the optical path length of the reference light LR and the optical path length of the measurement light LS. The dispersion compensation member 113 acts together with the dispersion compensation member 42 disposed on the measurement arm to match the dispersion characteristics between the reference light LR and the measurement light LS. The retroreflector 114 is movable along the optical path of the reference light LR incident thereon, thereby changing the length of the reference arm. Changing the reference arm length is used, for example, to correct the optical path length according to the axial length or to adjust the interference state.

The reference light LR that has passed through the retroreflector 114 passes through the dispersion compensating member 113 and the optical path length correcting member 112, is converted from a parallel beam into a convergent beam by the collimator 116, and enters the optical fiber 117. The reference light LR incident on the optical fiber 117 is guided to a polarization controller 118 to have its polarization state adjusted, guided to an attenuator 120 through an optical fiber 119 to have its light amount adjusted, and then sent to a fiber coupler 122 through an optical fiber 121. be guided.

On the other hand, the measurement light LS generated by the fiber coupler 105 is guided by the optical fiber 127 and converted into a parallel light beam by the collimator lens unit 40. The light passes through the relay lens 45, is reflected by the dichroic mirror 46, is refracted by the objective lens 22, and is projected onto the eye E to be examined. The measurement light LS is scattered and reflected at various depth positions of the eye E to be examined. The return light of the measurement light LS from the eye E to be examined travels the same path as the outgoing path in the opposite direction, is guided to the fiber coupler 105, and reaches the fiber coupler 122 via the optical fiber 128.

The fiber coupler 122 superimposes the measurement light LS incident on the optical fiber 128 and the reference light LR incident on the optical fiber 121 to generate interference light. The fiber coupler 122 generates a pair of interference lights LC by branching the generated interference lights at a predetermined branching ratio (for example, 1:1). A pair of interference lights LC are guided to a detector 125 through optical fibers 123 and 124, respectively.

Detector 125 includes, for example, a balanced photodiode. The balanced photodiode includes a pair of photodetectors that respectively detect a pair of interference lights LC, and outputs a difference between a pair of detection results obtained by these photodetectors. Detector 125 sends this output (detection signal) to data acquisition system (DAS) 130.

The data collection system 130 is supplied with a clock KC from the light source unit 101. The clock KC is generated in the light source unit 101 in synchronization with the output timing of each wavelength swept within a predetermined wavelength range by the variable wavelength light source. For example, the light source unit 101 splits the light L0 of each output wavelength to generate two branched lights, optically delays one of these branched lights, combines these branched lights, and generates the resulting composite light. A clock KC is generated based on the detection result. The data acquisition system 130 performs sampling of the detection signal input from the detector 125 based on the clock KC. Data acquisition system 130 sends the results of this sampling to arithmetic and control unit 200 .

In this example, both an element for changing the measurement arm length (for example, retroreflector 41) and an element for changing the reference arm length (for example, retroreflector 114 or reference mirror) are provided. However, only one element may be provided. Moreover, the elements for changing the difference (optical path length difference) between the measurement arm length and the reference arm length are not limited to these, and may be any elements (optical members, mechanisms, etc.).

<Control system/processing system>
Configuration examples of the control system and processing system of the ophthalmologic apparatus 1 are shown in FIGS. 3, 4A, and 4B. The control section 210, the image construction section 220, and the data processing section 230 are provided in the arithmetic control unit 200, for example. The ophthalmological apparatus 1 may include a communication device for performing data communication with an external device. The ophthalmological apparatus 1 may include a drive device (reader/writer) for reading data from a recording medium and writing data to the recording medium.

<Control unit 210>
The control unit 210 executes various controls. Control section 210 includes a main control section 211 and a storage section 212. Further, as shown in FIG. 4A, in this embodiment, the main control unit 211 includes a scan control unit 2111, and the storage unit 212 stores a scan protocol 2121.

<Main control unit 211>
The main control unit 211 includes a processor and controls each element of the ophthalmologic apparatus 1 (including the elements shown in FIGS. 1 to 4B). The main control unit 211 is realized by cooperation between hardware including a processor and control software. The scan control unit 2111 controls OCT scanning regarding a scan area of a predetermined shape and a predetermined size.

The photographic focus drive unit 31A moves the photographic focus lens 31 disposed in the photographic optical path and the focus optical system 60 disposed in the illumination optical path under the control of the main control unit 211. The retroreflector (RR) driving section 41A moves the retroreflector 41 provided on the measurement arm under the control of the main control section 211. The OCT focusing drive section 43A moves the OCT focusing lens 43 disposed on the measurement arm under the control of the main control section 211. A retroreflector (RR) drive unit 114A provided on the measurement arm moves the retroreflector 114 placed on the reference arm under the control of the main control unit 211. Each of the above-described drive units includes an actuator such as a pulse motor that operates under the control of the main control unit 211. The optical scanner 44 operates under the control of the main control section 211 (scanning control section 2111).

The moving mechanism 150 typically moves the fundus camera unit 2 three-dimensionally, for example, an x-stage that can move in the ±x direction (left-right direction), an x-movement mechanism that moves the x-stage, and a Includes a y stage that can move in the y direction (up and down direction), a y movement mechanism that moves the y stage, a z stage that can move in the ±z direction (depth direction), and a z movement mechanism that moves the z stage. . Each movement mechanism includes an actuator such as a pulse motor that operates under the control of the main control unit 211.

<Storage unit 212>
The storage unit 212 stores various data. The data stored in the storage unit 212 includes OCT images, fundus images, eye information to be examined, control information, and the like. The eye information to be examined includes patient information such as patient ID and name, left eye/right eye identification information, electronic medical record information, and the like. Control information is information regarding specific control. The control information in this embodiment includes a scanning protocol 2121.

The scanning protocol 2121 is an agreement regarding the contents of control for OCT scanning regarding a scanning area of a predetermined shape and a predetermined size, and includes a set of various control parameters (scanning control parameters). The scanning protocol 2121 includes a protocol for each scanning mode. The scanning protocol 2121 of this embodiment includes at least a Lissajous scan protocol, and may further include protocols such as B scan (line scan), cross scan, radial scan, and raster scan.

The scanning control parameters of this embodiment include at least parameters indicating the content of control for the optical scanner 44. This parameter includes, for example, a parameter indicating a scan pattern, a parameter indicating a scan speed, a parameter indicating a scan interval, and the like. The scan pattern indicates the shape of a scan path, and examples thereof include a Lissajous pattern, a line pattern, a cross pattern, a radial pattern, and a raster pattern. The scan speed is defined, for example, as the A-scan repetition rate. The scan interval is defined, for example, as the interval between adjacent A scans, that is, as the arrangement interval of scan points.

Note that, similar to the conventional techniques disclosed in Patent Documents 1 to 4 and Non-Patent Documents 1 and 2, the "Lissajous scan" of this embodiment is based on the point that two mutually orthogonal simple harmonic motions are obtained as an ordered pair. Not only Lissajous scan in a “narrow sense” where the path follows a pattern drawn by a trajectory (Lissajous pattern, Lissajous figure, Lissajous curve, Lissajous function, Bowditch curve), but also Lissajous in a “broad sense” that follows a predetermined two-dimensional pattern including a series of cycles. It may also be a scan.

In this embodiment, the optical scanner 44 includes a first galvano mirror that deflects the measurement light LS in the x direction and a second galvano mirror that deflects the measurement light LS in the y direction. In the Lissajous scan, the first galvanometer mirror is controlled so that the deflection direction along the x direction is repeated in the first cycle, and the second galvano mirror is controlled so that the deflection direction along the y direction is repeated in the second cycle. This is achieved by controlling the galvanometer mirror. Here, the first period and the second period are different from each other.

For example, the Lissajous scan of this embodiment not only scans a Lissajous pattern in a narrow sense obtained from a combination of two sine waves, but also scans a pattern in which a specific term (for example, an odd-order polynomial) is added to the Lissajous pattern, or scans a triangular wave pattern. It may also be a pattern scan based on .

"Cycle" generally means an object consisting of a plurality of sampling points of a certain length, which may be, for example, a closed or nearly closed curve (i.e., the start and end points of the cycle coincide or nearly coincide). ).

Typically, the scanning protocol 2121 is set based on a Lissajous function. As shown in equation (9) of Non-Patent Document 1, the Lissajous function is expressed, for example, by the following parametric equation system: x(t _i )=A・cos(2π・(f _A /n)・t _i ), y(t _i )=A・cos(2π・(f _A・(n-2)/n ² )・t _i ).

Here, x is the horizontal axis of the two-dimensional coordinate system in which the Lissajous curve is defined, y is the vertical axis, t _i is the acquisition time of the i-th A line in the Lissajous scan, A is the scan range (amplitude), f _A is the acquisition rate of the A-line (scan speed, repetition rate of A-scan), and n is the number of A-lines in each cycle in the x (horizontal axis) direction.

FIG. 5 shows an example of the distribution of scan lines applied to the fundus Ef in such a Lissajous scan.

Since any pair of cycles included in a Lissajous scan (in a narrow or broad sense) intersect with each other at one or more points (especially at two or more points), the difference between pairs of data collected from any pair of cycles in a Lissajous scan Registration can be performed. This makes it possible to use the image construction method and motion artifact correction method disclosed in Non-Patent Document 1 (or 2). Hereinafter, unless otherwise mentioned, the method described in Non-Patent Document 1 is applied.

The control information stored in the storage unit 212 is not limited to the above example. For example, the control information may include information for performing focus control (focus control parameters).

The focus control parameter is a parameter indicating the content of control for the OCT focusing drive unit 43A. Examples of the focus control parameters include a parameter indicating the focal position of the measurement arm, a parameter indicating the moving speed of the focal position, and a parameter indicating the moving acceleration of the focal position. The parameter indicating the focal position is, for example, a parameter indicating the position of the OCT focusing lens 43. The parameter indicating the moving speed of the focal position is, for example, a parameter indicating the moving speed of the OCT focusing lens 43. The parameter indicating the moving acceleration of the focal position is, for example, a parameter indicating the moving acceleration of the OCT focusing lens 43. The moving speed may or may not be constant. The same applies to movement acceleration.

According to such focus control parameters, it becomes possible to perform focus adjustment according to the shape of the fundus Ef (typically a concave shape that is deep at the center and shallow at the periphery) and the aberration distribution. Focus control is executed in conjunction with scanning control (Lissajous scan repetition control), for example. This corrects motion artifacts and provides a high-quality image that is in focus throughout the scan range.

<Scan control unit 2111>
The scan control unit 2111 controls at least the optical scanner 44 based on the scan protocol 2121. The scan control unit 2111 may further control the light source unit 101 in conjunction with the control of the optical scanner 44 based on the scan protocol 2121. The scan control unit 2111 is realized by cooperation between hardware including a processor and scan control software including a scan protocol 2121.

<Image construction unit 220>
The image construction unit 220 includes a processor, and constructs OCT image data of the fundus Ef based on the signal (sampling data) input from the data acquisition system 130. This OCT image data construction includes noise removal, filtering, fast Fourier transform (FFT), etc., similar to conventional Fourier domain OCT (swept source OCT). When another type of OCT technique is employed, the image construction unit 220 constructs OCT image data by executing known processing according to the type.

As described above, in this embodiment, the Lissajous scan is applied to the fundus Ef. The image construction unit 220, together with the data processing unit 230, applies the image construction method and motion disclosed in Non-Patent Document 1 or 2, for example, to data acquired through collection by Lissajous scan and sampling by the data collection system 130. Three-dimensional image data of the fundus Ef is constructed by applying an artifact correction method.

The image construction unit 220 (and/or the data processing unit 230) can form a display image by applying rendering to the three-dimensional image data. Examples of applicable rendering methods include volume rendering, surface rendering, maximum intensity projection (MIP), minimum intensity projection (MinIP), and multiplanar reconstruction (MPR).

The image construction unit 220 (and/or the data processing unit 230) can construct an enface OCT image based on the three-dimensional image data. For example, the image construction unit 220 (and/or the data processing unit 230) can construct projection data by projecting three-dimensional image data in the z direction (A-line direction, depth direction). Further, the image construction unit 220 (and/or the data processing unit 230) can construct a shadowgram by projecting partial three-dimensional image data, which is a part of the three-dimensional image data, in the z direction. This partial three-dimensional image data is set using, for example, segmentation. Segmentation is a process of identifying partial regions in an image. In this example, segmentation can be performed to identify an image region corresponding to a predetermined tissue of the fundus Ef.

The ophthalmological device 1 may be capable of performing OCT-Angiography. OCT angiography is an imaging technique that constructs images in which blood vessels are emphasized (for example, see Non-Patent Document 2, Japanese Patent Publication No. 2015-515894, etc.). Generally, the fundus tissue (structure) does not change over time, but the blood flow inside blood vessels changes over time. In OCT angiography, images are generated by emphasizing portions (blood flow signals) where such temporal changes exist. Note that OCT angiography is also called OCT motion contrast imaging. Furthermore, images obtained by OCT angiography are called angiography images, angiograms, motion contrast images, and the like.

When OCT angiography is performed, the ophthalmologic apparatus 1 repeatedly scans the same region of the fundus Ef a predetermined number of times. For example, the ophthalmologic apparatus 1 repeatedly performs the above-described scanning control (Lissajous scan repetition control) a predetermined number of times. Thereby, a plurality of three-dimensional data (three-dimensional data sets) corresponding to the Lissajous scan application area are collected by the data collection system 130. The image construction unit 220 and/or the data processing unit 230 can construct a motion contrast image from this three-dimensional data set. This motion contrast image is an angiographic image in which temporal changes in interference signals caused by blood flow in the fundus Ef are emphasized. This angiographic image is three-dimensional angiographic image data expressing the three-dimensional distribution of blood vessels in the fundus Ef.

The image construction unit 220 and/or the data processing unit 230 can construct arbitrary two-dimensional angiography image data and/or arbitrary pseudo three-dimensional angiography image data from this three-dimensional angiography image data. be. For example, the image constructing unit 220 can construct two-dimensional angiographic image data representing an arbitrary cross section of the fundus Ef by applying multi-sectional reconstruction to the three-dimensional angiographic image data. Furthermore, the image construction unit 220 can construct frontal angiography image data of the fundus Ef by applying projection imaging or shadowgram formation to the three-dimensional angiography image data.

In this embodiment, the image constructor 220 constructs a plurality of strips from data collected by the data collection system 130. As described in Non-Patent Document 1, the image construction unit 220 divides the volume (three-dimensional data) collected by Lissajous scan into a plurality of sub-volumes in which there is no relatively large movement, and calculates the size of each sub-volume. Build an en face projection image. This front projection image is a strip. By applying registration and merging processing to the plurality of strips obtained in this manner, an image with motion artifacts corrected is obtained. As will be described later, the registration and merging processing is executed by the data processing unit 230.

The image construction unit 220 is realized by cooperation between hardware including a processor and image construction software.

<Data processing unit 230>
The data processing unit 230 includes a processor and applies various data processing to the image of the eye E to be examined. For example, the data processing unit 230 is realized by cooperation between hardware including a processor and data processing software.

The data processing unit 230 may be configured to perform registration between two images acquired of the fundus Ef. For example, the data processing unit 230 is configured to perform registration between three-dimensional image data obtained by OCT and a frontal image obtained by the fundus camera unit 2. Further, the data processing unit 230 is configured to perform registration between two OCT images obtained by OCT. Further, the data processing unit 230 is configured to perform registration between two frontal images acquired by the fundus camera unit 2. Further, it may be configured to apply registration to the analysis results of OCT images or the analysis results of frontal images. These registrations can be performed using known techniques, and include, for example, feature point extraction and affine transformation.

Further, the data processing unit 230 is configured to process data obtained using Lissajous scan. As described above, the image construction unit 220 constructs a plurality of strips from the data collected in the Lissajous scan. The data processing unit 230 constructs an image in which motion artifacts are corrected by applying registration and merging processing to these strips.

As a characteristic of the Lissajous scan, any two strips have an overlapping region. Similar to the method described in Non-Patent Document 1, the data processing unit 230 is configured to perform registration between strips using overlap between strips. The data processing unit 230 first orders the plurality of strips constructed by the image construction unit 220 according to size (area, etc.) (first to Nth strips), and uses the first strip, which is the largest strip, as an initial reference. Specify strip. Next, the data processing unit 230 registers the second strip using the first strip as a reference, and merges (synthesizes) the first strip and the second strip. The data processing unit 230 performs registration of the third strip based on the merged strip obtained thereby, and merges this merged strip and the third strip. By sequentially executing such registration and merging processing in the above order, the first to Nth strips are aligned and pasted together, and an image with motion artifacts corrected is obtained.

In such registration, a cross-correlation function is used to determine the relative position between the reference strip and other strips, but since the strips have arbitrary shapes, each strip is ) Correlation calculation between strips is performed using a mask for handling as an image (see Appendix A of Non-Patent Document 1).

As mentioned above, the strip is an intensity image with a predetermined gradation, and the mask image is a binary image, so the difference between the absolute value of the pixel value of the strip and the absolute value of the pixel value of the mask becomes large, and the difference between the strip and the mask image becomes large. In the correlation calculation using the mask, the effect of the mask may be ignored and a correct correlation coefficient may not be obtained. In particular, when single-precision floating-point type (float type) arithmetic is adopted to calculate the correlation coefficient between strips from the viewpoint of cost, etc., this problem arises due to the effects of rounding errors due to the small number of significant digits. appears prominently.

In order to deal with this problem, this embodiment employs a data processing unit 230 having the configuration shown in FIG. 4B. The data processing unit 230 of this example includes a mask image generation unit 231, a range adjustment unit 232, a composite image generation unit 233, a cross-correlation function calculation unit 234, a correlation coefficient calculation unit 235, and a registration unit 236. , and a merge processing unit 237. In the example described below, any two strips are considered, one of which is designated as a reference strip, and the other strip (registering strip) is registered to this reference strip. .

<Mask image generation unit 231>
The mask image generation unit 231 generates a reference mask image corresponding to the reference strip and a target mask image corresponding to the target strip. Some examples of mask images will be described below, but the invention is not limited thereto.

The contour shape of the mask image is, for example, rectangular, and typically square. The shape of the reference mask image and the shape of the target mask image may be the same. Further, the dimensions of the reference mask image and the dimensions of the target mask image may be the same.

The range of pixel values of the mask image is set to be included in the closed interval [0, 1], for example. Typically, the mask image may be a binary image with pixel values of 0 or 1. As a specific example, the mask image is a binary image in which pixels in a region corresponding to the domain of the strip have a value of 1 and other pixels have a value of 0. In other words, as shown in equation (20) of Non-Patent Document 1, the pixel values of an exemplary mask image have a value of 1 in the image area of the corresponding strip and a value of 0 in other areas.

<Range adjustment section 232>
The range adjustment unit 232 adjusts the pixel value range of the reference strip and the pixel value range of the target strip based on the pixel value range of the reference mask image and the pixel value range of the target mask image. It is composed of Typically, the range adjustment unit 232 adjusts the pixel value range of the reference strip and the pixel value range of the target strip so as to reduce the difference between the pixel value range of the strip and the pixel value range of the mask image. Adjust the range.

Generally, the range adjustment unit 232 relatively adjusts the pixel value range of the reference strip, the pixel value range of the target strip, and the pixel value range of the mask image. In a typical example, unlike the reference mask image applied to the reference strip and the target mask image applied to the target strip, the range adjustment unit 232 adjusts the range of pixel values of the reference strip and the pixel value of the target strip. , the pixel value range of the reference mask image, and the pixel value range of the target strip are relatively adjusted.

The range adjustment unit 232 adjusts the pixel value range of the reference strip and the pixel value range of the reference mask image so as to reduce the pixel value range of the reference strip and the pixel value range of the reference mask image, and , configured to relatively adjust the pixel value range of the target strip and the pixel value range of the target strip so as to reduce the pixel value range of the target strip and the pixel value range of the target mask image. Ru.

The range adjustment unit 232 adjusts the pixel value range of the strip and the pixel value range of the mask image so that the pixel value range of one of the strip and the mask image matches the pixel value range of the other. may be configured. For example, the range adjustment unit 232 adjusts the pixel value range of the reference strip and the pixel value range of the reference mask image so that the pixel value range of the reference strip and the pixel value range of the reference mask image match. , and relatively adjust the pixel value range of the target strip and the pixel value range of the target strip so that the pixel value range of the target strip matches the pixel value range of the target mask image. may be configured.

For example, the range adjustment unit 232 is configured to normalize the range of pixel values of the strip according to the range of pixel values of the mask image. Some examples of this normalization (normalization) will be described below, but the invention is not limited to these.

A first example of normalization will be explained. When the range of pixel values of the mask image is included in the closed interval [0, 1], the range adjustment unit 232 divides the value of each pixel in the strip by the maximum pixel value in this strip. In this example, the range adjustment unit 232 first compares the values of all pixels in the strip, identifies the maximum value (maximum pixel value), and divides the value of each pixel in this strip by the maximum pixel value. Thereby, the range of pixel values of the strip is matched to the same closed interval [0, 1] as the range of pixel values of the mask image.

A second example of normalization will be explained. When the range of pixel values of the mask image is included in the closed interval [0, 1], the range adjustment unit 232 divides the value of each pixel of the strip by the maximum value of the range of pixel values of this strip. The range of pixel values of the strip is set in advance, and the range adjustment unit 232 divides the value of each pixel of the strip by the upper limit value (maximum value) of this range. Also in this example, the range of pixel values of the strip coincides with the same closed interval [0, 1] as the range of pixel values of the mask image.

Through the processing executed by the range adjustment unit 232, the difference between the absolute value of the pixel value of the strip and the absolute value of the pixel value of the mask can be reduced, and the effect of the mask can be ignored in the correlation calculation using the strip and the mask. Therefore, it becomes possible to accurately calculate the correlation coefficient. In particular, even when single-precision floating-point type (float type) arithmetic is employed, it is possible to eliminate (reduce) the influence of rounding errors caused by a small number of significant digits.

In this specification, we will mainly explain an example in which only the pixel value range of the strip is changed, but it is also possible to change only the pixel value range of the mask image, or the pixel value range of the strip and the pixel value of the mask image. Both the range and the range may be changed.

<Synthetic image generation unit 233>
Regarding the strip and mask image to which the pixel value range adjustment by the range adjustment unit 232 has been applied, the composite image generation unit 233 combines the mask image with each of the reference strip and the target strip to generate two composite images. Typically, the composite image generation unit 233 generates a reference composite image by combining a reference mask image with a reference strip, and generates a target composite image by combining a target mask image with a target strip.

The process of combining the strip and the mask image may be performed in the same manner as in Non-Patent Document 1. However, unlike the method of Non-Patent Document 1, in this embodiment, the difference between the pixel value range of the strip and the pixel value range of the mask image is adjusted to be small. Typically, the pixel value ranges of the reference strip and the target strip are normalized to match the pixel value ranges of the reference mask image and the target mask image.

For example, in the same manner as Equation (19) in Non-Patent Document 1, the composite image generation unit 233 is configured to embed a strip in which the pixel value range has been normalized into an image having the same dimensions and the same shape as the mask image. It's fine.

Further, the composite image generation unit 233 generates a composite image of the embedded image of the strip and the mask image. This composite image corresponds to "f'(r)m _f (r)" (page 1800, second line, etc.) in Non-Patent Document 1, but as mentioned above, the value of the embedded image of the strip is This is different from that of equation (19).

<Cross correlation function calculation unit 234>
The cross-correlation function calculation unit 234 calculates a plurality of cross-correlation functions based on the two composite images generated by the composite image generation unit 233. Calculation of the cross-correlation function is performed in the same manner as in Non-Patent Document 1. For example, the cross-correlation function calculation unit 234 calculates the formula ( The six cross-correlation functions (image cross-correlation) included in 33) are calculated.

<Correlation coefficient calculation unit 235>
The correlation coefficient calculation unit 235 calculates a correlation coefficient based on the plurality of cross-correlation functions calculated by the cross-correlation function calculation unit 234. This calculation follows Equation (33) of Non-Patent Document 1.

<Registration section 236>
The registration unit 236 performs registration in the xy direction (lateral direction) based on the correlation coefficient calculated by the correlation coefficient calculation unit 235. This registration corresponds to "rough lateral motion correction" (page 1787) of Non-Patent Document 1.

Furthermore, the registration unit 236 uses the "fine lateral motion correction" described in Non-Patent Document 1 to remove small motion artifacts in the lateral direction caused by eye movements such as slow drift and tremor. (Page 1789).

In addition, the registration unit 236 performs "axial motion correction (rough axial motion correction and/or fine axial motion correction") described in Non-Patent Document 1 in order to remove motion artifacts in the z direction (axial direction). (pages 1789 to 1790).

<Merge processing unit 237>
The merge processing unit 237 constructs a merged image of the reference strip and the target strip whose relative positions have been adjusted by the registration unit 236. This process is also executed in the same manner as the method described in Non-Patent Document 1.

As described above, the data processing unit 230 sequentially executes the above-described series of processes on the plurality of strips constructed by the image construction unit 220 in accordance with the ordering according to the size. As a result, an image with motion artifacts corrected can be obtained from the plurality of strips constructed by the image construction unit 220. The obtained image typically represents the entire area to which the Lissajous scan has been applied.

<User interface 240>
User interface 240 includes a display section 241 and an operation section 242. The display section 241 includes the display device 3. The operation unit 242 includes various operation devices and input devices. The user interface 240 may include a device such as a touch panel that has a display function and an operation function integrated. It is also possible to construct embodiments that do not include at least a portion of user interface 240. For example, the display device may be an external device connected to the ophthalmological apparatus 1.

<motion>
The operation of the ophthalmologic apparatus 1 will be explained. An example of the operation of the ophthalmologic apparatus 1 is shown in FIG. The same preparatory processes as before, such as entering the patient ID, setting the scan mode (specifying Lissajous scan), presenting a fixation target, alignment, focus adjustment, and OCT optical path length adjustment, have already been performed. do.

(S1: Lissage scan)
Upon receiving a predetermined scan start trigger signal, the scan control unit 2111 starts applying an OCT scan (Lissajous scan) to the eye E (fundus Ef) to be examined.

The scan start trigger signal is generated, for example, in response to completion of a predetermined preparatory operation (alignment, focus adjustment, OCT optical path length adjustment, etc.), or in response to a scan start instruction operation performed using the operation unit 242. is generated in response to.

The scan control unit 2111 applies the Lissajous scan to the fundus Ef by controlling the optical scanner 44, the OCT unit 100, etc. based on the scanning protocol 2121 (protocol compatible with Lissajous scan). The data collected by the Lissajous scan is sent to the image construction unit 220.

(S2: Build multiple strips)
The image construction unit 220 constructs a plurality of strips based on the data collected in step S1 in the manner described above. The constructed plurality of strips are sent to the data processing unit 230.

(S3: Set reference strip and target strip)
The data processing unit 230 orders the plurality of strips constructed in step S2 according to dimensions (area, etc.). In this example, it is assumed that the number of the plurality of strips constructed in step S2 is N (N is an integer of 2 or more). Further, these N strips are called a first strip, a second strip, . . . , an N-th strip according to the order specified by the ordering. Further, any strip among the first to Nth strips may be called an nth strip (n=1, 2, . . . , N). Thus, the first strip is the largest strip, the second strip is the second largest strip, the nth strip is the nth largest strip, and the Nth strip is the smallest strip. be.

In this example, the data processing unit 230 sets the first strip as the initial reference strip and sets the second strip as the initial target strip.

Note that the reference strip and the target strip correspond to arbitrary shaped images f(r) and g(r) in Appendix A of Non-Patent Document 1, respectively.

(S4: Set reference mask image and target mask image)
The mask image generation unit 231 generates a mask image (reference mask image) corresponding to the reference strip and a target mask image corresponding to the target strip.

At this stage, the mask image generation unit 231 generates two mask images corresponding to the first strip and the second strip, respectively. The data processing unit 230 sets the first mask image corresponding to the first strip as the initial reference mask image, and sets the second mask image corresponding to the second strip as the initial target mask image.

Note that the reference mask image and the target mask image correspond to rectangular shaped binary image masks m _f (r) and m _g (r) in Appendix A of Non-Patent Document 1, respectively.

(S5: Normalize the reference strip and target strip)
The range adjustment unit 232 normalizes the reference strip and target strip set in step S3 in the manner described above. Strip normalization is performed according to the range of pixel values of the mask image. At this stage, the range adjustment unit 232 applies normalization to the first strip (reference strip) and the second strip (target strip). Note that the process executed by the range adjustment unit 232 is not limited to normalization, and may be any of the range adjustment processes described above or a process similar thereto.

In this example, it is assumed that the range of pixel values of each mask image is included in the closed interval [0, 1]. In particular, each mask image in this example is a binary image in which pixel values in a region corresponding to the domain of the corresponding strip are set to 1, and values of other pixels are set to 0.

For example, the range adjustment unit 232 divides the value of each pixel of the strip by the maximum pixel value in this strip, or divides the value of each pixel of the strip by the maximum of the pixel value range (gradation range) of this strip. Normalize this strip by dividing by the value.

Furthermore, the data processing unit 230 (composite image generation unit 233) embeds the strip whose pixel value range has been normalized into an image having the same size and shape as the corresponding mask image. The embedded image of the normalized first strip f(r) is an image similar to the rectangular shaped image f′(r) in Appendix A of Non-Patent Document 1, but the embedded image of the first strip The absolute value (|f(r)|) of the range within the image area of f(r) is normalized to 1 or less. This embedded image is also expressed as f'(r).

Similarly, the embedded image of the normalized second strip g(r) is an image similar to the rectangular image g'(r), but the range of values within the image area of the second strip g(r) is The absolute value of (|g(r)|) is normalized to 1 or less. This embedded image is also expressed as g'(r).

(S6: Generate reference composite image and target composite image)
The composite image generation unit 233 generates a composite image of the normalized strip embedded image and the corresponding mask image.

At this stage, the composite image generation unit 233 generates a composite image of the normalized embedded image of the first strip (reference strip) and the first mask image (reference mask image), and A composite image of the embedded image of the converted second strip (target strip) and the second mask image (target mask image) is generated. The former composite image is called a reference composite image, and the latter is called a target composite image.

Here, the reference composite image and the target composite image are combinations of two rectangle shaped images f'(r)m _f (r) and g in Appendix A of Non-Patent Document 1, respectively. '(r)m _g (r). However, as described above, the absolute value (|f(r)|) of the range in the image area of the first strip f(r) is less than or equal to 1, and the image of the second strip g(r) The absolute value (|g(r)|) of the range within the area is 1 or less.

Image 311 in FIG. 7A is an example of the embedded image f'(r) of the normalized first strip f(r), and image 321 is an example of the first mask image. By combining these two images 311 and 321, a reference composite image f'(r)m _f (r) is obtained.

Similarly, image 312 of FIG. 7B is an example of an embedded image g'(r) of the normalized second strip f(r), and image 322 is an example of a second mask image. By combining these two images 312 and 322, a reference composite image g'(r)m _g (r) is obtained.

(S7: Calculate multiple cross-correlation functions)
The cross-correlation function calculation unit 234 calculates a plurality of cross-correlation functions based on the reference composite image and the target composite image generated in step S6. In this example, the cross-correlation function calculation unit 234 calculates six cross-correlation functions included in equation (33) of Non-Patent Document 1 based on the reference composite image and the target composite image generated in step S6. .

(S8: Calculate correlation coefficient)
The correlation coefficient calculation unit 235 calculates a correlation coefficient based on the plurality of cross-correlation functions calculated in step S7. In this example, the correlation coefficient calculation unit 235 calculates the correlation coefficient (ρ(r')) from the plurality of cross-correlation functions calculated in step S7 according to equation (33) of Non-Patent Document 1.

(S9: Registration between reference strip and target strip)
The registration unit 236 applies registration in the x and y directions (rough lateral motion correction) to the reference strip and the target strip based on the correlation coefficient calculated in step S8. At this stage, registration between the first strip f(r) and the second strip is performed.

In this example, the relative shift amount (Δx, Δy) between the reference strip and the target strip is determined by detecting the peak of the correlation coefficient ρ(r') calculated in step S8.

Further, the registration unit 236 applies registration such as the aforementioned "fine lateral motion correction" and "axial motion correction" to the reference strip and the target strip.

(S10: Build a merged image of the reference strip and target strip)
The merge processing unit 237 constructs a merged image of the reference strip and the target strip whose relative positions have been adjusted by the registration in step S9. Image 330 of FIG. 7C shows an example of a merged image of the first strip and the second strip.

(S11: Have all strips been processed?)
A series of processes from steps S3 to S10 are sequentially performed on all N strips obtained in step S2 in the above-described order.

If N is 3 or more and a merged image of the first strip and the second strip is created in step S10 (S11: No), the process returns to step S3. In step S3, the merged image of the first strip and the second strip is set as a new reference strip, and the third strip is set as a new target strip. By executing the processes of steps S4 to S10 based on the new reference strip and the new target strip, a merged image of the new reference strip and the new target strip is obtained. This new merged image is a merged image of the first to third strips. By sequentially applying such a series of processing to the N strips, all merged images of the N strips are obtained (S11: Yes).

(S12: Save the final merged image)
The merged image finally obtained through the repeated processing described above is an image in which motion artifacts have been corrected, and is an image that expresses the entire application range of the Lissajous scan in step S1. The main control unit 211 stores this final merged image in the storage unit 212 (and/or other storage device).

The data processing unit 230 utilizes the registration results based on the N strips to process the data (three-dimensional data) collected in step S1 and/or the image construction unit 220 constructs the data (three-dimensional data) from this three-dimensional data. Registration of three-dimensional image data can be performed. This registration includes a process of converting the Lissajous scan definition coordinate system into a three-dimensional orthogonal coordinate system (xyz coordinate system). This coordinate transformation corresponds to "remapping" in the method described in Non-Patent Document 1. The main control unit 211 can store the three-dimensional image data obtained in this way in the storage unit 212 (and/or other storage device) together with or instead of the final merged image ( end).

The main control unit 211 can display the final merged image on the display unit 241 (and/or other display device). Further, the data processing unit 230 can create an arbitrary rendered image from the three-dimensional image data constructed by the coordinate transformation described above. The main control unit 211 can display this rendered image on the display unit 241 (and/or other display device).

Here, an implementation example of the processing in steps S7 to S10 will be described. First, f'(r)m _f (r), g'(r)m _g (r), m _f (r), m _g (r), (f'(r) m _f (r)) ² , and (g'(r)m _g (r)) Set the imaginary part of each of ² to 0, leaving only the real part.

Next, f'(r)m _f (r), g'(r)m _g (r), m _f (r), m _g (r), (f'(r) m _f (r)) ² , and (g′(r)m _g (r)) ^2. Apply a fast Fourier transform (FFT) to the real parts of each of .

Next, six cross-correlation functions shown in step S7 of FIG. 6 are calculated based on the function group obtained by these fast Fourier transforms.

Next, an inverse fast Fourier transform (IFFT) is applied to each of the six calculated cross-correlation functions.

Next, the correlation coefficient ρ(r') of equation (33) of Non-Patent Document 1 is calculated.

Then, by specifying the peak position of the correlation coefficient ρ(r'), the relative shift amount (Δx, Δy) between the reference strip f(r) and the target strip g(r) is calculated. Registration and merging processing between the reference strip f(r) and the target strip g(r) is performed based on the shift amount.

By sequentially applying such a series of processing to the N strips, all merged images of the N strips are obtained.

<Action, effect, etc.>
Certain features of exemplary embodiments will be described, and some of the operations and advantages provided by them will be described.

The ophthalmic device 1 according to the exemplary embodiments described above provides several aspects of a scanning imaging device. In addition, the ophthalmological apparatus 1 according to the exemplary embodiments described above may include some aspects of the method of controlling a scanning imaging device, some aspects of the image processing device, and some aspects of the method of controlling the image processing device. Some aspects of image processing methods, and some aspects of scanning imaging methods are provided.

Note that some aspects of the method of controlling the scanning imaging device, some aspects of the image processing device, some aspects of the method of controlling the image processing device, some aspects of the image processing method, and What has been described in any of the aspects of the scanning imaging method may be applied to the ophthalmological device 1 (more generally any aspect of the scanning imaging device) according to the exemplary embodiments described above. It is.

Now, the scanning imaging apparatus (ophthalmological apparatus 1) according to the exemplary embodiment includes a scanning section, an image construction section 220, a mask image generation section 231, a range adjustment section 232, a composite image generation section 233, It includes a cross-correlation function calculation section 234 (first calculation section), a correlation coefficient calculation section 235 (second calculation section), a registration section 236, and a merge processing section 237.

The scanning unit is configured to collect data by applying optical scanning to the sample (eye E, fundus Ef). This optical scanning is performed according to a two-dimensional pattern that includes a series of cycles.

In some exemplary aspects, any two cycles of the series of cycles included in the two-dimensional pattern of light scanning may intersect each other at at least one point.

Furthermore, in some exemplary embodiments, the optical scanning applied to the sample may be performed according to a two-dimensional pattern based on a preset scanning protocol based on a Lissajous function. An example of such an optical scan is the Lissajous scan in the exemplary embodiments described above.

In some exemplary embodiments, the scanning unit includes a deflector (light scanner 44) capable of deflecting light in a first direction (x direction) and a second direction (y direction) that are different from each other, and By repeating the change in the deflection direction along the direction in a first period and the change in the deflection direction along the second direction in a second period different from the first period, optical scanning according to a two-dimensional pattern is applied to the sample. It may be configured.

In the above exemplary embodiment, the scanning section includes the OCT unit 100 and the elements in the fundus camera unit 2 that constitute the measurement arm (retroreflector 41, OCT focusing lens 43, optical scanner 44, objective lens 22, etc.) and applying an OCT scan to the eye E to be examined to collect OCT data.

The image construction unit 220 is configured to construct the first image and the second image based on the data collected by the scanning unit. The combination of the reference strip and the target strip in the above exemplary embodiment is an example of the combination of the first image and the second image.

In some exemplary aspects, if any two cycles of the series of cycles included in the two-dimensional pattern of light scanning intersect each other at at least one point, the image construction unit 220 The first image is constructed based on the first data collected by optical scanning according to the first cycle group, and the first image is collected by optical scanning according to the second cycle group different from the first cycle group among the series of cycles. The second image may be configured to be constructed based on the second data.

For example, when a two-dimensional pattern of optical scanning follows a preset scanning protocol based on a Lissajous function, such as the Lissajous scan in the above exemplary embodiment, the image construction unit 220 constructing a first image based on first data collected by optical scanning according to a first cycle group of a series of cycles; and optical scanning according to a second cycle group different from the first cycle group among the series of cycles. The second image may be constructed based on the second data collected at the second image.

In the above exemplary embodiment, the front projection image of the first sub-volume (corresponding to the first cycle group) of the volume obtained from the optical scanning according to the series of cycles is the first The frontal projection image of the second sub-volume (corresponding to the second cycle group, which is different from the first cycle group) is set to one image (the first strip) and there is no intervening relatively large movement. 2 strips).

Regarding the first image and second image constructed by the image construction section 220, the mask image generation section 231 generates a first mask image according to the first image and a second mask image according to the second image. It is configured as follows. Note that the "first mask image" and "second mask image" referred to herein refer to the "first mask image" and "second mask image" corresponding to the order in which they are applied to the plurality of strips in the above exemplary embodiment. mask image”.

In some exemplary aspects, the range of pixel values of the mask images (first mask image, second mask image) may be included in the closed interval [0, 1].

Further, in some exemplary embodiments, the mask images (first mask image, second mask image) may be binary images with pixel values of 0 or 1.

For example, as in the exemplary embodiments above, the mask images (first mask image, second mask image) have values of 1 for pixels in the region corresponding to the domain of the image of the sample, and other The image may be a binary image in which the pixel value is 0.

In some example aspects, the first mask image and the second mask image may have the same dimensions and the same shape.

The range adjustment unit 232 adjusts the pixel value range of the first image and the pixel value range of the second image based on the pixel value range of the first mask image and the pixel value range of the second mask image. It is configured.

In some exemplary aspects, the range adjustment unit 232 adjusts the pixel value range of the sample image (first image, second image) and the pixel value range of the mask image (first mask image, second mask image). The configuration may be configured to reduce the difference between the ranges. For example, as in the above exemplary embodiment, the range adjustment unit 232 adjusts the range of pixel values of the sample images (first image, second image) and the mask images (first mask image, second mask image). may be configured to match the pixel value range of .

Furthermore, in some exemplary aspects, the range adjustment unit 232 adjusts the range of pixel values of one of the sample image (first image, second image) and the mask image (first mask image, second mask image). may be configured to match the other pixel value range. For example, as in the above exemplary embodiment, the range adjustment unit 232 adjusts the range of pixel values of the sample images (first image, second image) to mask images (first mask image, second mask image). may be configured to match the pixel value range of . That is, the range adjustment unit 232 matches the pixel value range of the sample image (first image, second image) with the pixel value range of the mask image (first mask image, second mask image). Additionally, the pixel value range of the sample images (first image, second image) may be changed.

In some exemplary aspects, the range adjustment unit 232 adjusts the pixels of the sample images (the first image, the second image) according to the range of pixel values of the mask images (the first mask image, the second mask image). It may be configured to normalize (normalize) the range of values.

For example, when the range of pixel values of the mask images (first mask image, second mask image) is included in the closed interval [0,1], the range adjustment unit 232 adjusts the pixel values of the sample images (first image, second mask image). The range of pixel values of the image may be normalized by dividing the value of each pixel of the second image by the maximum pixel value of the image.

In another example, when the range of pixel values of the mask images (first mask image, second mask image) is included in the closed interval [0, 1], the range adjustment unit 232 The pixel value range of the image may be normalized by dividing the value of each pixel of the image (first image, second image) by the maximum value of the pixel value range of the image.

The composite image generation unit 233 generates a first composite image by combining the first mask image with the first image of the sample, and generates a second composite image by combining the second mask image with the second image. It is configured as follows.

In some example aspects, the first mask image and the second mask image may have the same dimensions and the same shape. Further, the first mask image may be a binary image in which the value of a pixel in a region corresponding to the domain of the first image is 1 and the value of other pixels is 0, and the second mask image The image may be a binary image in which pixels in a region corresponding to the domain of the second image have a value of 1 and other pixels have values of 0.

In the above exemplary embodiment, the first mask image and the second mask image have the same dimensions and the same shape. The composite image generation unit 233 is configured to convert each of the arbitrarily shaped first image and arbitrarily shaped second image into embedded images having the same dimensions and the same shape as these mask images. Furthermore, the composite image generation unit 233 is configured to composite the embedded image of the first image and the first mask image, and to composite the embedded image of the second image and the second mask image.

Alternatively, in the above exemplary embodiment, the composite image generation unit 233 is configured to convert the arbitrarily shaped first image and the arbitrarily shaped second image into embedded images having the same dimensions and the same shape as each other. It's okay to stay. Further, the mask image generation unit 231 may be configured to generate a first mask image and a second mask image, respectively, having the same dimensions and the same shape as these sample images. Furthermore, the composite image generation unit 233 may be configured to composite the embedded image of the first image and the first mask image, and to composite the embedded image of the second image and the second mask image.

The cross-correlation function calculation unit 234 is configured to calculate a plurality of cross-correlation functions based on the first composite image and the second composite image generated by the composite image generation unit 233.

The correlation coefficient calculation unit 235 is configured to calculate a correlation coefficient based on the plurality of cross-correlation functions calculated by the cross-correlation function calculation unit 234.

The registration unit 236 is configured to perform registration between the first image and the second image based on the correlation coefficient calculated by the correlation coefficient calculation unit 235.

The merge processing unit 237 is configured to construct a merged image of the first image and the second image based on the result of the registration performed by the registration unit 236.

Similar to the above exemplary embodiments, in some exemplary aspects, the cross-correlation function calculation section 234 (first calculation section), the correlation coefficient calculation section 235 (second calculation section), the registration section 236 , and the merge processing unit 237, and the configuration thereof (hardware configuration, software configuration) may be the same as the method described in Non-Patent Document 1. However, this does not preclude the adoption of other methods.

Several aspects of the image processing method that can be provided by the ophthalmologic apparatus 1 according to the above exemplary embodiment will be described. In some example aspects, an image processing method may include the steps described below.

(Image preparation process)
A first image and a second image of the sample are provided. For example, a first image and a second image of a sample are received via a communication line or a recording medium. Alternatively, data is collected by applying optical scanning to the sample and a first image and a second image are constructed. Each of the first and second images is constructed from data collected by applying a scan to the sample that follows a two-dimensional pattern that includes a series of cycles. For example, each of the first and second images is constructed based on data collected by applying a scan to the sample according to a two-dimensional pattern based on a preset scanning protocol based on a Lissajous function.

(Mask preparation process)
A mask image is prepared. For example, a mask image is received via a communication line or a recording medium. Alternatively, a mask image is generated (based on the image of the sample). The range of pixel values of the mask image may be included in the closed interval [0, 1]. Typically, the mask image may be a binary image with pixel values of 0 or 1. For example, the mask image may be a binary image in which pixels in a region corresponding to the domain of the sample image have a value of 1 and other pixels have values of 0. The mask image may include a first mask image and a second mask image. The first mask image and the second mask image may have the same dimensions and the same shape. The first mask image may be a binary image in which the value of a pixel in a region corresponding to the domain of the first image of the sample is 1, and the value of other pixels is 0; The image may be a binary image in which pixels in a region corresponding to the domain of the second image of the sample have a value of 1 and other pixels have a value of 0.

(Image value range adjustment process)
The pixel value range of the first image, the pixel value range of the second image, and the pixel value range of the mask image are relatively adjusted. Relative adjustment of the pixel value range is typically done to reduce the difference between the pixel value range of the sample image (first image, second image) and the pixel value range of the mask image. It will be held on. For example, the relative adjustment of the pixel value range is performed so that the pixel value range of one of the sample image (first image, second image) and the mask image matches the pixel value range of the other. In some example aspects, the relative adjustment of the pixel value range normalizes the pixel value range of the sample images (first image, second image) according to the pixel value range of the mask image. It may include processing to do so. When the range of pixel values of the mask image is included in the closed interval [0, 1], normalization converts the value of each pixel of the sample image (first image, second image) to the maximum pixel value in the image. It may include processing to divide by. Alternatively, when the range of pixel values of the mask image is included in the closed interval [0, 1], normalization can be performed by changing the value of each pixel of the sample image (first image, second image) to the pixel value of the image. It may include a process of dividing by the maximum value of the value range.

(Synthetic image generation process)
After the image value range adjustment step, a mask image is combined with each of the first image and the second image to generate a first combined image and a second combined image. In the case where the mask image includes a first mask image and a second mask image, the composite image generation step includes a step of combining the first mask image with the first image to generate a first composite image, and a step of generating a first composite image by combining the first mask image with the first image. The method may include a step of composing the image with a second image to generate a second composite image. Here, the first mask image and the second mask image may have the same size and the same shape. Further, the first mask image may be a binary image in which the value of a pixel in a region corresponding to the domain of the first image is 1, and the value of other pixels is 0, and the second mask image The image may be a binary image in which pixels in a region corresponding to the domain of the second image have a value of 1 and other pixels have values of 0.

(Cross correlation function calculation process)
A plurality of cross-correlation functions are determined based on the first composite image and the second composite image generated in the composite image generation step. This step is typically performed according to the method described in Non-Patent Document 1, but is not limited thereto.

(Correlation coefficient calculation process)
A correlation coefficient is calculated based on the plurality of cross-correlation functions obtained in the cross-correlation function calculation step. This step is typically performed according to the method described in Non-Patent Document 1, but is not limited thereto.

(Registration process)
Registration between the first image and the second image of the sample is performed based on the correlation coefficient calculated in the correlation coefficient calculation step. This step is typically performed according to the method described in Non-Patent Document 1, but is not limited thereto.

In some example embodiments, the image processing method may further include a merged image construction step. In the merge image construction step, the first image and the second image are combined based on the registration result (for example, the relative shift amount between the first image and the second image of the sample) performed in the registration step. A merged image is constructed.

In some example aspects, the image processing method may further include any of the steps described in the example embodiments above.

Several aspects of the scanning imaging method that can be provided by the ophthalmological device 1 according to the exemplary embodiments described above will now be described. In some example aspects, a scanning imaging method may include the steps described below. Note that in some example aspects, the scanning imaging method includes any of the steps described in the example embodiments above and/or the example aspects of the image processing method described above. The method may further include any of the above steps.

(scanning process)
A light scan following a two-dimensional pattern comprising a series of cycles is applied to the sample and data is collected. Typically, any two cycles in the series intersect each other at at least one point. For example, optical scanning according to a two-dimensional pattern is performed based on a preset scanning protocol based on a Lissajous function.

(Image construction process)
A first image and a second image are constructed based on the data collected during the scanning process. If any two cycles of the series intersect each other at at least one point (for example, when light scanning is performed according to a two-dimensional pattern based on a preset scanning protocol based on a Lissajous function), then the series A first image may be constructed based on first data collected in an optical scan according to a first group of cycles of the series, and an optical scan according to a second group of cycles different from the first group of cycles of the series of cycles. A second image may be constructed based on the second data collected at.

(Image value range adjustment process)
The pixel value range of the first image, the pixel value range of the second image, and the pixel value range of the mask image are relatively adjusted.

(Synthetic image generation process)
After the image value range adjustment step, a mask image is combined with each of the first image and the second image to generate a first combined image and a second combined image.

(Cross correlation function calculation process)
A plurality of cross-correlation functions are obtained based on the first composite image and the second composite image generated in the composite image generation step.

(Correlation coefficient calculation process)
A correlation coefficient is calculated based on the plurality of cross-correlation functions obtained in the cross-correlation function calculation step.

(Registration process)
Registration between the first image and the second image of the sample is performed based on the correlation coefficient calculated in the correlation coefficient calculation step.

(Merge image construction process)
A merged image of the first and second images of the sample is constructed based on the registration results.

In some example embodiments, the scanning imaging method may include the following additional steps.

In a first additional step, a third image is constructed based on the data collected from the sample in the scanning step. In the above exemplary embodiment, a third strip is constructed.

In the second additional step, the pixel value range of the third image, the pixel value range of the merged image constructed in the merged image construction step, and the pixel value range of the mask image are relatively adjusted. Ru. In the above exemplary embodiment, for a third strip, a mask image corresponding to the third strip, a merged image of the first strip and the second strip, and a mask image corresponding to this merged image. , pixel value range adjustment is applied.

In the third additional step, the mask image is combined with each of the third image and the merged image to generate a third combined image and a fourth combined image. In the exemplary embodiments described above, a composite image of the third strip and its corresponding mask image is generated, and a merged image of the first strip and the second strip and its corresponding mask image is generated. A composite image is generated.

In a fourth additional step, a plurality of cross-correlation functions are determined based on the third composite image and the fourth composite image. In the above exemplary embodiment, a plurality of cross-correlation functions are generated based on the composite image of the third strip and the corresponding mask image, and the composite image of the merged image and the corresponding mask image. is required.

In the fifth additional step, a correlation coefficient is calculated based on the plurality of cross-correlation functions determined in the fourth additional step. In the exemplary embodiment described above, a correlation coefficient is calculated based on a plurality of cross-correlation functions determined using a composite image based on the third strip and a composite image based on the merged image.

In the sixth additional step, the third image and the fourth image are registered based on the correlation coefficient calculated in the fifth additional step. In the above exemplary embodiment, based on the correlation coefficient calculated using the composite image based on the third strip and the composite image based on the merged image, this merged image (first strip and second strip (merged image) and the third strip.

In the seventh additional step, the third image and the merged image are merged based on the result of the registration performed in the sixth additional step. In the above exemplary embodiment, based on the registration result (relative shift amount) of the merged image and the third strip, this merged image (merged image of the first strip and the second strip) The third strip is merged with.

Note that although the processing for the merged image of the first strip and the second strip and the third strip has been described as an example, generally, as described in the above exemplary embodiment, the processing for the first to second strips is Processing for the merged image of the n-th strip and the n+1-th strip is also performed in the same manner (n=2, 3, . . . , N-1).

Some aspects of the image processing device that can be provided by the ophthalmological device 1 according to the above exemplary embodiment will be described. In some example aspects, an image processing device may include the elements described below. Note that in some exemplary aspects, the image processing apparatus performs any of the elements described in the exemplary embodiments above, or the steps described in the exemplary aspects of the image processing method above. and any of the elements configured to perform the steps described in the exemplary embodiments of the scanning imaging methods above. You can stay there.

An image processing device according to an exemplary embodiment includes a storage unit, a mask image generation unit, a range adjustment unit, a composite image generation unit, a first calculation unit, a second calculation unit, and a registration unit. .

The storage unit stores a first image and a second image of the sample. In the exemplary embodiment described above, this storage corresponds to storage 212.

The mask image generation unit is configured to generate a first mask image corresponding to the first image and a second mask image corresponding to the second image. In the above exemplary embodiment, the mask image generator corresponds to the mask image generator 231.

The range adjustment unit is configured to adjust the pixel value range of the first image and the pixel value range of the second image based on the pixel value range of the first mask image and the pixel value range of the second mask image. be done. In the exemplary embodiment described above, the range adjuster corresponds to the range adjuster 232.

The composite image generation unit is configured to generate a first composite image by combining the first mask image with the first image, and generate a second composite image by combining the second mask image with the second image. be done. In the above exemplary embodiment, the composite image generator corresponds to the composite image generator 233.

The first calculation unit is configured to calculate a plurality of cross-correlation functions based on the first composite image and the second composite image. In the above exemplary embodiment, the first calculation unit corresponds to the cross-correlation function calculation unit 234.

The second calculation unit is configured to calculate a correlation coefficient based on a plurality of cross-correlation functions. In the above exemplary embodiment, the second calculation unit corresponds to the correlation coefficient calculation unit 235.

The registration unit is configured to perform registration between the first image and the second image based on the correlation coefficient. In the exemplary embodiment described above, the registration section corresponds to registration section 236.

In some exemplary embodiments, the image processing device may further include a merge processing unit. The merge processing unit is configured to construct a merged image of the first image and the second image based on the registration result.

In some exemplary embodiments, the image processing device may further include a data reception unit and an image construction unit.

The data accepting unit is configured to accept data collected from the sample by optical scanning according to a two-dimensional pattern including a series of cycles. The data reception unit may be configured to receive data via a communication line or a recording medium, for example. In the exemplary embodiment described above, the data reception unit corresponds to a communication device provided in the ophthalmological apparatus 1.

The image construction section constructs the first image and the second image based on the data received by the data reception section. In the exemplary embodiments described above, the image construction unit corresponds to the image construction unit 220. The first image and second image constructed by the image construction section are stored in the storage section.

In some example aspects, the data acceptor is configured to accept first and second images constructed based on data collected from the sample by light scanning according to a two-dimensional pattern that includes a series of cycles. It's fine. In this case, the first image and second image accepted by the data accepting unit are stored in the storage unit.

In some example aspects, the data acceptor may be configured to accept three-dimensional image data constructed based on data collected from the sample by optical scanning according to a two-dimensional pattern that includes a series of cycles. In this case, the image construction section is configured to construct the first image and the second image based on the three-dimensional image data received by the data reception section. The first image and second image constructed by the image construction section are stored in the storage section.

Several aspects of the image processing device control method that can be provided by the ophthalmological device 1 according to the above exemplary embodiment will be described. In some exemplary embodiments, a method for controlling an image processing device may include the steps described below. Note that in some exemplary embodiments, the method for controlling an image processing apparatus includes any of the steps described in the exemplary embodiments described above, or the steps described in the exemplary embodiments of the image processing method described above. any of the steps described in the exemplary embodiments of the scanning imaging method above, and any of the steps performed by the elements described in the exemplary embodiments of the image processing apparatus above; It may further contain any one of the following.

An exemplary aspect is a method of controlling an image processing device including a storage unit storing a first image and a second image of a sample, and a processor, the processor being controlled to perform the following steps. conduct. Note that in the above exemplary embodiment, the storage unit corresponds to the storage unit 212, and the processor corresponds to at least the data processing unit 230.

The processor is controlled to generate a first mask image corresponding to the first image and a second mask image corresponding to the second image. For example, in the exemplary embodiment described above, the data processing unit 230 is controlled to operate as the mask image generation unit 231.

The processor also controls to adjust the pixel value range of the first image and the pixel value range of the second image based on the pixel value range of the first mask image and the pixel value range of the second mask image. be done. For example, in the exemplary embodiment described above, the data processing unit 230 is controlled to operate as the range adjustment unit 232.

The processor is also controlled to generate a first composite image by combining the first mask image with the first image, and generate a second composite image by combining the second mask image with the second image. Ru. For example, in the exemplary embodiment described above, the data processing unit 230 is controlled to operate as a composite image generation unit 233.

The processor is also controlled to obtain a plurality of cross-correlation functions based on the first composite image and the second composite image. For example, in the exemplary embodiment described above, the data processing section 230 is controlled to operate as the cross-correlation function calculation section 234.

The processor is also controlled to calculate a correlation coefficient based on a plurality of cross-correlation functions. For example, in the exemplary embodiment described above, the data processing section 230 is controlled to operate as the correlation coefficient calculation section 235.

Further, the processor is controlled to perform registration between the first image and the second image based on the correlation coefficient. For example, in the exemplary embodiment described above, the data processing unit 230 is controlled to operate as the registration unit 236.

In some example embodiments, the processor may be controlled to construct a merged image of the first image and the second image based on the results of the registration. For example, in the exemplary embodiment described above, the data processing unit 230 is controlled to operate as the merge processing unit 237.

Several aspects of the method of controlling a scanning imaging device that can be provided by the ophthalmological device 1 according to the exemplary embodiment described above will be described. In some example aspects, a method of controlling a scanning imaging device may include the steps described below. Note that in some exemplary aspects, the method for controlling a scanning imaging device includes any of the steps described in the exemplary embodiments above, or any of the exemplary aspects of the image processing method described above. any of the steps described in the exemplary embodiments of the scanning imaging method above; any of the steps performed by the elements described in the exemplary embodiments of the image processing apparatus above; , any of the steps described in the exemplary embodiments of the image processing apparatus above.

An exemplary aspect is a method of controlling a scanning imaging device including a scanning unit that applies optical scanning to a sample to collect data, and a processor, the scanning unit and the processor performing the following steps. control. In the above exemplary embodiment, the scanning section includes the OCT unit 100 and the elements in the fundus camera unit 2 that constitute the measurement arm (retroreflector 41, OCT focusing lens 43, optical scanner 44, objective lens 22). etc.). Further, the processor corresponds to the image construction section 220 and the data processing section 230.

The scanning unit is controlled to collect data by applying a light scan to the sample that follows a two-dimensional pattern that includes a series of cycles.

The processor (image construction unit 220 in the above exemplary embodiment) is also controlled to construct the first image and the second image based on the data collected by the scanning unit.

Further, the processor (data processing unit 230 in the above exemplary embodiment) is controlled to generate a first mask image corresponding to the first image and a second mask image corresponding to the second image.

The processor (data processing unit 230 in the above exemplary embodiment) may also determine the range of pixel values of the first image based on the range of pixel values of the first mask image and the range of pixel values of the second mask image. Control is performed to adjust the range of pixel values of the second image.

The processor (data processing unit 230 in the above exemplary embodiment) also combines the first mask image with the first image to generate a first composite image, and combines the second mask image with the second image. The images are controlled to be combined to generate a second combined image.

The processor (data processing unit 230 in the exemplary embodiments described above) is also controlled to determine a plurality of cross-correlation functions based on the first composite image and the second composite image.

The processor (data processing unit 230 in the above exemplary embodiment) is also controlled to calculate a correlation coefficient based on a plurality of cross-correlation functions.

The processor (data processing unit 230 in the above exemplary embodiment) is also controlled to register the first image and the second image based on the correlation coefficient.

The processor (data processing unit 230 in the above exemplary embodiment) is also controlled to construct a merged image of the first image and the second image based on the registration result.

In some exemplary embodiments, a program causes a computer to execute any aspect of a method for controlling a scanning imaging device, a program causes a computer to execute any aspect of a method for controlling an image processing device, and image processing. It is possible to provide a program that causes a computer to perform any aspect of the method or a program that causes a computer to perform any aspect of the scanning imaging method.

Furthermore, it is possible to create a computer-readable non-temporary recording medium that records such a program. This non-transitory recording medium may be in any form, examples of which include magnetic disks, optical disks, magneto-optical disks, and semiconductor memory.

According to some exemplary aspects described above, in image registration using a mask image, the pixel value range of the sample image and the pixel value range of the mask image are relatively adjusted, and then , it is possible to perform processing for registration.

This makes it possible to reduce the difference between the pixel value range of the sample image and the pixel value range of the mask image. For example, it is possible to (approximately) match the size of the pixel values of the sample image and the size of the pixel values of the mask image.

Therefore, it is possible to solve the conventional problem that the effect of the mask image is ignored in a correlation calculation using a sample image and a mask image, and it becomes possible to obtain a correct correlation coefficient.

In particular, even when single-precision floating-point type (float type) arithmetic is used to calculate the correlation coefficient between sample images, the effects of rounding errors due to the small number of significant digits can be eliminated. It is possible to make it smaller.

Furthermore, in an algorithm that calculates a correlation coefficient from a plurality of cross-correlation functions, there is a possibility that the error in the final correlation coefficient becomes large due to the superposition of errors caused by the individual cross-correlation functions. Regarding such problems, according to exemplary embodiments, it is possible to eliminate or reduce errors caused by individual cross-correlation functions, thereby making it possible to obtain highly accurate correlation coefficients.

As such, several exemplary aspects described above contribute to improving the accuracy of image registration using mask images.

This disclosure is merely illustrative of some embodiments and is not intended to be a limitation of the invention. Those who wish to carry out this invention can make arbitrary modifications (omissions, substitutions, additions, etc.) within the scope of the gist of this invention.

1 Ophthalmology apparatus 44 Optical scanner 100 OCT unit 211 Main control section 2111 Scanning control section 212 Storage section 2121 Scanning protocol 220 Image construction section 230 Data processing section 231 Mask image generation section 232 Range adjustment section 233 Composite image generation section 234 Cross-correlation function calculation Section 235 Correlation coefficient calculation section 236 Registration section 237 Merge processing section

Claims (24)

receiving a first image and a second image of the sample via a communication line or a recording medium, or constructing the first image and the second image based on data collected by applying optical scanning to the sample;
Accepting a mask image via a communication line or a recording medium, or generating a mask image based on a sample image,
The pixel value range of the first image and the pixel value range of the mask image are reduced so as to reduce the difference between the pixel value range of the first image, the pixel value range of the second image, and the pixel value range of the mask image. relatively adjusting the pixel value range of the second image and the pixel value range of the mask image;
compositing the mask image with each of the first image and the second image to generate a first composite image and a second composite image;
determining a plurality of cross-correlation functions based on the first composite image and the second composite image;
Calculating a correlation coefficient based on the plurality of cross-correlation functions,
registering the first image and the second image based on the correlation coefficient;
Image processing method. matching the pixel value range of one of the sample image and the mask image to the pixel value range of the other;
The image processing method according to claim 1 . normalizing the pixel value range of the sample image according to the pixel value range of the mask image;
The image processing method according to claim 2 . The range of pixel values of the mask image is included in the closed interval [0, 1],
dividing the value of each pixel of the sample image by the maximum pixel value in the image;
The image processing method according to claim 3 . The range of pixel values of the mask image is included in the closed interval [0, 1],
dividing the value of each pixel of the sample image by the maximum value of the range of pixel values of the image;
The image processing method according to claim 3 . The mask image is a binary image with a pixel value of 0 or 1.
The image processing method according to any one of claims 1 to 5 . The mask image is a binary image in which the value of a pixel in a region corresponding to the domain of the sample image is 1, and the value of other pixels is 0.
The image processing method according to claim 6 . The mask image includes a first mask image and a second mask image,
compositing the first mask image with the first image to generate the first composite image;
compositing the second mask image with the second image to generate the second composite image;
The image processing method according to any one of claims 1 to 7 . the first mask image and the second mask image have the same dimensions and the same shape;
The image processing method according to claim 8 . The first mask image is a binary image in which the value of a pixel in a region corresponding to the domain of the first image is 1, and the value of other pixels is 0,
The second mask image is a binary image in which the value of a pixel in a region corresponding to the domain of the second image is 1, and the value of other pixels is 0.
The image processing method according to claim 8 or 9 . each of the first image and the second image is constructed based on data collected by applying a scan to the sample according to a two-dimensional pattern comprising a series of cycles;
The image processing method according to any one of claims 1 to 10 . Each of the first image and the second image is constructed based on data collected by applying a scan to the sample according to a two-dimensional pattern based on a preset scanning protocol based on a Lissajous function.
The image processing method according to claim 11 . collecting data by applying an optical scan to the sample that follows a two-dimensional pattern comprising a series of cycles;
constructing a first image and a second image based on the data;
The pixel value range of the first image and the pixel value range of the mask image are reduced so as to reduce the difference between the pixel value range of the first image, the pixel value range of the second image, and the pixel value range of the mask image. relatively adjusting the pixel value range of the second image and the pixel value range of the mask image,
compositing the mask image with each of the first image and the second image to generate a first composite image and a second composite image;
determining a plurality of cross-correlation functions based on the first composite image and the second composite image;
Calculating a correlation coefficient based on the plurality of cross-correlation functions,
registering the first image and the second image based on the correlation coefficient;
constructing a merged image of the first image and the second image based on the registration result;
Scanning imaging method. constructing a third image based on the data;
The pixel value range of the third image and the merged image are adjusted so as to reduce the difference between the pixel value range of the third image, the pixel value range of the merged image, and the pixel value range of the mask image. relatively adjust the pixel value range of and the pixel value range of the mask image ,
compositing the mask image with each of the third image and the merged image to generate a third composite image and a fourth composite image;
determining a plurality of cross-correlation functions based on the third composite image and the fourth composite image,
Calculate a correlation coefficient based on the plurality of cross-correlation functions,
registering the third image and the fourth image based on the correlation coefficient;
merging the third image and the merged image based on the registration result;
14. The scanning imaging method of claim 13 . any two cycles of the series of cycles intersect each other at at least one point;
constructing the first image based on first data collected by optical scanning according to a first cycle group of the series of cycles;
constructing the second image based on second data collected by optical scanning according to a second cycle group different from the first cycle group among the series of cycles;
The scanning imaging method according to claim 13 or 14 . performing optical scanning according to the two-dimensional pattern based on a preset scanning protocol based on a Lissajous function;
16. The scanning imaging method of claim 15 . a storage unit that stores a first image and a second image of the sample;
a mask image generation unit that generates a first mask image according to the first image and a second mask image according to the second image;
Based on the pixel value range of the first mask image and the pixel value range of the second mask image , determine the pixel value range of the first image, the pixel value range of the second image, and the first mask image. a range in which the pixel value range of the first image and the pixel value range of the second image are adjusted so as to reduce the difference between the pixel value range of the first image and the pixel value range of the second mask image; an adjustment section;
a composite image generation unit that generates a first composite image by combining the first mask image with the first image, and generates a second composite image by combining the second mask image with the second image; ,
a first calculation unit that calculates a plurality of cross-correlation functions based on the first composite image and the second composite image;
a second calculation unit that calculates a correlation coefficient based on the plurality of cross-correlation functions;
An image processing device, comprising: a registration unit that performs registration of the first image and the second image based on the correlation coefficient. a data reception unit that receives data collected from the sample by optical scanning according to a two-dimensional pattern including a series of cycles;
further comprising: an image construction unit that constructs the first image and the second image based on the data;
The image processing device according to claim 17 . A method for controlling an image processing device including a storage unit storing a first image and a second image of a sample, and a processor, the method comprising:
The processor,
generating a first mask image corresponding to the first image and a second mask image corresponding to the second image;
Based on the pixel value range of the first mask image and the pixel value range of the second mask image , determine the pixel value range of the first image, the pixel value range of the second image, and the first mask image. adjusting the pixel value range of the first image and the pixel value range of the second image so as to reduce the difference between the pixel value range of the first image and the pixel value range of the second mask image,
compositing the first mask image with the first image to generate a first composite image, and compositing the second mask image with the second image to generate a second composite image,
determining a plurality of cross-correlation functions based on the first composite image and the second composite image;
calculating a correlation coefficient based on the plurality of cross-correlation functions, and
performing control to register the first image and the second image based on the correlation coefficient;
A method for controlling an image processing device. a scanning unit that collects data by applying optical scanning to the sample according to a two-dimensional pattern including a series of cycles;
an image construction unit that constructs a first image and a second image based on the data;
a mask image generation unit that generates a first mask image according to the first image and a second mask image according to the second image;
Based on the pixel value range of the first mask image and the pixel value range of the second mask image , determine the pixel value range of the first image, the pixel value range of the second image, and the first mask image. a range in which the pixel value range of the first image and the pixel value range of the second image are adjusted so as to reduce the difference between the pixel value range of the first image and the pixel value range of the second mask image; an adjustment section;
a composite image generation unit that generates a first composite image by combining the first mask image with the first image, and generates a second composite image by combining the second mask image with the second image; ,
a first calculation unit that calculates a plurality of cross-correlation functions based on the first composite image and the second composite image;
a second calculation unit that calculates a correlation coefficient based on the plurality of cross-correlation functions;
a registration unit that registers the first image and the second image based on the correlation coefficient;
A scanning imaging apparatus, comprising: a merge processing unit that constructs a merged image of the first image and the second image based on the registration result. The scanning unit includes a deflector capable of deflecting light in a first direction and a second direction that are different from each other, and repeats a change in the deflection direction along the first direction in a first cycle while deflecting the light along the second direction. applying optical scanning to the sample according to the two-dimensional pattern by repeating changes in the deflection direction in a second period different from the first period;
21. The scanning imaging device of claim 20 . A method for controlling a scanning imaging device including a scanning unit that collects data by applying optical scanning to a sample, and a processor, the method comprising:
controlling the scanning unit to collect data by applying an optical scan to the sample according to a two-dimensional pattern including a series of cycles;
The processor,
constructing a first image and a second image based on the data;
generating a first mask image corresponding to the first image and a second mask image corresponding to the second image;
Based on the pixel value range of the first mask image and the pixel value range of the second mask image , determine the pixel value range of the first image, the pixel value range of the second image, and the first mask image. adjusting the pixel value range of the first image and the pixel value range of the second image so as to reduce the difference between the pixel value range of the first image and the pixel value range of the second mask image,
compositing the first mask image with the first image to generate a first composite image, and compositing the second mask image with the second image to generate a second composite image,
determining a plurality of cross-correlation functions based on the first composite image and the second composite image;
Calculating a correlation coefficient based on the plurality of cross-correlation functions,
registering the first image and the second image based on the correlation coefficient; and
controlling to construct a merged image of the first image and the second image based on the registration result;
A method for controlling a scanning imaging device. A program that causes a computer to execute the method according to any one of claims 1 to 16 , 19 , or 22 . A computer-readable non-transitory recording medium on which the program according to claim 23 is recorded.

Images