JP2021090631A - Image processing method, scan type imaging method, image processing apparatus, its control method, scan type imaging device, its control method, program, and recording medium - Google Patents

Abstract

To provide a method for improving accuracy of image registration using a mask.SOLUTION: In an image processing method of exemplary manner, first of all, a first image of a sample and a second image and a mask image are prepared. A pixel value range of the first image and the pixel value range of the second image and the pixel value range of the mask image are relatively adjusted. After this relative pixel value range adjustment, the mask image is synthesized in each of the first image and the second image, and the first synthetic image and the second synthetic image are generated, and thereby multiple cross correlation functions are found on the basis of these synthetic images. The multiple cross correlation functions being found are used for calculation of the correlation coefficient, and the registration with the first image and the second image is carried out on the basis of this correlation coefficient.SELECTED DRAWING: Figure 4B

Description

The present invention relates to an image processing method, a scanning imaging method, an image processing apparatus, a control method thereof, a scanning imaging apparatus, a control method thereof, a program, and a recording medium.

Scanning imaging is one of the imaging techniques. Scanning imaging is a technique in which a plurality of parts of a sample are sequentially irradiated with a beam to collect data, and an image of the sample is constructed from the collected data.

Optical coherence tomography (OCT) is known as a typical example of scanning imaging using light. OCT is a technique capable of imaging a light scattering medium with a resolution of a micrometer level or less, and is applied to medical imaging, nondestructive inspection, and the like. OCT is a technique based on low coherence interferometry, typically utilizing near-infrared light to ensure the depth of light scattering media into a sample.

For example, in ophthalmic diagnostic imaging, OCT devices are becoming more widespread, and not only two-dimensional imaging but also three-dimensional imaging, structural analysis, functional analysis, etc. are put into practical use and widely used as a powerful diagnostic tool. Has reached. Further, in the field of ophthalmology, scanning imaging other than OCT such as scanning laser ophthalmoscope (SLO) is also used. Scanning imaging using light (electromagnetic waves) in a wavelength band other than near-infrared light or ultrasonic waves is also known.

There are various scanning modes used in OCT and SLO, but the so-called "Lissajous scan" for the purpose of correcting motion artifacts has been attracting attention in recent years (for example, Patent Documents 1 to 4). See Non-Patent Documents 1 and 2).

In a typical resage scan, the measurement light is scanned at high speed so as to draw a loop (cycle) having a certain size, so that the difference in data acquisition time from one cycle can be substantially ignored. Also, since the alignment between cycles can be performed by referring to the intersecting regions of different cycles, it is possible to correct the artifacts caused by the movement of the sample. Focusing on these characteristics of Lissajous scan, we are trying to deal with artifacts caused by eye movements in the field of ophthalmology.

In the technique described in Non-Patent Document 1, the data collected by the resage scan is divided into a plurality of subvolumes in which relatively large movement does not intervene, and a front projection image (en face projection) of each subvolume is constructed. .. Such a front projection image is called a strip. By performing registration between these strips, an image with corrected motion artifacts can be obtained.

A characteristic of Lissajous scans is that the two strips overlap (intersect) at four points. In the image processing described in Non-Patent Document 1, the largest strip is adopted as the initial reference strip, and the strips are sequentially registered and synthesized in order of size with respect to the initial reference strip. Registration uses a cross-correlation function to determine the relative position between the reference strip and the other strips. Since the strips have an arbitrary shape, the correlation calculation of the overlap region between the strips is performed using a mask for treating each strip as an image having a specific shape (for example, a square shape) (Appendix of Non-Patent Document 1). See A).

Here, the strip is an intensity image of a predetermined gradation, and the mask is binary data. Therefore, the absolute value of the pixel value of the mask is always small (0 or 1), while the absolute value of the pixel value of the strip may be large. Due to this effect, the effect of the mask may be ignored in the correlation calculation using the strip and the mask, and the correct correlation coefficient may not be obtained. In particular, when a single-precision floating-point type (float type) operation is adopted to obtain the correlation coefficient between strips, this problem becomes prominent due to the influence of rounding error due to the small number of significant digits.

Japanese Unexamined Patent Publication No. 2016-17915 Japanese Unexamined Patent Publication No. 2018-68578 Japanese Unexamined Patent Publication No. 2018-140004 Japanese Unexamined Patent Publication No. 2018-140049

Yiwei Chen, Young-Joo Hong, Shuichi Makita, and Yoshiaki Yasuno, "Three-dimensional eye motion correction by Lissajous scan optical coherence tomography", Biomedical Optics EXPRESS, Vol. 8, No. 3, 1 Mar 2017, PP. 1783- 1802 Yiwei Chen, Young-Joo Hong, Shuichi Makita, and Yoshiaki Yasuno, "Eye-motion-corrected optical coherence tomography angiography using Lissajous scanning", Biomedical Optics EXPRESS, Vol. 9, No. 3, 1 Mar 2018, PP. 1111- 1129

One object of the present invention is to improve the accuracy of image registration using a mask.

Some aspects are image processing methods, in which the first and second images of the sample are prepared, the mask image is prepared, and the range of the pixel values of the first image and the pixel values of the second image are set. The range and the range of the pixel values of the mask image are relatively adjusted, and the mask image is combined with each of the first image and the second image to generate a first composite image and a second composite image. A plurality of intercorrelation functions are obtained based on the first composite image and the second composite image, a correlation coefficient is calculated based on the plurality of intercorrelation functions, and with the first image based on the correlation coefficient. Registration with the second image is performed.

In some embodiments, the image processing method reduces the difference between the pixel value range of the sample image and the pixel value range of the mask image.

In some embodiments, the image processing method aligns one pixel value range of the sample image and the mask image with the other pixel value range.

In some embodiments, the image processing method normalizes the pixel value range of the sample image according to the pixel value range of the mask image.

In some embodiments, the range of pixel values of the mask image is included in the closed interval [0,1], and the image processing method divides the value of each pixel of the sample image by the maximum pixel value of the image. To do.

In some embodiments, the pixel value range of the mask image is included in the closed interval [0,1], and the image processing method sets the value of each pixel of the sample image to the pixel value range of the image. Divide by the maximum value.

In some embodiments, the mask image is a binary image with a pixel value of 0 or 1.

In some embodiments, the mask image is a binary image in which the pixel values in the region corresponding to the domain of the sample image are 1 and the other pixel values are 0.

In some embodiments, the mask image includes a first mask image and a second mask image, and the image processing method combines the first mask image with the first image to generate the first composite image. Then, the second mask image is combined with the second image to generate the second composite image.

In some embodiments, the first mask image and the second mask image have the same dimensions and the same shape.

In some embodiments, the first mask image is a binary image in which the pixel values in the region corresponding to the domain of the first image are 1 and the values of the other pixels are 0. The second mask image is a binary image in which the value of the pixel in the region corresponding to the domain of the second image is 1, and the value of the other pixel is 0.

In some embodiments, each of the first image and the second image is constructed based on data collected by applying a scan according to a two-dimensional pattern involving a series of cycles to the sample.

In some embodiments, each of the first image and the second image is constructed based on data collected by applying a scan according to a two-dimensional pattern based on a preset scanning protocol based on a resage function to the sample. Will be done.

In some embodiments, a scanning imaging method applies optical scanning according to a two-dimensional pattern involving a series of cycles to a sample to collect data and constructs first and second images based on the data. Then, the range of the pixel value of the first image, the range of the pixel value of the second image, and the range of the pixel value of the mask image are relatively adjusted, and the first image and the second image are each described. The mask images are combined to generate a first composite image and a second composite image, a plurality of intercorrelation functions are obtained based on the first composite image and the second composite image, and based on the plurality of intercorrelation functions. The correlation coefficient is calculated, the first image and the second image are registered based on the correlation coefficient, and the first image and the second image are merged based on the result of the registration. Build an image.

In some embodiments, the scanning imaging method constructs a third image based on the data, the range of the pixel values of the third image, the range of the pixel values of the merged image, and the range of the pixel values of the mask image. And are relatively adjusted, and the mask image is combined with each of the third image and the merged image to generate a third composite image and a fourth composite image, and the third composite image and the fourth composite image are generated. A plurality of mutual correlation functions are obtained based on the above, a correlation coefficient is calculated based on the plurality of mutual correlation functions, and registration between the third image and the fourth image is performed based on the correlation coefficient. The third image and the merged image are merged based on the result of the registration.

In some embodiments, any two cycles of the series of cycles intersect each other at at least one point and the scanning imaging method is collected by optical scanning according to the first cycle group of the series of cycles. The first image is constructed based on the first data obtained, and the second image is based on the second data collected by optical scanning according to a second cycle group different from the first cycle group in the series of cycles. To build.

In some embodiments, the scanning imaging method performs optical scanning according to the two-dimensional pattern based on a preset scanning protocol based on a resage function.

In some aspects, the image processing apparatus is an image processing apparatus, in which a storage unit for storing the first image and the second image of the sample, a first mask image corresponding to the first image, and a second image corresponding to the second image. A mask image generation unit that generates a mask image, a range of pixel values of the first image, and a range of pixel values of the first image and the second image based on the range of pixel values of the first mask image and the range of pixel values of the second mask image. A range adjusting unit that adjusts the range of the pixel value of the above, the first mask image is combined with the first image to generate a first composite image, and the second mask image is combined with the second image. Based on the composite image generation unit that generates the second composite image, the first calculation unit that obtains a plurality of intercorrelation functions based on the first composite image and the second composite image, and the plurality of intercorrelation functions. It includes a second calculation unit that calculates the correlation coefficient, and a registration unit that performs registration between the first image and the second image based on the correlation coefficient.

In some embodiments, the image processing apparatus comprises a data receiving unit that receives data collected from a sample by optical scanning according to a two-dimensional pattern including a series of cycles, and the first image and the second image based on the data. Further includes an image construction unit for constructing.

In some embodiments, a method of controlling an image processing apparatus including a storage unit for storing a first image and a second image of a sample and a processor, wherein the processor is subjected to a first image according to the first image. A mask image and a second mask image corresponding to the second image are generated, and the pixel value of the first image is based on the range of the pixel values of the first mask image and the range of the pixel values of the second mask image. And the range of pixel values of the second image are adjusted, the first mask image is combined with the first image to generate a first composite image, and the second mask image is combined with the second image. To generate a second composite image, obtain a plurality of intercorrelation functions based on the first composite image and the second composite image, and calculate the correlation coefficient based on the plurality of mutual correlation functions. Then, the registration between the first image and the second image is controlled based on the correlation coefficient.

In some embodiments, a scanning imaging apparatus is a scanning unit that collects data by applying optical scanning according to a two-dimensional pattern including a series of cycles to a sample, and a first image and a second image based on the data. An image construction unit that constructs an image, a mask image generation unit that generates a first mask image corresponding to the first image and a second mask image corresponding to the second image, and a pixel value of the first mask image. A range adjusting unit that adjusts the range of the pixel value of the first image and the range of the pixel value of the second image based on the range of the second mask image and the range of the pixel value of the second image, and the first image. A composite image generation unit that synthesizes one mask image to generate a first composite image, and synthesizes the second mask image with the second image to generate a second composite image, and the first composite image and A first calculation unit that obtains a plurality of intercorrelation functions based on the second composite image, a second calculation unit that calculates a correlation coefficient based on the plurality of intercorrelation functions, and the above-mentioned based on the correlation coefficient. It includes a registration unit that performs registration between the first image and the second image, and a merge processing unit that constructs a merged image of the first image and the second image based on the registration result.

In some embodiments, the scanning section comprises deflectors capable of deflecting light in different first and second directions, and the changes in the deflection direction along the first direction are repeated in a first cycle. Optical scanning according to the two-dimensional pattern is applied to the sample by repeating the change in the deflection direction along the second direction in a second cycle different from the first cycle.

In some embodiments, a method of controlling a scanning imaging apparatus including a scanning unit that applies optical scanning to a sample to collect data and a processor, wherein the scanning unit is two-dimensional including a series of cycles. Light scanning according to the pattern is applied to the sample to control the collection of data, and the processor constructs a first image and a second image based on the data, and a first mask image corresponding to the first image. And the second mask image corresponding to the second image are generated, and the range of the pixel values of the first image is based on the range of the pixel values of the first mask image and the range of the pixel values of the second mask image. And the range of pixel values of the second image is adjusted, the first mask image is combined with the first image to generate a first composite image, and the second mask image is combined with the second image. A second composite image is generated, a plurality of mutual correlation functions are obtained based on the first composite image and the second composite image, a correlation coefficient is calculated based on the plurality of mutual correlation functions, and the phase is described. Control to perform registration between the first image and the second image based on the number of relationships, and to construct a merged image of the first image and the second image based on the result of the registration. To do.

Some embodiments are programs that cause a computer to perform any of the above methods.

Some embodiments are computer-readable non-temporary recording media on which a program that causes a computer to perform any of the above methods is recorded.

It is possible to combine any two or more aspects at least in part. In addition, any matter according to the present disclosure can be combined at least partially with respect to any aspect. In addition, any matter according to the present disclosure can be combined at least partially with respect to at least a partial combination of any two or more aspects.

According to an exemplary embodiment, it is possible to improve the accuracy of image registration with a mask.

It is the schematic which shows an example of the structure of the scanning type imaging apparatus (ophthalmic apparatus) which concerns on an exemplary embodiment. It is the schematic which shows an example of the structure of the scanning type imaging apparatus (ophthalmic apparatus) which concerns on an exemplary embodiment. It is the schematic which shows an example of the structure of the scanning type imaging apparatus (ophthalmic apparatus) which concerns on an exemplary embodiment. It is the schematic which shows an example of the structure of the scanning type imaging apparatus (ophthalmic apparatus) which concerns on an exemplary embodiment. It is the schematic which shows an example of the structure of the scanning type imaging apparatus (ophthalmic apparatus) which concerns on an exemplary embodiment. It is the schematic which shows the example of the pattern of the resage scan performed by the scanning type imaging apparatus (ophthalmic apparatus) which concerns on an exemplary embodiment. It is a flowchart which shows an example of the operation of the scanning type imaging apparatus (ophthalmic apparatus) which concerns on an exemplary embodiment. It is a figure for demonstrating an example of the operation of the scanning type imaging apparatus (ophthalmic apparatus) which concerns on an exemplary embodiment. It is a figure for demonstrating an example of the operation of the scanning type imaging apparatus (ophthalmic apparatus) which concerns on an exemplary embodiment. It is a figure for demonstrating an example of the operation of the scanning type imaging apparatus (ophthalmic apparatus) which concerns on an exemplary embodiment.

An image processing method, a scanning imaging method, an image processing apparatus, a control method thereof, a scanning imaging apparatus, a control method thereof, a program, and a recording medium according to some exemplary embodiments will be described with reference to the drawings. ..

The techniques disclosed in the literature cited herein and any other known techniques can be incorporated into exemplary embodiments. Further, unless otherwise specified, the "image data" and the "image" based on the "image data" are not distinguished, and the "site" of the eye to be inspected and the "image" thereof are not distinguished.

The scanning imaging apparatus according to the exemplary embodiment described below is an ophthalmologic apparatus capable of measuring the fundus of a living eye using a Fourier domain OCT (eg, swept source OCT). The type of OCT that can be adopted in the embodiment is not limited to the swept source OCT, and may be, for example, a spectral domain OCT or a time domain OCT.

The scanning imaging apparatus according to some exemplary embodiments may utilize scanning modality other than OCT. For example, some exemplary embodiments can employ any optical scanning modality such as SLO.

Further, the scanning modality applicable to the scanning imaging apparatus according to some exemplary embodiments is not limited to the optical scanning modality. For example, some exemplary embodiments can employ scanning modality using electromagnetic waves other than light, scanning modality using ultrasonic waves, and the like.

The ophthalmic apparatus according to some exemplary embodiments can process data acquired by OCT scans and / or data collected by SLOs, as well as data acquired by other modality. You can. Other modality may be, for example, a fundus camera, a slit lamp microscope, and an ophthalmologic surgical microscope. Ophthalmic devices according to some exemplary embodiments may have such modality functionality.

The target (sample) to which OCT is applied is not limited to the fundus of the eye, and may be any part of the eye such as the anterior segment of the eye or the vitreous body, or a part or tissue of a living body other than the eye. It may be an object other than a living body.

The ophthalmic apparatus according to an exemplary embodiment described below includes some embodiments of a scanning imaging apparatus, some aspects of a method of controlling the scanning imaging apparatus, some aspects of an image processing apparatus, and image processing. Provided are some aspects of a method of controlling an apparatus, some aspects of an image processing method, and some aspects of a scanning imaging method.

<Configuration of scanning imaging device>
The ophthalmic apparatus 1 shown in FIG. 1 includes a fundus camera unit 2, an OCT unit 100, and an arithmetic control unit 200. The fundus camera unit 2 is provided with an element group (optical element, mechanism, etc.) for photographing the eye E to be inspected from the front. The OCT unit 100 is provided with a part of an element group (optical element, mechanism, etc.) for applying an OCT scan to the eye E to be inspected. The other part of the element group for the OCT scan is provided in the fundus camera unit 2. The arithmetic control unit 200 includes one or more processors that execute various arithmetic operations and controls. In addition to these, the ophthalmic apparatus 1 may include an element for supporting the face of the subject and an element for switching the site to which the OCT scan is applied. Examples of the former element are chin rests and forehead pads. An example of the latter element is a lens unit used to switch the OCT scan application site from the fundus to the anterior segment of the eye.

The functions of the elements disclosed herein are implemented using circuitry or processing circuits. Circuit configurations or processing circuit configurations are general purpose processors, dedicated processors, integrated circuits, CPUs (Central Processing Units), GPUs (Graphics Processing Units), ASICs, configured and / or programmed to perform the disclosed functions. Any of Application Specific Integrated Circuits), programmable logic devices (eg SPLDs (Simple Programmable Logic Devices), CPLDs (Complex Programmable Logic Devices), FPGAs (Field Programmable Gate Arrays), conventional circuit configurations, and any combination thereof. A processor is considered to be a processing circuit configuration or circuit configuration, including transistors and / or other circuit configurations. In the present disclosure, circuit configurations, units, means, or similar terms refer to the disclosed functions. The hardware to perform, or the hardware programmed to perform the disclosed functions. The hardware may be the hardware disclosed herein or perform the functions described. It may be known hardware programmed and / or configured to do so. If the hardware is a processor that can be considered as a type of circuit configuration, a circuit configuration, unit, means, or similar terminology. Is a combination of hardware and software, which software is used to configure the hardware and / or processor.

<Fundus camera unit 2>
The fundus camera unit 2 is provided with an optical system for photographing the fundus Ef of the eye to be inspected E. The acquired image of the fundus Ef (called a fundus image, a fundus photograph, etc.) is a front image such as an observation image, a photographed image, or the like. The observed image is obtained, for example, by photographing a moving image using near infrared light, and is used for alignment, focusing, tracking, and the like. The captured image is, for example, a still image using flash light in the visible region or the infrared region.

The fundus camera unit 2 includes an illumination optical system 10 and a photographing optical system 30. The illumination optical system 10 irradiates the eye E to be inspected with illumination light. The photographing optical system 30 detects the return light of the illumination light from the eye E to be inspected. The measurement light from the OCT unit 100 is guided to the eye E to be inspected through the optical path in the fundus camera unit 2, and the return light is guided to the OCT unit 100 through the same optical path.

The light (observation illumination light) output from the observation light source 11 of the illumination optical system 10 is reflected by the concave mirror 12, passes through the condenser lens 13, passes through the visible cut filter 14, and becomes near-infrared light. Further, the observation illumination light is once focused in the vicinity of the photographing light source 15, reflected by the mirror 16, and passes through the relay lens system 17, the relay lens 18, the diaphragm 19, and the relay lens system 20. Then, the observation illumination light is reflected at the peripheral portion of the perforated mirror 21 (the region around the perforated portion), passes through the dichroic mirror 46, is refracted by the objective lens 22, and illuminates the eye E (fundus Ef) to be inspected. To do. The return light of the observation illumination light from the eye E to be inspected is refracted by the objective lens 22, passes through the dichroic mirror 46, passes through the hole formed in the central region of the perforated mirror 21, and passes through the dichroic mirror 55. , It is reflected by the mirror 32 via the photographing focusing lens 31. Further, this return light passes through the half mirror 33A, is reflected by the dichroic mirror 33, and is imaged on the light receiving surface of the image sensor 35 by the imaging lens 34. The image sensor 35 detects the return light at a predetermined frame rate. The focus (focus position) of the photographing optical system 30 is adjusted so as to match the fundus Ef or the anterior segment of the eye.

The light output from the photographing light source 15 (photographing illumination light) is applied to the fundus Ef through the same path as the observation illumination light. The return light of the photographing illumination light from the eye E to be examined is guided to the dichroic mirror 33 through the same path as the return light of the observation illumination light, passes through the dichroic mirror 33, is reflected by the mirror 36, and is reflected by the imaging lens 37. An image is formed on the light receiving surface of the image sensor 38.

The liquid crystal display (LCD) 39 displays a fixation target (fixation target image). A part of the light flux output from the LCD 39 is reflected by the half mirror 33A, reflected by the mirror 32, passes through the photographing focusing lens 31 and the dichroic mirror 55, and passes through the hole portion of the perforated mirror 21. The luminous flux that has passed through the hole of the perforated mirror 21 passes through the dichroic mirror 46, is refracted by the objective lens 22, and is projected onto the fundus Ef. By changing the display position of the fixation target image on the LCD 39, the direction (fixation direction, fixation position) for guiding the line of sight of the eye E to be inspected is changed.

Instead of a display device such as an LCD, for example, a light emitting element array or a combination of a light emitting element and a mechanism for moving the light emitting element may be used.

The alignment optical system 50 generates an alignment index used for alignment of the optical system with respect to the eye E to be inspected. The alignment light output from the light emitting diode (LED) 51 passes through the diaphragm 52, the diaphragm 53, and the relay lens 54, is reflected by the dichroic mirror 55, passes through the hole portion of the perforated mirror 21, and passes through the dichroic mirror 46. It is transmitted and projected onto the eye E to be inspected through the objective lens 22. The return light (corneal reflex light, etc.) from the eye E to be inspected for the alignment light is guided to the image sensor 35 through the same path as the return light of the observation illumination light. Manual alignment and auto alignment can be executed based on the received light image (alignment index image).

As in the conventional case, the alignment index image of this example consists of two bright spot images whose positions change depending on the alignment state. When the relative position of the eye E to be inspected and the optical system changes in the xy direction, the two bright spot images are integrally displaced in the xy direction. When the relative position of the eye E to be inspected and the optical system changes in the z direction, the relative position (distance) between the two bright spot images changes. When the distance between the eye E to be inspected and the optical system in the z direction matches the predetermined working distance, the two bright spot images overlap. When the position of the eye E to be inspected and the position of the optical system match in the xy direction, two bright spot images are presented in or near a predetermined alignment target. When the distance between the eye to be inspected E and the optical system in the z direction matches the working distance, and the position of the eye to be inspected E and the position of the optical system in the xy direction match, the two bright spot images overlap and align. Presented within the target.

In the auto alignment, the data processing unit 230 detects the positions of the two bright spot images, and the main control unit 211 controls the movement mechanism 150 described later based on the positional relationship between the two bright spot images and the alignment target. .. In the manual alignment, the main control unit 211 displays two bright spot images on the display unit 241 together with the observation image of the eye E to be inspected, and the user uses the operation unit 242 while referring to the two displayed bright spot images. To operate the moving mechanism 150.

The focus optical system 60 generates a split index used for focus adjustment with respect to the eye E to be inspected. The focus optical system 60 is moved along the optical path (illumination optical path) of the illumination optical system 10 in conjunction with the movement of the imaging focusing lens 31 along the optical path (optical path) of the imaging optical system 30. The reflection rod 67 is inserted and removed from the illumination optical path. When adjusting the focus, the reflecting surface of the reflecting rod 67 is inclined and arranged in the illumination optical path. The focus light output from the LED 61 passes through the relay lens 62, is separated into two light beams by the split index plate 63, passes through the two-hole diaphragm 64, is reflected by the mirror 65, and is reflected by the condensing lens 66. Once imaged on the reflecting surface of, it is reflected. Further, the focus light is reflected by the perforated mirror 21 via the relay lens 20, passes through the dichroic mirror 46, and is projected onto the eye E to be inspected via the objective lens 22. The return light (reflected light from the fundus, etc.) of the focus light from the eye E to be inspected is guided to the image sensor 35 through the same path as the return light of the alignment light. Manual focusing or auto focusing can be executed based on the received light image (split index image).

The diopter correction lenses 70 and 71 can be selectively inserted into the photographing optical path between the perforated mirror 21 and the dichroic mirror 55. The diopter correction lens 70 is a plus lens (convex lens) for correcting high-intensity hyperopia. The diopter correction lens 71 is a minus lens (concave lens) for correcting intense myopia.

The dichroic mirror 46 synthesizes a fundus photography optical path and an OCT optical path (measurement arm). The dichroic mirror 46 reflects light in the wavelength band used for OCT and transmits light for fundus photography. The measuring arm is provided with a collimator lens unit 40, a retroreflector 41, a dispersion compensation member 42, an OCT focusing lens 43, an optical scanner 44, and a relay lens 45 in this order from the OCT unit 100 side.

The retroreflector 41 is movable in the direction of the arrow shown in FIG. 1, thereby changing the length of the measuring arm. The change in the measurement arm length is used, for example, for correcting the optical path length according to the axial length and adjusting the interference state.

The dispersion compensating member 42, together with the dispersion compensating member 113 (described later) arranged on the reference arm, acts to match the dispersion characteristics of the measurement light LS and the dispersion characteristics of the reference light LR.

The OCT focusing lens 43 is moved along the measuring arm to adjust the focus of the measuring arm. The movement of the photographing focusing lens 31, the movement of the focus optical system 60, and the movement of the OCT focusing lens 43 can be controlled in a coordinated manner.

The optical scanner 44 is substantially located at a position optically conjugate with the pupil of the eye E to be inspected. The optical scanner 44 deflects the measurement light LS guided by the measurement arm. The optical scanner 44 is a galvano scanner capable of two-dimensional scanning, including, for example, a galvano mirror for scanning in the x direction and a galvano mirror for scanning in the y direction.

<OCT unit 100>
As illustrated in FIG. 2, the OCT unit 100 is provided with an optical system for applying the swept source OCT. This optical system includes an interference optical system. This interference optical system divides the light from the variable wavelength light source (wavelength sweep type light source) into the measurement light and the reference light, and superimposes the return light of the measurement light from the eye E to be inspected and the reference light passing through the reference optical path. Together, it generates interfering light and detects this interfering light. The data (detection signal) obtained by the interference optical system is a signal representing the spectrum of the interference light and is sent to the arithmetic control unit 200.

The light source unit 101 includes, for example, a near-infrared wavelength tunable laser that changes the emission wavelength at high speed at least in the near-infrared wavelength band. The light L0 output from the light source unit 101 is guided by the optical fiber 102 to the polarization controller 103, and its polarization state is adjusted. Further, the optical L0 is guided by the optical fiber 104 to the fiber coupler 105 and divided into the measurement optical LS and the reference optical LR. The optical path of the measurement light LS is called a measurement arm, and the optical path of the reference light LR is called a reference arm.

The reference optical LR is guided by the optical fiber 110 to the collimator 111, converted into a parallel luminous flux, and guided to the retroreflector 114 via the optical path length correction member 112 and the dispersion compensation member 113. The optical path length correction member 112 acts to match the optical path length of the reference light LR with the optical path length of the measurement light LS. The dispersion compensating member 113, together with the dispersion compensating member 42 arranged on the measurement arm, acts to match the dispersion characteristics between the reference light LR and the measurement light LS. The retroreflector 114 is movable along the optical path of the reference light LR incident on it, thereby changing the length of the reference arm. The change of the reference arm length is used, for example, for correcting the optical path length according to the axial length and adjusting the interference state.

The reference light LR that has passed through the retro-reflector 114 is converted from a parallel luminous flux to a focused luminous flux by a collimator 116 via a dispersion compensating member 113 and an optical path length correcting member 112, and is incident on the optical fiber 117. The reference light LR incident on the optical fiber 117 is guided by the polarization controller 118 to adjust its polarization state, is guided to the attenuator 120 through the optical fiber 119 to adjust the amount of light, and is guided to the fiber coupler 122 through the optical fiber 121. Be guided.

On the other hand, the measurement optical LS generated by the fiber coupler 105 is guided by the optical fiber 127 and converted into a parallel light beam by the collimator lens unit 40, and is converted into a parallel light beam by the collimator lens unit 40, the retroreflector 41, the dispersion compensation member 42, the OCT focusing lens 43, and the optical scanner 44. The light is reflected by the dichroic mirror 46 via the relay lens 45, refracted by the objective lens 22, and projected onto the eye E to be inspected. The measurement light LS is scattered and reflected at various depth positions of the eye E to be inspected. The return light from the eye E to be inspected of the measurement light LS travels in the same direction as the outward path in the opposite direction, is guided to the fiber coupler 105, and reaches the fiber coupler 122 via the optical fiber 128.

The fiber coupler 122 superimposes the measurement light LS incidented via the optical fiber 128 and the reference light LR incidented via the optical fiber 121 to generate interference light. The fiber coupler 122 generates a pair of interference light LCs by branching the generated interference light at a predetermined branching ratio (for example, 1: 1). The pair of interference light LCs are guided to the detector 125 through the optical fibers 123 and 124, respectively.

The detector 125 includes, for example, a balanced photodiode. The balanced photodiode has a pair of photodetectors that detect each pair of interference light LCs, and outputs the difference between the pair of detection results obtained by these. The detector 125 sends this output (detection signal) to the data collection system (DAS) 130.

A clock KC is supplied to the data collection system 130 from the light source unit 101. The clock KC is generated in the light source unit 101 in synchronization with the output timing of each wavelength swept within a predetermined wavelength range by the tunable light source. The light source unit 101, for example, branches the light L0 of each output wavelength to generate two branched lights, optically delays one of the branched lights, synthesizes the branched lights, and produces the obtained combined light. It is detected, and a clock KC is generated based on the detection result. The data collection system 130 samples the detection signal input from the detector 125 based on the clock KC. The data collection system 130 sends the result of this sampling to the arithmetic control unit 200.

In this example, both an element for changing the measurement arm length (for example, the retroreflector 41) and an element for changing the reference arm length (for example, the retroreflector 114 or the reference mirror) are provided. However, only one element may be provided. Further, the element for changing the difference (optical path length difference) between the measurement arm length and the reference arm length is not limited to these, and may be any element (optical member, mechanism, etc.).

<Control system / processing system>
A configuration example of the control system and the processing system of the ophthalmic apparatus 1 is shown in FIGS. 3, 4A and 4B. The control unit 210, the image construction unit 220, and the data processing unit 230 are provided in, for example, the arithmetic control unit 200. The ophthalmic apparatus 1 may include a communication device for performing data communication with an external device. The ophthalmic apparatus 1 may include a drive apparatus (reader / writer) for reading data from the recording medium and writing data to the recording medium.

<Control unit 210>
The control unit 210 executes various controls. The control unit 210 includes a main control unit 211 and a storage unit 212. Further, as shown in FIG. 4A, in the present embodiment, the main control unit 211 includes the scan control unit 2111, and the storage unit 212 stores the scan protocol 2121.

<Main control unit 211>
The main control unit 211 includes a processor and controls each element of the ophthalmic apparatus 1 (including the elements shown in FIGS. 1 to 4B). The main control unit 211 is realized by the cooperation between the hardware including the processor and the control software. The scanning control unit 2111 controls OCT scanning for a scanning area having a predetermined shape and a predetermined size.

The photographing focusing drive unit 31A moves the photographing focusing lens 31 arranged in the photographing optical path and the focusing optical system 60 arranged in the illumination optical path under the control of the main control unit 211. The retroreflector (RR) drive unit 41A moves the retroreflector 41 provided on the measurement arm under the control of the main control unit 211. The OCT focusing drive unit 43A moves the OCT focusing lens 43 arranged on the measurement arm under the control of the main control unit 211. The retroreflector (RR) drive unit 114A provided on the measuring arm moves the retroreflector 114 arranged on the reference arm under the control of the main control unit 211. Each of the above-mentioned drive units includes an actuator such as a pulse motor that operates under the control of the main control unit 211. The optical scanner 44 operates under the control of the main control unit 211 (scanning control unit 2111).

The moving mechanism 150 typically moves the fundus camera unit 2 three-dimensionally, and for example, an x-stage capable of moving in the ± x direction (horizontal direction), an x-moving mechanism for moving the x-stage, and ± Includes a y stage that can move in the y direction (vertical direction), a y movement mechanism that moves the y stage, a z stage that can move in the ± z direction (depth direction), and a z movement mechanism that moves the z stage. .. Each moving mechanism includes an actuator such as a pulse motor that operates under the control of the main control unit 211.

<Memory unit 212>
The storage unit 212 stores various types of data. The data stored in the storage unit 212 includes an OCT image, a fundus image, eye examination information, control information, and the like. The eye test information includes subject information such as patient ID and name, left eye / right eye identification information, electronic medical record information, and the like. Control information is information about a particular control. The control information of this embodiment includes scanning protocol 2121.

Scanning Protocol 2121 is an agreement on the content of control for OCT scanning of a scanning area of a given shape and size and includes a set of various control parameters (scanning control parameters). Scanning protocol 2121 includes a protocol for each scanning mode. The scanning protocol 2121 of the present embodiment includes at least the protocol of the resage scan, and may further include protocols such as B scan (line scan), cross scan, radial scan, and raster scan.

The scanning control parameter of the present embodiment includes at least a parameter indicating the content of control for the optical scanner 44. This parameter includes, for example, a parameter indicating a scan pattern, a parameter indicating a scan speed, a parameter indicating a scan interval, and the like. The scan pattern indicates the shape of the scan path, and examples thereof include a resage pattern, a line pattern, a cross pattern, a radial pattern, and a raster pattern. The scan speed is defined as, for example, the repeat rate of the A scan. The scan interval is defined, for example, as the interval between adjacent A scans, that is, as the array interval of scan points.

Similar to the prior art as disclosed in Patent Documents 1 to 4 and Non-Patent Documents 1 and 2, the "Lissajous scan" of the present embodiment is obtained in that two simple vibrations orthogonal to each other are obtained as an order pair. A "broad" Lissajous that follows a predetermined two-dimensional pattern that includes a series of cycles, as well as a "narrow" Lissajous scan that is routed through a pattern drawn by a locus (Lissajous pattern, Lissajous figure, Lissajous curve, Lissajous function, Bauditch curve). It may be a scan.

In the present embodiment, the optical scanner 44 includes a first galvano mirror that deflects the measurement light LS in the x direction and a second galvano mirror that deflects the measurement light LS in the y direction. In the resage scan, the first galvanometer mirror is controlled so that the change in the deflection direction along the x direction is repeated in the first cycle, and the second change in the deflection direction along the y direction is repeated in the second cycle. It is realized by controlling the galvanometer mirror. Here, the first cycle and the second cycle are different from each other.

For example, the lithage scan of the present embodiment is not only a scan of a lisage pattern in a narrow sense obtained from a combination of two sine waves, but also a scan of a pattern to which a specific term (for example, an odd-order polynomial) is added, or a triangular wave. It may be a scan of a pattern based on.

"Cycle" generally means an object composed of multiple sampling points of a certain length, which may be, for example, a closed curve or a nearly closed curve (ie, the start and end points of the cycle coincide with or nearly coincide with each other. You can do it).

Typically, the scan protocol 2121 is set based on the Lissajous function. As shown in Equation (9) of Non-Patent Document 1, the Lissajous function is expressed by, for example, the following parametric equation system: x (t _i ) = A · cos (2π · (f _A / n) · t. _i ), y (t _i ) = A ・ cos (2π ・ (f _A・ (n-2) / n ² ) ・ t _i ).

Here, x is the horizontal axis of the two-dimensional coordinate system in which the Lissajous curve is defined, y is the vertical axis, t _i is the collection point of the i-th A line in the Lissajous scan, A is the scan range (amplitude), and f _A. Is the collection rate of A-lines (scan speed, repetition rate of A-scans), and n is the number of A-lines in each cycle in the x (horizontal axis) direction.

An example of the distribution of scan lines applied to the fundus Ef in such a Lissajous scan is shown in FIG.

Between any cycle pairs (pairs) included in a (narrowly or broadly defined) lisage scan intersect each other at one or more points (especially two or more points) and therefore between data pairs collected from any cycle pair in a lisage scan. Registration can be performed. This makes it possible to use the image construction method and the motion artifact correction method disclosed in Non-Patent Document 1 (or 2). Hereinafter, unless otherwise specified, the method described in Non-Patent Document 1 is applied.

The control information stored in the storage unit 212 is not limited to the above example. For example, the control information may include information for performing focus control (focus control parameter).

The focus control parameter is a parameter indicating the content of control for the OCT focusing drive unit 43A. Examples of focus control parameters include a parameter indicating the focal position of the measuring arm, a parameter indicating the moving speed of the focal position, and a parameter indicating the moving acceleration of the focal position. The parameter indicating the focal position is, for example, a parameter indicating the position of the OCT focusing lens 43. The parameter indicating the moving speed of the focal position is, for example, a parameter indicating the moving speed of the OCT focusing lens 43. The parameter indicating the moving acceleration of the focal position is, for example, a parameter indicating the moving acceleration of the OCT focusing lens 43. The moving speed may or may not be constant. The same applies to the moving acceleration.

According to such a focus control parameter, the focus can be adjusted according to the shape of the fundus Ef (typically, a concave shape having a deep central portion and a shallow peripheral portion) and an aberration distribution. Focus control is executed in cooperation with, for example, scanning control (repeated control of Lissajous scan). This results in a high quality image with motion artifacts corrected and in focus over the entire scan range.

<Scanning control unit 2111>
The scan control unit 2111 controls at least the optical scanner 44 based on the scan protocol 2121. The scan control unit 2111 may further control the light source unit 101 in conjunction with the control of the optical scanner 44 based on the scan protocol 2121. The scan control unit 2111 is realized by the cooperation of hardware including a processor and scan control software including scan protocol 2121.

<Image construction unit 220>
The image construction unit 220 includes a processor and constructs OCT image data of the fundus Ef based on the signal (sampling data) input from the data collection system 130. This OCT image data construction includes noise removal (noise reduction), filtering, fast Fourier transform (FFT), and the like, similar to the conventional Fourier domain OCT (swept source OCT). When another type of OCT method is adopted, the image construction unit 220 constructs OCT image data by executing a known process according to the type.

As described above, in this embodiment, the Lissajous scan is applied to the fundus Ef. The image construction unit 220, together with the data processing unit 230, refers to the data acquired through collection by the resage scan and sampling by the data collection system 130, for example, the image construction method and motion disclosed in Non-Patent Document 1 or 2. By applying the artifact correction method, three-dimensional image data of the fundus Ef is constructed.

The image construction unit 220 (and / or the data processing unit 230) can apply rendering to the three-dimensional image data to form a display image. Examples of applicable rendering methods include volume rendering, surface rendering, maximum value projection (MIP), minimum value projection (MinIP), and multi-section reconstruction (MPR).

The image construction unit 220 (and / or the data processing unit 230) can construct an OCT front image (enface OCT image) based on the three-dimensional image data. For example, the image construction unit 220 (and / or the data processing unit 230) can construct projection data by projecting the three-dimensional image data in the z direction (A line direction, depth direction). Further, the image construction unit 220 (and / or the data processing unit 230) can construct a shadow gram by projecting partial 3D image data which is a part of the 3D image data in the z direction. This partial three-dimensional image data is set by using, for example, segmentation. Segmentation is a process of identifying a partial area in an image. In this example, segmentation can be performed to identify an image region corresponding to a predetermined tissue of the fundus Ef.

The ophthalmic apparatus 1 may be capable of performing OCT-Angiography. OCT angiography is an imaging technique for constructing an image in which blood vessels are emphasized (see, for example, Non-Patent Document 2, Japanese Patent Application Laid-Open No. 2015-515894, etc.). Generally, the fundus tissue (structure) does not change with time, but the blood flow portion inside the blood vessel changes with time. In OCT angiography, an image is generated by emphasizing a portion (blood flow signal) in which such a temporal change exists. In addition, OCT angiography is also called OCT motion contrast imaging (motion contrast imaging) or the like. The images acquired by OCT angiography are called angiographic images, angiograms, motion contrast images, and the like.

When OCT angiography is performed, the ophthalmologic apparatus 1 repeatedly scans the same area of the fundus Ef a predetermined number of times. For example, the ophthalmic apparatus 1 repeatedly performs the above-mentioned scanning control (repeated control of Lissajous scan) a predetermined number of times. As a result, a plurality of three-dimensional data (three-dimensional data sets) corresponding to the application area of the resage scan are collected by the data collection system 130. The image construction unit 220 and / or the data processing unit 230 can construct a motion contrast image from this three-dimensional data set. This motion contrast image is an angiographic image in which the temporal change of the interference signal due to the blood flow of the fundus Ef is emphasized. This angiographic image is three-dimensional angiographic image data expressing the three-dimensional distribution of blood vessels of the fundus Ef.

The image construction unit 220 and / or the data processing unit 230 can construct arbitrary 2D angiographic image data and / or arbitrary pseudo 3D angiographic image data from the 3D angiographic image data. is there. For example, the image construction unit 220 can construct two-dimensional angiography image data representing an arbitrary cross section of the fundus Ef by applying the multi-section reconstruction to the three-dimensional angiography image data. Further, the image construction unit 220 can construct the angiographic image data of the fundus Ef by applying projection imaging or shadow gramming to the three-dimensional angiography image data.

In this embodiment, the image construction unit 220 constructs a plurality of strips from the data collected by the data collection system 130. As described in Non-Patent Document 1, the image construction unit 220 divides the volume (three-dimensional data) collected by the resage scan into a plurality of subvolumes in which relatively large movement does not intervene, and of each subvolume. Build a front projection image (en face projection). This front projection image is a strip. By applying the registration and merging process to the plurality of strips thus obtained, an image corrected with motion artifacts can be obtained. As will be described later, the registration and merge processing is executed by the data processing unit 230.

The image construction unit 220 is realized by the collaboration between the hardware including the processor and the image construction software.

<Data processing unit 230>
The data processing unit 230 includes a processor and applies various data processing to the image of the eye E to be inspected. For example, the data processing unit 230 is realized by the collaboration between the hardware including the processor and the data processing software.

The data processing unit 230 may be configured to perform alignment (registration) between the two images acquired for the fundus Ef. For example, the data processing unit 230 is configured to perform registration between the three-dimensional image data acquired by OCT and the front image acquired by the fundus camera unit 2. Further, the data processing unit 230 is configured to perform registration between two OCT images acquired by OCT. Further, the data processing unit 230 is configured to perform registration between the two front images acquired by the fundus camera unit 2. Further, the registration may be applied to the analysis result of the OCT image or the analysis result of the front image. These registrations can be performed by known techniques, including, for example, feature point extraction and affine transformation.

Further, the data processing unit 230 is configured to process the data obtained by using the Lissajous scan. As described above, the image construction unit 220 constructs a plurality of strips from the data collected by the Lissajous scan. The data processing unit 230 constructs an image corrected with motion artifacts by applying registration and merging processing to these strips.

As a characteristic of the Lissajous scan, any two strips have an overlapping region. Similar to the method described in Non-Patent Document 1, the data processing unit 230 is configured to perform registration between strips by utilizing the overlap between strips. The data processing unit 230 first orders a plurality of strips constructed by the image construction unit 220 according to the size (area, etc.) (first to Nth strips), and uses the first strip, which is the largest strip, as an initial reference. Specify on the strip. Next, the data processing unit 230 registers the second strip with the first strip as a reference, and merges (synthesizes) the first strip and the second strip. The data processing unit 230 registers the third strip with reference to the merge strip obtained thereby, and merges the merge strip with the third strip. By sequentially executing such registration and merging processing in the above order, the first to Nth strips are aligned and bonded, and an image corrected with motion artifacts is obtained.

In such registrations, a cross-correlation function is used to determine the relative position between the reference strip and the other strips, but since the strips have any shape, each strip has a specific shape (eg square). ) Is used as an image to perform correlation calculation between strips (see Appendix A of Non-Patent Document 1).

As described above, since the strip is an intensity image of a predetermined gradation and the mask image is a binary image, the difference between the absolute value of the pixel value of the strip and the absolute value of the pixel value of the mask becomes large, and the strip and the strip The effect of the mask in the correlation calculation using the mask may be ignored and the correct correlation coefficient may not be obtained. In particular, when a single-precision floating-point type (float type) operation is adopted to obtain the correlation coefficient between strips from the viewpoint of cost, etc., this problem is caused by the influence of rounding error due to the small number of significant digits. Appears prominently.

In order to deal with this problem, in this embodiment, the data processing unit 230 having the configuration shown in FIG. 4B is adopted. The data processing unit 230 of this example includes a mask image generation unit 231, a range adjustment unit 232, a composite image generation unit 233, a cross-correlation function calculation unit 234, a correlation coefficient calculation unit 235, and a registration unit 236. , The merge processing unit 237 and the like. In the example described below, any two strips are considered, one of which is designated as the reference strip, and the other strip (registering strip) is registered for this reference strip. ..

<Mask image generation unit 231>
The mask image generation unit 231 generates a reference mask image corresponding to the reference strip and a target mask image corresponding to the target strip. Some examples of mask images will be described below, but are not limited thereto.

The contour shape of the mask image is, for example, a rectangle, typically a square. The shape of the reference mask image and the shape of the target mask image may be the same. Further, the dimensions of the reference mask image and the dimensions of the target mask image may be the same.

The range of pixel values of the mask image is set so as to be included in, for example, the closed interval [0,1]. Typically, the mask image may be a binary image with a pixel value of 0 or 1. As a specific example, the mask image is a binary image in which the value of the pixel in the region corresponding to the domain of the strip is 1 and the value of the other pixel is 0. In other words, as shown in the equation (20) of Non-Patent Document 1, the pixel value of the exemplary mask image has a value of 1 in the image area of the corresponding strip and 0 in the other area.

<Range adjustment unit 232>
The range adjusting unit 232 adjusts the pixel value range of the reference strip and the pixel value range of the target strip based on the pixel value range of the reference mask image and the pixel value range of the target mask image. It is composed of. Typically, the range adjuster 232 sets the pixel value range of the reference strip and the pixel value of the target strip so as to reduce the difference between the pixel value range of the strip and the pixel value range of the mask image. Adjust the range.

Generally, the range adjusting unit 232 relatively adjusts the pixel value range of the reference strip, the pixel value range of the target strip, and the pixel value range of the mask image. In a typical example, unlike the reference mask image applied to the reference strip and the target mask image applied to the target strip, the range adjustment unit 232 sets the pixel value range of the reference strip and the pixel value of the target strip. The range of the pixel value of the reference mask image and the range of the pixel value of the target strip are relatively adjusted.

The range adjustment unit 232 adjusts the pixel value range of the reference strip and the pixel value range of the reference mask image so as to reduce the pixel value range of the reference strip and the pixel value range of the reference mask image. , The pixel value range of the target strip and the pixel value range of the target strip are relatively adjusted so as to reduce the pixel value range of the target strip and the pixel value range of the target mask image. To.

The range adjusting unit 232 adjusts the pixel value range of the strip and the pixel value range of the mask image so as to match the pixel value range of one of the strip and mask images with the pixel value range of the other. It may be configured. For example, the range adjustment unit 232 adjusts the pixel value range of the reference strip and the pixel value range of the reference mask image so as to match the pixel value range of the reference strip with the pixel value range of the reference mask image. In addition, the pixel value range of the target strip and the pixel value range of the target strip are relatively adjusted so as to match the pixel value range of the target strip with the pixel value range of the target mask image. It may be configured.

For example, the range adjusting unit 232 is configured to normalize the pixel value range of the strip according to the pixel value range of the mask image. Some examples of this normalization will be described below, but the present invention is not limited thereto.

A first example of normalization will be described. When the range of pixel values of the mask image is included in the closed interval [0,1], the range adjustment unit 232 divides the value of each pixel of the strip by the maximum pixel value in this strip. In this example, the range adjusting unit 232 first compares the values of all the pixels of the strip to specify the maximum value (maximum pixel value), and divides the value of each pixel of the strip by the maximum pixel value. As a result, the pixel value range of the strip coincides with the same closed interval [0,1] as the pixel value range of the mask image.

A second example of normalization will be described. When the range of pixel values of the mask image is included in the closed interval [0,1], the range adjustment unit 232 divides the value of each pixel of the strip by the maximum value of the pixel value range of this strip. The range of pixel values of the strip is set in advance, and the range adjusting unit 232 divides the value of each pixel of the strip by the upper limit value (maximum value) of this range. Also in this example, the pixel value range of the strip coincides with the same closed interval [0,1] as the pixel value range of the mask image.

By the process executed by the range adjustment unit 232, the difference between the absolute value of the pixel value of the strip and the absolute value of the pixel value of the mask can be reduced, and the effect of the mask is ignored in the correlation calculation using the strip and the mask. It becomes possible to calculate the correlation coefficient accurately. In particular, even when a single-precision floating-point type (float type) operation is adopted, the influence of rounding error due to a small number of significant digits can be eliminated (reduced).

Although the example of changing only the pixel value range of the strip is mainly described in the present specification, only the pixel value range of the mask image may be changed, or the pixel value range of the strip and the pixel value of the mask image may be changed. Both with the range of may be changed.

<Composite image generation unit 233>
For the strip and mask image to which the pixel value range adjustment by the range adjustment unit 232 is applied, the composite image generation unit 233 synthesizes the mask image with each of the reference strip and the target strip to generate two composite images. Typically, the composite image generation unit 233 synthesizes the reference mask image with the reference strip to generate the reference composite image, and synthesizes the target mask image with the target strip to generate the target composite image.

The process of synthesizing the strip and the mask image may be executed in the same manner as in Non-Patent Document 1. However, unlike the method of Non-Patent Document 1, in the present embodiment, the difference between the pixel value range of the strip and the pixel value range of the mask image is adjusted to be small. Typically, the pixel value ranges of the reference strip and the target strip are normalized to match the pixel value ranges of the reference mask image and the target mask image.

For example, in the same manner as in the formula (19) of Non-Patent Document 1, the composite image generation unit 233 is configured to embed a strip having a normalized pixel value range in an image having the same dimensions and the same shape as the mask image. It's okay.

Further, the composite image generation unit 233 generates a composite image of the embedded image of the strip and the mask image. This composite image corresponds to "f'(r) m _f (r)" (second line on page 1800, etc.) in Non-Patent Document 1, but as described above, the value of the embedded image of the strip is It is different from that of equation (19).

<Cross-correlation function calculation unit 234>
The cross-correlation function calculation unit 234 obtains a plurality of cross-correlation functions based on the two composite images generated by the composite image generation unit 233. The calculation of the cross-correlation function is performed in the same manner as in Non-Patent Document 1. For example, the cross-correlation function calculation unit 234 uses the formula of Non-Patent Document 1 based on the reference composite image generated from the reference strip and the reference mask image and the target composite image generated from the target strip and the target mask image. The six cross-correlation functions (image cross-correlation) included in 33) are calculated.

<Correlation coefficient calculation unit 235>
The correlation coefficient calculation unit 235 calculates the correlation coefficient based on a plurality of cross-correlation functions calculated by the cross-correlation function calculation unit 234. This operation follows the equation (33) of Non-Patent Document 1.

<Registration unit 236>
The registration unit 236 performs registration in the xy direction (lateral direction) based on the correlation coefficient calculated by the correlation coefficient calculation unit 235. This registration corresponds to "rough lateral motion correction" (page 1787) of Non-Patent Document 1.

Further, the registration unit 236 uses "fine lateral motion correction" of Non-Patent Document 1 in order to remove small motion artifacts in the lateral direction caused by eye movements such as slow drift and tremor. It may be configured to perform the registration corresponding to page 1789).

In addition, the registration unit 236 uses "axial motion correction (rough axial motion correction and / or fine axial motion correction") of Non-Patent Document 1 in order to remove motion artifacts in the z direction (axial direction, axial direction). It may be configured to perform the registration corresponding to pages 1789 to 1790).

<Merge processing unit 237>
The merge processing unit 237 constructs a merged image of the reference strip and the target strip whose relative positions have been adjusted by the registration unit 236. This process is also performed in the same manner as the method described in Non-Patent Document 1.

As described above, the data processing unit 230 sequentially executes the above-mentioned series of processes according to the ordering of the plurality of strips constructed by the image construction unit 220 according to the size. As a result, an image corrected with motion artifacts can be obtained from the plurality of strips constructed by the image constructing unit 220. The resulting image is typically an image representing the entire area to which the Lissajous scan has been applied.

<User interface 240>
The user interface 240 includes a display unit 241 and an operation unit 242. The display unit 241 includes a display device 3. The operation unit 242 includes various operation devices and input devices. The user interface 240 may include a device such as a touch panel in which a display function and an operation function are integrated. It is also possible to build embodiments that do not include at least part of the user interface 240. For example, the display device may be an external device connected to the ophthalmic device 1.

<motion>
The operation of the ophthalmic apparatus 1 will be described. An example of the operation of the ophthalmic apparatus 1 is shown in FIG. It should be noted that the same preparatory processing as before, such as inputting the patient ID, setting the scanning mode (designating the resage scan), presenting the fixation target, alignment, focus adjustment, and OCT optical path length adjustment, has already been performed. To do.

(S1: Lissajous scan)
Upon receiving the predetermined scanning start trigger signal, the scanning control unit 2111 starts applying the OCT scan (risage scan) to the eye E (fundus Ef) to be inspected.

The scanning start trigger signal indicates, for example, that a scanning start instruction operation has been performed in response to the completion of predetermined preparatory operations (alignment, focus adjustment, OCT optical path length adjustment, etc.) or by using the operation unit 242. Is generated corresponding to.

The scan control unit 2111 applies the lisage scan to the fundus Ef by controlling the optical scanner 44, the OCT unit 100, and the like based on the scan protocol 2121 (protocol corresponding to the lisage scan). The data collected by the Lissajous scan is sent to the image construction unit 220.

(S2: Build multiple strips)
The image construction unit 220 constructs a plurality of strips based on the data collected in step S1 as described above. The constructed strips are sent to the data processing unit 230.

(S3: Set the reference strip and the target strip)
The data processing unit 230 orders the plurality of strips constructed in step S2 according to the dimensions (area, etc.). In this example, it is assumed that the number of the plurality of strips constructed in step S2 is N (N is an integer of 2 or more). Also, according to the order specified by the ordering, these N strips are referred to as a first strip, a second strip, ..., Nth strip. Further, any strip among the first to Nth strips may be referred to as an nth strip (n = 1, 2, ..., N). Thus, the first strip is the largest strip, the second strip is the second largest strip, the nth strip is the nth largest strip, and the Nth strip is the smallest strip. is there.

In this example, the data processing unit 230 sets the first strip as the initial reference strip and the second strip as the initial target strip.

The reference strip and the target strip correspond to the arbitrary shaped images f (r) and g (r) in Appendix A of Non-Patent Document 1, respectively.

(S4: Set the reference mask image and the target mask image)
The mask image generation unit 231 generates a mask image (reference mask image) corresponding to the reference strip and a target mask image corresponding to the target strip.

At this stage, the mask image generation unit 231 generates two mask images corresponding to the first strip and the second strip, respectively. The data processing unit 230 sets the first mask image corresponding to the first strip as the initial reference mask image, and sets the second mask image corresponding to the second strip as the initial target mask image.

_{The reference mask image and the target mask image correspond to rectangular shaped binary image masks m f} (r) and _mg (r) in Appendix A of Non-Patent Document 1, respectively.

(S5: Normalize the reference strip and the target strip)
The range adjusting unit 232 normalizes the reference strip and the target strip set in step S3 as described above. Strip normalization is performed according to the range of pixel values of the mask image. At this stage, the range adjuster 232 applies normalization to the first strip (reference strip) and the second strip (target strip). The process executed by the range adjustment unit 232 is not limited to normalization, and may be any of the above-mentioned range adjustment processes or a process similar thereto.

In this example, the range of pixel values of each mask image is included in the closed interval [0,1]. In particular, it is assumed that each mask image of this example is a binary image in which the pixel value in the region corresponding to the domain of the corresponding strip is set to 1 and the values of the other pixels are set to 0.

The range adjustment unit 232, for example, divides the value of each pixel of the strip by the maximum pixel value in this strip, or divides the value of each pixel of the strip by the maximum pixel value range (gradation range) of this strip. Normalize this strip by dividing by a value.

Further, the data processing unit 230 (composite image generation unit 233) embeds a strip having a normalized pixel value range in an image having the same size and shape as the corresponding mask image. The embedded image of the normalized first strip f (r) is an image similar to the rectangular shaped image f'(r) in Appendix A of Non-Patent Document 1, but the first strip. The absolute value (| f (r) |) of the range in the image area of f (r) is normalized to 1 or less. This embedded image is also represented as f'(r).

Similarly, the embedded image of the normalized second strip g (r) is an image similar to the rectangular image g'(r), but with a range within the image area of the second strip g (r). The absolute value of (| g (r) |) is normalized to 1 or less. This embedded image is also represented as g'(r).

(S6: Generate reference composite image and target composite image)
The composite image generation unit 233 generates a composite image of the embedded image of the normalized strip and the corresponding mask image.

At this stage, the composite image generation unit 233 generates a composite image of the normalized first strip (reference strip) embedded image and the first mask image (reference mask image), and is normal. A composite image of the embedded image of the second strip (target strip) and the second mask image (target mask image) is generated. The former composite image is called a reference composite image, and the latter is called a target composite image.

Here, the reference composite image and the target composite image are the combination of two rectangular shaped images in Appendix A of Non-Patent Document 1, respectively (combination of two rectangular shaped images) f'(r) m _f (r) and g. It corresponds to ´ (r) _mg (r). However, as described above, the absolute value (| f (r) |) of the range in the image area of the first strip f (r) is 1 or less, and the image of the second strip g (r). The absolute value (| g (r) |) of the range in the area is 1 or less.

Image 311 of FIG. 7A is an example of the embedded image f'(r) of the normalized first strip f (r), and image 321 is an example of the first mask image. By combining these two images 311 and 321, a reference composite image f'(r) m _f (r) can be obtained.

Similarly, image 312 of FIG. 7B is an example of the embedded image g'(r) of the normalized second strip f (r), and image 322 is an example of the second mask image. By combining these two images 312 and 322, a reference composite image _g '(r) mg (r) is obtained.

(S7: Calculate multiple cross-correlation functions)
The cross-correlation function calculation unit 234 calculates a plurality of cross-correlation functions based on the reference composite image and the target composite image generated in step S6. In this example, the cross-correlation function calculation unit 234 calculates six cross-correlation functions included in the equation (33) of Non-Patent Document 1 based on the reference composite image and the target composite image generated in step S6. ..

(S8: Calculate the correlation coefficient)
The correlation coefficient calculation unit 235 calculates the correlation coefficient based on the plurality of cross-correlation functions calculated in step S7. In this example, the correlation coefficient calculation unit 235 calculates the correlation coefficient (ρ (r')) from the plurality of cross-correlation functions calculated in step S7 according to the equation (33) of Non-Patent Document 1.

(S9: Registration between the reference strip and the target strip)
The registration unit 236 applies registration (rough lateral motion correction) in the xy direction to the reference strip and the target strip based on the correlation coefficient calculated in step S8. At this stage, registration between the first strip f (r) and the second strip is performed.

In this example, the relative shift amount (Δx, Δy) between the reference strip and the target strip is obtained by detecting the peak of the correlation coefficient ρ (r ′) calculated in step S8.

Further, the registration unit 236 applies registrations such as the above-mentioned "fine lateral motion correction" and "axial motion correction" to the reference strip and the target strip.

(S10: Build a merged image of the reference strip and the target strip)
The merge processing unit 237 constructs a merged image of the reference strip and the target strip whose relative positions have been adjusted by the registration in step S9. Image 330 of FIG. 7C shows an example of a merged image of the first strip and the second strip.

(S11: Did you process all the strips?)
A series of processes of steps S3 to S10 are sequentially executed in the order described above for all of the N strips obtained in step S2.

When N is 3 or more and a merged image of the first strip and the second strip is created in step S10 (S11: No), the process returns to step S3. In step S3, the merged image of the first strip and the second strip is set as the new reference strip, and the third strip is set as the new target strip. By executing the processes of steps S4 to S10 based on the new reference strip and the new target strip, a merged image of the new reference strip and the new target strip is obtained. This new merged image is a merged image of the first to third strips. By sequentially applying such a series of processes to N strips, all merged images of N strips can be obtained (S11: Yes).

(S12: Save the final merged image)
The merged image finally obtained by the iterative process described above is an image in which motion artifacts have been corrected and represents the entire applicable range of the resage scan in step S1. The main control unit 211 stores the final merged image in the storage unit 212 (and / or another storage device).

The data processing unit 230 was constructed by the image construction unit 220 from the data (three-dimensional data) collected in step S1 and / or the three-dimensional data by utilizing the registration result based on N strips. Registration of 3D image data can be performed. This registration includes a process of converting the definition coordinate system of the Lissajous scan into a three-dimensional Cartesian coordinate system (xyz coordinate system). This coordinate transformation corresponds to "remapping" in the method described in Non-Patent Document 1. The main control unit 211 can store the three-dimensional image data thus obtained in the storage unit 212 (and / or another storage device) together with or instead of the final merged image (and / or other storage device). End).

The main control unit 211 can display the final merged image on the display unit 241 (and / or another display device). In addition, the data processing unit 230 can create an arbitrary rendered image from the three-dimensional image data constructed by the coordinate transformation. The main control unit 211 can display this rendered image on the display unit 241 (and / or another display device).

Here, an implementation example of the processing of steps S7 to S10 will be described. First, f'(r) m _f (r), _g '(r) mg (r), m _f (r), _mg (r), (f'(r) m _f (r)) ² , _{And (g} '(r) mg (r)) ² is set to 0 for each imaginary part, and only the real part is set.

Next, f'(r) m _f (r), _g '(r) mg (r), m _f (r), _mg (r), (f'(r) m _f (r)) ² _{, And (g} '(r) mg (r)) Apply a fast Fourier transform (FFT) to each real part of ^2.

Next, the six cross-correlation functions shown in step S7 of FIG. 6 are calculated based on the function group obtained by these fast Fourier transforms.

Next, an inverse fast Fourier transform (IFFT) is applied to each of the calculated six cross-correlation functions.

Next, the correlation coefficient ρ (r') of the formula (33) of Non-Patent Document 1 is calculated.

Then, the relative shift amount (Δx, Δy) between the reference strip f (r) and the target strip g (r) is calculated by specifying the peak position of the correlation coefficient ρ (r ′). A registration and merge process between the reference strip f (r) and the target strip g (r) is performed based on the shift amount.

By sequentially applying such a series of processes to N strips, all merged images of N strips can be obtained.

<Action, effect, etc.>
Some features of the exemplary embodiments will be described, and some actions and some effects exerted by them will be described.

The ophthalmic apparatus 1 according to the above exemplary embodiment provides some aspects of a scanning imaging apparatus. In addition, the ophthalmologic apparatus 1 according to the above exemplary embodiment includes some aspects of the method of controlling the scanning imaging apparatus, some aspects of the image processing apparatus, and some methods of controlling the image processing apparatus. The present aspect, some aspects of the image processing method, and some aspects of the scanning imaging method are provided.

It should be noted that some aspects of the method of controlling the scanning imaging apparatus, some aspects of the image processing apparatus, some aspects of the method of controlling the image processing apparatus, some aspects of the image processing method, and scanning. The matters described in any of several aspects of the type imaging method can be applied to the ophthalmic apparatus 1 (more generally, any aspect of the scanning imaging apparatus) according to the above exemplary embodiment. Is.

The scanning imaging apparatus (ophthalmic apparatus 1) according to the exemplary embodiment includes a scanning unit, an image construction unit 220, a mask image generation unit 231, a range adjustment unit 232, a composite image generation unit 233, and the like. It includes a cross-correlation function calculation unit 234 (first calculation unit), a correlation coefficient calculation unit 235 (second calculation unit), a registration unit 236, and a merge processing unit 237.

The scanning unit is configured to apply optical scanning to a sample (eye to be inspected E, fundus Ef) to collect data. This optical scan is performed according to a two-dimensional pattern that includes a series of cycles.

In some exemplary embodiments, any two cycles of the series of cycles included in the two-dimensional pattern of optical scanning may intersect each other at at least one point.

Further, in some exemplary embodiments, the optical scan applied to the sample may be performed according to a two-dimensional pattern based on a preset scan protocol based on the resage function. An example of such an optical scan is the Lissajous scan in the above exemplary embodiment.

In some exemplary embodiments, the scanning unit comprises a deflector (optical scanner 44) capable of deflecting light in different first directions (x directions) and second directions (y directions) and in the first direction. By repeating the change in the deflection direction along the first cycle and the change in the deflection direction along the second direction in the second cycle different from the first cycle, the optical scanning according to the two-dimensional pattern is applied to the sample. It may be configured.

In the above exemplary embodiment, the scanning unit includes an OCT unit 100 and elements in the fundus camera unit 2 constituting the measurement arm (retroreflector 41, OCT focusing lens 43, optical scanner 44, objective lens 22, etc.). OCT data is collected by applying an OCT scan to the eye E to be inspected.

The image construction unit 220 is configured to construct a first image and a second image based on the data collected by the scanning unit. The combination of the reference strip and the target strip in the above exemplary embodiment is an example of the combination of the first image and the second image.

In some exemplary embodiments, if any two cycles of the series of cycles included in the two-dimensional pattern of optical scanning intersect each other at at least one point, the image builder 220 will see the series of cycles. The first image was constructed based on the first data collected by the optical scan according to the first cycle group, and was collected by the optical scan according to the second cycle group different from the first cycle group in the series of cycles. It may be configured to construct a second image based on the second data.

For example, when the two-dimensional pattern of optical scanning follows a preset scanning protocol based on the resage function, as in the resage scan in the above exemplary embodiment, the image construction unit 220 is included in this two-dimensional pattern. An optical scan that constructs a first image based on the first data collected by an optical scan that follows the first cycle group of a series of cycles and that follows a second cycle group that is different from the first cycle group of the series of cycles. It may be configured to construct a second image based on the second data collected in.

In the above exemplary embodiment, of the volumes obtained from optical scanning following a series of cycles, the front projection image of the first subvolume (corresponding to the first cycle group) without the intervention of relatively large movements is the first. The front projection image of the second subvolume, which is set to one image (first strip) and does not involve relatively large movements (corresponding to the second cycle group different from the first cycle group), is the second image (second image). 2 strips).

Regarding the first image and the second image constructed by the image construction unit 220, the mask image generation unit 231 generates a first mask image corresponding to the first image and a second mask image corresponding to the second image. It is configured as follows. The "first mask image" and the "second mask image" referred to here are the "first mask image" and the "second mask image" corresponding to the order given to the plurality of strips in the above exemplary embodiment. It is different from "mask image".

In some exemplary embodiments, the range of pixel values of the mask image (first mask image, second mask image) may be included in the closed interval [0,1].

Further, in some exemplary embodiments, the mask image (first mask image, second mask image) may be a binary image with a pixel value of 0 or 1.

For example, as in the above exemplary embodiment, the mask image (first mask image, second mask image) has a pixel value of 1 in the region corresponding to the domain of the sample image, and other mask images. It may be a binary image in which the pixel value is 0.

In some exemplary embodiments, the first mask image and the second mask image may have the same dimensions and the same shape.

The range adjusting unit 232 adjusts the pixel value range of the first image and the pixel value range of the second image based on the pixel value range of the first mask image and the pixel value range of the second mask image. It is configured.

In some exemplary embodiments, the range adjuster 232 sets the pixel value range of the sample image (first image, second image) and the pixel value of the mask image (first mask image, second mask image). It may be configured to reduce the difference from the range. For example, as in the above exemplary embodiment, the range adjusting unit 232 uses the pixel value range of the sample image (first image, second image) and the mask image (first mask image, second mask image). It may be configured to match the range of pixel values of.

Further, in some exemplary embodiments, the range adjuster 232 ranges the pixel values of one of the sample image (first image, second image) and mask image (first mask image, second mask image). May be configured to match the range of the other pixel value. For example, as in the above exemplary embodiment, the range adjusting unit 232 masks the pixel value range of the sample image (first image, second image) as a mask image (first mask image, second mask image). It may be configured to match the range of pixel values of. That is, the range adjusting unit 232 is for matching the pixel value range of the sample image (first image, second image) with the pixel value range of the mask image (first mask image, second mask image). In addition, the range of pixel values of the sample images (first image, second image) may be changed.

In some exemplary embodiments, the range adjuster 232 has pixels of the sample image (first image, second image) depending on the range of pixel values of the mask image (first mask image, second mask image). It may be configured to normalize (standardize) the range of values.

For example, when the range of pixel values of the mask image (first mask image, second mask image) is included in the closed section [0,1], the range adjustment unit 232 uses the sample image (first image, first image, The value of each pixel of the second image) may be configured to normalize the range of pixel values of the image by dividing by the maximum pixel value of the image.

In another example, when the range of pixel values of the mask image (first mask image, second mask image) is included in the closed section [0,1], the range adjustment unit 232 uses the sample image (first mask image, second mask image). The value of each pixel of the 1st image and the 2nd image) may be divided by the maximum value of the pixel value range of the image to normalize the pixel value range of the image.

The composite image generation unit 233 synthesizes the first mask image with the first image of the sample to generate the first composite image, and synthesizes the second mask image with the second image to generate the second composite image. It is configured as follows.

In some exemplary embodiments, the first mask image and the second mask image may have the same dimensions and the same shape. Further, the first mask image may be a binary image in which the value of the pixel in the region corresponding to the domain of the first image is 1 and the value of the other pixel is 0, and the second mask. The image may be a binary image in which the value of the pixel in the region corresponding to the domain of the second image is 1 and the value of the other pixel is 0.

In the above exemplary embodiment, the first mask image and the second mask image have the same dimensions and the same shape. The composite image generation unit 233 is configured to convert each of the first image having an arbitrary shape and the second image having an arbitrary shape into embedded images having the same dimensions and the same shape as the mask images. Further, the composite image generation unit 233 is configured so that the embedded image of the first image and the first mask image are combined, and the embedded image of the second image and the second mask image are combined.

Alternatively, in the above exemplary embodiment, the composite image generation unit 233 is configured to convert the first image of the arbitrary shape and the second image of the arbitrary shape into embedded images having the same dimensions and the same shape. You may be. Further, the mask image generation unit 231 may be configured to generate a first mask image and a second mask image having the same dimensions and the same shape as the images of these samples, respectively. Further, the composite image generation unit 233 may be configured so that the embedded image of the first image and the first mask image are combined, and the embedded image of the second image and the second mask image are combined.

The cross-correlation function calculation unit 234 is configured to obtain a plurality of cross-correlation functions based on the first composite image and the second composite image generated by the composite image generation unit 233.

The correlation coefficient calculation unit 235 is configured to calculate the correlation coefficient based on a plurality of cross-correlation functions obtained by the cross-correlation function calculation unit 234.

The registration unit 236 is configured to perform registration between the first image and the second image based on the correlation coefficient calculated by the correlation coefficient calculation unit 235.

The merge processing unit 237 is configured to construct a merged image of the first image and the second image based on the result of the registration executed by the registration unit 236.

Similar to the above exemplary embodiments, in some exemplary embodiments, the cross-correlation function calculation unit 234 (first calculation unit), the correlation coefficient calculation unit 235 (second calculation unit), and the registration unit 236. , And the processing executed by each of the merging processing unit 237 and the configuration (hardware configuration, software configuration) for the same may be the same as the method described in Non-Patent Document 1. However, it does not exclude the adoption of other methods.

Some aspects of the image processing method that can be provided by the ophthalmic apparatus 1 according to the above exemplary embodiment will be described. In some exemplary embodiments, the image processing method may include the steps described below.

(Image preparation process)
A first image and a second image of the sample are prepared. For example, the first image and the second image of the sample are accepted via a communication line or a recording medium. Alternatively, light scanning is applied to the sample to collect data and construct the first and second images. Each of the first and second images is constructed based on the data collected by applying a scan according to a two-dimensional pattern involving a series of cycles to the sample. For example, each of the first image and the second image is constructed based on the data collected by applying a scan according to a two-dimensional pattern based on a preset scanning protocol based on the resage function to the sample.

(Mask preparation process)
A mask image is prepared. For example, a mask image is accepted via a communication line or a recording medium. Alternatively, a mask image is generated (based on the sample image). The range of pixel values of the mask image may be included in the closed interval [0,1]. Typically, the mask image may be a binary image with a pixel value of 0 or 1. For example, the mask image may be a binary image in which the value of the pixel in the region corresponding to the domain of the sample image is 1 and the value of the other pixel is 0. The mask image may include a first mask image and a second mask image. The first mask image and the second mask image may have the same dimensions and the same shape. The first mask image may be a binary image in which the value of the pixel in the region corresponding to the domain of the first image of the sample is 1 and the value of the other pixel is 0, and the second mask. The image may be a binary image in which the value of the pixel in the region corresponding to the domain of the second image of the sample is 1 and the value of the other pixel is 0.

(Image value range adjustment process)
The pixel value range of the first image, the pixel value range of the second image, and the pixel value range of the mask image are relatively adjusted. Relative adjustment of the pixel value range typically reduces the difference between the pixel value range of the sample image (first and second images) and the pixel value range of the mask image. It is done in. For example, the relative adjustment of the pixel value range is performed so that the pixel value range of one of the sample images (first image, second image) and the mask image matches the pixel value range of the other. In some exemplary embodiments, the relative adjustment of the pixel value range normalizes the pixel value range of the sample image (first image, second image) according to the pixel value range of the mask image. It may include the processing to be performed. When the range of pixel values of the mask image is included in the closed interval [0,1], the normalization sets the value of each pixel of the sample image (first image, second image) to the maximum pixel value in the image. It may include a process of dividing by. Alternatively, when the range of pixel values of the mask image is included in the closed interval [0,1], the normalization sets the value of each pixel of the sample image (first image, second image) to the pixel of the image. It may include a process of dividing by the maximum value in the range of values.

(Composite image generation process)
After the image value range adjusting step, the mask image is combined with each of the first image and the second image to generate the first composite image and the second composite image. When the mask image includes the first mask image and the second mask image, the composite image generation step includes a step of synthesizing the first mask image with the first image to generate the first composite image and a second mask. It may include a step of synthesizing an image with a second image to generate a second composite image. Here, the first mask image and the second mask image may have the same dimensions and the same shape. Further, the first mask image may be a binary image in which the value of the pixel in the region corresponding to the domain of the first image is 1 and the value of the other pixel is 0, and the second mask. The image may be a binary image in which the value of the pixel in the region corresponding to the domain of the second image is 1 and the value of the other pixel is 0.

(Cross-correlation function calculation process)
A plurality of cross-correlation functions are obtained based on the first composite image and the second composite image generated by the composite image generation step. This step is typically performed according to the method described in Non-Patent Document 1, but is not limited thereto.

(Correlation coefficient calculation process)
The correlation coefficient is calculated based on a plurality of cross-correlation functions obtained in the cross-correlation function calculation step. This step is typically performed according to the method described in Non-Patent Document 1, but is not limited thereto.

(Registration process)
Registration is performed between the first image and the second image of the sample based on the correlation coefficient calculated in the correlation coefficient calculation step. This step is typically performed according to the method described in Non-Patent Document 1, but is not limited thereto.

In some exemplary embodiments, the image processing method may further include a merged image construction step. In the merged image construction step, the first image and the second image are combined based on the result of the registration performed in the registration step (for example, the relative shift amount between the first image and the second image of the sample). The merged image is constructed.

In some exemplary embodiments, the image processing method may further comprise any of the steps described in the exemplary embodiments described above.

Some aspects of the scanning imaging method that can be provided by the ophthalmic apparatus 1 according to the above exemplary embodiment will be described. In some exemplary embodiments, the scanning imaging method may include the steps described below. In some exemplary embodiments, the scanning imaging method is described in any of the steps described in the exemplary embodiments above and / or in the exemplary embodiments of the image processing method described above. Any of the above steps may be further included.

(Scanning process)
An optical scan that follows a two-dimensional pattern involving a series of cycles is applied to the sample and data is collected. Typically, any two cycles of the series intersect each other at at least one point. For example, optical scanning according to a two-dimensional pattern is performed based on a preset scanning protocol based on a Lissajous function.

(Image construction process)
The first image and the second image are constructed based on the data collected in the scanning process. When any two cycles of the series intersect each other at at least one point (eg, when optical scanning is performed according to a two-dimensional pattern based on a preset scanning protocol based on the Lissajous function). The first image may be constructed based on the first data collected by the optical scan according to the first cycle group of the cycles, and the optical scan according to the second cycle group different from the first cycle group in the series of cycles. A second image may be constructed based on the second data collected in.

(Image value range adjustment process)
The pixel value range of the first image, the pixel value range of the second image, and the pixel value range of the mask image are relatively adjusted.

(Composite image generation process)
After the image value range adjusting step, the mask image is combined with each of the first image and the second image to generate the first composite image and the second composite image.

(Cross-correlation function calculation process)
A plurality of cross-correlation functions are obtained based on the first composite image and the second composite image generated in the composite image generation step.

(Correlation coefficient calculation process)
The correlation coefficient is calculated based on a plurality of cross-correlation functions obtained in the cross-correlation function calculation step.

(Registration process)
Registration is performed between the first image and the second image of the sample based on the correlation coefficient calculated in the correlation coefficient calculation step.

(Merge image construction process)
A merged image of the first and second images of the sample is constructed based on the registration result.

In some exemplary embodiments, the scanning imaging method may include the following additional steps:

In the first additional step, a third image is constructed based on the data collected from the sample in the scanning step. In the above exemplary embodiment, a third strip is constructed.

In the second additional step, the pixel value range of the third image, the pixel value range of the merged image constructed in the merged image construction step, and the pixel value range of the mask image are relatively adjusted. To. In the above exemplary embodiment, the third strip, the mask image corresponding to the third strip, the merged image of the first strip and the second strip, and the mask image corresponding to the merged image. , Pixel value range adjustment is applied.

In the third additional step, the mask image is combined with the third image and the merged image, respectively, to generate the third composite image and the fourth composite image. In the above exemplary embodiment, a composite image of the third strip and the corresponding mask image is generated, and the merged image of the first and second strips and the corresponding mask image A composite image of is generated.

In the fourth additional step, a plurality of cross-correlation functions are obtained based on the third composite image and the fourth composite image. In the above exemplary embodiment, a plurality of cross-correlation functions are based on a composite image of the third strip and the corresponding mask image, and a composite image of the merged image and the corresponding mask image. Is required.

In the fifth additional step, the correlation coefficient is calculated based on the plurality of cross-correlation functions obtained in the fourth additional step. In the above exemplary embodiment, the correlation coefficient is calculated based on a plurality of cross-correlation functions obtained using the composite image based on the third strip and the composite image based on the merged image.

In the sixth additional step, registration between the third image and the fourth image is performed based on the correlation coefficient calculated in the fifth additional step. In the above exemplary embodiment, the merged image (first strip and second strip) is based on the correlation coefficient calculated using the composite image based on the third strip and the composite image based on the merged image. Registration between the merged image) and the third strip is performed.

In the seventh additional step, the third image and the merged image are merged based on the result of the registration performed in the sixth additional step. In the above exemplary embodiment, the merged image (the merged image of the first and second strips) is based on the result of registration (relative shift amount) between the merged image and the third strip. The third strip is merged against.

Here, the processing for the merged image of the first strip and the second strip and the third strip has been described as an example, but generally, as described in the above exemplary embodiment, the first to first strips have been described. The processing for the merged image of the nth strip and the n + 1th strip is also executed in the same manner (n = 2, 3, ..., N-1).

Some aspects of the image processing apparatus that can be provided by the ophthalmic apparatus 1 according to the above exemplary embodiment will be described. In some exemplary embodiments, the image processing apparatus may include the elements described below. In some exemplary embodiments, the image processing apparatus performs any of the elements described in the exemplary embodiments above, the steps described in the exemplary embodiments of the image processing method. Further comprising any of the elements configured as described above and any of the elements configured to perform the steps described in the exemplary embodiments of the scanning imaging method described above. You may.

The image processing apparatus according to the exemplary embodiment includes a storage unit, a mask image generation unit, a range adjustment unit, a composite image generation unit, a first calculation unit, a second calculation unit, and a registration unit. ..

The storage unit stores the first image and the second image of the sample. In the above exemplary embodiment, this storage unit corresponds to the storage unit 212.

The mask image generation unit is configured to generate a first mask image corresponding to the first image and a second mask image corresponding to the second image. In the above exemplary embodiment, the mask image generation unit corresponds to the mask image generation unit 231.

The range adjusting unit is configured to adjust the pixel value range of the first image and the pixel value range of the second image based on the pixel value range of the first mask image and the pixel value range of the second mask image. Will be done. In the above exemplary embodiment, the range adjusting unit corresponds to the range adjusting unit 232.

The composite image generation unit is configured to synthesize the first mask image with the first image to generate the first composite image, and to synthesize the second mask image with the second image to generate the second composite image. Will be done. In the above exemplary embodiment, the composite image generation unit corresponds to the composite image generation unit 233.

The first calculation unit is configured to obtain a plurality of cross-correlation functions based on the first composite image and the second composite image. In the above exemplary embodiment, the first calculation unit corresponds to the cross-correlation function calculation unit 234.

The second calculation unit is configured to calculate the correlation coefficient based on a plurality of cross-correlation functions. In the above exemplary embodiment, the second calculation unit corresponds to the correlation coefficient calculation unit 235.

The registration unit is configured to perform registration between the first image and the second image based on the correlation coefficient. In the above exemplary embodiment, the registration unit corresponds to the registration unit 236.

In some exemplary embodiments, the image processing apparatus may further include a merge processing unit. The merge processing unit is configured to construct a merged image of the first image and the second image based on the registration result.

In some exemplary embodiments, the image processing apparatus may further include a data receiving unit and an image building unit.

The data receiving unit is configured to receive data collected from a sample by optical scanning according to a two-dimensional pattern including a series of cycles. The data receiving unit may be configured to receive data via, for example, a communication line or a recording medium. In the above exemplary embodiment, the data receiving unit corresponds to a communication device provided in the ophthalmic apparatus 1.

The image construction unit constructs the first image and the second image based on the data received by the data reception unit. In the above exemplary embodiment, the image construction unit corresponds to the image construction unit 220. The first image and the second image constructed by the image construction unit are stored in the storage unit.

In some exemplary embodiments, the data receiver is configured to accept first and second images constructed based on data collected from a sample by optical scanning according to a two-dimensional pattern involving a series of cycles. You can. In this case, the first image and the second image received by the data receiving unit are stored in the storage unit.

In some exemplary embodiments, the data receiving unit may be configured to receive 3D image data constructed based on data collected from a sample by optical scanning according to a 2D pattern involving a series of cycles. In this case, the image construction unit is configured to construct the first image and the second image based on the three-dimensional image data received by the data reception unit. The first image and the second image constructed by the image construction unit are stored in the storage unit.

Some aspects of the image processing apparatus control method that can be provided by the ophthalmic apparatus 1 according to the above exemplary embodiment will be described. In some exemplary embodiments, the control method of the image processing apparatus may include the steps described below. In some exemplary embodiments, the control method of the image processing apparatus is any of the steps described in the above exemplary embodiments, or the steps described in the exemplary embodiments of the image processing method. Any of the steps described in the exemplary aspects of the scanning imaging method described above, and any of the steps performed by the elements described in the exemplary embodiments of the image processing apparatus above. Any of these may be further included.

An exemplary embodiment is a method of controlling an image processing unit including a storage unit for storing first and second images of a sample and a processor, which controls the processor to perform the following steps. Do. In the above exemplary embodiment, the storage unit corresponds to the storage unit 212, and the processor corresponds to at least the data processing unit 230.

The processor is controlled to generate a first mask image corresponding to the first image and a second mask image corresponding to the second image. For example, in the above exemplary embodiment, the data processing unit 230 is controlled to operate as the mask image generation unit 231.

Further, the processor controls to adjust the pixel value range of the first image and the pixel value range of the second image based on the pixel value range of the first mask image and the pixel value range of the second mask image. Will be done. For example, in the above exemplary embodiment, the data processing unit 230 is controlled to operate as the range adjusting unit 232.

Further, the processor is controlled so as to synthesize the first mask image with the first image to generate the first composite image, and to synthesize the second mask image with the second image to generate the second composite image. To. For example, in the above exemplary embodiment, the data processing unit 230 is controlled to operate as a composite image generation unit 233.

Further, the processor is controlled to obtain a plurality of cross-correlation functions based on the first composite image and the second composite image. For example, in the above exemplary embodiment, the data processing unit 230 is controlled to operate as the cross-correlation function calculation unit 234.

In addition, the processor is controlled to calculate the correlation coefficient based on a plurality of cross-correlation functions. For example, in the above exemplary embodiment, the data processing unit 230 is controlled to operate as the correlation coefficient calculation unit 235.

Further, the processor is controlled to perform registration between the first image and the second image based on the correlation coefficient. For example, in the above exemplary embodiment, the data processing unit 230 is controlled to operate as the registration unit 236.

In some exemplary embodiments, the processor may be controlled to build a merged image of the first and second images based on the results of registration. For example, in the above exemplary embodiment, the data processing unit 230 is controlled to operate as the merge processing unit 237.

Some aspects of the control method of the scanning imaging apparatus that can be provided by the ophthalmologic apparatus 1 according to the above exemplary embodiment will be described. In some exemplary embodiments, the method of controlling the scanning imaging apparatus may include the steps described below. In some exemplary embodiments, the control method of the scanning imaging apparatus has been described in any of the steps described in the above exemplary embodiments, or in the exemplary embodiment of the image processing method described above. Any of the steps, any of the steps described in the exemplary embodiments of the scanning imaging method, any of the steps performed by the elements described in the exemplary embodiments of the image processing apparatus, and. , Any of the steps described in the exemplary aspects of the image processing apparatus described above may be further included.

An exemplary embodiment is a method of controlling a scanning imaging apparatus including a scanning unit that applies optical scanning to a sample to collect data and a processor, such that the scanning unit and the processor perform the following steps. To control. In the above exemplary embodiment, the scanning unit includes the OCT unit 100 and the elements (retroreflector 41, OCT focusing lens 43, optical scanner 44, objective lens 22) in the fundus camera unit 2 constituting the measurement arm. Etc.) and include. The processor corresponds to the image construction unit 220 and the data processing unit 230.

The scanning unit is controlled to apply optical scanning according to a two-dimensional pattern including a series of cycles to the sample to collect data.

In addition, the processor (image construction unit 220 in the above exemplary embodiment) is controlled to construct the first image and the second image based on the data collected by the scanning unit.

Further, the processor (data processing unit 230 in the above exemplary embodiment) is controlled to generate a first mask image corresponding to the first image and a second mask image corresponding to the second image.

Further, the processor (data processing unit 230 in the above exemplary embodiment) has a range of pixel values of the first image and a range of pixel values of the first image based on the range of pixel values of the first mask image and the range of pixel values of the second mask image. It is controlled to adjust the range of pixel values of the second image.

Further, the processor (data processing unit 230 in the above exemplary embodiment) synthesizes the first mask image with the first image to generate the first composite image, and combines the second mask image with the second image. It is controlled to combine to generate a second composite image.

Further, the processor (data processing unit 230 in the above exemplary embodiment) is controlled to obtain a plurality of cross-correlation functions based on the first composite image and the second composite image.

Further, the processor (data processing unit 230 in the above exemplary embodiment) is controlled to calculate the correlation coefficient based on a plurality of cross-correlation functions.

Further, the processor (data processing unit 230 in the above exemplary embodiment) is controlled to perform registration between the first image and the second image based on the correlation coefficient.

Further, the processor (data processing unit 230 in the above exemplary embodiment) is controlled to construct a merged image of the first image and the second image based on the registration result.

In some exemplary embodiments, a program that causes the computer to execute any aspect of the method of controlling the scanning imaging apparatus, a program that causes the computer to execute any aspect of the method of controlling the image processing apparatus, and image processing. It is possible to provide a program that causes a computer to execute any aspect of the method, or a program that causes a computer to execute any aspect of the scanning imaging method.

It is also possible to create a computer-readable non-temporary recording medium on which such a program is recorded. The non-temporary recording medium may be in any form, and examples thereof include a magnetic disk, an optical disk, a magneto-optical disk, and a semiconductor memory.

According to some exemplary embodiments described above, in image registration using a mask image, the pixel value range of the sample image and the pixel value range of the mask image are relatively adjusted. , It is possible to perform processing for registration.

Thereby, the difference between the pixel value range of the sample image and the pixel value range of the mask image can be reduced. For example, it is possible to (almost) match the size of the pixel value of the sample image with the size of the pixel value of the mask image.

Therefore, it is possible to solve the conventional problem that the effect of the mask image is ignored in the correlation calculation using the sample image and the mask image, and it is possible to obtain a correct correlation coefficient.

In particular, even when a single-precision floating-point type (float type) operation is used to obtain the correlation coefficient between sample images, the effect of rounding error due to the small number of significant digits should be eliminated. It can be made smaller.

Further, in the algorithm for calculating the correlation coefficient from a plurality of cross-correlation functions, the error of the final correlation coefficient may become large due to the superposition of the errors caused by the individual cross-correlation functions. With respect to such a problem, according to an exemplary embodiment, it is possible to eliminate or reduce the error caused by each cross-correlation function, so that it is possible to obtain a highly accurate correlation coefficient.

As described above, some of the exemplary embodiments described above contribute to the improvement of the accuracy of image registration using the mask image.

The present disclosure merely illustrates some aspects and is not intended to limit the invention. A person who intends to carry out the present invention can make arbitrary modifications (omission, substitution, addition, etc.) within the scope of the gist of the present invention.

1 Ophthalmology device 44 Optical scanner 100 OCT unit 211 Main control unit 2111 Scanning control unit 212 Storage unit 2121 Scanning protocol 220 Image construction unit 230 Data processing unit 231 Mask image generation unit 232 Range adjustment unit 233 Synthetic image generation unit 234 Cross-correlation function calculation Part 235 Correlation coefficient calculation part 236 Registration part 237 Merge processing part

Claims (25)

Prepare the first and second images of the sample,
Prepare the mask image and
The pixel value range of the first image, the pixel value range of the second image, and the pixel value range of the mask image are relatively adjusted.
The mask image is combined with each of the first image and the second image to generate a first composite image and a second composite image.
A plurality of cross-correlation functions were obtained based on the first composite image and the second composite image.
The correlation coefficient is calculated based on the plurality of cross-correlation functions, and the correlation coefficient is calculated.
Registration between the first image and the second image is performed based on the correlation coefficient.
Image processing method. To reduce the difference between the pixel value range of the sample image and the pixel value range of the mask image.
The image processing method of claim 1. Match the range of one pixel value of the sample image and the mask image to the range of the other pixel value.
The image processing method of claim 2. Normalize the pixel value range of the sample image according to the pixel value range of the mask image.
The image processing method of claim 3. The range of pixel values of the mask image is included in the closed interval [0,1].
Divide the value of each pixel in the sample image by the maximum pixel value in the image.
The image processing method of claim 4. The range of pixel values of the mask image is included in the closed interval [0,1].
Divide the value of each pixel of the sample image by the maximum value in the pixel value range of the image.
The image processing method of claim 4. The mask image is a binary image having a pixel value of 0 or 1.
The image processing method according to any one of claims 1 to 6. The mask image is a binary image in which the value of a pixel in the region corresponding to the domain of the sample image is 1 and the value of the other pixel is 0.
The image processing method of claim 7. The mask image includes a first mask image and a second mask image.
The first mask image is combined with the first image to generate the first composite image.
The second mask image is combined with the second image to generate the second composite image.
The image processing method according to any one of claims 1 to 8. The first mask image and the second mask image have the same dimensions and the same shape.
The image processing method of claim 9. The first mask image is a binary image in which the value of the pixel in the region corresponding to the domain of the first image is 1 and the value of the other pixel is 0.
The second mask image is a binary image in which the value of the pixel in the region corresponding to the domain of the second image is 1 and the value of the other pixel is 0.
The image processing method according to claim 9 or 10. Each of the first image and the second image is constructed based on the data collected by applying a scan according to a two-dimensional pattern including a series of cycles to the sample.
The image processing method according to any one of claims 1 to 11. Each of the first image and the second image is constructed based on data collected by applying a scan according to a two-dimensional pattern based on a preset scanning protocol based on a resage function to the sample.
The image processing method according to claim 12. Data is collected by applying an optical scan to the sample that follows a two-dimensional pattern involving a series of cycles.
The first image and the second image are constructed based on the above data, and the first image and the second image are constructed.
The pixel value range of the first image, the pixel value range of the second image, and the pixel value range of the mask image are relatively adjusted.
The mask image is combined with each of the first image and the second image to generate a first composite image and a second composite image.
A plurality of cross-correlation functions were obtained based on the first composite image and the second composite image.
The correlation coefficient is calculated based on the plurality of cross-correlation functions, and the correlation coefficient is calculated.
Registration between the first image and the second image is performed based on the correlation coefficient.
A merged image of the first image and the second image is constructed based on the result of the registration.
Scanning imaging method. A third image is constructed based on the above data,
The pixel value range of the third image, the pixel value range of the merged image, and the pixel value range of the mask image are relatively adjusted.
The mask image is combined with each of the third image and the merged image to generate a third composite image and a fourth composite image.
A plurality of cross-correlation functions were obtained based on the third composite image and the fourth composite image.
The correlation coefficient is calculated based on the plurality of cross-correlation functions, and the correlation coefficient is calculated.
Registration between the third image and the fourth image is performed based on the correlation coefficient.
The third image and the merged image are merged based on the result of the registration.
The scanning imaging method of claim 14. Any two cycles of the series of cycles intersect each other at at least one point.
The first image is constructed based on the first data collected by optical scanning according to the first cycle group of the series of cycles.
The second image is constructed based on the second data collected by optical scanning according to the second cycle group different from the first cycle group in the series of cycles.
The scanning imaging method according to claim 14 or 15. Optical scanning according to the two-dimensional pattern is performed based on a preset scanning protocol based on the Lissajous function.
The scanning imaging method of claim 16. A storage unit that stores the first and second images of the sample,
A mask image generation unit that generates a first mask image corresponding to the first image and a second mask image corresponding to the second image.
A range adjusting unit that adjusts the pixel value range of the first image and the pixel value range of the second image based on the pixel value range of the first mask image and the pixel value range of the second mask image. ,
A composite image generation unit that synthesizes the first mask image with the first image to generate a first composite image, and synthesizes the second mask image with the second image to generate a second composite image. ,
A first calculation unit that obtains a plurality of cross-correlation functions based on the first composite image and the second composite image, and
A second calculation unit that calculates the correlation coefficient based on the plurality of cross-correlation functions, and
An image processing apparatus including a registration unit that performs registration between the first image and the second image based on the correlation coefficient. A data reception unit that accepts data collected from samples by optical scanning that follows a two-dimensional pattern that includes a series of cycles.
Further includes the first image and an image construction unit that constructs the second image based on the data.
The image processing apparatus according to claim 18. A method of controlling an image processing device including a storage unit for storing a first image and a second image of a sample and a processor.
The processor
A first mask image corresponding to the first image and a second mask image corresponding to the second image are generated.
The pixel value range of the first image and the pixel value range of the second image are adjusted based on the pixel value range of the first mask image and the pixel value range of the second mask image.
The first mask image is combined with the first image to generate a first composite image, and the second mask image is combined with the second image to generate a second composite image.
A plurality of cross-correlation functions were obtained based on the first composite image and the second composite image.
The correlation coefficient is calculated based on the plurality of cross-correlation functions, and
Control so as to perform registration between the first image and the second image based on the correlation coefficient.
Image processing device control method. A scanning unit that collects data by applying optical scanning that follows a two-dimensional pattern that includes a series of cycles to a sample.
An image construction unit that constructs the first image and the second image based on the data, and
A mask image generation unit that generates a first mask image corresponding to the first image and a second mask image corresponding to the second image.
A range adjusting unit that adjusts the pixel value range of the first image and the pixel value range of the second image based on the pixel value range of the first mask image and the pixel value range of the second mask image. ,
A composite image generation unit that synthesizes the first mask image with the first image to generate a first composite image, and synthesizes the second mask image with the second image to generate a second composite image. ,
A first calculation unit that obtains a plurality of cross-correlation functions based on the first composite image and the second composite image, and
A second calculation unit that calculates the correlation coefficient based on the plurality of cross-correlation functions, and
A registration unit that performs registration between the first image and the second image based on the correlation coefficient, and
A scanning imaging apparatus including a merge processing unit that constructs a merged image of the first image and the second image based on the result of the registration. The scanning unit includes deflectors capable of deflecting light in different first and second directions, and follows the second direction while repeating changes in the deflection direction along the first direction in the first cycle. Optical scanning according to the two-dimensional pattern is applied to the sample by repeating the change in the deflection direction in a second cycle different from the first cycle.
The scanning imaging apparatus according to claim 21. A method of controlling a scanning imaging device that includes a scanning unit that applies optical scanning to a sample to collect data and a processor.
The scanning unit is controlled to apply optical scanning according to a two-dimensional pattern including a series of cycles to the sample to collect data.
The processor
The first image and the second image are constructed based on the above data, and the first image and the second image are constructed.
A first mask image corresponding to the first image and a second mask image corresponding to the second image are generated.
The pixel value range of the first image and the pixel value range of the second image are adjusted based on the pixel value range of the first mask image and the pixel value range of the second mask image.
The first mask image is combined with the first image to generate a first composite image, and the second mask image is combined with the second image to generate a second composite image.
A plurality of cross-correlation functions were obtained based on the first composite image and the second composite image.
The correlation coefficient is calculated based on the plurality of cross-correlation functions, and the correlation coefficient is calculated.
Registration between the first image and the second image is performed based on the correlation coefficient, and
Control to construct a merged image of the first image and the second image based on the registration result.
How to control a scanning imaging device. A program that causes a computer to execute the method of claims 1 to 17, 20 or 23. A computer-readable non-temporary recording medium on which the program of claim 24 is recorded.

Images