JP7058901B1 - 3D imager - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a three-dimensional image pickup apparatus capable of simultaneously realizing three-dimensional resolution and detection of a spectroscopic image for a subject with a simple structure. SOLUTION: A three-dimensional image pickup device combines a light source that supplies illumination light for illuminating a subject by sweeping the frequency of light or the frequency of amplitude modulation of light, and the reflected light and reference light from the subject. Interference fringes depending on the combination of an optical interferometer that generates interference fringes, a two-dimensional array of light receiving elements, a one-dimensional array of light receiving elements and one-dimensional scanning, and a single light receiving element and one-dimensional scanning combination. A two-dimensional detection mechanism that detects as an electric signal at a two-dimensional position, and an optical path difference calculation means that calculates the optical path difference between the reflected light and the reference light at the two-dimensional detection position of the two-dimensional detection mechanism for each three-dimensional pixel. And, the subject is resolved three-dimensionally by the processing using the information of the interference fringe and the optical path difference. [Selection diagram] Fig. 1

Description

The present invention relates to a three-dimensional image pickup device. In particular, the amplitude and phase of the reflected light are detected by the optical interferometry, and three-dimensional resolution is performed by electrical processing using the detection results. It relates to a three-dimensional image pickup device capable of recovering a deteriorated resolution and analyzing a spectrum.

There is a need to resolve the subject in three dimensions, detect the surface and internal images (cross-sectional image, tomographic image, transmission image) of the subject from the information, and measure the shape of the three-dimensional subject. Strong regardless of use or medical use. Therefore, as will be described later, there are various imaging techniques and measurement techniques.

Further, in recent years, there has been a growing need for non-contact analysis of the composition components of each three-dimensional pixel by analyzing the reflection spectrum at the same time as the above-mentioned three-dimensional imaging (for example, for both industrial and medical applications). , Patent Document 1).

(3D shape detection technology)
As a technique for measuring a three-dimensional shape in a non-contact manner, a focal movement method, a confocal movement method, an optical interference method, a fringe projection method, and the like are known.

(Spectroscopic image detection technology)
As a spectroscopic image detection technique, a hyperspectral camera using a line spectroscopic method is known.

Japanese Unexamined Patent Publication No. 2006-153654

As described above, there are various imaging techniques and measurement techniques for detecting a spectroscopic image in order to analyze the composition component of the subject by analyzing the reflection spectrum.

However, if an attempt is made to construct an image pickup device capable of simultaneously performing three-dimensional resolution and detection of a spectroscopic image by combining these three-dimensional shape detection techniques and spectroscopic image detection techniques, the following problems will occur in either combination. It is difficult to realize.
・ In principle, it is impossible to combine
・ The scale of hardware will increase, the structure will become complicated,
・ The processing time will be enormous.
-The detection accuracy is significantly reduced.

For this reason, there is no existing image pickup device that simultaneously achieves high-precision three-dimensional resolution and detection of a spectroscopic image with high spectral accuracy.

The present invention has been made in view of such circumstances. That is, an object of the present invention is to provide a three-dimensional image pickup apparatus capable of simultaneously realizing three-dimensional resolution and detection of a spectroscopic image for a subject with a simple structure.

In order to solve the above-mentioned problems and achieve the object, the three-dimensional image pickup apparatus according to at least some embodiments of the present invention sweeps the frequency of light or the frequency of amplitude modulation of light to illuminate the subject. Light source, an optical interferometer that combines reflected light and reference light from the subject to generate interference fringes, a two-dimensional array of light-receiving elements, a one-dimensional array of light-receiving elements and a combination of one-dimensional scanning, and a single light-receiving device. Depending on the combination of the element and the one-dimensional scanning, the two-dimensional detection mechanism that detects the interference fringes as an electric signal at the two-dimensional position, and the optical path of the reflected light and the reference light at the two-dimensional detection position of the two-dimensional detection mechanism. It is provided with an optical path difference calculation means for calculating the difference for each three-dimensional pixel, and the subject is resolved three-dimensionally by processing using the information of the interference fringes and the optical path difference.

Further, according to the second preferred embodiment of the present embodiment, a Fourier transform processing unit that Fourier transforms the interference fringe signal to resolve the light receiving direction and detects a three-dimensional data string of a complex signal of amplitude and phase.
Data that matches the optical path length from the two-dimensional detection position to the pixel to be resolved is extracted from the three-dimensional data string using the optical path difference, and the filter coefficient calculated from the optical path difference is superimposed and integrated to receive the light receiving direction. It is provided with a two-dimensional filter processing unit that resolves a surface intersecting with.

Further, according to the third preferred embodiment of the present embodiment, a storage unit for storing a three-dimensional data string and a storage unit.
An address generation unit that uses the optical path difference to generate an address that matches the optical path length from the detection position of the two-dimensional detection mechanism to the reflection point to be resolved from the storage unit.
A filter coefficient generator that reads data using the address, interpolates the data in the light receiving direction, matches the initial phase, and weights the aperture of the image.
A low-pass filter unit that superimposes and integrates the filter coefficient on complex signal data,
To prepare for.

Further, according to the fourth preferred embodiment of the present embodiment, the opening for which the two-dimensional filter processing is performed is divided into a plurality of blocks, and each block is centered on the pixel to be resolved by the same processing as the two-dimensional filter processing. The disturbance of the light wave surface is detected from the mutual correlation calculation of the complex signal data of the neighboring pixels obtained in each block and reflected in the address generation of the address part. A correction means for correcting the above is provided.

Further, according to the fifth preferred embodiment of the present embodiment, there is provided a correction means for detecting the distortion and fluctuation of the frequency sweep of the light source and correcting the dispersion of the frequency component of the interference fringe caused by the distortion by the phase matching filter.

Further, according to the sixth preferred embodiment of the present embodiment, the spectral components are calculated from the reflection spectrum of the subject whose cluster is known in descending order of the Fisher ratio, and the spectral component is used from the reflection spectrum of the subject whose cluster is unknown. It is provided with an identification means for identifying a subject.

Further, according to the seventh preferred embodiment of the present embodiment, the identification means uses an AI that performs deep learning.

Further, according to the eighth preferred embodiment of the present embodiment, instead of the light source, a low coherence light source and a spectroscope are provided, the interference fringe signal is detected, and the Fourier transform processing unit and the two-dimensional structure are provided. Three-dimensional resolution is performed by the filter processing unit.

Further, according to the ninth preferred embodiment of the present embodiment, a data format creation unit that adds information necessary for three-dimensional resolution and spectral analysis to the interference fringe signal detected by the two-dimensional detection mechanism, and three. It is provided with a storage unit for storing interference fringe signals to which information necessary for dimensional resolution and spectrum analysis is added as RAW data.

Further, according to the tenth preferred embodiment of the present embodiment, the coherence degree of the light source, the band characteristics (including distortion) and the directivity of the frequency sweep, the coordinates of the detection position of the two-dimensional detection mechanism, and the directivity of the light receiving element. It includes three-dimensional coordinates of the emission position of the illumination light and the reference light with respect to the detection position of the two-dimensional detection mechanism, and information about the subject.

According to the present invention, it is possible to provide a three-dimensional image pickup apparatus capable of simultaneously realizing three-dimensional resolution and detection of a spectroscopic image for a subject with a simple structure.

It is a block diagram which shows the structure of the 3D image pickup apparatus which concerns on embodiment. It is a figure explaining the principle which causes the interference fringe in a 3D image pickup apparatus. It is another figure explaining the principle which causes the interference fringe in a 3D image pickup apparatus. It is a figure which shows the structure from a reflection point to a two-dimensional filter processing. It is a figure explaining the processing operation of 2D filter processing. (A)-(g) is a figure explaining the arrangement interval and directivity of a light receiving element. It is a figure explaining the structure which recovers the resolution deteriorated by the turbulence of a light wave surface by a two-dimensional filter processing. It is a figure which shows the structure which generates the RGB image from the RAW data of the interference fringe. It is a figure which shows the wavelength band of various spectrum images generated by a Fourier transform. It is a figure explaining the non-linear separation of the substance to be specified by AI. It is a figure explaining the case where the linearity of the frequency sweep of a laser light source is distorted. It is a figure explaining the structure which detects the linear distortion of a frequency sweep. It is a figure which shows the structure of the application example of this embodiment. (A) shows the imaged luminous flux when the image formation position of the reflection point is in front of the image pickup element, and (b) shows the image formation light flux when the image formation position of the reflection point is after the image pickup element. It is a figure which shows. It is a figure which shows the structure which used the low coherence light source in this embodiment. (a), (b), and (c) are diagrams showing an optical path when the reflected light at the reflection point is detected while scanning in the vertical direction of the paper surface, respectively. It is a figure which shows the focusing range necessary for diagnosing a coronary artery. It is a figure which shows the structure which applied this embodiment to the intravascular OCT apparatus. It is a figure explaining the method of detecting the image of the coronary artery using the intravascular OCT apparatus. It is a figure explaining the structure which applies this embodiment to X-ray image pickup and γ-ray image pickup. (A) is another diagram showing the configuration from the reflection point to the two-dimensional filter processing. (B) is a diagram showing a three-dimensional resolution process in which a plane perpendicular to the optical axis is resolved by an imaging lens and the light receiving direction is resolved by a Fourier transform process.

Prior to the description of the examples, the effects of the embodiments according to certain aspects of the present invention will be described. In addition, when concretely explaining the action and effect of this embodiment, a concrete example will be shown and explained. However, as in the case of the examples described later, those exemplary embodiments are merely a part of the embodiments included in the present invention, and there are many variations in the embodiments. Therefore, the present invention is not limited to the exemplary embodiments.

(Embodiment)
The three-dimensional image pickup apparatus according to the embodiment of the present invention detects the amplitude and phase of the reflected light by the optical interferometry, and performs three-dimensional resolution by electrical processing using the same. Then, the three-dimensional image pickup device performs focusing, recovery of the resolution deteriorated due to the disturbance of the light wave surface, and spectrum analysis for each three-dimensional pixel.

The three-dimensional image pickup device two-dimensionally detects the interference fringes of the reflected light generated by the optical interferometer. Next, the light receiving direction is resolved at each position detected two-dimensionally by the Fourier transform process described later. Then, using the amplitude and phase (hereinafter referred to as complex signal) of the reflected light obtained by the Fourier transform process and the optical path length to the reflection point, the resolution of the surface intersecting the light receiving direction is obtained by the two-dimensional filter process described later. conduct. The three-dimensional image pickup device resolves the subject in three dimensions by these two processes.

By the above two-dimensional filter processing, focusing (dynamic focusing) for each pixel and recovery of the resolution deteriorated due to the disturbance of the light wave surface, which will be described later, are performed. Further, the spectrum of the reflected light is analyzed by utilizing the frequency sweep of the illumination light used for the resolution processing in the light receiving direction, and the composition of the subject is identified for each pixel.

Further, the present embodiment is not limited to the visible light band, but has a configuration in which an imaging optical system does not exist, or an electromagnetic wave wavelength band that is expensive even if an imaging optical system exists, for example, infrared light or terahertz. It can also be applied to waves, millimeter waves, X-rays, γ-rays, and the like.

The basic configuration of the three-dimensional image pickup apparatus will be described with reference to FIG.
FIG. 1 is a configuration diagram showing a configuration of a three-dimensional image pickup apparatus according to an embodiment.
The light source 1 emits frequency-swept light within the imaging time. The sweeping light, which is the illumination light emitted from the light source 1, is separated by the beam splitter 2 of the optical interferometer 13. One of the sweep lights reflected by the dividing surface illuminates the subject 3. The other sweep light transmitted through the dividing surface is reflected by the mirror 4. The reference light reflected by the mirror 4 is combined with the reflected light 7 from the subject 3 by the beam splitter 2 to generate interference fringes.

The generated interference fringes are received by the light receiving element 8 (hereinafter referred to as "imaging element") having a two-dimensional arrangement. The method of detecting the interference fringe signal by the image pickup device 8 will be described later.

The interference fringe signal received by the image pickup device 8 is stored in the memory 5 as RAW data. Then, the interference fringe signal required for resolution is read from the memory 5, and the light receiving direction is resolved by the Fourier transform process 11.

After that, the surface perpendicular to the optical axis 9 is resolved by the two-dimensional filter processing 12 described later, and the reflection point 6 is detected three-dimensionally. The Fourier transform process 11 and the two-dimensional filter process 12 will be described later.

When the reflected light reflected from the reflection point 6 of the subject 3 is combined with the reference light by the beam splitter 2, interference fringes having a frequency proportional to the optical path difference between the reflected light at the reflection point 6 and the reference light are described later. Produces.

The above optical path difference is the optical path length of the reflected light until the illumination light emitted from the light source 1 is reflected at the reflection point 6 via the beam splitter 2 and is received by each light receiving element of the image pickup element 8, and the light source 1. It is the difference in the optical path length of the reference light from the beam splitter 2 to the light receiving element of the image pickup element 8 through the reflection mirror 4 and the beam splitter 2.

Therefore, interference fringes having a frequency corresponding to the position of the reflection point 6 and the position of each light receiving element of the image pickup element 8 are generated. Utilizing this, the reflection point 6 can be resolved three-dimensionally by the Fourier transform process 11 and the two-dimensional filter process 12, which will be described later.

Here, the light source 1 has spatial coherence (point light source property) required for resolution of a plane perpendicular to the optical axis 9, and frequency sweeping has linearity and frequency band required for resolution in the light receiving direction. Has temporal coherence. As a light source that satisfies these conditions, a frequency sweep type laser light source using a deflection element such as MEMS (Micro Electro-Mechanical System) or KTN (potassium niobate tantalate) and a spectroscope can be used.

Alternatively, the light source 1 is an incoherent light source (partial coherent light source) having spatial coherence (point light source property) necessary for resolving a surface perpendicular to the optical axis 9, and the amplitude of the emitted light is modulated by frequency sweep. The configuration may be made.

The former light source is used to resolve a small subject at a relatively short distance from the point of view of coherence length in three dimensions with high resolution. The latter light source is used when resolving a large subject at a long distance in three dimensions.

In FIG. 1, the image pickup device 8 is shown as a mechanism for detecting interference fringes in two dimensions. However, the mechanism is not limited to this, and a mechanism for detecting interference fringes two-dimensionally by a combination of a one-dimensional array of light-receiving elements and one-dimensional scanning, or a combination of a single light-receiving element and two-dimensional scanning may be used.

Further, if the optical path lengths of the reflected light and the reference light from the light source 1 to each light receiving element can be calculated without impairing the coherence, the optical system may be arranged in each optical path of the illumination light, the reflected light, and the reference light.

Further, the optical interferometer 13 may be arranged anywhere as long as it is on the light receiving path, and the beam splitter 2 may be arranged separately for the combined wave of the reference wave and for the separation of the illumination light.

The optical interferometer 13 of FIG. 1 shows a basic configuration for explaining the principle. Not limited to this, there are various types of optical interferometers, and the method can be selected according to the application. For example, an optical interferometer such as the Mirau method may be used to reduce the constituent dimensions.

Further, if the wavelength band is satisfied, an optical circulator using a Faraday rotator may be used in order to increase the efficiency of light utilization.

However, in any of the configurations, the optical system in the middle and the beam splitter 2 and the mirror 4 constituting the optical interferometer 13 do not impair the coherence of the illumination light, the reflected light, and the reference light, and the light source 1 to the image pickup element 8 are not impaired. The shape must be such that the optical path lengths of the reflected light and the reference light to each light receiving element can be calculated.

For example, the mirror 4 has a sufficiently small surface accuracy of 1/16 or less of the wavelength, has a focal point such as a concave surface, a convex surface, and an ellipsoidal surface in addition to a point reflector and a flat plate, and it is easy to calculate the optical path length. Things are used.

If the optical path lengths of the reflected light and the reference light from the light source 1 to each light receiving element 8 can be calculated without impairing the coherence, in any of the above configurations, the Fourier transform process 11 and the two-dimensional filter process 12 described later 3 Dimensional resolution can be performed.

(Explanation of Fourier transform processing)
The Fourier transform process 11 of FIG. 1 that resolves the light receiving direction will be described below.
When two lights having different frequencies and phases are combined, an interference fringe consisting of the frequency of the difference and the phase of the difference is generated. This is called optical heterodyne detection.

Optical heterodyne detection can convert carriers of very high frequency light into carriers of low frequency interference fringes. Then, the interference fringes holding the information on the amplitude and phase of the light can be converted into an electric signal by the light receiving element. Optical heterodyne detection can also be applied to detect the amplitude and phase of amplitude-modulated incoherent light.

FIG. 2 is a diagram illustrating a principle of generating interference fringes in a three-dimensional image pickup apparatus.
The Fourier transform process for resolving the light receiving direction is based on this principle of optical heterodyne detection. As shown in FIG. 2, a slight time difference (optical path difference) 14 is generated from the difference in the optical path length between the frequency-swept reference light 18 and the reflected light 19. This causes a slight time difference 15 in the frequency and phase of the reference light 18 and the reflected light 19. Then, an interference fringe consisting of the frequency of the difference and the phase of the difference is generated.

As can be seen from FIG. 2, if the linearity of the frequency sweep is high, the frequency of the difference and the phase of the difference become constant over the sweep time. Therefore, interference fringes having a constant frequency are generated according to the frequency sweep of the reflected light and the reference light.

When the optical path difference 14 between the reference light 18 and the reflected light 19 increases as shown by the dotted line 16, the frequency 15 of the interference fringes also increases as shown by the dotted line 17.

FIG. 3 is another diagram illustrating the principle of causing interference fringes in a three-dimensional image pickup apparatus.
Further, as shown in FIG. 3, when the bandwidth 21 of the frequency sweep to be swept is widened as shown by the dotted line 22, the frequency 23 of the interference fringe becomes higher as shown by the dotted line 24 even if the optical path difference 25 is the same.

When such an interference fringe signal is Fourier transformed, the frequency of the interference fringe is detected as a spectrum (complex signal) on the frequency axis. Here, the position of the spectrum on the frequency axis is proportional to the optical path difference between the reflected light from the light source (point light source) 1 to the light receiving element 8 in FIG. 1 and the reference light. Then, the distance from the light receiving element of the image pickup element 8 to the reflection point 6 can be detected.

The resolution of the spectrum (width of a single spectrum) is determined by the Fourier transform of the envelope of the frequency sweep. When the bandwidth 23 of the frequency sweep of the interference fringes in FIG. 3 is widened as shown by the dotted line 24, the number of spectra after the Fourier transform with respect to the optical path difference increases, so that the resolution in the light receiving direction can be increased.

Then, the above-mentioned processing can be applied as it is even when the amplitude modulation of the incoherent light is frequency-swept.
The reference light Es and the reflected light Er can be expressed by the following equations (1) and (2), respectively.
Es ＝ As × cos {2π [f0 ＋ (Δf / 2T) t] t ＋ θ ₀ } (1)
Er ＝ Ar × cos {2π [f0 ＋ (Δf / 2T) (t-td)] (t-td) ＋ θ ₀ } (2)
here,
Δf is the frequency sweep bandwidth,
T is the sweep time,
f0 is the frequency at which the sweep starts.
θ ₀ is the initial phase of the start of sweep,
t is the time,
td is the time difference between the reference light and the reflected light (optical path difference),
As the amplitude of the reference light,
Ar, the amplitude of the reflected light
Is.

When the reference light and the reflected light are combined and received by the light receiving element, the term including the high frequency of the light becomes the DC component from the frequency response (LPF: low pass filter) of the light receiving element, and only the term of the interference fringe of the low frequency component becomes. Remain. Then, the following equation (3) is obtained from the equation of the sum of products of trigonometric functions.
LPF [(Es ＋ Er) ² ] ＝ A × cos {2π [f0 ＋ (Δf / 2T) t] t-2π [f0 ＋ (Δf / 2T) (t-td)] (t-td)} ＋ K (3)
here,
A and K are constants determined by the amplitude As of the reference light and the amplitude Ar of the reflected light, respectively.

When the constants A and K are further removed from the equation (3), the following equation (4) is obtained.
cos {2π [2 (Δf / 2T) td] t + 2π [(Δf / 2T) td ² + f0 × td]} (4)

From the first term of the equation (4), it can be seen that 2 (Δf / 2T) td is the frequency of the interference fringe signal, and the frequency of the interference fringe changes linearly when the time difference (optical path difference) td changes.
Further, from the second term of the equation (4), 2π [(Δf / 2T) td ² + f0 × td] is the initial phase of the interference fringe signal, and the initial phase may change parabolic with respect to td. I understand.

When the frequency sweep envelope is a square wave, the spectral resolution is half-value full width = 1 / T of the sinc function ((sinx) / x) obtained by Fourier transforming the square wave. Therefore, when the time difference td corresponding to the spectral resolution is obtained, it becomes td = 1 / Δf from 2 (Δf / 2T) td = 1 / T. Therefore, the resolution ρ in the light receiving direction is the resolution, and considering that the speed of light in the propagation medium is C ₀ and the reflection is reciprocating, ρ = C _0/2 Δf.

The principle described above can be applied as it is even when an incoherent light source is used and frequency sweep is performed for the amplitude modulation thereof.

When the envelope is a square wave, a sidelobes of the sinc function obtained by Fourier transforming the square wave is generated. If the envelope is Gaussian (Gaussian function), the sidelobes can be suppressed, but the resolution will be slightly reduced, so the sweep bandwidth will be increased accordingly.

The three-dimensional point image distribution function (three-dimensional PSF (Point Spread Function)) when the interference fringe signal at the reflection point 6 is detected by the single light receiving element m shown in FIG. 1 and Fourier transformed is obtained by the curve 10 in FIG. As shown in the above, the point image distribution function in the light receiving direction 7 is expanded in a spherical shape within the directional range of the light receiving element m.

The notation "resolving the light receiving direction" in the description of the present application indicates that the complex signal of this three-dimensional point image distribution function is detected as a data string in the light receiving direction 7.

The coherence degree of the light source, the band characteristics and directionality of frequency sweep, the directionality of the light receiving element, the number of elements, and the arrangement spacing are used for the interference fringe signal obtained by each light receiving element or the complex signal resolved by Fourier transforming it. If the three-dimensional coordinates of the emission position of the illumination light and the reference light with respect to the light receiving surface of the image pickup element and the information of the subject are added and archived as RAW data, various processing using the phase information can be performed later. can.

It is technically difficult to detect a short time for the reflected light to reach the light receiving element from the reflection point. However, as described above, if the interference fringe with the reference light is detected and the Fourier transform is performed, a slight time difference (optical path difference) can be converted into a difference in the frequency of the interference fringe and detected.

Since the interference fringes generated by the optical interferometry are superimposed on the reflected light by the number of reflection points of the subject, the light receiving direction can be resolved by performing the Fourier transform. The resolution in the light receiving direction by the Fourier transform process has the same principle as the resolution of radar pulse compression. That is, the phases of the frequency components of the reflected light are matched and added (passed through a phase matching filter), and pulses of light on the order of microns are transmitted and received like a radar to resolve the light receiving direction. It will be.

(Explanation of detection of interference fringe signal)
Next, a method of detecting the interference fringe signal by the image pickup device 8 of FIG. 1 will be described.
As described above in the "(Explanation of Fourier transform process)" column, when the interference fringe signal is Fourier transformed, a point image distribution function in the light receiving direction is obtained, and the full width at half maximum is the resolution in the light receiving direction. According to the sampling theorem, the sampling interval in the light receiving direction is set smaller than the resolution. Therefore, the number of pixels in the light receiving direction is obtained by dividing the resolution range in the light receiving direction by the sampling interval.

By repeating the imaging of the image pickup element 8 as many times as the number of pixels in the light receiving direction during the frequency sweep of the illumination light, the interference fringe signal can be detected by satisfying the sampling theorem from the relationship of the Fourier transform pair. When the detection time is shortened, the frequency sweep time is shortened, and the image sensor 8 having a high frame rate is used accordingly. As the image sensor 8, basically, an image sensor 8 capable of a global shutter operation is used.

When the number of pixels in the light receiving direction is 1000 and the interference fringe signal is sampled using a general 60 frame / sec image sensor, the time required to detect the interference fringe signal is 1000 ÷ 60 = 16.7 seconds. become. In this case, the sweep time of the light source 1 is set to 16.7 seconds. Due to the length of the detection time, it is applied to three-dimensional resolution and shape measurement of stationary objects.

When a commercially available high-speed image sensor (2000 × 1000 pixels, 1000 frames / sec) is used, the detection time is 1 second.

When the existing fastest image sensor (2000 × 1000 pixels, 20000 frames / sec) is used, the image pickup time is 50 ms. Therefore, it can be used for moving subjects. Further, if the timings of the plurality of image pickup elements are staggered by the multi-plate prism, the image pickup time can be further shortened.

In addition, if the image pickup time of the image sensor is shortened, it seems that the sensitivity cannot be obtained. However, when the Fourier transform process 11 of FIG. 1 is performed, the sensitivity is improved by the number of pixels in the light receiving direction, in other words, the SN ratio of the single spectrum is improved by the square root of the number of pixels.

In addition, when the two-dimensional filter processing 12 of FIG. 1 is performed, the sensitivity is improved by the number of light receiving elements of the virtual lens 35 of FIG. Therefore, as a result, the sensitivity becomes the same as that of the shutter operation of the image pickup apparatus using the optical system, and no problem occurs.

When performing a shutter operation (1 ms or less), an image sensor that can receive interference fringes by each light receiving element, digitize them in parallel, and store them in a memory in parallel is required. Considering the number of wires required for parallel processing and the power consumption due to its stray capacity, it is not feasible unless the parallel processing circuit is built in the image sensor.

The feasibility is high if the semiconductor multi-layer structuring technology, which is making remarkable progress, is used. For example, there is known a development report of a back-illuminated image sensor with 2 million pixels and 2.5 million frames / sec, which is provided with a transfer memory unit for recording for each pixel. If the memory is similarly configured by the multi-layer structuring technique, a shutter operation of 0.4 ms becomes possible.

(Explanation of 2D filtering)
Next, the two-dimensional filter processing 12 of FIG. 1 for resolving a plane perpendicular to the optical axis using the complex signal of the reflected light obtained by the Fourier transform process 11 (FIG. 1) will be described.

FIG. 4 is a diagram showing a configuration from a reflection point to a two-dimensional filter processing.
As shown in FIG. 4, the interference fringes generated by the combined wave with the reference light in the combined wave unit 32 are received by the light receiving elements 33-1 to 33-n of the image pickup element. The detected interference fringe signal is stored in the memory 5 (FIG. 1). After that, the interference fringe signal corresponding to the aperture of the virtual lens 35 is read out from the memory, and the Fourier transform process 34 (11 in FIG. 1) is performed. As a result, the light receiving direction of each light receiving element 33-1 to 33-n is resolved, and the three-dimensional data strings 36-1 to 36-n of the complex signal in the light receiving direction are obtained.

Then, from the three-dimensional data strings 36-1 to 36-n of the complex signal in the light receiving direction by the two-dimensional filter processing 37, as if the light was focused by the virtual lens (colimating lens) 35 having the reflection point 31 as the focal point. , The complex signal of the pixel corresponding to the optical path length from the reflection point 31 to each light receiving element 33-1 to 33-n is extracted. Then, the reflection point 31 can be resolved by matching the phase with the complex signal at the center position of the aperture of the image formation and adding the phase. This process is performed for all reflection points (pixels) in the subject space to resolve the subject in three dimensions.

The configuration of the process of FIG. 4 is shown in FIG. 21 (a).
FIG. 21A is another diagram showing the configuration from the reflection point to the two-dimensional filter processing.

一つの反射点からの反射光Ｐ１～Ｐｎは、上記の式（５）で表すことができる。
ここで、
Ｒは、参照光、
Ｌｐは、受光素子によるローパスフィルタの係数、
Ｆ（カリグラフィ書体）は、受光方向のフーリエ変換、である。

The reflected light P1 to Pn from one reflection point can be expressed by the above equation (5).
here,
R is the reference light,
Lp is the coefficient of the low-pass filter by the light receiving element.
F (calligraphic typeface) is a Fourier transform in the light receiving direction.

式（５）を分配則に従って整理すると、上記の式（６）で表すことができる。

If the equation (5) is arranged according to the distributive law, it can be expressed by the above equation (6).

Equation (6) represents a three-dimensional resolution process in which a plane perpendicular to the optical axis is resolved by an imaging lens and the light receiving direction is resolved by a Fourier transform process, as shown in FIG. 21 (b). There is.

As described in the above-mentioned "Explanation of Fourier Transform Process" column, when the optical path difference td changes, the frequency of the interference fringes changes linearly.
And the initial phase of the interference fringe signal is
2π [(Δf / T) td ² + f0 × td]
It becomes a parabolic shape with respect to td.

This initial phase matching is performed when complex signals of pixels matching the optical path lengths from the reflection points 31 to the light receiving elements 33-1 to 33-n are extracted and added. The initial phase matching is performed by the low-pass filters 42-1 to 42-n of FIG. 5, which will be described later, together with the data interpolation processing. In the filter coefficient generation unit 50 of FIG. 5, which will be described later, the coefficient for data interpolation is multiplied by the coefficient of the complex signal for phase matching, and the coefficients 47-1 to 47- of the low-pass filters 42-1 to 42-n are multiplied. n is generated.

(Explanation of processing operation of two-dimensional filter processing 37)
FIG. 5 is a diagram illustrating the processing operation of the two-dimensional filter processing 37.
The data strings 36-1 to 36-n of the complex signal in the light receiving direction of FIG. 4 are stored in the line memories 41-1 to 41-n of FIG. From the line memory 41-1 to 41-n, from the data string corresponding to the opening to be imaged by the addresses 44-1 to 44-n, from the reflection point 31 in FIG. 4, each light receiving element 33-1 to 33- The complex signals 48-1 to 48-n stored in the address corresponding to the optical path length up to n are read out.

Then, in order to suppress the influence of the quantization error of the optical path length, the low-pass filters 42-1 to 42-n perform data interpolation in the light receiving direction of the complex signals 48-1 to 48-n and the above-mentioned phase matching. After that, addition is performed by the adder 49.

The accuracy of data interpolation is preferably 1/16 or less of the resolution in the light receiving direction. Spline interpolation is desirable for data interpolation. However, linear interpolation using neighborhood data is also sufficient. Data in the vicinity of the complex signal corresponding to the optical path length from the reflection point 31 to each light receiving element 33-1 to 33-n is read from the line memory 41-1 to 41-n. The read data is input to the low-pass filters 42-1 to 42-n, and data interpolation is performed.

The coefficients 47-1 to 47-n of the filter for data interpolation and phase matching are generated by the filter coefficient generation unit 50 according to the addresses 44-1 to 44-p. In order to suppress the sidelobes, the addition may be performed after multiplying by the weighting coefficient for correction. The multiplication of the weighting coefficients is performed by multiplying the filter coefficients 47-1 to 47-n of the low-pass filters 42-1 to 42-n in the filter coefficient generation unit 50.

Addresses 44-1 to 44-n for reading complex signals 48-1 to 48-n having the same optical path length from line memories 41-1 to 41-n, and addresses 44 to which lower addresses for data interpolation are added. -1-44-p are generated by the address generation unit 45.

These addresses 44-1 to 44-p may be generated by calculation, stored in a look-up table in advance, or in consideration of the balance between the calculation time and the memory scale. It is generated by the combined method.

The optical path length of the reflected light and the reference light depends on the optical system arranged in the optical path including the positions of the light receiving elements 33-1 to 33-n in FIG. 4 and the shape and position of the reflecting mirror. There is a difference in the optical path length from each light receiving element 33-1 to 33-n. Therefore, the optical path lengths of the reflected light and the reference light are accurately calculated by the address generation unit 45 and reflected in the addresses 44-1 to 44-p.

For example, the optical path lengths of the reflected light and the reference light having the configuration of FIG. 1 are calculated.
The position of the center of the light receiving surface of the image sensor 8 is set as the origin (0,0,0) of the three-dimensional coordinates.
The direction perpendicular to the paper surface is the X-axis, the vertical direction is the Y-axis, and the direction of the optical axis 9 is the Z-axis.

The optical path length of the reflected light is such that the position of the light source 1 is folded back at the reflecting surface of the beam splitter 2, and the position of the light source 1 (0,0, s) on the optical axis 9 at that time is changed to the position of the reflection point 6 (x, y). , Z), the optical path length from the position of the reflection point 6 (x, y, z) to the position of each light receiving element of the image pickup element 8 (dx, dy, 0) is added.

The optical path length of the reflected light is expressed by the following equation (7).
[x ² + y ² + (zs) ² ] ^1/2 + [(x-dx) ² + (y-dy) ² + z ² ] ^1/2 (7)

Further, the optical path length of the reference light is such that the position of the light source 1 is folded back at the reflecting surface of the reflection mirror 4, and further folded back at the reflecting surface of the beam splitter 2, and the position (0) of the light source 1 at that time on the optical axis 9. , 0, r) to the position (dx, dy, 0) of each light receiving element of the image pickup element 8.

The optical path length of the reference light is expressed by the following equation (8).
[dx ² + dy ² + r ² ] ^1/2 (8)

As described above, in the case of FIG. 1, since the optical path lengths of the reflected light and the reference light can be easily calculated, the optical path difference between the reflected light and the reference light can be calculated. Then, the value obtained by dividing the optical path difference between the reflected light and the reference light by the sampling interval in the light receiving direction corresponds to the address of the pixel when the light receiving direction is resolved by the Fourier transform process 11. In this way, addresses 44-1 to 44-n can be generated.

Further, when performing the two-dimensional filter processing, it is possible to convert the pixels into three-dimensional pixels having a cubic or equal array by data interpolation of the low-pass filters 42-1 to 42-n. Therefore, the addresses 44-1 to 44-p are generated correspondingly.

Further, for three-dimensional resolution, the reflected light from the subject space may be Fourier transformed into three dimensions. In the Fourier transform, when the characteristics of the filter to be multiplied after the Fourier transform are constant (space invariant filter), the total multiplication number can be greatly reduced by the effect of the butterfly operation.

However, in reality, it is necessary to change the size of the aperture and the focal position (change the characteristics of the filter) for each pixel. Further, in the correction of the disturbance of the light wave surface, which will be described later, it is necessary to change the coefficient of the filter for each pixel according to the disturbance of the light wave surface. Therefore, in the present embodiment, the one-dimensional Fourier transform process in the light receiving direction is combined with the two-dimensional filter process 12 that optimizes the coefficient of the filter for each three-dimensional pixel and performs superimposition integration.

In addition, for the convenience of explaining the principle of two-dimensional filter processing, the system configuration that processes complex signals 36-1 to 36-n in the light receiving direction in parallel has been mainly described, but in reality, in consideration of the circuit scale. , Series and parallel processing will be interwoven as appropriate.

Further, as described above, since the two-dimensional filter processing 12 often switches parameters adaptively, GPU (Graphics Processing), which is good at high-speed CPU processing and high-speed parallel processing and can be programmed in a general-purpose language, is possible. It is desirable to combine Unit) etc. to balance the circuit scale and processing time, and make it easier to change the system configuration and upgrade the functions.

(Explanation of arrangement spacing and directivity of light receiving elements required for 2D filter processing)
Next, the arrangement spacing and directivity of the light receiving elements required for the two-dimensional filter processing 12 of FIG. 1 will be described. The two-dimensional filter processing 12 is the same as the two-dimensional Fourier transform of the reflected light.

6 (a)-(g) are diagrams for explaining the arrangement spacing and directivity of the light receiving elements.
In order to simplify the explanation, FIGS. 6A-(g) show a Fourier transform pair in which reflected light is received by a light receiving element having a one-dimensional arrangement and Fourier transformed with respect to the arrangement direction. I will explain the directionality. The y-axis indicates the position in the arrangement direction, and the Y-axis indicates the position of the focal plane obtained by Fourier transforming the y-axis.

FIG. 6A shows a light receiving sensitivity distribution 51 when the reflected light from the reflection point on the optical axis is received by the aperture 52. The light receiving sensitivity distribution 51 is the product of the set aperture 52 and the directivity (light receiving sensitivity distribution on the focal plane) 53 of the light receiving element alone. Therefore, it is meaningless to set the opening 52 beyond the range of the directivity 53.

In other words, the maximum resolution that can be detected is determined by the directivity 53 of the light receiving element. The directivity 53 is formed by the aperture of the light receiving element alone and the microlens. The waveform of the light receiving sensitivity distribution 51 shown in FIG. 6A shows the case where the aperture 52 is smaller than the directivity 53. Further, since the directivity 53 of the light receiving element alone is always in the direction of the optical axis, the waveform of the light receiving sensitivity distribution 51 changes when the reflection point to be detected is separated from the optical axis. The waveform of FIG. 6A shows the case where the reflection point is on the optical axis.

FIG. 6B shows an array of light receiving elements having an interval P. FIG. 6C shows the sensitivity distribution on the light receiving surface of the light receiving element alone.

The Fourier transform of the function of FIG. 6 (a) yields the function of FIG. 6 (d). The Fourier transform of the function of FIG. 6 (b) yields the function of FIG. 6 (e). The Fourier transform of the function of FIG. 6 (c) yields the function of FIG. 6 (f).

According to the fundamental theorem of superimposition integral, the Fourier transform replaces multiplication with superimposition integral (convolution) and superimposition integral with multiplication.

FIG. 6D shows a point image distribution function (half width at half maximum is resolution) 54 on the focal plane obtained by Fourier transforming the light receiving sensitivity distribution 51. FIG. 6 (e) shows the poles of diffraction caused by the arrangement of the light receiving elements. The distance between the poles is 1 / P. FIG. 6 (f) shows the directivity (light receiving sensitivity distribution) 53 on the focal plane of the light receiving element unit formed by the micro lens (formed by the Fourier transform of the micro lens). Incidentally, the actual value on the Y-axis is the value obtained by multiplying the reciprocal of the focal length by a coefficient proportional to the center wavelength, but the figure is omitted because it is not directly related to the explanation in this section.

When the function of FIG. 6 (d) and the function of FIG. 6 (e) are superimposed and integrated, and further multiplied by the function of FIG. 6 (f), the waveform shown in FIG. 6 (g) is obtained. The resolution 57 is proportional to the reciprocal of the aperture τ of the light receiving sensitivity distribution 51, 1 / τ, due to the relationship of the Fourier transform pair. As described above, the resolution (numerical aperture) that can be combined is determined by the directivity 53 of the light receiving element. The directivity of the microlens is set according to the target resolution.

Then, as can be seen from the waveform of FIG. 6 (g), the diffraction of the second main pole 55 or more (which becomes a factor of the ghost image) is removed by multiplying the directivity 53 of the light receiving element alone. The diffraction pole spacing 1 / P must be set larger than the position 56 where the directivity 53 is null (0). In other words, the array spacing P of the light receiving elements needs to be set smaller than the resolution.

For example, when a resolution of 1 μm is obtained by the two-dimensional filter processing 12 of FIG. 1, the arrangement spacing of the light receiving elements needs to be 1 μm or less. Incidentally, the manufacturing limit of the pixel spacing of the image sensor is currently slightly less than 1 μm. The directivity of the microlens can also be controlled during the manufacturing process.

When detecting the reflection point at the end of the angle of view, the position of ± 2nd main pole 55 is closest to the optical axis. On the other hand, the directivity 53 of the light receiving element is always in the optical axis direction. Therefore, it is necessary to set the arrangement interval P small or narrow the angle of view so that the ± second main pole 55 does not enter the directivity 53 of the light receiving element.

Therefore, the Fourier transform process 11 (FIG. 1) is the same as performing orthogonal detection of the interference fringe signal for each frequency component. By the Fourier transform process, the carriers (carrier components) of the interference fringes disappear, and a complex signal having a point image distribution function in the light receiving direction is obtained.

It becomes a two-channel signal called a complex signal, but the frequency band becomes narrower as the carriers of the interference fringes disappear, and the bandwidth of the envelope of the point image distribution function becomes. As a result, the arrangement spacing of the light receiving elements required for the two-dimensional filter processing 12 (FIG. 1) performed after converting the light into a complex signal may be on the order of microns, which is ½ or less of the resolution.

On the other hand, in the case of an imaging lens, since it is a Fourier transform using the frequency of light as a carrier, its surface accuracy is required to be as high as 1/16 or less (several tens of nanometers) of the wavelength of light. ..

However, the imaging lens is a very excellent two-dimensional Fourier transform that can perform imaging instantly, and does not require processing time unlike the two-dimensional filter processing. However, when changing the focal position, aperture, magnification, etc., or correcting the disturbance of the light wave surface, a complicated optical system and mechanism are required, and it takes time to switch between them.

In that respect, the two-dimensional filter processing electrically switches between them, optimizes each pixel, recovers the deteriorated resolution, and expands the depth of field with high resolution. Things can be done.

Therefore, by combining the optical system and the two-dimensional filter processing and making the best use of the characteristics of each, it is possible to optimize the system according to the application and purpose. Further, an application example will be described later.

If a composition having a different refractive index is mixed or an aberration of the optical system is present in the medium through which the reflected light passes, the light wave surface is disturbed and the resolution is deteriorated. Next, a configuration and a method for recovering the resolution deteriorated due to the disturbance of the light wave surface by the two-dimensional filter processing will be described.

(Explanation to restore resolution by 2D filtering)
FIG. 7 is a diagram illustrating a configuration in which the resolution deteriorated due to the disturbance of the light wave surface is restored by the two-dimensional filter processing.

As shown in FIG. 7, the opening is divided into a plurality of blocks 61-1 to 61-m to 61-n. The interference fringe signal corresponding to each block is read from the memory 5 (FIG. 1), and Fourier transforms 62-1 to 62-n are performed.

After obtaining a complex signal in the light receiving direction, two-dimensional filtering processing 63-1 to 63-n is performed for each block, and several pixels before and after the pixel of the reflection point 66, for example, in the direction of the main ray 67 of each block, for example. A complex signal with a total of 5 pixels is detected.

Next, cross-correlation processing 64-1 to 64-n of the 5-pixel complex signal detected in each block and the 5-pixel complex signal in the central block 61-m is performed to detect the deviation in the optical path length. The cross-correlation processing 64-1 to 64-n is performed by superimposing and integrating the 5-pixel complex conjugate signal of the central block 61-m on the 5-pixel complex signal of the other block.

At that time, the superimposition integration is performed by data interpolation for 5 pixels so that the detection accuracy of the peak value indicating the deviation of the optical path length is 1/16 or less of the resolution in the light receiving direction.

If the turbulence of the light wave surface is large, it can be dealt with by increasing the number of 5 pixels that perform cross-correlation processing. If the spatial frequency of the turbulence of the light wave surface is high, increase the number of blocks to increase the number of samplings. Gaussian weighting may be applied to the output of the light receiving element of the block, and the number of blocks may be doubled by overlapping the openings.

The difference in the optical path length between the central block and each block detected by the cross-correlation process 64-1 to 64-n indicates the disturbance of the optical wave surface. The data interpolation unit 66 interpolates the difference in the optical path length for each block so as to correspond to the light receiving elements 33-1 to 33-n in FIG. Then, it is sent to the address generation unit 45 of the optical path length matching shown in FIG. 5 and reflected in the address 46 (added to the address 46). This makes it possible to perform a two-dimensional filter process that corrects the disturbance of the light wave surface.

Further, when the numerical aperture is large as in the case of microscopic imaging, the correlation between the central block and the block at the open end is weakened. Explain how to avoid fading of correlation. First, the correlation process is performed between the first block in the center and the second block next to it, which has a high correlation. Next, a correlation process is performed between the second block and the third block. This may be detected by repeating this while sequentially shifting to the outside and accumulating the deviation of the optical path length.

Detection errors are also accumulated, but if the resolution in the light receiving direction is on the order of microns and the SN ratio is 40db or more, if cross-correlation processing is performed by data interpolation of the complex signal in the light receiving direction, the deviation detection accuracy will be ten. High accuracy on the order of several nanometers can be obtained. Therefore, the cumulative error can be ignored. It is desirable to select the correlation processing method in consideration of the numerical aperture and the SN ratio.

The disturbance of the light wave surface caused by the aberration of the optical system on the way is gentle, and the spatial frequency component in the light wave surface is low. On the other hand, when compositions having different refractive indexes are mixed in the passing medium with a size comparable to that of the resolution, the spatial frequency component due to the disturbance of the light wave surface becomes high.

If the spatial frequency component is high, the number of blocks will be increased according to the sampling theorem. There is a trade-off between the number of blocks and the numerical aperture of blocks (NA). Therefore, increasing the number of blocks reduces the accuracy of cross-correlation.

Therefore, the turbulence of the light wave surface is statistically grasped for each subject, and the SN ratio is used as a constraint condition. Next, the combination of the number of blocks, the numerical aperture, and the pixel range of the cross-correlation is solved in advance for each subject by the combinatorial optimization problem. Then, the correction is performed after switching to the optimum balance for each subject.

In addition, the optimum combination is detected for each subject by annealing and iteration using the growth of OTF of the image after the two-dimensional filter processing as an index, with the number of blocks, the numerical aperture, the range of cross-correlation, and the SN ratio as variables. .. Then, the correction may be performed by switching to the optimum balance for each subject.

Alternatively, the statistically grasped mixed state of the refractive index in the passing medium is added to those variables as a parameter, and the elongation of OTF after the two-dimensional filter processing is used as teacher information to learn the optimum balance for AI. Then, the optimum combination may be determined, and the balance may be switched to the optimum balance for each subject to perform correction.

Incidentally, the principle of correcting the disturbance of the light wave surface of this embodiment is basically the same as that of adaptive optics performed in astronomy or the like. However, in the case of adaptive optics, it is necessary to set a guide star (point image) in the subject space in order to detect the disturbance of the light wave surface. A guide star is set by irradiating a sodium atomic layer at an altitude of 90 km with laser light and exciting sodium to shine.

Similarly, a point image may be set on the surface of the subject by infrared light or the like, but in the method of the present embodiment, the disturbance of the light wave surface is detected by the cross-correlation processing using the signal of the subject. Therefore, it is not necessary to set the same point image as the guide star in the subject space.

Further, for electrical processing, the number and size of blocks 61-1 to 61-n in FIG. 7, which correspond to the wavefront sensor and the wavefront controller of adaptive optics, can be appropriately set according to the application. Then, the balance can be optimized by utilizing a process such as an optimization problem.

Further, a method for correcting the disturbance of the light wave surface with higher accuracy will be described. By the two-dimensional filter processing of each block 61-1 to 61-m to 61-n, three-dimensional complex signal data of 5 × 5 × 5 pixels centered on the detection point is detected. Cross-correlation processing of 6 axes (x, y, z, x _θ , y _θ , z _θ ) using the three-dimensional complex signal is performed between blocks. From the result, in addition to the correction of the disturbance of the light wave surface, the light receiving position of the light receiving element is corrected. After that, a two-dimensional filter process is performed.

As a result, it is possible to perform the two-dimensional filter processing while correcting the blur caused by the mechanical scanning with high accuracy as well as the correction of the disturbance of the light wave surface. However, since this correction method requires a huge number of processes, it is applied to applications where processing time is allowed.

Next, a configuration and processing for generating an RGB image from the RAW data of the interference fringes stored in the memory 5 of FIG. 1 will be described with reference to FIG.

As shown in FIG. 8, the interference fringe signal 71 corresponding to the opening is read out sequentially or in parallel from the memory 5 of FIG. The read signal is subjected to Fourier transform of the visible light band 81a shown in FIG. 9 by FFT72. Then, a W (White) complex signal in which the light receiving direction is resolved is generated. FIG. 9 is a diagram showing wavelength bands of various spectral images generated by the Fourier transform.

When the subject is a living body or the like and the transmission image including the inner layer thereof is detected, the band is set to W including the near infrared region 82 having high biotransparency shown in FIG. 9, and the range is subjected to Fourier transform to perform W. May be generated.

In parallel with the generation of the W complex signal, the FFT 73 performs the Fourier transform of the R band 83 shown in FIG. 9, and the R complex signal is generated. Similarly, the FFT 74 shown in FIG. 8 performs the Fourier transform of the B band 84 shown in FIG. 9, and the complex signal of B is generated.
FIG. 9 is a diagram showing wavelength bands of various spectral images generated by the Fourier transform.

Then, two-dimensional filter processing is performed on each of the above W, R, and B complex signals, and three-dimensional resolution of W, R, and B is performed. When an optical system is used together in the optical path, chromatic aberration (deviation in the optical path length) can be corrected during the two-dimensional filter processing of R and B. Further, as described above, when performing the two-dimensional filter processing, the pixels may be converted into pixels having a cubic or equal arrangement.

Each FFT and each two-dimensional filter process shown in FIG. 8 have the same functions as those described in the Fourier transform process 11 and the two-dimensional filter 12 of FIG.

Next, matrix conversion is performed by the matrix converter 75 of FIG. 8 to generate a three-dimensionally resolved RGB signal. Then, an image according to the purpose such as a surface image, a cross-sectional image, a transmission image, and a three-dimensional construction image by CG is displayed.

Incidentally, the R signal 83 and the B signal 84 in FIG. 9 have a narrower wavelength bandwidth than the W signal and have different center wavelengths. Therefore, the resolution of the R signal 83 and the B signal 84 in the light receiving direction is about 1/3 of that of the W signal. However, there is no problem because the resolution of the human eye for R and B is also about 1/3.

The reason why the complex signal of the R signal 83 and the B signal 84 can be generated by Fourier transforming the interference fringe signal of the R band and the B band obtained by dividing the band of the visible light 81a will be described. The broadband sweep light is considered to consist of a linear sum of a plurality of sweep lights such as R, G, B, and infrared.

All processes such as illumination, reflection, interference with reference light, interference fringe detection, and Fourier transform are linear processes. Therefore, from the principle of superposition, the frequency sweep portion corresponding to the band of R or B is extracted from the interference fringe signal, and the Fourier transform is performed to solve the optical interference by using the sweep light source of R or B alone. The same result as the image processing is obtained.

From this principle, when accurate color reproduction is required, an XYZ complex signal can be obtained by performing a Fourier transform by multiplying the swept visible light band interference fringe signal by an XYZ color matching function.

(Explanation of the configuration for obtaining a complex signal in the light receiving direction that is orthographically projected onto the feature axis)
Further, from the same principle, a complex signal in the light receiving direction that is orthographically projected onto the feature axis can be obtained by performing a Fourier transform by multiplying the interference fringe signal by a coefficient that is orthogonally transformed to the feature axis.

A multispectral analysis performed using the RAW data of the interference fringes stored in the memory 5 of FIG. 1 will be described.

Taking the reflection spectrum of the visible light band as an example, it mainly absorbs wavelengths that excite the outer shell electrons of atoms, absorbs wavelengths that excite molecular vibrations, spins, and intermolecular vibrations, and diffraction scattering due to the arrangement of refractive coefficients. Changes the spectral component.

The changes in the spectral components due to these have a high correlation with the composition and structure of the substance, but they occur multiple times and also change to some extent depending on the illumination and the angle of imaging, such as diffraction and scattering. .. Therefore, it is treated as a cluster for each substance having similar spectral components.

Good results can be obtained by using statistical analysis, such as multivariate analysis, deep learning AI, etc., as a method of identifying such clusters. The procedure for identifying the two clusters by such a method is described below.

First, as many interference fringe signals of the two substances as identified by the three-dimensional image pickup apparatus of the present embodiment are acquired. Next, the data format creation unit 81 shown in FIG. 8 adds additional information 70 and sends it to an external computer as RAW data. Then, it is archived in a recording device.

The additional information 70 includes information on normalization processing that makes it easier to distinguish between clusters by reducing the dispersion of the clusters to be identified, an address that cuts out the substance to be identified from the image, and various information necessary for generating the image. Information and is included.

The information of the normalization process includes the brightness of the illumination light, the wavelength band characteristics of the illumination light, and the like. The cutout address is specified by a specialist who can identify the substance by observing an RGB image or an analysis image of a spectrum described later with a mouse or the like. You may specify it after generating the image on an external computer. The information required to generate the image corresponds to the frequency sweep band, linearity, array spacing and directivity of the light receiving elements, and the like.

The interference fringe signal is read from the recording device, and the acquired data is normalized by the computer. After that, a Fourier transform process and a two-dimensional filter process are performed to generate a three-dimensional image.

Next, the image portion of the substance to be identified from the three-dimensional image is cut out according to the cutout address. At this time, the complex signal in the light receiving direction of the cut out pixel is mainly the complex signal of the reflected light from the surface of the subject. In particular, in the case of visible light, since the propagation attenuation is large, only the reflection on the surface of the subject is performed. In the case of a transparent subject, the pixel data in the target light receiving direction is cut out in three dimensions.

Inverse Fourier transform is performed on the complex signal in the light receiving direction of the cut out pixel. By extracting the amplitude signal from the result of the inverse Fourier transform, the multispectral data of the cut out portion can be detected.

By the way, when the subject is a living body or the like, the reflected light in the living body can be obtained by using the near-infrared band having high transparency. However, the attenuation when propagating in the living body varies greatly depending on the wavelength. Also, the attenuation of the tissue in the propagation path is superimposed.

Therefore, the dispersion of the cluster becomes too large, and it is difficult to analyze the spectrum quantitatively. Therefore, the spectrum analysis is mainly performed on the image on the surface of the subject, except for the case of a highly transparent subject.

Next, a computer performs FS (Folley-Sammon) conversion on a large amount of multispectral data of two substances to be identified in a multidimensional coordinate space with each spectral component as an orthogonal axis.

Then, the feature axes having a large Fisher ratio of the two clusters are narrowed down, and the matrix conversion coefficients for projective transformation to the feature axes, 77-1 to 77-n in FIG. 8, are acquired. The FS transformation is an orthogonal transformation in which the feature axes in which the Fisher ratios of the two clusters increase are calculated in descending order. Similar to data compression, it can be narrowed down to at most 5 to 6 feature axes based on the cumulative contribution rate and experience.

Next, a large amount of data obtained by projecting and transforming the multispectral data to the feature axis is trained by an AI on a computer having the same configuration as the AI80 in FIG. Then, the learned coefficient 76 (neuron coefficient) is acquired.

The number of input terminals of AI is 5 to 6 of the feature axes narrowed down by FS conversion. Therefore, the scale of AI is much smaller including the number of layers. As shown in FIG. 10, the feature of the discrimination by AI is that the discrimination by the non-linear separation Z becomes possible.

Then, as shown in FIG. 8, the matrix conversion coefficients 77-1 to 77-n sent from the computer to the control unit 78 and stored are multiplied by the interference fringe signal 71 by the multipliers 79-1 to 79-n. As a result, the projective transformation of the interference fringe signal 71 into the feature axes EU1 to EU6 is performed.

FIG. 10 is a diagram illustrating a non-linear separation of a substance to be specified by AI.
The interference fringe signal projected onto the feature axes EU1 to EU6 is subjected to a Fourier transform process and a two-dimensional filter process to generate an analysis image of a spectrum which is an image of the feature axes EU1 to EU6.

The upper EU1, EU2, and EU3 of the spectrum analysis image may be assigned to YIQ (component method used in the internal processing of NTSC) in descending order of visual sensitivity, and may be displayed after performing matrix conversion to RGB. After that, the observer's visual brain makes a non-linear discrimination.

Alternatively, the analysis image of the spectrum may be input to the AI80 for each pixel, and the two substances may be identified for each pixel. At this time, the neuron coefficient 76 of AI is generally loaded from the computer into AI80 via the control unit 78.

The result identified by AI80 may be displayed by fusing with an RGB image by pseudo-coloring the pixel portion of the identified substance.

The above-mentioned identification by FS conversion is a method for identifying two clusters. When specifying a plurality of substances, the characteristic axis is switched each time. Even if the switching is performed a plurality of times in combination in a tree shape, the feature axis is narrowed down to 5 to 6 and the circuit scale of the AI80 is small, so that the tree shape identification operation can be performed at high speed.

Alternatively, the multispectral waveforms of the plurality of substances to be specified may be directly learned (supervised) by AI to specify the plurality of substances. However, the number of AI input terminals at that time is required as many as the number of multi-spectrums. Therefore, the scale of AI becomes large.

Next, a correction method will be described when the frequency sweep of the light source is distorted or when it fluctuates with time or temperature. Since there are few laser light sources that are compensated for the linearity and fluctuation of frequency sweep and they are expensive, the applications of this embodiment will be expanded if they can be electrically corrected.

FIG. 11 is a diagram illustrating a case where the linearity of the frequency sweep of the laser light source is distorted.
When the linearity of the frequency sweep is distorted, as shown in FIG. 11, the optical path difference 103 between the reflected light 101 and the reference light 102 causes frequency modulation (frequency dispersion) 104 in the interference fringe signal. As a result, the spectral width after the Fourier transform is widened and the resolution is lowered. Then, when the optical path difference 103 changes, the frequency modulation 104 also changes.

The frequency modulation 104 caused by such sweep distortion can be corrected by performing phase matching filtering on the spectral dispersion after the Fourier transform. As shown in FIG. 5, the phase matching is performed by superimposing and integrating the complex conjugate signal generated by Fourier transforming the frequency modulation 104 using an FIR filter (Filine Impulse Response Filter) 86-1 to 86-n. .. The length of the FIR filter is set in anticipation of the range of frequency dispersion after the Fourier transform.

As described above, when the optical path difference 103 in the light receiving direction shown in FIG. 11 changes, the frequency modulation of the interference fringes changes. Therefore, the coefficients 87-1 to 87-n (FIG. 5) of the phase matching filter are switched and multiplied for each pixel in the light receiving direction.

However, since the change in time due to the change in the optical path difference 103 is extremely small with respect to the frequency sweep time, there is almost no change in the spectral variance after the Fourier transform. Since the change in the spectral variance changes depending on the time of frequency sweep, the bandwidth, and the magnitude of distortion, the necessity of switching the coefficients 87-1 to 87-n (FIG. 5) of the phase matching filter is determined accordingly.

As the coefficients 87-1 to 87-n (FIG. 5) of the phase matching filter, the complex conjugate signal generated by Fourier transforming the frequency modulation 104 shown in FIG. 11 is used. Changes in the linear distortion of the light source frequency sweep are detected at an appropriate time cycle, and the coefficients 87-1 to 87-n of the phase matching filter are calculated in the FIR coefficient generating section 88 of FIG. 5, and the FIR coefficient generating section 88 ( Save and use in the memory in Fig. 5).

Further, the coefficients 87-1 to 87-n of the phase matching filter are added to the additional information 70 via the control unit 78 of FIG. 8 and sent to the data format creation unit 81.

Next, a method of detecting the linear distortion of the frequency sweep will be described. A spectroscope is used to detect frequency sweeps. As shown in FIG. 12, the light emitted from the frequency sweep point light source 1 (FIG. 1) is split by the beam splitter 111 and converted into parallel light by the collimating optical system 112.
FIG. 12 is a diagram illustrating a configuration for detecting a linear distortion of frequency sweep.

After that, it is separated into wavelength components by a spectroscope 113, imaged on a light receiving element (line sensor) 114 having a one-dimensional arrangement arranged in the spectral direction by an imaging optical system 115, and received light. The light emitted from the light source 1 is imaged in a spot shape and moves on the one-dimensional light receiving element 114 according to the frequency sweep.

During the frequency sweep of the light source, the reading of the one-dimensional light receiving element 114 is repeated a plurality of times to detect the movement of the spot light and detect the distortion of the sweep frequency. The peak value detection circuit 115 interpolates the pixel data of the light receiving element to detect the peak value, and improves the accuracy of the position of the spot light.

The phase of the frequency modulation 104 (FIG. 11) is calculated by the FIR coefficient generator 88 (FIG. 5) using the time integration formula when calculating the phase of the FM modulation.

The position of the spot light detected by the one-dimensional light receiving element 114 is temporarily stored in the memory 116, converted into a frequency modulation waveform, and then sent to the FIR coefficient generating unit 88 in FIG. 5 to correct the linear distortion. The coefficients 87-1 to 87-n of the FIR filter are generated by calculation and sent to the FIR filters 86-1 to 86-n for correction.

The repetition frequency of reading of the one-dimensional light receiving element 114 does not require a large value because the distortion of the frequency sweep of the light source is gentle. The characteristics of frequency sweep can be reproduced by data interpolation. The frequency sweep distortion detection accuracy requires a value corresponding to the resolution in the light receiving direction.

Alternatively, as described above, when the change in the spectral variance after the Fourier transform with respect to the optical path difference 103 can be ignored, it is not necessary to switch the coefficients of the phase matching filter (87-1 to 87-n in FIG. 5). Therefore, the complex signal of the reflected light from the reference reflection point is detected by using an optical interferometer, and the complex conjugate signal obtained by Fourier transforming the complex signal is the coefficient of the phase matching filter (87-1 to 87-n in FIG. 5). ) May be used.

(Application Example) Next, an application example of this embodiment will be described.
(First application example)
A single light receiving element is used instead of the image pickup element 8 (FIG. 1), the reflected light from the subject is detected in two dimensions by the two-dimensional scanning mechanism, and the three-dimensional by Fourier transform processing and two-dimensional filter processing. Resolve. Since a single light receiving element can detect an ultrasensitive element or a special wavelength band other than visible light, it can be applied to a three-dimensional image pickup device or an inspection device using such a wavelength band.

(Second application example)
Further, instead of the image pickup element 8 (FIG. 1), a light receiving element (line sensor) having a one-dimensional arrangement is used, scanning is performed by a one-dimensional scanning mechanism in a direction intersecting the arrangement, and the reflected light from the subject is two-dimensional. Is detected, and three-dimensional resolution is performed by Fourier transform processing and two-dimensional filter processing.

Line sensors include those with a large number of pixels, those with high sensitivity, and those that can detect a special wavelength band, such as inspection equipment for FA (Factory Automation) that can identify the composition by three-dimensional measurement and spectral analysis. It can be applied to the visual sensor of FA robots.

(Third application example)
In this embodiment, three-dimensional resolution can be achieved without using an imaging optical system. However, as described above, by combining this embodiment with the imaging optical system, the number of processing of the two-dimensional filter can be reduced.

FIG. 13 is a diagram showing a configuration of an application example of the present embodiment.
As shown in FIG. 13, the resolution in the direction of the main ray 123 is performed by the Fourier transform process. The resolution of the plane perpendicular to the optical axis 121 is performed by the imaging optical system 122. Using the obtained three-dimensional complex signal, the depth of field of the imaging optical system 122 can be expanded and the resolution deterioration due to the disturbance of the light wave surface can be recovered by the two-dimensional filtering process. Aberrations in the optical system can also be corrected by correcting the disturbance of the light wave surface. Reference numeral 4a is a reflection mirror.

FIG. 14A shows an imaged luminous flux 126 when the image forming position of the reflection point 124 (FIG. 13) is in front of the image pickup element 125 (FIG. 13) (subject side). FIG. 14B shows the image luminous flux 127 when it is behind (image side) the image sensor.

The dotted line luminous fluxes 128-1 and 128-2 show the luminous flux when re-imaged by the virtual lenses 129-1 and 129-2, which are processed by a two-dimensional filter. By performing the two-dimensional filter processing on the pixels in the direction of the main ray 123, it is possible to increase the depth of field and recover the resolution deteriorated due to the above-mentioned disturbance of the light wave surface.

By combining the imaging optical system and the two-dimensional filter processing in this way, the range in which the two-dimensional filter processing is performed can be reduced, and the number of processes can be greatly reduced. Therefore, processing in real time becomes possible.

This can be applied to a three-dimensional shape measuring device, a visual sensor of a robot, and the like. Compared to a white interference microscope, it has excellent horizontal resolution, measurement speed, and angular characteristics, and enables three-dimensional shape measurement that enables simultaneous spectral analysis.

Further, if this embodiment is applied to a fundus imaging device, unnecessary reflections on the surface of the objective optical system and the eyeball optical system and unnecessary reflections due to turbidity of the vitreous body can be removed from the difference in the optical path length. Further, in order to avoid unnecessary reflection from the eyeball optical system, all the openings of the eyeball optical system, which have been divided into rings for illumination and for imaging, can be used. Therefore, high-definition and high-contrast fundus imaging can be performed at high speed, and a tomographic image of the retina can be detected in three dimensions.

In FIG. 13, a tomographic image is detected by scanning the imaging mechanism one-dimensionally, and resolution and depth of field are expanded only in the optical axis direction 121 by Fourier transform processing and two-dimensional filter processing. As a result, the number of image sensor elements required to increase the depth of field is 100 pixels or less. Therefore, the number of processing of the two-dimensional filter can be further reduced, and a tomographic image having a high horizontal resolution and a deep depth of field can be detected in real time.

If it takes time to correct the disturbance of the light wave surface, it is desirable to recover the resolution deterioration when the still image is frozen.

Further, the frequency sweep light source is replaced with a low coherence light source (for example, SLD: Super Luminescent Diode), and the interference fringe signal is separated by a spectroscope. As a result, a frequency-swept interference fringe signal can be obtained in the same manner as that obtained with a frequency-swept light source, and this can be subjected to Fourier transform processing and two-dimensional filtering to perform three-dimensional resolution. Can be done.

FIG. 15 is a diagram showing a configuration using a low coherence light source in the present embodiment.
As shown in FIG. 15, the wideband light emitted from the low coherence light source (SLD) 131 is reflected by the beam splitter 132 and illuminates the subject 133. The reflected light from the reflection point 130 is imaged by the objective optical system 134, and unnecessary light is removed through the slit 135 installed on the image forming surface.

After that, it is converted into parallel light by the collimated optical system 136 and is input to the spectroscope 137. The center of the opening of the slit 135 is located on the optical axis of the objective optical system 134. The reflected light is separated by the spectroscope 137 and then imaged on the image pickup device 139 by the imaging optical system 138.

One of the lights separated by the beam splitter 132 is imaged as a line segment 142 on the reflection mirror 141 by the cylindrical optical system 140. The light reflected from the line segment 142 is converted into parallel light via the cylindrical optical system 140, the objective optical system 134, the slit 135, and the collimated optical system 136.

The parallel light is separated by the spectroscope 137 and then imaged on the image pickup device 139 by the imaging optical system 138. Then, a combined wave of the reflected light and the reference light is formed on the light receiving surface of the image pickup element 139, and the generated interference fringes are converted into an electric signal by the image pickup element.

At this time, the optical axis of the cylindrical optical system 140 folded back at the reflection surface of the beam splitter 132 coincides with the optical axis of the objective optical system 134, and the openings of the line segment 142 and the slit 135 are optically coupled. , Cylindrical optical system 140 and reflection mirror 141 are arranged.

Then, the spectroscope 137, the image pickup element 139, and the imaging optical system 138 are arranged so that the direction and the range of spectroscopy by the spectroscope 137 coincide with the direction of the vertical arrangement of the pixels of the image pickup element 139 and the range thereof. There is.

With the above configuration, the subject image formed in the opening of the slit 135 by the objective optical system 134 is formed in the pixel array on the side of the image pickup device 139. Then, the interference fringe signal generated by being dispersed for each horizontal array pixel is detected from the vertical pixel array.

Interference fringe signals are sequentially read from the image sensor 139 and stored in a memory (not shown). Then, the subject 133 or the optical axis 144 is scanned by the scanning mechanism 143 in the vertical direction 130 of FIG. 15, and the interference fringe signal is acquired two-dimensionally and stored in the memory.

Then, the interference fringe signal is read out from the memory, and the light receiving direction is resolved by the Fourier transform process to obtain a three-dimensional resolution. Then, the depth of field is expanded and the disturbance of the light wave surface is corrected by the two-dimensional filter processing.

16 (a), (b), and (c) show the reflected light of the reflection point 146 located outside the in-focus position 145 of the objective optical system 134 (FIG. 15), and the optical axis 144 in the vertical direction of the paper surface. The optical path of the reflected light when detecting while scanning is shown as 147-1, 147-m, and 147-n.

16 (a), (b), and (c) show three patterns 147-1, 147-m, and 147- when the reflection point 146 is on the optical axis 144 and when it is at both ends of the light receiving bundle 148. Shows n.

As can be seen from FIGS. 16A, 16B, and 16C, even if the reflection point 146 is far from the in-focus position 145, when it is detected by scanning in the vertical direction, the reflected light is reflected in the range of the light receiving bundle 148. 147-1 to 147-n can be detected.

Using these reflected lights 147-1 to 147-n, the reflection point 146 can be resolved by the Fourier transform process in the light receiving direction and the two-dimensional filter process. When the reflection point 146 deviates to the rear side from the in-focus position 145, the resolution can be similarly obtained.

When the reflection point 146 deviates from the in-focus position 145 of the objective optical system 134 (FIG. 15), the light beam is kicked by the slit 149 and the sensitivity drops. Since the amplitudes of the reflected light 147-1 to 147-n to be performed are added, the same sensitivity as when the reflection point 146 is at the in-focus position 145 can be obtained.

(Application to intravascular OCT device)
Next, an example of applying the present invention to an intravascular OCT device will be described.
An intravascular OCT (Optical Coherence Tomography) device inserts a guide wire percutaneously from a blood vessel such as the base of a leg, an arm, or a wrist to the coronary artery of the heart under fluoroscopy. Then, a 1 mmφ OCT catheter is inserted along the guide wire and rotated to detect a tomographic image of a coronary artery (2 to 4 mmφ, length 15 cm).

It is a device for diagnosing the stenosis state of blood vessels, the installation status of stents installed to widen the stenosis, and the subsequent restenosis status. With the increase in percutaneous coronary intervention therapy (PCI), the demand for intravascular OCT is increasing.

The task of the intravascular OCT device lies in the qualitative diagnosis of identifying substances that cause stenosis in blood vessels, such as plaques (masses of fat and cholesterol), thrombi, and calcification, and determining the grade of their risk.

Currently, qualitative ones are diagnosed from the morphological information (shape, texture, and brightness) of the tomographic image, but advanced experience is required. Treatment depends on the substance causing the stenosis and its grade. In particular, oily plaques, when peeled off, clog small blood vessels and cause angina pectoris and myocardial infarction, so qualitative diagnosis is important.

In addition, if the presence and grade of the causative substance are known over 15 cm of the coronary artery, various preventive measures can be taken not only for treatment but also for angina pectoris and myocardial infarction.

In order to make a qualitative diagnosis, it is effective to analyze the reflection spectrum that has a high correlation with the causative substance, but in the tomographic image (OCT), the attenuation when propagating in the living body differs greatly depending on the wavelength. In addition, it is difficult to quantitatively analyze the spectrum because the attenuation of the tissue existing in the propagation path is superimposed (integrated). Further, in the near-infrared band used for OCT, the spectral component indicating the characteristics of the causative substance such as plaque has not been clarified.

On the other hand, the visible light image of the fiber angiography makes it clear that plaques, thrombi and calcification can be distinguished by color. For example, plaque is yellowish, thrombus is reddish, which is determined by mixing with fibrin, and calcification and normal mucosa are both whiteish, but the color tone including transparency is slightly different.

By analyzing the spectrum of the visible light band as described above, extracting and emphasizing the optimum feature spectrum for identifying the causative substance, it is possible to improve the accuracy of identifying the causative substance and determining the grade. ..

For this reason, it is desirable that the intravascular OCT device can simultaneously perform qualitative diagnosis by analyzing the spectrum of the blood vessel wall in addition to morphological diagnosis using a tomographic image. And it is desirable that these diagnoses can be made over a length of 15 cm in the coronary arteries.

FIG. 17 is a diagram showing a focus range required when diagnosing a coronary artery.
Another problem with the intravascular OCT device is that, as shown in FIG. 17, the in-focus range 150 required for diagnosing a coronary artery having a maximum diameter of 4 mm is as wide as 1 mm to 4 mm, so that the tomographic image is horizontal. The resolution cannot be set high. If the horizontal resolution is low, the horizontal reflections will be superimposed, and the resolution in the depth direction will also be reduced as a result.

Yet another task is to perform pullback while rotating the OCT catheter at high speed to detect an image of the blood vessel wall of a 15 cm coronary artery. Since the pullback is performed while flashing the optically transparent contrast medium under fluoroscopy, the OCT catheter is rotated at high speed during the time limit of the contrast medium flash of 2 to 3 seconds (recommended time according to the safety of the living body). Even so, the image of the blood vessel wall can only obtain millimeter-order resolution. Further, as described above, in the near-infrared band, the spectrum showing the characteristics of the causative substance is not as clear as in the visible light band.

By applying this embodiment to an intravascular OCT device, the above problems can be solved, and a tomographic image with high resolution and a deep depth of field and an image of the blood vessel wall can be detected, and the blood vessel wall image can be detected. The accuracy of qualitative diagnosis can be improved by analyzing the spectrum of. The application example will be described below.

FIG. 18 is a diagram showing a configuration in which the present embodiment is applied to an intravascular OCT device.
The imaging catheter 151 of FIG. 18 is rotated and pulled back in a sheath inserted from the lower limb aorta into the coronary artery via a guide wire. Guide wires and sheaths are not shown because they are existing treatment instruments.

The connector 152 has a role of fixing (chucking) the image pickup catheter 151 to the rotor portion 153 in addition to attaching / detaching the image pickup catheter 151.

The connector 152 rotates the image pickup catheter 151 together with the rotor portion 153, and the one-dimensional array of fiber rows 154 and the pixel array of the line sensor 155 contained in the image pickup catheter 151 are paired via the telecentric optical system 168. Fix it so that it corresponds to one.

The imaging catheter 151 and the rotor portion 153 are equipped with a mechanism described below.
A frequency-swept light source (not shown) installed in the main body of the device emits light that has been frequency-swept from visible to near-infrared. The emitted light is guided to the fiber coupler 158 by the fiber 157 via the optical rotary joint 156, and is separated into illumination light and reference light.

The illumination light is guided by the fiber 159, converted into parallel light by the collimated optical system 160, reflected by the beam splitter 162 via the cylindrical optical system 161 and is reflected by the beam splitter 162, and the fiber row 154 of about 100 fibers arranged in one dimension. It is focused on the end 163.

The NA (numerical aperture) of the cylindrical optical system 161 is set to match the NA of the fiber row 154. The arrangement of the fiber row 154 may be made into a one-dimensional staggered arrangement, and the number of arrangements may be increased to 200.

Then, the illumination light guided by the fiber row 154 is emitted from the end 164 of the fiber row 154 and illuminates the inside of the blood vessel via the objective optical system 165 and the mirror 166. The objective optical system 165 is an image-side telecentric system, and its focal point is set to the center of the range shown in FIG. 17, 150. Then, the reflected light from the inside of the blood vessel, the blood vessel wall, and the inner layer of the blood vessel wall is imaged at the end 164 of the fiber row 154 by the objective optical system 165 and guided to the rotor portion 153.

In the rotor portion 153, the reflected light emitted from the end 163 of the fiber row 154 is combined with the reference light by the beam splitter 162 to generate interference fringes. The length of the fiber 167 that guides the reference light from the fiber coupler 158 corresponds to the reciprocating length of the fiber row 154.

The interference fringes are imaged on the line sensor 155 by the telecentric optical system 168. The telecentric optical system 168 is a telecentric optical system on both sides so that the image of the end 163 of the fiber row 154 is enlarged and the directivity of the NA of the fiber row 154 and the one-dimensional light receiving element 155 have a one-to-one correspondence. ing.

Then, by driving the one-dimensional light receiving element 155 at high speed, the interference fringe signal received by each element of the one-dimensional light receiving element 155 is sampled. Since the number of pixels of the one-dimensional light receiving element 155 is as small as 100, high-speed driving can be sufficiently achieved.

In addition, because of the high-speed drive, it seems that there is no margin in the sensitivity of the one-dimensional light receiving element 155, but the amplitude (SN ratio) after the interference fringes are Fourier transformed is the light receiving direction due to the phase matching of the Fourier transform. Since it is multiplied by the number of pixels (in the SN ratio of the bandwidth of a single spectrum), no problem occurs.

Then, the drive system 169 that rotates and pulls back rotates and pulls back the imaging catheter 151 integrally with the rotor portion 153, and the interference fringe signal is sequentially detected over the coronary artery 15 cm. The interference fringe signal is sent to the device main body via the rotary transformer 170 and stored in a memory (not shown) in the device main body.

Instead of the rotary transformer 170, the optical rotary joint 156 may be made into multiple channels, optical modulation including other control signals may be performed, and an interface may be performed with the main body of the apparatus. Use a slip ring for the power supply.

The interference fringe signal is read out from the memory of the main body of the apparatus, divided into the interference fringe signal in the visible light band and the near infrared band, and the Fourier transform is performed. Then, using the complex signal obtained by the Fourier transform, the expansion of the depth of field and the correction of the disturbance of the light wave surface described in FIGS. 14 (a) and 14 (b) are performed by the two-dimensional filter processing.

This makes it possible to detect RGB images and three-dimensional shape measurements of blood vessel walls and analysis images of spectra, and at the same time, it is possible to detect three-dimensional images and tomographic images in blood vessels by near infrared rays and in blood vessel walls. Detection becomes possible.

3D images can be constructed from a free viewpoint by CG technology, transmission images and tomographic images can be displayed, and as described above, qualitative diagnosis of the causative substance can be performed by analyzing the spectra of each pixel. ..

When observing an image in real time, the depth of field is expanded and the disturbance of the light wave surface, which requires a lot of processing time, is corrected only when the still image is displayed, and the focus is moved manually. If you do it in, you can observe it in real time.

Next, a method of detecting an image of a coronary artery 15 cm using an intravascular OCT device to which the present embodiment is applied will be described with reference to FIG. The angle of view of the objective optical system 165 is set so that the imaging range 171 (corresponding to the width 181 of the image in FIG. 19) closest to the imaging catheter 151 in FIG. 18 is 1.5 mm. Then, while rotating the imaging catheter 151 and the rotor portion 153 at a speed of 75 rotations / second, the image is pulled back in the 15 cm coronary artery for 2 seconds to acquire a three-dimensional image.

FIG. 19 shows an image of a blood vessel wall obtained by pullback and cut open. 150 images of the blood vessel wall having a width of 1 mm excluding the overlapping portion 182 are detected over a blood vessel length of 15 cm. The width of the overlap portion 182 depends on the distance to the vessel wall.

From the detected pixel position data of the 3D image, the position and magnification of the pixel of the overlapping portion 182 are corrected by CG technology, and the amplitude intensity of the image is smoothed and added in the pullback direction to obtain the image. Can be pasted together.

It is possible to display three-dimensionally resolved image data over a coronary artery of 15 cm with a free viewpoint and magnification. Then, the accuracy of the qualitative diagnosis can be improved by the above-mentioned analysis image of the spectrum.

Further, the resolution in the light receiving direction obtained by the Fourier transform process is higher than the horizontal resolution determined by the fiber arrangement spacing. Therefore, by adjusting the angle of the mirror 166 (FIG. 18) to irradiate the blood vessel wall at an angle to take an image, the resolution of the image of the blood vessel wall can be increased.

Further, in the description of FIG. 18, a wideband optical fiber that guides both the visible light band and the near infrared band is used for the fiber row 155, but the fiber rows for the visible light band and the near infrared band are used. It may be arranged vertically in parallel and two processing circuits may be prepared. Further, the frequency sweep light source may be prepared separately for the visible light band and the near infrared band.

Further, instead of detecting the tomographic image by near-infrared light, an ultrasonic vibrator is installed at the tip of the imaging catheter 151 to detect the tomographic image by ultrasonic waves, and the above-mentioned image detection and spectrum of the blood vessel wall. A combination of analysis mechanisms may be used. The resolution of the ultrasonic tomography is lower than that of the near infrared, but the detection depth of the tomographic image is deep. In addition, each has its own characteristics in morphological diagnosis.

Next, FIG. 20 is a diagram illustrating an example of applying this embodiment to X-ray imaging and γ-ray imaging.
The X-rays emitted from the X-ray source 191 of FIG. 20 have frequency sweep and coherence commensurate with the resolution to be detected. Alternatively, the amplitude of the X-ray is modulated by frequency sweep according to the resolution to be detected.

The X-rays emitted from the X-ray source 191 pass through the beam splitter 192 for X-rays and irradiate the subject 193. The shape of the beam splitter 192 is an elliptical spherical surface, one of which is located at the exit of the X-ray source 191 and the other of which is located at the reflecting surface of the reflector 194.

A part of the X-rays emitted from the X-ray source 191 is reflected by the beam splitter 192 for X-rays, and further reflected by the reflector 194, and irradiates the two-dimensional light receiving element 195 surface as X-rays for reference. ..

The beam splitter 192 for X-rays is a mirror dedicated to X-rays having a very high surface accuracy of ± 1 to 2 nm whose surface is polished by an EEM (Elastic Emission Machining) method. In recent years, such mirrors dedicated to X-rays have become commercially available. By adjusting the angle at which the X-ray mirror is installed, the reflectance and transmittance are adjusted and used as a beam splitter 192.

The X-rays reflected (backscattered) from the subject are combined with the reference X-rays reflected from the reflector 194 by the beam splitter 192 to generate interference fringes. The interference fringes are converted into an electric signal by a CMOS or CCD image sensor, or a two-dimensional light receiving element 195 such as a flat panel detector (FPD).

By repeating the imaging of the two-dimensional light receiving element 195 at high speed within the X-ray sweep time, the X-ray interference fringes in the detection direction are detected, and the detection direction is resolved by the Fourier transform. Then, the surface perpendicular to the detection direction is resolved by the above-mentioned two-dimensional filter processing, and the subject is resolved three-dimensionally.

With the latest CT, the time to acquire 3D data is as fast as 1 second. However, the resolution remains on the order of millimeters because the resolution is achieved only by the absorption information without using the phase information.

On the other hand, according to the imaging method of the present embodiment, the time for acquiring the three-dimensional data may be set to several ms, which is the same as the shutter operation, and the phase information is used for resolution. Therefore, it is possible to obtain a resolution on the order of microns from the interval of the pixel arrangement of the two-dimensional light receiving element 195. Then, the angle of view, the magnification, and the resolution can be freely set according to the purpose. The scale of the device can be simpler than that of CT.

In addition, since the wavelength of X-rays is about two orders of magnitude shorter than that of visible light, the frequency sweep required to obtain micron-order resolution requires a very narrow specific bandwidth, and non-linear scattering such as X-ray fluorescence occurs. It can be set avoiding the wavelength. Further, since the opening of the image pickup device 195 can be reduced, the number of three-dimensional filter processes can be reduced. In recent years, the announcement of X-ray sources capable of frequency sweeping has become popular.

As described above, when the arrangement interval of the image sensor is set to the manufacturing limit of 1 μm, three-dimensional resolution on the order of several μm becomes possible, and the image pickup magnification and the resolution corresponding to it can be freely set for the purpose. Accordingly, a transmission image, a cross-sectional image, and a three-dimensionally constructed image can be displayed. In addition, if the frequency sweep is adjusted to the spectral absorption band where the characteristics of the substance appear, there is a possibility of identifying the substance by analyzing the spectrum.

Although the embodiments to which the present invention is applied and the modified examples thereof have been described above, the present invention is not limited to the respective embodiments and the modified examples as they are, and at the implementation stage, the present invention is within a range that does not deviate from the gist of the invention. The components can be transformed and embodied with. In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the above-described embodiments and modifications. For example, some components may be deleted from all the components described in each embodiment or modification. Further, the components described in different embodiments and modifications may be combined as appropriate. As described above, various modifications and applications are possible within a range that does not deviate from the gist of the invention.

As described above, the present invention is suitable for a three-dimensional image pickup apparatus having a simple structure and capable of simultaneously realizing three-dimensional resolution and detection of a spectroscopic image on a subject.

1 Light source 2 Beam splitter 3 Subject 4 Mirror 5 Memory 6 Reflection point 7 Reflected light 8 Imaging element 9 Optical axis 10 Curve 11 Fourier conversion processing 12 Two-dimensional filter processing 13 Optical interferometer 14 Optical path difference 15 Frequency 16 dotted line 17 dotted line 18 Reference light 19 Reflected light
21 Sweep bandwidth 22 Sweep bandwidth 23 Interference fringe frequency 24 Dotted line 25 Optical path difference 32 Combined part 33-1 to 33-n Light receiving element 34 Fourier transform processing 36-1 to 36-n Complex signal 37 Two-dimensional filter process

Claims (10)

A light source that supplies illumination light to illuminate the subject by sweeping the frequency of light or the frequency of amplitude modulation of light.
An optical interferometer that combines the reflected light from the subject and the reference light to generate interference fringes.
Two-dimensional detection of the interference fringes by two-dimensionally arranged light-receiving elements, by one-dimensional scanning of the one-dimensionally arranged light-receiving elements, or by two -dimensional scanning of a single light-receiving element. A two-dimensional detection mechanism that detects as an interference fringe signal at the position,
The optical path length of the reflected light that reflects the reflection points distributed in three dimensions of the subject from the light source to the two-dimensional detection position of the two-dimensional detection mechanism, and the reference light emitted by the light source. An optical path difference calculation means for calculating the optical path difference from the optical path length from the light source to the two-dimensional detection position of the two-dimensional detection mechanism for each of the two-dimensional detection positions for all the reflection points to be resolved .
A detection unit that resolves the light receiving direction for each of the two-dimensional detection positions and acquires a three-dimensional data string by detecting the frequency of the interference fringe signal.
Using the three-dimensional data string acquired by the detection unit and the optical path difference information calculated by the optical path difference calculation means, an image is formed by electrical processing to obtain the light receiving direction. It is equipped with a two-dimensional filter processing unit that resolves intersecting surfaces .
A three-dimensional image pickup apparatus characterized in that the reflection points distributed in three dimensions of the subject are resolved in three dimensions. The detection unit detects the three-dimensional data string by Fourier transforming the interference fringe signal, and the three-dimensional data string is a complex signal of amplitude and phase.
The two-dimensional filter processing unit selects a data string corresponding to the opening of the image formation from the three-dimensional data string, and from the selected data string, from the two-dimensional detection position to the reflection point . Data matching the optical path length is extracted using the information of the optical path difference, and the filter coefficient calculated from the optical path difference is multiplied and added to perform the image formation and resolve the reflection point. Similarly, the three-dimensional image pickup apparatus according to claim 1, wherein the surface intersecting the light receiving direction is resolved by superimposing and integrating the filter coefficients for all the reflection points to be resolved. A storage unit that stores the three-dimensional data string and
An address generation unit that generates an address for reading the data corresponding to the optical path length from the detection position of the two-dimensional detection mechanism to the reflection point to be resolved from the storage unit by using the optical path difference.
Using the address, the data is read out, the data is interpolated in the light receiving direction, the initial phase is matched, and the filter coefficient generator for generating the filter coefficient for weighting the aperture of the image formation is provided.
The three-dimensional image pickup apparatus according to claim 2, wherein the two-dimensional filter processing unit superimposes and integrates the filter coefficient on the data of the complex signal. The opening of the image formation to be subjected to the two-dimensional filter processing is divided into a plurality of blocks, and the reflection points in the vicinity centering on the reflection point to be resolved by the same processing as the two-dimensional filter processing for each block. By performing resolution, the disturbance of the light wave surface is detected from the mutual correlation calculation of the complex signal data of the reflection points in the vicinity obtained in each of the blocks, and reflected in the generation of the address by the address generation unit. The three-dimensional image pickup apparatus according to claim 3 , further comprising a correction means for correcting the disturbance of the light wave surface. Any one of claims 1 to 4, further comprising a correction means for detecting the distortion and fluctuation of the frequency sweep of the light source and correcting the dispersion of the frequency component of the interference fringes caused by the distortion by a phase matching filter. The three-dimensional image pickup device according to the section. It is characterized in that a spectral component is calculated from the reflection spectrum of a subject whose cluster is known in descending order of Fisher ratio, and the cluster is provided with an identification means for identifying the subject from the reflection spectrum of an unknown subject by using the spectral component. The three-dimensional image pickup apparatus according to any one of claims 1 to 5. The three-dimensional imaging device according to claim 6, wherein the identification means uses an AI that performs deep learning. Any one of claims 1 to 7, further comprising a low coherence light source and a spectroscope instead of the light source, and performing three-dimensional resolution by the detection unit and the two-dimensional filter processing unit. The three-dimensional image pickup device according to the section. A data format creation unit that adds information necessary for three-dimensional resolution and spectrum analysis to the interference fringe signal detected by the two-dimensional detection mechanism, and information necessary for analysis of the three-dimensional resolution and the spectrum. The three-dimensional image pickup apparatus according to any one of claims 1 to 8, further comprising a storage unit for storing the interference fringe signal to which the above-mentioned interference fringe signal is added as RAW data. The coherence degree of the light source, the band characteristics (including distortion) and directionality of the frequency sweep, the coordinates of the detection position of the two-dimensional detection mechanism and the directionality of the light receiving element, and the illumination light with respect to the detection position of the two-dimensional detection mechanism. The three-dimensional image pickup apparatus according to claim 9, wherein the three-dimensional coordinates of the emission position of the reference light, information about the subject, and the like are provided.

Images