WO2021193179A1

WO2021193179A1 - Endoscopic device

Info

Publication number: WO2021193179A1
Application number: PCT/JP2021/010282
Authority: WO
Inventors: のりこ安間; 達夫長▲崎▼; 広朗長▲崎▼
Original assignee: のりこ安間
Priority date: 2020-03-21
Filing date: 2021-03-15
Publication date: 2021-09-30
Also published as: JP2021146008A; JP6748932B1

Abstract

[Problem] To provide an endoscopic device that does not require an image forming optical system and an image sensor, that can be made small and with a small diameter, and that performs imaging and spectrum analysis. [Solution] An endoscopic device is provided with: an optical fiber 1 that guides light supplied from a light source 209, emits the light as diffused light 10, and receives reflected light that returns from a subject 3; an optical interference resolution processing unit 207 that carries out resolution processing on an advancing direction of a wave surface 5 of the diffused light 10 by the imaging principle of optical coherence tomography; a scanning mechanism 8 that scans the diffused light 10 in a scanning direction Y that crosses a center axis X of the diffused light 10; a storage unit 211 that stores a data string 177 generated by the resolution processing carried out by the optical interference resolution processing unit 207; and a combined aperture processing unit 213 that extracts and adds data AD-1 to AD-n, from the data string 177 stored in the storage unit 211, that match an optical path length OPL from one end 51 of the optical fiber 1 to a position of a pixel 173 to be detected.

Description

Endoscope device

The present invention enables high-definition imaging with a deep depth of focus and wide-field to microscopic enlargement by enabling pixel-by-pixel focusing without using an optical system and an image sensor. The present invention relates to a small and small-diameter endoscope device that enables continuous switching of imaging and, in addition, enables pixel-level spectral analysis.

The main issues of the conventional endoscope device are as follows: -To reduce the pain of the patient and to make it possible to observe a narrow space in the body, the size and diameter are reduced.

・ Achieve wide-field and high-definition imaging in order to quickly detect diseases of 3 mm or less (which cannot be detected by X-ray imaging or CT).
-Achieve microscopic magnified imaging to enable observation at the cellular level.
-Realize pixel-level spectral analysis to obtain qualitative information on the diseased part.

The current status of existing endoscopic devices for the above problems is described below.
-Although there is a 2 mmφ small-diameter endoscope device that uses an image fiber bundle, the number of pixels is limited to about 40,000 due to the wave-optical limit of the diameter of the image fiber.

-Although there is a 2 mmφ small-diameter endoscope device with an image sensor manufactured according to the latest design rules, the number of pixels is only about 140,000.

-Although there are endoscopic devices that can magnify from a wide field of view to a microscope, the diameter is as large as 10 mmφ due to the large optical system. Moreover, since the electronic zoom function is also used, it is not possible to provide a high-definition image.
-There is no endoscope device that has a spectrum analysis function.

-In the imaging method using an image sensor, it is difficult to solve the problems collectively because the methods for solving the problems of the endoscope device described above are in a trade-off relationship.

-For example, if you try to install an HD (2000 x 1000 pixels) image sensor in a space of 2 mmφ at the tip of the endoscope, the diagonal dimension of the image sensor's imaging surface must be suppressed to 1.4 mm or less. At this time, the pixel pitch is required to be 0.65 μm, which exceeds the limit values in the pixel pitch and sensitivity of the image sensor and the resolution of the optical system.

-In addition, if the image sensor is to be equipped with a spectrum detection function, it is necessary to separately prepare a pixel equipped with an optical filter for spectrum detection in addition to the RGB filter, which makes the image sensor more compact. It gets even more difficult.

-In addition, the optical system that enables wide-field observation to microscopic magnified observation has a large shape, and the diameter of the endoscopic device must be increased. If the electronic zoom function is used together to increase the magnification in order to reduce the size of the optical system, it is not possible to provide a high-definition image. Displaying a wide-field, high-definition image on a high-definition screen that matches the visual resolution leads to quicker detection of the diseased part, rather than displaying an enlarged display that is unnecessarily lower than the visual resolution by electronic zoom. This is because it is much faster to visually screen on a wide-field, high-definition screen than to operate an endoscope device for screening.

Japanese Unexamined Patent Publication No. 2008-165236

-Patent Document 1 discloses an endoscope device that does not use an image sensor. This is an endoscope device that scans spot light in two dimensions for imaging. However, when trying to obtain an SD image by such a two-dimensional scanning imaging method, high-speed scanning of 15.7 kHz is required for the horizontal line, and 65.7 kHz is required for the HD image of progressive scanning. Difficult to achieve. In addition, since the optical fiber is oscillated and scanned, fatigue (transmission loss) of the optical fiber is started in minutes due to the bending resistance of the optical fiber. Furthermore, it is difficult to realize a circuit that performs spectrum analysis for each pixel.

Further, in Patent Document 1, the realization of a thinner endoscope device is limited by the size of the imaging optical system. In particular, if the imaging optical system is to have functions from wide-field imaging to microscopic magnified imaging, the optical system becomes larger and it becomes difficult to reduce the diameter.

The present invention has been made in view of such circumstances. That is, in the present invention, the imaging optical system, the image sensor, and the like provided in the conventional endoscope device are replaced with one optical fiber and a one-dimensional scanning mechanism, and the reflected light received by the optical fiber is solved outside the imaging unit. By processing the optical information necessary for image and spectrum analysis, the problems caused by providing an image sensor and the like and the problems caused by high-speed horizontal scanning in Patent Document 1 can be solved, and the human body of the endoscopy device can be used. The purpose is to reduce the size and diameter of the inserted part.

[Means to solve problems]
The present invention is an imaging type endoscope device that does not use an imaging optical system and an image sensor in order to collectively solve the above-mentioned problems.
-The outline of the imaging method of the present invention will be described with reference to FIG. The diffused light 10 is obliquely irradiated to the subject surface 3, and the obtained reflected light is subjected to OCI (Optical Coherence Imaging) processing to perform resolution in the line 4 direction, and while repeating resolution in the line 4 direction, The diffused light 10 is scanned in the scanning direction Y that intersects the line 4 direction. Resolution in the scanning direction Y is performed by performing a synthetic aperture process using a plurality of data strings relating to the reflected light obtained by scanning in the line 4 direction and the scanning direction Y. As described above, the endoscope device includes a method of performing pipeline processing to image the resolution of the scanning direction Y orthogonal to the line 4 direction and the line 4 direction on the subject surface 3. The OCI process is technically based on the OCT (Optical Coherence Tomography) process, but since the detection target and purpose are different, the name OCI is used in this application. The details of OCI processing will be described later.

-The principle of the above resolution will be described with reference to FIGS. 2 (a) to 2 (c). When the resolution in the direction in which the illumination light is diffused is performed by the OCI process, the same resolution as that transmitted and received by the spherical light pulse P1 is obtained as shown in FIG. 2 (a), and the diffused light is obtained by the synthetic aperture processing. When the resolution of the scanning direction Y orthogonal to the central axis X (see FIG. 1) of 10 is performed, the resolution of the fan-shaped P2 is obtained as shown in FIG. 2 (b), and the product of both is integrated. It becomes the resolution of. As shown in FIG. 2C, the overall resolution is the same as transmitting and receiving arc-shaped light pulses 5 having a thickness on the order of microns, like a radar, to obtain reflected light.

-If the arc-shaped light pulse 5 enters the subject surface 3 diagonally within the diffused range, the reflected light that is in focus can be obtained at any position on the line 4, and the resolution is made for each pixel. Therefore, the depth of focus is deep, and by switching between the OCI range OC and the synthetic aperture range SA, continuous switching from wide-field imaging to microscopic magnified imaging becomes possible.

-As can be seen from the above, the imaging method of the present invention has dynamic focusing that greatly expands the depth of field while maintaining high resolution by electrical processing, and microscopic enlargement from wide-field / high-definition imaging. Super zooming that continuously switches up to imaging is possible.

The features of the image obtained by the endoscope device of the present invention will be described below.
As shown in FIG. 3, the image obtained by the imaging method of the present invention has a wave surface at a reflection point 6 where the line 4 and the wave surface 5 of the illumination light intersect when the diffused light 10 is illuminated at an oblique angle θ with respect to the subject. It becomes the same as the image observed from the observation direction 7 which is the tangential direction of 5.

-When the subject is translucent, as shown in the enlarged view 3A at the lower left of FIG. 3, the reflected signal (reflected brightness) from the same wave plane inside the subject is superimposed, and the transmitted image is transmitted from the tangential direction (observation direction 7). You will get the same image as you observed. By including near-infrared light (wavelength 0.68 μm to 1.5 μm) with high biotransparency in the illumination, it is possible to obtain a transmitted image at a depth of several mm from the surface of the living body (subject surface).

-Since the image obtained by illuminating the subject from an angle is observed from the direction orthogonal to the irradiation direction (observation direction 7), the specular reflection on the mucosal surface escapes forward (to the left in FIG. 3). Since the shadow of the tissue T such as a blood vessel is easily cast and the outline of the blood vessel is emphasized, the structure of the blood vessel can be easily grasped. By the way, the malignancy and progression of cancer are diagnosed from the structure of blood vessels.

-In addition, by adjusting the irradiation angle θ of the illumination, it is possible to emphasize the slight unevenness of the subject surface 3 and the shadow of a small tissue under the mucous membrane. For example, the contours of submucosal capillaries, slight irregularities on the joint surface, and slight steps at the edges of the epiretinal membrane on the retina can be easily observed. The irradiation angle θ of the illumination is manipulated according to the purpose.

A first aspect of the endoscope device of the present invention is an optical fiber that guides light supplied from a light source, emits it as diffused light, and receives reflected light returned from a subject.
An optical interference resolution processing unit that performs resolution processing in the traveling direction of the wavefront of the diffused light according to the imaging principle of the optical coherence tomography method.
A scanning mechanism that scans the diffused light in a scanning direction that intersects the central axis of the diffused light.
A storage unit that stores a data string generated by resolution processing by the optical interference resolution processing unit, and a storage unit.
It is provided with a synthetic aperture processing unit that extracts and adds data that matches the optical path length from one end of the optical fiber to the position of the pixel to be detected from the data string stored in the storage unit.

The second aspect of the endoscope device of the present invention is the endoscope device according to the first aspect, in which the scanning direction is orthogonal to the central axis of the diffused light.

The third aspect of the endoscope device of the present invention is the endoscope device according to the first or second aspect, and the light source produces wideband light or wideband wavelength sweep light.

A fourth aspect of the endoscope device of the present invention is the endoscope device according to any one of the first to third aspects, and the optical interference resolution processing unit is the reflection received by the optical fiber. A reflector for generating reference light with respect to light, an interference coupler that interferes with the reference light, a spectroscope that extracts a predetermined wavelength band component, and a Fourier conversion of the predetermined wavelength band component to perform a Fourier conversion to a predetermined wavelength. It includes a Fourier transform unit that generates a predetermined signal corresponding to the band.

A sixth aspect of the endoscope device of the present invention is the endoscope device according to any one of the first to fifth aspects, wherein a plurality of endoscope devices having clusters are known in descending order of Fisher ratio from the reflection spectrum of a subject. A discriminating means is provided for calculating a spectral component and using the spectral component to discriminate from the reflection spectrum of an unknown subject by a cluster.

The seventh aspect of the endoscope device of the present invention is the endoscope device according to the sixth aspect, and the identification means uses an AI that performs deep learning.

-According to the present invention, since the imaging unit of the endoscope device has a simple configuration including an optical fiber and a scanning mechanism in a one-dimensional direction, it is possible to make the diameter of the insertion unit of the endoscope device extremely small.
According to the endoscope device of the present invention, since horizontal scanning is performed by electrical processing (for example, Fourier conversion processing), high-speed horizontal scanning required for HD, 4K, and 8K can be realized, and images are repeated. For vertical scanning, a scanning mechanism having a low vertical scanning frequency (for example, 60 Hz) can be used.

-The endoscope device of the present invention is capable of high-definition imaging of HD, 4K, and 8K in spite of its small size and small diameter, and can continuously switch from such an image to a microscopic magnified image, and is covered. It is possible to electrically increase the depth of field.

-Although the endoscope device of the present invention is small and has a small diameter, it is possible to identify the substance of the subject at the pixel level by spectral analysis.

-Since the endoscope device of the present invention does not use a heat-sensitive image sensor, autoclave sterilization is possible.

It is a block diagram which shows the structure of the endoscope apparatus which is an embodiment. It is a figure for demonstrating the principle of imaging by an endoscopic apparatus which is an embodiment, (a) is a figure which shows the resolution by OCI, (b) is a figure which shows the resolution by a synthetic aperture. Yes, (c) is a schematic diagram schematically showing an optical pulse diffused on a subject surface. It is a figure for demonstrating the observation image. It is a figure for demonstrating the relationship of the reciprocating optical path length between the fiber end of the irradiation light which irradiates from an optical fiber, and an arc. It is a block diagram which shows the structure of a scanning mechanism. (A) is a diagram for explaining a method of imaging a living body by OCT processing, and (a) is a diagram for explaining a method of imaging a living body by OCI processing. It is a block diagram which shows OCI processing using a wide band light source. It is a figure for demonstrating OCI processing by Fourier analysis. It is a block diagram which shows the OCI processing using the wavelength sweep light source. It is a conceptual diagram which shows the concept of a synthetic aperture in a sector scan system. It is a block diagram which shows the example of the completion processing which complements the acquired data, and the synthesis opening processing by the correlation calculation. It is a block diagram which shows the example of the completion processing which complements the acquired data, and the synthesis opening processing by a Fourier transform. It is explanatory drawing for demonstrating an example of performing microscope imaging by the configuration combined with the existing endoscope. It is a block diagram which shows the structure of the image pickup part of a small-diameter endoscope apparatus. It is sectional drawing explanatory drawing for demonstrating microscopic imaging. It is a figure explaining the synthetic opening process. It is a figure explaining the scanning system which scans diffused light, (a) is a linear system, (b) is a convex system, (c) is a sector scan system, (d) is an example of a multi-scan system, (e) is an example. This is another example of the multi-scan method. It is a conceptual diagram which shows the concept of phase matching in a sector scan system. It is a block diagram which shows the detection of RGB and a spectrum image. It is a figure explaining the range of the Fourier transform. It is a figure explaining FS (Foley Sammon) conversion. It is a block diagram which shows the synthetic aperture processing and the matrix conversion processing of RGB and a spectrum image. It is a block diagram explaining that the spectrum image is identified by AI. It is a figure explaining the cluster identification flow. It is a graph which shows the reflection spectrum characteristic of blood and a superficial mucosa. It is a graph which shows the absorption coefficient characteristic of oxidized hemoglobin (artery) and reduced hemoglobin (vein). It is a block diagram which shows the structure of the scanning mechanism which rotates the end of an optical fiber by a micromotor. It is a block diagram which shows the structure of the scanning mechanism which uses a MEMS mirror. (A) and (b) are configuration diagrams showing the configuration of a scanning mechanism that vibrates the optical fiber end by a piezoelectric bimorph, and (c) shows the optical fiber end by two piezoelectric bimorph oscillators whose vibration directions are orthogonal to each other. It is a block diagram which shows the structure of the scanning mechanism which causes rotational vibration.

Hereinafter, the endoscope device according to the embodiment of the present invention will be described with reference to the drawings. In the drawings, the same parts are indicated by the same reference numerals.

1. 1. [Examples of the present invention]
Examples of the imaging method of the present invention will be described below.
1) Basic operation of the embodiment of the imaging method of the present invention-As shown in FIG. 1, the light of the broadband light source 209 is guided to the endoscope tip 201 by the optical fiber 1 via the optical circulator 203, and the optical fiber end. It is emitted from 51, and the subject surface 3 is illuminated obliquely in a predetermined diffusion range DA. The position of the endoscope tip 201 with respect to the subject surface 3 so that the angle θ at which the central axis X of the diffused light and the subject surface 3 intersect is in the range of 0 ° <θ <90 °, preferably 45 degrees. Is operated.

At this time, as shown in FIG. 4, the reflected light from the arc ARCs (ARC1 and ARC2) is superimposed in the same phase because the round-trip optical path lengths from the optical fiber end 51 are the same, and is received by the optical fiber end 51. Line 4 shows one of the lines on the subject surface 3 orthogonal to the arc ARC. The optical path length OPL from the optical fiber end 51 to the arc ARC gradually increases as the position of the arc ARC becomes farther from one end (line end) 104 of the line 4.

The reflected light received at the optical fiber end 51 is guided to the OCI processing unit 207 via an optical circulator (see reference numeral 203 in FIG. 1). In the OCI processing unit 207, by the same processing as OCT, the direction orthogonal to the arc ARC (longitudinal direction of the line 4) is utilized by utilizing the difference in the round-trip optical path length OPL from the optical fiber end 51 to the arc ARC shown in FIG. Resolution is made. The operation of the OCI processing unit 207 will be described in detail in sections 2) to 5) described later.

Then, while repeating the operation of the OCI processing unit 207, the diffused light 10 is scanned in the scanning direction Y intersecting the central axis X by the one-dimensional scanning mechanism 8, and the data string required for the composite opening is acquired and stored. It is stored in the unit 211.

Figure 5 shows an example of the scanning mechanism. Reference numeral 511 indicates an optical fiber that rotates and reciprocates while illuminating the subject surface 3 at an angle, 555 indicates a microgalvano scanner, 555 indicates an optical rotary joint, and 112 indicates an optical fiber for transmission. Reference numeral 12 indicates a GRIN (Gradient Index) lens. The GRIN lens 12 is used not for forming an image but for adjusting the diffusion distribution of light intensity, and the aperture diameter can be made relatively small. Instead of the microgalvano scanner 555, a rotary scanning micromotor may be used. When using a micromotor, the coil spring 533 becomes unnecessary. Micromotors of 0.9 mmφ and 0.6 mmφ are commercially available, and there is no concern about their size. Examples of other scanning mechanisms will be described later.

Next, the data required for the composite aperture is read from the storage unit 211, the composite aperture processing unit 213 performs a process of synthesizing the aperture in the scanning direction Y, and resolution in the scanning direction Y is performed. The operation of the synthetic opening processing unit 213 will be described in detail in Sections 6) and 7) described later.

-Finally, the display processing unit 205 performs matrix conversion to an RGB image, processing for displaying the AI determination result, scanning conversion according to the display device 215, and the like, and the display device 215 performs RGB image and spectrum image. Is displayed.

2) Difference between OCI processing and OCT processing As described above, OCI processing is optical interference resolution processing (processing based on the imaging principle of optical coherence tomography) based on OCT processing, but the purpose is as shown below. Are different from each other, so there are differences in both processes.

-Since the purpose of the OCT process is to detect an internal tomographic image from the living body surface OS, the depth of field D1 is set deeply as shown in FIG. 6A. Therefore, the numerical aperture NA1 of the optical system cannot be increased (the resolution is increased) (NA1 is relatively small). In addition, the wavelength band used in OCT processing is limited to near-infrared light with high biotransparency, and attenuation and scattering for each wavelength are superimposed during propagation in the living body, so RGB images can be generated and quantitatively generated. Spectral analysis is not possible.

On the other hand, since the OCI process aims to detect an image of the biological surface OS which is the surface of the subject, the numerical aperture NA2 can be set relatively large as shown in FIG. 6 (b). In addition, according to the OCI process, unlike the OCT process, the irradiation light is not attenuated when propagating in the living body, so there is a margin in sensitivity, and by adjusting the opening of the optical fiber end 51 with a pinhole, high microscopic resolution can be achieved. realizable. In addition, in OCI processing, wideband light can be used, and unlike OCT processing, there is no attenuation or scattering for each wavelength that occurs superimposed when the irradiation light propagates in the body, so RGB image detection and quantitative spectrum Analysis is possible.

-Therefore, in OCI processing, OCT processing technology is used as a base, OCT processing is performed using wideband light, and in addition, processing for RGB image detection and spectrum analysis is performed. The operation of RGB image detection and spectrum analysis will be described in [2. Spectrum analysis] will be described in detail.

3) OCI processing Next, the OCI processing unit using a wideband light source will be described.
As shown in FIG. 7, the light of the broadband light source 21 is divided into two by the branch coupler 22, and one of the branched lights is reflected by the reflector 24 via the optical circulator 23, and the reference light. 25 is obtained.

The other branched light is guided to the endoscope tip portion 201 by the optical fiber 1 via the optical circulator 26, and is obliquely irradiated to the subject surface 3 as shown in FIG.

Next, the reflected light from the arc ARC shown in FIG. 4 is received by the optical fiber 1, and is interfered with the reference light by the interference coupler 27 via the optical circulator 26. When interference occurs, interference fringes are generated at a frequency proportional to the difference between the reflected light and the optical path length OPL of the reference light 25, and these are superimposed by the number of reflection points from the line 4. The longer the optical path length OPL of the reflected light is longer than the optical path length of the reference light 25, the higher the frequency of the interference fringes.

Next, the output light of the interference coupler 27 is split into a wave number (inverse wavelength) component by the toroidal type grating spectroscope 28, and the light carrier disappears by receiving the wave number component with the light receiving line detector 29. , An electric signal in which the wavenumber components are in chronological order can be obtained. This electric signal has components indicating interference fringes generated according to the difference in the round-trip optical path lengths of the reflected light and the reference light 25, which are superimposed by the number of reflection points on the line 4.

Next, when the Fourier transform is performed on this electric signal by the FFT30, the frequency component after the Fourier transform corresponds to the difference between the reference light 25 and the optical path length OPL of each arcuate reflection point ARC, and the amplitude of each frequency component. However, it corresponds to the reflection intensity. As a result, the resolution is made in the direction orthogonal to the arc-shaped reflection point ARC, and the reflected signal reflected from the arc-shaped reflection point ARC is obtained.

Then, while scanning the diffused light 10 in the scanning direction Y by the scanning mechanism described later, the above operation is repeated, and the data string of the amplitude and phase of the reflected signal of the arc-shaped reflection point ARC obtained is stored in the storage unit 211. Is sent to and remembered.

When the wavelength (wave number) band of the wideband light source 21 shown in FIG. 7 is widened, the wave number of the interference fringes increases in proportion to it, so that the width of the single spectrum when the wavelength signal is Fourier transformed becomes narrow and the line 4 direction. Will increase the resolution of.

^{The resolution σ in the line 4 direction shown in FIG. 1 is σ = [0.44 · λc 2} / Δλ] / cos θ, where the optical fiber end 51 is regarded as a point light source and the shape of the wavelength band of the broadband light source 209 is a normal function. It is represented by. The resolution σ is proportional to the width of the wavelength band. Δλ indicates the wavelength bandwidth, λc indicates the central wavelength, and θ indicates the angle at which the central axis X of the diffused light and the line 4 intersect.

-In addition, as the repetition cycle of the OCI process becomes longer, the output of the light receiving line detector 29 (see FIG. 7) becomes larger, so that the sensitivity of the OCI process increases.

4) Supplementary explanation of OCI processing The following is a supplementary explanation of OCI processing by Fourier analysis.
The first term of the equation shown in FIG. 8A schematically represents a wide band illumination light. The second term in FIG. 8A expresses the relationship between the reference light 25 and the optical path length OPL of the reflected light by a δ function when the number of reflection points on the line 4 is one. The value of L indicates the reciprocating optical path length of the reflected light when the optical path length OPL of the reference light 25 is 0. The process of interfering the reference light 25 with the reflected light can be expressed as a superposition integration (*) of the illumination light on the reflection positions of the reference light 25 and the reflected light as shown in the equation of FIG. 8A. The second term k in FIG. 8A indicates the amplitude of the reference wave, and A indicates the amplitude of the reflected wave.

The composite product obtained by the formula of FIG. 8 (a) is separated by a toroidal type grating spectroscope 28 (see FIG. 7), and the electric signal converted by the light receiving line detector 29 is shown in FIG. 8 (a). Since it is equivalent to detecting the carriers of light by Fourier transforming (spectroscopically) the equation, the wavenumber band and the second term of the illumination light of the first term of the equation of FIG. 8 (b) are based on the theorem of superposition integration. The electrical signal of the third term obtained by multiplying the interference fringes of (X) is obtained. The first term τ in FIG. 8 (b) means the wave number component of the broadband light. The second term of the equation of FIG. 8 (b) shows the interference fringes of the frequency proportional to the distance L (the period is inversely proportional to the distance L). In the actual reflected light, all the reflected light from the reflected position on the line 4 is superimposed, so that all the interference fringes having the corresponding frequencies are superimposed. The amplitude k of the reference light shown in the second term of the equation of FIG. 8B becomes the DC component of the interference fringes, and the intensity A of the reflected light becomes the amplitude of the interference fringes.

-Since the second term of FIGS. 8 (a) and 8 (b) is a Fourier transform pair, the waveform after the Fourier transform can be obtained by exchanging the position axis (time axis) and the wave number axis (frequency axis). easy to understand. By the way, when the position (distance) axis is Fourier transformed, it becomes the wave number (spatial frequency) axis, but since it is converted into a time-series electric signal by the light receiving line detector 29 (see FIG. 7), FIG. 8 (b) As shown in the third term of the above, the wave number axis is the time axis.

Next, when the equation of FIG. 8 (b) converted into an electric signal is inverse Fourier transformed by FFT30, the phases of the wavenumber band components of the first term of the equation of FIG. 8 (b) are matched, and FIG. 8 ( It becomes the pulse shown in the first term of the equation of c). The interference fringe of the second term of the equation of FIG. 8 (b) returns to the δ function as shown in the second term of the equation of FIG. 8 (c). According to the superimposition integral theorem, the multiplication of the equation of FIG. 8 (b) is the superimposition integral in the equation of FIG. 8 (c). A pulse having an amplitude of A at the position and a width inversely proportional to the wave number band of the illumination light can be obtained. The full width at half maximum of the pulse of the third term of the equation of FIG. 8C is the resolution in the direction orthogonal to the arc ARC (line 4 direction).

5) OCI processing using a wavelength sweep light source Next, the OCI processing when a wavelength sweep light source 31 is used instead of the wideband light source 21 will be described with reference to FIG.

-Within the repeated scanning time (TIME) of the OCI process, the light whose wave number (reciprocal of the wavelength) of the illumination light is linearly modulated by the wavelength sweep light source 31 is input to the branch coupler 32. Further, the output light from the interference coupler 33 is converted into an electric signal by the light receiving detector 34. Other processing is the same as the processing performed in the OCI processing of the wideband light source 21 described with reference to FIG. 7.

When the wavelength sweep light source 31 is used, the grating spectroscope 28 and the light receiving line detector 29 shown in FIG. 7 are not required, and a single light receiving detector 34 having good sensitivity and SN can be used. However, since it is necessary to synthesize a plurality of wavelength sweep light sources for a wide band wavelength sweep light source 31 such as visible light, the light source is selected according to the purpose in consideration of applications such as multispectral analysis described later. To do or use a light source in combination.

6) Synthetic opening processing Next, the operation of the synthetic opening processing unit 213 (see FIG. 1) will be described.
As shown in FIG. 10, the data strings DL-1 to DL-n (177) required for the composite opening from the data sequence stored in the storage unit 211 of FIG. 1 according to the size of the opening 171 to be combined. Is selected, and the data AD-1 to AD-n corresponding to the optical path lengths from the optical fiber end 51 to the detection pixel 173 are read from the data strings DL-1 to DL-n. In order to suppress the quantization error when reading the data AD-1 to AD-n, the interpolation circuit 46 shown in FIG. 11 performs amplitude and phase interpolation from the data of the preceding and following addresses, and reads the data AD-1. -Improve the accuracy of AD-n. The addresses of the data AD-1 to AD-n to be read are parabolic 175 by the first-order approximation.

Next, when the phase shifts corresponding to the optical path differences OPD-1 to OPD-n of the read data AD-1 to AD-n were matched and added, pixel 173 was detected by the optical system having the same aperture. It will be the same as. By matching and adding the phases, the amplitude of the detected pixel 173 is increased by the number of the data strings DL-1 to DL-n, and the amplitude of the other pixels used in the synthetic aperture 171 is the arc of FIG. As the distance from the line 4 on the ARC increases, the phases (positions) shift and cancel each other out. A supplementary explanation of this phase shift will be given in Section 7) using Fourier analysis.

-The phase of the data DL-1 to DL-n described above is matched, the addition is performed, and the process is repeated in the scanning direction Y. As shown in FIG. 11, the correlation calculation unit 48 causes the data DL-1 to be added. The correlation calculation of the reference signal RS for matching with DLn is performed in the direction of the data string 177. In order to obtain a constant resolution (numerical aperture) on the line 4, the reference signal RS selects the data string DL-, which is selected according to the optical path length of the pixel 173 to be detected (according to the reflection position on the line 4). The range 177 of 1 to DL-n (the range 171 of the synthetic aperture) and the reference signal RS must be adaptively switched. The reference signal RS required for the pixel 173 to be detected (combined) is stored in the lookup table of the reference signal generation unit 45. Alternatively, since the reference signal RS is a function of the optical path length of the pixel 173 to be detected, it may be generated by calculation.

-As for the correlation calculation, according to the theorem of superimposition integration, as shown in FIG. 12, the correlation calculation unit of FIG. The same result can be obtained by the method according to 48. The range of data for which the Fourier transform 401 is performed is the range of data stored in the scanning direction Y. When the reference signal in the scanning direction Y is constant as in the scanning method of FIG. 5, the method of Fourier transform 401 in FIG. 12 requires fewer multiplications than the method of correlation calculation in FIG.

7) Supplementary explanation of synthetic aperture processing Next, a specific example of a small-diameter endoscope to which the endoscope device of the present invention is applied will be shown, and synthetic aperture processing in a small-diameter endoscope will be explained using Fourier analysis. conduct.

-As shown in FIG. 13, the small-diameter endoscope (2 mmφ) shown in FIG. 5 is passed through the forceps opening (2 to 3 mmφ) 405 of the existing endoscope (6 mmφ thin endoscope) 407. A case where the 500 is pressed against the target site to perform microscopic imaging will be described. The reason for pressing it against the subject surface is to prevent blurring and to increase the numerical aperture of the synthetic aperture.

-The operation performed by the observer is as follows: first, the diseased part is screened by the wide-field image of the parent endoscope 407, and then the high-definition observation of the part of interest is performed by the small-diameter endoscope 500 through the forceps opening 405. I do. Then, the observer continues the screening while shortening the observation distance of the high-definition image (while enlarging the image), and presses the small-diameter endoscope 500 against the screened site where cytodiagnosis is required. And observe the magnified microscopic image. As described above, the small-diameter endoscope 500 has a deep depth of focus and can automatically switch from high-definition imaging to microscopic magnified imaging simply by changing the distance of the subject. Can be easily done. In FIG. 13, the small-diameter endoscope 500 has a first flexible portion 409 that can be refracted by wire control and a spring-shaped second flexible portion 411 that can be refracted.

Next, the numerical aperture of the synthetic aperture in this microscopic magnified imaging and the interval P (see FIG. 10) for acquiring the data sequence 177 for the synthetic aperture will be described.

As shown in FIGS. 14 and 15, the optical fiber end 51 is rotationally scanned in an arcuate scanning range 343 0.7 mm from the center (rotary scanning axis) R of the endoscope device 301, and the optical fiber end 51 is 1 mm from the optical fiber end 51. When imaging a 2 × 2 mm imaging range 571 at a distance, the resolution for obtaining 1000 × 1000 pixels for the 2 × 2 mm imaging range 571 needs to be 2 μm or less according to the sampling theorem. When the resolution σ in the line 4 direction is calculated from the above-mentioned OCI resolution formula, a resolution of 2 μm or less can be sufficiently achieved by using the visible light band for the diffused light. The aperture size T required to set the resolution ξ in the scanning direction Y to 2 μm (see Res. In FIG. 16 (c)) is that the focal length is 1 mm and the center wavelength λc of the diffused light 10 is 550 nm. Then, when calculated back from the approximate expression of imaging, ξ = 0.61 × λc / NA (NA = sinα), T = 0.34 mm is required. In FIG. 14, reference numeral 511 is an optical fiber for lighting, 525 is a bearing, 259 is a magnet, 556 is a micromotor, 533 is a coil spring, 533 is an optical rotary joint, 513 is an electric coil, and 571 is a 2x2 mm imaging range. Reference numeral 341 in FIG. 15 indicates a surface mucous membrane, and 343 indicates a scanning range of 1.34 mm at the end 51 of the optical fiber.

Next, under the above conditions, the interval P when the diffused light 10 is rotationally scanned to acquire the data strings DL-1 to DL-n in FIG. 10 is obtained by Fourier analysis.

-Since the openings 171 to be combined are discrete, as shown in the equation of FIG. 16A, the COM (comb) function of the first term representing the interval P for acquiring the data strings DL-1 to DL-n can be used. , The window function (T = 0.34 mm) of the second term representing the opening T is multiplied, and the light receiving sensitivity distribution of the optical fiber end face shown in the third term is superimposed and integrated.

-Since the composite aperture process is a Fourier transform, according to the theorem of superimposed integration, as shown in the equation of Fig. 16 (b), the square wave is Fouriered into the com function with an interval of 1 / P that can be obtained by Fourier transforming the COM function. It can be expressed by an equation obtained by superimposing and integrating the sync function formed by the transform and multiplying it by the distribution of diffused light (light receiving sensitivity distribution of the optical fiber) on the focal point of the third term. The axis unit of the equation of FIG. 16 (b) is represented by the spatial frequency, but when converted into the unit of distance, it is represented by the waveform of FIG. 16 (c).

・ As can be seen from the waveform in FIG. 16 (c), as the interval P becomes wider, the interval between the 0th and ± 1st orders of the waveform in FIG. 16 (c) becomes narrower, and the 0th order PSF (Point Spread Function: light receiving sensitivity) The ± 1st order PSF enters the distribution) as an artifact (a virtual image that does not originally exist). To be precise, the integral value (artifact) of the ± 1st order PSF (side lobe) overlapping the 0th order PSF must be smaller than the signal noise. Here, since the diffusion range DA of the diffused light 10 (see FIG. 1) needs to be larger than the aperture in synthesizing the aperture, the diffusion range DA of the diffused light 10 (see FIG. 1) is set. By narrowing it, it is not possible to suppress ± 1st order or higher. By adjusting the interval P, the effect of ± 1st order can be suppressed. The smaller the interval P, the wider the interval between 0th order and ± 1st order, and the influence of ± 1st order becomes smaller, but the number of data increases by that amount, so consider the balance between SN and the amount of calculation (circuit scale). Set.

-When the interval P in the equation shown in FIG. 16 (a) is set to 1 μm, the interval between the 0th order and the ± 1st order is within the range required for the diffused light 10 as shown in the waveform of FIG. 16 (c). Since it is doubled to 0.68 mm (= 0.61 x λc x 2 f / P / cos θ, f: focal length, θ: aperture angle), ± 1st-order artifacts can be suppressed. When imaging a 2 mm imaging range 571, as shown in FIG. 15, the optical fiber end 51 is scanned and scanned over a 1.34 mm scanning range 343 (scanning angle β is 68 degrees) including the opening to be combined. In the meantime, 1340 data strings are acquired, and the synthetic aperture processing is performed using the 340 data strings corresponding to the size of the aperture in the acquired data strings.

8) Scanning method of diffused light ・ FIG. 17 shows a scanning method for scanning diffused light 10 (see FIG. 3). a is a linear system, c is a sector scan system, and d is a multi-scan system in which a, b, and c are combined. Apertures can be combined in any scanning method. For example, the convex method of b is convenient when imaging the inside of a blood vessel or the respiratory tract, and the linear method is convenient when imaging a joint or the like, and the scanning method can be used properly according to the shape of the subject. Items to be described later 2. In the spectral analysis of the above, when the spectral band showing the characteristics of the subject is narrowed and a speckle pattern is generated in the image, the scanning method of d is used to suppress the interference pattern.
-The scanning method in FIG. 5 corresponds to the convex method in b.

Further, in the sector scanning method c, the diameter of the endoscope can be reduced because the scanning range 343 (see FIG. 15) for scanning the diffused light 10 may be the size of the opening. However, since the scanning range 343 where the synthetic aperture is possible is an area where the diffusion range DA (see FIG. 1) of the scanned diffused light 10 overlaps, it is necessary to widen the diffusion range DA of the illumination light by the amount of widening the viewing angle. There is. Further, in order to deflect the central axis X, the reference signal RS is generated in accordance with the deflection of the central axis X, so that the number of reference signals RS (see FIGS. 11 and 12) increases. By the way, the amount of calculation of the synthetic aperture can be considerably reduced only by reducing the peripheral resolution ξ by about -3 dB as compared with the center of the diffusion range DA.

Further, in the sector scan method c, in order to make the resolution ξ of all pixels constant, the opening AP (scanning range of diffused light) required for detecting the pixel 42 (see FIG. 18) having the longest optical path length OPER. Is selected, the interval P required for the pixel 42 having the shortest optical path length is selected, and the diffusion range DA of the diffused light 10 required for the pixel 42 having the largest deflection angle α (see FIG. 18) (see FIG. 18). (See FIG. 1) may be selected. FIG. 18 shows a conceptual diagram of phase matching of the sector scan method c. Reference numeral AP in FIG. 18 indicates an opening required for detection.

2. Spectrum analysis Next, the spectrum analysis function of the endoscope device of the present invention will be described. First, the significance of spectrum analysis will be described, then the properties of the reflection spectrum will be described, then the problems and solutions of spectrum analysis will be described, and finally, examples of spectrum analysis will be described.

1) Significance of spectrum analysis ・ As for human visual function, the eyes and brain are integrated to realize a wide field of view, high resolution, and high dynamic range. The high-definition visual field (fovea) of the human eye has a viewing angle range of only 2 °, and the resolution drops sharply as it goes outside the visual field. By scanning this range of viewing angle of 2 ° at high speed and synthesizing the obtained image in the brain, it is as if the entire field of view is being observed in high definition. In addition, by adjusting the pupil at high speed according to the average brightness of the field of view with a viewing angle of 2 °, an image with a high dynamic range is synthesized over the entire field of view. Indeed, human visual function is an integrated function that awakens when viewed.

-Such visual functions demonstrate high recognition ability for the form, texture, and movement of the subject. However, human visual function has only the resolution of the three primary colors of XYZ in terms of spectrum, and the ability to recognize the spectral components of light and identify substances is by no means high.

-In diagnostic imaging, there is a strong need to perform non-invasive and non-contact tissue characterization (Tissue characterization) at the pixel level, and as one of the means, spectral analysis beyond human color vision is expected. There is. The wavelength band of spectrum analysis is the region from the visible light region excluding ultraviolet rays and X-rays, which are highly invasive, to the infrared region and terahertz.

2) Reflection and absorption in the wavelength band The mechanism of light reflection and absorption in the living body differs depending on the wavelength band.
-In the visible light band, changes in the spectral components that are absorbed by exciting the vibrations and spins of biomolecules are superimposed on the reflected light of Rayleigh scattering to form a color.

-In the infrared band, since the wavelength is long, molecular vibration and intermolecular vibration are easily excited, and absorption of a spectrum highly correlated with the molecular structure occurs strongly. There is a wavelength band called the fingerprint region, and infrared spectroscopy for identifying substances has been established. Similarly in the infrared band, changes in the absorption spectrum are superimposed on the reflected light of Rayleigh scattering.

-Also, regardless of the wavelength band, a phenomenon occurs in which part of the light energy is absorbed by molecular vibrations and the remaining energy is emitted as autofluorescence or Raman scattered light. The spectral components of fluorescence and Raman scattered light have a high correlation with the molecular structure, but the light intensity is weak. However, since the wavelength shift (ε = h · v) according to the emitted energy occurs, Rayleigh scattering of the excitation light as the background light can be removed by the wavelength band filter, and stability and SN can be ensured. Therefore, fluorescence and Raman scattered light spectroscopy are suitable for reflection-type detection.

3) Problems of spectrum analysis and means of solution ・ Since the spectral components of Rayleigh scattered light itself change to some extent depending on the environment, such as the angle of illumination and the distribution of the refractive index of the subject, it is superimposed on Rayleigh scattered light (reflected light). Absorption spectrum stability and SN are generally not high.

-However, human color vision can distinguish colors even if the state of Rayleigh scattering changes depending on the angle of illumination or the spectral characteristics of illumination change to some extent (color constancy in Retinex theory). This is because the color space formed by the eyes and the brain as one unit performs statistical discrimination and non-linear discrimination by experience.

-Similarly, by applying statistical analysis or non-linear discrimination such as AI to multispectral, it is possible to realize spectrum analysis beyond human color vision. When a large amount of reflected light spectrum data is acquired and multivariate analysis is performed by orthogonal transformation in a space where the spectral components are multidimensional orthogonal axes, clusters such as normal cells and cancer cells can be separated and identified. It is generally known that there are many.

-For each substance, the optimal spectral space axis for identifying the substance (the axis with the largest Fisher ratio between the clusters to be identified) is determined in advance by multivariate analysis, and the number of axes is calculated from the cumulative contribution rate. When narrowing down to the following, assigning the information obtained on that axis to the axis with high visual resolution in the color space (3D) (for example, YIQ with visual resolution of 4: 1.5: 0.5) is effective for diagnostic imaging. Can be supported. Once displayed in color space, the observer's visual brain then performs non-linear discrimination.

-If the number of axes is large, the clusters formed in that axis space are narrowed down to 3 axes by AI (Deep Learning), which is good at non-linear identification, and as described above, a color space that is easy for human vision to recognize. You may support the diagnosis by converting it to the axis of and displaying it.

-Although the multispectral itself may be input to the trained AI to identify the substance, the scale of AI can be significantly reduced by narrowing down the number of AI inputs by multivariate analysis, which is a preprocessing. As with data compression, the number of input axes can be narrowed down to at most 5-6.

・ AI is better than human beings in learning a large amount of information in a short time and giving an answer in a short time from a large amount of information if the application scene and range are limited. The reason is that AI does not get tired, so it is possible to learn a large amount of information in a short time by the speed of electricity day and night, and forget the large amount of information used for learning and the learning result. Since there is no such thing (in the case of human beings, forgetting is said to be a means to debug the brain), if the scenes and scope of application are limited, there are many cases where it is superior to human beings. AI is effective in cases such as multispectral analysis where simple spectral pattern recognition is performed, but the number of variables is large, and unstable noise is mixed with various generation factors such as Rayleigh scattering. ..

Based on the above problems and background, the procedure for performing spectrum analysis by the endoscope device of the embodiment to which the present invention is applied and the functions required for the spectrum analysis will be described below.

・ First, in order to acquire as many multi-spectral image data as possible by dividing the visible light to infrared wavelength band as finely as possible by the endoscopy device, and to reduce the tags and cluster dispersion consistent with the definitive diagnosis. Information necessary for preprocessing of (for example, wavelength band characteristics of the light source at the time of data acquisition, wavelength band characteristics until conversion into an electric signal by the light receiving line detector, etc.) is attached to the image data, and the external computer It is sent to a storage device and stored.

-Next, after preprocessing the accumulated multispectral data with an external computer, the optimum spectral space axis is calculated to identify the target substance by principal component analysis or FS (Foley-Sammon) conversion. do. Since the spectral space axes are calculated in descending order of cumulative contribution rate, the number of spectral space axes is narrowed down in the same manner as for data compression.

-Next, the accumulated multi-spectral data is projected onto the narrowed-down spectral space axis, and the accumulated multi-spectral data is used as training data to cause AI on the computer to perform "supervised learning" to identify the cluster. The configuration of AI is the same as the configuration of AI installed in the endoscope device.

-Then, the basal vector component of the narrowed-down spectral space axis and the knowledge data learned by AI (coefficients of neurons of the hierarchical neural network that composes AI) are sent from the computer to the endoscopy device for storage and identification. These coefficients are switched and used for each substance. The number of spectral space axes narrowed down is at most 5 to 6 axes (according to the experience of the inventor, etc.), and since the scale of AI is small, it is easy to incorporate into an endoscope device and the identification speed is also high. Since it will be faster, real-time substance identification will be possible in vivo.

-As can be seen from the above explanation, the endoscope device acquires a large number of multispectral data in vivo and can store them in an external storage device, and detects the signal of the spectral axis determined by multivariate analysis. It has two functions that enable real-time substance identification in vivo. Examples of these two functions will be described in Section 5). Twice

4) Generation of RGB image Before explaining the embodiment of the spectrum analysis, the operation of generating the RGB image of the OCI processing unit (207 in FIG. 1) will be described.

The output of the light receiving line detector 29 shown in FIG. 7 is input to the FFT 61 of FIG. 19, Fourier transform is performed on the visible light band 80 shown in FIG. 20, and the W (white) signal 81 shown in FIG. 20 is generated. NS. FIG. 20 shows the wavelength bands of various spectral images generated by the Fourier transform.

-Instead of the W signal, the output of the light receiving line detector 29 may be multiplied by a predetermined coefficient to perform a Fourier transform to generate a Y (luminance) signal.

-In addition, when drawing a part deep from the surface of the living body to some extent, the band including the near-infrared region 85 having good biopermeability shown in FIG. 20 may be Fourier transformed and generated as a W signal 81.

-In parallel with the generation of the W signal 81, the FFT 62 of FIG. 19 performs the Fourier transform of the R band shown in FIG. 20 to generate the R signal 82. Pixels of the R signal 82 are interpolated by the interpolation memory unit 63 shown in FIG. 19, and the number of pixels and the time axis are synchronized with the W signal 81. Subsequently, the FFT 62 performs the Fourier transform of the B band shown in FIG. 20 to generate the B signal 83. The B signal 83 is also interpolated by the interpolation memory unit 64 shown in FIG. 19, and the number of pixels and the time axis are synchronized with the W signal 81.

Next, as shown in FIG. 22, the data strings of the W, R, and B signals are stored in the memories 211-1 to 211-3 of the storage unit 211, and then in the synthetic aperture processing unit 213, W , R, and B signals are subjected to synthetic aperture processing 213-1 to 213-3, and then the matrix conversion unit 205-1 performs matrix conversion to an RGB signal to generate a video signal. ..

-Since the wavelength bandwidth of the R signal 82 and the B signal 83 is 1/3 that of the W signal 81, the resolution is also 1/3, but the resolution for R and B of the human eye is compared with the luminance information. There is no problem because it is 1/3.

-If accurate color reproduction information is required, the output of the light receiving line detector 29 (see FIG. 7) can be multiplied by the coefficient of the XYZ color matching function to perform a Fourier transform to obtain an XYZ signal. It is possible.

-As described above, the R signal 82 and the B signal 83 can be generated by performing the Fourier transform on the divided R and B bands (see FIG. 20). It can be considered that the wideband light source 21 (see FIG. 7) consists of the linear sum of a plurality of light sources such as R, G, B and infrared, and everything from illumination to Fourier transform. Since it is a linear process, the process of extracting signals 82 and 83 corresponding to the bands of R and B from the output of the light receiving line detector 29 and performing Fourier transform by the principle of superposition is separate using a single light source of R and B. It is the same as acquiring the signal to and Fourier transforming it. In the same way, for the multispectral MS1 to MSn and the intrinsic spectra EU1 to EUn shown by reference numeral 84 in FIG. 20, which will be described later, the output of the light receiving line detector 29 is multiplied by a predetermined coefficient to obtain the band of each spectrum. By Fourier transforming it (see FFT74-1 to 74-n and FFT68 in FIG. 20), images corresponding to the respective spectra can be obtained.

5) Example of spectrum analysis Next, as an example of spectrum analysis, a case of distinguishing between two tissues, cancer and normal tissue, will be described.

-In order to perform multivariate analysis on an external computer, images of multispectral MS1 to MSn (reference numeral 84 in FIG. 20) obtained by dividing the wavelength band from visible light to infrared light as finely as possible are acquired by an endoscope device. do. First, the site of the cancer tissue and the site of the normal tissue for which the multispectral MS1 to MSn are acquired are designated on the displayed RGB image by an input means such as a mouse. Then, as shown in FIG. 19, the control unit 71 generates a gate signal 76 for cutting out a designated area based on the mouse information. Next, the multispectral MS1 to MSn, which are the outputs of the light receiving line detector (reference numeral 29 in FIG. 7), are Fourier transformed by FFT68 to generate images of the respective spectra. A designated area is cut out from those images by the gate signal 76 and stored in the memory unit 69.

Next, in the data format creation unit 70 shown in FIG. 19, tags for definitive diagnosis of cancer, tags indicating the type, malignancy, progression, etc. of cancer, and spectral characteristics of illumination required for pretreatment, etc. Information is attached to the cut out multispectral MS1 to MSn image data, sent to an external computer, and stored. Follow the same procedure for normal tissue as for cancer tissue. As many image data of multispectral MS1 to MSn of such cancer tissue and normal tissue are acquired and accumulated for each case. The processing by the data format creation unit 70 described above may be performed on an external computer by sending the images of the multispectral MS1 to MSn and the corresponding RGB images to a storage device managed by an external computer.

-In multispectral analysis, the resolution of the spectrum and the resolution of the spectral image are in a trade-off relationship. In any method of detecting multispectrums, a trade-off relationship based on SN, including the completion time, holds. If you want to apply multi-spectral analysis to texture enhancement and contour identification, focus on the resolution (bandwidth) of the spectral image, and if you focus on the accuracy of substance identification, you should focus on the spectral resolution. The number of multispectral to be acquired will be increased.

In this way, the balance between the number of multispectral MS1 to MSn and the bandwidth is appropriately set according to the purpose of application. Since the bandwidths of the multispectral MS1 to MSn used for multivariate analysis should be divided as finely as possible, the emphasis is on the resolution of the spectrum, but the target site is specified by an input means such as a mouse. Since a certain resolution is required for the spectral image in order to cut out from the image, the balance is set according to the purpose.

-Next, as described above, a large amount of multispectral image data of cancer tissue and normal tissue accumulated in an external computer is normalized by the computer, such as the spectral characteristics of illumination, the wavelength band characteristics of the optical processing circuit, and the average brightness. Preprocessing is performed. Preprocessing is important to reduce cluster distribution and improve identification accuracy.

-Next, a computer performs multivariate analysis of the preprocessed image data. When the spectral data of the cancer tissue and the normal cell is displayed pixel by pixel in the space where the multi-spectral component is the multidimensional orthogonal axis (O in FIG. 21), a cluster of the cancer tissue and the normal tissue is formed. If pretreatment is performed to normalize the spectral characteristics of the illumination, the average brightness, and the variation in the sensitivity of the endoscopic device, the dispersion of each cluster is mainly caused by the instability of the superimposed Rayleigh scattering. .. The variance of each of the

clusters

361 and 362 is the smallest, and the distance between the

clusters

361 and 362 is the largest (the Fisher ratio is the largest) so that the two

clusters

361 and 362 of the cancer tissue and the normal tissue can be separated and distinguished. ) Find the projective space by orthogonal transformation. For example, when this projective space is obtained by FS (Foley-Sammon) transformation, the projective axes (eigenvectors) EU1 to EUn are calculated in descending order of Fisher ratio. The inventors have obtained the finding that the number of EU1 to EUn can be reduced to at most 5 to 6 or less, similar to data compression.

-Next, a large amount of accumulated multispectral data MS1 to MSn of cancer tissue and normal tissue is reprojected on the projection axis of the intrinsic spectrum EU1 to EUn by a computer, and the data of the intrinsic spectrum is used on the computer. Have the AI (same configuration as AI137 in FIG. 23) perform "supervised learning" to distinguish between the two clusters. AI is composed of a multi-layer neural network capable of learning Deep Learning. In order to improve the accuracy of identification (suppress the local minimum (other minimum values of the minimum value)) and reduce the scale (number of layers) of AI, "supervised learning" is performed on AI for each identification target. The neuron coefficients obtained by learning are sent to AI137 in FIG. 23 and stored.

-Fig. 21 schematically shows the identification of clusters in the projected space of the calculated intrinsic spectra EU1 to EUn so that the non-linear identification by AI can be easily understood visually. Reference numeral 363 indicates the projection of clusters of cancerous tissue on the planes of EU1 and EU2, reference numeral 364 indicates the projection of clusters of normal tissue, and Z indicates the non-linear threshold value that AI identifies on that plane. Shown. It shows that the data of the two orthogonal axes EU1 and EU2 are converted into one-axis information (in this case, the axis that distinguishes between cancerous tissue and normal tissue) by the non-linear discrimination of AI. In this way, AI enables non-linear identification in the information space by performing identification for each minute area of the information space for each layer, and performs conversion to different information axes. The number of orthogonal axes increases or decreases as needed (due to backpropagation).

-The basis vector component values of EU1 to EUn are sent from an external computer to the control unit 71 shown in FIG. 19, and are set in the control unit 71 as the coefficients 72-1 to 72-n of the multipliers 73-1 to 73-n. It is stored in the memory of. Values necessary for preprocessing such as correction by monitoring the spectral characteristics of the wideband light source are added to the values of the coefficients 72-1 to 72-n.

-Then, the coefficients 72-1 to 72-n are read out from the memory of the control unit 71, and the spectral components output in time series from the light receiving line detector 29 (see FIG. 7) by the multipliers 73-1 to 73-n. To generate a time-series signal projected on the EU1 to EUn axes. By Fourier transforming the time-series signals with FFT 74-1 to 74-n, spectral images for EU1 to EUn can be obtained, respectively.

-When the EU axis with a high cumulative contribution rate is narrowed down to the three axes of EU1, EU2, and EU3, as shown in FIG. 22, after performing the synthetic aperture treatments 214-1 to 214-3 of the spectral components, the human eye. The YIQ signal with a resolution ratio of 4: 1.5: 0.5 is assigned in descending order of contribution on the EU axis, and then converted to RGB by the matrix conversion unit 205-2 and displayed. The identification accuracy on the image of the tissue can be maximized. After that, non-linear discrimination is made by the observer's visual brain in this color space. In addition, when visually identifying clusters from an image in this way, the element that senses the contrast of the image is not the difference in brightness between the pixel of interest and neighboring pixels, but the ratio of brightness (according to the Retinex theory of the visual model). By performing logarithmic conversion of multispectral data to convert the ratio between spectral data into a difference and then performing multivariate analysis, the optimum EU axis for visual identification can be obtained.

-When the EU axis, which has a high identification contribution rate, is 4 or more axes, the image of the spectral component corresponding to each axis is input to the trained AI137, narrowed down to 3 axes, and assigned to the YIQ signal described above for display. You may. Alternatively, since the number of EU axes having a high contribution rate is at most 5 to 6, the substance may be directly specified by AI137. Then, depending on the degree of ignition of the AI output, the hue of the image of the cancer part is changed and expressed, or the outline of the cancer is emphasized and displayed, and the display is converted into a display that is easy to visually identify. Then, the diagnosis support may be effectively performed.

-Although the spectral analysis for the identification of two clusters such as cancer tissue and normal tissue has been described above, it is also possible to collectively identify (grade) a plurality of clusters with AI. However, since the scale (number of layers) of AI becomes large, the identification method described below may be used.

-As shown in 1. of FIG. 24, first, the detected spectral components are used to distinguish between two clusters of normal tissue and cancer tissue for each region of the image, and then the cancer tissue is determined. As shown in 2. of FIG. 24, the type and malignancy of the cancer are determined from the combination of identification of the two clusters, and then the two clusters are shown in 3. of FIG. 24. The degree of progress is judged from the combination of identification of. By identifying the two clusters in a tree shape in this way, it is possible to identify a plurality of (three or more) clusters. For each identification, the coefficients 72-1 to 72-n of EU1 to EUn, which are optimal for identification, are read out from the control unit 71 and used, and by switching the knowledge data of AI137, the scale shown in FIG. 23 is small. AI137 can be used to identify multiple clusters without increasing the time required for identification.
6) Application example of spectrum analysis

-As shown in FIG. 25, since the characteristics of the reflection spectra of the blood BV and the surface mucosa MM are clearly different, the above-mentioned spectrum analysis enables an image display that can effectively distinguish between the two tissues. As described above, cancer can be diagnosed from the vascular structure by the spectral analysis of the endoscope device of the present invention.

-As shown in FIG. 26, since the absorption coefficient characteristics of oxidized hemoglobin (arterial) OH and reduced hemoglobin (vein) RH are different, the above-mentioned spectral analysis can be used to identify the tissues of arterial OH and venous RH and to make a cumulative contribution. It is possible to detect an image of oxygen saturation by converting the values of the high-rate spectrum with a lookup table. Oxygen saturation differs between cancerous tissue and normal tissue.
-As described above, it goes without saying that the endoscope device of the present invention can be used for the spectrum analysis of biological substances having unique reflection characteristics with respect to the spectrum of light (particularly infrared rays).

-In addition, the endoscope device of the present invention can also be used for spectrum analysis of biomolecules having the property of binding to a specific dye, or biomolecules having staining or fluorescence that develop color by reacting with a specific enzyme. There are many dyeing and fluorescent dyes such as ICG (indocyanine green), 5-ALA, BBG, triamcinolone acetonide, and fluorescein. The staining is based on the absorption spectrum, and Rayleigh scattered light is mixed in, but the absorption is strong, so the stability and SN are high. For such staining and fluorescence, the result of staining can be emphasized by using the above-mentioned spectrum analysis. Since some dyes, such as ICG, are weakly toxic, the amount used can be reduced by emphasizing the dyeing results.

-In addition, the fluorescent dye (probe) that attaches to the cancer, the photodynamic therapy that kills the attached cancer when exposed to near infrared rays is added to the fluorescent dye, and the cancer is displaced by the immune effect on the remains of the cancer that died by the photodynamic therapy. Various new reagents and therapeutic agents are being developed, including those with photoimmunotherapy that kills the cancer. For these as well, it is expected that the situation of the probe will be recognized by the above-mentioned spectrum analysis.

-However, in the fluorescence phenomenon, after the energy of an atom (outer shell electron) is once excited, a part of the energy is first absorbed as an intermolecular vibration, and the remaining energy is emitted as fluorescence. Since the non-linear phenomenon of absorption into intermolecular vibration occurs first, the phase correlation between fluorescence and excitation light (reference light) is lost. Therefore, fluorescence cannot be detected by the OCI treatment of the imaging method of the present invention.

However, as disclosed by the inventors in Japanese Patent Application No. 2019-087128, stimulated emission for promoting fluorescence emission when excitation light is applied to a fluorescent dye or probe injected into a biological material to cause spontaneous emission of fluorescence. The fluorescence phenomenon can be measured by irradiating light to amplify the absorption spectrum on the excitation light side and detecting the change in the spectrum on the excitation light side by the above-mentioned spectrum analysis.

-Although Raman scattered light has extremely low signal energy ( ^{10-6 of} excitation light), Rayleigh scattering of excitation light can be removed by a spectral filter that utilizes wavelength shift, so it is detected by the reflection method. Suitable for. Since the Raman scattered light is emitted first, which is a linear phenomenon, and the remaining energy is absorbed as intermolecular vibration, it is possible to detect the Raman spectrum of the subject by the above-mentioned spectral analysis. In addition, a method for improving the image resolution of the Raman spectrum as disclosed in Japanese Patent Application No. 2019-087128 and optimization of the detection band for each substance specified by multivariate analysis (see FIG. 21) are performed by AI. By identifying the substance, it is possible to improve the sensitivity of identification. Further, in the spectral analysis of the endoscope device of the present invention, it is possible to utilize the combined use of the SERS (Surface Enhanced Raman Scattering) effect of Raman scattering and the process of CARS (Coherent anti-Stokes Raman Scattering).

3. 3. Example of scanning mechanism Depending on the purpose, the scanning mechanism as shown below can be used.
-Fig. 27 shows an example of using a micromotor 231 (0.9 mmφ) as a scanning mechanism. The tip of the micromotor 231 is connected to the optical fiber end 51 via the flexible joint 237, and when the micromotor 231 is driven, the flexible joint 237 and the optical fiber end 51 are rotated. Since the tip of the endoscope device having the optical fiber end 51 can be further thinned, it is suitable for observing a luminal-shaped subject such as a blood vessel or an airway. The length of the flexible joint 237 is appropriately set within a range in which the rigidity of the flexible joint 237 can be maintained so that the rotation speed does not become uneven due to twisting. In FIG. 27, reference numeral 235 indicates a sliding member, 233 indicates an optical rotary joint, and RX1 indicates a rotation axis.

-Fig. 28 is an example of a scanning mechanism using the MEMS mirror 131. In FIGS. 28 and 27, reference numeral 51 indicates an optical fiber end, and 137 indicates a virtual scanning line of the optical fiber end 51.

-Fig. 29 is a scanning mechanism that vibrates and scans the optical fiber end with a voice coil or a piezoelectric bimorph such as piezo or polyvinylidene fluoride because the optical fiber has high bending resistance. The tip becomes thinner, and it is possible to perform scanning according to the shape of the subject.

Although the embodiments, examples, examples, etc. of the present invention have been described above, the present invention is not limited to these embodiments, examples, examples, etc., and various modifications and changes can be made within the scope of the gist thereof. be.

AD data; AP opening; ARC arc; BV blood;
DA diffusion range; DL data sequence; L distance; MM surface mucosa;
NA1 numerical aperture; NA2 numerical aperture; OH artery; OPL optical path length;
OS biological surface; P1 optical pulse; P2 optical pulse; R rotating scanning axis;
RH vein; RS reference signal; RX1 optical rotary joint;
X central axis; Y scanning direction;
1 optical fiber; 4 lines; 10 diffused light; 12 lenses;
21 wideband light source; 22 branch couplers; 23,26
Optical circulator; 25 Reference light; 27 Interfering coupler;
28 Grating spectroscope; 29 Light receiving line detector;
31 wavelength sweep light source; 32 branch coupler; 34 light receiving detector;
42 pixels; 45 Reference signal generator; 46 Interpolation circuit; 48 Correlation calculation unit; 51 Optical fiber end; 63, 64 Interpolation memory unit;
69 Memory section; 70 Data format creation section;
71 Control section; 73 Multiplier; 76 Gate signal;
104 line end; 131 mirror; 137 optical fiber end;
171 Synthetic aperture; 173 pixels; 177 data sequence;
201 Endoscope tip; 203 Optical circulator;
205-1, 205-2 Matrix converter; 205
Display processing unit; 207 processing unit;
209 Wideband light source; 211 Storage unit; 213 Synthetic aperture processing unit;
215 Display device; 231,556 micromotors; 233 sliding materials;
233 Optical rotary joint; 237 Flexible joint;
301 Endoscope device; 343 scanning range; 361,362 clusters;
401 Fourier transform; 403 inverse Fourier transform; 407 endoscope;
407 Small-diameter endoscope; 409 1st flexible part; 411 2nd flexible part;
413 Forceps opening; 525 Optical fiber for lighting; 259 Bearings;
553 optical rotary joint; 533 coil spring;
555 Micro Galvano Scanner; 557 Transparent Cover 529 Magnet; 571 Imaging Range; 531 Electric Coil This application claims the benefit of Japanese Patent Application No. 2020-050385 filed on March 21, 2020. Is incorporated herein by reference in its entirety.

Claims

An optical fiber that guides the light supplied from the light source, emits it as diffused light, and receives the reflected light returned from the subject.
An optical interference resolution processing unit that performs resolution processing in the traveling direction of the wavefront of the diffused light according to the imaging principle of the optical coherence tomography method.
A scanning mechanism that scans the diffused light in a scanning direction that intersects the central axis of the diffused light.
A storage unit that stores a data string generated by resolution processing by the optical interference resolution processing unit, and a storage unit.
It is characterized by including a synthetic aperture processing unit that extracts and adds data that matches the optical path length from one end of the optical fiber to the position of the pixel to be detected from the data string stored in the storage unit. Endoscope device.
The endoscope device according to claim 1, wherein the scanning direction is orthogonal to the central axis of the diffused light.
The endoscope device according to claim 1 or 2, wherein the light source produces wideband light or wideband wavelength sweep light.
The optical interference resolution processing unit extracts a reflecting plate for generating reference light, an interference coupler that interferes with the reference light, and a predetermined wavelength band component with respect to the reflected light received by the optical fiber. The invention according to any one of claims 1 to 3, further comprising a device and a Fourier conversion unit that performs Fourier conversion of the predetermined wavelength band component to generate a predetermined signal corresponding to the predetermined wavelength band. Endoscope device.
The synthetic aperture processing unit includes a correlation calculation unit or a Fourier transform unit that matches the phase difference generated between a plurality of data included in the data strings corresponding to the reflected light having the same optical path length. Item 2. The endoscope device according to any one of Items 1 to 4.
It is characterized in that a plurality of spectral components are calculated from the reflection spectrum of a subject whose cluster is known in descending order of Fisher ratio, and the cluster is provided with an identification means for distinguishing from the reflection spectrum of an unknown subject by using the spectral components. The endoscope device according to any one of claims 1 to 5.
The endoscope device according to claim 6, wherein the identification means uses an AI that performs deep learning.