JP6748932B1 - Endoscope device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an endoscope device which does not require an imaging optical system and an image sensor, can be miniaturized and reduced in diameter, and performs imaging and spectrum analysis. SOLUTION: An optical fiber 1 that guides light supplied from a light source 209, emits it as diffused light 10 and receives reflected light returned from a subject 3, and a wave surface 5 of diffused light 10 according to an imaging principle of an optical interference tomography method. The optical interference resolution processing unit 207 that performs resolution processing in the traveling direction, the scanning mechanism 8 that scans the diffused light 10 in the scanning direction Y that intersects the central axis X of the diffused light 10, and the optical interference resolution processing unit 207. The storage unit 211 that stores the data string 177 generated by the resolution processing according to the above, and the data AD-1 to AD-n that match the optical path length OPL from one end 51 of the optical fiber 1 to the position of the pixel 173 to be detected are stored. An endoscopic device including a synthetic aperture processing unit 213 that extracts and adds light from a data string 177 stored in unit 211. [Selection diagram] Fig. 1

Description

INDUSTRIAL APPLICABILITY The present invention enables high-definition imaging with a deep depth of focus and wide-field to microscopic enlargement by enabling focusing for each pixel by electrical processing without using an optical system and an image sensor. The present invention relates to a small-sized and small-diameter endoscope device that enables continuous switching of imaging and additionally enables pixel-level spectrum analysis.

The main problems of the conventional endoscope apparatus are as follows. ・To reduce the pain of the patient and enable observation of narrow spaces in the body, downsizing and downsizing are realized.

・A wide field of view and high-definition imaging are realized to quickly detect diseases of 3 mm or less (which cannot be detected by X-ray imaging or CT).
Realize microscopic magnified imaging to enable observation at the cell level.
Realize pixel-level spectral analysis to obtain qualitative information on the diseased part.

The current state of the existing endoscope apparatus for the above problems will be described below.
-There is a 2mmφ small-diameter endoscopic device that uses an image fiber bundle, but due to the wave-optic limit of the diameter of the image fiber, the number of pixels remains at around 40,000.

-Also, there is a 2 mmφ small-diameter endoscopic device that incorporates an image sensor manufactured according to the latest design rules, but the number of pixels is only about 140,000.

-There is an endoscopic device that can be used from a wide field of view to microscopic expansion, but the diameter is as large as 10 mmφ due to the large optical system. Moreover, since the electronic zoom function is also used, a high-definition image cannot be provided.
-There is no endoscopic device with a spectrum analysis function.

In the imaging method using the image sensor, it is difficult to collectively solve the problems to be solved because the methods for solving the problems of the endoscope device described above have a trade-off relationship.

-For example, if an HD (2000 x 1000 pixel) image sensor is to be installed inside a 2mmφ space at the tip of the endoscope, the diagonal dimension of the image pickup surface of the image sensor must be kept below 1.4mm. At that time, the pixel pitch is required to be 0.65 μm, which exceeds the limit values in the pixel pitch and sensitivity of the image sensor and the resolution of the optical system.

・In addition, if the image sensor is to be equipped with a function to detect the spectrum, it is necessary to prepare a pixel equipped with an optical filter for spectrum detection separately in addition to the RGB filter, which further reduces the size of the image sensor. It gets harder.

In addition, the optical system that enables wide-field observation to microscopic magnifying has a large shape, and the diameter of the endoscope device must be increased. If the electronic zoom function is used together to increase the magnification to reduce the size of the optical system, it is impossible to provide a high-definition image. Displaying a wide-field, high-definition image on a high-definition screen that matches the visual resolution leads to quick detection of a diseased part, rather than using an electronic zoom to magnify and display the image below the resolution of the visual inspection. This is because it is much faster to perform visual screening on a wide-field/high-definition screen than to operate the endoscopic device to perform screening.

JP2008-165236

-Patent Document 1 discloses an endoscope apparatus that does not use an image sensor. This is an endoscope apparatus that scans spot light two-dimensionally and picks up an image. However, when trying to obtain an SD image by such a two-dimensional scanning imaging method, high-speed scanning of 15.7 kHz is required for horizontal lines, and 65.7 kHz is required for HD images of progressive scanning. It is difficult to realize. Further, since the optical fiber is swung and scanned, fatigue (transmission loss) of the optical fiber starts in minutes due to bending resistance of the optical fiber.

Further, in Patent Document 1, realization of a thinner endoscope apparatus is limited by the size of the imaging optical system. In particular, if the imaging optical system is to be provided with functions from wide-field imaging to microscopic magnifying imaging, the optical system becomes even larger and it becomes difficult to reduce the diameter.

The present invention has been made in view of such circumstances. That is, the present invention replaces the imaging optical system, the image sensor, and the like included in the conventional endoscope apparatus with one optical fiber and a one-dimensional scanning mechanism, and the reflected light received by the optical fiber is analyzed outside the imaging unit. By processing the optical information necessary for the image and spectrum analysis, the problems caused by the provision of the image sensor and the like and the problems caused by the high-speed horizontal scanning of Patent Document 1 are solved, and the human body of the endoscope apparatus is solved. The purpose is to realize a smaller size and smaller diameter of the inserted part.

[Means for solving the problem]
The present invention is an imaging-type endoscope apparatus that does not use an imaging optical system and an image sensor in order to solve the problems described above.
-The outline of the imaging method of the present invention will be described with reference to FIG. The diffused light 10 is obliquely irradiated to the subject surface 3, and the obtained reflected light is subjected to OCI (Optical Coherence Imaging) processing to perform line 4 direction resolution, and while repeating line 4 direction resolution, The diffused light 10 is scanned in the scanning direction Y intersecting the line 4 direction. The resolution in the scanning direction Y is performed by performing synthetic aperture processing using a plurality of data strings regarding reflected light obtained by scanning in the line 4 direction and scanning direction Y. As described above, the endoscope apparatus has a method of performing pipeline processing on the subject plane 3 in the line 4 direction and the resolution in the scanning direction Y orthogonal to the line 4 direction to capture an image. Although the OCI processing is technically based on the processing of OCT (Optical Coherence Tomography), the object to be detected and the purpose are different, and hence the name OCI is used in the present application. Details of OCI processing will be described later.

The principle of the above resolution will be described with reference to FIGS. 2(a) to 2(c). When resolution is performed in the direction in which the illumination light is diffused by the OCI processing, the same resolution as that transmitted and received by the spherical optical pulse P1 is obtained as shown in FIG. When the resolution in the scanning direction Y orthogonal to the central axis X of 10 (see FIG. 1) is performed, the fan-shaped P2 is resolved as shown in FIG. It becomes the resolution of. As shown in FIG. 2(c), the overall resolution is the same as that when a circular arc-shaped optical pulse 5 having a thickness on the order of microns is transmitted and received like a radar to obtain reflected light.

When the arc-shaped light pulse 5 obliquely enters the object plane 3 within the range in which it diffuses, focused reflected light is obtained at any position on the line 4, and resolution is made for each pixel. Therefore, the depth of focus is deep, and by switching between the OCI range OC and the synthetic aperture range SA, it is possible to continuously switch from wide-field imaging to microscopic magnifying imaging.

-As can be seen from the above, the image pickup method of the present invention, by means of electrical processing, dynamically focuses on widening the depth of field while maintaining high resolution, and wide-field/high-definition image pickup allows microscopic expansion. Super zooming that continuously switches from imaging to imaging is possible.

The characteristics of the image obtained by the endoscope apparatus of the present invention will be described below.
As shown in FIG. 3, when an image obtained by the imaging method of the present invention is illuminated with diffused light 10 at an oblique angle θ, the reflection point 6 at which the line 4 and the wavefront 5 of the illumination light intersect is reflected by the wavefront. It is the same as the image observed from the observation direction 7, which is the tangential direction of 5.

When the subject is semi-transparent, as shown in the enlarged diagram 3A in the lower left of FIG. 3, the reflection signal (reflection brightness) from the same wavefront inside the subject is superimposed, and the transmission image is obtained from the tangential direction (observation direction 7). The same image as that observed is obtained. If near-infrared light (wavelength 0.68 μm to 1.5 μm) having high biological permeability is included in the illumination, it is possible to obtain a transmission image at a depth of several mm from the surface of the living body (subject surface).

Since the image obtained by illuminating the subject obliquely will be observed from the direction (observation direction 7) orthogonal to the irradiation direction, the specular reflection of the mucous membrane surface escapes forward (to the left in FIG. 3), Since the shadow of the tissue T such as a blood vessel is likely to appear and the contour of the blood vessel is emphasized, the structure of the blood vessel can be easily grasped. Incidentally, the malignancy and progression of cancer have been diagnosed from the structure of blood vessels.

Also, by adjusting the irradiation angle θ of the illumination, it is possible to emphasize slight irregularities on the subject surface 3 and shadows of small tissues under the mucous membrane. For example, it becomes easy to observe contours of capillaries under the mucous membrane, slight irregularities on the joint surface, and slight steps on the edge of the epiretinal membrane on the retina. The irradiation angle θ of the illumination is operated according to the purpose.

A first aspect of the endoscope apparatus of the present invention is an optical fiber that guides light supplied from a light source, emits it as diffused light, and receives reflected light that returns from a subject,
An optical interference resolution processing unit that performs resolution processing in the traveling direction of the wavefront of the diffused light by the imaging principle of optical coherence tomography,
A scanning mechanism that scans the diffused light in a scanning direction that intersects the central axis of the diffused light,
A storage unit that stores a data string generated by the resolution processing by the optical interference resolution processing unit;
And a synthetic aperture processing unit that extracts data that matches the optical path length from one end of the optical fiber to the position of the pixel to be detected from the data string stored in the storage unit and adds the data.

A second aspect of the endoscope apparatus of the present invention is the endoscope apparatus according to the first aspect, wherein the scanning direction is orthogonal to the central axis of the diffused light.

A third aspect of the endoscope apparatus of the present invention is the endoscope apparatus according to the first or second aspect, wherein the light source generates broadband light or broadband wavelength swept light.

A fourth aspect of the endoscope apparatus of the present invention is the endoscope apparatus according to any one of the first to third aspects, wherein the optical interference resolution processing section receives the reflection light by the optical fiber. For light, a reflector for generating reference light, an interference coupler for interfering with the reference light, a spectroscope for extracting a predetermined wavelength band component, and a predetermined wavelength for performing Fourier transform of the predetermined wavelength band component A Fourier transform unit for generating a predetermined signal corresponding to the band.

A sixth aspect of the endoscope apparatus of the present invention is the endoscope apparatus according to any one of the first to fifth aspects, wherein a plurality of clusters are arranged in descending order of Fisher ratio from a reflection spectrum of a known subject. An identification unit is provided which calculates a spectral component and uses the spectral component to identify from a reflection spectrum of an object whose cluster is unknown.

A seventh aspect of the endoscopic device of the present invention is the endoscopic device according to the sixth aspect, wherein the identifying means uses AI for executing deep learning.

According to the present invention, since the imaging unit of the endoscope device has a simple configuration including an optical fiber and a scanning mechanism in a one-dimensional direction, the insertion portion of the endoscope device can be made extremely thin.
According to the endoscope apparatus of the present invention, since horizontal scanning is performed by electrical processing (for example, Fourier transform processing), high-speed horizontal scanning required for HD, 4K, and 8K can be realized, and images are repeated. A vertical scanning mechanism having a low vertical scanning frequency (for example, 60 Hz) can be used for the vertical scanning.

The endoscopic device of the present invention is capable of high-definition imaging in HD, 4K, and 8K even though it is small and small in diameter, and continuously switches from such an image to a microscopic magnified image. It is possible to electrically increase the depth of field.

The endoscope apparatus of the present invention is small in size and small in diameter, but can identify the substance of the subject at the pixel level by spectrum analysis.

Since the endoscopic device of the present invention does not use an image sensor that is weak against heat, autoclave sterilization is possible.

It is a block diagram which shows the structure of the endoscope apparatus which is embodiment. 3A and 3B are diagrams for explaining the principle of imaging by the endoscope apparatus according to the embodiment, FIG. 7A is a diagram showing resolution by OCI, and FIG. 7B is a diagram showing resolution by synthetic aperture. Yes, (c) is a schematic diagram schematically showing an optical pulse diffused on the object surface. It is a figure for demonstrating an observation image. It is a figure for demonstrating the relationship of the reciprocal optical path length of the fiber end and the circular arc of the irradiation light irradiated from an optical fiber. It is a block diagram which shows the structure of a scanning mechanism. (A) is a figure explaining an imaging method of a living body by OCT processing, and (a) is a figure explaining an imaging method of a living body by OCI processing. It is a block diagram which shows the OCI process using a broadband light source. It is a figure for explaining OCI processing by Fourier analysis. It is a block diagram which shows the OCI process using a wavelength swept light source. It is a conceptual diagram which shows the concept of the synthetic aperture in a sector scan system. It is a block diagram which shows the example of the synthetic|combination aperture process by the complementation process and correlation calculation which complement acquisition data. It is a block diagram which shows the example of the complementary process which complements acquired data, and the synthetic aperture process by Fourier transform. It is explanatory drawing for demonstrating the example which performs microscope imaging by the structure combined with the existing endoscope. It is a block diagram which shows the structure of the imaging part of a small diameter endoscope apparatus. It is a section explanatory view for explaining microscope imaging. It is a figure explaining synthetic aperture processing. It is a figure explaining the scanning system which scans diffused light, (a) is a linear system, (b) is a linear system, (c) is a sector scan system, (d) is an example of a multi-scan system, (e) is an example. It is another example of the multi-scan method. It is a conceptual diagram which shows the concept of phase matching in a sector scan system. It is a block diagram which shows detection of RGB and a spectrum image. It is a figure explaining the range of Fourier transformation. It is a figure explaining FS (Foley Sammon) conversion. It is a block diagram which shows the synthetic aperture process of RGB and a spectrum image, and a matrix conversion process. It is a block diagram explaining identifying a spectral image with AI. It is a figure explaining a cluster identification flow. It is a graph which shows the reflection spectrum characteristic of blood and a surface mucous membrane. 6 is a graph showing absorption coefficient characteristics of oxyhemoglobin (arteries) and deoxyhemoglobin (veins). It is a block diagram which shows the structure of the scanning mechanism which rotates an optical fiber end by a micromotor. It is a block diagram which shows the structure of the scanning mechanism which uses a MEMS mirror. (A), (b) is a block diagram which shows the structure of the scanning mechanism which vibrates an optical fiber end by a piezoelectric bimorph, (c) shows an optical fiber bar edge by two piezoelectric bimorph vibrators whose vibration directions are orthogonal. It is a block diagram which shows the structure of the scanning mechanism which rotates and vibrates.

Hereinafter, an endoscope apparatus according to an embodiment of the present invention will be described with reference to the drawings. In the drawings, the same parts are designated by the same reference numerals.

1. Example of the present invention
An embodiment of the imaging method of the present invention will be described below.
1) Basic operation of the embodiment of the image pickup system of the present invention: As shown in FIG. 1, the light from the broadband light source 209 is guided to the endoscope distal end portion 201 by the optical fiber 1 via the optical circulator 203, and the optical fiber end. It is emitted from 51 and the subject surface 3 is illuminated obliquely within a predetermined diffusion range DA. The position of the endoscope distal end portion 201 with respect to the object plane 3 is set so that the angle θ at which the central axis X of the diffused light and the object plane 3 intersect is in the range of 0°<θ<90°, preferably 45 degrees. Is operated.

At this time, as shown in FIG. 4, the reflected light from the arc ARC (ARC1, ARC2) has the same round-trip optical path lengths from the optical fiber end 51, and therefore is superimposed in the same phase and received by the optical fiber end 51. The line 4 indicates one of the lines on the subject plane 3 that is orthogonal to the arc ARC. The optical path length OPL from the optical fiber end 51 to the arc ARC gradually becomes longer as the position of the arc ARC becomes farther from one end (line end) 104 of the line 4.

The reflected light received by the optical fiber end 51 is guided to the OCI processing unit 207 via the optical circulator (see reference numeral 203 in FIG. 1). In the OCI processing unit 207, by the same processing as OCT, the difference in the round-trip optical path length OPL from the optical fiber end 51 to the arc ARC shown in FIG. Resolution is made. The operations of the OCI processing unit 207 will be described in detail in sections 2) to 5) described later.

Then, while repeating the operation of the OCI processing unit 207, the one-dimensional scanning mechanism 8 scans the diffused light 10 in the scanning direction Y intersecting the central axis X, and a data string required for the synthetic aperture is acquired and stored. It is stored in the unit 211.

FIG. 5 shows an example of the scanning mechanism. Reference numeral 511 denotes an optical fiber that performs rotary reciprocating scanning while obliquely illuminating the subject surface 3, 555 denotes a microgalvano scanner, 553 denotes an optical rotary joint, and 112 denotes an optical fiber for transmission. Reference numeral 12 indicates a GRIN (Gradient Index) lens. The GRIN lens 12 is used not for forming an image but for adjusting the diffusion distribution of the light intensity, and the aperture diameter can be made relatively small. Instead of the micro galvano scanner 555, a micro motor that performs rotational scanning may be used. The coil spring 533 becomes unnecessary when using a micromotor. Micromotors with 0.9mmφ or 0.6mmφ are commercially available, so there is no concern about the size. Examples of other scanning mechanisms will be described later.

Next, the data required for the synthetic aperture is read from the storage unit 211, and the synthetic aperture processing unit 213 performs a process of synthesizing the aperture in the scanning direction Y to perform resolution in the scanning direction Y.
The operation of the synthetic aperture processing unit 213 will be described in detail later in 6) and 7).

Finally, the display processing unit 205 performs matrix conversion into an RGB image, processing for displaying the AI determination result, scan conversion according to the display device 215, and the like, and the display device 215 displays the RGB image and spectrum image. Is displayed.

2) Difference between OCI processing and OCT processing As described above, OCI processing is optical coherent resolution processing based on OCT processing (processing based on the imaging principle of optical coherence tomography), but the purpose is as follows. Are different from each other, there are different parts in both processes.

Since the OCT processing aims to detect a tomographic image inside from the biological surface OS, the depth of field D1 is set deep as shown in FIG. 6(a). Therefore, the numerical aperture NA1 of the optical system cannot be increased (the resolution is increased) (NA1 is relatively small). In addition, the wavelength band used in OCT processing is limited to near-infrared light with high biological permeability, and since attenuation and scattering for each wavelength are superimposed during propagation in the living body, RGB image generation and quantitative Spectral analysis is not possible.

On the other hand, since the OCI process aims to detect the image of the living body surface OS which is the subject surface, the numerical aperture NA2 can be set relatively large as shown in FIG. 6B. Further, unlike the OCT process, the irradiation light is not attenuated when it propagates in the living body, so there is a margin in sensitivity, and by adjusting the opening of the optical fiber end 51 with a pinhole, high resolution microscopically can be realized. In addition, since the OCI processing can use a wide band light and there is no attenuation or scattering for each wavelength that is superposed when the irradiation light propagates in the living body unlike the OCT processing, RGB image detection and quantitative spectrum Analysis is possible.

-Therefore, in the OCI processing, the technology of OCT processing is used as a base, the OCT processing is performed using broadband light, and in addition, the processing for RGB image detection and spectrum analysis is performed. The operations of RGB image detection and spectrum analysis will be described in detail later in [Spectrum Analysis].

3) OCI Processing Next, an OCI processing section using a broadband light source will be described.
As shown in FIG. 7, the light from the broadband light source 21 is divided into two by the branching coupler 22, and one of the branched light is reflected by the reflecting plate 24 via the optical circulator 23, and then the reference light. 25 is obtained.

The other branched light is guided to the endoscope front end portion 201 by the optical fiber 1 via the optical circulator 26, and obliquely irradiates the subject surface 3 as shown in FIG.

Next, the reflected light from the arc ARC shown in FIG. 4 is received by the optical fiber 1 and interferes with the reference light by the interference coupler 27 via the optical circulator 26. When the interference occurs, interference fringes are generated at a frequency proportional to the difference between the optical path length OPL of the reflected light and the reference light 25, and they become the light superposed by the number of reflection points from the line 4. As the optical path length OPL of the reflected light becomes longer than the optical path length of the reference light 25, the frequency of the interference fringe becomes higher.

Next, the output light of the interference coupler 27 is split into a wave number (reciprocal number of wavelength) component by the toroidal grating spectroscope 28, and the wave number component is received by the light receiving line detector 29, so that the light carrier disappears. , An electric signal whose wave number component is time-series is obtained. This electric signal has a component in which the components indicating the interference fringes generated according to the difference in the round-trip optical path lengths of the reflected light and the reference light 25 are superimposed by the number of the reflection points on the line 4.

Next, when Fourier transform is performed on this electric signal by the FFT 30, the frequency component after the Fourier transform corresponds to the difference between the optical path length OPL of the reference light 25 and each arcuate reflection point ARC, and the amplitude of each frequency component is increased. Corresponds to the reflection intensity. As a result, the resolution is made in the direction orthogonal to the arc-shaped reflection point ARC, and the reflection signal reflected from the arc-shaped reflection point ARC is obtained.

The data sequence of the amplitude and phase of the reflection signal of the arc-shaped reflection point ARC obtained by repeating the above operation while scanning the diffused light 10 in the scanning direction Y by the scanning mechanism described later is stored in the storage unit 211. Sent to and stored in.

When the wavelength (wave number) band of the broadband light source 21 shown in FIG. 7 is widened, the wave number of the interference fringes increases in proportion thereto, so the width of the single spectrum when Fourier transforming the wavelength signal becomes narrow, and the line 4 direction Will increase the resolution.

The resolution σ in the line 4 direction shown in FIG. 1 is σ=[0.44·λc ² /Δλ]/cos θ when the optical fiber end 51 is regarded as a point light source and the shape of the wavelength band of the broadband light source 209 is a normal function. It is represented by. The resolution σ is proportional to the width of the wavelength band. Δλ indicates the wavelength bandwidth, λc indicates the center wavelength, and θ indicates the angle at which the central axis X of the diffused light and the line 4 intersect.

Further, the longer the repetition cycle of the OCI processing, the larger the output of the light receiving line detector 29 (see FIG. 7), so that the sensitivity of the OCI processing increases.

4) Supplementary explanation of OCI processing A supplementary explanation of OCI processing by Fourier analysis will be given below.
The first term of the equation shown in FIG. 8(a) is a schematic representation of broadband illumination light. The second term in FIG. 8A represents the relationship between the reference light 25 and the optical path length OPL of the reflected light as a δ function when the number of reflection points on the line 4 is one. The value of L indicates the round-trip optical path length of the reflected light when the optical path length OPL of the reference light 25 is set to 0. The process of causing the reference light 25 to interfere with the reflected light can be expressed as a process in which the illumination light is superimposed and integrated (*) at the reflection position of the reference light 25 and the reflected light, as in the formula of FIG. In the second term of FIG. 8A, k indicates the amplitude of the reference wave, and A indicates the amplitude of the reflected wave.

The electric signal converted by the light-receiving line detector 29 after the composite product obtained by the equation of FIG. 8A is separated by the toroidal grating spectroscope 28 (see FIG. 7) is shown in FIG. Since this is equivalent to detecting the light carrier by Fourier transform (spectral) of the equation, the wave number band and the second term of the illumination light of the first term of the equation of FIG. The electric signal of the third term obtained by multiplying (X) the interference fringes of is obtained. The first term τ in FIG. 8B means the wave number component of broadband light. The second term of the equation in FIG. 8B shows an interference fringe having a frequency proportional to the distance L (the cycle is inversely proportional to the distance L). Since all the reflected light from the reflection position on the line 4 is superposed on the actual reflected light, all the interference fringes of the corresponding frequency are superposed. The amplitude k of the reference light shown in the second term of the equation of FIG. 8B becomes the DC component of the interference fringes, and the intensity A of the reflected light becomes the amplitude of the interference fringes.

Since the second term in FIGS. 8A and 8B is a Fourier transform pair, if the position axis (time axis) and the wave number axis (frequency axis) are interchanged, the waveform after the Fourier transform is obtained. easy to understand. Incidentally, when the position (distance) axis is Fourier transformed, it becomes a wave number (spatial frequency) axis, but since it is converted into a time-series electric signal by the light receiving line detector 29 (see FIG. 7), FIG. The wave number axis is the time axis as shown in the third term of.

Next, when the equation of FIG. 8(b) converted into an electric signal is inverse Fourier transformed by the FFT 30, the phase of the wave number band component of the first term of the equation of FIG. 8(b) is matched, and The pulse shown in the first term of the equation (c) is obtained. The interference fringes of the second term in the equation of FIG. 8B return to the δ function, as shown in the second term of the equation of FIG. 8C. According to the theorem of superposition integral, the multiplication of the equation of FIG. 8(b) is the superposition integral of the equation of FIG. 8(c). Therefore, as shown in the third term of the equation of FIG. A pulse whose amplitude is A at the position and whose width is inversely proportional to the wave number band of the illumination light is obtained. The full width at half maximum of the pulse in the third term of the equation in FIG. 8C is the resolution in the direction orthogonal to the arc ARC (line 4 direction).

5) OCI Processing Using Wavelength Swept Light Source Next, the OCTI processing when the wavelength swept light source 31 is used instead of the broadband light source 21 will be described with reference to FIG.

The light in which the wave number (the reciprocal of the wavelength) of the illumination light is linearly modulated by the wavelength sweeping light source 31 is input to the branching coupler 32 within the repeated scanning time (TIME) of the OCI processing. Further, the output light from the interference coupler 33 is converted into an electric signal by the light receiving detector 34. The other processing is the same as the processing performed in the OCI processing of the broadband light source 21 described with reference to FIG. 7.

When the wavelength swept light source 31 is used, the grating spectroscope 28 and the light receiving line detector 29 shown in FIG. 7 are not required, and a single light receiving detector 34 having good sensitivity and SN can be used. However, since the broadband wavelength swept light source 31 such as visible light needs to synthesize a plurality of wavelength swept light sources, the light source is selected according to the purpose in consideration of applications such as multi-spectral analysis described later. What to do and to use the light source in combination.

6) Synthetic Aperture Processing Next, the operation of the synthetic aperture processing unit 213 (see FIG. 1) will be described.
As shown in FIG. 10, according to the size of the opening 171 to be combined, the data strings DL-1 to DL-n (177) necessary for the combining opening from the data strings stored in the storage unit 211 of FIG. Is selected, and the data AD-1 to AD-n that match the optical path length from the optical fiber end 51 to the detection pixel 173 are read from the data strings DL-1 to DL-n. In order to suppress the quantization error when reading the data AD-1 to AD-n, the interpolation circuit 46 shown in FIG. 11 performs the amplitude and phase interpolation from the data of the addresses before and after and reads the data AD-1. ~ Increase the accuracy of AD-n. The addresses of the data AD-1 to AD-n to be read become parabolic 175 by the first-order approximation.

Next, when the phase shifts corresponding to the optical path differences OPD-1 to OPD-n of the read data AD-1 to AD-n are aligned and added, the pixel 173 is detected by the optical system having the same aperture. Will be the same as. By matching and adding the phases, the amplitude of the pixel 173 to be detected is increased by the number of the data strings DL-1 to DL-n, and the amplitudes of the other pixels used in the synthetic aperture 171 are the arcs of FIG. As the distance from the line 4 is increased on the ARC, the phases (positions) are deviated to cancel each other. A supplementary explanation of this phase shift will be given using Fourier analysis in section 7).

The fact that the phases of the above-mentioned data DL-1 to DL-n are matched and addition is performed and the processing is repeated in the scanning direction Y means that the data DL-1 .About.DL-n, the correlation calculation of the reference signal RS is performed in the direction of the data string 177. In order to obtain a constant resolution (numerical aperture) on the line 4, the reference signal RS is selected according to the optical path length of the pixel 173 to be detected (corresponding to the reflection position on the line 4). It is necessary to adaptively switch between the range 177 of 1 to DL-n (range 171 of synthetic aperture) and the reference signal RS. The reference signal RS required for the pixel 173 to be detected (synthesized) is stored in the lookup table of the reference signal generation unit 45. Alternatively, since the reference signal RS is a function of the optical path length of the pixel 173 to be detected, it may be generated by calculation.

Regarding the correlation calculation, according to the theorem of the superposition integral, as shown in FIG. 12, the correlation calculation unit of FIG. The method according to 48 gives the same result. The range of data for which the Fourier transform 401 is performed is the range of data stored in the scanning direction Y. When the reference signal in the scanning direction Y is constant as in the scanning method of FIG. 5, the Fourier transform 401 method of FIG. 12 requires a smaller number of multiplications than the correlation calculation method of FIG.

7) Supplementary Explanation of Synthetic Aperture Processing Next, a specific example of the small-diameter endoscope to which the endoscope apparatus of the present invention is applied will be described, and the synthetic-aperture processing in the small-diameter endoscope will be described using Fourier analysis. To do.

・Now, as shown in FIG. 13, through the forceps port (2 to 3 mmφ) 405 of the existing endoscope (6 mmφ thin endoscope) 407, the thin endoscope (2 mmφ) shown in FIG. A case will be described where 500 is pressed against a target portion to perform microscopic imaging. The reason for pressing against the subject surface is to prevent blurring and to increase the numerical aperture of the synthetic aperture.

The operation performed by the observer is such that a diseased part is screened by a wide-field image of the parent endoscope 407, and then a high-definition observation is performed on the attention site by the small-diameter endoscope 500 through the forceps port 405. I do. Then, the observer continues the screening while shortening the observation distance of the high-definition image (while enlarging the image), and presses the small-diameter endoscope 500 on the site requiring cytology among the screened sites. The microscopic enlarged image. As described above, the small-diameter endoscope 500 has a deep depth of focus, and switching from high-definition imaging to microscopic magnifying imaging can be automatically performed only by changing the distance to the subject. Can be done easily. In FIG. 13, the small-diameter endoscope 500 has a first flexible portion 409 that can be bent by wire control, and a spring-shaped second flexible portion 411 that can be bent.

Next, the numerical aperture of the synthetic aperture and the interval P (see FIG. 10) for acquiring the data row 177 for the synthetic aperture in the microscopic magnified image will be described.

As shown in FIGS. 14 and 15, the optical fiber end 51 is rotationally scanned within an arcuate scanning range 343 of 0.7 mm from the center (rotational scanning axis) R of the endoscope device 301, and the optical fiber end 51 is rotated by 1 mm from the optical fiber end 51. When the imaging is performed in the imaging range 571 of 2×2 mm at a distance, the resolution for obtaining 1000×1000 pixels in the imaging range 571 of 2×2 mm needs to be 2 μm or less according to the sampling theorem. The resolution σ in the direction of the line 4 is calculated from the above-described OCT resolution formula, and a resolution of 2 μm or less can be sufficiently achieved by using a visible light band for diffused light. The size T of the aperture required to set the resolution ξ in the scanning direction Y to 2 μm (see Res. in FIG. 16C) has a focal length of 1 mm and a central wavelength λc of the diffused light 10 of 550 nm. At this time, when back-calculated from the approximate formula of image formation, ξ=0.61×λc/NA (NA=sinα), T=0.34 mm is required. In FIG. 14, reference numeral 511 is an optical fiber for illumination, 525 is a bearing, 529 is a magnet, 556 is a micromotor, 533 is a coil spring, 553 is an optical rotary joint, 531 is an electric coil, and 571 is an imaging range of 2×2 mm. Reference numeral 341 in FIG. 15 indicates the surface mucosa, and 343 indicates the scanning range of the optical fiber end 51 of 1.34 mm.

Next, under the above conditions, the interval P when the diffused light 10 is rotationally scanned to acquire the data strings DL-1 to DL-n in FIG. 10 is obtained by Fourier analysis.

Since the apertures 171 to be combined are discrete, as shown in the expression of FIG. 16A, the COM (comb) function of the first term representing the interval P for acquiring the data strings DL-1 to DL-n is obtained. , Is multiplied by the window function (T=0.34 mm) of the second term representing the aperture T, and is further expressed by the formula in which the light receiving sensitivity distribution of the end face of the optical fiber shown in the third term is superposed and integrated.

-Since the synthetic aperture processing is a Fourier transform, according to the theorem of superposition integral, as shown in the equation of Fig. 16(b), the square wave is Fourier-transformed into the com function with the interval 1/P that can be obtained by Fourier transforming the COM function. It can be expressed by a formula in which a sync function formed by conversion is superposed and integrated, and the distribution of diffused light on the focus of the third term (light-receiving sensitivity distribution of optical fiber) is multiplied. The axis unit of the equation of FIG. 16(b) is represented by the spatial frequency, but when converted into the unit of distance, it is represented by the waveform of FIG. 16(c).

・As can be seen from the waveform in Fig. 16(c), when the interval P becomes wider, the 0th order and ±1st order intervals in the waveform in Fig. 16(c) become narrower, and the 0th order PSF (Point Spread Function: photosensitivity) PSF of ±1st order enters into the distribution) as an artifact (virtual image that does not exist originally). To be precise, the integral value (artifact) of the ±1st-order PSF (sidelobe) that overlaps with the 0th-order PSF must be smaller than the signal noise. Here, the diffusion range DA (see FIG. 1) of the diffused light 10 needs to be larger than the aperture in order to combine the apertures, so the diffusion range DA (see FIG. 1) of the diffused light 10 is used. By making it narrower, it is not possible to suppress more than ±1st order. By adjusting the interval P, the ±1st order effect can be suppressed. The smaller the interval P, the wider the 0th order and the ±1st order, and the less the effect of the ±1st order, but the number of data increases accordingly, so consider the balance between SN and the amount of calculation (circuit scale). Set.

・Now, when the interval P in the equation of FIG. 16(a) is set to 1 μm, the 0th order and the ±1st order are within the range necessary for the diffused light 10, as shown in the waveform of FIG. 16(c). Since it is doubled to 0.68 mm (=0.61×λc×2f/P/cos θ, f: focal length, θ: aperture angle), ±1st order artifacts can be suppressed. When imaging is performed on the 2 mm imaging range 571, as shown in FIG. 15, the optical fiber end 51 is scanned and scanned for the 1.34 mm scanning range 343 (scanning angle β is 68 degrees) including the synthetic aperture. In the meantime, 1340 data strings are acquired, and synthetic aperture processing is performed using 340 data strings corresponding to the size of the openings in the acquired data strings.

8) Scanning method of diffused light-FIG. 17 shows a scanning method of scanning the diffused light 10 (see FIG. 3). a is a linear system, c is a sector scan system, and d is a multi-scan system that combines a, b, and c. It is possible to combine apertures in any scanning method. For example, the scanning method can be selectively used according to the shape of the subject, such as the convex method when imaging the inside of a blood vessel or the airway and the linear method when imaging the joint. In the spectral analysis described in Section 10, when the spectral band that is characteristic of the subject becomes narrow and a speckle pattern occurs in the image, the d scanning method is used to suppress it.
-The scanning method of Fig. 5 corresponds to the convex method of b.

Further, in the sector scan method c, since the scanning range 343 (see FIG. 15) for scanning the diffused light 10 may be the size of the opening, the diameter of the endoscope can be reduced. However, since the scanning range 343 where the synthetic aperture is possible is an area where the diffusion range DA (see FIG. 1) of the scanned diffused light 10 overlaps, it is necessary to widen the diffusion range DA of the illumination light as much as the viewing angle is widened. There is. Further, in order to deflect the central axis X, the reference signal RS is generated in accordance with the deflection of the central axis X, so that the number of reference signals RS (see FIGS. 11 and 12) increases. By the way, the amount of calculation of the synthetic aperture can be considerably reduced only by lowering the resolution ξ in the periphery by −3 dB compared with the center of the diffusion range DA.

In addition, in the sector scan method c, in order to keep the resolution ξ of all pixels constant, the aperture AP (scanning range of diffused light) required to detect the pixel 42 (see FIG. 18) having the longest optical path length OPL. Is selected, and the interval P required for the pixel 42 having the shortest optical path length is selected, and the diffusion range DA( of the diffused light 10 required for the pixel 42 having the largest deflection angle α (see FIG. 18) ( 1 (see FIG. 1). FIG. 18 shows a conceptual diagram of phase matching in the sector scan method c. Reference numeral AP in FIG. 18 indicates an aperture required for detection.

2. Spectrum Analysis Next, the spectrum analysis function of the endoscope apparatus of the present invention will be described. First, the significance of the spectrum analysis will be described, then the properties of the reflection spectrum will be described, then the problems and means for solving the spectrum analysis will be described, and finally, an example of the spectrum analysis will be described.

1) Significance of spectral analysis-The human visual function is that the eye and brain are integrated to realize a wide field of view, high resolution, and high dynamic range. The high-definition visual field (fovea) of the human eye has a range of a viewing angle of only 2°, and the resolution sharply decreases as it goes outside the visual field. By scanning this range with a viewing angle of 2° at high speed and synthesizing the obtained images in the brain, it is possible to capture the entire visual field as if observing it in high definition. In addition, by adjusting the pupil at high speed according to the average luminance of the visual field with a viewing angle of 2°, a high dynamic range image is synthesized over the entire visual field. Indeed, the visual function of humans is a function that can be seen when viewed.

-These visual functions demonstrate high recognition ability for the shape, texture, and movement of the subject. However, the human visual function has only the resolution of the three primary colors of XYZ regarding the spectrum, and the ability to identify the substance by recognizing the spectral components of light is by no means high.

・In image diagnosis, there is a strong need for non-invasive, non-contact tissue characterization to be performed at the pixel level. As one of the means, spectral analysis that exceeds human color vision is expected. There is. The wavelength band of the spectrum analysis is the region from the visible light region excluding the highly invasive ultraviolet rays and X-rays to the infrared region and terahertz.

2) Reflection and absorption in the wavelength band The mechanism of light reflection and absorption in the living body differs depending on the wavelength band.
In the visible light band, a change in the spectral component that is absorbed by exciting the vibrations and spins of biomolecules is superimposed on the reflected light of Rayleigh scattering to form a color.

In the infrared band, the longer the wavelength, the easier it is to excite molecular vibrations and intermolecular vibrations, and strong absorption of the spectrum highly correlated with the molecular structure occurs. There is a wavelength band called the fingerprint area, and infrared spectroscopy for identifying substances has been established. Similarly in the infrared band, the change in absorption spectrum is superimposed on the reflected light of Rayleigh scattering.

-Also, regardless of the wavelength band, a part of light energy is absorbed by molecular vibration, and the remaining energy is emitted as autofluorescence or Raman scattered light. The spectral components of fluorescence and Raman scattered light are highly correlated with the molecular structure, but the light intensity is weak. However, since the wavelength shift (ε=h·v) according to the emission energy occurs, Rayleigh scattering of the excitation light that becomes the background light can be removed by the wavelength band filter, and the stability and SN can be secured. Therefore, the spectroscopic method of fluorescence or Raman scattered light is suitable for the reflection type detection.

3) Issues and solutions for spectrum analysis-The spectral components of the Rayleigh scattered light itself change to some extent depending on the environment such as the angle of illumination and the distribution of the refractive index of the subject, so it is superimposed on the Rayleigh scattered light (reflected light). The stability of absorption spectrum and SN are not high in general.

・However, human color vision can be distinguished even if the state of Rayleigh scattering changes depending on the angle of illumination or the spectral characteristics of illumination change to some extent (Retinex theory color constancy). This is because empirical statistical identification and non-linear identification are performed in the color space formed by the eyes and the brain as a unit.

-Similarly, by applying statistical analysis or non-linear identification such as AI to multi-spectrum, it is possible to realize spectrum analysis beyond human color vision. When a large amount of spectral data of reflected light is acquired and multivariate analysis such as orthogonal transformation is performed in a space where spectral components are multidimensional orthogonal axes, it is possible to distinguish between normal cells and cancer cells by clusters. It is generally known that there are many.

・For each substance, the spectrum space axis (the axis that maximizes the Fisher ratio between the clusters to be identified) optimal for identifying the substance is obtained in advance by multivariate analysis, and the number of axes is calculated from the cumulative contribution ratio to 3 When narrowing down to the following, assigning the information obtained on that axis to the axis with high visual resolution in the color space (three-dimensional) (for example, YIQ with visual resolution of 4:1.5:0.5) and displaying it will be effective for image diagnosis. Can be assisted. Once displayed in color space, the observer's visual brain then performs non-linear identification.

・When the number of axes is large, clusters formed in that axis space are narrowed down to 3 axes by AI (Deep Learning), which is good at non-linear identification, and as described above, a color space that can be easily recognized by human eyes. Diagnosis may be supported by converting and displaying the axis of.

・The substance may be specified by inputting the multi-spectrum itself into a trained AI, but if the number of AI inputs is narrowed down by multivariate analysis, which is preprocessing, the scale of AI can be greatly reduced. Like the data compression, the number of input shafts can be narrowed down to 5-6 at most.

・AI is better at learning a large amount of various kinds of information in a short time, and giving an answer from a large amount of large quantities of information in a short time, if the application scene and range are limited. The reason is that AI does not get tired, so you can learn a large amount of various information in a short time by the speed of electricity regardless of day and night, and forget about the large amount of information used for learning and the result of learning. Since there is no such thing (although it is said that forgetting is a means of debugging the brain in the case of humans), if the applicable scene and range are limited, there are many cases where it is superior to humans. The use of AI is effective in cases where a large number of variables are present, such as multi-spectral analysis, but there are a large number of variables, and various unstable factors such as Rayleigh scattering are mixed with unstable noise. ..

Based on the above problems and background, the procedure of performing spectrum analysis by the endoscope apparatus of the embodiment to which the present invention is applied and the function required for the spectrum analysis will be described below.

First of all, to obtain as many multi-spectral image data as possible by dividing the visible to infrared wavelength band as finely as possible with an endoscopic device to reduce tags and cluster dispersion consistent with definitive diagnosis. The information necessary for pre-processing (for example, the wavelength band characteristic of the light source at the time of data acquisition, the wavelength band characteristic until it is converted into an electric signal by the light receiving line detector) is attached to the image data and Send to a storage device for storage.

・Next, after preprocessing the accumulated multispectral data by an external computer, calculate the optimal spectrum space axis to identify the target substance by principal component analysis or FS (Foley-Sammon) conversion. To do. Since the spectrum space axes are calculated in the descending order of cumulative contribution rate, the number of spectrum space axes is narrowed down similarly to data compression.

-Next, project the accumulated multi-spectral data on the narrowed spectrum space axis, and let the AI on the computer perform "supervised learning" for identifying clusters by using it as learning data. The configuration of AI is the same as the configuration of AI installed in the endoscopic device.

・And the basis vector component of the narrowed spectrum space axis and the knowledge data learned by AI (coefficients of neurons of the hierarchical neural network forming AI) are sent from the computer to the endoscopic device to be stored and specified. These coefficients are switched and used for each substance. The number of narrowed spectrum space axes is at most 5 to 6 (according to the experience of the inventors), and because the scale of AI is small, it can be easily incorporated into the endoscope device and the identification speed is also high. Because of the high speed, real-time substance identification is possible in vivo.

-As can be seen from the above description, the endoscopic device acquires a large number of multispectral data in vivo and accumulates it in an external storage device, and detects the signal of the spectrum axis determined by multivariate analysis, It has two functions that enable real-time substance identification in vivo. An example of these two functions will be described in section 5).

4) Generation of RGB Image Before describing an embodiment of spectrum analysis, an operation of generating an RGB image by the OCI processing unit (207 in FIG. 1) will be described.

The output of the light receiving line detector 29 shown in FIG. 7 is input to the FFT 61 of FIG. 19 and Fourier transform is performed on the visible light band 80 shown in FIG. 20 to generate the W (white) signal 81 shown in FIG. It FIG. 20 shows wavelength bands of various spectrum images generated by Fourier transform.

Instead of the W signal, the output of the light receiving line detector 29 may be multiplied by a predetermined coefficient to perform the Fourier transform to generate the Y (luminance) signal.

When a part deeper than the surface of the living body is drawn, the W signal 81 may be generated by performing the Fourier transform on the band including the near-infrared region 85 with good biological permeability shown in FIG.

In parallel with the generation of the W signal 81, the FFT 62 of FIG. 19 performs the Fourier transform of the R band shown in FIG. 20 to generate the R signal 82. The R signal 82 is subjected to pixel interpolation by the interpolation memory unit 63 shown in FIG. 19, and the W signal 81 is synchronized with the number of pixels and the time axis. Then, the FFT 62 performs the Fourier transform of the B band shown in FIG. 20 to generate the B signal 83. The B signal 83 is also subjected to pixel interpolation by the interpolation memory unit 64 shown in FIG. 19, and the W signal 81 is synchronized with the number of pixels and the time axis.

Next, as shown in FIG. 22, data strings of W, R, and B signals are stored in the memories 211-1 to 211-3 of the storage unit 211, and then, in the synthetic aperture processing unit 213, W , R, and B signals are subjected to synthetic aperture processing 213-1 to 213-3, and then the matrix conversion unit 205-1 performs matrix conversion into RGB signals to generate video signals. ..

The R signal 82 and B signal 83 have a wavelength bandwidth of 1/3 that of the W signal 81, so the resolution is also 1/3, but the resolution for human eyes R and B is less than that of luminance information. Since it is 1/3, there is no problem.

Also, if accurate color reproduction information is required, the XYZ signal can be obtained by performing Fourier transform by multiplying the output of the light receiving line detector 29 (see FIG. 7) by the coefficient of the XYZ color matching function. It is possible.

The R signal 82 and the B signal 83 can be generated by performing the Fourier transform on the divided R and B bands as described above (see FIG. 20). This can be considered that the broadband light source 21 (see FIG. 7) is made up of a linear sum of a plurality of light sources such as R, G, B, and infrared, and all from illumination to Fourier transform. Since it is a linear process, the process of extracting the signals 82 and 83 corresponding to the R and B bands from the output of the light receiving line detector 29 and performing the Fourier transform by the principle of superposition is separately performed by using the R and B independent light sources. This is the same as acquiring the signal and performing the Fourier transform. In a similar way, by multiplying the output of the light receiving line detector 29 by a predetermined coefficient, multi-spectrum MS1 to MSn and unique spectra EU1 to EUn shown by reference numeral 84 in FIG. Can be generated and subjected to Fourier transform (see FFT74-1 to 74-n, FFT68 in FIG. 20) to obtain images corresponding to the respective spectra.

5) Example of Spectrum Analysis Next, as an example of spectrum analysis, a case of distinguishing between two tissues of cancer and normal tissue will be described.

・In order to perform multivariate analysis with an external computer, the endoscopic device acquires images of multispectral MS1 to MSn (reference numeral 84 in FIG. 20) obtained by dividing the wavelength band from visible light to infrared as finely as possible. To do. First, on the displayed RGB image, the site of the cancer tissue and the site of the normal tissue from which the multispectral MS1 to MSn are acquired are designated by the input means such as a mouse. Then, as shown in FIG. 19, the control unit 71 generates a gate signal 76 for cutting out the designated area based on the mouse information. Next, the multi-spectrum MS1 to MSn, which are the outputs of the light receiving line detector (reference numeral 29 in FIG. 7), are Fourier-transformed by the FFT 68, so that images of respective spectra are generated. A designated area is cut out from those images by the gate signal 76 and stored in the memory unit 69.

Next, in the data format creation unit 70 shown in FIG. 19, a tag for definitive diagnosis of cancer, a tag indicating the type, malignancy, progression, etc. of cancer, and spectral characteristics of illumination necessary for preprocessing, etc. Information is attached to the cut-out image data of multispectral MS1 to MSn and sent to an external computer for storage. For normal tissue, the same procedure as for cancer tissue is performed. Image data of multispectral MS1 to MSn of such cancer tissue and normal tissue are acquired and accumulated as much as possible for each case. The processing by the data format creation unit 70 described above may be performed on the external computer by sending the images of the multispectral MS1 to MSn and the RGB images corresponding thereto to the storage device managed by the external computer.

-In multispectral analysis, the resolution of the spectrum and the resolution of the spectral image are in a trade-off relationship. In any method of detecting multispectral, a trade-off relationship based on SN including the complete image time is established. If you want to apply multispectral analysis to enhance textures and identify contours, place emphasis on the resolution (bandwidth) of the spectral image, and if you place importance on the substance-specific accuracy, use the spectral resolution. It will increase the number of multi-spectra to be acquired.

In this way, the balance between the number of multispectrals MS1 to MSn and the bandwidth is appropriately set according to the application purpose. The multispectral MS1 to MSn used for multivariate analysis should have the bandwidth divided as finely as possible, so the emphasis will be on the resolution of the spectrum, but the target site should be specified by input means such as a mouse. Since a certain degree of resolution is required for the spectral image in order to cut out from the image, the balance is set according to the purpose.

-Then, the large amount of multi-spectral image data of cancer tissue and normal tissue accumulated in the external computer as described above is normalized by the computer with the spectral characteristic of illumination, the wavelength band characteristic of the optical processing circuit, the average brightness, etc. Pre-processing is performed. The pre-processing is important for suppressing the cluster dispersion and improving the identification accuracy.

-Next, a computer performs multivariate analysis of the preprocessed image data. When the spectrum data of the cancer tissue and normal cells is displayed for each pixel in the space where the multispectral component is the multidimensional orthogonal axis (O in FIG. 21), clusters of cancer tissue and normal tissue are formed. If preprocessing is performed to normalize the spectral characteristics of illumination, average brightness, and variations in the sensitivity of the endoscopic device, the dispersion of each cluster is caused mainly by the instability of the superimposed Rayleigh scattering. .. The clusters 361 and 362 have the smallest variance and the largest distance between the clusters 361 and 362 (the Fisher ratio is the largest so that the two clusters 361 and 362 of the cancer tissue and the normal tissue can be discriminated and distinguished from each other. ) Obtain the projective space by orthogonal transformation. For example, when this projection space is obtained by FS (Foley-Sammon) conversion, the projection axes (eigenvectors) EU1 to EUn are calculated in descending order of Fisher ratio. The inventors have found that the number of EU1 to EUn can be narrowed down to 5 to 6 or less at the same time as data compression.

-Next, the computer accumulates a large amount of multispectral data MS1 to MSn of cancer tissues and normal tissues onto the projection axes of the characteristic spectra EU1 to EUn by the computer, and uses the data of the characteristic spectra on the computer. AI (same configuration as AI 137 in FIG. 23) is made to perform “supervised learning” for identifying two clusters. AI is composed of a multi-layered neural network capable of learning deep learning. In order to improve the accuracy of identification (suppress local minimum (other minimum value of minimum value)) and reduce the scale of AI (number of layers), "supervised learning" is performed on AI for each identification target. The coefficient of the neuron obtained by learning is sent to and stored in the AI 137 of FIG.

FIG. 21 schematically shows the cluster identification in the projective space of the calculated eigen spectra EU1 to EUn so that the AI-based nonlinear identification can be easily understood visually. Reference numeral 363 indicates the projection of the cluster of cancer tissue on the plane of EU1 and EU2, reference numeral 364 indicates the projection of the cluster group of normal tissue, and Z indicates the nonlinear threshold that AI identifies in that plane. Showing. It shows that the data of the two orthogonal axes of EU1 and EU2 are converted into the information of one axis (in this case, the axis that identifies cancer tissue or normal tissue) by the nonlinear identification of AI. In this way, the AI enables the non-linear identification in the information space by performing the identification for each minute area of the information space for each layer, and performs the conversion to different information axes. The number of orthogonal axes is increased or decreased as needed (by backpropagation).

The base vector component values of EU1 to EUn are sent from the external computer to the control unit 71 shown in FIG. 19, and are stored in the control unit 71 as the coefficients 72-1 to 72-n of the multipliers 73-1 to 73-n. Stored in memory. To the values of the coefficients 72-1 to 72-n, values necessary for preprocessing such as correction by monitoring the spectral characteristics of the broadband light source are added.

Then, the coefficients 72-1 to 72-n are read out from the memory of the control unit 71, and the multipliers 73-1 to 73-n output spectral components in time series from the light receiving line detector 29 (see FIG. 7). To generate a time-series signal projected on the axes EU1 to EUn. By performing a Fourier transform on the time-series signals using FFTs 74-1 to 74-n, it is possible to obtain spectral images for EU1 to EUn, respectively.

When the EU axis having a high cumulative contribution rate is narrowed down to the three axes of EU1, EU2, and EU3, as shown in FIG. 22, after performing synthetic aperture processing 214-1 to 214-3 of spectral components, the human eye By assigning to the YIQ signal having a resolution ratio of 4:1.5:0.5 in descending order of the contribution ratio of the EU axis, the matrix conversion unit 205-2 converts the RGB into an RGB image and displays it. The identification accuracy on the tissue image can be maximized. The non-linear discrimination in this color space is then made by the observer's visual brain. Further, when visually identifying clusters from an image in this way, the element that feels the contrast of the image is the brightness ratio, not the difference in brightness between the pixel of interest and neighboring pixels (from the Retinex theory of the visual model), By performing logarithmic conversion of multispectral data to convert the ratio between spectral data into a difference, and then performing multivariate analysis, the EU axis most suitable for visual identification can be obtained.

・When the EU axis with high contribution of discrimination is 4 or more, the image of the spectrum component corresponding to each axis is input to the learned AI 137, narrowed down to 3 axes, and assigned to the YIQ signal and displayed. May be. Alternatively, since the number of EU axes having a high contribution ratio is at most 5 to 6, the substance may be directly specified by AI137. Then, according to the degree of firing of the AI output, the hue of the image of the cancer part is changed and expressed, or the outline of the cancer is emphasized and displayed, and converted into a display that is easy to visually identify. Then, the diagnosis support may be effectively performed.

-As described above, the spectral analysis for identifying two clusters such as a cancer tissue and a normal tissue has been described, but it is also possible to collectively identify (classify) a plurality of clusters by AI. However, since the scale of AI (the number of layers) becomes large, the identification method described below may be performed.

As shown in 1 of FIG. 24, first, the detected spectral components are used to identify two clusters of normal tissue or cancer tissue for each region of the image, and then the determination of cancer tissue is performed. As for the part that appeared, as shown in 2. of FIG. 24, the type of cancer and the grade of malignancy are judged from the combination of identification of two clusters, and then, as shown in 3. of FIG. The degree of progress is determined from the combination of the identifications. In this way, by identifying two clusters in a tree shape, it is possible to identify a plurality of (three or more) clusters. For each identification, the optimum EU1 to EUn coefficients 72-1 to 72-n are read from the control unit 71 and used, and the knowledge data of the AI 137 is switched, so that the scale shown in FIG. 23 is small. AI137 can be used to identify multiple clusters without increasing the time required for identification.
6) Application examples of spectrum analysis

As shown in FIG. 25, since the characteristics of the reflection spectra of blood BV and superficial mucosa MM are clearly different, the above-mentioned spectrum analysis enables image display in which two tissues can be effectively distinguished. Thus, cancer can be diagnosed from the blood vessel structure by the spectrum analysis of the endoscope apparatus of the present invention.

Further, as shown in FIG. 26, since the absorption coefficient characteristics of oxyhemoglobin (artery) OH and deoxyhemoglobin (vein) RH are different, the tissue analysis of arterial OH and vein RH and the cumulative contribution are performed by the above-mentioned spectrum analysis. It is possible to detect an image of oxygen saturation by converting the value of the spectrum having a high rate by a look-up table. Oxygen saturation is different between cancer tissue and normal tissue.
-As described above, it goes without saying that the endoscope apparatus of the present invention can be used for spectrum analysis of a biological material having unique reflection characteristics with respect to the spectrum of light (in particular, infrared).

The endoscopic device of the present invention can also be used for spectral analysis of biomolecules having a property of binding a specific dye, or of biomolecules having staining or fluorescence that develops color by reacting with a specific enzyme. As specific dyes, there are many dyes for staining and fluorescence such as ICG (indocyanine green), 5-ALA, BBG, triamcinolone acetonide, and fluorescein. Staining is due to absorption spectrum, and Rayleigh scattered light is mixed in, but because of strong absorption, stability and SN are high. With respect to such staining and fluorescence, the result of staining can be emphasized by using the above-described spectrum analysis. Some dyes, like ICG, are weakly toxic, so the amount used can be suppressed by emphasizing the staining results.

・In addition, a fluorescent dye (probe) that attaches to cancer, a photodynamic therapy that kills the attached cancer by applying near-infrared rays to the fluorescent dye, and an immune effect on the remains of the cancer that died by photodynamic therapy are transposed A variety of new reagents and therapeutic agents are being developed, including those to which photoimmunotherapy that kills cancers that have been killed is added. Also for these, it is expected that the situation of the probe is recognized by the above-described spectrum analysis.

-However, in the fluorescence phenomenon, after the energy of atoms (outer shell electrons) is once excited, part of the energy is first absorbed as intermolecular vibration and the remaining energy is emitted as fluorescence. Since the non-linear phenomenon of absorption due to intermolecular vibrations occurs first, the correlation between the phases of fluorescence and excitation light (reference light) disappears. Therefore, fluorescence cannot be detected by the OCI processing of the imaging method of the present invention.

However, as disclosed by the inventors in Japanese Patent Application No. 2019-087128, stimulated emission for promoting the emission of fluorescence when excitation light is applied to a fluorescent dye or probe injected into a biological substance to cause spontaneous emission of fluorescence. The fluorescence phenomenon can be measured by applying light to amplify the absorption spectrum on the excitation light side and detecting the change in the excitation light side spectrum by the above-described spectrum analysis.

・Although Raman scattered light has extremely small signal energy (10 ^{−6 of} excitation light), it can be detected by reflection method because Rayleigh scattering of excitation light can be removed by a spectral filter that uses wavelength shift. Suitable for The Raman scattered light is emitted as a linear phenomenon first, and the remaining energy is absorbed as intermolecular vibrations, so that the Raman spectrum of the object can be detected by the above-described spectrum analysis. In addition, the detection band is optimized for each substance specified by the method for improving the resolution of the Raman spectrum image as disclosed in Japanese Patent Application No. 2019-087128 and the multivariate analysis (see FIG. 21), and the AI is used. It is possible to improve the sensitivity of identification by specifying the substance. Further, the spectral analysis of the endoscope apparatus of the present invention can utilize the combined use of SERS (Surface Enhanced Raman Scattering) effect of Raman scattering and the process of CARS (Coherent anti-Stokes Raman Scattering).

3. Example of Scanning Mechanism Depending on the purpose, the following scanning mechanism can be used.
FIG. 27 shows an example of using the micromotor 231 (0.9 mmφ) as the scanning mechanism. The tip of the micromotor 231 is connected to the optical fiber end 51 via the flexible joint 237, and when the micromotor 231 is driven, the flexible joint 237 and the optical fiber end 51 are rotated. Since the distal end portion of the endoscope device having the optical fiber end 51 can be further thinned, it is suitable for observing a luminal subject that is gradually thinned like a blood vessel or an airway. The length of the flexible joint 237 is appropriately set within a range in which the rigidity of the flexible joint 237 can be maintained so that unevenness of the rotation speed due to twisting does not occur. In FIG. 27, reference numeral 235 is a sliding member, 233 is an optical rotary joint, and RX1 is a rotation axis.

-FIG. 28 is an example of a scanning mechanism using the MEMS mirror 131. 28 and 27, reference numeral 51 indicates an optical fiber end, and 137 indicates a virtual scanning line of the optical fiber end 51.

FIG. 29 shows a scanning mechanism in which an optical fiber end is vibrated and scanned by a voice coil or a piezoelectric bimorph such as piezo or polyvinylidene fluoride since the optical fiber has high bending resistance. The tip is thin, and it is possible to perform scanning according to the shape of the subject.

Although the embodiments, examples, and examples of the present invention have been described above, the present invention is not limited to these embodiments, examples, examples, and the like, and various modifications and changes can be made within the scope of the gist thereof. is there.

AD data; AP aperture; ARC arc; BV blood;
DA diffusion range; DL data string; L distance; MM surface mucosa;
NA1 numerical aperture; NA2 numerical aperture; OH artery; OPL optical path length;
OS biological surface; P1 light pulse; P2 light pulse; R rotational scan axis;
RH vein; RS reference signal; RX1 optical rotary joint;
X central axis; Y scanning direction;
1 optical fiber; 4 lines; 10 diffused light; 12 lenses;
21 Broadband light source; 22 Branch coupler; 23, 26
Optical circulator; 25 reference light; 27 interference coupler;
28 grating spectroscope; 29 light receiving line detector;
31 wavelength swept light source; 32 branch coupler; 34 photodetector;
42 pixels; 45 reference signal generation unit; 46 interpolation circuit; 48 correlation calculation unit; 51 optical fiber end; 63, 64 interpolation memory unit;
69 memory unit; 70 data format creation unit;
71 control unit; 73 multiplier; 76 gate signal;
104 line end; 131 mirror; 137 optical fiber end;
171 synthetic aperture; 173 pixels; 177 data string;
201 End of endoscope; 203 Optical circulator;
205-1, 205-2 matrix conversion unit; 205
Display processing unit; 207 Processing unit;
209 broadband light source; 211 storage unit; 213 synthetic aperture processing unit;
215 display device; 231,556 micro motor; 233 sliding material;
233 Optical rotary joint; 237 Flexible joint;
301 endoscope device; 343 scanning range; 361, 362 clusters;
401 Fourier transform; 403 Inverse Fourier transform; 407 Endoscope;
407 small-diameter endoscope; 409 first flexible portion; 411 second flexible portion;
413 forceps mouth; 525 optical fiber for illumination; 529 bearing;
553 Optical rotary joint; 533 Coil spring;
555 Micro Galvano Scanner; 557 Transparent Cover 529 Magnet; 571 Imaging Range; 531 Electric Coil

Claims (7)

An optical fiber that guides the light supplied from the light source, emits it as diffused light, and receives the reflected light that returns from the subject,
An optical interference resolution processing unit that performs resolution processing in the traveling direction of the wavefront of the diffused light by the imaging principle of optical coherence tomography,
A scanning mechanism that scans the diffused light in a scanning direction that intersects the central axis of the diffused light,
A storage unit that stores a data string generated by the resolution processing by the optical interference resolution processing unit;
A synthetic aperture processing unit that extracts data that matches the optical path length from one end of the optical fiber to the position of the pixel to be detected from the data string stored in the storage unit and adds the data. Endoscopic device. The endoscope apparatus according to claim 1, wherein the scanning direction is orthogonal to the central axis of the diffused light. The endoscope apparatus according to claim 1, wherein the light source generates broadband light or broadband wavelength swept light. The optical interference resolution processing unit includes a reflection plate for generating reference light for the reflected light received by the optical fiber, an interference coupler for interfering the reference light, and a spectral component for extracting a predetermined wavelength band component. vessel and, according to any one of claims 1 to 3 and the Fourier transform unit for generating a predetermined signal corresponding to a predetermined wavelength band performs Fourier transform, characterized in that it comprises a predetermined wavelength band component Endoscopic device. The synthetic aperture processing unit includes a correlation calculation unit or a Fourier transform unit that matches a phase difference generated between a plurality of data included in a data string corresponding to the reflected light having the same optical path length. Item 5. The endoscope device according to any one of items 1 to 4. A plurality of spectral components are calculated from a reflection spectrum of a subject whose cluster is known in descending order of Fisher ratio, and the cluster is provided with a discriminating means for discriminating from a reflection spectrum of an unknown subject. The endoscope apparatus according to any one of claims 1 to 5. 7. The endoscope apparatus according to claim 6, wherein the identifying unit uses AI that executes deep learning.

Images