CN115586630A - Optical endoscope imaging system and method based on deep learning and single multimode optical fiber - Google Patents

Abstract

The invention discloses an optical endoscope imaging system and method based on deep learning and single multimode optical fiber, comprising the following steps: the optical path of the laser transmitter is sequentially provided with a microscope objective, a pinhole filter, a first lens, a polaroid, a first adjustable diaphragm and a beam splitter; the spatial light modulator is arranged on one side of the beam splitter, the second adjustable diaphragm, the first collimating objective, the multimode optical fiber, the second collimating objective, the second lens and the CCD are arranged on the other side of the beam splitter along a light path, the first collimating objective and the second collimating objective are respectively arranged at the input end and the output end of the multimode optical fiber, and the microscope objective, the pinhole filter and the lens 1 are used for focusing, filtering, collimating and expanding light emitted by the laser. According to the invention, the method is used for improving the imaging resolution of the multimode optical fiber endoscope and realizing high-definition transmission of complex images in a single multimode optical fiber.

Description

Optical endoscope imaging system and method based on deep learning and single multimode optical fiber

Technical Field

The invention relates to the technical field of optical imaging, in particular to an optical endoscope imaging system and method based on deep learning and a single multimode optical fiber.

Background

The endoscope can go deep into the tissue to image the region which is difficult to observe by human eyes, and presents a focus which is difficult to detect by in vitro imaging. At present, an optical endoscope mainly adopts an optical fiber bundle as a carrier for information transmission, and is influenced by the size of an optical fiber and a dummy pixel, so that the intervention and the imaging definition of the optical fiber bundle endoscope are limited.

Compared with a single-mode optical fiber with the same diameter, the core diameter of the multimode optical fiber is usually tens of times that of the single-mode optical fiber, and multiple optical modes can be transmitted simultaneously. Theoretically, instead of a bundle of optical fibers several millimeters thick, multimode optical fibers several hundred micrometers thick can be used to image tiny spaces that cannot be reached by existing optical endoscopes. However, affected by multimode dispersion, the multimode fiber output presents speckle as a chaotic message. In the past, optical methods relying on high-precision phase measurements have been used for image transmission studies of multimode optical fibers, and have achieved some success in the biomedical field. However, phase drift limits imaging accuracy, making it difficult for multimode fiber optic endoscopes based on conventional optical methods to exit the laboratory. Recently, researchers have proposed a speckle imaging method based on compressed sensing, which combines a single-pixel imaging system and can realize image transmission through multimode fiber without complex wavefront shaping. However, the imaging sharpness of the compressive sensing algorithm depends on the number of samples, and each imaging requires independent calculation and pre-calibration.

Disclosure of Invention

Aiming at the defects in the prior art, the invention aims to provide an optical endoscope imaging system and method based on deep learning and a single multimode optical fiber, which are used for improving the imaging resolution of a multimode optical fiber endoscope and realizing high-definition transmission of complex images in the single multimode optical fiber. To achieve the above objects and other advantages in accordance with the present invention, there is provided an optical endoscopic imaging system based on deep learning and a single multimode optical fiber, comprising:

the optical path of the laser transmitter is sequentially provided with a microscope objective, a pinhole filter, a first lens, a polaroid, a first adjustable diaphragm and a beam splitter;

one side of the beam splitter is provided with a spatial light modulator, the other side of the beam splitter is provided with a second adjustable diaphragm, a first collimating objective lens, a multimode optical fiber, a second collimating objective lens, a second lens and a CCD along a light path, and the input end and the output end of the multimode optical fiber are respectively provided with the first collimating objective lens and the second collimating objective lens.

Preferably, the first collimating objective and the second collimating objective are used for coupling optical carriers carrying image information into and out of the multimode optical fiber, so as to complete transmission of the image information in the multimode optical fiber.

Preferably, the first lens is used for expanding, collimating and transmitting the laser light in parallel;

the second lens is used for adjusting the speckle size of the output end of the multimode fiber received by the CCD.

Preferably, the system further comprises a computer for controlling the optical system and learning an approximate model of the multimode fiber transmission characteristics through the deep neural network and the training set of image-speckle pairs, and further recovering the image from the unknown speckles.

Preferably, the spatial light modulator is a digital micromirror device.

Preferably, the laser emitter is a He-Ne laser having a wavelength of 632.8 nm.

The optical endoscope imaging method applied to the optical endoscope imaging system based on the deep learning and the single multimode optical fiber in claim 1 comprises the following steps:

s1, emitting laser with 632.8nm wavelength through a laser emitter to serve as an experimental light source;

s2, focusing, filtering, collimating and expanding the light source through the microscope objective, the pinhole filter and the first lens;

s3, transmitting the gray image information to a light path of the laser by combining the polaroid, the beam splitter and the spatial light modulator;

s4, coupling optical information into the multimode optical fiber by using a first collimating objective at the input end of the multimode optical fiber, and coupling the optical information out of the multimode optical fiber by using a second collimating objective at the output end of the multimode optical fiber to finish the transmission of the optical information in the multimode optical fiber;

s5, collecting speckles at the output end of the multimode fiber through a second lens, and imaging the speckles onto a CCD camera;

s6, storing speckles at the output end of the multimode fiber corresponding to a large number of images acquired by the CCD through a computer to form an image-speckle pair training set;

s7, sending the image-speckle pair training set into a full convolution neural network for training to obtain an approximate model of multimode fiber transmission characteristics;

s8, continuously collecting speckles by using the endoscope imaging system of claim 1, sending the speckles into the trained model to recover the image, and further realizing high-definition transmission of the complex image in a single multimode optical fiber.

Compared with the prior art, the invention has the beneficial effects that: a single multimode fiber endoscope imaging system is used for obtaining speckles of a large number of complex images, an image-speckle pair is constructed and sent into a deep neural network for learning and training, an approximate model for simulating multimode fiber transmission characteristics is further obtained, the trained model is used for recovering images from unknown speckles, and high-definition transmission of the complex images in the single multimode fiber is achieved. Compared with the traditional optical fiber bundle endoscope, the single multimode optical fiber has lower cost and smaller diameter, and can promote the application of the optical fiber endoscope in interventional medicine. Compared with the traditional optical imaging method such as phase conjugation and full-wave holographic transmission matrix, the deep learning method does not need pre-calibration and high-precision phase measurement equipment, and has the advantages of high imaging speed, high imaging definition and high robustness to changes of optical fiber forms and external environments.

Drawings

FIG. 1 is a three-dimensional block diagram of an optical endoscopic imaging system and method based on deep learning and a single multimode optical fiber in accordance with the present invention;

FIG. 2 is a Digital Micromirror Device (DMD) schematic diagram of an optical endoscopic imaging system and method based on deep learning and single multimode fiber according to the present invention;

FIG. 3 is a diagram of a deep neural network model for an optical endoscopic imaging system and method based on deep learning and a single multimode fiber according to the present invention;

FIG. 4 is a flow chart of an optical endoscopic imaging system and method based on deep learning and a single multimode optical fiber according to the present invention;

FIG. 5 is a diagram of an image-speckle pair training set in embodiment 1 of a deep learning and single multimode fiber based optical endoscopic imaging system and method according to the present invention;

fig. 6 is a reconstruction effect diagram in embodiment 1 of the optical endoscope imaging system and method based on deep learning and single multimode fiber according to the invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Referring to fig. 1-6, an optical endoscopic imaging system based on deep learning and a single multimode fiber, comprising:

the laser device comprises a laser transmitter 1, wherein a microscope objective 2, a pinhole filter 3, a first lens 4, a polaroid 5, a first adjustable diaphragm 6 and a beam splitter 7 are sequentially arranged along the light path of the laser transmitter 1, the microscope objective 2 is at a magnification of 10 times, the microscope objective 2 is arranged behind the laser transmitter and is used for focusing laser, the pinhole filter 3 is arranged behind the microscope objective and is used for filtering, and the polaroid 5 is used for allowing light in one vibration direction to pass through;

a spatial light modulator 8 is arranged on one side of the beam splitter 7, a second adjustable diaphragm 9, a first collimating objective lens 10, a multimode optical fiber 11, a second collimating objective lens 12, a second lens 13 and a CCD14 are arranged on the other side of the beam splitter along a light path, the input end and the output end of the multimode optical fiber 11 are respectively provided with the first collimating objective lens 10 and the second collimating objective lens 12, and the first collimating objective lens 10 and the second collimating objective lens 12 are 20 × magnification.

Further, the first collimating objective lens 10 and the second collimating objective lens 12 are used for coupling optical carriers carrying image information into and out of the multimode optical fiber, so as to complete transmission of the image information in the multimode optical fiber.

Further, the first lens 4 is used for expanding, collimating and transmitting the laser light in parallel;

the second lens 13 is used for adjusting the speckle size of the output end of the multimode fiber received by the CCD 14.

Further, the method comprises a computer, which is used for controlling the optical system and learning the approximate model of the multimode fiber transmission characteristic through the deep neural network and the training set of image-speckle pairs, and further recovering the image from the unknown speckles, as shown in fig. 3, the used deep neural network model diagram is improved compared with the traditional U-NET network as follows:

1. by setting the proper convolution kernel size, step length and Padding type, the feature extraction and dimensionality reduction are realized by using the convolution layer at the same time, the pooling layer is deleted, and the training time is reduced on the premise of ensuring the imaging precision.

2. And an MSE loss function is used for replacing a cross entropy loss function, so that the network is more suitable for the image generation problem.

3. In the down-sampling module, leaky ReLU is used for replacing a traditional ReLU activation function, and the problem of neuron death is reduced.

Further, the spatial light modulator 8 is a digital micromirror device, as shown in fig. 2, the spatial light modulator is preferably a DMD, the DMD is composed of 1024 × 1024 tiny mirrors, and each mirror can realize ± 12 ° inversion in two directions during operation.

Further, the laser emitter 1 is a He — Ne laser having a wavelength of 632.8 nm.

An optical endoscopic imaging method applied to the optical endoscopic imaging system based on deep learning and single multimode optical fiber of claim 1, comprising the following steps:

s1, emitting laser with 632.8nm wavelength through a laser emitter 1 to serve as an experimental light source;

s2, focusing, filtering, collimating and expanding a light source through the microscope objective 2, the pinhole filter 3 and the first lens 4;

s3, transmitting the gray image information to a laser light path by combining the polarizing plate 5, the beam splitter 7 and the spatial light modulator 8;

s4, coupling optical information into the multimode optical fiber by using a first collimating objective 10 at the input end of the multimode optical fiber 11, and coupling the optical information out of the multimode optical fiber by using a second collimating objective 12 at the output end of the multimode optical fiber to finish the transmission of the optical information in the multimode optical fiber;

s5, collecting speckles at the output end of the multimode fiber 11 through a second lens 13, and imaging the speckles onto a CCD14 camera;

s6, storing speckles at the output end of the multimode fiber 11 corresponding to a large number of images acquired by the CCD14 through a computer to form an image-speckle pair training set;

s7, sending the image-speckle pair training set into a full convolution neural network for training to obtain an approximate model of the transmission characteristic of the multimode optical fiber 11;

s8, continuously collecting speckles by using the endoscope imaging system of claim 1, sending the speckles into the trained model to recover the image, and further realizing high-definition transmission of the complex image in a single multimode optical fiber.

Example 1

As shown in fig. 4, the method comprises the steps of:

the Step1 laser emits laser with 632.8nm wavelength as an experimental light source;

step 2, focusing, filtering, collimating and expanding the light source through the microscope objective, the pinhole filter and the lens 1;

step3, transmitting the gray image information to a light path of the laser by combining a polaroid, a beam splitter and a spatial light modulator;

step4, coupling optical information into the multimode optical fiber at the input end of the optical fiber by using an objective lens, and coupling the optical information out of the multimode optical fiber at the output end of the optical fiber by using the same objective lens to finish the transmission of the optical information in the multimode optical fiber;

speckles at the output end of the Step5 optical fiber are collected through an imaging lens (lens 2) and then imaged on a CCD camera;

step6, storing speckles at the output end of the multimode fiber corresponding to a large number of images acquired by the CCD through a computer to form an image-speckle pair training set, as shown in FIG. 5;

step7, sending the image-speckle pair training set into a full convolution neural network for training to obtain an approximate model of multimode fiber transmission characteristics;

step8, continuously acquiring speckles by using the endoscope imaging system of claim 1, sending the speckles into the trained model to recover the image, and further realizing high-definition transmission of the complex image in a single multimode optical fiber, wherein the reconstruction effect is shown in fig. 6.

Compared with the traditional optical fiber bundle endoscope imaging system and the traditional optical imaging method, the optical endoscope imaging system and the method based on the deep learning and the single multimode optical fiber have the advantages that:

1. the conventional fiber optic endoscope imaging system usually uses 20000-30000 single mode fibers to form a fiber bundle, which has a high cost and a large diameter, while the single multimode fiber used in the above embodiments has a smaller diameter and a lower cost. Theoretically, a single multimode optical fiber with the thickness of hundreds of microns can be used for replacing an optical fiber bundle with the thickness of a few millimeters, and tiny space which cannot be reached by the existing optical endoscope can be imaged.

2. Under the influence of multimode dispersion, the output end of the multimode fiber presents a speckle pattern with disordered information, so that the key point of multimode fiber endoscope imaging is to reconstruct a high-definition image from speckles. Conventional optical imaging methods such as phase conjugation and full-wave holographic transmission matrices require highly accurate phase measurements, which means costly phase measurement equipment and places extremely high demands on the stability of the experimental system. The deep learning method does not need pre-calibration and high-precision phase measurement equipment, and has the advantages of high imaging speed, high imaging definition and high robustness on optical fiber form and external environment change.

The number of devices and the scale of the processes described herein are intended to simplify the description of the invention, and applications, modifications and variations of the invention will be apparent to those skilled in the art. While embodiments of the invention have been described above, it is not limited to the applications set forth in the description and the embodiments, which are fully applicable in various fields of endeavor to which the invention pertains, and further modifications may readily be made by those skilled in the art, it being understood that the invention is not limited to the details shown and described herein without departing from the general concept defined by the appended claims and their equivalents.

Claims (7)

1. An optical endoscopic imaging system based on deep learning and a single multimode fiber, comprising:

the device comprises a laser transmitter (1), wherein a microscope objective (2), a pinhole filter (3), a first lens (4), a polaroid (5), a first adjustable diaphragm (6) and a beam splitter (7) are sequentially arranged along the light path of the laser transmitter (1);

a spatial light modulator (8) is arranged on one side of the beam splitter (7), a second adjustable diaphragm (9), a first collimating objective (10), a multimode fiber (11), a second collimating objective (12), a second lens (13) and a CCD (14) are arranged on the other side of the beam splitter along a light path, and the input end and the output end of the multimode fiber (11) are respectively provided with the first collimating objective (10) and the second collimating objective (12).

2. The optical endoscope imaging system based on deep learning and single multimode fiber as claimed in claim 1, wherein the first collimating objective (10) and the second collimating objective (12) are used for coupling the optical carrier carrying the image information into and out of the multimode fiber, so as to complete the transmission of the image information in the multimode fiber.

3. The optical endoscopic imaging system based on deep learning and single multimode fiber according to claim 2, characterized in that said first lens (4) is used for expanding, collimating and parallel transmitting laser light;

the second lens (13) is used for adjusting the speckle size of the output end of the multimode fiber received by the CCD (14).

4. The deep-learning and single-multimode-fiber-based optical endoscopic imaging system of claim 3, further comprising a computer for controlling the optical system and learning an approximate model of the multimode fiber transmission characteristics through the deep neural network and the training set of image-speckle pairs to recover images from unknown speckles.

5. An optical endoscopic imaging system based on deep learning and single multimode fiber as claimed in claim 1 characterized by spatial light modulator (8) selection of digital micromirror device.

6. The optical endoscopic imaging system based on deep learning and single multimode fiber according to claim 5, characterized in that the laser emitter (1) is a He-Ne laser with a wavelength of 632.8 nm.

7. The optical endoscope imaging method applied to the optical endoscope imaging system based on the deep learning and the single multimode optical fiber in the claim 1 is characterized by comprising the following steps:

s1, emitting laser with 632.8nm wavelength as an experimental light source through a laser emitter (1);

s2, focusing, filtering, collimating and expanding a light source through the microscope objective (2), the pinhole filter (3) and the first lens (4);

s3, transmitting the gray image information to a laser light path by combining a polarizing plate (5), a beam splitter (7) and a spatial light modulator (8);

s4, coupling optical information into the multimode optical fiber by using a first collimating objective (10) at the input end of the multimode optical fiber (11), and coupling the optical information out of the multimode optical fiber by using a second collimating objective (12) at the output end of the multimode optical fiber to finish the transmission of the optical information in the multimode optical fiber;

s5, collecting speckles at the output end of the multimode fiber (11) through a second lens (13), and imaging the speckles to a CCD (14) camera;

s6, storing speckles at the output end of the multimode fiber (11) corresponding to a large number of images acquired by the CCD (14) through a computer to form an image-speckle pair training set;

s7, sending the image-speckle pair training set into a full convolution neural network for training to obtain an approximate model of the transmission characteristic of the multimode optical fiber (11);

s8, continuously collecting speckles by using the endoscope imaging system of claim 1, sending the speckles into the trained model to recover the image, and further realizing high-definition transmission of the complex image in a single multimode optical fiber.

Images