CN118021243A - Single-path double-spectrum real-time endoscope device based on depth network reconstruction - Google Patents

Single-path double-spectrum real-time endoscope device based on depth network reconstruction Download PDF

Info

Publication number
CN118021243A
CN118021243A CN202410438802.4A CN202410438802A CN118021243A CN 118021243 A CN118021243 A CN 118021243A CN 202410438802 A CN202410438802 A CN 202410438802A CN 118021243 A CN118021243 A CN 118021243A
Authority
CN
China
Prior art keywords
spectrum
video
reconstruction
double
channel
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202410438802.4A
Other languages
Chinese (zh)
Inventor
李金红
栾衡
冯英
索津莉
闫燕
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Peking University Third Hospital Peking University Third Clinical Medical College
Original Assignee
Peking University Third Hospital Peking University Third Clinical Medical College
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Peking University Third Hospital Peking University Third Clinical Medical College filed Critical Peking University Third Hospital Peking University Third Clinical Medical College
Priority to CN202410438802.4A priority Critical patent/CN118021243A/en
Publication of CN118021243A publication Critical patent/CN118021243A/en
Pending legal-status Critical Current

Links

Landscapes

  • Endoscopes (AREA)

Abstract

The invention provides a depth network reconstruction-based single-channel double-spectrum real-time endoscope device, which comprises a double-spectrum alternate illumination module, a single-channel imaging module, a synchronous control module and an image reconstruction module, wherein incident light rays form broadband and narrowband double-channel illumination after passing through a spectrum switching assembly, then enter an endoscope light path through an illumination optical fiber, an imaging lens assembly continuously collects video image data under the double-channel illumination and transmits the video image data to an image sensor, a broadband and narrowband alternate frame video sequence obtained by the image sensor is fused and reconstructed through a double-channel reconstruction model, and a synchronous continuous video with two-channel frame rate of N frames/second is obtained and output and displayed in real time. The invention integrates a wide-narrow-band light source, adopts a depth network model to fuse the definition of the wide-band video and the imaging contrast of the narrow-band video to obtain a clear and synchronous double-channel video, can synchronously develop the anatomical outline and pathological features of tissues, clearly observe multi-view information in the same visual field, and can greatly improve the inspection efficiency and the operation precision.

Description

Single-path double-spectrum real-time endoscope device based on depth network reconstruction
Technical Field
The invention relates to the technical field of medical imaging devices, in particular to a single-path double-spectrum real-time endoscope device based on depth network reconstruction.
Background
In surgical diagnosis and operation, accurate observation and identification of pathological conditions of human tissue is critical to the establishment of appropriate treatment regimens. The traditional endoscope generally adopts only a single-wavelength light source, and can not effectively display the difference of tissue microstructure and pathological characteristics, thereby influencing the judgment of doctors on the pathological change range. The endoscope can directly observe the morphological change of the mucosal surface and obtain histological evidence, thus being a key means for finding and diagnosing early cancers. Malignant tumors occurring in the nasopharyngeal larynx are mainly squamous cancers, and are only the redness, congestion and the like of mucous membranes in early stages of cancer occurrence, are basically similar to surrounding light red normal tissues, are difficult to find in conventional endoscopy, and are very easy to miss. To address this problem, scientists and endoscopists employ narrow band imaging endoscopes to view lesion diagnostic patients.
Narrowband imaging (NBI) technology can irradiate human tissues with light of specific wavelength, clearly display the capillary vessels and cell morphology of the mucosal surface layer, and has been widely applied to diagnosis and biopsy localization of early cancers. The broadband spectrum in the red, blue and green light waves emitted by the endoscope light source is filtered by the optical filter, and only a narrowband spectrum is left for diagnosing various diseases of the digestive tract. However, NBI only displays information at a narrow band of wavelengths, and the entire outline of the tissue cannot be observed at the same time. In contrast, broadband light sources can better display the anatomy of the tissue, but with lower resolution for the lesion area. Therefore, the dual-spectrum endoscope integrated with wide-narrow-band imaging is expected to realize synchronous observation of tissue pathology and morphology, and improve the accuracy of diagnosis and operation. However, most endoscope systems currently realize single-path development through optical path switching, cannot present two kinds of information at the same time, and brings inconvenience to operation.
Although some existing endoscope systems with two-way imaging can realize simultaneous imaging, a complex light splitting system needs to be arranged in front of an imaging element, so that the system volume and the operation complexity are increased. The existing multi-channel endoscope system using the rotary optical filter has the problems of lower photon utilization rate, higher noise and lower imaging quality, and a plurality of channels of the multi-channel endoscope system are not synchronous, which may cause trouble for doctors to use.
Disclosure of Invention
The invention aims to solve the problems in the prior art, provides a single-path double-spectrum real-time endoscope device for in-vivo development in operation, aims to realize double-spectrum real-time imaging of a focus area, and provides more valuable information for clinical diagnosis and operation treatment.
The invention aims to achieve the aim, and is specifically realized by the following technical scheme:
the invention provides a single-path double-spectrum real-time endoscope device based on depth network reconstruction, which comprises:
The double-spectrum alternating illumination module comprises a broadband white light source, a spectrum switching assembly and an illumination optical fiber, wherein incident light rays of the broadband white light source form broadband and narrowband double-path illumination after passing through the spectrum switching assembly, and then enter an internal optical path of the endoscope through the illumination optical fiber;
The single-path imaging module is arranged in the endoscope and comprises an imaging lens assembly and an image sensor, wherein the imaging lens assembly continuously collects video image data under double-path illumination and transmits the video image data to the image sensor to obtain a wide-narrow-band alternate frame video sequence with a frame rate of 2N frames/sec, N is an integer and is more than or equal to 24 and less than or equal to 30;
the synchronous control module is respectively connected with the double-spectrum alternating illumination module and the single-path imaging module in a synchronous control manner;
And the image reconstruction module is used for carrying out fusion reconstruction on the wide-narrow band alternate frame video sequence acquired by the image sensor through a double-channel reconstruction model to acquire synchronous two paths of continuous videos with the frame rate of N frames/second, and outputting and displaying the continuous videos in real time.
Further, the spectrum switching assembly comprises a filter wheel and a switching motor for driving the filter wheel to rotate, a narrow-band filter is arranged on the filter wheel along a half of the evenly divided rotation center, and the rotation speed of the filter wheel is N revolutions per second.
Further, the wavelength of the narrowband filter is 580-650nm.
Further, the dual-channel reconstruction model is a depth network model trained by using open-source endoscope multispectral video data, and the training process is as follows:
s11, representing the broadband spectrum video and the narrowband spectrum video of the same sample as The acquisition frame rate is 2N frames/second; then, the spectral video of two N frames/second, which are not synchronized, is extracted at intervals of one frame, and is expressed as/>Narrowband spectral video/>Plus noise, broadband spectral data/>No noise is added as input;
s12, taking two synchronous N frames/second spectrum videos which are extracted at intervals of one frame and are not staggered as true values, wherein the training process is expressed as follows:
Wherein the method comprises the steps of Noise representing gaussian distribution,/>And obtaining a two-channel reconstruction model after training is completed as the model parameters.
Further, an Adam optimizer is used for optimization during the training process.
Further, the fusion reconstruction process is as follows:
S21, extracting a wide-narrow band alternate frame video sequence output by an image sensor of the single-channel double-spectrum real-time endoscope device to obtain a width video sequence with the frame rate of N frames/second respectively And narrowband video sequences/>Expressed as:
a delay of 1 frame exists between the two paths of video;
s22, the reconstruction process using the two-channel reconstruction model is expressed as:
Output after model reconstruction Namely, two paths of synchronous continuous videos.
Further, N is 30.
The single-path double-spectrum real-time endoscope device is used for an endoscope to develop the outlines of broadband and narrowband spectrums simultaneously, and improves the accuracy of observing human tissues by doctors. The wide-narrow-band light source is integrated, the depth network model is adopted for reconstruction, the definition of the wide-band video and the imaging contrast of the narrow-band video are fused to obtain a clear and synchronous double-channel video, the anatomical outline and pathological features of tissues can be synchronously developed, multi-view information can be clearly observed in the same visual field, and the inspection efficiency and the operation precision can be greatly improved. In addition, the optics of the present invention are primarily focused on the light source portion and do not substantially increase the volume of the endoscopic portion that enters the body. The device is convenient to operate, can be widely applied to various endoscopic operations such as laparoscopes, arthroscopes and the like, and is hopeful to promote the improvement of clinical diagnosis and treatment level.
Drawings
Fig. 1 is a schematic diagram of the overall structure of the present invention.
FIG. 2 is a training process of a two-channel reconstruction model according to the present invention;
Fig. 3 is a video reconstruction process of the dual channel reconstruction model of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples.
As shown in fig. 1 to 3, the present invention provides a single-path dual-spectrum real-time endoscope apparatus based on depth network reconstruction, comprising:
The double-spectrum alternating illumination module 1 comprises a broadband white light source 11, a spectrum switching assembly 12 and an illumination optical fiber 13, wherein incident light rays of the broadband white light source 11 form broadband and narrowband double-path illumination after passing through the spectrum switching assembly 12, and then enter an endoscope optical path through the illumination optical fiber 13;
The single-path imaging module 2 is arranged in the endoscope and comprises an imaging lens assembly 21 and an image sensor 22, the imaging lens assembly 21 continuously collects video image data under double-path illumination and transmits the video image data to the image sensor 22 to obtain a wide-narrow-band alternate frame video sequence with a frame rate of 2N frames/second, N is an integer and 24-30, N is a common video frame rate such as 24, 25, 30 and the like, and the following embodiments take N as 30 as an example;
The synchronous control module 3 is respectively connected with the double-spectrum alternating illumination module 1 and the single-path imaging module 2 in a synchronous control manner;
And the image reconstruction module 4 performs fusion reconstruction on the wide-narrow band alternate frame video sequence acquired by the image sensor 22 through a double-channel reconstruction model to acquire synchronous two paths of continuous videos with the frame rate of N frames/second, and outputs and displays the continuous videos in real time. The dual-channel reconstruction model acquires a wide-and-narrow-band alternating video frame sequence output by the image sensor 22, and reconstructs a wide-band video sequence and a narrow-band video sequence through image processing algorithms such as segmentation, filtering and the like, wherein the frame rate is 30 frames/second, and continuous and uninterrupted dual-spectrum imaging display is realized.
Preferably, the spectrum switching assembly 12 comprises a filter wheel 121 and a switching motor 122 for driving the filter wheel 121 to rotate, wherein the filter wheel 121 is provided with a narrow-band filter 123 along a half of the rotation center after being evenly divided, and the rotation speed of the filter wheel 121 is 30 revolutions per second.
In the embodiment, the broadband white light source 11 is a light emitting device capable of providing broadband white light, such as a halogen lamp or an LED light source, and the illumination power can be increased if necessary, so as to ensure that the imaging image quality is satisfied after passing through the narrowband filter and the optical path splitting element.
The spectrum switching device 12 is provided with a filter wheel 121 which can rotate at a speed of 30 revolutions per second under the control of a circuit of the synchronous control module 3, wherein the filter wheel 121 is circular in the embodiment, a semicircular narrow-band filter 123 is arranged on the filter wheel, the rest semicircular area is empty or transparent, and illumination light with broadband and narrowband alternately switched is obtained in the rotation process, wherein half of the time is narrow-band illumination, and the other half of the time is broadband illumination. The narrowband region of the filter wheel 121 is set to a spectral band that facilitates observation of the microvascular morphology of the mucosal surface.
Preferably, the wavelength of the narrowband filter is 580-650nm. The narrow-band filter 123 adopts a narrow band in the range of 580-650nm, can enhance the imaging contrast of details such as mucous membrane microvessels, tissues and the like, and is convenient for doctors to observe and diagnose relevant physiological and pathological features.
The illumination fiber 13 is used for leading the broadband and narrowband dual-path illumination fibers formed after the spectrum switching assembly into an endoscope optical path and irradiating the tissue surface of the focus area. The reflected light is imaged by the imaging lens assembly 21 onto the image sensor 22. The imaging lens assembly 21 includes a combination of optical elements such as an objective lens, an interface lens, etc. for imaging, and is capable of imaging an image of the internal surface region of the subject to the image sensor 22 at the distal end; the image sensor 22 is a high-resolution CCD or CMOS sensor and is used for acquiring the target image in real time, and the image sensor 22 images according to the frame rate of 60 frames/second under the control of the synchronous control module 3 so as to meet the requirement of real-time imaging. The lens system and housing portions of the endoscope head need to be miniaturized and designed for isolation to accommodate the needs of endoscopy.
The synchronous control module 3 can accurately control the precise synchronization of the double-spectrum alternative lighting system and the endoscope imaging system through a circuit, control the rotation of the filter wheel, the switching of the lighting source, the exposure of the camera and other time sequences, and ensure that the alternately generated wide and narrow band lighting light is precisely matched with the image exposure. The filter wheel 121 is caused to rapidly switch between broadband and narrowband spectra at a set rate of 30 revolutions per second while precisely triggering imaging of the image sensor 22 at a rate of 60 frames per second.
Preferably, the dual-channel reconstruction model is a depth network model trained by using open-source endoscope multispectral video data, and the training process is as follows:
s11, representing the broadband spectrum video and the narrowband spectrum video of the multispectral video data as The acquisition frame rate is 2N frames/second; the two N frames/second spectral video which are not synchronized and are extracted at intervals of one frame and dislocation are expressed asNarrowband spectral video/>Plus noise, broadband spectral data/>No noise is added as input;
S12, extracting two synchronous N frames/second spectrum videos which are separated by one frame and are not staggered As a true value, its training process is expressed as:
Wherein the method comprises the steps of Noise representing gaussian distribution,/>And obtaining a two-channel reconstruction model after training is completed as the model parameters. Preferably, the Adam optimizer is used for optimization during the training process. Open source endoscopic multispectral video data refers to a public dataset, such as Basque Biobank-PICCOLO RGB/NBI (WIDEFIELD) image collection (bioef. The two-channel reconstruction model obtained through training can accurately capture and separate different components in the image, and performs self-adaptive fusion reconstruction according to different performance characteristics of the components in the wide-narrow-band video.
Preferably, the fusion reconstruction process is as follows:
S21, extracting a wide-narrow band alternate frame video sequence output by an image sensor of the single-channel double-spectrum real-time endoscope device to obtain a width video sequence with the frame rate of N frames/second respectively And narrowband video sequences/>Expressed as:
a delay of 1 frame exists between the two paths of video;
s22, the reconstruction process using the two-channel reconstruction model is expressed as:
Output after model reconstruction Namely, two paths of synchronous continuous videos.
When the dual-channel reconstruction model is actually used, the wide-and-narrow-band alternate frame video sequence output by the image sensor 22 is acquired, one-frame delay exists between two paths of videos, and the frame rates are respectively 30 frames/second and are respectively expressed as. And then, carrying out fusion reconstruction on the two paths of videos through the reconstruction process by utilizing a two-channel reconstruction model, and combining the definition of the visible light broadband video and the imaging contrast advantage of the narrowband video to generate enhanced and synchronous two-path video output.
The fused video output by the model can improve the signal-to-noise ratio of the narrow-band video, so that imaging details are clearer, and the time synchronization of two paths of videos can be ensured without delay offset.
The specific embodiments of the present invention are intended to be illustrative, rather than limiting, of the invention, and modifications thereof will be suggested to persons skilled in the art to which the present invention pertains without inventive contribution, as desired, after having read the present specification, but are to be protected by the patent law within the scope of the appended claims.

Claims (7)

1. A single-path dual-spectrum real-time endoscope device based on depth network reconstruction, comprising:
The double-spectrum alternating illumination module (1) comprises a broadband white light source (11), a spectrum switching assembly (12) and an illumination optical fiber (13), wherein incident light rays of the broadband white light source form broadband and narrowband double-path illumination after passing through the spectrum switching assembly, and then enter an internal optical path of the endoscope through the illumination optical fiber;
The single-path imaging module (2) is arranged in the endoscope and comprises an imaging lens assembly (21) and an image sensor (22), wherein the imaging lens assembly continuously collects video image data under double-path illumination and transmits the video image data to the image sensor to obtain a wide-narrow-band alternate frame video sequence with a frame rate of 2N frames/sec, N is an integer and is more than or equal to 24 and less than or equal to 30;
the synchronous control module (3) is respectively connected with the double-spectrum alternating illumination module and the single-path endoscope imaging module in a synchronous control manner;
and the image reconstruction module (4) is used for carrying out fusion reconstruction on the wide-narrow band alternate frame video sequence acquired by the image sensor through a double-channel reconstruction model to acquire synchronous two paths of continuous videos with the frame rate of N frames/second, and outputting and displaying the continuous videos in real time.
2. The depth network reconstruction-based single-path dual-spectrum real-time endoscope apparatus according to claim 1, wherein the spectrum switching assembly comprises a filter wheel (121) and a switching motor (122) for driving the filter wheel to rotate, the filter wheel is provided with a narrow-band filter (123) along a half of the evenly divided rotation center, and the rotation speed of the filter wheel is N revolutions per second.
3. The depth network reconstruction-based single-pass dual-spectrum real-time endoscope apparatus according to claim 1, wherein said narrowband filter has a wavelength of 580-650nm.
4. The depth network reconstruction-based single-path dual-spectrum real-time endoscope apparatus according to claim 1, wherein the dual-channel reconstruction model is a depth network model trained by using open-source endoscope multi-spectrum video data, and the training process is as follows:
s11, representing the broadband spectrum video and the narrowband spectrum video of the multispectral video data as The acquisition frame rate is 2N frames/second; the two N frames/second spectral video, which are not synchronized by one frame and are extracted in a staggered way, are then expressed asNarrowband spectral video/>Plus noise, broadband spectral data/>No noise is added as input;
S12, extracting two synchronous N frames/second spectrum videos which are separated by one frame and are not staggered As a true value, its training process is expressed as:
Wherein the method comprises the steps of Noise representing gaussian distribution,/>And obtaining a two-channel reconstruction model after training is completed as the model parameters.
5. The depth network reconstruction-based single-pass dual-spectrum real-time endoscopic device of claim 4, wherein optimization is performed using Adam optimizer during training.
6. The depth network reconstruction-based single-path dual-spectrum real-time endoscope apparatus according to claim 4, wherein said fusion reconstruction process is:
S21, extracting a wide-narrow band alternate frame video sequence output by an image sensor of the single-channel double-spectrum real-time endoscope device to obtain a width video sequence with the frame rate of N frames/second respectively And narrowband video sequences/>Expressed as:
a delay of 1 frame exists between the two paths of video;
s22, the reconstruction process using the two-channel reconstruction model is expressed as:
Output after model reconstruction Namely, two paths of synchronous continuous videos.
7. The depth network reconstruction based single-pass dual-spectrum real-time endoscopic device according to any of claims 1 to 6, wherein N is 30.
CN202410438802.4A 2024-04-12 2024-04-12 Single-path double-spectrum real-time endoscope device based on depth network reconstruction Pending CN118021243A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202410438802.4A CN118021243A (en) 2024-04-12 2024-04-12 Single-path double-spectrum real-time endoscope device based on depth network reconstruction

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202410438802.4A CN118021243A (en) 2024-04-12 2024-04-12 Single-path double-spectrum real-time endoscope device based on depth network reconstruction

Publications (1)

Publication Number Publication Date
CN118021243A true CN118021243A (en) 2024-05-14

Family

ID=90984449

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202410438802.4A Pending CN118021243A (en) 2024-04-12 2024-04-12 Single-path double-spectrum real-time endoscope device based on depth network reconstruction

Country Status (1)

Country Link
CN (1) CN118021243A (en)

Similar Documents

Publication Publication Date Title
US6678398B2 (en) Dual mode real-time screening and rapid full-area, selective-spectral, remote imaging and analysis device and process
US7697975B2 (en) Methods and apparatus for fluorescence imaging using multiple excitation-emission pairs and simultaneous multi-channel image detection
US9654745B2 (en) Rapid multi-spectral imaging methods and apparatus and applications for cancer detection and localization
JP6057921B2 (en) Living body observation device
JP6883627B2 (en) Imaging device for tissues containing blood
WO1999001749A1 (en) Fluorescence imaging system
RU2011128383A (en) EQUIPMENT FOR INFRARED OBSERVATION OF ANATOMIC STRUCTURES AND METHOD OF PROCESSING SIGNALS FROM Mentioned STRUCTURES
KR102567918B1 (en) Optical speculum
WO2006062987A2 (en) Apparatus, system and method for optically analyzing substrate
WO2019100450A1 (en) Multi-functional endoscope system
CN102215732A (en) High resolution digital video colposcope with built-in polarized led illumination and computerized clinical data management system
CN107625513B (en) Enhancing display Narrow-Band Imaging endoscopic system and its imaging method
US9271640B2 (en) Optical speculum
WO2023024701A1 (en) Panoramic endoscope and image processing method thereof
CN111579498B (en) Hyperspectral endoscopic imaging system based on push-broom imaging
CN208693238U (en) Endoscopic optical imaging system that is a kind of while obtaining otherwise visible light color image and blood-stream image
CN118021243A (en) Single-path double-spectrum real-time endoscope device based on depth network reconstruction
CN110115557B (en) Hyperspectral endoscopic imaging device and imaging method
JP4109132B2 (en) Fluorescence determination device
CN213588232U (en) Laryngoscope device
CN114271766A (en) Endoscope device
KR20190035017A (en) Optical device using liquid crystal tunable wavelength filter
CN111568359A (en) Laryngoscope device
WO2020206261A1 (en) Medical instrumentation utilizing narrowband imaging
Zeng 4 White Light Endoscopy

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination