CN115607105A - Endoscopic hyperspectral imaging method and system - Google Patents

Endoscopic hyperspectral imaging method and system Download PDF

Info

Publication number
CN115607105A
CN115607105A CN202111322810.5A CN202111322810A CN115607105A CN 115607105 A CN115607105 A CN 115607105A CN 202111322810 A CN202111322810 A CN 202111322810A CN 115607105 A CN115607105 A CN 115607105A
Authority
CN
China
Prior art keywords
endoscopic
image
hyperspectral
rgb
images
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202111322810.5A
Other languages
Chinese (zh)
Inventor
张金刚
聂云峰
苏润木
王雄智
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Zhongke Photoelectric Beijing Science And Technology Co ltd
Original Assignee
Zhongke Photoelectric Beijing Science And Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Zhongke Photoelectric Beijing Science And Technology Co ltd filed Critical Zhongke Photoelectric Beijing Science And Technology Co ltd
Priority to CN202111322810.5A priority Critical patent/CN115607105A/en
Publication of CN115607105A publication Critical patent/CN115607105A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/0059—Measuring for diagnostic purposes; Identification of persons using light, e.g. diagnosis by transillumination, diascopy, fluorescence
    • A61B5/0075—Measuring for diagnostic purposes; Identification of persons using light, e.g. diagnosis by transillumination, diascopy, fluorescence by spectroscopy, i.e. measuring spectra, e.g. Raman spectroscopy, infrared absorption spectroscopy
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/00163—Optical arrangements
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
    • A61B5/7225—Details of analog processing, e.g. isolation amplifier, gain or sensitivity adjustment, filtering, baseline or drift compensation
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
    • A61B5/7235—Details of waveform analysis
    • A61B5/7264—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems
    • A61B5/7267—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems involving training the classification device

Abstract

The invention provides an endoscopic hyperspectral imaging method and system, wherein the method comprises the following steps: collecting endoscopic RGB images; carrying out geometric correction and image enhancement pretreatment on the endoscopic RGB image to obtain a high-quality RGB image; and performing hyperspectral reconstruction on the endoscopic RGB image based on a deep learning method to obtain an endoscopic hyperspectral image. The invention uses the RGB image which is easier to obtain for the construction of the hyperspectral data, thereby breaking through the limitation of obtaining the hyperspectral image with improved performance by depending on more hardware.

Description

Endoscopic hyperspectral imaging method and system
Technical Field
The invention belongs to the field of hyperspectral images, and particularly relates to an endoscopic hyperspectral imaging method and system.
Background
The existing endoscopic imaging (hard lens and soft lens) only acquires RGB images, even doctors with abundant experience are difficult to perceive superficial lesions, an imaging technology capable of seeing and diagnosing is urgently needed, and the hyperspectral imaging technology can solve the problem.
The hyperspectral image refers to a three-dimensional data cube containing two-dimensional spatial information and one-dimensional spectral information. Each image point of the spatial dimension image corresponds to a complete reflection spectrum curve of the target, and the substance leaves a unique 'fingerprint' in the electromagnetic spectrum. Compared with two-dimensional image information, the hyperspectral image additionally provides rich spectral curve information, so that substances can be identified and detected, and information which is difficult to find in RGB images, such as green pests in green vegetables or colorless pesticides left on fruits, can be found. For the same red blood vessels and the inner walls of human organs, the lesion areas can be effectively highlighted under the hyperspectral image.
However, conventional hyperspectral imaging systems not only require a complete recording of the spatial dimension, but also require the spectroscopic splitting of the target for sampling of the spectral dimension to obtain spectral information. A conventional detector can only record two-dimensional information, so that in order to obtain a complete data cube, scanning in a spatial dimension or a spectral dimension is required, and thus, a precise moving part is introduced, which is not only expensive, but also relatively complex and heavy. Fig. 1 is a schematic diagram of a conventional hyperspectral imaging system, in which a front objective lens, a collimator lens and a focusing lens portion all include more than one lens, and fig. 1 is a schematic diagram.
Most of the existing hyperspectral endoscopic systems are in theoretical research stages and are based on structural designs of endoscopic hard lenses (patent CN 111579498A) and fiberscopes (patent CN 208876461U and patent CN 110859585A). From the research results, no report is found on a flexible electronic endoscope with hyperspectral imaging. The main reasons are that: firstly, most of hyperspectral instruments are heavy and have large volumes; secondly, most mature hyperspectral imaging instruments need to obtain a finished data cube by pushing and sweeping, and due to the introduction of moving parts, the hyperspectral technology is subjected to great elbow dragging in the endoscopic field.
Therefore, the existing endoscopic hyperspectral imaging system cannot go deep into relevant internal organs such as ears, noses, throats, respiratory tracts, digestive tracts and the like of a human body to generate a required hyperspectral image due to inherent defects.
Disclosure of Invention
In order to solve the problems, the invention provides an endoscopic hyperspectral imaging method and system.
An endoscopic hyperspectral imaging method, the method comprising: collecting endoscopic RGB images; carrying out geometric correction and image enhancement pretreatment on the endoscopic RGB image to obtain a high-quality RGB image; and performing hyperspectral reconstruction on the endoscopic RGB image based on a deep learning method to obtain an endoscopic hyperspectral image.
And (3) acquiring an endoscopic RGB image through an endoscopic RGB imaging system.
The hyperspectral reconstruction of the endoscopic RGB image based on the deep learning comprises the step of inputting the endoscopic RGB image into a neural network to reconstruct the endoscopic hyperspectral image.
The neural network is established according to network parameters; the network parameters are trained from existing hyperspectral images and RGB images.
The hyperspectral image reconstruction based on the deep learning has the reconstruction formula
Figure BDA0003345953020000021
F is described θ Representing a neural network, said theta referring to a network parameter.
The existing endoscopic hyperspectral image is obtained according to a priori method.
The prior method sequentially comprises data training and endoscopic hyperspectral image reconstruction; the reconstruction of the endoscopic hyperspectral image is performed by using a data training result.
The data training includes learning a set of basis spectra from an existing endoscopic hyperspectral image database, projecting the basis spectra to corresponding RGB image spectra.
The endoscopic hyperspectral image reconstruction comprises the steps of calculating the expression coefficients of a newly input RGB image on an RGB spectrum; and combining the representation coefficient with the base spectrum to obtain a reconstructed endoscopic hyperspectral image.
And constructing a diversified database by adopting a reversing and cutting method according to the reconstructed endoscopic hyperspectral image.
After the reconstruction of the endoscopic hyperspectral image, the method further comprises the following steps: storing the newly input endoscopic RGB image and the reconstructed endoscopic hyperspectral image so as to update the existing hyperspectral image database; training according to the updated hyperspectral image data and the RGB image to obtain new network parameters; the neural network is initialized according to the new network parameters.
The neural network is generated through dense connection, residual learning and attention mechanism.
The method further comprises the identification and marking of the endoscopic hyperspectral image; the identification and marking of the endoscopic hyperspectral images comprise interpreting the endoscopic hyperspectral images by adopting a mode identification method.
The invention also protects an endoscopic hyperspectral imaging system, which is used for operating the method; the system comprises an endoscopic RGB imaging system, a data processing unit and an image display unit; the endoscopic RGB imaging system is used for collecting endoscopic RGB images; the data processing unit is used for generating an endoscopic hyperspectral image according to the acquired endoscopic RGB image; the image display unit is used for displaying the generated endoscopic hyperspectral image. A (c)
The system also comprises an image identification and marking unit which is used for interpreting the endoscopic hyperspectral image.
Compared with the prior art, the technical scheme of the invention has the following advantages:
the invention provides an endoscopic hyperspectral imaging method and system, which are used for reconstructing an endoscopic hyperspectral image with high precision by using a high-quality RGB image, not only keep the high-definition characteristic of a flexible electronic endoscopic image, but also obtain a three-dimensional hyperspectral data cube. The RGB image which is easier to obtain is used for constructing hyperspectral endoscopic data, so that the limitation that a hyperspectral endoscope with improved performance is obtained by depending on more hardware is broken through. In medicine, the acquisition of hyperspectral images is used for assisting doctors to identify substances of internal organs and tissues of observed human bodies, and can automatically identify lesion areas and highlight superficial lesions, so that the examination efficiency of doctors and the diagnosis accuracy of early lesions can be greatly improved.
Drawings
In order to more clearly illustrate the embodiments or technical solutions of the present invention, the drawings used in the embodiments or technical solutions in the prior art are briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to these drawings without creative efforts.
FIG. 1 is a schematic diagram of a hyperspectral imaging system principle and hyperspectral data cube acquisition intensity.
Fig. 2 is a schematic diagram of a hyperspectral endoscopic imaging system based on a hard mirror.
Fig. 3 is a schematic diagram of a fiberscope-based hyperspectral endoscopic imaging system.
FIG. 4 is a flowchart of an endoscopic hyperspectral imaging method according to an embodiment of the invention.
Fig. 5 is a schematic diagram of endoscopic hyperspectral image reconstruction based on deep learning according to an embodiment of the invention.
FIG. 6 is a flow chart of the diversified database construction according to the embodiment of the present invention.
Fig. 7 is a flowchart of an endoscopic hyperspectral imaging application of an embodiment of the invention.
FIG. 8 is a schematic view of an endoscopic hyperspectral imaging system in accordance with an embodiment of the invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that: the relative arrangement of the components, units, lines and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present invention unless it is specifically stated otherwise.
The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.
Techniques, methods, and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail but are intended to be part of the specification where appropriate.
In all examples shown and discussed herein, any particular value should be construed as merely illustrative, and not limiting. Thus, other examples of the exemplary embodiments may have different values.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be discussed further in subsequent figures.
In order to specifically illustrate the technical scheme protected by the invention, the embodiment of the invention specifically illustrates a medical hyperspectral imaging implementation scheme.
Some hyperspectral endoscopic imaging systems in the prior art are mostly in a theoretical research stage and are based on structural designs of endoscopic hard lenses and fiberscopes.
The whole optical path of the hyperspectral endoscopic imaging system based on the hard mirror is usually complicated, and more than ten lenses are involved as shown in fig. 2. The system comprises an endoscope hard lens, an illumination light source, a first light splitting module, a second light splitting module, a first detector, a second detector, a motion platform, a controller and a computer, and is shown in figure 2. The optical path of the endoscopic hard mirror is only used as a schematic diagram, and the number of lenses contained in the first light splitting module and the second light splitting module is far more than that of the schematic optical path. The system combines an endoscopic imaging technology, a prism grating light splitting technology, a push-broom imaging technology and a bright field imaging technology, and obtains a hyperspectral endoscopic image by fusing a bright field image and a hyperspectral image. The relevant principle can be briefly described as follows: the light emitted by the illumination light source irradiates on the biological tissue sample, the reflected light is divided into two paths by the first light splitting module, and one path of the reflected light enters the first detector to carry out bright field imaging. And the other path of light enters a second light splitting module, enters a second detector after being subjected to spectrum dimension uniform dispersion, and is subjected to hyperspectral imaging. When the hyperspectral data are obtained, the controller needs to control the motion platform to carry out push-broom, and therefore a data cube containing two-dimensional space information and one-dimensional spectral information is obtained.
A hyperspectral endoscopic imaging system based on a fiberscope generally comprises a set of externally connected complete hyperspectral imaging system (including a front objective lens, a slit, a collimating lens group, a focusing lens group and the like), an endoscopic lens (including a light source, a lens group and the like) and an image transmission fiber bundle, as shown in fig. 3. The method comprises the steps of generating continuous spectrums by a wide-spectrum light source to illuminate organs and tissues in a human body, transmitting image information imaged by a lens group to one end of an optical fiber, receiving the image information by a set of complete hyperspectral imaging system at the other end of the optical fiber after passing through an image transmission optical fiber bundle, obtaining parallel light and grating light splitting suitable for gratings after passing through a series of pre-objective lenses for one-time imaging, spatial filtering of slits and a collimating objective lens, and finally obtaining a hyperspectral endoscopic image of a certain spatial dimension through a focusing lens. And the moving platform pushes and sweeps the primary image transmitted by the image optical fiber bundle, and finally the acquisition of the three-dimensional hyperspectral data cube is completed. The working principle can be summarized as follows: the endoscopic light source irradiates the inner tube wall of a human body, an image or a video imaged by the endoscopic lens is transmitted to the front objective lens through the image transmission optical fiber bundle, and then hyperspectral data is finally obtained through the collimating lens group, the grating (or other feasible light splitting elements) and the focusing lens. The hyperspectral imaging system part also needs to carry out push scanning to obtain three-dimensional hyperspectral data. Due to the honeycomb structure of the image-transmitting fiber bundle, the obtained image requires special processing and the quality is usually degraded.
As can be seen from the above description of the prior art, the existing hyperspectral imaging system has the disadvantages of complex structural design, low light energy utilization rate, poor reliability, high cost and large imaging noise; the medical requirement of high-quality hyperspectral imaging on human organs cannot be met.
In order to solve the problems in the prior art, the invention provides a simple and quick hyperspectral endoscopic imaging system solution, so that the acquisition of spectral information is quick and efficient, the performance of the endoscopic spectral imaging system is greatly enhanced, and a new idea is provided for doctors to make relevant diagnosis and analysis. The novel hyperspectral endoscopic imaging system is realized by constructing the mapping relation between the existing RGB three image channels and the hyperspectral channels and finally reconstructing hyperspectral data. The invention fully integrates the technologies of optics, big data image algorithm, medical diagnosis and the like, and is a great advantage of the current multidisciplinary development.
An embodiment of the present invention provides an endoscopic hyperspectral imaging method, as shown in fig. 4, the method includes acquiring an endoscopic RGB image; carrying out geometric correction and image enhancement pretreatment on the endoscopic RGB image to obtain a high-quality RGB image; and performing hyperspectral reconstruction on the endoscopic RGB image based on a deep learning method to obtain an endoscopic hyperspectral image.
The endoscopic RGB image is acquired through an endoscopic RGB imaging system; the endoscopic RGB imaging system comprises an RGB endoscopic imaging system.
The reconstruction of the hyperspectral image depends on an imaging model, namely the RGB endoscopic image (I) is essentially to integrate the product of the hyperspectral image (H) and a camera sensitivity function (S) in a spectral range (w)
Figure BDA0003345953020000071
And (4) obtaining the product. Therefore, the inverse mapping problem is solved from the endoscopic RGB image to the endoscopic hyperspectral image. The inverse mapping problem is solved by adopting a reconstruction algorithm, so that the mode that only a hardware method is added to obtain the hyperspectral endoscopic image in the past is changed.
Specifically, the deep learning is an end-to-end training process, that is, the RGB images are input into the neural network to obtain corresponding hyperspectral images, as shown in fig. 5. Some latest machine learning methods can be adopted to improve the spectral reconstruction accuracy, for example, the simplest network model is upgraded into a more sophisticated neural network, and some special mechanisms, such as residual learning, dense connection, channel attention mechanism, etc., are added to extract representative features.
The neural network is established according to network parameters; the network parameters are trained from existing hyperspectral images and RGB images. Based on the above description canTo obtain the hyperspectral image reconstruction based on the deep learning, the reconstruction formula is
Figure BDA0003345953020000081
F is θ Representing a neural network, said theta referring to a network parameter.
The main problem of the deep learning-based endoscopic hyperspectral image reconstruction is that hyperspectral image data is lacking, and due to the shortage of the hyperspectral image data, the network parameters of a neural network are not accurate enough when being calculated, so that the establishment of the neural network is further influenced. In order to solve the problem, at present, three methods are provided, wherein the first method is to construct a set of high-precision endoscopic model through three-dimensional software and acquire enough hyperspectral data; secondly, a priori method is adopted to recover a small part of hyperspectral data, and methods such as inversion and cutting are adopted to construct a diverse database; thirdly, technologies such as intensive connection, residual learning and attention mechanism are comprehensively applied, a neural network with higher performance (such as generation of a confrontation network) is designed, and deep learning can be carried out only by a small amount of real data.
The prior method sequentially comprises data training and endoscopic hyperspectral image reconstruction, and a diversified database construction process is shown in FIG. 6; the reconstruction of the endoscopic hyperspectral image is performed by using a data training result. The data training includes learning a set of basis spectra from an existing hyperspectral image database, projecting the basis spectra to corresponding RGB image spectra. The endoscopic hyperspectral image reconstruction comprises the steps of calculating the representation coefficients of a newly input RGB image on an RGB spectrum; and combining the representation coefficient with the base spectrum to obtain a reconstructed endoscopic hyperspectral image.
Further, after the reconstructed hyperspectral image is obtained, a diversified database is constructed by adopting a reversing and cutting method according to the reconstructed endoscopic hyperspectral image.
After reconstructing the endoscopic hyperspectral image, the method further comprises the following steps: storing the newly input RGB image and the reconstructed endoscopic hyperspectral image so as to update the existing hyperspectral image database; training according to the updated hyperspectral image data and the RGB image to obtain new network parameters; the neural network is initialized according to the new network parameters. The reconstruction of the endoscopic hyperspectral image is a continuous learning and perfecting process, after a new endoscopic RGB image and the endoscopic hyperspectral image reconstructed by the image are available, new network parameters are trained according to the new image data, an application network is initialized according to the new network parameters, the network parameters are more accurate through continuous learning and perfection, and a neural network established according to the network parameters is more perfected.
After reconstruction is completed, further identifying and marking the peeping hyperspectral image; the identification and marking of the endoscopic hyperspectral images comprises the step of interpreting the hyperspectral images by adopting a pattern recognition method.
In the embodiment of the invention, the RGB reconstruction algorithm is combined with the RGB image acquisition system to obtain the RGB image, and the hyperspectral data is recovered by utilizing the spectrum reconstruction algorithm. It can be seen that the invention greatly simplifies the hardware system, and the reconstruction of the hyperspectrum from the RGB image has high feasibility step by step due to the rapid development of the computer technology and machine learning at present.
The technical scheme protected by the invention is further explained by an embodiment of an application of medical endoscopic hyperspectral imaging.
As shown in fig. 7, the method includes:
checking whether the RGB endoscopic imaging system is within the examination tissue;
if yes, acquiring an endoscopic RGB image: the endoscope collects RGB images of organs and tissues corresponding to a human body, and the RGB images are received and stored by the upper computer through the transmission cable;
carrying out data preprocessing on the acquired endoscopic RGB image: the upper computer performs geometric correction, image enhancement and other operations on the received endoscopic RGB image in real time to obtain a high-quality RGB image rich in details;
hyperspectral reconstruction of RGB images: and the upper computer initializes the network according to the stored parameters of the spectrum reconstruction network, and then outputs the reconstructed endoscopic hyperspectral image by taking the endoscopic RGB image as input. After the hyperspectral image is reconstructed, based on deep learning, storing a newly input endoscopic RGB image and a reconstructed endoscopic hyperspectral image so as to update an existing hyperspectral image database; training according to the updated hyperspectral image data and the RGB image to obtain new network parameters; and initializing the neural network according to the new network parameters to prepare for the next reconstruction of the endoscopic hyperspectral image.
Storing the endoscopic hyperspectral image;
identification and marking of endoscopic hyperspectral images to determine the presence of lesions: interpreting the endoscopic hyperspectral image by adopting pattern recognition, and marking and prompting a doctor to perform further diagnosis at a corresponding position if the endoscopic hyperspectral image has an abnormal area; if there are no abnormal areas, the endoscope is continued to the next position for observation until the examination is finished.
According to the embodiment of the invention, the endoscopic hyperspectral image is reconstructed through the endoscopic RGB image, the use of expensive precise light-splitting optical elements such as adjustable optical filters and gratings and related push-broom moving parts is avoided, and the hyperspectral three-dimensional data is obtained by utilizing the existing RGB endoscope system to observe the narrow space occasions such as the throat, nasal cavity, abdominal cavity, intestinal tract, pancreas, bronchus and ureter in the human body, so that the suspicious lesion area is automatically searched, the diagnosis efficiency of a doctor is greatly improved, and a powerful guarantee is provided for the big data of the health medicine.
An endoscopic hyperspectral imaging system is further provided in an embodiment of the present invention, as shown in fig. 8, the system is configured to operate the hyperspectral imaging method, and the system includes an endoscopic RGB imaging system, a data processing unit, and an image display unit; the endoscopic RGB imaging system is used for collecting endoscopic RGB images; the data processing unit is used for generating an endoscopic hyperspectral image according to the acquired endoscopic RGB image; the image display unit is used for displaying the generated hyperspectral image.
The RGB imaging system comprises an RGB endoscopic imaging system; the data processing unit is a device with data processing capability, such as a computer, an industrial personal computer, a server and the like; the image display unit comprises equipment with an image display function, such as a display and the like, and can display the identification and marking results of the hyperspectral images in addition to displaying the hyperspectral images; further, in medical diagnosis, in order to have a comparison effect, the image display unit also has a function of simultaneously displaying the RGB image and the hyperspectral image.
The endoscopic hyperspectral image imaging system further comprises an image recognition and marking unit which is used for interpreting the endoscopic hyperspectral image, and marking and prompting a doctor at a corresponding position if the endoscopic hyperspectral image has an abnormal area in medical diagnosis.
Therefore, the embodiment of the invention avoids the defect of high-complexity endoscopic hyperspectral imaging, and the reconstruction of the RGB image hyperspectrum gradually has high feasibility due to the rapid development of the computer technology and machine learning at present.
Although some specific embodiments of the present invention have been described in detail by way of illustration, it should be understood by those skilled in the art that the above illustration is only for the purpose of illustration and is not intended to limit the scope of the invention. It will be understood by those skilled in the art that various changes may be made in the above embodiments and equivalents may be substituted for elements thereof without departing from the scope and spirit of the invention.

Claims (15)

1. An endoscopic hyperspectral imaging method, the method comprising:
collecting endoscopic RGB images;
carrying out geometric correction and image enhancement pretreatment on the endoscopic RGB image to obtain a high-quality RGB image;
and performing hyperspectral reconstruction on the endoscopic RGB image based on a deep learning method to obtain an endoscopic hyperspectral image.
2. The method according to claim 1, wherein the endoscopic RGB images are acquired by an endoscopic RGB imaging system.
3. The method according to claim 1, wherein the hyperspectral reconstruction of endoscopic RGB images based on deep learning comprises inputting endoscopic RGB images into a neural network to reconstruct endoscopic hyperspectral images.
4. The method of claim 3, wherein the neural network is established based on network parameters;
the network parameters are trained from existing hyperspectral images and RGB images.
5. The method according to claim 4, wherein the hyperspectral image reconstruction based on deep learning is based on a reconstruction formula
Figure FDA0003345953010000011
F is θ Representing a neural network, said theta referring to a network parameter.
6. A method according to claim 4, wherein the existing endoscopic hyperspectral image is obtained from a priori methods.
7. The method according to claim 6, wherein the a priori method comprises data training and endoscopic hyperspectral image reconstruction in sequence;
the reconstruction of the endoscopic hyperspectral image is performed by using a data training result.
8. The method according to claim 7, wherein the data training comprises learning a set of basis spectra from an existing endoscopic hyperspectral image database, projecting the basis spectra to corresponding RGB image spectra.
9. The method according to claim 8, wherein the endoscopic hyperspectral image reconstruction comprises calculating a representation coefficient of a newly input RGB image over an RGB spectrum;
and combining the representation coefficient with the base spectrum to obtain a reconstructed endoscopic hyperspectral image.
10. The method according to claim 3 or 9, characterized in that a diversified database is constructed by inversion and cropping according to the reconstructed endoscopic hyperspectral image.
11. The method according to claim 4 or 9, further comprising, after reconstructing the endoscopic hyperspectral image:
storing the newly input endoscopic RGB image and the reconstructed endoscopic hyperspectral image so as to update the existing hyperspectral image database;
training according to the updated hyperspectral image data and the RGB image to obtain new network parameters;
the neural network is initialized according to the new network parameters.
12. The method of claim 3, wherein the neural network is generated by dense connectivity, residual learning, attention mechanism.
13. The method according to any one of claims 1 to 12, further comprising identification and marking of the endoscopic hyperspectral image;
the identification and marking of the endoscopic hyperspectral images comprise interpreting the endoscopic hyperspectral images by adopting a mode identification method.
14. An endoscopic hyperspectral imaging system, wherein the system is adapted to operate the method of any of claims 1 to 13; the system comprises an endoscopic RGB imaging system, a data processing unit and an image display unit;
the endoscopic RGB imaging system is used for collecting endoscopic RGB images;
the data processing unit is used for generating an endoscopic hyperspectral image according to the acquired endoscopic RGB image;
the image display unit is used for displaying the generated endoscopic hyperspectral image.
15. The system according to claim 14, further comprising an image recognition and marking unit for interpreting the endoscopic hyperspectral image.
CN202111322810.5A 2021-11-09 2021-11-09 Endoscopic hyperspectral imaging method and system Pending CN115607105A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111322810.5A CN115607105A (en) 2021-11-09 2021-11-09 Endoscopic hyperspectral imaging method and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111322810.5A CN115607105A (en) 2021-11-09 2021-11-09 Endoscopic hyperspectral imaging method and system

Publications (1)

Publication Number Publication Date
CN115607105A true CN115607105A (en) 2023-01-17

Family

ID=84856732

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111322810.5A Pending CN115607105A (en) 2021-11-09 2021-11-09 Endoscopic hyperspectral imaging method and system

Country Status (1)

Country Link
CN (1) CN115607105A (en)

Similar Documents

Publication Publication Date Title
Clancy et al. Surgical spectral imaging
Tian et al. Deep learning in biomedical optics
US11141044B2 (en) Method and apparatus for estimating the value of a physical parameter in a biological tissue
WO2015035229A2 (en) Apparatuses and methods for mobile imaging and analysis
US11213189B2 (en) Endoscopic device and method for endoscopic examination
WO2009009414A2 (en) Simultaneous acquisition of differing image types
Peterson et al. Feasibility of a video‐mosaicking approach to extend the field‐of‐view for reflectance confocal microscopy in the oral cavity in vivo
US11445915B2 (en) Compact briefcase OCT system for point-of-care imaging
CN107427202B (en) Device, system and method for illuminating a structure of interest inside a human or animal body
US10226180B2 (en) System, method, and apparatus for performing histopathology
CN111667447A (en) Intravascular image fusion method and system and image acquisition device
KR102267509B1 (en) The method for measuring microcirculation in cochlea and the apparatus thereof
CN109752377B (en) Spectroscopic bimodal projection tomography tissue blood vessel imaging device and method
Gu et al. Image enhancement based on in vivo hyperspectral gastroscopic images: a case study
Marsden et al. FLImBrush: dynamic visualization of intraoperative free-hand fiber-based fluorescence lifetime imaging
CN107957401A (en) A kind of EO-1 hyperion microscopic imaging apparatus available for insertion type diagnosing tumor
CN114300095A (en) Image processing apparatus, image processing method, image processing device, image processing apparatus, and storage medium
KR102047247B1 (en) Multi-modal fusion endoscope system
CN115607105A (en) Endoscopic hyperspectral imaging method and system
KR102393661B1 (en) Endoscope system for multi image, image providing method of the system, and a recording medium having computer readable program for executing the method
JP6710853B2 (en) Probe-type confocal laser microscope endoscopic image diagnosis support device
WO2019088008A1 (en) Image processing apparatus, image processing method, program, and endoscope system
CN115607104A (en) Endoscopic hyperspectral imaging method and system
Wu et al. Learned end-to-end high-resolution lensless fiber imaging toward intraoperative real-time cancer diagnosis
JP2012010776A (en) Tomographic image processing apparatus and method, and optical interference tomographic image diagnostic apparatus

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination