CN111579498A - Hyperspectral endoscopic imaging system based on push-broom imaging - Google Patents
Hyperspectral endoscopic imaging system based on push-broom imaging Download PDFInfo
- Publication number
- CN111579498A CN111579498A CN202010340445.XA CN202010340445A CN111579498A CN 111579498 A CN111579498 A CN 111579498A CN 202010340445 A CN202010340445 A CN 202010340445A CN 111579498 A CN111579498 A CN 111579498A
- Authority
- CN
- China
- Prior art keywords
- image
- hyperspectral
- imaging
- bright field
- light
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003384 imaging method Methods 0.000 title claims abstract description 82
- 238000000701 chemical imaging Methods 0.000 claims abstract description 31
- 230000003595 spectral effect Effects 0.000 claims abstract description 26
- 230000033001 locomotion Effects 0.000 claims abstract description 16
- 230000004927 fusion Effects 0.000 claims abstract description 6
- 238000001228 spectrum Methods 0.000 claims description 31
- 230000015556 catabolic process Effects 0.000 claims description 12
- 238000006731 degradation reaction Methods 0.000 claims description 12
- 239000011159 matrix material Substances 0.000 claims description 11
- 238000005286 illumination Methods 0.000 claims description 8
- 239000013307 optical fiber Substances 0.000 claims description 8
- 238000012937 correction Methods 0.000 claims description 7
- 239000006185 dispersion Substances 0.000 claims description 5
- 230000005540 biological transmission Effects 0.000 claims description 4
- 238000013507 mapping Methods 0.000 claims description 3
- 238000007500 overflow downdraw method Methods 0.000 claims description 3
- 239000000835 fiber Substances 0.000 claims description 2
- 238000012546 transfer Methods 0.000 claims description 2
- 238000005516 engineering process Methods 0.000 abstract description 20
- 230000000877 morphologic effect Effects 0.000 abstract description 4
- 210000001519 tissue Anatomy 0.000 description 21
- 206010028980 Neoplasm Diseases 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 238000000034 method Methods 0.000 description 5
- 238000001514 detection method Methods 0.000 description 4
- 238000011161 development Methods 0.000 description 2
- 238000010521 absorption reaction Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 210000004204 blood vessel Anatomy 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 210000000981 epithelium Anatomy 0.000 description 1
- 210000001035 gastrointestinal tract Anatomy 0.000 description 1
- 238000001727 in vivo Methods 0.000 description 1
- 230000003902 lesion Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000035515 penetration Effects 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
- G01N21/17—Systems in which incident light is modified in accordance with the properties of the material investigated
- G01N21/25—Colour; Spectral properties, i.e. comparison of effect of material on the light at two or more different wavelengths or wavelength bands
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/06—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor with illuminating arrangements
- A61B1/07—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor with illuminating arrangements using light-conductive means, e.g. optical fibres
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
- G01J3/00—Spectrometry; Spectrophotometry; Monochromators; Measuring colours
- G01J3/28—Investigating the spectrum
- G01J3/2823—Imaging spectrometer
Landscapes
- Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Spectroscopy & Molecular Physics (AREA)
- Pathology (AREA)
- Surgery (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Veterinary Medicine (AREA)
- Optics & Photonics (AREA)
- Heart & Thoracic Surgery (AREA)
- Medical Informatics (AREA)
- Molecular Biology (AREA)
- Animal Behavior & Ethology (AREA)
- Radiology & Medical Imaging (AREA)
- Public Health (AREA)
- Engineering & Computer Science (AREA)
- Biophysics (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Chemical & Material Sciences (AREA)
- Analytical Chemistry (AREA)
- Biochemistry (AREA)
- Immunology (AREA)
- Endoscopes (AREA)
Abstract
The invention discloses a hyperspectral endoscopic imaging system based on push-broom imaging, which comprises an endoscope, a bright field light source, a first light splitting module, a second light splitting module, a first detector, a second detector, a motion platform, a controller and a computer, wherein the controller is connected with the controller; the light emitted by the bright field light source irradiates a biological tissue sample, the reflected light is divided into two paths by the first light splitting module, and one path of the reflected light enters the first detector for bright field imaging; the other path of light enters a second detector after passing through a second light splitting module to perform hyperspectral imaging; the controller controls the bright field imaging and the hyperspectral imaging to be carried out synchronously, and the bright field image and the hyperspectral image are fused to obtain endoscopic image data. The invention combines the endoscopic imaging technology, the prism grating light splitting technology, the push-broom imaging technology and the bright field imaging technology, can simultaneously and accurately acquire the spectral information and the morphological information of the living organism tissue, and obtains the hyperspectral and high-resolution image of the living organism tissue through the image fusion algorithm.
Description
Technical Field
The invention relates to the technical field of endoscopic imaging, in particular to a hyperspectral endoscopic imaging system based on push-broom imaging.
Background
With the development of science and technology, endoscopic imaging systems have been widely used in the medical field, especially in the field of tumor diagnosis and treatment. Most of the current endoscopic imaging systems are white light source illumination imaging, and tiny tumor focuses with similar colors and forms to normal tissues are easy to miss. How to distinguish lesion tissues from normal tissues and improve the tumor recognition rate is an important direction for endoscopic development. For example, as a new endoscopic narrowband Imaging (NBI), three kinds of Narrow-Band light sources of red (605nm), green (540nm) and blue (415nm) are adopted for illumination, and the contrast of the mucosal epithelium form and the form of the epithelial blood vessel network is enhanced through different penetration depths of different wavelengths, so that the detection rate of digestive tract tumors is improved. However, the method only has three bands of data, and cannot provide more function information.
The hyperspectral imaging technique is an imaging technique which is non-contact and non-ionizing and can simultaneously acquire morphological information and spectral information of biological tissues. Because different substances of biological tissues have different spectral absorption characteristics, the method provides a basis for identifying normal tissues and cancerous tissues. The hyperspectral imaging technology and the endoscopic imaging technology are combined to provide possibility for the in-vivo noninvasive detection of tumors.
At present, no commercial hyperspectral endoscopic imaging product exists, related technologies are introduced in part of documents and patents, and the currently adopted spectral information acquisition mode adopts an optical filter light splitting technology, a liquid crystal tunable light splitting technology or an acousto-optic tunable light splitting technology. The method cannot acquire all spectral information of the sample at the same time, and the detection object is a living body and cannot be kept unchanged for a long time, so that the position and the shape of the detection object are changed to inevitably cause inaccuracy of the spectral information.
Disclosure of Invention
In order to overcome the problems, the invention provides a hyperspectral endoscopic imaging system based on push-broom imaging.
In order to solve the technical problems, the invention adopts the technical scheme that: a hyperspectral endoscopic imaging system based on push-broom imaging comprises: the system comprises an endoscope, a bright field light source, a first light splitting module, a second light splitting module, a first detector, a second detector, a motion platform, a controller and a computer;
the first light splitting module is used for splitting light reflected by a biological tissue sample into two paths, and the two paths of light respectively enter the first detector and the second light splitting module, the second light splitting module is used for dispersing and splitting light, the first detector is used for bright field imaging, the second detector is used for hyperspectral imaging, the motion platform is used for realizing a push-broom function, the controller is connected with the first detector, the second detector and the motion platform, and the computer is connected with the controller;
the light emitted by the bright field light source irradiates a biological tissue sample, the reflected light is divided into two paths by the first light splitting module, wherein one path enters the first detector for bright field imaging; the other path of light enters the second light splitting module, enters the second detector after being subjected to spectrum dimension uniform dispersion, and is subjected to hyperspectral imaging; the controller controls the bright field imaging and the hyperspectral imaging to be carried out synchronously, and the bright field image and the hyperspectral image are fused to obtain endoscopic image data with hyperspectral resolution and high spatial resolution.
The image fusion method of the bright field image and the hyperspectral image comprises the following steps:
1) geometric correction of the hyperspectral image:
synchronously imaging the bright field system and the hyperspectral imaging system, and determining the spatial relationship between the bright field system and the hyperspectral imaging system;
let t1Time of day, bright field imaging acquisition image I1Hyperspectral imaging acquisition image H1The spatial relationship between the two is determined by a matrix R11Is represented by I1＝R11H1；tnAcquisition of image I by time brightfield imagingnHyperspectral imaging acquisition image HnThe spatial relationship between the two is determined by a matrix RnnIs represented byn＝RnnHn；
Suppose a sample is at t1To tnMoving within time, and obtaining an image I in an image characteristic point matching modenAnd I1Spatial relationship R betweenn1，tnThe hyperspectral image at any moment can pass through Rn1RnnMapping to I1Space, thereby realizing geometric correction of the hyperspectral image;
2) super-resolution reconstruction of hyperspectral images:
constrained by a light splitting system, the spatial resolution of the hyperspectral image is smaller than that of the bright field image, and in order to acquire the high-resolution hyperspectral image, the image fusion is carried out through the bright field image and the hyperspectral image, and the hyperspectral image super-resolution reconstruction is carried out;
the hyperspectral image H is assumed to be a three-color image with M X N of spatial pixels, S of spectral dimension, M X N of spatial pixels of a bright field image I, RGB of the image, a hyperspectral super-resolution image X of the image, M X N of spatial pixels and S of spectral dimension, wherein M is km, N is kn, and k is a positive integer;
images H and I can be viewed as degradations in the spatial and spectral dimensions, respectively, of image X, as shown by the following equation:
H＝XG+NH(1)；
I＝PX+NI(2)；
where G is the spatial degradation matrix and P is lightSpectral degradation matrices, all known, NH，NIRespectively, gaussian noise introduced by observation; according to the formulas (1) and (2), the hyperspectral super-resolution image X cannot be accurately solved, and according to the spectrum linear mixed model, each pixel X of the image X pixel can be considered to be a linear combination of a limited number of spectrum end members, as shown in the following formula.
Therefore, the image X can be expressed as
X＝ΨC (4)；
The hyperspectral image H and the bright field image I are combined with the priori knowledge of the target object spectrum end member, the optimal spectrum end member combination psi can be obtained through a spectrum degradation equation, then spectrum unmixing is carried out according to the spectrum information of the hyperspectral image H and the spectrum end member information to obtain the abundance C of the end member, and finally the hyperspectral super-resolution image X is obtained through the spectrum end member combination psi, the abundance C and the reconstruction of the bright field image I.
Preferably, the bright field light source is used for providing illumination, and the light source spectral range is 400-1000 nm.
Preferably, the bright field light source is connected to the endoscope by an optical fiber.
Preferably, the endoscope is used for biological tissue image transmission, and comprises a hard tube endoscope and a fiber optic endoscope.
Preferably, the endoscope is connected to the first light splitting module through an adapter ring.
Preferably, the first light splitting module includes a first collimating mirror, a first converging mirror, a semi-transparent semi-reflecting mirror and a second converging mirror, light reflected by the biological tissue sample reaches the semi-transparent semi-reflecting mirror after passing through the first collimating mirror, and a part of the light is reflected by the semi-transparent semi-reflecting mirror and enters the first detector after passing through the first converging mirror; and the other part of light transmits the semi-transmitting and semi-reflecting mirror and enters the second light splitting module after passing through the second converging mirror.
Preferably, the second light splitting module includes a slit, a second collimating mirror, a prism grating group, and a third converging mirror, which are sequentially disposed along an incident light path.
Preferably, the first detector is a color CCD or CMOS camera.
Preferably, the second detector is a hyperspectral imaging detector.
The invention has the beneficial effects that: the invention combines the endoscopic imaging technology, the prism grating light splitting technology, the push-broom imaging technology and the bright field imaging technology, can simultaneously and accurately acquire the spectral information and the morphological information of the living organism tissue, obtains the hyperspectral and high-resolution images of the living organism tissue through the image fusion algorithm, and can overcome the defect of image distortion caused by the movement of the living organism in the prior hyperspectral endoscopic imaging technology.
Drawings
FIG. 1 is a schematic structural diagram of a hyperspectral endoscopic imaging system based on push-broom imaging according to the present invention;
fig. 2 is a schematic structural diagram of a first light splitting module according to the present invention;
FIG. 3 is a schematic structural diagram of a second light splitting module according to the present invention;
FIG. 4 is a schematic view of the working flow of the hyperspectral endoscopic imaging system based on push-broom imaging of the present invention;
FIG. 5 is a schematic diagram of geometric correction of a hyperspectral image according to the present invention;
FIG. 6 is a schematic diagram of reconstruction of a hyperspectral super-resolution image in the invention.
Description of reference numerals:
1-bright field light source; 2-an optical fiber; 3, an endoscope; 4, a transfer ring; 5-a first light splitting module; 6 — a first detector; 7-a second light splitting module; 8, a motion platform; 9 — a second detector; 10-a controller; 11-a computer; 501, a first collimating mirror; 502 — a first converging mirror; 503-half mirror; 504-second converging mirror; 701-a slit; 702 — a second collimating mirror; 703-prism grating group; 704 — third converging mirror.
Detailed Description
The present invention is further described in detail below with reference to examples so that those skilled in the art can practice the invention with reference to the description.
It will be understood that terms such as "having," "including," and "comprising," as used herein, do not preclude the presence or addition of one or more other elements or groups thereof.
The invention combines the endoscopic imaging technology, the prism grating light splitting technology, the push-broom imaging technology and the bright field imaging technology, simultaneously accurately obtains the spectral information and the morphological information of the living organism tissue, obtains the hyperspectral and high-resolution image of the living organism tissue through the image fusion algorithm, and overcomes the image distortion caused by the movement of the living organism.
As shown in fig. 1, a hyperspectral endoscopic imaging system based on push-broom imaging of the present embodiment includes: the system comprises an endoscope 3, a bright field light source 1, a first light splitting module 5, a second light splitting module 7, a first detector 6, a second detector 9, a motion platform 8, a controller 10 and a computer 11;
the first light splitting module 5 is used for splitting light reflected by the biological tissue sample into two paths, and the two paths of light respectively enter the first detector 6 and the second light splitting module 7; the second light splitting module 7 is used for dispersion splitting; the first detector 6 is used for bright field imaging, and the second detector 9 is used for hyperspectral imaging; the motion platform 8 controls the second light splitting module 7 to move and is used for realizing a push-broom function; the controller 10 is connected with the first detector 6, the second detector 9 and the motion platform 8, and the computer 11 is connected with the controller 10;
the light emitted by the bright field light source 1 irradiates a biological tissue sample, the reflected light is divided into two paths by the first light splitting module 5, and one path of the reflected light enters the first detector 6 for bright field imaging; the other path of light enters a second light splitting module 7, enters a second detector 9 after being subjected to spectrum dimension uniform dispersion, and is subjected to hyperspectral imaging; the controller 10 controls the bright field imaging and the hyperspectral imaging to be carried out synchronously, and after the bright field image and the hyperspectral image are fused, endoscopic image data with hyperspectral resolution and high spatial resolution are obtained.
The image fusion method of the bright field image and the hyperspectral image comprises the following steps:
1) geometric correction of the hyperspectral image:
synchronously imaging the bright field system and the hyperspectral imaging system, and determining the spatial relationship between the bright field system and the hyperspectral imaging system;
let t1Time of day, bright field imaging acquisition image I1Hyperspectral imaging acquisition image H1The spatial relationship between the two is determined by a matrix R11Is represented by I1＝R11H1；tnAcquisition of image I by time brightfield imagingnHyperspectral imaging acquisition image HnThe spatial relationship between the two is determined by a matrix RnnIs represented byn＝RnnHn；
Suppose a sample is at t1To tnMoving within time, and obtaining an image I in an image characteristic point matching modenAnd I1Spatial relationship R betweenn1，tnThe hyperspectral image at any moment can pass through Rn1RnnMapping to I1Space, thereby realizing geometric correction of the hyperspectral image; referring to fig. 5;
2) super-resolution reconstruction of hyperspectral images:
due to the restriction of a light splitting system, the spatial resolution of a hyperspectral image is generally smaller than that of a bright field image, and in order to acquire a high-resolution hyperspectral image, image fusion is carried out on the bright field image and the hyperspectral image, and hyperspectral image super-resolution reconstruction is carried out;
the hyperspectral image H is assumed to be a three-color image with M X N of spatial pixels, S of spectral dimension, M X N of spatial pixels of a bright field image I, RGB of the image, a hyperspectral super-resolution image X of the image, M X N of spatial pixels and S of spectral dimension, wherein M is km, N is kn, and k is a positive integer;
images H and I can be viewed as degradations in the spatial and spectral dimensions, respectively, of image X, as shown by the following equation:
H＝XG+NH(1)；
I＝PX+NI(2)；
where G is the spatial degradation matrix, P is the spectral degradation matrix, both known, NH，NIRespectively, gaussian noise introduced by observation; the high spectrum cannot be accurately solved according to the formulas (1) and (2)Super-resolution image X, and according to the spectral linear mixture model, each pixel X of image X pixels can be considered as a linear combination of a finite number of spectral end members, as shown in the following equation:
therefore, the image X can be expressed as
X＝ΨC (4)；
The hyperspectral image H and the bright field image I are combined with the priori knowledge of the target object spectrum end member, the optimal spectrum end member combination psi can be obtained through a spectrum degradation equation, then spectrum unmixing is carried out according to the spectrum information of the hyperspectral image H and the spectrum end member information to obtain the abundance C of the end member, and finally the hyperspectral super-resolution image X is obtained through the spectrum end member combination psi, the abundance C and the reconstruction of the bright field image I. Refer to fig. 6.
The bright field light source 1 is used for providing illumination, and in one embodiment, the light source spectrum range is 400-1000 nm.
In one embodiment, the bright field light source 1 is connected to the endoscope 3 by an optical fiber 2, the optical fiber 2 being used for illumination light transmission. The endoscope 3 is used for biological tissue image transmission, and comprises a hard tube type endoscope 3 and an optical fiber 2 endoscope 3.
In one embodiment, the endoscope 3 is connected with the first light splitting module 5 through the adapter ring 4, and can realize focusing adjustment.
Referring to fig. 2, in an embodiment, the first light splitting module 5 includes a first collimator 501, a first focusing mirror 502, a half mirror 503 and a second focusing mirror 504, and light reflected by the biological tissue sample reaches the half mirror 503 after passing through the first collimator 501, wherein a part of the light is reflected by the half mirror 503, passes through the first focusing mirror 502 and enters the first detector 6; the other part of the light is transmitted through the half mirror 503, passes through the second converging mirror 504, and enters the second light splitting module 7.
Referring to fig. 3, in one embodiment, the second light splitting module 7 includes a slit 701, a second collimating mirror 702, a prism grating group 703 and a third converging mirror 704, which are sequentially disposed along an incident light path.
In one embodiment, the first detector 6 is a color CCD or CMOS camera for bright field imaging. The second detector 9 is a hyperspectral imaging detector, which is a high-sensitivity detector for hyperspectral imaging. The controller 10 is used for synchronous control of hyperspectral imaging and bright field imaging and motion control of the motion platform 8. The computer 11 is used to acquire and process bright field and hyperspectral image data.
In one embodiment, the system operates on the principle of: the illumination emitted by the bright field light source 1 is irradiated on a biological tissue sample through the optical fiber 2 and the endoscope 3 illumination optical fiber 2, the reflected light enters the first light splitting module 5 through the endoscope 3 imaging light path and the adapter ring 4, the light reflected by the biological tissue sample in the first light splitting module 5 reaches the semi-transmitting semi-reflecting mirror 503 after passing through the first collimating mirror 501, and a part of the light is reflected by the semi-transmitting semi-reflecting mirror 503 and enters the first detector 6 after passing through the first converging mirror 502; the other part of the light is transmitted through the half-transmitting and half-reflecting mirror 503, passes through the second converging mirror 504 and enters the second light splitting module 7; in the second light splitting module 7, light sequentially passes through a slit 701, a second collimating mirror 702, a prism grating group 703 (for realizing spectral dimension uniform dispersion) and a third converging mirror 704 and then reaches the second detector 9, and spectral dimension information and one-dimensional spatial information are imaged on the second detector 9 for high-spectrum imaging; and the moving platform 8 controls the second light splitting module 7 to move to the next position until push-broom imaging is completed. The controller 10 implements the push-broom of the motion platform 8 to obtain another dimension of spatial information, and obtains a data cube containing the spatial information and the spectral information. The controller 10 controls the bright field imaging and the hyperspectral imaging to be carried out synchronously, and after the bright field image and the hyperspectral image are fused, endoscopic image data with hyperspectral resolution and high spatial resolution are obtained. The workflow is shown in fig. 4.
While embodiments of the invention have been disclosed above, it is not limited to the applications listed in the description and the embodiments, which are fully applicable in all kinds of fields of application of the invention, and further modifications may readily be effected by those skilled in the art, so that the invention is not limited to the specific details without departing from the general concept defined by the claims and the scope of equivalents.
Claims (10)
1. A hyperspectral endoscopic imaging system based on push-broom imaging is characterized by comprising: the system comprises an endoscope, a bright field light source, a first light splitting module, a second light splitting module, a first detector, a second detector, a motion platform, a controller and a computer;
the first light splitting module is used for splitting light reflected by a biological tissue sample into two paths, and the two paths of light respectively enter the first detector and the second light splitting module, the second light splitting module is used for dispersing and splitting light, the first detector is used for bright field imaging, the second detector is used for hyperspectral imaging, the motion platform is used for realizing a push-broom function, the controller is connected with the first detector, the second detector and the motion platform, and the computer is connected with the controller;
the light emitted by the bright field light source irradiates a biological tissue sample, the reflected light is divided into two paths by the first light splitting module, wherein one path enters the first detector for bright field imaging; the other path of light enters the second light splitting module, enters the second detector after being subjected to spectrum dimension uniform dispersion, and is subjected to hyperspectral imaging; the controller controls the bright field imaging and the hyperspectral imaging to be carried out synchronously, and the bright field image and the hyperspectral image are fused to obtain endoscopic image data with hyperspectral resolution and high spatial resolution.
2. The push-broom imaging-based hyperspectral endoscopic imaging system according to claim 1, wherein the image fusion method of the bright field image and the hyperspectral image comprises the following steps:
1) geometric correction of the hyperspectral image:
synchronously imaging the bright field system and the hyperspectral imaging system, and determining the spatial relationship between the bright field system and the hyperspectral imaging system;
let t1Time of day, bright field imaging acquisition image I1Hyperspectral imaging acquisition image H1The spatial relationship between the two is determined by a matrix R11Is represented by I1＝R11H1；tnTime of day brightfield imaging acquiring imagesInHyperspectral imaging acquisition image HnThe spatial relationship between the two is determined by a matrix RnnIs represented byn＝RnnHn；
Suppose a sample is at t1To tnMoving within time, and obtaining an image I in an image characteristic point matching modenAnd I1Spatial relationship R betweenn1，tnThe hyperspectral image at any moment can pass through Rn1RnnMapping to I1Space, thereby realizing geometric correction of the hyperspectral image;
2) super-resolution reconstruction of hyperspectral images:
constrained by a light splitting system, the spatial resolution of the hyperspectral image is smaller than that of the bright field image, and in order to acquire the high-resolution hyperspectral image, the image fusion is carried out through the bright field image and the hyperspectral image, and the hyperspectral image super-resolution reconstruction is carried out;
the hyperspectral image H is assumed to be a three-color image with M X N of spatial pixels, S of spectral dimension, M X N of spatial pixels of a bright field image I, RGB of the image, a hyperspectral super-resolution image X of the image, M X N of spatial pixels and S of spectral dimension, wherein M is km, N is kn, and k is a positive integer;
images H and I can be viewed as degradations in the spatial and spectral dimensions, respectively, of image X, as shown by the following equation:
H＝XG+NH(1)；
I＝PX+NI(2)；
where G is the spatial degradation matrix, P is the spectral degradation matrix, both known, NH，NIRespectively, gaussian noise introduced by observation; according to the formulas (1) and (2), the hyperspectral super-resolution image X cannot be accurately solved, and according to the spectrum linear mixed model, each pixel X of the image X pixel can be considered to be a linear combination of a limited number of spectrum end members, as shown in the following formula.
Therefore, the image X can be expressed as
X＝ΨC (4)；
The hyperspectral image H and the bright field image I are combined with the priori knowledge of the target object spectrum end member, the optimal spectrum end member combination psi can be obtained through a spectrum degradation equation, then spectrum unmixing is carried out according to the spectrum information of the hyperspectral image H and the spectrum end member information to obtain the abundance C of the end member, and finally the hyperspectral super-resolution image X is obtained through the spectrum end member combination psi, the abundance C and the reconstruction of the bright field image I.
3. The hyperspectral endoscopic imaging system based on push-broom imaging according to claim 2, wherein the bright field light source is used for providing illumination, and the light source spectral range is 400-1000 nm.
4. The push-broom imaging-based hyperspectral endoscopic imaging system according to claim 3, wherein the bright field light source is connected to the endoscope by an optical fiber.
5. The push-broom imaging-based hyperspectral endoscopic imaging system according to claim 4, wherein the endoscope is used for biological tissue image transmission and comprises a hard-tube endoscope and a fiber optic endoscope.
6. The push-broom imaging-based hyperspectral endoscopic imaging system according to claim 5, wherein the endoscope is connected to the first light splitting module by a transfer ring.
7. The hyperspectral endoscopic imaging system based on push-broom imaging according to claim 6, wherein the first light splitting module comprises a first collimating mirror, a first converging mirror, a semi-transparent semi-reflecting mirror and a second converging mirror, light reflected by the biological tissue sample reaches the semi-transparent semi-reflecting mirror after passing through the first collimating mirror, and a part of the light is reflected by the semi-transparent semi-reflecting mirror and enters the first detector after passing through the first converging mirror; and the other part of light transmits the semi-transmitting and semi-reflecting mirror and enters the second light splitting module after passing through the second converging mirror.
8. The push-broom imaging-based hyperspectral endoscopic imaging system according to claim 1, wherein the second light splitting module comprises a slit, a second collimating mirror, a prism grating group and a third converging mirror arranged in sequence along an incident light path.
9. The push-broom imaging-based hyperspectral endoscopic imaging system according to claim 1, wherein the first detector is a color CCD or CMOS camera.
10. The push-broom imaging-based hyperspectral endoscopic imaging system according to claim 1, wherein the second detector is a hyperspectral imaging detector.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010340445.XA CN111579498B (en) | 2020-04-26 | 2020-04-26 | Hyperspectral endoscopic imaging system based on push-broom imaging |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010340445.XA CN111579498B (en) | 2020-04-26 | 2020-04-26 | Hyperspectral endoscopic imaging system based on push-broom imaging |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111579498A true CN111579498A (en) | 2020-08-25 |
| CN111579498B CN111579498B (en) | 2021-08-06 |
Family
ID=72112532
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010340445.XA Active CN111579498B (en) | 2020-04-26 | 2020-04-26 | Hyperspectral endoscopic imaging system based on push-broom imaging |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111579498B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113440093A (en) * | 2021-07-19 | 2021-09-28 | 山东第一医科大学附属省立医院(山东省立医院) | Digestive tract secretion microscopic detection system |
| WO2022195222A1 (en) * | 2021-03-17 | 2022-09-22 | Institut Hospitalo-Universitaire De Strasbourg | Medical imaging method employing a hyperspectral camera |
| CN115598075A (en) * | 2022-12-14 | 2023-01-13 | 自然资源部第二海洋研究所（Cn） | Deep sea hyperspectral imaging detection system and method based on two-channel coaxial light path |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2013017507A (en) * | 2011-07-07 | 2013-01-31 | Olympus Corp | Spectral image device |
| CN103501683A (en) * | 2012-03-30 | 2014-01-08 | 奥林巴斯医疗株式会社 | Endoscopic device |
| CN104068823A (en) * | 2014-07-21 | 2014-10-01 | 中国科学院遥感与数字地球研究所 | In-vivo microendoscopic spectral imaging system |
| CN109640868A (en) * | 2016-09-09 | 2019-04-16 | 直观外科手术操作公司 | Simultaneous with the imaging system of white light and EO-1 hyperion light |
| CN209074529U (en) * | 2018-07-24 | 2019-07-09 | 苏州网颢信息科技有限公司 | Spectrum and image Double channel endoscope system |
| CN110478625A (en) * | 2019-08-21 | 2019-11-22 | 中国科学院苏州生物医学工程技术研究所 | The accurate treatment system of light power guided based on EO-1 hyperion and fluorescence imaging |
| CN210043986U (en) * | 2019-05-17 | 2020-02-11 | 山东本然生物科技有限公司 | Noninvasive intelligent diagnosis device for benign and malignant skin tumors by utilizing hyperspectral imaging technology |
-
2020
- 2020-04-26 CN CN202010340445.XA patent/CN111579498B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2013017507A (en) * | 2011-07-07 | 2013-01-31 | Olympus Corp | Spectral image device |
| CN103501683A (en) * | 2012-03-30 | 2014-01-08 | 奥林巴斯医疗株式会社 | Endoscopic device |
| CN104068823A (en) * | 2014-07-21 | 2014-10-01 | 中国科学院遥感与数字地球研究所 | In-vivo microendoscopic spectral imaging system |
| CN109640868A (en) * | 2016-09-09 | 2019-04-16 | 直观外科手术操作公司 | Simultaneous with the imaging system of white light and EO-1 hyperion light |
| CN209074529U (en) * | 2018-07-24 | 2019-07-09 | 苏州网颢信息科技有限公司 | Spectrum and image Double channel endoscope system |
| CN210043986U (en) * | 2019-05-17 | 2020-02-11 | 山东本然生物科技有限公司 | Noninvasive intelligent diagnosis device for benign and malignant skin tumors by utilizing hyperspectral imaging technology |
| CN110478625A (en) * | 2019-08-21 | 2019-11-22 | 中国科学院苏州生物医学工程技术研究所 | The accurate treatment system of light power guided based on EO-1 hyperion and fluorescence imaging |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2022195222A1 (en) * | 2021-03-17 | 2022-09-22 | Institut Hospitalo-Universitaire De Strasbourg | Medical imaging method employing a hyperspectral camera |
| FR3120940A1 (en) * | 2021-03-17 | 2022-09-23 | Institut Hospitalo-Universitaire De Strasbourg | Medical imaging process using a hyperspectral camera |
| CN113440093A (en) * | 2021-07-19 | 2021-09-28 | 山东第一医科大学附属省立医院(山东省立医院) | Digestive tract secretion microscopic detection system |
| CN115598075A (en) * | 2022-12-14 | 2023-01-13 | 自然资源部第二海洋研究所（Cn） | Deep sea hyperspectral imaging detection system and method based on two-channel coaxial light path |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111579498B (en) | 2021-08-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111579498B (en) | Hyperspectral endoscopic imaging system based on push-broom imaging | |
| US10182708B2 (en) | Simultaneous acquisition of differing image types | |
| US11744439B2 (en) | Micro CMOS scopes for medical imaging | |
| CA2912401C (en) | Microscopy of a tissue sample using structured illumination | |
| US20170223316A1 (en) | Rapid multi-spectral imaging methods and apparatus and applications for cancer detection and localization | |
| US20060184040A1 (en) | Apparatus, system and method for optically analyzing a substrate | |
| Peterson et al. | Feasibility of a video‐mosaicking approach to extend the field‐of‐view for reflectance confocal microscopy in the oral cavity in vivo | |
| CA2527205A1 (en) | Methods and apparatus for fluorescence imaging using multiple excitation-emission pairs and simultaneous multi-channel image detection | |
| JP2022529366A (en) | Systems and methods for deep learning-based color holographic microscopes | |
| US9271640B2 (en) | Optical speculum | |
| US20230239583A1 (en) | Method and system for joint demosaicking and spectral signature estimation | |
| WO2020222307A1 (en) | Endoscopic device capable of high-definition imaging and spectral analysis | |
| US11619548B2 (en) | Hybrid spectral imaging devices, systems and methods | |
| CN117314754A (en) | Double-shot hyperspectral image imaging method and system and double-shot hyperspectral endoscope | |
| Coole et al. | Multimodal optical imaging with real-time projection of cancer risk and biopsy guidance maps for early oral cancer diagnosis and treatment | |
| AU2017346249B2 (en) | Multi-wavelength endoscopic system and image processing method using same | |
| KR20120012191A (en) | Optimal LED Light for Endoscope Maximizing RGB Distsnce between Object | |
| US20170019575A1 (en) | Optical Methods and Devices For Enhancing Image Contrast In the Presence of Bright Background | |
| US20190090726A1 (en) | Optical device using liquid crystal tunable wavelength filter | |
| CN118021243B (en) | Single-path double-spectrum real-time endoscope device based on depth network reconstruction | |
| US10264190B2 (en) | Adaptive optical methods and devices for enhancing image contrast in the presence of bright background | |
| US20190239730A1 (en) | Multi-wavelength endoscopic system and image processing method using same | |
| Clancy et al. | A triple endoscope system for alignment of multispectral images of moving tissue | |
| Chua et al. | A Hyperspectral Endomicroscopy System for Medical Diagnosis | |
| Vega et al. | Development of a Hyperspectral Colposcope for Early Detection and Assessment of Cervical Dysplasia |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |