CN108040243A - Multispectral 3-D visual endoscope device and image interfusion method - Google Patents

Multispectral 3-D visual endoscope device and image interfusion method Download PDF

Info

Publication number
CN108040243A
CN108040243A CN201711258979.2A CN201711258979A CN108040243A CN 108040243 A CN108040243 A CN 108040243A CN 201711258979 A CN201711258979 A CN 201711258979A CN 108040243 A CN108040243 A CN 108040243A
Authority
CN
China
Prior art keywords
image
pyramid
infrared
top layer
green channel
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201711258979.2A
Other languages
Chinese (zh)
Other versions
CN108040243B (en
Inventor
陈春晓
李建飞
董琰彪
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nanjing University of Aeronautics and Astronautics
Original Assignee
Nanjing University of Aeronautics and Astronautics
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nanjing University of Aeronautics and Astronautics filed Critical Nanjing University of Aeronautics and Astronautics
Priority to CN201711258979.2A priority Critical patent/CN108040243B/en
Publication of CN108040243A publication Critical patent/CN108040243A/en
Application granted granted Critical
Publication of CN108040243B publication Critical patent/CN108040243B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/50Constructional details
    • H04N23/54Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/50Constructional details
    • H04N23/55Optical parts specially adapted for electronic image sensors; Mounting thereof
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/50Constructional details
    • H04N23/555Constructional details for picking-up images in sites, inaccessible due to their dimensions or hazardous conditions, e.g. endoscopes or borescopes

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Endoscopes (AREA)
  • Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)

Abstract

The invention discloses a kind of multispectral 3-D visual endoscope device and image interfusion method, belong to medical instruments field.It includes:Light source portion, image pickup part, image processing part, display unit.Image interfusion method is:Step 1, image is obtained by endoscopic images acquiring unit;Step 2, gaussian pyramid and laplacian pyramid are built respectively;Step 3, handle and merge near-infrared image and each layer laplacian pyramid of the green image in addition to top layer;Step 4, target area segmentation is carried out to near-infrared pyramid top layer images;Step 5, handle and merge top layer green image and near-infrared image;Step 6, according to the Laplacian pyramid reconstruction bottom green image after fusion;Step 7, the green image of reconstruct is merged with other passages to obtain final blending image;Step 8, two groups of blending images are handled using anaglyph displacement method, generates stereo pairs.Real three-dimensional structure information has been obtained using this method.

Description

Multispectral 3-D visual endoscope device and image interfusion method
Technical field
The invention discloses a kind of multi-spectrum endoscopic device and image interfusion method, more particularly to one kind to be based on binocular tri-dimensional The multi-spectrum endoscopic device and image interfusion method of feel, belong to medical instruments field.
Background technology
Endoscope is a kind of optical instrument, by cold light source camera lens, fiber optic conducting wire, image delivering system, screen display system The composition such as system, can expand surgical field of view.The outstanding advantages of endoscope are that operation flexible and convenient, operative incision be small, after-operation response Gently, the diagnosis and treatment ability of doctor can be improved, thus in sides such as clinical diagnosis, treatment and monitorings to pathogenesis, pathological change Face is applied widely.
What conventional endoscope was observed is the structural images of histoorgan level, it is impossible to realizes body vessel, lymphatic vessel, swells The functional image of knurl etc., reduces the accuracy of operation.And existing multi-spectrum endoscopic passes through in the preoperative to noting in patient body Shoot at the target to or non-targeted optical molecular developer, can be gathered in operation and show reflection anatomical structure information colour The optical molecular image near-infrared image of image and marked tumor, blood vessel, lymphatic vessel etc..But existing multi-spectrum endoscopic is only Anatomical three-dimensional structure actual in two dimensional image, and surgical procedure is shown there are deviation, therefore can not accurate judgement lesion position Put, be easy to cause doctor and make a fault in operation.There is presently no the effective ways that can be solved the above problems.
The content of the invention
Problem to be solved by this invention is to overcome the 2 d plane picture obtained in traditional multi-spectrum endoscopic course of work Deficiency as that can not reflect steric information, there is provided a kind of multispectral 3-D visual endoscope device and image interfusion method, make to obtain The information for the image reflection got is more comprehensive, so as to improve accuracy and the accuracy that doctor performs the operation, reduces error Occur.
People can be perceived between object and the distance of oneself and object when observing object while body form is perceived Relative position relation, but the depth information of object can be lost during the display of existing 2D displays.Can be with using 3D display technology The stereo-picture with depth feelings is effectively showed, overcomes the shortcomings of that three-dimensional depth information is lost in 2D displays.Binocular solid is shown One kind of 3D display is shown as, human-eye stereoscopic vision characteristic is mainly realized by technical modellings such as optics, by space object with vertical Body information mode reproduces.The technology can help doctor to judge the opposite position between lesion position in space and different tissues Put.
Single channel collection camera is replaced with simulation binocular vision by the endoscope apparatus on the basis of traditional multi-spectrum endoscopic The two-way camera of feel, the two-way coloured image collected according to endoscope, using described image fusion method and near-infrared image It is based respectively on image pyramid algorithm to be merged, and the two-way image after fusion is inputted after anaglyph displacement method processing Observed in stereoscopic display by polaroid glasses, so as to restore real anatomical three-dimensional structure.
From the principle of multi-spectrum endoscopic, the structure letter for the color light image reflection target that multi-spectrum endoscopic obtains Breath, and near-infrared image reflects its functional information, extracts the detailed information of near-infrared image and is merged with color light image, Obtained blending image can reflect the structural information and functional information of target at the same time, improve the identification of image.First, adopt Spatial scene is watched with dual-channel camera structural simulation human eye, same scape is shot from two shooting points in same horizontal line Thing, obtaining two width has the anaglyph of parallax information, and the anaglyph that the shooting of run-in index stereoscopic camera obtains does not have trapezoidal mistake True and vertical parallax, the only negative horizontal parallax of whole scene;Secondly, processing obtains the blending image of two passages respectively, passes through The method of space division includes two images on polarisation display screen;Finally, beholder's images of left and right eyes is made using technologies such as polaroid glasses The horizontal parallax image that originally shooting point is shot from left and right is respectively seen, so as to fulfill stereoscopic display, greatly improves display Image and the compatible degree of real anatomy structure.
Based on above-mentioned thinking, the present invention proposes a kind of multispectral 3-D visual endoscope device, it includes:
Light source portion, there is provided visible ray and exciting light, are made of white light source, excitation source and collector lens, the collector lens The exciting light of the illumination light from the white light source and the excitation source is set to converge to the incident end face of optical fiber;
Image pickup part, wherein image pickup part have:For guiding the optical fiber for the light assembled by the light source portion;Make by the light Light that is fine and being directed to front end spreads and is irradiated to the illuminating lens of observation object;
And there are two to be used to receive the visible of convergence for the camera unit for the imaging assembled for detection, the camera unit The photographing element of the imaging of light and two photographing elements for being used to receive the near infrared light assembled;
Image processing part, it includes:Image acquiring section, it reads and stores the image obtained by the camera unit;
Image co-registration portion, its two groups of RGB color image obtained from described image obtaining section and near-infrared image, and from described RGB color image extraction green channel images, red channel image and blue channel image, by green channel images and right with it The near-infrared image answered builds respective gaussian pyramid and laplacian pyramid respectively, according to the gaussian pyramid and drawing The green channel images are merged the green channel images after reconstructing fusion by this pyramid of pula with the near-infrared image, By the green channel images after the fusion with being obtained after the red channel image and the blue channel image merging treatment Two groups have parallax information and have merged the coloured image after the merging of coloured image and near-infrared image detailed information;
Stereo-picture generating unit, it handles the colour after two groups of fusions with parallax information using anaglyph displacement method Image, generates stereo pairs;
Display unit, the stereo pairs that the stereo-picture generating unit generates are shown as with stereo-picture by it.
Described image fusion portion includes image separative element, it will obtain two from described image obtaining section and be used to receive meeting The RGB color image of one and two near infrared lights for being used to receive convergence in the photographing element of the imaging of poly- visible ray Photographing element in the near-infrared image of one, it is logical that coloured image progress channel separation is obtained into red, green, blueness three The image in road, chooses wherein green channel images;Pyramid construction unit, it will be from described in the acquisition of described image separative element Near-infrared image and the green channel images build the two respective gaussian pyramid as bottom layer image, separately down sampling And laplacian pyramid;
Pyramid updating block, it is respectively compared the near-infrared image and green channel images of the pyramid construction cell formation The numerical values recited of each layer of each pixel of laplacian pyramid in addition to top layer simultaneously takes higher value to save as new La Pula This pyramid;
Top layer pyramid processing unit, it is by the gaussian pyramid of the near-infrared image top layer of the pyramid construction cell formation Image carries out Boundary Extraction and holes filling, then is multiplied with original top layer near-infrared image to obtain the near-infrared figure for removing background Picture;
Top layer pyramid updating block, its be respectively compared the top layer of the green channel images of the pyramid construction cell formation with The value of each pixel of the near-infrared image for the removal background that the pyramid processing unit obtains simultaneously takes higher value to save as The top layer gaussian pyramid of new green channel images;
Image reconstruction unit, its by top layer pyramid updating block according to the top layer gaussian pyramid image after the fusion of acquirement to Up-sampling, is then added with the laplacian pyramid of this layer and continues up sampling, and so circulation is until reconstruct bottom green Channel image;
Image combining unit, the bottom green channel images that its described image reconfiguration unit reconstructs and described image separative element The red and the image of blue channel isolated merge into row of channels, the coloured image after finally being merged.
The camera unit further includes:Two object lens, the object lens assemble the reflected light that object returns from;Two points Light microscopic, spectroscope projection near infrared light, reflects visible ray.
The camera unit further includes:Two object lens, the object lens assemble the reflected light that object returns from;Two points Light microscopic, spectroscope projection near infrared light, reflects visible ray.
A kind of image processing apparatus of the present invention, including:
Image acquiring section, it reads and stores the image obtained by the camera unit;
Image co-registration portion, its two groups of RGB color image obtained from described image obtaining section and near-infrared image, and from described RGB color image extraction green channel images, red channel image and blue channel image, by green channel images and right with it The near-infrared image answered builds respective gaussian pyramid and laplacian pyramid respectively, according to the gaussian pyramid and drawing The green channel images are merged the green channel images after reconstructing fusion by this pyramid of pula with the near-infrared image, By the green channel images after the fusion with being obtained after the red channel image and the blue channel image merging treatment Two groups have parallax information and have merged the coloured image after the merging of coloured image and near-infrared image detailed information, its bag Include;
Image separative element, it takes the photograph the imaging that two visible rays for being used to receive convergence are obtained from described image obtaining section The RGB color image of one in element and two are used to receiving the near of one in the photographing element for the near infrared light assembled Infrared image, obtains the image of red, green, blue three passages by coloured image progress channel separation, chooses its Green Channel image;
Pyramid construction unit, it is by the near-infrared image obtained from described image separative element and the green channel figure As being used as bottom layer image, sampling separately down builds the two respective gaussian pyramid and laplacian pyramid;
Pyramid updating block, it is respectively compared the near-infrared image and green channel images of the pyramid construction cell formation The numerical values recited of each layer of each pixel of laplacian pyramid in addition to top layer simultaneously takes higher value to save as new La Pula This pyramid;
Top layer pyramid processing unit, it is by the gaussian pyramid of the near-infrared image top layer of the pyramid construction cell formation Image carries out Boundary Extraction and holes filling, then is multiplied with original top layer near-infrared image to obtain the near-infrared figure for removing background Picture;
Top layer pyramid updating block, its be respectively compared the top layer of the green channel images of the pyramid construction cell formation with The value of each pixel of the near-infrared image for the removal background that the pyramid processing unit obtains simultaneously takes higher value to save as The top layer gaussian pyramid of new green channel images;
Image reconstruction unit, its by top layer pyramid updating block according to the top layer gaussian pyramid image after the fusion of acquirement to Up-sampling, is then added with the laplacian pyramid of this layer and continues up sampling, and so circulation is until reconstruct bottom green Channel image;
Image combining unit, the bottom green channel images that its described image reconfiguration unit reconstructs and described image separative element The red and the image of blue channel isolated merge into row of channels, the coloured image after finally being merged;
Stereo-picture generating unit, it handles the colour after two groups of fusions with parallax information using anaglyph displacement method Image, generates stereo pairs.
Present invention also offers a kind of multispectral stereo-picture fusion method, it comprises the following steps:
Step 1:The one group of near-infrared image and RGB color image obtained from image acquisition equipment, wherein, one group of near-infrared Image and RGB color image include a frame RGB color image and the frame near-infrared obtained with its same optical channel, synchronization Image;By its isolated green channel images of image, red channel image and the blueness of three passages of RGB color image Channel image;
Step 2:The near-infrared image and the green channel images are built the two as bottom layer image, separately down sampling Respective gaussian pyramid and laplacian pyramid;
Step 3:For each layer of laplacian pyramid of the near-infrared image and the green image in addition to top layer, difference Compare the numerical values recited of two kinds of each pixels of image and take higher value to save as new laplacian pyramid;
Step 4:Carry out Boundary Extraction and holes filling to the gaussian pyramid image of near-infrared image top layer, then with original top Layer near-infrared image is multiplied to obtain the near-infrared image for removing background;
Step 5:It is respectively compared the value of each pixel of near-infrared image of the green channel images of top layer with removing background simultaneously Higher value is taken to save as new top layer gaussian pyramid;
Step 6:By the top layer gaussian pyramid image that step 5 obtains to up-sampling, then with the laplacian pyramid of this layer Addition continues up sampling, and so circulation is until reconstruct bottom green channel images;
Step 7:Red that bottom green channel images that step 6 obtains are isolated with step 1 and the image of blue channel into Row of channels merges, and obtains the coloured image after the first fusion;
Step 8:Another group of near-infrared image and RGB color image obtained from image acquisition equipment, repeat step 1-7, obtains Coloured image after second fusion, after handling two groups of first fusions with parallax information using anaglyph displacement method Coloured image and it is described second fusion after coloured image, generate stereo pairs, wherein another group of near-infrared image and RGB color image includes a frame RGB color image and the frame near-infrared image obtained with its same optical channel, synchronization, The frame RGB color image that wherein this step obtains is identical at the time of acquisition with the frame RGB color image that step 1 obtains, and obtains The optical channel taken is different.
Beneficial effect
The present invention compared with prior art, has following technique effect using above technical scheme:
The present invention can lost part high frequency when artwork is built gaussian pyramid to down-sampling using image pyramid algorithm The characteristics of detailed information, establish the laplacian pyramid for including the high-frequency information lost, by comparing each layer of green of processing The laplacian pyramid of channel image and near-infrared image can amplify and merge detailed information, meanwhile, select RGB color figure The green channel images of picture carry out fusion and have the following advantages:
Firstly, since needing to open wound structure endoscope path in clinical operation, therefore inevitably there are bleeding feelings Condition, and organism intracorporeal organ takes on a red color mostly under normal circumstances, thus using RGB color image red channel image into During row fusion the factors such as blood will necessarily be subject to disturb;Secondly, human eye for the blue channel in RGB color image image not Sensitivity, causes the change of the subtle blue channel information in observation, thus blue channel image is also unsuitable for blending image. Consider, final green channel images of choosing are merged with near-infrared image, and then are reconstructed and merged color light image And the blending image with more prominent detailed information of near-infrared image.Processing represents left eye and the right side respectively in this way The image of the dual-channel camera collection of eye, and it is that can obtain with stereoeffect to shift method to handle blending image by anaglyph Image pair.Using the image that this method obtains to that can be observed after input 3 d display device by polaroid glasses with three-dimensional The stereo-picture of structure, there is provided real three-dimensional structure information, helps to improve surgical precision and accuracy.
Brief description of the drawings
Fig. 1 is the system structure diagram of the multispectral 3-D visual endoscope device of the present invention;
Fig. 2 is the camera unit of the multispectral 3-D visual endoscope device of the present invention;
Fig. 3 is the structure chart of the image processing part of the multispectral 3-D visual endoscope device of the present invention;
Fig. 4 is the flow chart of image interfusion method of the present invention;
The near-infrared image of Fig. 5 single channels camera collection;
Green channel images after the colourama channel separation of Fig. 6 single channels camera collection;
The gaussian pyramid image that it is 4 as the number of plies that bottom layer image is built using near-infrared image that Fig. 7, which is,.
Embodiment
The present invention is explained in further detail below in conjunction with the accompanying drawings, it is noted that described embodiment only purport Easy to the understanding of the present invention, without playing any restriction effect to it.
Present embodiment is multichannel endoscope, is preferably Double channel endoscope, Fig. 1 is the light more for showing the embodiment Compose the integrally-built block diagram of 3-D visual endoscope device.The multispectral 3-D visual endoscope device of the present invention is formed by light Source portion 100, image pickup part 200, image processing part 300, display unit 400 are formed.
Light source portion 100 is made of white light source 101 and excitation source 102, collector lens 103, which makes to come The exciting light of illumination light and excitation source 102 from white light source 101 converges to the incident end face of optical fiber 201.
Image pickup part 200 is formed as elongated and can be with curved structure in order to be inserted into body cavity.Image pickup part 200 has:With In the optical fiber 201 for the light that guiding is assembled by light source portion 100;Make to spread and shine being directed to the light of front end by the optical fiber 201 It is mapped to the illuminating lens 202 of observation object;For detecting the camera unit for the imaging assembled.
Illustrate the camera unit of the multispectral 3-D visual endoscope device of the present invention with reference to Fig. 2.Camera unit includes Two object lens 203 for assembling the reflected light that object returns from;Two spectroscopes 211, spectroscope projection near infrared light, instead Penetrate visible ray;Two are used to receive and detect the photographing element 209 of the imaging of the visible ray of convergence, its color camera spectrum phase It is 400-700nm to answer scope;Two photographing elements 210 for being used to receive the near infrared light assembled, its near infrared camera spectrum are rung It is 700-900nm to answer scope, and it is 830-850nm to set filter spectral scope in front of it.What white light source 101 was sent can See illumination to be mapped to after observation object and light is penetrated while respectively by a meeting coalescence point in two object lens 203 by returning for observation object reflection The reflected light back is used to detect the visible of convergence by one not be transferred in two spectroscopes 211, spectroscope 211 to two The photographing element 209 of the imaging of light, the photographing element 209 are ccd image sensor;The excitation sent by excitation source 102 Illumination is mapped to the near infrared light produced after observation object and also while is respectively respectively transmitted to by a meeting coalescence in two object lens 203 One in two spectroscopes 211, spectroscope 211 is by the transmission of near infra red light to the shooting for receiving the near infrared light assembled Element 210, the photographing element 210 are ccd image sensor.White light source 101 and excitation source 102 shine at the same time, shooting member Part 209 and 210 receives light at the same time.
Fig. 3 shows that image processing part 300 possesses A/D converter sections 310, image acquiring section 320, image co-registration portion 330, vertical Body image production part 340 and control unit 350.A/D converter sections 310 are by the process opto-electronic conversion from photographing element 209 and 210 Analog signal afterwards is converted to digital signal, and exports to image acquiring section 320, image acquiring section 320 and read and store by taking the photograph The image obtained as unit.
Control unit 350 is by hardware realizations such as CPU, by itself and A/D converter sections 310, image acquiring section 320, image co-registration portion 330 are connected with stereo-picture generating unit 340, and generate and control their control signal, and control unit is uniformly controlled at image The action of reason portion entirety.
Image co-registration portion 330 carries out the fusion treatment of visible ray and near-infrared image, and image co-registration portion 330 is obtained from image Portion 320 is obtained can obtain the image for representing right and left eyes vision using the dual-channel camera structure shown in Fig. 2, and processing respectively obtains two After the blending image of a passage, by blending image and follow-up stereogram can be passed through by the processing of binocular vision related algorithm Image is presented on display unit 400 in three dimensions as generating unit 340 generates 3-D view, greatly improve display image with it is true The compatible degree of real anatomical structure.
Image co-registration portion 330 include separation green channel images image separative element 3301, structure gaussian pyramid and The pyramid of each layer laplacian pyramid of the pyramid construction unit 3302, renewal of laplacian pyramid in addition to top layer Updating block 3303, the top layer pyramid processing unit 3304 of processing and renewal top layer gaussian pyramid and the renewal of top layer pyramid Unit 3305, the image reconstruction unit 3306 for reconstructing bottom green channel images and the image conjunction for obtaining final blending image And unit 3307.
Then the action in image co-registration portion 330 is illustrated, Fig. 4 is the action flow chart in image co-registration portion 330.
First, the acquisition that image separative element 3301 will obtain in two photographing elements 209 from image acquiring section 320 RGB color image and with one two photographing elements positioned at same optical channel in described two photographing elements 209 A near-infrared image obtained in synchronization in 210(As shown in Figure 5), coloured image progress channel separation is obtained red Color, green, the image of blue three passages, choose wherein green channel images, as shown in Figure 6.
Secondly, the near-infrared image and green channel that pyramid construction unit 3302 will be obtained from image separative element 3301 Image builds the two respective gaussian pyramid and laplacian pyramid as bottom layer image, separately down sampling.Wherein, exist During building gaussian pyramid, level is QUOTE in order to obtain Pyramid diagram picture, it is necessary to be adopted downwards Sample, i.e.,:
(1)To upper layer images QUOTE Carry out Gaussian kernel convolution;
(2)Remove all even number lines and even column;
Obtain QUOTE After repeat this process and can construct complete gaussian pyramid, what is finally built is near red The gaussian pyramid of outer image, as shown in Figure 7.
There is equation below when building laplacian pyramid:
QUOTE =QUOTE –pyrUP( QUOTE
Wherein pyrUP () is image to be carried out to up-sampling, i.e.,:
1. it is by i+1 tomographic image position(x,y)Point be mapped to the i-th tomographic image(2x+1,2y+1)Position;
2. with identical 4 convolution of Gaussian kernel *;
Repeat this process and can obtain complete laplacian pyramid.
Next, pyramid updating block 3303 be respectively compared pyramid construction unit 3302 structure near-infrared image and Each layer laplacian pyramid QUOTE of the green channel images in addition to top layer The numerical values recited of each pixel is simultaneously Higher value is taken to save as new laplacian pyramid.
Then, the near-infrared image top layer that top layer pyramid processing unit 3304 builds pyramid construction unit 3302 Gaussian pyramid image QUOTE , go out border in image by using Canny operator extractions, calculated using holes filling Method closes target area boundaries, and pixel value in target area is set to 1, region exterior pixel value zero setting, then with near-infrared image Top layer artwork be multiplied, obtain removing background influence and only include the near-infrared image QUOTE of target area
Furthermore top layer pyramid updating block 3305 is respectively compared the green channel built by pyramid construction unit 3302 The near-infrared image QUOTE for the removal background that the gaussian pyramid top layer of image is obtained with pyramid processing unit 3304 Each pixel pixel value, by QUOTE In pixel value replace with higher value in both, merged Top layer green channel images QUOTE afterwards
Then, the top layer after the fusion that image reconstruction unit 3306 obtains top layer pyramid updating block 3305(N-th Layer)Green channel images QUOTE To up-sampling, next layer after being obscured((n-1)th layer)Image pyrUP( QUOTE ), then the Laplacian-pyramid image QUOTE of this layer updated with pyramid updating block 3303 Fusion, obtains (n-1)th layer of blending image QUOTE , i.e.,:
QUOTE =QUOTE + pyrUP( QUOTE
By QUOTE Above-mentioned sampling is repeated as upper layer images and fusion process obtains every layer Blending image, finally reconstructs the bottom green channel images QUOTE after fusion
Finally, the bottom green channel images QUOTE that image combining unit 3307 reconstructs image reconstruction unit 3306 The red and the image of blue channel isolated with image separative element 3301 merge into row of channels, obtain the first fusion Coloured image afterwards.
In addition, RGB color image and and above-mentioned two that another in two photographing elements 209 is obtained in synchronization Another in photographing element 209 be located in same two photographing elements 210 of optical channel another obtained in synchronization Coloured image after the second fusion that near-infrared image obtains after image co-registration portion 330 is handled.
Stereo-picture generating unit 340 obtains the coloured image after first fusion and the cromogram after the second fusion Picture.The anaglyph obtained by the shooting of run-in index stereoscopic camera does not have trapezoidal distortion and vertical parallax, but whole scene is only negative Horizontal parallax.Stereo-picture generating unit 340 shifts method to change the disparity range in anaglyph by anaglyph for this, obtains Positive and negative horizontal parallax is obtained, passes through anaglyph and shifts the coloured image after first fusion of the method processing with parallax information Stereo pairs are obtained with after the coloured image after the described second fusion.
Display unit 400 is polarisation display screen, and the stereo pairs that the stereo-picture generating unit 340 generates are shown as by it Stereo-picture, the graphics for being observed object that observer has stereoscopic vision effect by wearing polaroid glasses to can be observed Picture.
In addition, the present invention proposes that a kind of image interfusion method specifically includes:
Step 1:The one group of near-infrared image and RGB color image obtained from image acquisition equipment, wherein, one group of near-infrared Image and RGB color image include a frame RGB color image and the frame near-infrared obtained with its same optical channel, synchronization Image;By its isolated green channel images of image, red channel image and the blueness of three passages of RGB color image Channel image;
Step 2:The near-infrared image and the green channel images are built the two as bottom layer image, separately down sampling Respective gaussian pyramid and laplacian pyramid;
Step 3:For each layer of laplacian pyramid of the near-infrared image and the green image in addition to top layer, difference Compare the numerical values recited of two kinds of each pixels of image and take higher value to save as new laplacian pyramid;
Step 4:Carry out Boundary Extraction and holes filling to the gaussian pyramid image of near-infrared image top layer, then with original top Layer near-infrared image is multiplied to obtain the near-infrared image for removing background;
Step 5:It is respectively compared the value of each pixel of near-infrared image of the green channel images of top layer with removing background simultaneously Higher value is taken to save as new top layer gaussian pyramid;
Step 6:By the top layer gaussian pyramid image that step 5 obtains to up-sampling, then with the laplacian pyramid of this layer Addition continues up sampling, and so circulation is until reconstruct bottom green channel images;
Step 7:Red that bottom green channel images that step 6 obtains are isolated with step 1 and the image of blue channel into Row of channels merges, and obtains the coloured image after the first fusion;
Step 8:Another group of near-infrared image and RGB color image obtained from image acquisition equipment, repeat step 1-7, obtains Coloured image after second fusion, after handling two groups of first fusions with parallax information using anaglyph displacement method Coloured image and it is described second fusion after coloured image, generate stereo pairs, wherein another group of near-infrared image and RGB color image includes a frame RGB color image and the frame near-infrared image obtained with its same optical channel, synchronization, The frame RGB color image that wherein this step obtains is identical at the time of acquisition with the frame RGB color image that step 1 obtains, and obtains The optical channel taken is different.
The above is only the preferred embodiment of the present invention, it should be pointed out that:For the ordinary skill people of the art For member, various improvements and modifications may be made without departing from the principle of the present invention, these improvements and modifications also should It is considered as protection scope of the present invention.

Claims (6)

  1. A kind of 1. multispectral 3-D visual endoscope device, it is characterised in that including:
    Light source portion, there is provided visible ray and exciting light, are made of white light source, excitation source and collector lens, the collector lens The exciting light of the illumination light from the white light source and the excitation source is set to converge to the incident end face of optical fiber;
    Image pickup part, wherein image pickup part have:
    For guiding the optical fiber for the light assembled by the light source portion;
    Make the illuminating lens for spreading and being irradiated to observation object being directed to the light of front end by the optical fiber;And
    For detecting the camera unit for the imaging assembled, there are the camera unit two to be used for the visible ray that reception is assembled The photographing element of imaging and two photographing elements for being used to receive the near infrared light assembled;
    Image processing part, it includes:
    Image acquiring section, it reads and stores the image obtained by the camera unit;
    Image co-registration portion, its two groups of RGB color image obtained from described image obtaining section and near-infrared image, and from described RGB color image extraction green channel images, red channel image and blue channel image, by green channel images and right with it The near-infrared image answered builds respective gaussian pyramid and laplacian pyramid respectively, according to the gaussian pyramid and drawing The green channel images are merged the green channel images after reconstructing fusion by this pyramid of pula with the near-infrared image, By the green channel images after the fusion with being obtained after the red channel image and the blue channel image merging treatment Two groups have parallax information and have merged the coloured image after the merging of coloured image and near-infrared image detailed information;
    Stereo-picture generating unit, it handles the colour after two groups of fusions with parallax information using anaglyph displacement method Image, generates stereo pairs;
    Display unit, the stereo pairs that the stereo-picture generating unit generates are shown as with stereo-picture by it.
  2. 2. multispectral 3-D visual endoscope device according to claim 1, it is characterised in that wrap in described image fusion portion Include:
    Image separative element, it takes the photograph the imaging that two visible rays for being used to receive convergence are obtained from described image obtaining section The RGB color image of one in element and two are used to receiving the near of one in the photographing element for the near infrared light assembled Infrared image, obtains the image of red, green, blue three passages by coloured image progress channel separation, chooses its Green Channel image;
    Pyramid construction unit, it is by the near-infrared image obtained from described image separative element and the green channel figure As being used as bottom layer image, sampling separately down builds the two respective gaussian pyramid and laplacian pyramid;
    Pyramid updating block, it is respectively compared the near-infrared image and green channel images of the pyramid construction cell formation The numerical values recited of each layer of each pixel of laplacian pyramid in addition to top layer simultaneously takes higher value to save as new La Pula This pyramid;
    Top layer pyramid processing unit, it is by the gaussian pyramid of the near-infrared image top layer of the pyramid construction cell formation Image carries out Boundary Extraction and holes filling, then is multiplied with original top layer near-infrared image to obtain the near-infrared figure for removing background Picture;
    Top layer pyramid updating block, its be respectively compared the top layer of the green channel images of the pyramid construction cell formation with The value of each pixel of the near-infrared image for the removal background that the pyramid processing unit obtains simultaneously takes higher value to save as The top layer gaussian pyramid of new green channel images;
    Image reconstruction unit, its by top layer pyramid updating block according to the top layer gaussian pyramid image after the fusion of acquirement to Up-sampling, is then added with the laplacian pyramid of this layer and continues up sampling, and so circulation is until reconstruct bottom green Channel image;
    Image combining unit, the bottom green channel images that its described image reconfiguration unit reconstructs and described image separative element The red and the image of blue channel isolated merge into row of channels, the coloured image after finally being merged.
  3. 3. multispectral 3-D visual endoscope device according to claim 1, it is characterised in that the camera unit also wraps Include:
    Two object lens, the object lens assemble the reflected light that object returns from;
    Two spectroscopes, spectroscope projection near infrared light, reflect visible ray.
  4. 4. multispectral 3-D visual endoscope device according to claim 3, it is characterised in that the white light source is sent Radiation of visible light to by returning for observation object reflection penetrating light while respectively by a meeting in described two object lens after observation object Coalescence is respectively transmitted to one in described two spectroscopes, which is used to detect meeting by spectroscope to described two The photographing element of the imaging of poly- visible ray;The exciting light that is sent by the excitation source produces after being irradiated to observation object Near infrared light also while is respectively respectively transmitted to one in described two spectroscopes by a meeting coalescence in described two object lens, Photographing element of the spectroscope by the transmission of near infra red light to the near infrared light for being used to receive convergence.
  5. A kind of 5. image processing apparatus, it is characterised in that including:
    Image acquiring section, it reads and stores the image obtained by the camera unit;
    Image co-registration portion, its two groups of RGB color image obtained from described image obtaining section and near-infrared image, and from described RGB color image extraction green channel images, red channel image and blue channel image, by green channel images and right with it The near-infrared image answered builds respective gaussian pyramid and laplacian pyramid respectively, according to the gaussian pyramid and drawing The green channel images are merged the green channel images after reconstructing fusion by this pyramid of pula with the near-infrared image, By the green channel images after the fusion with being obtained after the red channel image and the blue channel image merging treatment Two groups have parallax information and have merged the coloured image after the merging of coloured image and near-infrared image detailed information, its bag Include;
    Image separative element, it takes the photograph the imaging that two visible rays for being used to receive convergence are obtained from described image obtaining section The RGB color image of one in element and two are used to receiving the near of one in the photographing element for the near infrared light assembled Infrared image, obtains the image of red, green, blue three passages by coloured image progress channel separation, chooses its Green Channel image;
    Pyramid construction unit, it is by the near-infrared image obtained from described image separative element and the green channel figure As being used as bottom layer image, sampling separately down builds the two respective gaussian pyramid and laplacian pyramid;
    Pyramid updating block, it is respectively compared the near-infrared image and green channel images of the pyramid construction cell formation The numerical values recited of each layer of each pixel of laplacian pyramid in addition to top layer simultaneously takes higher value to save as new La Pula This pyramid;
    Top layer pyramid processing unit, it is by the gaussian pyramid of the near-infrared image top layer of the pyramid construction cell formation Image carries out Boundary Extraction and holes filling, then is multiplied with original top layer near-infrared image to obtain the near-infrared figure for removing background Picture;
    Top layer pyramid updating block, its be respectively compared the top layer of the green channel images of the pyramid construction cell formation with The value of each pixel of the near-infrared image for the removal background that the pyramid processing unit obtains simultaneously takes higher value to save as The top layer gaussian pyramid of new green channel images;
    Image reconstruction unit, its by top layer pyramid updating block according to the top layer gaussian pyramid image after the fusion of acquirement to Up-sampling, is then added with the laplacian pyramid of this layer and continues up sampling, and so circulation is until reconstruct bottom green Channel image;
    Image combining unit, the bottom green channel images that its described image reconfiguration unit reconstructs and described image separative element The red and the image of blue channel isolated merge into row of channels, the coloured image after finally being merged;
    Stereo-picture generating unit, it handles the colour after two groups of fusions with parallax information using anaglyph displacement method Image, generates stereo pairs.
  6. 6. a kind of multispectral stereo-picture fusion method, it is characterised in that it comprises the following steps:
    Step 1:The one group of near-infrared image and RGB color image obtained from image acquisition equipment, wherein, one group of near-infrared Image and RGB color image include a frame RGB color image and the frame near-infrared obtained with its same optical channel, synchronization Image;By its isolated green channel images of image, red channel image and the blueness of three passages of RGB color image Channel image;
    Step 2:The near-infrared image and the green channel images are built the two as bottom layer image, separately down sampling Respective gaussian pyramid and laplacian pyramid;
    Step 3:For each layer of laplacian pyramid of the near-infrared image and the green image in addition to top layer, difference Compare the numerical values recited of two kinds of each pixels of image and take higher value to save as new laplacian pyramid;
    Step 4:Carry out Boundary Extraction and holes filling to the gaussian pyramid image of near-infrared image top layer, then with original top Layer near-infrared image is multiplied to obtain the near-infrared image for removing background;
    Step 5:It is respectively compared the value of each pixel of near-infrared image of the green channel images of top layer with removing background simultaneously Higher value is taken to save as new top layer gaussian pyramid;
    Step 6:By the top layer gaussian pyramid image that step 5 obtains to up-sampling, then with the laplacian pyramid of this layer Addition continues up sampling, and so circulation is until reconstruct bottom green channel images;
    Step 7:Red that bottom green channel images that step 6 obtains are isolated with step 1 and the image of blue channel into Row of channels merges, and obtains the coloured image after the first fusion;
    Step 8:Another group of near-infrared image and RGB color image obtained from image acquisition equipment, repeat step 1-7, obtains Coloured image after second fusion, after handling two groups of first fusions with parallax information using anaglyph displacement method Coloured image and it is described second fusion after coloured image, generate stereo pairs, wherein another group of near-infrared image and RGB color image includes a frame RGB color image and the frame near-infrared image obtained with its same optical channel, synchronization, The frame RGB color image that wherein this step obtains is identical at the time of acquisition with the frame RGB color image that step 1 obtains, and obtains The optical channel taken is different.
CN201711258979.2A 2017-12-04 2017-12-04 Multispectral 3-D visual endoscope device and image interfusion method Active CN108040243B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201711258979.2A CN108040243B (en) 2017-12-04 2017-12-04 Multispectral 3-D visual endoscope device and image interfusion method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201711258979.2A CN108040243B (en) 2017-12-04 2017-12-04 Multispectral 3-D visual endoscope device and image interfusion method

Publications (2)

Publication Number Publication Date
CN108040243A true CN108040243A (en) 2018-05-15
CN108040243B CN108040243B (en) 2019-06-14

Family

ID=62094817

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201711258979.2A Active CN108040243B (en) 2017-12-04 2017-12-04 Multispectral 3-D visual endoscope device and image interfusion method

Country Status (1)

Country Link
CN (1) CN108040243B (en)

Cited By (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109215079A (en) * 2018-07-17 2019-01-15 艾瑞迈迪医疗科技(北京)有限公司 Image processing method, operation navigation device, electronic equipment, storage medium
CN109688342A (en) * 2019-01-11 2019-04-26 南京理工大学 A kind of multispectral stereo imaging system
CN110602476A (en) * 2019-08-08 2019-12-20 南京航空航天大学 Hole filling method of Gaussian mixture model based on depth information assistance
CN110811498A (en) * 2019-12-19 2020-02-21 中国科学院长春光学精密机械与物理研究所 Visible light and near-infrared fluorescence 3D fusion image endoscope system
CN112089403A (en) * 2020-10-02 2020-12-18 深圳市中安视达科技有限公司 Multispectral medical multi-path imaging method and system thereof
CN112102171A (en) * 2020-09-18 2020-12-18 贝壳技术有限公司 Image processing method, image processing device, computer-readable storage medium and electronic equipment
CN112089404A (en) * 2020-10-10 2020-12-18 南京航空航天大学 Endoscopic OCT-Raman dual-mode imaging device and imaging method
CN112184604A (en) * 2020-09-15 2021-01-05 杭州电子科技大学 Color image enhancement method based on image fusion
WO2021051222A1 (en) * 2019-09-16 2021-03-25 北京数字精准医疗科技有限公司 Endoscope system, mixed light source, video acquisition device and image processor
CN113129243A (en) * 2021-03-10 2021-07-16 同济大学 Blood vessel image enhancement method and system based on infrared and visible light image fusion
CN113688707A (en) * 2021-03-04 2021-11-23 黑芝麻智能科技(上海)有限公司 Face anti-spoofing method
CN114027765A (en) * 2020-11-20 2022-02-11 上海微创医疗机器人(集团)股份有限公司 Fluorescence endoscope system, control method, and storage medium
CN115299914A (en) * 2022-08-05 2022-11-08 上海微觅医疗器械有限公司 Endoscope system, image processing method and device
WO2022267645A1 (en) * 2021-06-21 2022-12-29 中兴通讯股份有限公司 Photography apparatus and method, electronic device, and storage medium
CN116471466A (en) * 2023-06-12 2023-07-21 深圳市博盛医疗科技有限公司 Dual-wafer endoscope imaging method and imaging device

Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100302376A1 (en) * 2009-05-27 2010-12-02 Pierre Benoit Boulanger System and method for high-quality real-time foreground/background separation in tele-conferencing using self-registered color/infrared input images and closed-form natural image matting techniques
CN102298769A (en) * 2011-06-11 2011-12-28 浙江理工大学 Colored fusion method of night vision low-light image and infrared image based on color transmission
CN102789640A (en) * 2012-07-16 2012-11-21 中国科学院自动化研究所 Method for fusing visible light full-color image and infrared remote sensing image
CN103300812A (en) * 2013-06-27 2013-09-18 中国科学院自动化研究所 Endoscope-based multispectral video navigation system and method
CN103530853A (en) * 2013-10-17 2014-01-22 中北大学 Infrared intensity image and infrared polarization image enhancement and fusion method
CN104274148A (en) * 2014-09-28 2015-01-14 安徽中科医药成像技术科技有限公司 Imaging system
CN105096285A (en) * 2014-05-23 2015-11-25 南京理工大学 Image fusion and target tracking system based on multi-core DSP
CN105447838A (en) * 2014-08-27 2016-03-30 北京计算机技术及应用研究所 Method and system for infrared and low-level-light/visible-light fusion imaging
CN106236006A (en) * 2016-08-31 2016-12-21 杨晓峰 3D optical molecular image peritoneoscope imaging system
CN106960428A (en) * 2016-01-12 2017-07-18 浙江大立科技股份有限公司 Visible ray and infrared double-waveband image co-registration Enhancement Method
CN107103596A (en) * 2017-04-27 2017-08-29 湖南源信光电科技股份有限公司 A kind of color night vision image interfusion method based on yuv space

Patent Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100302376A1 (en) * 2009-05-27 2010-12-02 Pierre Benoit Boulanger System and method for high-quality real-time foreground/background separation in tele-conferencing using self-registered color/infrared input images and closed-form natural image matting techniques
CN102298769A (en) * 2011-06-11 2011-12-28 浙江理工大学 Colored fusion method of night vision low-light image and infrared image based on color transmission
CN102789640A (en) * 2012-07-16 2012-11-21 中国科学院自动化研究所 Method for fusing visible light full-color image and infrared remote sensing image
CN103300812A (en) * 2013-06-27 2013-09-18 中国科学院自动化研究所 Endoscope-based multispectral video navigation system and method
CN103530853A (en) * 2013-10-17 2014-01-22 中北大学 Infrared intensity image and infrared polarization image enhancement and fusion method
CN105096285A (en) * 2014-05-23 2015-11-25 南京理工大学 Image fusion and target tracking system based on multi-core DSP
CN105447838A (en) * 2014-08-27 2016-03-30 北京计算机技术及应用研究所 Method and system for infrared and low-level-light/visible-light fusion imaging
CN104274148A (en) * 2014-09-28 2015-01-14 安徽中科医药成像技术科技有限公司 Imaging system
CN106960428A (en) * 2016-01-12 2017-07-18 浙江大立科技股份有限公司 Visible ray and infrared double-waveband image co-registration Enhancement Method
CN106236006A (en) * 2016-08-31 2016-12-21 杨晓峰 3D optical molecular image peritoneoscope imaging system
CN107103596A (en) * 2017-04-27 2017-08-29 湖南源信光电科技股份有限公司 A kind of color night vision image interfusion method based on yuv space

Cited By (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109215079A (en) * 2018-07-17 2019-01-15 艾瑞迈迪医疗科技(北京)有限公司 Image processing method, operation navigation device, electronic equipment, storage medium
CN109688342A (en) * 2019-01-11 2019-04-26 南京理工大学 A kind of multispectral stereo imaging system
CN110602476A (en) * 2019-08-08 2019-12-20 南京航空航天大学 Hole filling method of Gaussian mixture model based on depth information assistance
CN110602476B (en) * 2019-08-08 2021-08-06 南京航空航天大学 Hole filling method of Gaussian mixture model based on depth information assistance
WO2021051222A1 (en) * 2019-09-16 2021-03-25 北京数字精准医疗科技有限公司 Endoscope system, mixed light source, video acquisition device and image processor
CN110811498A (en) * 2019-12-19 2020-02-21 中国科学院长春光学精密机械与物理研究所 Visible light and near-infrared fluorescence 3D fusion image endoscope system
CN112184604B (en) * 2020-09-15 2024-02-20 杭州电子科技大学 Color image enhancement method based on image fusion
CN112184604A (en) * 2020-09-15 2021-01-05 杭州电子科技大学 Color image enhancement method based on image fusion
CN112102171A (en) * 2020-09-18 2020-12-18 贝壳技术有限公司 Image processing method, image processing device, computer-readable storage medium and electronic equipment
CN112102171B (en) * 2020-09-18 2021-10-08 贝壳找房(北京)科技有限公司 Image processing method, image processing device, computer-readable storage medium and electronic equipment
CN112089403A (en) * 2020-10-02 2020-12-18 深圳市中安视达科技有限公司 Multispectral medical multi-path imaging method and system thereof
CN112089404A (en) * 2020-10-10 2020-12-18 南京航空航天大学 Endoscopic OCT-Raman dual-mode imaging device and imaging method
CN114027765A (en) * 2020-11-20 2022-02-11 上海微创医疗机器人(集团)股份有限公司 Fluorescence endoscope system, control method, and storage medium
CN113688707A (en) * 2021-03-04 2021-11-23 黑芝麻智能科技(上海)有限公司 Face anti-spoofing method
US12002294B2 (en) 2021-03-04 2024-06-04 Black Sesame Technologies Inc. RGB-NIR dual camera face anti-spoofing method
CN113129243A (en) * 2021-03-10 2021-07-16 同济大学 Blood vessel image enhancement method and system based on infrared and visible light image fusion
WO2022267645A1 (en) * 2021-06-21 2022-12-29 中兴通讯股份有限公司 Photography apparatus and method, electronic device, and storage medium
CN115299914A (en) * 2022-08-05 2022-11-08 上海微觅医疗器械有限公司 Endoscope system, image processing method and device
CN116471466A (en) * 2023-06-12 2023-07-21 深圳市博盛医疗科技有限公司 Dual-wafer endoscope imaging method and imaging device
CN116471466B (en) * 2023-06-12 2023-09-05 深圳市博盛医疗科技有限公司 Dual-wafer endoscope imaging method and imaging device

Also Published As

Publication number Publication date
CN108040243B (en) 2019-06-14

Similar Documents

Publication Publication Date Title
CN108040243B (en) Multispectral 3-D visual endoscope device and image interfusion method
CN106236006B (en) 3D optical molecular image laparoscope imaging systems
CN104883946B (en) Image processing device, electronic apparatus, endoscope device, and image processing method
US10307046B2 (en) Method for spatial 3D stereoscopic intraoral camera
CN102036599B (en) Imaging system for combined full-color reflectance and near-infrared imaging
KR101608848B1 (en) System and method for generating a multi-dimensional image
CN109222865B (en) Multimode imaging endoscope system
Geng et al. Review of 3-D endoscopic surface imaging techniques
US10117579B2 (en) Simultaneous capture of filtered images of the eye
US10327627B2 (en) Use of plenoptic otoscope data for aiding medical diagnosis
CN102300498A (en) Equipment For Infrared Vision Of Anatomical Structures And Signal Processing Methods Thereof
WO2016048911A1 (en) Hyperspectral imager
JP6156787B2 (en) Imaging observation device
WO2015023990A1 (en) Method and apparatus for quantitative and depth resolved hyperspectral fluorescence and reflectance imaging for surgical guidance
JP6972049B2 (en) Image processing method and image processing device using elastic mapping of vascular plexus structure
JP2011120916A (en) System for medial imaging
KR20160089355A (en) Device for non-invasive detection of predetermined biological structures
DE112015002821T5 (en) observing system
CN107977950A (en) Rapid and effective video image fusion method based on multi-scale guide filtering
CN107105987A (en) Image processing device, method for operating image processing device, program for operating image processing device, and endoscope device
CN109068035A (en) A kind of micro- camera array endoscopic imaging system of intelligence
JP5740559B2 (en) Image processing apparatus and endoscope
WO2021245374A1 (en) Method and system for joint demosaicking and spectral signature estimation
CN116158718A (en) Imaging and display method for endoscope system and endoscope system
CN116172493A (en) Imaging and display method for endoscope system and endoscope system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant