CN108040243A - Multispectral 3-D visual endoscope device and image interfusion method - Google Patents
Multispectral 3-D visual endoscope device and image interfusion method Download PDFInfo
- Publication number
- CN108040243A CN108040243A CN201711258979.2A CN201711258979A CN108040243A CN 108040243 A CN108040243 A CN 108040243A CN 201711258979 A CN201711258979 A CN 201711258979A CN 108040243 A CN108040243 A CN 108040243A
- Authority
- CN
- China
- Prior art keywords
- image
- pyramid
- infrared
- top layer
- green channel
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000000007 visual effect Effects 0.000 title claims abstract 6
- 238000000034 method Methods 0.000 title abstract description 24
- 230000004927 fusion Effects 0.000 claims abstract description 32
- 238000011549 displacement method Methods 0.000 claims abstract 4
- 238000003384 imaging method Methods 0.000 claims description 48
- 238000010276 construction Methods 0.000 claims description 21
- 238000005070 sampling Methods 0.000 claims description 18
- 238000000926 separation method Methods 0.000 claims description 15
- 230000005284 excitation Effects 0.000 claims description 13
- 230000003287 optical effect Effects 0.000 claims description 12
- 238000000605 extraction Methods 0.000 claims description 9
- 239000013307 optical fiber Substances 0.000 claims description 9
- 238000007500 overflow downdraw method Methods 0.000 claims description 8
- 238000005286 illumination Methods 0.000 claims description 5
- 238000011282 treatment Methods 0.000 claims description 3
- 230000005540 biological transmission Effects 0.000 claims description 2
- 230000015572 biosynthetic process Effects 0.000 claims 6
- 238000004581 coalescence Methods 0.000 claims 2
- 230000000149 penetrating effect Effects 0.000 claims 1
- 230000005855 radiation Effects 0.000 claims 1
- 238000002156 mixing Methods 0.000 abstract 2
- 230000011218 segmentation Effects 0.000 abstract 1
- 230000008569 process Effects 0.000 description 8
- 210000003484 anatomy Anatomy 0.000 description 4
- 238000006243 chemical reaction Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 239000011521 glass Substances 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 206010028980 Neoplasm Diseases 0.000 description 2
- 230000000740 bleeding effect Effects 0.000 description 2
- 210000004204 blood vessel Anatomy 0.000 description 2
- 230000007812 deficiency Effects 0.000 description 2
- 239000000284 extract Substances 0.000 description 2
- 238000007499 fusion processing Methods 0.000 description 2
- 230000001678 irradiating effect Effects 0.000 description 2
- 230000003902 lesion Effects 0.000 description 2
- 210000001365 lymphatic vessel Anatomy 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 210000000056 organ Anatomy 0.000 description 2
- 230000003595 spectral effect Effects 0.000 description 2
- 208000002847 Surgical Wound Diseases 0.000 description 1
- 206010052428 Wound Diseases 0.000 description 1
- 208000027418 Wounds and injury Diseases 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 239000008280 blood Substances 0.000 description 1
- 210000004369 blood Anatomy 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000003759 clinical diagnosis Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000008506 pathogenesis Effects 0.000 description 1
- 230000036285 pathological change Effects 0.000 description 1
- 231100000915 pathological change Toxicity 0.000 description 1
- 230000002980 postoperative effect Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 238000001228 spectrum Methods 0.000 description 1
- 238000001356 surgical procedure Methods 0.000 description 1
- 238000004148 unit process Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/50—Constructional details
- H04N23/54—Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/50—Constructional details
- H04N23/55—Optical parts specially adapted for electronic image sensors; Mounting thereof
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/50—Constructional details
- H04N23/555—Constructional details for picking-up images in sites, inaccessible due to their dimensions or hazardous conditions, e.g. endoscopes or borescopes
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Endoscopes (AREA)
- Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
Abstract
Description
技术领域technical field
本发明公开了一种多光谱内窥镜装置及图像融合方法，尤其涉及一种基于双目立体视觉的多光谱内窥镜装置及图像融合方法，属于医疗器械领域。The invention discloses a multispectral endoscope device and an image fusion method, in particular to a multispectral endoscope device and an image fusion method based on binocular stereo vision, and belongs to the field of medical equipment.
背景技术Background technique
内窥镜是一种光学仪器，由冷光源镜头、纤维光导线、图像传输系统、屏幕显示系统等组成，能够扩大手术视野。内窥镜的突出优点是操作灵活简便、手术切口小、术后反应轻、能提高医生的诊疗能力，因而在临床诊断、治疗以及对发病机制、病理变化的监测等方面获得了广泛应用。Endoscope is an optical instrument composed of cold light source lens, fiber optic wire, image transmission system, screen display system, etc., which can expand the surgical field of view. The outstanding advantages of endoscope are flexible and easy operation, small surgical incision, mild postoperative reaction, and can improve the doctor's ability to diagnose and treat. Therefore, it has been widely used in clinical diagnosis, treatment, and monitoring of pathogenesis and pathological changes.
传统内窥镜观察到的是组织器官水平的结构图像，不能实现体内血管、淋巴管、肿瘤等的功能显像，降低了手术的精确度。而现有的多光谱内窥镜通过在术前向病人体内注射靶向或非靶向的光学分子显影剂，能够在手术中采集并显示反映解剖学结构信息的彩色图像及标记肿瘤、血管、淋巴管等的光学分子影像近红外图像。然而现有的多光谱内窥镜仅显示二维图像，与手术过程中实际的解剖学三维结构存在偏差，因此无法准确判断病灶位置，容易导致医生在手术中出现失误。目前还没有能够解决上述问题的有效方法。Traditional endoscopes observe structural images at the level of tissues and organs, and cannot realize functional imaging of blood vessels, lymphatic vessels, tumors, etc. in the body, which reduces the accuracy of surgery. However, the existing multispectral endoscope can collect and display color images reflecting anatomical structure information and mark tumors, blood vessels, Optical molecular imaging near-infrared images of lymphatic vessels, etc. However, the existing multispectral endoscopes only display two-dimensional images, which deviate from the actual anatomical three-dimensional structure during the operation. Therefore, it is impossible to accurately determine the location of the lesion, which may easily lead to doctors making mistakes during the operation. At present, there is no effective method capable of solving the above problems.
发明内容Contents of the invention
本发明所要解决的问题是克服传统多光谱内窥镜工作过程中获取的二维平面图像无法反映立体信息的不足，提供一种多光谱立体视觉内窥镜装置及图像融合方法，使获取到的图像反映的信息更加全面，从而提高医生进行手术的精确度和准确度，减少失误的发生。The problem to be solved by the present invention is to overcome the deficiency that the two-dimensional plane images obtained in the working process of the traditional multispectral endoscope cannot reflect the stereoscopic information, and provide a multispectral stereoscopic vision endoscope device and image fusion method, so that the obtained The information reflected in the image is more comprehensive, thereby improving the precision and accuracy of the doctor's operation and reducing the occurrence of mistakes.
人在观察物体时，能够在感知物体形状的同时感知物体与自己的距离及物体间的相对位置关系，然而现有的2D显示器显示时会丢失物体的深度信息。利用3D显示技术可以有效地呈现出具有纵深感的立体图像，克服2D显示丢失三维深度信息的不足。双目立体显示作为3D显示的一种，主要通过光学等技术模拟实现人眼立体视觉特性，将空间物体以立体信息方式再现。该技术能够帮助医生判断病灶在空间中的位置及不同组织间的相对位置。When people observe an object, they can perceive the distance between the object and themselves and the relative positional relationship between the objects while perceiving the shape of the object. However, the existing 2D display will lose the depth information of the object when it is displayed. The use of 3D display technology can effectively present a stereoscopic image with a sense of depth, and overcome the deficiency of 2D display that loses three-dimensional depth information. As a kind of 3D display, binocular stereoscopic display mainly realizes the stereoscopic vision characteristics of human eyes through optical technology simulation, and reproduces spatial objects in the form of stereoscopic information. This technology can help doctors judge the position of the lesion in space and the relative position of different tissues.
所述内窥镜装置在传统多光谱内窥镜基础上将单路采集相机替换为模拟双目视觉的双路相机，根据内窥镜采集到的两路彩色图像，采用所述图像融合方法与近红外图像分别基于图像金字塔算法进行融合，并将融合后的双路图像经视差图像移位法处理后输入立体视觉显示装置中通过偏光眼镜进行观察，从而还原出真实的解剖学三维结构。The endoscope device replaces the single-channel acquisition camera with a dual-channel camera simulating binocular vision on the basis of a traditional multispectral endoscope, and uses the image fusion method and The near-infrared images are fused based on the image pyramid algorithm, and the fused two-way images are processed by the parallax image shift method and then input into the stereoscopic display device for observation through polarized glasses, thereby restoring the real three-dimensional anatomical structure.
由多光谱内窥镜的原理可知，多光谱内窥镜获取的彩色光图像反映目标的结构信息，而近红外图像反映其功能信息，提取近红外图像的细节信息并与彩色光图像进行融合，得到的融合图像可以同时反映目标的结构信息及功能信息，提高了图像的辨识度。首先，采用双通道相机结构模拟人眼观看空间场景，从两个位于同一水平线上的拍摄点拍摄同一景物，得到两幅具有视差信息的视差图像，平行式立体相机拍摄获得的视差图像没有梯形失真及垂直视差，整个场景只有负水平视差；其次，分别处理得到两个通道的融合图像，通过空分的方法将两幅图像显示在偏光显示屏上；最后，采用偏光眼镜等技术使观看者左、右眼分别看到原先从左右拍摄点拍摄得到的左右视差图像，从而实现立体显示，大大提高显示图像与真实解剖学结构的契合度。According to the principle of multi-spectral endoscope, the colored light image acquired by multi-spectral endoscope reflects the structural information of the target, while the near-infrared image reflects its functional information, and the detailed information of the near-infrared image is extracted and fused with the colored light image. The obtained fused image can reflect the structural information and functional information of the target at the same time, which improves the recognition of the image. First, a dual-channel camera structure is used to simulate the human eye watching the space scene, and the same scene is photographed from two shooting points on the same horizontal line to obtain two parallax images with parallax information. The parallax images obtained by parallel stereo cameras have no trapezoidal distortion and vertical parallax, the entire scene has only negative horizontal parallax; secondly, the fused images of the two channels are processed separately, and the two images are displayed on the polarized display screen through the method of space division; finally, polarized glasses and other technologies are used to make the viewer left The left and right eyes respectively see the left and right parallax images obtained from the left and right shooting points, thereby achieving stereoscopic display and greatly improving the fit between the displayed image and the real anatomical structure.
基于上述思路，本发明提出一种多光谱立体视觉内窥镜装置，其包括：Based on the above thinking, the present invention proposes a multispectral stereoscopic vision endoscope device, which includes:
光源部，提供可见光和激发光，由白色光源、激发光源和聚光透镜构成，所述聚光透镜使来自所述白色光源的照明光以及所述激发光源的激发光会聚到光纤的入射端面；The light source part provides visible light and excitation light, and is composed of a white light source, an excitation light source and a condenser lens, and the condenser lens converges the illumination light from the white light source and the excitation light from the excitation light source to the incident end face of the optical fiber;
摄像部，其中摄像部具有：用于引导由所述光源部会聚的光的所述光纤；使通过所述光纤而被引导至前端的光扩散并照射到观察对象的照明透镜；an imaging unit, wherein the imaging unit has: the optical fiber for guiding the light condensed by the light source unit; an illumination lens for diffusing the light guided to the front end through the optical fiber and irradiating the observation object;
以及用于检测会聚的成像光的摄像单元，所述摄像单元具有两个用于接收会聚的可见光的成像光的摄像元件和两个用于接收会聚的近红外光的摄像元件；And an imaging unit for detecting convergent imaging light, the imaging unit has two imaging elements for receiving convergent visible light imaging light and two imaging elements for receiving convergent near-infrared light;
图像处理部，其包括：图像取得部，其读取并存储由所述摄像单元获取的图像；an image processing unit including: an image acquisition unit that reads and stores an image acquired by the imaging unit;
图像融合部，其从所述图像取得部获取的两组RGB彩色图像和近红外图像，并从所述RGB彩色图像提取绿色通道图像、红色通道图像和蓝色通道图像，将绿色通道图像和与其对应的近红外图像分别构建各自的高斯金字塔及拉普拉斯金字塔，根据所述高斯金字塔及拉普拉斯金字塔将所述绿色通道图像与所述近红外图像融合重构出融合后的绿色通道图像，将所述融合后的绿色通道图像与所述红色通道图像和所述蓝色通道图像合并处理后得到两组具有视差信息且融合了彩色图像与近红外图像细节信息的融合后的彩色图像；an image fusion unit, which obtains two sets of RGB color images and near-infrared images from the image acquisition unit, and extracts a green channel image, a red channel image and a blue channel image from the RGB color images, and combines the green channel image with the The corresponding near-infrared images construct their respective Gaussian pyramids and Laplacian pyramids, and according to the Gaussian pyramids and Laplacian pyramids, the green channel image is fused with the near-infrared image to reconstruct a fused green channel image, combining the fused green channel image with the red channel image and the blue channel image to obtain two sets of fused color images with disparity information and fused color image and near-infrared image detail information ;
立体图像生成部，其利用视差图像移位法处理两组具有视差信息的所述融合后的彩色图像，生成立体图像对；a stereoscopic image generating unit, which uses a parallax image shift method to process two sets of fused color images having parallax information to generate a stereoscopic image pair;
显示部，其将所述立体图像生成部生成的立体图像对显示为以立体图像。The display unit displays the stereoscopic image pair generated by the stereoscopic image generation unit as a stereoscopic image.
所述图像融合部包括图像分离单元，其将从所述图像取得部获取两个用于接收会聚的可见光的成像光的摄像元件中的一个的RGB彩色图像和两个用于接收会聚的近红外光的摄像元件中的一个的近红外图像，将彩色图像进行通道分离得到红色、绿色、蓝色三个通道的图像，选取其中绿色通道图像；金字塔构建单元，其将从所述图像分离单元获取的所述近红外图像和所述绿色通道图像作为底层图像，分别向下采样构建二者各自的高斯金字塔及拉普拉斯金字塔；The image fusion section includes an image separation unit that acquires from the image acquisition section an RGB color image of one of the imaging elements for receiving converged visible light imaging light and two images for receiving convergent near-infrared light. The near-infrared image of one of the imaging elements of the light, the channel separation of the color image is carried out to obtain the images of the three channels of red, green and blue, and the image of the green channel is selected; the pyramid construction unit will be obtained from the image separation unit The near-infrared image and the green channel image are used as the underlying image, and the Gaussian pyramid and the Laplacian pyramid are constructed by down-sampling respectively;
金字塔更新单元，其分别比较所述金字塔构建单元构建的近红外图像和绿色通道图像除顶层外的每一层拉普拉斯金字塔每个像素点的数值大小并取较大值保存为新的拉普拉斯金字塔；Pyramid update unit, which compares the near-infrared image constructed by said pyramid construction unit and the value of each pixel of the green channel image except the top layer of the Laplacian pyramid and takes the larger value and saves it as a new Laplacian Pyramid of Plath;
顶层金字塔处理单元，其将所述金字塔构建单元构建的近红外图像顶层的高斯金字塔图像进行边界提取及孔洞填充，再与原来的顶层近红外图像相乘得到去除背景的近红外图像；A top-level pyramid processing unit, which performs boundary extraction and hole filling on the Gaussian pyramid image of the near-infrared image top layer constructed by the pyramid construction unit, and then multiplies the original top-level near-infrared image to obtain a near-infrared image that removes the background;
顶层金字塔更新单元，其分别比较所述金字塔构建单元构建的绿色通道图像的顶层与所述金字塔处理单元获取的去除背景的近红外图像的每个像素点的值并取较大值保存为新的绿色通道图像的顶层高斯金字塔；The top layer pyramid update unit, which compares the value of each pixel of the top layer of the green channel image constructed by the pyramid construction unit with the background-removed near-infrared image acquired by the pyramid processing unit and takes a larger value and saves it as a new The top level Gaussian pyramid of the green channel image;
图像重构单元，其将顶层金字塔更新单元根据取得的融合后的顶层高斯金字塔图像向上采样，然后与该层的拉普拉斯金字塔相加继续向上采样，如此循环直至重构出底层绿色通道图像；The image reconstruction unit, which up-samples the top-level pyramid update unit according to the obtained fused top-level Gaussian pyramid image, and then adds it to the Laplacian pyramid of this layer to continue up-sampling, and so on until the bottom green channel image is reconstructed ;
图像合并单元，其所述图像重构单元重构出的底层绿色通道图像与所述图像分离单元分离出的红色及蓝色通道的图像进行通道合并，得到最终融合后的彩色图像。The image merging unit performs channel merging on the bottom layer green channel image reconstructed by the image reconstruction unit and the red and blue channel images separated by the image separation unit to obtain a final fused color image.
所述摄像单元还包括：两个物镜，所述物镜会聚从观察对象返回的反射光；两个分光镜，该分光镜投射近红外光，反射可见光。The imaging unit further includes: two objective lenses that converge reflected light returned from the observed object; and two beam splitters that project near-infrared light and reflect visible light.
所述摄像单元还包括：两个物镜，所述物镜会聚从观察对象返回的反射光；两个分光镜，该分光镜投射近红外光，反射可见光。The imaging unit further includes: two objective lenses that converge reflected light returned from the observed object; and two beam splitters that project near-infrared light and reflect visible light.
本发明一种图像处理装置，包括：An image processing device of the present invention, comprising:
图像取得部，其读取并存储由所述摄像单元获取的图像；an image acquisition unit that reads and stores an image acquired by the imaging unit;
图像融合部，其从所述图像取得部获取的两组RGB彩色图像和近红外图像，并从所述RGB彩色图像提取绿色通道图像、红色通道图像和蓝色通道图像，将绿色通道图像和与其对应的近红外图像分别构建各自的高斯金字塔及拉普拉斯金字塔，根据所述高斯金字塔及拉普拉斯金字塔将所述绿色通道图像与所述近红外图像融合重构出融合后的绿色通道图像，将所述融合后的绿色通道图像与所述红色通道图像和所述蓝色通道图像合并处理后得到两组具有视差信息且融合了彩色图像与近红外图像细节信息的融合后的彩色图像，其包括；an image fusion unit, which obtains two sets of RGB color images and near-infrared images from the image acquisition unit, and extracts a green channel image, a red channel image and a blue channel image from the RGB color images, and combines the green channel image with the The corresponding near-infrared images construct their respective Gaussian pyramids and Laplacian pyramids, and according to the Gaussian pyramids and Laplacian pyramids, the green channel image is fused with the near-infrared image to reconstruct a fused green channel image, combining the fused green channel image with the red channel image and the blue channel image to obtain two sets of fused color images with disparity information and fused color image and near-infrared image detail information , which includes;
图像分离单元，其将从所述图像取得部获取两个用于接收会聚的可见光的成像光的摄像元件中的一个的RGB彩色图像和两个用于接收会聚的近红外光的摄像元件中的一个的近红外图像，将彩色图像进行通道分离得到红色、绿色、蓝色三个通道的图像，选取其中绿色通道图像；an image separation unit that acquires an RGB color image of one of the two imaging elements for receiving condensed visible light imaging light and an RGB color image of one of the two imaging elements for receiving condensed near-infrared light from the image acquisition unit A near-infrared image, the channel separation of the color image to obtain images of three channels of red, green, and blue, and select the image of the green channel;
金字塔构建单元，其将从所述图像分离单元获取的所述近红外图像和所述绿色通道图像作为底层图像，分别向下采样构建二者各自的高斯金字塔及拉普拉斯金字塔；A pyramid construction unit, which uses the near-infrared image and the green channel image obtained from the image separation unit as the underlying image, and down-samples to construct their respective Gaussian pyramids and Laplacian pyramids;
金字塔更新单元，其分别比较所述金字塔构建单元构建的近红外图像和绿色通道图像除顶层外的每一层拉普拉斯金字塔每个像素点的数值大小并取较大值保存为新的拉普拉斯金字塔；Pyramid update unit, which compares the near-infrared image constructed by said pyramid construction unit and the value of each pixel of the green channel image except the top layer of the Laplacian pyramid and takes the larger value and saves it as a new Laplacian Pyramid of Plath;
顶层金字塔处理单元，其将所述金字塔构建单元构建的近红外图像顶层的高斯金字塔图像进行边界提取及孔洞填充，再与原来的顶层近红外图像相乘得到去除背景的近红外图像；A top-level pyramid processing unit, which performs boundary extraction and hole filling on the Gaussian pyramid image of the near-infrared image top layer constructed by the pyramid construction unit, and then multiplies the original top-level near-infrared image to obtain a near-infrared image that removes the background;
顶层金字塔更新单元，其分别比较所述金字塔构建单元构建的绿色通道图像的顶层与所述金字塔处理单元获取的去除背景的近红外图像的每个像素点的值并取较大值保存为新的绿色通道图像的顶层高斯金字塔；The top layer pyramid update unit, which compares the value of each pixel of the top layer of the green channel image constructed by the pyramid construction unit with the background-removed near-infrared image acquired by the pyramid processing unit and takes a larger value and saves it as a new The top level Gaussian pyramid of the green channel image;
图像重构单元，其将顶层金字塔更新单元根据取得的融合后的顶层高斯金字塔图像向上采样，然后与该层的拉普拉斯金字塔相加继续向上采样，如此循环直至重构出底层绿色通道图像；The image reconstruction unit, which up-samples the top-level pyramid update unit according to the obtained fused top-level Gaussian pyramid image, and then adds it to the Laplacian pyramid of this layer to continue up-sampling, and so on until the bottom green channel image is reconstructed ;
图像合并单元，其所述图像重构单元重构出的底层绿色通道图像与所述图像分离单元分离出的红色及蓝色通道的图像进行通道合并，得到最终融合后的彩色图像；An image merging unit, the bottom layer green channel image reconstructed by the image reconstruction unit and the images of the red and blue channels separated by the image separation unit are channel merged to obtain a final fused color image;
立体图像生成部，其利用视差图像移位法处理两组具有视差信息的所述融合后的彩色图像，生成立体图像对。The stereoscopic image generating unit processes two sets of fused color images with parallax information by using a parallax image shifting method to generate a stereoscopic image pair.
本发明还提供了一种多光谱立体图像融合方法，其包括以下步骤：The present invention also provides a multi-spectral stereoscopic image fusion method, which includes the following steps:
步骤1：从图像获取设备获取的一组近红外图像和RGB彩色图像，其中，所述一组近红外图像和RGB彩色图像包括一帧RGB彩色图像和与其同一光通道、同一时刻获取的一帧近红外图像；将所述RGB彩色图像三个通道的图像分离得到其绿色通道图像、红色通道图像和蓝色通道图像；Step 1: A group of near-infrared images and RGB color images obtained from the image acquisition device, wherein the group of near-infrared images and RGB color images includes a frame of RGB color images and a frame acquired at the same optical channel and at the same time Near-infrared image; separating the images of the three channels of the RGB color image to obtain its green channel image, red channel image and blue channel image;
步骤2：将所述近红外图像及所述绿色通道图像作为底层图像，分别向下采样构建二者各自的高斯金字塔及拉普拉斯金字塔；Step 2: using the near-infrared image and the green channel image as the underlying image, respectively down-sampling to construct their respective Gaussian pyramids and Laplacian pyramids;
步骤3：对于所述近红外图像和所述绿色图像除顶层外的每一层拉普拉斯金字塔，分别比较两种图像每个像素点的数值大小并取较大值保存为新的拉普拉斯金字塔；Step 3: For each layer of the Laplacian pyramid except the top layer of the near-infrared image and the green image, compare the numerical values of each pixel of the two images and take the larger value and save it as a new Laplacian Pyramid of Las;
步骤4：对近红外图像顶层的高斯金字塔图像进行边界提取及孔洞填充，再与原来的顶层近红外图像相乘得到去除背景的近红外图像；Step 4: Perform boundary extraction and hole filling on the Gaussian pyramid image on the top layer of the near-infrared image, and then multiply it with the original top-layer near-infrared image to obtain a near-infrared image with the background removed;
步骤5：分别比较顶层的绿色通道图像与去除背景的近红外图像的每个像素点的值并取较大值保存为新的顶层高斯金字塔；Step 5: Compare the value of each pixel of the top-level green channel image and the background-removed near-infrared image and take the larger value and save it as a new top-level Gaussian pyramid;
步骤6：将步骤5取得的顶层高斯金字塔图像向上采样，然后与该层的拉普拉斯金字塔相加继续向上采样，如此循环直至重构出底层绿色通道图像；Step 6: Up-sample the top-level Gaussian pyramid image obtained in step 5, and then add it to the Laplacian pyramid of this layer to continue up-sampling, and so on until the bottom-level green channel image is reconstructed;
步骤7：将步骤6得到的底层绿色通道图像与步骤1分离出的红色及蓝色通道的图像进行通道合并，得到第一融合后的彩色图像；Step 7: The bottom layer green channel image obtained in step 6 is combined with the images of the red and blue channels separated in step 1 to obtain the first fused color image;
步骤8：从图像获取设备获取的另一组近红外图像和RGB彩色图像，重复步骤1-7，获取第二融合后的彩色图像，利用视差图像移位法处理两组具有视差信息的所述第一融合后的彩色图像和所述第二融合后的彩色图像，生成立体图像对，其中所述另一组近红外图像和RGB彩色图像包括一帧RGB彩色图像和与其同一光通道、同一时刻获取的一帧近红外图像，其中此步骤获取的一帧RGB彩色图像与步骤1获取的一帧RGB彩色图像获取的时刻相同，获取的光通道不同。Step 8: Another group of near-infrared images and RGB color images obtained from the image acquisition device, repeat steps 1-7, obtain the second fused color image, and use the parallax image shift method to process the two groups of images with parallax information The first fused color image and the second fused color image generate a stereoscopic image pair, wherein the other group of near-infrared images and RGB color images include a frame of RGB color image and the same optical channel and at the same time A frame of near-infrared image obtained, wherein the frame of RGB color image obtained in this step is obtained at the same time as the frame of RGB color image obtained in step 1, and the light channels obtained are different.
有益效果Beneficial effect
本发明采用以上技术方案与现有技术相比，具有以下技术效果：Compared with the prior art, the present invention adopts the above technical scheme and has the following technical effects:
本发明利用图像金字塔算法的在将原图向下采样构建高斯金字塔时会丢失部分高频细节信息的特点，建立包含丢失的高频信息的拉普拉斯金字塔，通过比较处理每一层绿色通道图像和近红外图像的拉普拉斯金字塔可以放大并融合细节信息，同时，选择RGB彩色图像的绿色通道图像进行融合具有以下优点：The present invention utilizes the feature of the image pyramid algorithm that some high-frequency detail information will be lost when the original image is down-sampled to construct a Gaussian pyramid, and a Laplacian pyramid containing the lost high-frequency information is established, and each layer of green channel is processed by comparison The Laplacian pyramid of the image and the near-infrared image can amplify and fuse the detail information. At the same time, choosing the green channel image of the RGB color image for fusion has the following advantages:
首先，由于在临床手术时需要打开创口构建内窥镜通路，因此不可避免会出现出血情况，并且正常情况下生物体体内器官大多呈红色，因此采用RGB彩色图像的红色通道图像进行融合时必然会受到血液等因素干扰；其次，人眼对于RGB彩色图像中的蓝色通道的图像不敏感，导致在观察时不易觉察蓝色通道信息的变化，因而蓝色通道图像亦不适于融合图像。综合考虑，最终选取绿色通道图像与近红外图像进行融合，进而重建出融合了彩色光图像及近红外图像的具有更突出的细节信息的融合图像。采用这种方式分别处理代表左眼与右眼的双通道相机采集的图像，并通过视差图像移位法处理融合图像即可得到具有立体效应的图像对。使用该方法得到的图像对输入立体显示装置后可通过偏光眼镜观察到具有三维结构的立体图像，提供真实的三维结构信息，有助于提高医生手术精度和准确度。First of all, due to the need to open the wound to build the endoscopic access during clinical operations, bleeding will inevitably occur, and under normal circumstances, most of the organs in the body are red, so when the red channel image of the RGB color image is used for fusion, there will inevitably be bleeding. It is interfered by factors such as blood; secondly, the human eye is not sensitive to the image of the blue channel in the RGB color image, which makes it difficult to detect the change of the blue channel information during observation, so the blue channel image is not suitable for fusion images. After comprehensive consideration, the green channel image and the near-infrared image are finally selected for fusion, and then the fused image with more prominent detail information is reconstructed by fusing the color light image and the near-infrared image. In this way, the images collected by the dual-channel cameras representing the left eye and the right eye are respectively processed, and the fused images are processed by the parallax image shift method to obtain an image pair with a stereoscopic effect. After the image pair obtained by using the method is input into a stereoscopic display device, a stereoscopic image with a three-dimensional structure can be observed through polarized glasses, providing real three-dimensional structural information, and helping to improve the precision and accuracy of doctors' operations.
附图说明Description of drawings
图1是本发明多光谱立体视觉内窥镜装置的系统结构示意图；Fig. 1 is a schematic diagram of the system structure of the multi-spectral stereo vision endoscope device of the present invention;
图2是本发明多光谱立体视觉内窥镜装置的摄像单元；Fig. 2 is the imaging unit of the multispectral stereo vision endoscope device of the present invention;
图3是本发明多光谱立体视觉内窥镜装置的图像处理部的结构图；Fig. 3 is a structural diagram of the image processing unit of the multispectral stereoscopic vision endoscope device of the present invention;
图4是本发明图像融合方法的流程图；Fig. 4 is a flowchart of the image fusion method of the present invention;
图5单通道相机采集的近红外图像；Figure 5 is a near-infrared image collected by a single-channel camera;
图6单通道相机采集的彩色光通道分离后的绿色通道图像；Figure 6 is the green channel image collected by a single-channel camera after color light channel separation;
图7是将近红外图像作为底层图像构建的层数为4的高斯金字塔图像。Figure 7 is a Gaussian pyramid image with 4 layers constructed using the near-infrared image as the underlying image.
具体实施方式Detailed ways
下面结合附图对本发明进行进一步详细描述，应指出的是，所描述的实施例仅旨在便于对本发明的理解，而不对其起任何限定作用。The present invention will be further described in detail below in conjunction with the accompanying drawings. It should be noted that the described embodiments are only intended to facilitate the understanding of the present invention, and do not limit it in any way.
本实施方式为多通道内窥镜，优选为双通道内窥镜，图1是示出该实施方式的多光谱立体视觉内窥镜装置的整体结构的框图。构成本发明的多光谱立体视觉内窥镜装置由光源部100、摄像部200、图像处理部300、显示部400构成。This embodiment is a multi-channel endoscope, preferably a dual-channel endoscope, and FIG. 1 is a block diagram showing the overall structure of the multi-spectral stereoscopic endoscope device of this embodiment. The multi-spectral stereoscopic endoscope device constituting the present invention is composed of a light source unit 100 , an imaging unit 200 , an image processing unit 300 , and a display unit 400 .
光源部100由白色光源101和激发光源102、聚光透镜103构成，该聚光透镜103使来自白色光源101的照明光以及激发光源102的激发光会聚到光纤201的入射端面。The light source unit 100 is composed of a white light source 101 , an excitation light source 102 , and a condensing lens 103 that condenses the illumination light from the white light source 101 and the excitation light from the excitation light source 102 onto the incident end face of the optical fiber 201 .
摄像部200为了能插入体腔而形成为细长且可以弯曲的结构。摄像部200具有：用于引导由光源部100会聚的光的光纤201 ；使通过该光纤201而被引导至前端的光扩散并照射到观察对象的照明透镜202；用于检测会聚的成像光的摄像单元。The imaging unit 200 is formed in an elongated and bendable structure so that it can be inserted into a body cavity. The imaging unit 200 has: an optical fiber 201 for guiding the light condensed by the light source unit 100; an illumination lens 202 for diffusing the light guided to the tip through the optical fiber 201 and irradiating the observation object; camera unit.
结合图2来说明本发明的多光谱立体视觉内窥镜装置的摄像单元。摄像单元包括两个会聚从观察对象返回的反射光的物镜203；两个分光镜211，该分光镜投射近红外光，反射可见光；两个用于接收并检测会聚的可见光的成像光的摄像元件209，其彩色相机光谱相应范围为400-700nm；两个用于接收会聚的近红外光的摄像元件210，其近红外相机光谱响应范围为700-900nm，设置在其前面的滤光片光谱范围为830-850nm。白色光源101发出的可见光照射到观察对象后由观察对象反射的返射光同时分别由两个物镜203中一个会聚并分别传输到两个分光镜211中的一个，分光镜211将该反射光反射到两个用于检测会聚的可见光的成像光的摄像元件209，该摄像元件209为CCD图像传感器；由激发光源102发出的激发光照射到观察对象后产生的近红外光也同时分别由两个物镜203中一个会聚并分别传输到两个分光镜211中的一个，分光镜211将该近红外光透射到用于接收会聚的近红外光的摄像元件210，该摄像元件210为CCD图像传感器。白色光源101和激发光源102同时发光，摄像元件209和210同时接收光。The imaging unit of the multi-spectral stereo vision endoscope device of the present invention will be described with reference to FIG. 2 . The imaging unit includes two objective lenses 203 that condense reflected light returned from the observation object; two beam splitters 211 that project near-infrared light and reflect visible light; and two imaging elements for receiving and detecting the converged visible light imaging light 209, the corresponding range of the color camera spectrum is 400-700nm; two imaging elements 210 for receiving converging near-infrared light, the spectral response range of the near-infrared camera is 700-900nm, and the spectral range of the filter set in front of it is 830-850nm. After the visible light emitted by the white light source 101 irradiates the observed object, the reflected light reflected by the observed object is simultaneously converged by one of the two objective lenses 203 and transmitted to one of the two beam splitters 211 respectively, and the beam splitter 211 reflects the reflected light to Two imaging elements 209 for detecting the imaging light of the converged visible light, the imaging element 209 is a CCD image sensor; the near-infrared light generated after the excitation light emitted by the excitation light source 102 irradiates the observation object is also simultaneously captured by the two objective lenses One of 203 converges and transmits to one of the two beam splitters 211 respectively, and the beam splitter 211 transmits the near-infrared light to the imaging element 210 for receiving the converged near-infrared light, and the imaging element 210 is a CCD image sensor. The white light source 101 and the excitation light source 102 emit light at the same time, and the imaging elements 209 and 210 receive light at the same time.
图3示出了图像处理部300具备A/D转换部310、图像取得部320、图像融合部330、立体图像生成部340和控制部350。A/D转换部310将来自摄像元件 209和210的经过光电转换后的模拟信号转换为数字信号，并输出至图像取得部320，图像取得部320读取并存储由摄像单元获取的图像。FIG. 3 shows that the image processing unit 300 includes an A/D conversion unit 310 , an image acquisition unit 320 , an image fusion unit 330 , a stereoscopic image generation unit 340 and a control unit 350 . The A/D conversion unit 310 converts the photoelectrically converted analog signals from the imaging elements 209 and 210 into digital signals, and outputs them to the image acquisition unit 320, and the image acquisition unit 320 reads and stores the images acquired by the imaging unit.
控制部350 由CPU等硬件实现，将其与A/D转换部310、图像取得部320、图像融合部330和立体图像生成部340相连，并生成控制它们的控制信号，并且控制部统一控制图像处理部整体的动作。The control unit 350 is realized by hardware such as a CPU, and is connected to the A/D conversion unit 310, the image acquisition unit 320, the image fusion unit 330, and the stereoscopic image generation unit 340, and generates control signals for controlling them, and the control unit uniformly controls the image The overall operation of the processing unit.
图像融合部330进行可见光与近红外图像的融合处理，图像融合部330从图像取得部320获取采用图2所示的双通道相机结构可获取代表左右眼视觉的图像，分别处理得到两个通道的融合图像后，经过双目视觉相关算法的处理可以将融合图像并通过后续的立体图像生成部340生成三维图像将图像以三维形式展现在显示部400上，大大提高显示图像与真实解剖学结构的契合度。The image fusion unit 330 performs fusion processing of visible light and near-infrared images. The image fusion unit 330 obtains from the image acquisition unit 320 the images representing the vision of the left and right eyes obtained by using the dual-channel camera structure shown in FIG. After the fused images, the fused images can be processed by the binocular vision correlation algorithm to generate a three-dimensional image through the subsequent stereo image generation unit 340, and the image can be displayed on the display unit 400 in three-dimensional form, which greatly improves the relationship between the displayed image and the real anatomical structure. compatibility.
图像融合部330包括分离绿色通道图像的图像分离单元3301、构建高斯金字塔及拉普拉斯金字塔的金字塔构建单元3302、更新除顶层外的每一层拉普拉斯金字塔的金字塔更新单元3303、处理和更新顶层高斯金字塔的顶层金字塔处理单元3304和顶层金字塔更新单元3305、重构出底层绿色通道图像的图像重构单元3306和得到最终的融合图像的图像合并单元3307。The image fusion part 330 includes an image separation unit 3301 for separating the green channel image, a pyramid construction unit 3302 for constructing a Gaussian pyramid and a Laplacian pyramid, a pyramid update unit 3303 for updating each layer of the Laplacian pyramid except the top layer, processing And the top pyramid processing unit 3304 and the top pyramid update unit 3305 for updating the top Gaussian pyramid, the image reconstruction unit 3306 for reconstructing the bottom green channel image, and the image merging unit 3307 for obtaining the final fused image.
接着说明图像融合部330的动作，图4是图像融合部330的动作流程图。Next, the operation of the image fusion unit 330 will be described. FIG. 4 is a flowchart of the operation of the image fusion unit 330 .
首先，图像分离单元3301将从图像取得部320获取两个摄像元件209中的一个获取的RGB彩色图像和与所述两个摄像元件209中的所述一个位于同一光通道的两个摄像元件210中的一个在同一时刻获取的近红外图像（如图5所示），将彩色图像进行通道分离得到红色、绿色、蓝色三个通道的图像，选取其中绿色通道图像，如图6所示。First, the image separation unit 3301 acquires the RGB color image acquired by one of the two imaging elements 209 and the two imaging elements 210 located in the same optical channel as the one of the two imaging elements 209 from the image acquisition unit 320 One of the near-infrared images acquired at the same time (as shown in Figure 5), the color image is separated to obtain images of three channels of red, green, and blue, and the green channel image is selected, as shown in Figure 6.
其次，金字塔构建单元3302将从图像分离单元3301获取的近红外图像和绿色通道图像作为底层图像，分别向下采样构建二者各自的高斯金字塔及拉普拉斯金字塔。其中，在构建高斯金字塔过程中，为了得到层级为 QUOTE 的金字塔图像，需要进行向下采样，即：Secondly, the pyramid construction unit 3302 uses the near-infrared image and the green channel image acquired from the image separation unit 3301 as the underlying image, and down-samples to construct their respective Gaussian pyramids and Laplacian pyramids. Among them, in the process of building the Gaussian pyramid, in order to get the level of QUOTE The pyramid image needs to be down-sampled, namely:
（1）对上层图像 QUOTE 进行高斯内核卷积；(1) QUOTE to the upper image Perform Gaussian kernel convolution;
（2）去除所有偶数行和偶数列；(2) Remove all even-numbered rows and even-numbered columns;
得到 QUOTE 后重复此过程即可构建出完整的高斯金字塔，最终构建的近红外图像的高斯金字塔，如图7所示。get QUOTE After repeating this process, a complete Gaussian pyramid can be constructed, and the Gaussian pyramid of the near-infrared image is finally constructed, as shown in FIG. 7 .
在构建拉普拉斯金字塔时有如下公式：There are the following formulas when constructing the Laplacian pyramid:
QUOTE =QUOTE –pyrUP（ QUOTE ）QUOTE =QUOTE –pyrUP(QUOTE )
其中pyrUP()是对图像进行向上采样，即：Among them, pyrUP() is to upsample the image, namely:
①将第i+1层图像位置为（x,y）的点映射到第i层图像的（2x+1,2y+1）位置；① Map the point where the i+1 layer image position is (x, y) to the (2x+1, 2y+1) position of the i layer image;
②与相同的高斯内核*4卷积；②Convolution with the same Gaussian kernel*4;
重复此过程即可得到完整的拉普拉斯金字塔。Repeat this process to get the complete Laplacian pyramid.
接下来，金字塔更新单元3303分别比较金字塔构建单元3302构建的近红外图像和绿色通道图像除顶层外的每一层拉普拉斯金字塔 QUOTE 每个像素点的数值大小并取较大值保存为新的拉普拉斯金字塔。Next, the pyramid update unit 3303 compares the near-infrared image and the green channel image constructed by the pyramid construction unit 3302 with each layer of the Laplacian pyramid except the top layer QUOTE The numerical value of each pixel and take the larger value and save it as a new Laplacian pyramid.
然后，顶层金字塔处理单元3304将金字塔构建单元3302构建的近红外图像顶层的高斯金字塔图像 QUOTE ，通过采用Canny算子提取出图像中边界，利用孔洞填充算法将目标区域边界封闭，将目标区域内像素值置为1，区域外像素值置零，再与近红外图像的顶层原图相乘，得到去除背景影响只包含目标区域的近红外图像 QUOTE 。Then, the top-level pyramid processing unit 3304 QUOTE the Gaussian pyramid image at the top level of the near-infrared image constructed by the pyramid construction unit 3302 , by using the Canny operator to extract the boundary of the image, using the hole filling algorithm to close the boundary of the target area, setting the pixel value in the target area to 1, and setting the value of the pixel outside the area to zero, and then multiplied by the top original image of the near-infrared image , get the near-infrared image that only contains the target area after removing the background influenceQUOTE .
再者，顶层金字塔更新单元3305分别比较由金字塔构建单元3302构建的绿色通道图像的高斯金字塔顶层与金字塔处理单元3304获取的去除背景的近红外图像 QUOTE 的每个像素点的像素值，将 QUOTE 中的像素值替换为两者中较大值，得到融合后的顶层绿色通道图像 QUOTE 。Furthermore, the top pyramid update unit 3305 compares the top layer of the Gaussian pyramid of the green channel image constructed by the pyramid construction unit 3302 with the background-removed near-infrared image QUOTE obtained by the pyramid processing unit 3304. The pixel value of each pixel, will QUOTE The pixel value in is replaced with the larger value of the two to get the fused top-level green channel imageQUOTE .
接着，图像重构单元3306将顶层金字塔更新单元3305获得的融合后的顶层（第n层）绿色通道图像 QUOTE 向上采样，得到模糊后的下一层（第n-1层）图像pyrUP（QUOTE ），再与金字塔更新单元3303更新的该层的拉普拉斯金字塔图像 QUOTE 融合，得到第n-1层的融合图像 QUOTE ，即：Next, the image reconstruction unit 3306 QUOTE the fused top layer (nth layer) green channel image obtained by the top layer pyramid update unit 3305 Sampling up to get the blurred next layer (layer n-1) image pyrUP(QUOTE ), and then update the Laplacian pyramid image QUOTE of this layer with the pyramid update unit 3303 Fusion, get the fusion image of the n-1th layerQUOTE ,which is:
QUOTE =QUOTE + pyrUP（ QUOTE ）QUOTE =QUOTE +pyrUP(QUOTE )
将 QUOTE 作为上层图像重复上述采样及融合过程得到每层融合图像，最终重建出融合后的底层绿色通道图像 QUOTE 。Will QUOTE Repeat the above sampling and fusion process as the upper layer image to obtain the fusion image of each layer, and finally reconstruct the fused bottom layer green channel imageQUOTE .
最后，图像合并单元3307将图像重构单元3306重构出的底层绿色通道图像 QUOTE 与图像分离单元3301分离出的红色及蓝色通道的图像进行通道合并，得到第一融合后的彩色图像。Finally, the image merging unit 3307 reconstructs the underlying green channel image QUOTE from the image reconstruction unit 3306 Channel merge with the images of the red and blue channels separated by the image separation unit 3301 to obtain the first fused color image.
另外，两个摄像元件209中的另一个在同一时刻获取的RGB彩色图像和与上述两个摄像元件209中的另一个位于同一光通道两个摄像元件210中的另一个在同一时刻获取的近红外图像经图像融合部330处理后得到的第二融合后的彩色图像。In addition, the RGB color image acquired by the other of the two imaging elements 209 at the same time and the close-up image acquired by the other of the two imaging elements 210 located in the same optical channel as the other of the above two imaging elements 209 at the same time The infrared image is processed by the image fusion unit 330 to obtain a second fused color image.
立体图像生成部340获取所述第一融合后的彩色图像以及第二融合后的彩色图像。由平行式立体相机拍摄获得的视差图像没有梯形失真及垂直视差，但整个场景只有负水平视差。为此立体图像生成部340通过视差图像移位法来改变视差图像中的视差范围，获得正、负水平视差，通过视差图像移位法处理具有视差信息的所述第一融合后的彩色图像和所述第二融合后的彩色图像后得到立体图像对。The stereoscopic image generator 340 acquires the first fused color image and the second fused color image. The parallax image captured by the parallel stereo camera has no trapezoidal distortion and vertical parallax, but the whole scene has only negative horizontal parallax. For this reason, the stereoscopic image generator 340 changes the parallax range in the parallax image by the parallax image shift method to obtain positive and negative horizontal parallax, and processes the first fused color image and the color image having the parallax information by the parallax image shift method. Stereoscopic image pairs are obtained from the second fused color images.
显示部400为偏光显示屏，其将所述立体图像生成部340生成的立体图像对显示为立体图像，观察者通过佩戴偏光眼镜即可观察到具有立体视觉效应的被观测物体的三维图像。The display unit 400 is a polarized display screen, which displays the stereoscopic image pair generated by the stereoscopic image generating unit 340 as a stereoscopic image, and the observer can observe a three-dimensional image of the observed object with stereoscopic effect by wearing polarized glasses.
另外，本发明提出一种图像融合方法具体包括：In addition, the present invention proposes an image fusion method specifically including:
步骤1：从图像获取设备获取的一组近红外图像和RGB彩色图像，其中，所述一组近红外图像和RGB彩色图像包括一帧RGB彩色图像和与其同一光通道、同一时刻获取的一帧近红外图像；将所述RGB彩色图像三个通道的图像分离得到其绿色通道图像、红色通道图像和蓝色通道图像；Step 1: A group of near-infrared images and RGB color images obtained from the image acquisition device, wherein the group of near-infrared images and RGB color images includes a frame of RGB color images and a frame acquired at the same optical channel and at the same time Near-infrared image; separating the images of the three channels of the RGB color image to obtain its green channel image, red channel image and blue channel image;
步骤2：将所述近红外图像及所述绿色通道图像作为底层图像，分别向下采样构建二者各自的高斯金字塔及拉普拉斯金字塔；Step 2: using the near-infrared image and the green channel image as the underlying image, respectively down-sampling to construct their respective Gaussian pyramids and Laplacian pyramids;
步骤3：对于所述近红外图像和所述绿色图像除顶层外的每一层拉普拉斯金字塔，分别比较两种图像每个像素点的数值大小并取较大值保存为新的拉普拉斯金字塔；Step 3: For each layer of the Laplacian pyramid except the top layer of the near-infrared image and the green image, compare the numerical values of each pixel of the two images and take the larger value and save it as a new Laplacian Pyramid of Las;
步骤4：对近红外图像顶层的高斯金字塔图像进行边界提取及孔洞填充，再与原来的顶层近红外图像相乘得到去除背景的近红外图像；Step 4: Perform boundary extraction and hole filling on the Gaussian pyramid image on the top layer of the near-infrared image, and then multiply it with the original top-layer near-infrared image to obtain a near-infrared image with the background removed;
步骤5：分别比较顶层的绿色通道图像与去除背景的近红外图像的每个像素点的值并取较大值保存为新的顶层高斯金字塔；Step 5: Compare the value of each pixel of the top-level green channel image and the background-removed near-infrared image and take the larger value and save it as a new top-level Gaussian pyramid;
步骤6：将步骤5取得的顶层高斯金字塔图像向上采样，然后与该层的拉普拉斯金字塔相加继续向上采样，如此循环直至重构出底层绿色通道图像；Step 6: Up-sample the top-level Gaussian pyramid image obtained in step 5, and then add it to the Laplacian pyramid of this layer to continue up-sampling, and so on until the bottom-level green channel image is reconstructed;
步骤7：将步骤6得到的底层绿色通道图像与步骤1分离出的红色及蓝色通道的图像进行通道合并，得到第一融合后的彩色图像；Step 7: The bottom layer green channel image obtained in step 6 is combined with the images of the red and blue channels separated in step 1 to obtain the first fused color image;
步骤8：从图像获取设备获取的另一组近红外图像和RGB彩色图像，重复步骤1-7，获取第二融合后的彩色图像，利用视差图像移位法处理两组具有视差信息的所述第一融合后的彩色图像和所述第二融合后的彩色图像，生成立体图像对，其中所述另一组近红外图像和RGB彩色图像包括一帧RGB彩色图像和与其同一光通道、同一时刻获取的一帧近红外图像，其中此步骤获取的一帧RGB彩色图像与步骤1获取的一帧RGB彩色图像获取的时刻相同，获取的光通道不同。Step 8: Another group of near-infrared images and RGB color images obtained from the image acquisition device, repeat steps 1-7, obtain the second fused color image, and use the parallax image shift method to process the two groups of images with parallax information The first fused color image and the second fused color image generate a stereoscopic image pair, wherein the other group of near-infrared images and RGB color images include a frame of RGB color image and the same optical channel and at the same time A frame of near-infrared image obtained, wherein the frame of RGB color image obtained in this step is obtained at the same time as the frame of RGB color image obtained in step 1, and the light channels obtained are different.
以上所述仅是本发明的优选实施方式，应当指出：对于本技术领域的普通技术人员来说，在不脱离本发明原理的前提下，还可以做出若干改进和润饰，这些改进和润饰也应视为本发明的保护范围。The above is only a preferred embodiment of the present invention, it should be pointed out that for those of ordinary skill in the art, without departing from the principle of the present invention, some improvements and modifications can also be made, and these improvements and modifications are also possible. It should be regarded as the protection scope of the present invention.
Claims (6)
- A kind of 1. multispectral 3-D visual endoscope device, it is characterised in that including：Light source portion, there is provided visible ray and exciting light, are made of white light source, excitation source and collector lens, the collector lens The exciting light of the illumination light from the white light source and the excitation source is set to converge to the incident end face of optical fiber；Image pickup part, wherein image pickup part have：For guiding the optical fiber for the light assembled by the light source portion；Make the illuminating lens for spreading and being irradiated to observation object being directed to the light of front end by the optical fiber；AndFor detecting the camera unit for the imaging assembled, there are the camera unit two to be used for the visible ray that reception is assembled The photographing element of imaging and two photographing elements for being used to receive the near infrared light assembled；Image processing part, it includes：Image acquiring section, it reads and stores the image obtained by the camera unit；Image co-registration portion, its two groups of RGB color image obtained from described image obtaining section and near-infrared image, and from described RGB color image extraction green channel images, red channel image and blue channel image, by green channel images and right with it The near-infrared image answered builds respective gaussian pyramid and laplacian pyramid respectively, according to the gaussian pyramid and drawing The green channel images are merged the green channel images after reconstructing fusion by this pyramid of pula with the near-infrared image, By the green channel images after the fusion with being obtained after the red channel image and the blue channel image merging treatment Two groups have parallax information and have merged the coloured image after the merging of coloured image and near-infrared image detailed information；Stereo-picture generating unit, it handles the colour after two groups of fusions with parallax information using anaglyph displacement method Image, generates stereo pairs；Display unit, the stereo pairs that the stereo-picture generating unit generates are shown as with stereo-picture by it.
- 2. multispectral 3-D visual endoscope device according to claim 1, it is characterised in that wrap in described image fusion portion Include：Image separative element, it takes the photograph the imaging that two visible rays for being used to receive convergence are obtained from described image obtaining section The RGB color image of one in element and two are used to receiving the near of one in the photographing element for the near infrared light assembled Infrared image, obtains the image of red, green, blue three passages by coloured image progress channel separation, chooses its Green Channel image；Pyramid construction unit, it is by the near-infrared image obtained from described image separative element and the green channel figure As being used as bottom layer image, sampling separately down builds the two respective gaussian pyramid and laplacian pyramid；Pyramid updating block, it is respectively compared the near-infrared image and green channel images of the pyramid construction cell formation The numerical values recited of each layer of each pixel of laplacian pyramid in addition to top layer simultaneously takes higher value to save as new La Pula This pyramid；Top layer pyramid processing unit, it is by the gaussian pyramid of the near-infrared image top layer of the pyramid construction cell formation Image carries out Boundary Extraction and holes filling, then is multiplied with original top layer near-infrared image to obtain the near-infrared figure for removing background Picture；Top layer pyramid updating block, its be respectively compared the top layer of the green channel images of the pyramid construction cell formation with The value of each pixel of the near-infrared image for the removal background that the pyramid processing unit obtains simultaneously takes higher value to save as The top layer gaussian pyramid of new green channel images；Image reconstruction unit, its by top layer pyramid updating block according to the top layer gaussian pyramid image after the fusion of acquirement to Up-sampling, is then added with the laplacian pyramid of this layer and continues up sampling, and so circulation is until reconstruct bottom green Channel image；Image combining unit, the bottom green channel images that its described image reconfiguration unit reconstructs and described image separative element The red and the image of blue channel isolated merge into row of channels, the coloured image after finally being merged.
- 3. multispectral 3-D visual endoscope device according to claim 1, it is characterised in that the camera unit also wraps Include：Two object lens, the object lens assemble the reflected light that object returns from；Two spectroscopes, spectroscope projection near infrared light, reflect visible ray.
- 4. multispectral 3-D visual endoscope device according to claim 3, it is characterised in that the white light source is sent Radiation of visible light to by returning for observation object reflection penetrating light while respectively by a meeting in described two object lens after observation object Coalescence is respectively transmitted to one in described two spectroscopes, which is used to detect meeting by spectroscope to described two The photographing element of the imaging of poly- visible ray；The exciting light that is sent by the excitation source produces after being irradiated to observation object Near infrared light also while is respectively respectively transmitted to one in described two spectroscopes by a meeting coalescence in described two object lens, Photographing element of the spectroscope by the transmission of near infra red light to the near infrared light for being used to receive convergence.
- A kind of 5. image processing apparatus, it is characterised in that including：Image acquiring section, it reads and stores the image obtained by the camera unit；Image co-registration portion, its two groups of RGB color image obtained from described image obtaining section and near-infrared image, and from described RGB color image extraction green channel images, red channel image and blue channel image, by green channel images and right with it The near-infrared image answered builds respective gaussian pyramid and laplacian pyramid respectively, according to the gaussian pyramid and drawing The green channel images are merged the green channel images after reconstructing fusion by this pyramid of pula with the near-infrared image, By the green channel images after the fusion with being obtained after the red channel image and the blue channel image merging treatment Two groups have parallax information and have merged the coloured image after the merging of coloured image and near-infrared image detailed information, its bag Include；Image separative element, it takes the photograph the imaging that two visible rays for being used to receive convergence are obtained from described image obtaining section The RGB color image of one in element and two are used to receiving the near of one in the photographing element for the near infrared light assembled Infrared image, obtains the image of red, green, blue three passages by coloured image progress channel separation, chooses its Green Channel image；Pyramid construction unit, it is by the near-infrared image obtained from described image separative element and the green channel figure As being used as bottom layer image, sampling separately down builds the two respective gaussian pyramid and laplacian pyramid；Pyramid updating block, it is respectively compared the near-infrared image and green channel images of the pyramid construction cell formation The numerical values recited of each layer of each pixel of laplacian pyramid in addition to top layer simultaneously takes higher value to save as new La Pula This pyramid；Top layer pyramid processing unit, it is by the gaussian pyramid of the near-infrared image top layer of the pyramid construction cell formation Image carries out Boundary Extraction and holes filling, then is multiplied with original top layer near-infrared image to obtain the near-infrared figure for removing background Picture；Top layer pyramid updating block, its be respectively compared the top layer of the green channel images of the pyramid construction cell formation with The value of each pixel of the near-infrared image for the removal background that the pyramid processing unit obtains simultaneously takes higher value to save as The top layer gaussian pyramid of new green channel images；Image reconstruction unit, its by top layer pyramid updating block according to the top layer gaussian pyramid image after the fusion of acquirement to Up-sampling, is then added with the laplacian pyramid of this layer and continues up sampling, and so circulation is until reconstruct bottom green Channel image；Image combining unit, the bottom green channel images that its described image reconfiguration unit reconstructs and described image separative element The red and the image of blue channel isolated merge into row of channels, the coloured image after finally being merged；Stereo-picture generating unit, it handles the colour after two groups of fusions with parallax information using anaglyph displacement method Image, generates stereo pairs.
- 6. a kind of multispectral stereo-picture fusion method, it is characterised in that it comprises the following steps：Step 1：The one group of near-infrared image and RGB color image obtained from image acquisition equipment, wherein, one group of near-infrared Image and RGB color image include a frame RGB color image and the frame near-infrared obtained with its same optical channel, synchronization Image；By its isolated green channel images of image, red channel image and the blueness of three passages of RGB color image Channel image；Step 2：The near-infrared image and the green channel images are built the two as bottom layer image, separately down sampling Respective gaussian pyramid and laplacian pyramid；Step 3：For each layer of laplacian pyramid of the near-infrared image and the green image in addition to top layer, difference Compare the numerical values recited of two kinds of each pixels of image and take higher value to save as new laplacian pyramid；Step 4：Carry out Boundary Extraction and holes filling to the gaussian pyramid image of near-infrared image top layer, then with original top Layer near-infrared image is multiplied to obtain the near-infrared image for removing background；Step 5：It is respectively compared the value of each pixel of near-infrared image of the green channel images of top layer with removing background simultaneously Higher value is taken to save as new top layer gaussian pyramid；Step 6：By the top layer gaussian pyramid image that step 5 obtains to up-sampling, then with the laplacian pyramid of this layer Addition continues up sampling, and so circulation is until reconstruct bottom green channel images；Step 7：Red that bottom green channel images that step 6 obtains are isolated with step 1 and the image of blue channel into Row of channels merges, and obtains the coloured image after the first fusion；Step 8：Another group of near-infrared image and RGB color image obtained from image acquisition equipment, repeat step 1-7, obtains Coloured image after second fusion, after handling two groups of first fusions with parallax information using anaglyph displacement method Coloured image and it is described second fusion after coloured image, generate stereo pairs, wherein another group of near-infrared image and RGB color image includes a frame RGB color image and the frame near-infrared image obtained with its same optical channel, synchronization, The frame RGB color image that wherein this step obtains is identical at the time of acquisition with the frame RGB color image that step 1 obtains, and obtains The optical channel taken is different.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711258979.2A CN108040243B (en) | 2017-12-04 | 2017-12-04 | Multispectral stereoscopic vision endoscope device and image fusion method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711258979.2A CN108040243B (en) | 2017-12-04 | 2017-12-04 | Multispectral stereoscopic vision endoscope device and image fusion method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108040243A true CN108040243A (en) | 2018-05-15 |
| CN108040243B CN108040243B (en) | 2019-06-14 |
Family
ID=62094817
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711258979.2A Active CN108040243B (en) | 2017-12-04 | 2017-12-04 | Multispectral stereoscopic vision endoscope device and image fusion method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108040243B (en) |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109215079A (en) * | 2018-07-17 | 2019-01-15 | 艾瑞迈迪医疗科技（北京）有限公司 | Image processing method, operation navigation device, electronic equipment, storage medium |
| CN109688342A (en) * | 2019-01-11 | 2019-04-26 | 南京理工大学 | A kind of multispectral stereo imaging system |
| CN110602476A (en) * | 2019-08-08 | 2019-12-20 | 南京航空航天大学 | Hole filling method of Gaussian mixture model based on depth information assistance |
| CN110811498A (en) * | 2019-12-19 | 2020-02-21 | 中国科学院长春光学精密机械与物理研究所 | Visible light and near-infrared fluorescence 3D fusion image endoscopy system |
| CN112089404A (en) * | 2020-10-10 | 2020-12-18 | 南京航空航天大学 | Endoscopic OCT-Raman dual-mode imaging device and imaging method |
| CN112102171A (en) * | 2020-09-18 | 2020-12-18 | 贝壳技术有限公司 | Image processing method, image processing device, computer-readable storage medium and electronic equipment |
| CN112089403A (en) * | 2020-10-02 | 2020-12-18 | 深圳市中安视达科技有限公司 | Multispectral medical multi-path imaging method and system thereof |
| CN112184604A (en) * | 2020-09-15 | 2021-01-05 | 杭州电子科技大学 | Color image enhancement method based on image fusion |
| WO2021051222A1 (en) * | 2019-09-16 | 2021-03-25 | 北京数字精准医疗科技有限公司 | Endoscope system, mixed light source, video acquisition device and image processor |
| CN113129243A (en) * | 2021-03-10 | 2021-07-16 | 同济大学 | Blood vessel image enhancement method and system based on infrared and visible light image fusion |
| CN113688707A (en) * | 2021-03-04 | 2021-11-23 | 黑芝麻智能科技(上海)有限公司 | Face anti-spoofing method |
| CN114027765A (en) * | 2020-11-20 | 2022-02-11 | 上海微创医疗机器人（集团）股份有限公司 | Fluorescence endoscope system, control method, and storage medium |
| CN115299914A (en) * | 2022-08-05 | 2022-11-08 | 上海微觅医疗器械有限公司 | Endoscope system, image processing method and device |
| WO2022267645A1 (en) * | 2021-06-21 | 2022-12-29 | 中兴通讯股份有限公司 | Photography apparatus and method, electronic device, and storage medium |
| CN116471466A (en) * | 2023-06-12 | 2023-07-21 | 深圳市博盛医疗科技有限公司 | Dual-wafer endoscope imaging method and imaging device |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100302376A1 (en) * | 2009-05-27 | 2010-12-02 | Pierre Benoit Boulanger | System and method for high-quality real-time foreground/background separation in tele-conferencing using self-registered color/infrared input images and closed-form natural image matting techniques |
| CN102298769A (en) * | 2011-06-11 | 2011-12-28 | 浙江理工大学 | Colored fusion method of night vision low-light image and infrared image based on color transmission |
| CN102789640A (en) * | 2012-07-16 | 2012-11-21 | 中国科学院自动化研究所 | Method for fusing visible light full-color image and infrared remote sensing image |
| CN103300812A (en) * | 2013-06-27 | 2013-09-18 | 中国科学院自动化研究所 | Endoscope-based multispectral video navigation system and method |
| CN103530853A (en) * | 2013-10-17 | 2014-01-22 | 中北大学 | Infrared intensity image and infrared polarization image enhancement and fusion method |
| CN104274148A (en) * | 2014-09-28 | 2015-01-14 | 安徽中科医药成像技术科技有限公司 | Imaging system |
| CN105096285A (en) * | 2014-05-23 | 2015-11-25 | 南京理工大学 | Image fusion and target tracking system based on multi-core DSP |
| CN105447838A (en) * | 2014-08-27 | 2016-03-30 | 北京计算机技术及应用研究所 | Method and system for infrared and low-level-light/visible-light fusion imaging |
| CN106236006A (en) * | 2016-08-31 | 2016-12-21 | 杨晓峰 | 3D optical molecular image peritoneoscope imaging system |
| CN106960428A (en) * | 2016-01-12 | 2017-07-18 | 浙江大立科技股份有限公司 | Visible ray and infrared double-waveband image co-registration Enhancement Method |
| CN107103596A (en) * | 2017-04-27 | 2017-08-29 | 湖南源信光电科技股份有限公司 | A kind of color night vision image interfusion method based on yuv space |
-
2017
- 2017-12-04 CN CN201711258979.2A patent/CN108040243B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100302376A1 (en) * | 2009-05-27 | 2010-12-02 | Pierre Benoit Boulanger | System and method for high-quality real-time foreground/background separation in tele-conferencing using self-registered color/infrared input images and closed-form natural image matting techniques |
| CN102298769A (en) * | 2011-06-11 | 2011-12-28 | 浙江理工大学 | Colored fusion method of night vision low-light image and infrared image based on color transmission |
| CN102789640A (en) * | 2012-07-16 | 2012-11-21 | 中国科学院自动化研究所 | Method for fusing visible light full-color image and infrared remote sensing image |
| CN103300812A (en) * | 2013-06-27 | 2013-09-18 | 中国科学院自动化研究所 | Endoscope-based multispectral video navigation system and method |
| CN103530853A (en) * | 2013-10-17 | 2014-01-22 | 中北大学 | Infrared intensity image and infrared polarization image enhancement and fusion method |
| CN105096285A (en) * | 2014-05-23 | 2015-11-25 | 南京理工大学 | Image fusion and target tracking system based on multi-core DSP |
| CN105447838A (en) * | 2014-08-27 | 2016-03-30 | 北京计算机技术及应用研究所 | Method and system for infrared and low-level-light/visible-light fusion imaging |
| CN104274148A (en) * | 2014-09-28 | 2015-01-14 | 安徽中科医药成像技术科技有限公司 | Imaging system |
| CN106960428A (en) * | 2016-01-12 | 2017-07-18 | 浙江大立科技股份有限公司 | Visible ray and infrared double-waveband image co-registration Enhancement Method |
| CN106236006A (en) * | 2016-08-31 | 2016-12-21 | 杨晓峰 | 3D optical molecular image peritoneoscope imaging system |
| CN107103596A (en) * | 2017-04-27 | 2017-08-29 | 湖南源信光电科技股份有限公司 | A kind of color night vision image interfusion method based on yuv space |
Cited By (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109215079A (en) * | 2018-07-17 | 2019-01-15 | 艾瑞迈迪医疗科技（北京）有限公司 | Image processing method, operation navigation device, electronic equipment, storage medium |
| CN109688342A (en) * | 2019-01-11 | 2019-04-26 | 南京理工大学 | A kind of multispectral stereo imaging system |
| CN110602476A (en) * | 2019-08-08 | 2019-12-20 | 南京航空航天大学 | Hole filling method of Gaussian mixture model based on depth information assistance |
| CN110602476B (en) * | 2019-08-08 | 2021-08-06 | 南京航空航天大学 | A Hole Filling Method Based on Depth Information Aided Gaussian Mixture Models |
| WO2021051222A1 (en) * | 2019-09-16 | 2021-03-25 | 北京数字精准医疗科技有限公司 | Endoscope system, mixed light source, video acquisition device and image processor |
| CN110811498A (en) * | 2019-12-19 | 2020-02-21 | 中国科学院长春光学精密机械与物理研究所 | Visible light and near-infrared fluorescence 3D fusion image endoscopy system |
| CN112184604B (en) * | 2020-09-15 | 2024-02-20 | 杭州电子科技大学 | Color image enhancement method based on image fusion |
| CN112184604A (en) * | 2020-09-15 | 2021-01-05 | 杭州电子科技大学 | Color image enhancement method based on image fusion |
| CN112102171A (en) * | 2020-09-18 | 2020-12-18 | 贝壳技术有限公司 | Image processing method, image processing device, computer-readable storage medium and electronic equipment |
| CN112102171B (en) * | 2020-09-18 | 2021-10-08 | 贝壳找房(北京)科技有限公司 | Image processing method, image processing device, computer-readable storage medium and electronic equipment |
| CN112089403A (en) * | 2020-10-02 | 2020-12-18 | 深圳市中安视达科技有限公司 | Multispectral medical multi-path imaging method and system thereof |
| CN112089404A (en) * | 2020-10-10 | 2020-12-18 | 南京航空航天大学 | Endoscopic OCT-Raman dual-mode imaging device and imaging method |
| CN114027765A (en) * | 2020-11-20 | 2022-02-11 | 上海微创医疗机器人（集团）股份有限公司 | Fluorescence endoscope system, control method, and storage medium |
| CN113688707A (en) * | 2021-03-04 | 2021-11-23 | 黑芝麻智能科技(上海)有限公司 | Face anti-spoofing method |
| US12002294B2 (en) | 2021-03-04 | 2024-06-04 | Black Sesame Technologies Inc. | RGB-NIR dual camera face anti-spoofing method |
| CN113129243A (en) * | 2021-03-10 | 2021-07-16 | 同济大学 | Blood vessel image enhancement method and system based on infrared and visible light image fusion |
| WO2022267645A1 (en) * | 2021-06-21 | 2022-12-29 | 中兴通讯股份有限公司 | Photography apparatus and method, electronic device, and storage medium |
| CN115299914A (en) * | 2022-08-05 | 2022-11-08 | 上海微觅医疗器械有限公司 | Endoscope system, image processing method and device |
| CN116471466A (en) * | 2023-06-12 | 2023-07-21 | 深圳市博盛医疗科技有限公司 | Dual-wafer endoscope imaging method and imaging device |
| CN116471466B (en) * | 2023-06-12 | 2023-09-05 | 深圳市博盛医疗科技有限公司 | Dual-wafer endoscope imaging method and imaging device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108040243B (en) | 2019-06-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108040243A (en) | Multispectral 3-D visual endoscope device and image interfusion method | |
| CN106236006B (en) | 3D optical molecular image laparoscope imaging systems | |
| Geng et al. | Review of 3-D endoscopic surface imaging techniques | |
| CN109222865B (en) | Multimode imaging endoscope system | |
| CN104883946B (en) | Image processing device, electronic apparatus, endoscope device, and image processing method | |
| CN110638527A (en) | Augmented Reality Surgical Microscopic Imaging System Based on Optical Coherence Tomography | |
| WO2015037340A1 (en) | Image processing device, image processing method, and program | |
| CN106999019B (en) | Image processing device, image processing method, recording medium, and endoscope device | |
| JP7449736B2 (en) | Medical image processing device and medical observation system | |
| CN109310278A (en) | Image processing device, image processing method, program, and image processing system | |
| WO2018211854A1 (en) | 3d endoscope device and 3d video processing device | |
| US11779210B2 (en) | Ophthalmic imaging apparatus and system | |
| JP2010057619A (en) | Stereoscopic image capturing and displaying system | |
| JP6972049B2 (en) | Image processing method and image processing device using elastic mapping of vascular plexus structure | |
| CN109068035B (en) | Intelligent micro-camera array endoscopic imaging system | |
| CN209172253U (en) | A kind of multi-modality imaging endoscopic system | |
| JP5740559B2 (en) | Image processing apparatus and endoscope | |
| Liu et al. | Bio-inspired multimodal 3D endoscope for image-guided and robotic surgery | |
| JP2015231498A (en) | Endoscope device | |
| JP2012110481A (en) | Endoscope apparatus, focus control method, and program | |
| CN218792184U (en) | Monocular 3D stereoscopic endoscope system | |
| US11310481B2 (en) | Imaging device, system, method and program for converting a first image into a plurality of second images | |
| CN110115557A (en) | A kind of EO-1 hyperion based endoscopic imaging device and imaging method | |
| Kwan et al. | Development of a Light Field Laparoscope for Depth Reconstruction | |
| CN117398043B (en) | 3D fluorescent endoscope, imaging method, device and debugging method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |