WO2017024422A1

WO2017024422A1 - Endoscope stereo vision system utilizing shape-from-shading technique and method thereof

Info

Publication number: WO2017024422A1
Application number: PCT/CN2015/000587
Authority: WO
Inventors: 库马·阿兔; 汪彦佑; 刘楷哲; 王民良
Original assignee: 明士股份有限公司
Priority date: 2015-08-13
Filing date: 2015-08-13
Publication date: 2017-02-16
Also published as: JP6600442B2; JP2018529404A

Abstract

The invention provides an endoscope stereo vision system utilizing a shape-from-shading technique and a method thereof. The endoscope stereo vision system comprises an image conversion device connected between a single lens endoscope and a stereoscopic display. The image conversion device has a built-in shape-from-shading algorithm, and is configured to compute depth information by applying shape-from-shading to a flat image obtained from the single lens endoscope, and then generate a depth map, generate, according to the flat image and the depth map, a required stereoscopic image, and then output the stereoscopic image to the external stereoscopic display. The endoscope stereo vision system facilitates a user to view a stereoscopic image, requiring neither to replace an existing single lens endoscope with a dual-camera endoscope system, nor to modify a hardware structure of the present single lens endoscope, and addressing problems of being unable to provide a stereoscopic image in a conventional single lens endoscope and an overpriced dual-camera endoscope equipment.

Description

Endoscope stereoscopic visualization system and method using colorimetry

Technical field

The invention relates to an endoscope stereoscopic visualization system, in particular to an endoscope stereoscopic visualization system for generating stereoscopic images by using a chromaticity forming method.

Background technique

Minimally invasive surgery in existing medical behavior has become an indispensable part of the treatment of many diseases. The minimally invasive surgery is performed by an auxiliary surgical instrument of an endoscope, which has a small incision and reduces tissue trauma, and can Shorten the cycle of patient recovery and reduce the cost of overall treatment. However, the existing minimally invasive surgery uses a single-lens endoscope, which can only display a planar (2D) image. Because the planar image lacks depth information, the surgeon according to the planar image It is a challenge to accurately move the surgical instrument to the correct position in the patient. Surgeons often rely on motion parallax, monocular cues, and other indirect evidence to give depth to the aforementioned planar image to improve positioning accuracy. However, stereoscopic images (including 2D images and depth information) provide depth perception and no additional experience (motion parallax, monocular cues, and other indirect evidence) are still the best solution to the aforementioned poor positioning accuracy. the way. The existing stereoscopic image to be obtained in minimally invasive surgery is a dual-lens endoscope, which can obtain the depth information or stereoscopic image required by the surgeon, but the cost is much more expensive than the single-lens endoscope, which makes it difficult Disadvantages of popularity.

Disclosure of invention

As described above, the existing single-lens endoscope can only display a planar image, and the planar image lacks depth information and has difficulty in accurately positioning. The dual-lens interior mirror has the advantage of obtaining a stereoscopic image, but its cost is It is expensive and not easy to popularize. Therefore, the main object of the present invention is to provide an endoscope stereoscopic visualization system and a method thereof using a chromaticity forming method, which mainly produces a depth map by using a chromaticity forming method for a plane image obtained by a single lens endoscope. The stereo image can be generated by combining the planar image without setting up the dual camera and without modifying the hardware structure of the existing single lens endoscope, and solving the existing single lens endoscope without The method provides a problem that the stereo image and the dual camera endoscope device are expensive.

The main technical means adopted to achieve the aforementioned objectives is to make the aforementioned endoscope stereoscopic visualization system using the colorimetric method include:

The image conversion device has an endoscope input port, a 2D/3D image conversion unit and an image output port, and the endoscope input port is configured to connect the single lens endoscope to obtain a plane through the single lens endoscope The 2D/3D image conversion unit generates a depth map of the planar image and converts it into a stereo image by using a built-in chromaticity forming method, and the image output port is connected with a stereoscopic display, and the stereoscopic display is used to display the 2D a stereoscopic image converted by the /3D image conversion unit;

Wherein, the chromaticity forming method refers to the 2D/3D image converting unit calculating a light source of the plane image and the shadow distribution information thereof, wherein the distribution information is a gradient repeating iterative operation performed by the pixel value, and the direction and position of the combined light source illumination Information is added to the source illumination direction estimate to obtain the depth map.

The main technical means adopted to achieve the aforementioned objectives is to make the aforementioned stereoscopic visualization method using the chromaticity forming method include:

Capture a planar image: obtain a planar image of the endoscopic camera with an image capture device;

Generating a depth map by using a chromaticity forming method: calculating a light source of the planar image and a shadow distribution information thereof by a chromaticity forming method, wherein the distribution information is a result of a gradient repeating iterative operation performed by a pixel value, and a direction in which the light source is combined with the light source The position information is further added to the illumination direction estimation to improve the accuracy of the relative position judgment. The calculated result is a relative depth map, and the pixel value in the depth map includes the pixel intensity, the direction of the light source, and the natural logarithm of the coordinates. And accelerate its iterative process with fast solution equations and parallel operations;

Generating a disparity map by using a depth map: the depth map is composed of a gray scale image, which is represented by a context of an object in a third dimension, and the depth map generates a disparity map during the process of transforming the stereo image, the parallax map The pixel value is inversely proportional to the depth map but proportional to the focal length of the camera and the spacing between the two eyes;

The left and right images of the stereoscopic image are generated: the disparity map obtained from the stereoscopic image is used to generate left and right eye images of the stereoscopic image, and the pixel value of the disparity map also represents the offset of each pixel between the two eyes, thereby generating a left eye or a right eye. A stereoscopic image of the image.

An endoscope stereoscopic visualization system using the chromaticity forming method and the method thereof, wherein the planar image obtained by the single lens endoscope is calculated by the chromaticity forming method by the image conversion device The depth information is further generated to generate a depth map, and the desired stereoscopic image is generated through the planar image and the depth map, and then output to an external stereoscopic display, so that the user can view the converted stereoscopic image without the need of the existing single lens internal view. The mirror is replaced with a dual-camera endoscope system, and the hardware structure of the existing single-lens endoscope is not required to be modified, thereby solving the problem that the existing single-lens endoscope cannot provide stereoscopic images and the equipment of the dual-camera endoscope is expensive.

BRIEF DESCRIPTION OF THE DRAWINGS

BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 is a block diagram of a circuit in accordance with a preferred embodiment of the present invention.

2 is a flow chart of a preferred embodiment of the present invention.

Reference mark:

10: Image conversion device

11: Endoscope input port

12:2D/3D image conversion unit

13: image output port

20: Single lens endoscope

30: Stereoscopic display.

The best way to implement the invention

Referring to FIG. 1 , the image conversion device 10 is connected to a single-lens endoscope 20 and a stereoscopic display 30. The image conversion device 10 has an endoscope. The input port 11, the 2D/3D image converting unit 12 and the image output port 13 are electrically connected to the endoscope input port 11 and the image output port 13, respectively. The endoscope input port 11 is Connected to the single lens endoscope 20, the image output port 13 is connected to the stereoscopic display 30; the 2D/3D image conversion unit 12 obtains a planar image through the single lens endoscope 20, and has built-in chromaticity The forming method converts the planar image into a stereoscopic image, and the stereoscopic image is output from the image output port 13 to the stereoscopic display 30, so that the stereoscopic display 30 displays the stereoscopic image converted by the 2D/3D image converting unit 12.

For a flowchart of a preferred embodiment of the present invention, as shown in FIG. 2, the 2D/3D image conversion unit 12 performs the following steps to convert a planar image of the endoscope into a stereoscopic image.

Endoscope camera correction (S1): The internal parameters of the endoscope camera are calculated using the camera calibration method ^[1] , which performs camera pose estimation with rotation and translation of a calibration template, and then obtains a nonlinear solution. The camera's internal parameters and external parameters.

Capture Plane Image (S2): Obtain a planar image of the endoscope camera with an image capture device; the image capture device is a resolution of SD or HD, and the lens of the endoscope camera is a 30° or wide-angle lens.

The depth map is generated by the chromaticity forming method (S3): the light source and its shadow distribution information are calculated by the chromaticity forming method ^[2] , and the distribution information is the result of the gradient repeat iterative operation performed by the pixel value, and the direction of the combined light source illumination is The position information is added to the illumination direction estimation ^[3] to improve the accuracy of the relative position judgment. The calculated result is a relative depth map, and the pixel value in the depth map includes the pixel intensity, the direction of the light source and the natural coordinates. Logarithm, and accelerate its iterative process with fast solution equation ^[4] and parallel operations.

Wherein, the chromaticity forming method can complete the calculation of the illumination distribution by using the following description: assuming the camera position C(α, β, γ), there is a surface normal vector n and a light source vector 1 in the image space domain x=(x, y) The pixel point at which the three-dimensional point M depends in the image plane is m, which can be described as:

Where u(x) represents the depth of the x point and u _x , u _y is the spatial derivative. Therefore, the illumination equation of the image can be used to solve the Lambertian SFS method which does not include the distance attenuation of the light source and the surface reflection.

Where: ρ is the reflectivity of the surface, by displacement

You can get the Hamiltonian equation:

among them,

The depth image of the illumination distribution can be generated by an iterative equation, since the source of the endoscope is almost identical to the camera vector of the endoscope, it can be simplified to have the same vector.

The depth map is used to generate a disparity map (S4): the depth map is composed of gray scale images, which is represented by the context of the object in the third dimension, and the depth map generates a disparity map during the process of transforming the stereo image ^[5] . The pixel value of the disparity map is inversely proportional to the depth map but proportional to the focal length of the camera and the distance between the two eyes.

The left and right images of the stereoscopic image are generated (S5): the disparity map obtained from the stereoscopic image is used to generate the left and right eye images of the stereoscopic image, and the pixel value of the disparity map also represents the offset of each pixel between the two eyes, thereby generating A stereoscopic image of the left or right eye image. The generated left-eye and right-eye images can be converted into different 3D playback formats, such as side-by-side or interlaced or other 3D playback formats, and then played through the stereoscopic display.

It can be seen from the above that the planar image is calculated by the chromaticity forming method, and after the depth map is generated, the desired stereoscopic image is generated by the planar image and the depth map, and the existing single lens endoscope is not required to be replaced. The camera endoscope system does not need to modify the hardware structure of the existing single-lens endoscope, and solves the problem that the existing single-lens endoscope cannot provide stereoscopic images and the equipment of the dual-camera endoscope is expensive.

Reference materials:

[1] "Image processing, analysis and machine vision, 2nd ed., vol. 68PWS, 1998, pp. 448-457"

[2] Visentini-Scarzanella et al. (IEEE Internal Conference on Image Processing) "Using Chroma Forming and Specular Reflection from Monocular Images" Metric depth recovery from monocular images using shape-from-shading and specularities

[3] Danail Stoyanov et al. 2009 IEEE/RSJ International Conference on Intelligent Robots and System (IROS) Illumination position estimation for 3D soft tissue reconstruction in robotic minimally invasive surgery

[4] Chiu-Yen Kao et al. SIAM J. Numerical Analysis 2005 “Fast Sweeping Methods for Fast Sweeping Methods for Static Hamiton-Jacobi Equation)

[5] Berretty et al. 2006 International Society for Optics and Photonics 2006 “Real-time rendering for multi-view auto” (Real-time rendering for multi-view auto) -stereoscopic displays).

Claims

An endoscope stereoscopic visualization system using a chromatic shaping method, comprising:

The image conversion device has an endoscope input port, a 2D/3D image conversion unit and an image output port, and the endoscope input port is configured to connect the single lens endoscope to obtain a plane through the single lens endoscope The 2D/3D image conversion unit generates a depth map of the planar image and converts it into a stereo image by using a built-in chromaticity forming method, and the image output port is connected with a stereoscopic display, and the stereoscopic display is used to display the 2D a stereoscopic image converted by the /3D image conversion unit;

Wherein, the chromaticity forming method refers to the 2D/3D image converting unit calculating a light source of the plane image and the shadow distribution information thereof, wherein the distribution information is a gradient repeating iterative operation performed by the pixel value, and the direction and position of the combined light source illumination Information is added to the source illumination direction estimate to obtain the depth map.
A stereoscopic visualization method for an endoscope using a chromatic shaping method, comprising:

Capture a flat image: obtain a planar image of the endoscope camera with an image capture device;

Generating a depth map by using a chromaticity forming method: calculating a light source of the planar image and a shadow distribution information thereof by a chromaticity forming method, wherein the distribution information is a result of a gradient repeating iterative operation performed by a pixel value, and a direction in which the light source is combined with the light source The position information is further added to the illumination direction estimation to improve the accuracy of the relative position judgment. The calculated result is a relative depth map, and the pixel value in the depth map includes the pixel intensity, the direction of the light source, and the natural logarithm of the coordinates. And accelerate its iterative process with fast solution equations and parallel operations;

Generating a disparity map by using a depth map: the depth map is composed of a gray scale image, which is represented by a context of an object in a third dimension, and the depth map generates a disparity map during the process of transforming the stereo image, the parallax map The pixel value is inversely proportional to the depth map but proportional to the focal length of the camera and the spacing between the two eyes;

The left and right images of the stereoscopic image are generated: the disparity map obtained from the stereoscopic image is used to generate left and right eye images of the stereoscopic image, and the pixel value of the disparity map also represents the offset of each pixel between the two eyes, thereby generating a left eye or a right eye. The stereoscopic image of the image.
The endoscopic stereoscopic visualization method using the chromaticity forming method according to claim 2, wherein the step of correcting the planar image is performed by performing an endoscope camera correction step, and the endoscope camera correction is performed using a camera correction The method calculates the internal parameters of the endoscope camera, and the camera correction method performs camera attitude estimation by correcting the rotation and translation of the template, and then obtains the internal parameters and external parameters of the camera by a nonlinear solution.
The endoscopic stereoscopic visualization method using the chromaticity forming method according to claim 2 or 3, wherein the chromaticity forming method performs the calculation of the illumination distribution by the following description:

Assuming that the camera position C(α, β, γ) has a surface normal vector n and a light source vector l in the image space domain x=(x, y), the pixel point of the three-dimensional point M in the image plane is m is described as :

Where u(x) represents the depth of the x point, and u x , u y are the spatial derivatives; therefore, the illumination equation of the image is used to solve the Lambert characteristic forming method that does not include the distance attenuation of the light source and the surface reflection;

Where: ρ is the reflectivity of the surface, by substituting v'=lnu to obtain the Hamiltonian equation:

among them,

The depth image of the illumination distribution is generated via an iterative equation, since the source of the endoscope is almost identical to the camera vector of the endoscope, it is simplified here to have the same vector.