JP2006305332A - Image processor and endoscope using the same - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To form a clear three-dimensional image from a plurality of two-dimensional images. <P>SOLUTION: The image processor 10 is equipped with an image pickup means 11 for picking up two-dimensional images and an image forming means 12 forming a three-dimensional image 16 from a plurality of two-dimensional images 15. Since the image forming means 12 is equipped with units U each comprising a plurality of lenses disposed in a matrix, the image forming means 12 can acquire two-dimensional images corresponding to the number of the units U. Since relative positions of units U can be extremely accurately grasped in advance, the calculation for calculating the three-dimensional positions can be accurately and easily carried out. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an image processing apparatus and an endoscope using the same, and more particularly to an image processing apparatus that generates a three-dimensional image from a plurality of two-dimensional images with high accuracy and an endoscope using the same.

  As a method of obtaining a three-dimensional image, there is a method of taking a two-dimensional image from cameras arranged at two or more viewpoints and generating a three-dimensional image using parallax of these images (the following non-patent document). 1). In this method, in order to generate a three-dimensional image, it is necessary to associate a plurality of corresponding points of the two-dimensional image.

  Specific methods of the association include a correlation method and an evaluation function method.

  In the association method using the correlation method, association is performed by comparing local patterns of two-dimensional images. Specifically, first, a window having a predetermined number of pixels per one point of one two-dimensional image is set. Then, the degree of similarity of the windows in the other two-dimensional image is measured, and one point corresponding to the window with the highest degree of similarity is made to correspond.

  The association method using the evaluation function can be applied to pixels for which no correspondence is found by the correlation method. In this method, parallax can be obtained by defining an evaluation function and minimizing this function. By using the evaluation function, parallax can be given to pixels whose correspondence is not determined only by the brightness pattern.

On the other hand, endoscopes are frequently used in the medical field. Particularly, in recent years, endoscopic surgical operations with less burden on patients have been performed. Imaging means such as a CCD camera is provided at the distal end of the endoscope, and image data acquired by the CCD camera is displayed on a monitor. A doctor who is a user has operated the endoscope while confirming an image displayed on the monitor.
Tsuyoshi Tsuyoshi, 3D Vision, Kyoritsu Publishing, p95-p103

  However, in the above-described 3D image generation method, when the number of pixels of the base 2D image is not sufficient and the image is rough, it may be difficult to find the correspondence between the pixels. Accordingly, there is a problem that the positional accuracy of the generated three-dimensional image is lowered and it is difficult to obtain a clear three-dimensional image. In addition, since shooting was performed using cameras arranged at different positions, it was difficult to accurately grasp the position of the camera, and it was difficult to obtain a highly reproducible three-dimensional image. .

  An image acquired by a conventional endoscope is a two-dimensional image, and the image is coarser and the field of view is narrow. Therefore, it is difficult to know in detail the situation inside the human body from the image obtained by the endoscope. As a result, doctors who use the endoscope are excessively fatigued or cause medical accidents. Here, if a plurality of cameras are provided at the distal end portion of the endoscope, a three-dimensional image can be acquired. However, it is very difficult to provide a large number of high-performance cameras at the distal end of a thin endoscope having a diameter of about 1 cm.

  The present invention has been made in view of the above problems. A main object of the present invention is to provide an image processing apparatus capable of acquiring a clear three-dimensional image from a plurality of two-dimensional images and an endoscope using the same.

  An image processing apparatus according to the present invention includes a plurality of units arranged in a matrix at different positions on the same plane, and integrates the plurality of two-dimensional images with imaging means for acquiring a plurality of two-dimensional images. Image generating means for generating a three-dimensional image.

  Furthermore, in the image processing apparatus of the present invention, the image generation means generates the three-dimensional image by an illuminance difference stereo method.

  Furthermore, in the image processing apparatus of the present invention, the image generation means detects a corresponding point between the two-dimensional images by calculating a cross-correlation coefficient.

  Furthermore, in the image processing apparatus of the present invention, the unit is composed of one lens and an image sensor disposed below the lens.

  The endoscope of the present invention is an endoscope having an elongated insertion portion that is inserted into a body cavity, and is provided with the insertion portion, and a plurality of units arranged in a matrix at different positions on the same plane And imaging means for acquiring a plurality of two-dimensional images, and image generation means for generating a three-dimensional image by integrating the plurality of two-dimensional images.

  Furthermore, in the endoscope of the present invention, the image generation means generates the three-dimensional image by an illuminance difference stereo method.

  Furthermore, in the endoscope of the present invention, the image generation means detects a corresponding point between the two-dimensional images by calculating a cross-correlation coefficient.

  Furthermore, in the endoscope of the present invention, the unit includes one lens and an image pickup device disposed below the lens.

  According to the image processing apparatus of the present invention, a three-dimensional image is generated based on a two-dimensional image obtained by an imaging means in which a large number of units are arranged in a plane. Therefore, since the relative positional relationship of each unit can be grasped very accurately, the generated three-dimensional image can be refined.

  In addition, according to the endoscope of the present invention, a clearer three-dimensional image can be obtained, so that the burden on the doctor who uses the endoscope can be reduced and the occurrence of a medical accident can be prevented. .

  Embodiments of the present invention will be described below with reference to the drawings.

<First Embodiment>
Here, the configuration and operation of an image processing apparatus and an endoscope using the same will be described.

  Referring to FIG. 1, an image processing apparatus 10 according to the present invention includes an imaging unit 11 that acquires a plurality of two-dimensional images and an image generation unit 12 that generates a three-dimensional image from the plurality of two-dimensional images.

  The imaging unit 11 can employ an individual imaging device such as a CCD (Charged Coupled Device). Here, imaging means 11 having a plurality of units is prepared, and a plurality of (for example, several hundred) two-dimensional images 15A, 15B, and 15C are acquired. The imaging means 11 has units arranged in a matrix on a plane. Therefore, when the same subject is simultaneously photographed by the imaging means 11, the two-dimensional images obtained by the individual units have slightly different (shifted) designs. Details of the unit will be described later.

  The image generation unit 12 generates a three-dimensional image 16 from a plurality of two-dimensional images photographed by the imaging unit 11. As described above, the individual two-dimensional images photographed by the imaging means 11 having a plurality of units arranged at different positions in a plan are slightly different in each symbol. In the present embodiment, this property is used to generate a three-dimensional image 16 from a plurality of two-dimensional images. When the number of pixels of the two-dimensional image obtained by the imaging unit 11 is not sufficient, the three-dimensional position accuracy of the three-dimensional image may not be sufficient. In the present embodiment, by using a unit arranged with regularity on one substrate, the calculation of the position of the three-dimensional image is facilitated and the calculation accuracy is increased. Here, a triangular mesh lattice is used as the three-dimensional model.

  In this embodiment, the image processing means such as the image generating means 12 is realized by a semiconductor element such as an LSI or software.

  Here, examples of a method for generating a three-dimensional image from a plurality of two-dimensional images include an illuminance difference stereo method and a multi-baseline stereo method. In this embodiment, an illuminance difference stereo method is adopted, and a simulation result using this method will be described later. Further, as a method for detecting the relative positions of two-dimensional images, a method for calculating a cross-correlation coefficient is adopted in the present embodiment, but details of this method will be described in detail.

  With reference to FIG. 2 (A), the structure of the image input device 20 which can be used as an imaging means in this Embodiment is demonstrated. The image input device 20 includes a light receiving element array 23, a partition wall layer 26 and a lens array 21 disposed on the light receiving element array 23.

  On the surface of the light receiving element array 23, light receiving elements 27 made of CCD or CMOS are arranged in a matrix.

  The partition layer 26 includes partition walls 25 combined in a lattice shape, and has a function of preventing noise from entering the light receiving element. The partition wall 25 is formed from the light receiving element array 23 to the lens array 21.

  The lens array 21 includes a plurality of lenses 24 arranged in a matrix, and each lens 24 is fixed to the upper part of the partition wall layer 26.

  A plurality of light receiving elements 27 of the light receiving element array 23 correspond to the minute lens 24 of the lens array 21. Further, one lattice portion of the partition wall layer 26 corresponds to one lens 24. As indicated by a prismatic line with a chain line, one lens 24, one partition wall 25 and a light receiving element 27 positioned below constitute a unit U which is one signal processing unit. One unit U can obtain one two-dimensional image. In the image input device 20, for example, about 600 units U are formed. Therefore, the image input device 20 can simultaneously acquire, for example, 600 two-dimensional images. Each unit U has about 400 CCD elements.

  The specific planar size of the image input device 20 is, for example, about 8 mm × 7 mm. And about 600 units U are arranged at a pitch of about 200 μm.

  In the present embodiment, a very high relative positional accuracy between the lenses 24 (units U) is used. Therefore, a three-dimensional image with excellent positional accuracy can be obtained.

  Conventionally, for example, a three-dimensional image is generated by using about 2 to 3 cameras arranged in a studio. However, with this method, in order to calculate a three-dimensional image, it is necessary to measure the position of the cameras, the angle of the camera, and the like, and the calculation of the position of the three-dimensional image is also complicated. In this embodiment, as described above, the image sensor of each unit U is built with extremely high positional accuracy by a semiconductor manufacturing process. Therefore, the relative positional relationship between the units is basically unchanged, and there is no need to measure them individually. The design value of each element can be used as it is for the calculation of the three-dimensional position. This makes the obtained three-dimensional image clear and reduces the cost for calculation.

  With reference to FIG. 2B, a configuration of an endoscope in which the image input device 20 described above is incorporated will be described. Here, the image input device 20 is provided on the side surface of the distal end of the elongated insertion portion 28 to be inserted into the body vagina of the human body. Specifically, the image input device 20 provided with a large number of units in a plane is fixed to the side surface of the distal end portion of the insertion portion 28. In addition, a light-emitting unit 29 made of a light-emitting element such as an LED is provided at the other part of the distal end of the insertion unit 28 in proximity to the image input device 20.

  For example, the endoscope having the above-described configuration is used as follows. First, a doctor who is a user inserts an elongated insertion portion 28 of an endoscope into a body vagina such as a patient's esophagus. Next, light is emitted from the light emitting unit 29 into the body vagina, and the light reflected by the inner wall of the body vagina is photographed by the image input device 20. Hundreds of two-dimensional images are acquired by a large number of units U provided in the image input device 20, and these two-dimensional images are combined to generate a precise three-dimensional image. This three-dimensional image is displayed almost in real time (simultaneously) on a monitor such as a television set arranged outside. A doctor who is a user can accurately know the state of the patient's body and vagina from the projected three-dimensional image. Furthermore, it is possible to perform an operation while confirming the three-dimensional image.

  Next, with reference to FIGS. 3 and 4, a three-dimensional image generated using the image processing apparatus 10 having the above-described configuration will be introduced.

  FIG. 3 shows two-dimensional images 15A, 15B, and 15C taken from different viewpoints. Here, a two-dimensional image obtained by photographing the ground surface from the sky is displayed as an example. Since the two-dimensional images 15 are taken from different viewpoints, their designs are slightly different. In the present embodiment, a two-dimensional image 15 can be obtained from each of several hundred units U whose structure is shown in FIG.

  With reference to FIG. 4, a three-dimensional image 16 is generated based on the above-described many two-dimensional images 15. Here, the feature points of the two-dimensional image are extracted and matched, and the three-dimensional image 16 can be generated by a method of calculating the three-dimensional position of the subject by the illuminance difference stereo method or the like. As described above, in this embodiment, since the image pickup unit in which the units are densely arranged on one plane is used, it is possible to generate a three-dimensional image with relatively high accuracy. As described above, the illuminance difference stereo method and the multi-baseline stereo method can be considered as a method for generating a three-dimensional image. In this embodiment, the illuminance difference stereo method is used. Details of this method and simulation results will be described later.

  In the present embodiment, as shown in FIG. 4, a triangular mesh lattice is used as a surface shape model of a three-dimensional image. In the triangular mesh grid, the value of the vertex of the mesh is information in the height direction, and expresses a three-dimensional coordinate position.

<Second Embodiment>
In this embodiment, a method for generating a three-dimensional image from a plurality of two-dimensional images using the illuminance difference stereo method will be described together with a simulation result.

  First, referring to FIG. 5, in the present embodiment, a Lambert model is employed as a surface characteristic model of an object. The Lambert model is described to surface diffuse reflection on an object having a smooth surface without unevenness.

Here, when the height of the object surface that is the subject is z = h (x, y), the normal vector n _v = ( _av , b _v , c _v ) of the object surface that is the subject is expressed by the following equation (1). ).

Further, if the incident light amount from the light source toward the surface of the subject 32 is Θ ₀ and the incident light vector is n ₁ = (a ₁ , b ₁ , c ₁ ), the lens 24A when the surface of the subject 32 has a complete diffusion characteristic. The incident light quantity Θ ₁ is given by the following equation (2).

Similarly, the incident light quantity Θ ₂ to the lens 24B adjacent to the lens 24A is given by the following formula (3), where n ₂ = (a ₂ , b ₂ , c ₂ ).

The above formula (2) and (3), it is possible to obtain the normal vector a _v and b _v of the object, it is possible to obtain a height direction of the data.

  That is, in this embodiment, the normal vector (the data in the height direction) of the surface of the subject 32 is obtained from the incident light incident on the two adjacent lenses 24A and 24B. Therefore, data in the height direction can be obtained according to the number of two-dimensional images. From this, it is possible to obtain several hundreds of three-dimensional images corresponding to the number of units and use them to improve accuracy.

  Here, in order to calculate the three-dimensional position of the subject 32, it is preferable to accurately grasp the relative positions of adjacent lenses. In this embodiment, referring to FIG. 2, an image input device is configured by forming a partition wall layer 26 and a lens 24 on a single semiconductor substrate. Accordingly, the relative positions of the adjacent lenses can be calculated with extremely high accuracy and are invariant, so that the three-dimensional position of the subject can be calculated with high accuracy.

  Next, a method (correlation analysis method) for aligning two two-dimensional images will be described with reference to FIG. In order to generate a three-dimensional image from two two-dimensional images, it is necessary to specify a corresponding part between the images. A number of methods have been proposed as this method, but this embodiment uses a highly accurate correlation analysis method.

  As shown in FIG. 6, in the correlation analysis method (stereo image method), a certain limited range (nx × ny) in two two-dimensional images is set, and in this range, the following equation (4) is set. The cross-correlation coefficient ρ shown is obtained.

Next, one image is moved by one pixel, and the cross-correlation coefficient ρ is obtained again. If this is performed in the vertical and horizontal directions and the point where the obtained correlation coefficient is maximized is obtained, this becomes the corresponding point indicating the same place in the two images. By calculating Equation 4 for all pixels, the corresponding points of each pixel of the two images can be obtained.

<Third Embodiment>
Next, with reference to FIG. 7 and FIG. 8, the result of an experiment conducted to confirm the accuracy of a three-dimensional image generated using the above-described illuminance difference stereo method will be described. FIG. 7 is a diagram showing an outline of the experiment, FIG. 8A is a table showing the experimental results, and FIG. 8B is a graph showing the experimental results. In this experiment, a spherical ball was used as the subject 32, and it was confirmed how much the shape of the subject 32 was reproduced by the image apparatus of the present invention.

  First, the conditions of the experiment will be described with reference to FIG. The subject 32 as a ball was placed at the origin (0, 0, 0). The image pickup module 30 (the image pickup device 20 described above) was placed on (0, 0, 20) with a lens (not shown) facing the subject 32. Furthermore, the light 31 which is a light source is arrange | positioned at (10, 2.5, 20).

  After each device is arranged as described above, the imaging module 30 captures an image of the subject 32 irradiated with the light 31. Thus, for example, several hundred two-dimensional images are obtained by the imaging module 30. In this experiment, first, two two-dimensional images obtained by adjacent units are taken out from a large number of two-dimensional images. Next, after the two extracted images are converted into monochrome, the luminance of the shape extraction point is measured. Furthermore, a three-dimensional shape is extracted using the illuminance difference stereo method.

  FIG. 8A shows a part of the shape extraction result. From this result, the error rate is large at a steep portion (for example, a point where the X coordinate is 0 and the Y coordinate is 1.875). The cause seems to be that the image is rough on a steep slope and the brightness cannot be obtained with high accuracy.

  FIG. 8B is a graph of shape extraction data. The point where the lines intersect is the point where the shape was extracted this time. The points calculated by shape extraction are supplemented by straight lines. Since the distance between the points is long, the shape is rough, but the three-dimensional effect of the ball is visible. This graph has a substantially hemispherical shape because only the hemisphere of the ball that is the subject is photographed.

  From the above experiments, it has been clarified that by using the imaging module 30 of the present embodiment, a three-dimensional image in which the three-dimensional shape of the subject is faithfully reproduced can be generated.

<Other embodiments>
For example, the following means may be added to the image processing apparatus of the present embodiment that generates a three-dimensional image.

  For example, an image correction unit that corrects a three-dimensional image obtained by the image generation unit 12 may be added. In this image correction unit, first, the three-dimensional image 16 obtained by the image generation unit 12 is projected onto each two-dimensional image 15A, 15B, 15C. Next, a normal vector is calculated based on the pixel information (luminance) of the two-dimensional images 15A, 15B, and 15C using a Lambert model, and the three-dimensional position of the three-dimensional image 16 is corrected. A plurality of three-dimensional images are generated. Here, the same number of corrected three-dimensional images as the number of two-dimensional images (for example, several hundreds) is generated.

  Furthermore, a means for integrating the three-dimensional images corrected as described above may be provided. Specifically, a plurality of corrected three-dimensional images are integrated to generate one three-dimensional image. Furthermore, when this integration is performed, if the three-dimensional position information is optimized by the optimization means (Bayesian estimation), the position accuracy of the obtained three-dimensional image can be further improved.

  The image processing apparatus of the present invention can be applied to, for example, a stereoscopic video apparatus, an image input apparatus, a medical apparatus such as a stereoscopic endoscope, and a robot such as an eye of a small robot.

It is a figure which shows the structure of the image processing apparatus of this invention. (A) is a perspective view showing an image input device applicable to the image processing device of the present invention, and (B) is a perspective view showing an endoscope of the present invention. It is a figure which shows the two-dimensional image processed with the image processing apparatus of this invention. It is a figure which shows the three-dimensional image produced | generated using the some two-dimensional image in the image processing apparatus of this invention. It is a figure which shows the method of processing an image with the image processing apparatus of this invention, and is a figure which shows the principle which calculates | requires the three-dimensional position of a to-be-photographed object. It is a figure which shows the corresponding point search method of the two-dimensional image of this invention. It is a figure which shows the outline | summary of the experiment conducted using the image processing apparatus of this invention. It is a figure which shows the result of the experiment conducted using the image processing apparatus of this invention, (A) is a table | surface which shows a result, (B) makes it a graph.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Image processing apparatus 11 Imaging means 12 Image generation means 15A-15C Two-dimensional image 16 Three-dimensional image 20 Image input device 21 Lens array 23 Light receiving element array 24 Lens 24A, 24B Lens 25 Partition 26 Partition wall 27 Light receiving element 28 Insertion part 29 Light emitting unit 30 Imaging module 31 Light 32 Subject U unit

Claims (8)

An imaging means comprising a plurality of units arranged in a matrix at different positions on the same plane, and acquiring a plurality of two-dimensional images;
An image processing apparatus comprising: an image generating unit that generates a three-dimensional image by integrating the plurality of two-dimensional images.   The image processing apparatus according to claim 1, wherein the image generation unit generates the three-dimensional image by an illuminance difference stereo method.   The image processing apparatus according to claim 1, wherein the image generation unit detects a corresponding point between the two-dimensional images by calculating a cross-correlation coefficient.   The image processing apparatus according to claim 1, wherein the unit includes one lens and an image sensor disposed below the lens. In an endoscope having an elongated insertion portion to be inserted into a body cavity,
An imaging unit provided in the insertion unit, including a plurality of units arranged in a matrix at different positions on the same plane, and acquiring a plurality of two-dimensional images;
An endoscope comprising: an image generating unit that generates a three-dimensional image by integrating the plurality of two-dimensional images.   The endoscope according to claim 5, wherein the image generation unit generates the three-dimensional image by an illuminance difference stereo method.   6. The endoscope according to claim 5, wherein the image generation means detects a corresponding point between the two-dimensional images by calculating a cross-correlation coefficient.   The endoscope according to claim 5, wherein the unit includes one lens and an image pickup device disposed below the lens.