JP2013240590A - Three-dimensional shape acquisition device from stereoscopic endoscopic image - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a three-dimensional acquisition device which can acquire a three-dimensional shape for an image photographed by a stereoscopic endoscope with accuracy when a subject to be examined is concavo-convex and makes large movements such as a rhinopharynx.SOLUTION: An insertion portion 1 includes an image acquisition part 10 having a camera where a pair of image pickup devices are disposed on a distal end side and a pattern projection part 20 having a projector lens on a distal end side and is connected to a control processing part 40. The pattern projection part 20 has a projection pattern housing part 20A where a light source for projection and a projection pattern are housed, image light from the projection pattern irradiated with the light source is transmitted to a distal end side by an image fiber and is focused onto a position of a subject to be examined by the projector lens, and a three-dimensional image is acquired by performing stereo-matching processing on an image obtained by photographing the subject to be examined on which the projection pattern is projected by the right and left image pickup devices. The projector lens has an imaging depth of 15 mm or more and projects a pattern image having a size of 20 mm or more on a surface of the subject to be examined.

Description

  The present invention relates to an apparatus for acquiring a three-dimensional image from a stereoscopic endoscope image, and more particularly to an apparatus for acquiring a three-dimensional image from a stereoscopic endoscope image combined with pattern irradiation, which is effectively applied to a medical examination of a nasopharynx or the like.

  Cleft palate patients acquire normal language by acquiring nasopharyngeal closure function in the rehabilitation process after surgery in infants. Because approaches differ greatly, it is important to accurately evaluate nasopharyngeal closure function for effective speech therapy. In clinical practice, the movement of the nasopharynx is observed using an endoscope to determine the closed state of the nasopharynx, but this is only a qualitative evaluation, and this evaluation method lacks objectivity There is a problem that detailed analysis is not possible.

  CT and MR images may be used to analyze the movement of the nasopharynx, and CT and MR that are capable of high-speed imaging are being developed. Often, nasopharyngeal regions are not accurately imaged due to device metal artifacts. Therefore, at present, the approach is actually limited to an approach using an endoscope.

  With respect to the stereoscopic endoscope in the prior art, there are those shown in Patent Documents 1 to 3, and the endoscope shown in Patent Document 1 is an imaging unit that forms a left and right image on the distal end configuration portion of the endoscope. And the left and right images are switched and displayed by the video switching means to obtain a stereoscopic image. The endoscope shown in Patent Document 2 is an imaging unit for left and right images arranged at the distal end portion of the endoscope. The configuration is such that the imaging element and the substrate of the imaging unit on one side are arranged facing the other side in the vertical direction with respect to the imaging element and the substrate on the other side, and the interval between the objective lenses in both units is adjusted to the optimum interval It is what makes it possible. In addition, the endoscope disclosed in Patent Document 3 performs image conversion on an image obtained by imaging a subject from two viewpoints, and the spatial characteristics of the subject based on the optical data indicating the characteristics of the optical system and the image data. Is to measure.

  In Patent Document 1, the arrangement of image pickup elements for stereoscopic vision is optimized in an endoscope whose outer diameter is limited, and in Patent Document 2, the objective lens interval of the left and right image pickup units is set to the optimum interval. Further, in Patent Document 3, it is possible to measure the spatial characteristics of a subject without requiring measurement of optical data regarding the spatial characteristics of the optical system necessary for performing stereo measurement by triangulation. However, when the subject has a shape with an uneven depth and moves, the subject is not particularly effective in accurately and accurately observing the subject.

  The endoscope shown in Patent Document 4 projects a pattern image having a plurality of measurement points onto an observation target, and positions of the measurement points based on images obtained by photographing the pattern image by a plurality of photographing means from different positions. The coordinates are calculated to measure the three-dimensional shape of the observation target. With this endoscope, the three-dimensional shape of the observation target can be measured at high speed and with high accuracy, but when the observation target has irregularities, it is not guaranteed that the pattern is clearly imaged within the depth range. The shape measurement is not always performed with high accuracy.

JP-A-7-163517 JP 11-56757 A JP 2008-29497 A JP 2005-287900 A

When the object to be examined is observed with a stereoscopic endoscope to obtain a three-dimensional shape, for example, when grasping the closed state of the nasopharynx as the object to be inspected, the surface of the object to be inspected changes in shading. Since there are no feature points that serve as indices due to the small amount, a clear image or an image that can be easily discriminated cannot be obtained. As a method for dealing with this, a pattern image is projected to acquire a three-dimensional shape of an object to be inspected in an image, and a three-dimensional shape is acquired for the object to be inspected with the projection pattern combined.
However, when the object to be inspected has a concavo-convex shape like the nasopharynx and takes a large movement, the projection pattern as a whole is not clear within the observation object range of the object to be inspected. Observation and acquisition of a three-dimensional shape were not performed accurately. For this reason, it has been demanded to obtain a three-dimensional shape with high accuracy by projecting a pattern image onto an object to be inspected, which has unevenness, and generates an image with a stereoscopic endoscope.

  The present invention has been made to solve the above-described problems, and a three-dimensional shape acquisition device for stereoscopic endoscope images according to the present invention has an image having a camera in which a pair of image sensors are arranged on the distal end side. An acquisition portion and a pattern projection portion having a projection lens on the distal end side, an insertion portion formed in an elongated shape so as to be inserted toward the object to be inspected, a connection portion connected to the insertion portion, and a connection portion connected to the connection portion An apparatus for acquiring a three-dimensional shape from a stereoscopic endoscope image comprising a control processing unit, wherein the pattern projection unit is configured to project a projection pattern and the projection pattern at a rear position away from the distal end of the insertion unit. A projection pattern accommodating portion in which a projection light source that illuminates at the position is arranged, an image fiber that transmits image light from the projection pattern irradiated by the light source to the distal end side, and the transmitted image light is projected A projection lens, and configured to project an image of the projection pattern onto a position of the object to be inspected, and the control processing unit photographs the object to be inspected on which the projection pattern image is projected with a camera having a pair of imaging elements. A stereo matching process is performed on the obtained image to obtain a three-dimensional shape of the object to be inspected, and the projection lens has an F value and a focal length having an imaging depth of 15 mm or more.

  It is preferable that the projection lens project a pattern image having a size of at least 20 mm on the surface of the object to be projected on which the projection pattern image is projected, and the projection pattern placement unit can accommodate and arrange the projection pattern in an exchangeable manner. In addition, for the image obtained by photographing the inspected object on which the projection pattern image is projected with a camera having a pair of imaging elements, a pair of imaging elements facing the subject in the control processing unit After performing a rectifying process for converting an image based on the visual axis into an image based on a virtual parallel visual axis, a stereo matching process may be performed to acquire a three-dimensional shape of the object to be inspected.

  In the present invention, when acquiring a three-dimensional shape from an image obtained by photographing an object to be inspected by projecting a projection pattern image with a pair of cameras, at a rear position away from the distal end of the insertion portion of the stereoscopic endoscope. By arranging the projection pattern, transmitting the projection pattern image by the image fiber, projecting the projection pattern image to the position of the object to be inspected by the projection lens, and increasing the imaging depth of the projection lens, A clear projection pattern can be imaged even on the inspection object. By disposing the projection pattern at a position away from the distal end of the insertion portion, the size of the projection pattern is increased, and by using a high-luminance one such as for a gastroscope as a projection light source, A bright projection pattern image can be generated over a wide range of positions. As a result, it is possible to accurately acquire the three-dimensional shape of the object to be inspected, such as the nasopharynx, which has unevenness and movement.

It is a figure which shows roughly the three-dimensional shape acquisition apparatus from the stereoscopic endoscope image by combined pattern irradiation by this invention. It is a perspective view which shows an endoscope front-end | tip part. It is a figure which shows the structure of a pattern projection part as sectional drawing. It is a figure which shows the condition which acquires an image by an image acquisition part. (A) It is a figure which shows an example of a projection pattern, (b) It is a figure which shows the example of another projection pattern. It is a figure explaining how to obtain | require the depth using the parallax in an image. It is a figure which shows the flow of the arithmetic processing which calculates | requires three-dimensional shape from the image on either side.

The specific form of the three-dimensional shape acquisition apparatus from the stereoscopic endoscope image by combined pattern irradiation according to the present invention will be described below.
FIG. 1 shows an overall configuration of a three-dimensional shape acquisition apparatus from a stereoscopic endoscope image. An endoscope insertion unit 1 has an image acquisition unit 10 and a pattern projection unit 20 integrated. The image acquisition unit 10 is provided with an image pickup device for picking up an image of an object to be inspected via a photographing lens on the distal end side of the insertion unit, and an operation signal to the image pickup device and wiring for transmitting the picked-up image signal are image acquisition units. 10 and the connection part 30 connected to this are connected to the control processing part 40.

  The pattern projection unit 20 is integrally arranged with the image acquisition unit 10 so that the tip is at the same position, 20A is a projection pattern storage unit, and wiring for supplying power for operating the pattern projection unit is connected to the pattern projection unit 20 It is provided so as to be connected to the control processing section through the section 30. Note that when the endoscope has a viewing angle, the tip of the pattern projection unit 20 corresponds to the tip of the image acquisition unit 10 according to the viewing angle so that the pattern image is projected on the entire field of view of the endoscope. You may reverse | retreat suitably. At this time, the tip of the image acquisition unit 10 (that is, a photographic lens 11L and 11R described later) is positioned in the same direction as the side where the pattern projection unit 20 is connected to the image acquisition unit 10 or in the opposite direction. The endoscope is installed so that it can have the viewing angle. For example, in FIG. 2, the pattern projection unit 20 is appropriately retracted and connected to the lower side of the image acquisition unit 10, and the tip of the image acquisition unit 10 (the photographic lenses 11 </ b> L and 11 </ b> R) is located below the image acquisition unit 10. The endoscope is installed in a direction (the same direction as the continuous side of the pattern projection unit 20) or upward (a direction opposite to the continuous side of the pattern projection unit 20) so that the endoscope can have the viewing angle. The

  FIG. 2 shows a state in which the image acquisition unit 10 and the pattern projection unit 20 are viewed from the front end side, and photographing lenses 11L and 11R that form an image on a pair of left and right imaging elements are attached to the front end of the image acquisition unit 10. An image sensor is disposed at the imaging position behind it. A projection lens 21 is attached to the distal end side of the pattern projection unit 20, and an image fiber for light transmission is provided behind the projection lens 21.

  FIG. 3 is a sectional view showing the configuration of only the pattern projection unit 20. In order to accurately reproduce the three-dimensional shape of the object to be inspected, it is important to obtain the original left and right images as clear as possible. When the object to be inspected is a nasopharynx, etc., it has movement and has a large unevenness, and the properties of the mucosal surface of the nasopharynx have poor characteristics on the image, so accurate three-dimensional shape acquisition Is difficult. Therefore, an active stereo method is used in which a pattern image is projected on the object to be inspected, and a three-dimensional shape is acquired based on an image obtained by imaging the object to be inspected on which the pattern image is formed.

  In FIG. 3, a light source L, a transmissive projection pattern P, and an imaging lens 22 are disposed in a light source housing portion 20 </ b> A connected to the rear portion of the pattern projection portion 20. An image fiber 23 is arranged between the imaging lens 22 and the projection lens 21 on the distal end side in the shape portion. When inspecting the object to be inspected, the projection pattern P is irradiated from behind by the light source L via the condenser lens 24, imaged by the imaging lens 22 on the incident end of the image fiber 23, and emitted from the image fiber 23. An image of the projection pattern P is projected onto the inspection target part through the projection lens 21 on the tip side from the end. The light source L is supplied with power through the wiring 25.

  The projection pattern is projected as an image having a side of a substantially square on the surface of the object S to be inspected, and in this range, the pattern image is projected and brightened, and observation and photographing are performed. Further, the object to be inspected has a maximum of d unevenness in the optical axis direction of the endoscope, and a clear pattern image is projected on the surface of the object to be inspected in the range of such unevenness.

  With respect to the object to be inspected on which the projection pattern image is projected, an image of the object to be inspected is acquired by a pair of imaging elements on the distal end side of the endoscope via the imaging lens, and the image signal is provided from the inside of the image acquisition unit through the connecting unit. The data is transmitted to the image signal processing unit in the control processing unit 40 via the transmission line. The image signal processing unit performs image analysis processing based on the acquired left and right images, and generates three-dimensional shape data of the object to be inspected.

  FIG. 4 shows a situation in which the inspected object S on which the projection pattern image is projected is captured by the image acquisition unit 10, and each imaging element 12L is captured by a pair of imaging lenses 11L and 11R on the distal end side of the image acquisition unit 10. , 12R, an image with parallax is formed. Reference numerals 13L and 13R denote wirings for transmitting image signals.

  The projection pattern is used for obtaining a three-dimensional shape based on an image formed on the surface of the object to be inspected. Basically, the projection pattern is a lattice-like regular and periodic pattern as shown in FIG. It is made into a shape, or it is made as shown in FIG. 5B in which the same point shape is arranged in a lattice shape, and is produced as a transmissive projection pattern. Since a clear image is projected on the surface of the object to be inspected, the projection pattern should be prepared to have a certain size, and the projection pattern should be exchanged according to the type of object to be inspected. It is good to make it.

  In order to form an image with a clear projection pattern over the inspection range of the surface-shaped object to be inspected, such as a nasopharynx, it is important to increase the imaging depth of the projection optical system. Further, since the endoscope distal end side is an insertion portion to the inspected object side and is limited in size, the projection lens 21 on the distal end side has a small diameter in the configuration of the pattern projection unit in FIG. , The F value is small, and the imaging depth of the projected image is increased. When the object to be inspected is a nasopharynx cavity, the degree of unevenness is about 10 mm or more, or about 20 mm, and this movement is about 30 mm parallel to the endoscope. Therefore, it is necessary to make the projected pattern image clear with respect to an object to be inspected with large movement.

  The diameter of the image fiber 23 is about 2 mm, and the diameter of the projection lens 21 at the tip of the pattern projection unit 20 is also about the same. By setting the effective diameter of the projection lens 21 to about 1.5 mm and setting the F value (focal length / effective diameter) to a value of about 5.6, a pattern image on the exit end face of the image fiber 23 is 10 by the projection lens 21. The image can be projected in such a manner that the magnification is about double and the imaging depth is 15 mm or more, preferably 20 mm or more, more preferably 20 to 30 mm. In order to accurately observe an inspected object image and acquire a three-dimensional shape, such a projection optical system can provide a clear projection over an uneven inspected object surface over a size range of 20 mm or more, preferably 30 mm or more. A pattern image is preferably formed.

  Since the effective diameter of the projection lens on the front end side is reduced and the F-number is increased, the amount of projection light is reduced. Therefore, it is necessary to increase the projection pattern and increase the brightness of the projection light source accordingly. Along with this, it is also necessary in terms of the amount of light projected. Therefore, for example, when the object to be inspected is located at a position 15 mm from the distal end of the endoscope, a light source having sufficiently high light emission luminance is provided so that a bright projection pattern image can be formed in a range of about 20 mm. It is necessary to do so.

[Shooting the image of the object to be inspected and acquiring the 3D shape]
A process of acquiring a three-dimensional shape by photographing a test object on which a pattern image is projected using a stereoscopic endoscope having a pattern image projecting unit according to the present invention will be described.
The stereoscopic endoscope includes an image pickup device that captures an image formed by a pair of lenses 11L and 11R at the distal end of the image acquisition unit 10, and the distal end of the stereoscopic endoscope is set at a position facing the object to be inspected. Then, the object to be inspected in a state where the pattern is projected by the pattern projection unit 20 is photographed to generate left and right images.

  A three-dimensional shape is acquired using a stereo matching algorithm for the captured images of the pair of left and right inspected objects. In stereo matching, rectify processing is performed as preprocessing for three-dimensional restoration in order to reduce processing time. The rectifying process is a process of converting an image based on the visual axis of a pair of imaging elements facing the subject into an image based on a virtual parallel visual axis. By performing the rectifying process, the stereo matching is simplified to a search problem on the same horizontal operation line of two images, and the calculation time is shortened.

The parallax due to stereo vision is obtained from the parallel stereo vision image corrected by the rectifying process. There are several methods for this, but here the birch field method is used. In the birch field method, the following two steps are performed.
(1) A matching process is performed on a pair of scan lines that coincide with each other on the left and right images. At this time, it is not affected by the adjacent scan line.
(2) For the disparity distribution obtained by the matching process performed on each scan line, information is propagated from an area with high parallax reliability to an area with low reliability to sort out the parallax of the entire screen.

Thereby, parallax is calculated | required from the image of a right-and-left image sensor, this is converted into distance and a depth is calculated | required. FIG. 6 shows the relationship between parallax and depth. In FIG. 6, it is assumed that one point on the object to be inspected is a gazing point, and VP-L and VP-R are the center positions (viewpoints) of the left and right imaging elements of the camera on the imaging plane. The main surface of the camera's photographic lens is a projection plane, the distance between the centers of the left and right image sensors is dc, the distance from the viewpoint to the projection plane is fl, and the actual depth is l. The parallax is expressed as ds on the projection plane as the difference between the distance between the two viewpoints and the interline-of-sight distance from the viewpoint to the gazing point on the projection plane.
From the parallax, the actual depth can be obtained by the following equation.
l = dc / ds × fl

  The 3D model can be generated by obtaining the parallax and the depth by the matching process of the left and right images and obtaining the three-dimensional coordinates of the object to be inspected. A flow of 3D model generation is shown in FIG. Image distortion correction is performed on images obtained by the left and right imaging elements, and then parallelization is performed on each of the left and right images by rectifying processing. Depth coordinates are calculated by matching the parallelized left and right images, and a 3D model is generated from the three-dimensional coordinate values.

  In the acquisition of a three-dimensional shape from a stereoscopic endoscope image according to the present invention, projection patterns are projected on a subject to be inspected, which has irregularities such as the nasopharynx and has a large movement, and is photographed by the left and right imaging elements. A stereo matching process is performed on the obtained image to obtain a three-dimensional shape. In the case of an inspected object such as the nasopharynx, it is characterized by unevenness and large movement, and this movement is the same direction as the insertion direction of the endoscope, that is, it “closes or moves away” from the endoscope tip. The movement in the front-rear direction is at least 20 mm. In the present invention, the imaging depth of the projection optical system of the projection pattern is set to 15 mm or more, so that there is unevenness and movement such as the nasopharynx, so that the object to be inspected with large movement can be visualized. A sharp pattern image can be projected on a moving object to be inspected about 30 mm in the front-rear direction of the mirror tip, and a three-dimensional shape can be accurately acquired by stereo matching processing.

1 Insertion section
DESCRIPTION OF SYMBOLS 10 Image acquisition part 11L, 11R Image pickup lens 12L, 12R Image pick-up element 13L, 13R Wiring 20 Pattern projection part 20A Projection pattern accommodating part 21 Projection lens 22 Imaging lens 23 Image fiber 24 Condensing lens 25 Wiring 30 Connection part 40 Control processing part L Light source P Projection pattern S Inspected object

Claims (4)

  An insertion portion formed in an elongated shape so as to be inserted toward the object to be inspected, including an image acquisition unit having a camera having a pair of image pickup devices disposed on the distal end side and a pattern projection unit having a projection lens on the distal end side And a three-dimensional shape acquisition device from a stereoscopic endoscope image, comprising: a connecting portion connected to the inserting portion; and a control processing portion connected to the connecting portion, wherein the pattern projecting portion is a tip of the inserting portion. A projection pattern storage unit in which a projection pattern and a light source for projection that illuminates the projection pattern at a position behind the projection pattern are disposed at a rear position away from the projection pattern; and image light from the projection pattern irradiated by the light source An image fiber that transmits the image light and a projection lens that projects the transmitted image light, and is configured to project an image of the projection pattern onto the position of the object to be inspected. A stereo matching process is performed on an image obtained by photographing the object to be inspected with a camera having a pair of imaging elements to obtain a three-dimensional shape of the object to be inspected, and the projection lens is 15 mm or more Apparatus for obtaining a three-dimensional shape from a stereoscopic endoscope image, having an F-number and a focal length having an imaging depth of   2. The three-dimensional image from the stereoscopic endoscope image according to claim 1, wherein the projection lens projects a pattern image having a size of at least 20 mm or more on the surface of the object to be examined on which the projection pattern image is projected. Shape acquisition device.   The apparatus for acquiring a three-dimensional shape from a stereoscopic endoscope image according to any one of claims 1 and 2, wherein the projection pattern arrangement unit can accommodate and arrange the projection patterns in an exchangeable manner. With respect to an image obtained by photographing the object to be inspected on which the projection pattern image is projected with a camera having a pair of image sensors, an image of the pair of image sensors facing the subject in the control processing unit is obtained by the visual axis. 4. A three-dimensional shape of an object to be inspected is obtained by performing stereo matching after performing a rectification process for converting into an image with a virtual parallel line of sight axis. A three-dimensional shape acquisition apparatus from the described stereoscopic endoscope image.

Images