JP2001264033A - Three-dimensional shape-measuring apparatus and its method, three-dimensional modeling device and its method, and program providing medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a three-dimensional shape measuring apparatus, capable of producing a distance image with high accuracy by simultaneously capturing a texture image and a light projection pattern image for distance measurement, in a stereoscopically imaging method. SOLUTION: A light projection pattern is irradiated with infrared rays, or the like, having wavelengths which are different from those of visible rays. The texture image with visible rays and a light projection pattern image with infrared rays, or the like, are captured simultaneously by a standard camera and a reference camera. A mismatching area is detected in a distance image obtained from the pattern images. By using the images captured simultaneously with infrared rays, a distant image having a partial area corresponding to the mismatching area is produced, and it is substituted for the mismatching area in the distance image based on the pattern images, thereby producing a more accurate distance image.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to a three-dimensional shape measuring device and method, a three-dimensional modeling device and method, and a program providing medium. Particularly suitable three-dimensional shape measurement device and method used when measuring the three-dimensional shape of the subject surface using images taken from a plurality of different positions of the object present in a distance range on the three-dimensional space,
The present invention relates to a three-dimensional modeling device and method, and a program providing medium.

[0002]

2. Description of the Related Art Conventional methods for obtaining a three-dimensional shape of an object are roughly classified into an active method (active method) and a passive method (passive method). Active methods include a method of projecting light and measuring the time until the light is reflected back to determine the distance to each object to be measured, or a method of applying slit-shaped pattern light to the object to be measured. There is a method such as a light cutting method for measuring a three-dimensional shape by examining the shape of the pattern light projected on the target to be measured.

[0003] A typical passive method is as follows.
This is a stereo image method that uses the principle of triangulation, and is a method of measuring a distance by using two or more cameras to find a corresponding point between the images to obtain parallax.
For stereo image method, for example, Kinade, Nakahara, Okutomi "3D measurement by multi-baseline stereo method",
Image Lab, pp 53-57.

The principle of the stereo image method will be briefly described. The stereo image method uses two or more cameras.
The position of a measurement target in a three-dimensional space is to be obtained by associating pixels in a plurality of images obtained by photographing the same target from more than one direction (different line-of-sight directions). For example, the same object is photographed from different directions by the reference camera and the reference camera, and the distance between the measurement objects in each image is measured based on the principle of triangulation.

FIG. 1 is a diagram for explaining the principle of the stereo image method. The reference camera (Camera 1) and the reference camera (Camera 2) photograph the same object from different directions. "M" in the image taken by the reference camera
Consider finding the depth of the point "b".

[0006] An object that appears at a point "mb" in an image captured by the reference camera is located at a position in the depth direction (p1, p).
According to (2, p3), in an image captured by a reference camera capturing the same object from different directions,
It is developed on a certain straight line such as "m1,""m2," and "m3." This straight line is connected to the epipolar line (E
(pipolar line) Lp.

[0007] The position of the point "mb" in the reference camera appears on a straight line called an "epipolar line" in the image taken by the reference camera. Point P to be imaged
(Points existing on a straight line including P1, P2, and P3) are the same observation point “mb” on the reference image regardless of the depth, that is, the distance from the reference camera, as long as they are on the line of sight of the reference camera. Appears in On the other hand, the point P on the image captured by the reference camera appears on the epipolar line at a position corresponding to the magnitude of the distance between the reference camera and the observation point P.

FIG. 1 illustrates a correspondence between an epipolar line and an observation point “mb” in a reference image. As shown in the figure, as the position of the observation point P changes to P1, P2, and P3, the observation point in the reference image is “m1”,
Shift to “m2” and “m3”.

Using the above geometrical optical properties, the distance to the point P can be identified by searching for a point corresponding to the observation point "mb" on the reference image on the epipolar line. This is the basic principle of the “stereo image method”. In this way, three-dimensional information on all pixels on the screen is obtained. The acquired three-dimensional information can be used as pixel attribute data corresponding to each pixel.

Although the above-described stereo image method uses one reference camera and one reference camera, a multi-baseline stereo (Multi) using a plurality of reference cameras is used.
An evaluation value may be obtained by an iBaseline Stereo) method, and three-dimensional information for each pixel may be obtained based on the evaluation value. The multi-baseline stereo image method uses images taken by one reference camera and a plurality of reference cameras, obtains an evaluation value representing a correlation between each of the plurality of reference camera images and the reference camera image, and calculates each evaluation value. The sum is added and the sum is used as a final evaluation value. For details of the multi-baseline stereo image method, see, for example, “Stereo matching using a plurality of baseline lengths”, IEICE Transactions D
-IIVol. J75-D-II No. 8 pp. 1
317-1327, August 1992.

As described above, the stereo image method associates pixels in a plurality of images obtained by photographing the same object from two or more directions (different line-of-sight directions) using a plurality of cameras; That is, the position of the measurement object in the three-dimensional space is obtained by performing “corresponding scoring”.

Conventionally, the method of “corresponding scoring” that is often used is roughly classified into pixel-based matching, area-based matching, and feature-based matching. Pixel-bas
ed matching is a method of searching for the correspondence of points in one image as it is in the other image (C. Lawrence Z
itnick and Jon A. Webb: Multi-baseline Stereo Usin
g Surface Extraction, Technical Report, CMU-CS-96-
196, (1996)).

Area-based matching is a method of searching for a correspondence of a point in one image by using a local image pattern around the point when searching the other image (Okutomi, Kinide: Stereo matching using multiple baseline lengths, IEICE Transactions D-II, Vol.J75-D-II,
No.8, pp.1317-1327, (1992), Yokoyama, Miwa, Ashigahara, Koyanatsu, Hayashi, later: Stereo Camera System and Its Applic
ation, SRF'97, (1997), Kanade, Kimura: Video Rate Stereo Machine, Journal of the Robotics Society of Japan, Vol.13, N
o.3, pp.322-326, (1995), Kinade, Mosquito Field, Kimura, Kawamura,
Yoshida, Oda: Development of Video Rate Stereo Machine, Journal of the Robotics Society of Japan, Vol.15, No.2, pp.261-267, (199
7), Yamaguchi, Takachi, Iguchi: Stereo matching for stone image measurement using adaptive window method, Humanities and Computer, Vol.32, No.10, pp.55-60, (1996), Yokoya: Recent Special Issue on Signal Processing in Computers, Recent Topics on Computer Vision, Systems / Control / Information, Vol.38, No.8, pp.436
441, (1994)).

[0014] Feature-based matching is a method of extracting features such as shaded edges from images and associating them using only features between images (HHBaker and TOBi).
nford: Depth from edge and intensity based stereo,
In Proc. IJCAI'81, (1981), Ishiyama, Kakuho, Kawai, Ueshiba, Tomita: Searching for Correspondence Candidates in Segment-Based Stereo, IEICE Technical Report, Vol.96, No.136, (1997), WEL
Grimson: Computationalexperiments with a feature b
ased stereo algorithm, IEEE Trans. PAMI, Vol. 7, N
o.1, pp.17-34, (1985)).

According to the various methods described above, the corresponding pointing process in the stereo image method is performed. However,
For objects with few features (shades, shapes, colors, etc.) such as white walls and human faces, the corresponding points required to measure the three-dimensional shape using any of the above methods The attachment process becomes difficult. This is one of the major issues in processing by the stereo image method.

In order to more accurately perform an associating process on an object having almost no characteristic (shade, shape, color, etc.) such as a white wall or a human face, a periodic process such as a checker or a frequency-modulated sine wave is performed. Several techniques have been disclosed for reducing errors in image matching by a multi-baseline stereo system by projecting a simple pattern (SBKang, JAWebb, CLZitnick and T. Kana).
de: A Multibaseline Stereo System with Active Illu
mination and Real-time Image Acquisition, ProcIEEE
Int Conf.Comput.Vis., Vol.5, pp.88-93, (199
5), M. Noguchi and SKNayar: Microscopic Shape fro
m Focus Using Active Illumination, Proc Int. Conf.
Pattern Recogn., Vol.12, No.Vol 1, pp.147-152,
(1994)).

[0017]

However, in the above-mentioned conventional method, it was not possible to completely eliminate an error in the matching process, that is, a mismatch. Further, the above-described configuration of projecting the frequency-modulated periodic pattern and executing the associating process reduces the mismatching area in a multi-baseline stereo machine using a plurality of (three or more) cameras. However, for binocular stereo vision using two cameras, there is a possibility that an error occurs in associating similar patterns with each other because the generated patterns are periodic, There is a disadvantage that it is not always effective.

As another technique, there is a three-dimensional measurement method called a spatial code method. The spatial code method requires that (1) regular pattern light must be projected onto an object to be measured; 2) distance measurement cannot be performed unless the positional relationship between the light emitting system and the camera is strictly maintained; (3) distance images cannot be obtained unless plural regular patterns are always projected; (4) Since there are various restrictions such as the necessity of projecting a code pattern precisely, there are many disadvantages such as a complicated light projecting device and complicated measurement processing.

The present invention provides a three-dimensional shape measuring apparatus and a method thereof for solving the above-mentioned problem, and a three-dimensional modeling apparatus and a method using the three-dimensional shape measuring apparatus. This achieves a configuration for synchronously capturing a light projection pattern image.

That is, in the stereo image method, it is possible to simultaneously irradiate the texture image and the light projection pattern image for distance measurement with light of different wavelengths, and to simultaneously capture them separately. Specifically, for example, a photographing mechanism that captures a texture image with visible light and a light-projection pattern image for distance measurement with infrared light is configured so that the texture image and the light-projection pattern image can be captured in one photographing process. The present invention realizes a three-dimensional (3D) shape measuring device that captures images at the same time.

[0021]

Means for Solving the Problems The present invention has been made in consideration of the above problems, and a first aspect of the present invention is to provide a three-dimensional image of the object to be measured using images taken from different directions. In a three-dimensional shape measurement device that measures the shape,
A projection pattern irradiating unit that irradiates the measurement target with a projection pattern using light having a wavelength different from visible light, a projection pattern image irradiated with the projection pattern by the projection pattern irradiating unit, and a visible light A reference camera that separates and synchronizes the two projected images of the texture image observed by the image separating unit and captures the synchronized light projection pattern image and the texture image separately, and the illumination of the light projection pattern by the illumination pattern irradiation unit The projected light pattern image and the texture image, which are obtained by separating and synchronizing two images of the texture image observed by visible light by the image separating means, from a direction different from the reference camera and the reference image, A reference camera which is taken at a timing synchronized with the camera, and which is taken by the reference camera and the reference camera. Distance image based on the light pattern image,
And a distance image generating means for generating a distance image based on a texture image captured by the reference camera and the reference camera; and detecting a mismatched area in the distance image based on the light projection pattern image generated by the distance image generating means. A mismatching area detecting means, and a replacing means for replacing a distance image of a partial area including the mismatching area detected by the mismatching area detecting means with a distance image based on a texture image. It is in the shape measuring device.

Further, in the three-dimensional shape measuring apparatus according to the present invention, the light projection pattern irradiated by the light projection pattern irradiation means is a light projection pattern by infrared light.

Further, in the three-dimensional shape measuring apparatus according to the present invention, the light projection pattern irradiated by the light projection pattern irradiation means is an aperiodic light projection pattern.

Further, in the three-dimensional shape measuring apparatus according to the present invention, the image separating means includes a spectroscope and a filter arranged on a path separated by the spectrometer to cut off light of a specific wavelength. It is characterized by having.

Further, in the three-dimensional shape measuring apparatus according to the present invention, the mismatching area detecting means compares a difference between pixel values of adjacent pixels in the distance image with a predetermined threshold value, and the difference is smaller than the threshold value. The present invention is characterized in that a labeling process for giving the same label to pixels is executed, and a process for detecting a small area with a different label and surrounded by the same label area as a mismatching area is executed.

Further, in the three-dimensional shape measuring apparatus according to the present invention, the distance image generating means generates a distance image based on the texture image only for a partial region including a mismatching region in the distance image based on the light projection pattern image. It is characterized by generating.

Further, in the three-dimensional shape measuring apparatus according to the present invention, the replacement means compares the corresponding pixel values of the distance image based on the texture image and the distance image based on the light projection pattern to determine the distance image based on the texture image. It is characterized by having a configuration in which a reliability determination is executed and a distance image based on a texture image for which reliability has been confirmed is applied as replacement data for a distance image based on a light projection pattern.

Further, the three-dimensional shape measuring apparatus of the present invention
A plurality of pairs of reference cameras and reference cameras, wherein the distance image generating means includes a plurality of distance images based on a light projection pattern image captured by each pair of the reference cameras and reference cameras; and And a configuration for generating a plurality of distance images based on a texture image captured by each set of the reference camera and the reference camera.

Further, the three-dimensional shape measuring apparatus of the present invention
Having a plurality of reference camera and reference camera pairs,
It has a plurality of projection pattern irradiation means, each of the plurality of projection pattern irradiation means is different from visible light, and uses a light of a unique wavelength different from other light projection pattern irradiation means to form a projection pattern. It has a configuration to irradiate the measurement target, each set of the plurality of reference camera and reference camera sets, separately captures a projection pattern image based on light of different wavelengths from a texture image, and captures the distance. Image generating means, a plurality of distance images based on the light projection pattern image captured by each set of the reference camera and the reference camera,
And a configuration for generating a plurality of distance images based on a texture image captured by each set of the reference camera and the reference camera.

Further, a second aspect of the present invention relates to a three-dimensional shape measuring method for measuring a three-dimensional shape of a measurement object using images taken from different directions of the measurement object, wherein the method has a wavelength different from that of visible light. A light projection pattern irradiating step of irradiating the measurement target with a light projection pattern using light; and an image separation of two images of a light projection pattern image irradiated with the light projection pattern and a texture image observed by visible light. The projection pattern image and the texture image separated and synchronized by the means are separately captured by the reference camera, and the projection pattern image synchronized by the reference camera from a different direction from the reference camera and at a timing synchronized with the reference camera. Image capturing step of individually capturing a texture image and a texture image, and capturing by the reference camera and the reference camera. A first distance image generating step of generating a distance image based on the projected light projection pattern image, and a mistake detecting a mismatching region in the distance image based on the light projection pattern image generated in the first distance image generating step. A matching area detecting step, a second distance image generating step of generating a distance image of a partial area including the mismatching area detected by the mismatching area detecting step based on the texture image,
The partial area including the mismatched area of the distance image based on the projection pattern image generated in the distance image generating step is replaced with a distance image based on the texture image of the partial area generated in the second distance image generating step. And a replacement step.

Further, in the three-dimensional shape measuring method according to the present invention, the light projection pattern irradiated by the light projection pattern irradiation means is a non-periodic light projection pattern by infrared light.

Further, in the three-dimensional shape measuring method according to the present invention, in the mismatching area detecting step, a difference between pixel values of adjacent pixels in the range image is compared with a predetermined threshold value, and the difference is smaller than the threshold value. The method includes a labeling processing step of assigning the same label to pixels, and a detection step of detecting a small area with a different label and surrounded by the same label area as a mismatching area.

Further, in the three-dimensional shape measuring method according to the present invention, the replacing step includes comparing a distance image based on the texture image with a corresponding pixel value of the distance image based on the light projection pattern to determine a distance image based on the texture image. A reliability determination is performed, and a distance image based on the texture image whose reliability has been confirmed is applied as replacement data for the distance image based on the light projection pattern.

Further, a third aspect of the present invention is a three-dimensional shape measuring method for measuring the three-dimensional shape of the object to be measured using images taken of the object from different directions. A light projection pattern irradiating step of irradiating the measurement target with a light projection pattern using light; and an image separation of two images of a light projection pattern image irradiated with the light projection pattern and a texture image observed by visible light. The projection pattern image and the texture image separated and synchronized by the means are separately captured by the reference camera, and the projection pattern image synchronized by the reference camera from a different direction from the reference camera and at a timing synchronized with the reference camera. Image capturing step of individually capturing a texture image and a texture image, and capturing by the reference camera and the reference camera. A plurality of distance image parallel generations in which a distance image based on a textured image captured by the reference camera and the reference camera is generated in parallel as a second distance image, using a distance image based on the projected light projection pattern image as a first distance image. A step of detecting a mismatching area in a distance image of at least one of the first distance image and the second distance image generated in the multiple distance image parallel generation step; and detecting the mismatching area. A replacement step of replacing the distance image of the partial area including the mismatched area in one of the distance images detected by the step with the other distance image,
A three-dimensional shape measuring method characterized by having:

Further, in the three-dimensional shape measuring method according to the present invention, the light projection pattern irradiated by the light projection pattern irradiation means is a non-periodic light projection pattern by infrared light.

Further, in the three-dimensional shape measuring method according to the present invention, in the mismatching area detecting step, a difference between pixel values of adjacent pixels in the distance image is compared with a predetermined threshold value, and the difference is smaller than the threshold value. The method includes a labeling processing step of assigning the same label to pixels, and a detection step of detecting a small area with a different label and surrounded by the same label area as a mismatching area.

Further, in the three-dimensional shape measuring method according to the present invention, the replacing step includes comparing a distance image based on the texture image with a corresponding pixel value of the distance image based on the light projection pattern to calculate the distance image based on the texture image. A reliability determination is performed, and a distance image based on the texture image whose reliability has been confirmed is applied as replacement data for the distance image based on the light projection pattern.

Further, a fourth aspect of the present invention is a three-dimensional modeling apparatus for measuring a three-dimensional shape of a measurement object using images obtained by photographing the measurement object from different directions and performing three-dimensional modeling. A light projection pattern irradiating means for irradiating the measurement target with a light projection pattern using light having a wavelength different from that of light, a light projection pattern image irradiated with the light projection pattern by the light projection pattern irradiating means, and observing with visible light. A reference camera that separates and synchronizes the two projected texture images by an image separating unit and separately captures a synchronized light projection pattern image and a texture image; and a projection camera irradiated with the light projection pattern by the light projection pattern irradiation unit. A light projection pattern in which two images, a light pattern image and a texture image observed by visible light, are separated and synchronized by image separation means. Camera and a texture image from different directions from the reference camera and at a timing synchronized with the reference camera, and a distance image based on the projection pattern image captured by the reference camera and the reference camera A distance image generating means for generating a distance image based on a texture image captured by the reference camera and the reference camera; and a mismatching area in the distance image based on the light projection pattern image generated by the distance image generating means. A mismatching area detecting means for detecting, and a replacing means for replacing a distance image of a partial area including the mismatching area detected by the mismatching area detecting means with a distance image based on a texture image,
A three-dimensional shape data is generated based on a composite distance image of a distance image based on a light projection pattern image replaced and generated by the replacement unit and a distance image based on a texture image, and a texture image captured by the reference camera. And a three-dimensional shape data generating means.

Further, according to a fifth aspect of the present invention, there is provided a three-dimensional modeling method for measuring a three-dimensional shape of the measurement object using images obtained by photographing the measurement object from different directions and performing three-dimensional modeling. A light projection pattern irradiating step of irradiating the measurement target with a light projection pattern using light having a wavelength different from that of light, a light projection pattern image irradiated with the light projection pattern, and a texture image observed by visible light. The light projection pattern image and the texture image which are separated and synchronized by the image separation means are separately captured by the reference camera, and are synchronized by the reference camera from a different direction from the reference camera and at a timing synchronized with the reference camera. Capturing an image of the projected light pattern image and the texture image separately, and the reference camera First generating a distance image based on the projection pattern image interleaved taken by a preliminary reference camera
A distance image generating step, a mismatching area detecting step of detecting a mismatching area in the distance image based on the light projection pattern image generated in the first distance image generating step, and a mismatch detected by the mismatching area detecting step. A second distance image generating step of generating a distance image of a partial area including the area based on the texture image; and a mismatching area of the distance image based on the light projection pattern image generated in the first distance image generating step. A replacement step of replacing the partial area with a distance image based on the texture image of the partial area generated in the second distance image generation step; and a distance image based on the light projection pattern image generated and replaced by the replacement step. Of the distance image based on the texture image A three-dimensional shape data generating step of generating three-dimensional shape data based on a synthetic distance image and a texture image captured by the reference camera.

According to a sixth aspect of the present invention, there is provided a three-dimensional modeling method for measuring a three-dimensional shape of a measurement object using images obtained by photographing the measurement object from different directions and performing three-dimensional modeling. A light projection pattern irradiating step of irradiating the measurement target with a light projection pattern using light having a wavelength different from that of light, a light projection pattern image irradiated with the light projection pattern, and a texture image observed by visible light. The light projection pattern image and the texture image which are separated and synchronized by the image separation means are separately captured by the reference camera, and are synchronized by the reference camera from a different direction from the reference camera and at a timing synchronized with the reference camera. Capturing an image of the projected light pattern image and the texture image separately, and the reference camera And a distance image based on the projection pattern image captured by the reference camera as a first distance image, and a distance image based on a texture image captured by the reference camera and the reference camera as a second distance image. Multiple distance image parallel generation step, a mismatching area detection step of detecting a mismatching area in at least one of the first distance image or the second distance image generated in the multiple distance image parallel generation step, A replacement step of replacing the distance image of the partial area including the mismatched area in one of the distance images detected by the mismatched area detection step with the other distance image, and a light projection pattern image replaced and generated by the replacement step Of Range Image Based on Image and Texture Image Away images, and the 3D modeling method characterized by having a three-dimensional shape data generating step of generating a three-dimensional shape data based on the texture image interleaved taken by the reference camera.

Further, according to a seventh aspect of the present invention, there is provided a computer system for executing a process for performing three-dimensional shape measurement for measuring the three-dimensional shape of the measurement object using images obtained by photographing the measurement object from different directions. A program providing medium that tangibly provides a computer program to be executed on the computer program, wherein the computer program irradiates the measurement target with a light projection pattern using light having a wavelength different from visible light. An irradiating step, an illuminated pattern image illuminated with the illuminated pattern, and a texture image observed by visible light are separated by an image separating unit and synchronized to synchronize the illuminated pattern image and the texture image with a reference camera. To capture images individually from different directions and synchronized with the reference camera. An image capturing step of separately capturing a light projection pattern image and a texture image synchronized by a reference camera in a shooting, and a first distance for generating a distance image based on the light projection pattern image captured by the reference camera and the reference camera. An image generating step;
(1) a mismatching area detection step of detecting a mismatching area in the distance image based on the light projection pattern image generated in the distance image generation step, and a distance image of a partial area including the mismatching area detected by the mismatching area detection step Generating a second distance image based on the texture image, and a partial area including a mismatched area of the distance image based on the projection pattern image generated in the first distance image generating step. A replacement step of replacing the partial area generated in the image generation step with a distance image based on a texture image.

The program providing medium according to the seventh aspect of the present invention is, for example, a medium for providing a computer program in a computer-readable format to a general-purpose computer system capable of executing various program codes. . The form of the medium is not particularly limited, such as a storage medium such as a CD, an FD, and an MO, and a transmission medium such as a network.

Such a program providing medium defines a structural or functional cooperative relationship between the computer program and the providing medium for realizing a predetermined computer program function on a computer system. Things. In other words, by installing the computer program into the computer system via the providing medium, a cooperative operation is exerted on the computer system, and the same operation and effect as the other aspects of the present invention can be obtained. You can do it.

Still other objects, features and advantages of the present invention are:
It will become apparent from the following more detailed description based on the embodiments of the present invention and the accompanying drawings.

[0045]

DESCRIPTION OF THE PREFERRED EMBODIMENTS [3D Shape Measuring Apparatus-First Embodiment] FIG. 2 shows a configuration example of a 3D shape measuring apparatus to which the present invention is applied. The three-dimensional shape measuring apparatus of the present invention measures a three-dimensional shape of a measurement target by applying the stereo image method described above, and has a three-dimensional shape, for example, for a measurement target object such as a human face. An image is captured using two cameras arranged at different angles with respect to each other, that is, a so-called reference camera and a reference camera, and based on the images captured by these two cameras, the image is captured based on the stereo image method described in FIG. This is for measuring the surface shape of the measurement object.

The light projecting pattern irradiating means 220 of the three-dimensional shape measuring apparatus of the present invention measures the light to be measured by light having a wavelength different from visible light such as sunlight, fluorescent light, incandescent light such as infrared light. It is configured as a light emitting pattern irradiating means 220 for irradiating the object 200, and it is desirable that the light emitting pattern to be irradiated is an aperiodic light emitting pattern based on a uniform random number or a normal random number.

The light to be measured 2
The non-periodic light projection pattern that is projected to the light source 00 has no periodicity in, for example, dot size, line length, thickness, position, density, and the like. For example, FIG.
A pattern as shown in FIG. These non-periodic patterns can be obtained by previously forming on a slide or the like and irradiating light having a wavelength different from visible light, for example, infrared light, through the slide. By forming such a non-periodic pattern on the surface of the measurement target 200 and capturing the image, it is possible to easily associate the image data of the reference camera and the reference camera when generating the distance image.

FIG. 4 shows a specific example of the structure of the light projecting pattern irradiating means 220. FIG. 4 shows an example of a light projection pattern irradiation unit using infrared LED illumination. Irradiation of the infrared light emitted from the LED array 401 in which a plurality of LEDs are arranged is reduced through the light diffusion plate 402 to reduce unevenness in illumination. Further, the slide 40 is provided via the condenser lens 403.
4. The measurement target 410 is irradiated via the light projecting lens 405.

The slide 404 is a slide on which an aperiodic pattern is recorded. As described above, there is no periodicity in dot size, line length, thickness, position, density, etc., so-called non-periodic light projection pattern, for example, FIG.
It is a slide on which a pattern as shown in FIG.

The LED array 401 includes the light emission drive circuit 40
The light emission drive circuit 407 is driven by the shutter synchronization circuit 408 to output the reference camera 20 shown in FIG.
1. Control is performed so as to be synchronized with the timing of capturing an image by the reference camera 202, that is, the shutter timing.

In the configuration of the present invention in which the non-periodic pattern is irradiated on the object to be measured by infrared light, the pattern can be continuously irradiated for a predetermined time, and the illumination unevenness is reduced as compared with the configuration in which the pattern image is captured by flash light. In the configuration of the present invention, since it is possible to distinguish an image (texture image) by visible light by using infrared light, an image capture of a light projection pattern image (infrared light) and a texture image (visible light) is performed. Therefore, it is not necessary to switch the shutter speed, a non-periodic pattern having a uniform intensity can be formed over the entire measurement target, and a light projection pattern image can be captured by the reference camera and the reference camera. Therefore, an accurate distance image can be generated.

Returning to FIG. 2, the description of the configuration of the three-dimensional shape measuring apparatus of the present invention will be continued. The reference camera 201 and the reference camera 202 are arranged in different line-of-sight directions with respect to the measurement target 200, and capture images in each line-of-sight direction. Images captured by the reference camera 201 and the reference camera 202 are captured at synchronized timing. Images captured from the reference camera 201 and the reference camera 202 are respectively transmitted to the frame memories 203 to 20 via the A / D converter.
6 is stored.

Different image data is taken into two frame memories by the reference camera 201 and the reference camera 202, respectively. That is, a projection pattern image by infrared light is captured on one side, and a texture image by visible light is captured on the other side. This configuration will be described in detail with reference to FIG.

FIG. 5 is a block diagram showing a configuration in which each of the reference camera 201 and the reference camera 202 realizes capturing of two images at one image capturing timing.

The lens 50 of the reference camera is placed on the measurement object 200.
The lens 1 and the lens 507 of the reference camera are respectively directed from different directions, and a pattern of light having a wavelength different from visible light, here an infrared light, is projected and irradiated from the light projection pattern irradiating means 220 to the measurement target. The wavelength of the light emitted from the projection pattern irradiating means 220 in the three-dimensional shape measuring apparatus of the present invention is different from the wavelength of visible light used for capturing a texture image, that is, the projection pattern image is separated from the texture image. Any wavelength may be used as long as it can capture images. Here, the description will be made assuming that infrared light is used.

The object 200 to be measured is photographed as an image illuminated with the above-described infrared light pattern, and is placed in an environment where a visible image can be photographed by so-called environmental light such as sunlight, a fluorescent lamp, an incandescent lamp, or the like. . An image captured at a timing synchronized by the reference camera lens 501 and the reference camera lens 507 is split into two optical paths via respective spectroscopes 502 and 508, and the visible light cut filter 50
In the optical path where the light projection patterns 3 and 509 are installed, visible light is cut by the visible light cut filters 503 and 509, and the light projection pattern images 520 and 523 based on the infrared light irradiation pattern projected and irradiated by the light projection pattern irradiation means 220. Is captured.

That is, the light projection pattern images 1,520
Is a reference camera lens 5 that illuminates the measurement target irradiated with the infrared light projection pattern irradiated by the light projection pattern irradiation means 220.
01 is an image captured by the reference camera lens 507, and the light projection pattern images 2 and 523 are images captured by the reference camera lens 507 of the measurement target irradiated with the infrared light projection pattern irradiated by the light projection pattern irradiation unit 220. It is.

On the other hand, infrared light cut filters 505 and 51
In each installed optical path, the infrared light cut filter 50
The infrared light is cut by 5, 511, and the measurement target 200 is measured by so-called environmental light such as sunlight, fluorescent light, incandescent light, and the like.
Observed images of the CCD elements 506, 51
It is shot in 2.

That is, the texture images 1 and 521 are
A reference camera lens 501 captures an image of a measurement target irradiated with so-called environmental light such as sunlight, a fluorescent light, an incandescent light, and the like. Texture images 2 and 522 include sunlight,
The reference camera lens 507 captures an image of a measurement target irradiated with so-called environmental light such as a fluorescent lamp or an incandescent lamp.

Image Correlation & Distance Image Generation Means 530
Executes image association and generation of a distance image using a stereo image method based on the two light projection pattern images 520 and 523. The image association & distance image generation unit 531 executes image association and generation of a distance image using a stereo image method based on the two texture images 521 and 522. As will be described later, the generation process of the distance image based on the two texture images 521 and 522 by the image association & distance image generation unit 531 is performed by the light projection pattern image 520 by the image association & distance image generation unit 530. This may be performed as a supplementary process for the distance image generated based on 523, and it is not always necessary to generate the distance image for all pixel regions of the captured image.

The image association and distance image generation means 530,
The processing in 531 and the subsequent processing will be described with reference to FIG.

The frame memories 1a and 203 shown in FIG.
, A projection pattern image captured by the reference camera 201, a texture image captured by the reference camera 201 in the frame memories 1b and 204, and a texture image captured by the reference camera 202 in the frame memories 2a and 205. Light projection pattern image, frame memories 2b, 2
06 stores a texture image captured by the reference camera 202.

The image-to-image associating means A and 207 perform an associating process between the projection pattern images of the reference cameras and the reference cameras in the frame memories 1a and 203 and the frame memories 2a and 205. That is, a process of associating each pixel of the image captured by the reference camera 201 with a pixel of the reference image captured by the reference camera 202 is performed, and disparity data of the reference image with respect to the reference image is stored in the distance image generation unit A
209.

The light projection pattern emitted by the light projection pattern irradiating means 220 is a non-periodic pattern as described above, and it is relatively easy to associate images taken by each camera. However, the projection pattern irradiation means 2
20 is light (infrared light) irradiation from one direction, the camera may catch the specular reflected light of the irradiation light from the positional relationship between the projection pattern irradiation means 220, the measurement target 200, and the camera. In some cases, a region where the pixel value is saturated and the association process cannot be performed may occur. Also,
Depending on the surface shape of the measurement target 200, a partial periodicity may occur in the light projection pattern, and an accurate distance image may not be generated in such a region.

In the three-dimensional shape measuring apparatus of the present invention, such a region, that is, a region where a distance image is not accurately generated based on a light projection pattern image is detected, and the frame memories 1b and 204 and a frame Memory 2
b, a distance image based on the texture image stored in 206 is generated, complement processing is performed, and an accurate distance image is output.

The distance image generation means 209 shown in FIG. 2 firstly generates the light projection pattern reference image and the light projection pattern reference image based on the infrared non-periodic pattern light input from the inter-image correspondence means A and 207. A distance image is generated using the parallax data of each pixel. The distance image is a reference camera 20 for each pixel of the measurement target image captured by the reference camera 201.
It is an image in which the distance between 1 and the measurement target 200 is reflected.
The distance image generated by the distance image generation unit 209 is output and held in the image memory 210.

The mismatching area detecting means 211 reads out a distance image based on the light projection pattern image from the image memory 210 and executes a mismatching area detecting process by labeling. The mismatching area detection processing by labeling will be described in detail later.

The pixel position information of the mismatching area detected by the mismatching area detecting means 211 is output to the distance image generating means 209 and the replacing means 212.

The distance image generation means 209 generates a distance image for the partial area including the mismatched pixel area detected by the mismatched area, based on the correlation processing data of the image correlation means B and 208.

The frame memories 1b and 204 and the frame memories 2b and 206 correspond to the reference camera 201 and the reference camera 2 respectively.
02 stores an image photographed based on visible light, that is, a texture image.
8 generates parallax data based on the two texture images, and the distance image generation unit 209 generates a distance image of the partial region including the above-described mismatched pixel region based on the two texture images.

The distance image of the partial area including the mismatching pixel area generated based on the two texture images is output to the image memory 210 via the replacing means 212, and is output to the mismatching area in the distance image based on the light projection pattern. Is replaced with a distance image generated based on the texture image. Specifically, the mismatching region occurs, for example, in a regular reflection region.

The replacement means 212 is based on a distance image (a distance image based on a light projection pattern image) at pixel values around the mismatching area detected by the mismatching area detection means 211 and a texture image newly generated. After determining the continuity with the distance data of the partial region, the replacement is performed on the distance image data of each pixel while ensuring the consistency and output to the image memory 210.

The replacement means 212 corresponds to the pixel values (distance data) around the mismatching area in the partial distance image generated based on the texture image and the distance image previously generated based on the light projection pattern image. The difference between the pixel value and the pixel value is compared with a preset threshold value, and when the difference is smaller than the threshold value, it is determined that a distance image based on the texture image can be used as replacement data, and replacement processing is performed.
If the difference is larger than the threshold value, the region recognized as the mismatching region may be an extremely uneven portion existing in the measurement target, and in this case, the replacement process of that portion is not performed.

Since the light projection pattern image and the texture image captured by each camera are based on the same captured image, the distance image based on the light projection pattern image and the distance image based on the texture image are The pixels basically correspond to each other, and the replacement unit 212 can easily execute the replacement process.

The distance image stored in the image memory 210 is a composite image obtained by replacing the mismatched area in the distance image based on the projection pattern image with the distance image based on the texture image, and outputs the distance image as a more accurate distance image. Is output via.

The processing in each block having the configuration shown in FIG. 2 and the control of data transfer between blocks are controlled by control means (not shown), and predetermined memories, for example, semiconductor memories such as RAM and ROM, magnetic disks, and optical disks And the like can be controlled by a control program recorded in a storage medium such as. Further, the user can input commands, parameters, and the like using input means (not shown) to change each control mode.

In FIG. 2, the inter-image associating means A and 207 and the inter-image associating means B and 208 are shown as separate blocks. However, these may be configured as one processing block. Frame memory 1a, 203
After the image association processing of the frame memories 2a and 205, the image association processing of the frame memories 1b and 204 and the frame memories 2b and 206 is executed serially in the same processing block as necessary. Is also good.

FIG. 6 is a flowchart illustrating a method for measuring a three-dimensional shape using the three-dimensional shape measuring apparatus of the present invention. Hereinafter, each step in the flow of FIG. 6 will be described.

In step 601, a distance image D1 is obtained based on the light projection pattern image 1 captured by the reference camera and the light projection pattern image 2 captured by the reference camera.
(I, j) is calculated. The calculation of the distance image is shown in FIG.
Is generated by the distance image generation means 209 based on the parallax data generated by the image association means A and 207 in.

Next, in step 602, detection of a mismatching area in the distance image generated based on the light projection pattern image is executed. The detection process of the mismatching region will be described with reference to FIGS.

FIG. 7 shows an outline of the process of detecting a mismatch. In step 701, images from different viewpoint directions are taken in by two cameras, that is, a reference camera and a reference camera, and a distance image is generated in step 702. Then, in step 703, labeling processing of the distance image is executed, and a mismatching area is detected in step 705 based on the resulting labeling image 704.

FIG. 8 shows a detailed processing flow of the labeling processing in step 703 shown in FIG. First, in step 801 of FIG. 8, the mismatching area detecting unit 211 shown in FIG.
Read 1

In step 802, the mismatching area detecting means 211 initializes the number k of unprocessed pixels for labeling as initialization processing for setting coordinates at which labeling processing is started.
1 is a built-in coordinate storage buffer x [k], y [k]
Is initialized.

In step 803, the mismatching area detection means 211 sets arbitrary position coordinates of a portion corresponding to the measurement target (subject) in the distance image D1 as coordinates (i, j) of the pixel of interest.

At step 804, the mismatching area detecting means 211 extracts the pixel value g (i, j) of the target pixel of the distance image D1. The pixel value g (i, j) indicates a distance value of the pixel (i, j).

Next, at step 805, the mismatching area detecting means 211 outputs the pixel value g (i + i) of one unprocessed pixel among the pixels adjacent to the target pixel (i, j).
m, j + n) (where m, n = -1, 0, 1).

In step 806, the mismatching area detecting means 211 determines whether or not the pixel value g (i, j) of the pixel of interest (i, j) is the label value L0, and further determines the pixel of interest. It is determined whether or not the absolute value of the difference between the pixel value g (i, j) of the pixel and the pixel value g (i + m, j + n) of the adjacent pixel is smaller than a predetermined threshold.

By comparison with this threshold value, the pixel (i, j)
And the continuity of the adjacent pixel (i + m, j + n) is determined.
That is, the pixel (i, j) and the adjacent pixel (i + m, j + n)
If the difference between the pixel value of g (i, j) and g (i + m, j + n) is smaller than the threshold value, it is determined that the two pixels are in a continuous area. It is determined that there is.

The pixel value g (i, j) of the pixel of interest is L0
And the pixel value g (i + mj +
If it is determined that n) is continuous with the pixel value g (i, j), the process proceeds to step 807.

In step 807, the mismatching area detecting means 211 sets the number k of unprocessed pixels to be labeled to 1
Then, the x coordinate i + m of the adjacent pixel is stored in the coordinate storage buffer x [k], and the y coordinate j + n is stored in the coordinate storage buffer y [k].

In step 808, the mismatching area detecting means 211 determines whether or not the processing in steps 805 and 806 has been performed on all eight pixels adjacent to the pixel of interest. ,
If it is determined that there is any one that has not been subjected to the processing of 806, the process returns to step 805, and the subsequent processing is repeated. Thereafter, if it is determined in step 808 that the processing in steps 805 and 806 has been performed on all eight adjacent pixels, the process proceeds to step 809.

At step 809, the mismatching area detecting means 211 outputs the pixel value g (i, j) of the target pixel.
Is replaced with the label value L0. Here, the label value L0 is
This is added to the pixels of the part corresponding to the measurement target (subject) of the distance image D1.

In step 810, the mismatching area detecting means 211 determines that the number k of unprocessed pixels to be labeled is 0.
And whether the number k of unprocessed pixels to be labeled is 0 or not.
If it is determined that it is not, the process proceeds to step 811.

In step 811, the mismatching area detecting means 211 sets the value stored in the coordinate storage buffer x [k] as the x coordinate of the pixel of interest and stores the value in the coordinate storage buffer y [k]. Let the value be the y coordinate of the pixel of interest. Thereby, a pixel adjacent to the previous pixel of interest and having a continuous pixel value is set as a new pixel of interest. Thereafter, the mismatching area detection unit 211 increments the number k of unlabeled pixels by one and returns to step 804.

Thereafter, in step 810, the processing after step 804 is repeated until it is determined that the number k of unprocessed labeling pixels is 0, and when it is determined that the number k of unprocessed labeling pixels is 0, Step 812
Proceed to. In step 812, the mismatching area detection unit 211 replaces the pixel values of the pixels of the distance image D1 whose pixel values are not replaced with the label value L0 with the label value L1 indicating that the pixel is a mismatching area.

When the above-described labeling processing is completed, the same label is assigned to continuous areas in which the distance data has a difference equal to or less than a predetermined threshold in the distance image D1,
For example, a labeling image 704 in FIG. 7 is obtained.

The mismatching area detecting means 211 determines whether or not there is a small isolated area surrounded by pixels to which the same label is assigned and to which a label different from the surrounding area is present, according to the above label value. I do. In the labeling image 704 of FIG. 7, the areas B, C, and D correspond to these conditions, and the areas B, C, and D are determined to be mismatching areas (step 705).

Returning to FIG. 6, the description will be continued. In step 602, when the detection of the mismatching area is completed according to the flow of FIGS. 7 and 8, it is determined in step 603 whether the mismatching area is detected. If the mismatching area is not detected, the light is projected. A process of generating a distance image from only the pattern image (step 610) is performed.

If a mismatching area is detected in step 603, calculation area setting processing is performed in steps 604 and 605. This calculation region setting process is a process of setting a partial region in which a distance image generation process using a texture image is performed.
The coordinate area including the mismatched area detected in step 2 is set as a partial area for performing a distance image generation process using a texture image.

Next, in step 606, the distance image in the calculation area set in steps 604 and 605 is calculated as D2 (i, j). They are,
Based on the texture images captured by the reference camera stored in the frame memories 1b and 204 and the texture images captured by the reference camera stored in the frame memories 2b and 206 in FIG.
Is a process in which the associating process based on the stereo image method is executed to generate disparity data, and the distance image generating unit 209 further generates a distance image based on the disparity data. In addition,
These processes are performed only on the partial area set in steps 604 and 605, that is, on the pixel area including the mismatching area.

In step 606, when the distance image of the partial area using the texture image is generated, the process proceeds to step 607, in which the reliability of the generated distance image is determined. This reliability determination process is performed by determining the consistency between the distance image generated based on the texture image and the distance image already generated based on the light projection pattern image. Specifically, for example, for pixels around the mismatching area, the pixel values of the distance image generated based on the texture image, the pixel values of the distance image generated based on the light projection pattern image, and these two pixels This is performed by comparing value data, and when these are far apart, the reliability of the distance image generated based on the texture image is denied (N in step 608).
o) and the replacement process is stopped.

Specifically, the replacement means 212 shown in FIG.
Is the difference between the pixel value (distance data) around the mismatching region in the partial distance image generated based on the texture image and the pixel value of the corresponding pixel of the distance image generated previously based on the light projection pattern image. The difference is calculated, and the difference is compared with a preset threshold value. When the difference is smaller than the threshold value, the replacement process with the distance image based on the texture image is performed. If the difference is larger than the threshold value, the region recognized as the mismatching region may be an extremely uneven portion existing in the measurement target, and the replacement process is not performed.

The distance image generated based on the texture image has reliability (Yes in step 608).
Is determined in step 609, the distance image D2 (i,
j) is applied to the distance image D1 (i, j) generated based on the light projection pattern image. That is, the distance image of the mismatching area is replaced with the distance image D2 (i, j) generated based on the texture image.

With respect to the distance image D1 (i, j) generated based on the light projection pattern image, the mismatched area is defined as the distance image D2 generated based on the texture image.
The composite image replaced with (i, j) is generated as a final distance image (step 610) and output.

FIG. 9 shows an example of a light projection pattern image and a texture image taken by the reference camera and the reference camera in the three-dimensional shape measuring apparatus of the present invention.

FIG. 9A is a projection pattern image taken by the reference camera, FIG. 9B is a projection pattern image taken by the reference camera, and FIG. 9C is a texture image taken by the reference camera. 9 (d) is a texture image taken by the reference camera. Although FIGS. 9A and 9B are not clearly shown in the drawings, the aperiodic pattern shown in the upper part of FIG. 9 is irradiated with infrared light, and the aperiodic pattern is projected on the measurement target. This is captured by each camera. FIG.
(C) and (d) show the case where the measurement target is photographed in a state where the non-periodic pattern is not photographed but observed with visible light of sunlight, a fluorescent lamp, and an incandescent lamp.

In FIGS. 9A and 9B, regular reflection areas as shown in FIG. 9 occur, and it becomes difficult to generate an accurate distance image in these areas. That is, the regular reflection region is a portion where the irradiation pattern cannot be obtained, the association of pixels becomes difficult, and the possibility of becoming the above-described mismatching region is high. In the three-dimensional shape measuring apparatus of the present invention, an accurate distance image is obtained by replacing these parts with a distance image based on the texture images of FIGS. 9 (c) and 9 (d).

FIG. 10 shows an example of a distance image based on a light projection pattern image, a distance image based on a texture image, and a final output image. In FIG. 10, each distance image is
Each of the colors (shading) indicates a difference in distance.
FIG. 10 shows a schematic distance image for explaining the replacement area, which is different from an actual distance image.

In FIG. 10, the upper left diagram is a distance image based on the light projection pattern, a mismatching region is formed in a region almost in front of the measurement target, and an incorrect distance value is shown on the distance image. The upper right part of FIG. 10 is a distance image based on the texture image. Although a mismatching area exists on the right side of the measurement target in the figure, the front side of the measurement target has
There is no mismatching area, and by setting the distance images of these areas as replacement areas, the output distance image shown in the lower part of FIG. 10 is obtained. There is no mismatching area in the distance image in the lower part of FIG. 10, and an output as an accurate distance image is obtained.

In the above example, the example having one set of the reference camera and the reference camera has been mainly described.
As a configuration having a plurality of pairs of a reference camera and a reference camera, the distance image generation means generates a plurality of distance images based on the light emission pattern images captured by each pair of the reference camera and the reference camera, and further includes a reference As a configuration for generating a plurality of distance images based on a texture image captured by each set of the camera and the reference camera, a configuration for generating a plurality of distance images, for example, distance images from various viewpoints may be used.

Further, a plurality of sets of a reference camera and a reference camera are provided, and a plurality of light projecting pattern irradiating means are formed. Each of the plurality of light projecting pattern irradiating means is different from visible light and has another light projecting pattern. As a configuration for irradiating the measurement target with light having a unique wavelength different from the pattern irradiating means, each set of a plurality of sets of a reference camera and a reference camera is configured to emit light based on light having a different wavelength. The image is captured separately from the texture image, and the distance image generating means adjusts a plurality of distance images based on the light emitting pattern image captured by each set of the reference camera and the reference camera, and the reference camera and A configuration in which a plurality of distance images are generated based on texture images captured by each set of reference cameras may be adopted.

As described above, according to the three-dimensional shape measuring apparatus and the three-dimensional shape measuring method of the present invention, it is possible to simultaneously capture a texture image with visible light and an aperiodic light projection pattern image with infrared light. Even if a mismatching area occurs in the distance image obtained by the aperiodic light projection pattern image, the distance image of the partial area corresponding to the mismatching area using the simultaneously captured image with visible light can be used. Is generated, and this can be replaced with a mismatching area in the distance image based on the light projection pattern image, and a more accurate distance image can be obtained.

According to the three-dimensional shape measuring apparatus and the three-dimensional shape measuring method of the present invention, the texture image of visible light and the aperiodic light projection pattern image of infrared light have no time lag. Since it is possible to shoot at the same time,
The pixels in the range image are accurately associated with each other, and the replacement process can be easily performed.

According to the three-dimensional shape measuring apparatus and the three-dimensional shape measuring method of the present invention, the aperiodic light projection pattern image is formed by irradiation with infrared light, which is different from the conventional pattern irradiation using a flash. The pattern image capturing time (shutter release time) can be set long, and a stable image can be captured.

[Three-Dimensional Shape Measuring Apparatus—Second Embodiment] The configuration shown in FIG. 2 described in the above embodiment simultaneously captures a texture image with visible light and an aperiodic light projection pattern image with infrared light. First, a distance image is generated based on a non-periodic light projection pattern image by infrared light, a mismatching area is detected from the distance image, and a distance image based on a texture image is generated only for the area. Then, the configuration is such that the generated distance image of the partial area is used as the replacement area. However, the distance image generation processing based on both the texture image of visible light and the aperiodic light projection pattern image of infrared light is performed. A configuration for executing in parallel and selectively selecting and outputting a distance image from these two distance images will be described as a second embodiment.

FIG. 11 is a block diagram of a three-dimensional shape measuring apparatus according to the second embodiment. In the configuration shown in FIG. 11, unlike the configuration of the first embodiment shown in FIG. 2, two distance image generation units and two image memories are configured.
1 is a frame memory 1a, 203 and a frame memory 2
a, 205, generate a distance image based on the light projection pattern image stored therein, and the distance image generating means B, 1102 generates frame images 1b, 204 and frame memories 2b, 206.
Generate a distance image based on the texture image stored in the. These two distance image generating means generate distance images for all areas.

The generated distance images are stored in image memories A and 1103 and image memories B and 1104, respectively.

The mismatching area detecting means detects a mismatching area of the distance image stored in each image memory, and if any of the areas has a mismatching area, selects area data.
Then, the selection unit executes the replacement of the area with the distance image stored in the other memory, and outputs the finally synthesized distance image via the output unit 1107.

According to the configuration of the present embodiment, when two distance images are generated in parallel and a mismatching area is detected, the replacement processing with the other distance image can be executed immediately, so that the processing speed is increased. Becomes possible. In addition, since the mismatching area detecting unit 1105 or the selecting unit 1106 can execute comparison processing of each pixel value of the two distance images, it is easy to extract an area having an incorrect pixel value in each distance image.

[3D Modeling Apparatus] Next, a 3D modeling apparatus and a 3D modeling method using the 3D shape measuring apparatus of the present invention will be described.

FIG. 12 is a block diagram showing the configuration of the three-dimensional modeling apparatus according to the present invention, and FIG. FIG.
As shown in FIG. 7, the means for generating a distance image is the same method as the method described in the first embodiment. The distance image based on the light projection pattern image is complemented using the distance image based on the texture image, and the output distance is calculated. Generate an image.

The three-dimensional (3D) shape data generating means 1201 shown in FIG.
b, receiving the texture image data stored in 204, performing a process of pasting the texture image data to the distance image, and further, based on the calibration parameters of the space in which the measurement target is arranged, a 3D image in the xyz space. Generate shape data. The generated 3D shape data is displayed together with the texture image.

FIG. 13 shows a processing flow of the three-dimensional modeling device in the three-dimensional (3D) shape data generating means 1201 shown in FIG.
The shape data generating means 1201 inputs a distance image D (i, j) obtained by performing a complementation process on the distance image based on the light projection pattern image using the distance image based on the texture image (step 1301), and further receives the reference image from the reference camera. The captured texture image is input (step 1302).

Further, in step 1303, the three-dimensional (3D) shape data generating means 1201 executes processing for pasting the texture image data to the input distance image. Further, in step 1304, a calibration parameter of the space where the measurement target is arranged is input, and 3D shape data in the xyz space is generated (step 1305). In step 1306, 3D
Display shape data and texture.

The three-dimensional modeling apparatus of the present invention is not limited to a conventional three-dimensional image generated by a distance image based on one projection pattern, but a distance image in which a mismatching area is complemented based on a texture image captured at the same time. Since the three-dimensional image is generated based on the distance image, an inaccurate three-dimensional image is not generated based on the erroneous distance image, and a more accurate three-dimensional (3D) display can be performed.

The three-dimensional shape measuring apparatus is applied to the three-dimensional shape measuring apparatus in which the distance image based on the light projection pattern image and the distance image based on the texture image described in the second embodiment are generated by parallel processing. May be configured.

The present invention has been described in detail with reference to the specific embodiments. However, it is obvious that those skilled in the art can modify or substitute the embodiment without departing from the spirit of the present invention. That is, the present invention has been disclosed by way of example, and should not be construed as limiting. In the above-described embodiment, the irradiation of the light projection pattern is mainly described with infrared light. However, in the three-dimensional shape measuring device and method of the present invention, the three-dimensional modeling device and method are captured with visible light. Any configuration that allows simultaneous projection of the light projection pattern image as an image different from the texture image may be used. Irradiation of the light projection pattern is not limited to infrared light, and a wavelength different from visible light is used. Any configuration that can be captured as a different image, for example, a configuration capable of capturing only a light projection pattern image or only a texture image by a specific filter means may be used. In order to determine the gist of the present invention, the claims described at the beginning should be considered.

[0128]

As described above in detail, according to the present invention,
It is possible to simultaneously capture a texture image with visible light and a light projection pattern image with light having a wavelength different from visible light, such as infrared light, and a mismatching area is included in the distance image obtained by the light projection pattern image. Even if it occurs, a distance image of the partial region corresponding to the mismatching region is generated using the simultaneously captured image with visible light, and this is used as the mismatching region in the distance image based on the projection pattern image. , And a more accurate distance image can be obtained.

Further, according to the present invention, a light projection pattern image by light having a wavelength different from visible light such as infrared light is
Since the pattern is an aperiodic pattern, the number of mismatched areas in the association process can be reduced, and the pixels in the distance image can be associated with each other more accurately.

According to the present invention, a texture image with visible light and a light projection pattern image with light having a wavelength different from visible light, such as infrared light, can be captured simultaneously with no time lag. Is possible, the pixels in the range image are associated with each other accurately, and the replacement process can be easily executed.

Further, according to the present invention, the projected pattern image is formed by irradiation with infrared light, for example, and unlike the conventional pattern irradiation using a flash, the pattern image capturing time (shutter open time) is set longer. It is possible to capture a stable image.

[Brief description of the drawings]

FIG. 1 is a diagram illustrating a stereo image method applicable as a three-dimensional information acquisition configuration used in the present invention.

FIG. 2 is a block diagram showing a configuration of an embodiment of a three-dimensional shape measuring apparatus according to the present invention.

FIG. 3 is a diagram illustrating an example of an aperiodic light projection pattern used in a light projection pattern irradiation unit according to the present invention.

FIG. 4 is a diagram showing a configuration example of a light projection pattern irradiation unit in the present invention.

FIG. 5 is a diagram showing a configuration of capturing a light projection pattern image and a texture image by a reference camera and a reference camera according to the present invention.

FIG. 6 is a diagram showing a flowchart of a distance image generation process in the three-dimensional shape measuring apparatus of the present invention.

FIG. 7 is a diagram showing a flow of processing for detecting a mismatching region in the three-dimensional shape measuring apparatus of the present invention.

FIG. 8 is a diagram showing a processing flow of labeling processing based on a distance image in the three-dimensional shape measuring apparatus of the present invention.

FIG. 9 is a diagram showing an example of a light projection pattern image and a texture image captured by a reference camera and a reference camera in the three-dimensional shape measuring apparatus of the present invention.

FIG. 10 is a diagram illustrating the correspondence between a distance image based on a light projection pattern image and a texture image and a distance image to be output in the three-dimensional shape measuring apparatus of the present invention.

FIG. 11 is a block diagram showing the configuration of a second embodiment of the three-dimensional shape measuring apparatus of the present invention.

FIG. 12 is a block diagram showing a configuration of a three-dimensional modeling device of the present invention.

FIG. 13 is a diagram showing a processing flow of the three-dimensional modeling device of the present invention.

[Explanation of symbols]

 201 Reference camera 202 Reference camera 203-206 Frame memory 207, 208 Image correspondence means 209 Distance image generation means 210 Image memory 211 Mismatch area detection means 212 Replacement means 213 Output means 220 Projection pattern irradiation means 401 LED array 402 Light diffusion plate 403 Condensing lens 404 Slide 405 Projection lens 407 Light emission drive circuit 408 Shutter synchronization circuit 501 Reference camera lens 502, 508 Spectroscope 503 509: visible light cut filter 504, 510: CCD element 505, 511: infrared light cut filter 506, 512: CCD element 507: reference camera lens 1101, 1102: distance image generating means 1103, 1104: image memory 1105: mismap Ring region detection unit 1106 ... selection unit 1107 ... output unit 1201 ... three-dimensional (3D) shape data generating unit 1202 ... display means

Continued on the front page (72) Inventor Takayuki Ashigahara 1-14-10 Higashi-Gotanda, Shinagawa-ku, Tokyo Inside Sony Kihara Laboratory Co., Ltd. (72) Inventor Kazuyoshi Hayashi 1--14 Higashi-Gotanda, Shinagawa-ku, Tokyo No. 10 F-term in Sony Kihara Research Laboratories (reference)

Claims (21)

[Claims] 1. A three-dimensional shape measuring apparatus for measuring a three-dimensional shape of an object to be measured using images taken from different directions of the object to be measured, wherein the light projection pattern is formed using light having a wavelength different from visible light. A light projection pattern irradiating means for irradiating the object to be measured; a light projection pattern image irradiated with the light projection pattern by the light projection pattern irradiating means; and a texture image observed by visible light, which is separated by an image separating means. A reference camera that separately captures the synchronized light projection pattern image and the texture image, and a light projection pattern image irradiated by the light projection pattern by the light projection pattern irradiation unit, and a texture image observed by visible light. The light projection pattern image and the texture image, which are obtained by separating and synchronizing the two images by the image separating means, are output to a different one from the reference camera. And a reference camera that captures at a timing synchronized with the reference camera, a distance image based on the projection pattern image captured by the reference camera and the reference camera, and a texture captured by the reference camera and the reference camera. A range image generating unit that generates a range image based on an image; a mismatching region detecting unit that detects a mismatching region in the range image based on the light projection pattern image generated by the range image generating unit; A replacement unit configured to replace a distance image of a partial region including a mismatched region detected by the unit with a distance image based on a texture image. 2. The three-dimensional shape measuring apparatus according to claim 1, wherein the light projection pattern irradiated by the light projection pattern irradiation means is a light projection pattern using infrared light. 3. The three-dimensional shape measuring apparatus according to claim 1, wherein the light projection pattern irradiated by the light projection pattern irradiation means is an aperiodic light projection pattern. 4. The apparatus according to claim 1, wherein said image separating means includes a spectroscope and a filter arranged on a path separated by said spectroscope to cut light of a specific wavelength. The three-dimensional shape measuring device according to the above. 5. A labeling process for comparing a difference between pixel values of adjacent pixels in a distance image with a predetermined threshold value and assigning the same label to pixels having a smaller difference than the threshold value. 3. The three-dimensional shape measuring apparatus according to claim 1, wherein the processing is performed to execute a process of detecting, as a mismatching region, a small region with a different label surrounded by the same label region. 6. The distance image generating means for generating a distance image based on the texture image only for a partial area including a mismatching area in the distance image based on the light projection pattern image. 3. The three-dimensional shape measuring device according to 1. 7. The replacement means executes a reliability judgment of the distance image based on the texture image by comparing a corresponding pixel value of the distance image based on the texture image with a corresponding pixel value of the distance image based on the light projection pattern. The three-dimensional shape measuring apparatus according to claim 1, wherein the three-dimensional shape measuring apparatus according to claim 1, further comprising a configuration that applies a distance image based on the confirmed texture image as replacement data for the distance image based on the light projection pattern. 8. A plurality of sets of a reference camera and a reference camera, wherein the distance image generating means includes a plurality of range images based on a projection pattern image captured by each set of the reference camera and the reference camera. The three-dimensional shape measuring apparatus according to claim 1, further comprising a configuration for generating a plurality of distance images based on a texture image captured by each set of the reference camera and the reference camera. 9. A plurality of sets of reference cameras and reference cameras, and a plurality of light projecting pattern irradiating means, wherein each of the plurality of light projecting pattern irradiating means is different from the visible light and has another light projecting pattern. Light pattern irradiating means has a configuration of irradiating the measurement target with a light projection pattern using light of a unique wavelength, each set of the plurality of sets of reference cameras and reference cameras,
Separately captures a light projection pattern image based on light of different wavelengths from a texture image, and captures the distance image generating means, based on the light projection pattern images captured by each set of the reference camera and the reference camera. 3. The three-dimensional shape measurement according to claim 1, further comprising a step of generating a plurality of distance images based on a distance image and a texture image captured by each set of the reference camera and the reference camera. apparatus. 10. A three-dimensional shape measuring method for measuring a three-dimensional shape of a measurement object using images taken of the measurement object from different directions, wherein the light projection pattern is formed by using light having a wavelength different from visible light. A light-projection pattern irradiation step of irradiating the object to be measured, a light-projection pattern image illuminated with the light-projection pattern, and a light projection synchronized and separated by image separation means into two images of a texture image observed by visible light An image in which the pattern image and the texture image are separately captured by the reference camera, and the projection pattern image and the texture image separately captured by the reference camera from a different direction from the reference camera and at a timing synchronized with the reference camera. A capturing step; and a projection pattern image captured by the reference camera and the reference camera. A first distance image generating step of generating a distance image based on the first distance image generating step; a mismatching area detecting step of detecting a mismatching area in the distance image based on the light projection pattern image generated in the first distance image generating step; A second distance image generation step of generating a distance image of a partial area including the mismatched area detected by the matching area detection step based on the texture image; and a light projection pattern image generated in the first distance image generation step. Replacing a partial area including a mismatched area of the distance image based on the distance image based on the texture image of the partial area generated in the second distance image generating step. Shape measurement method. 11. The three-dimensional shape measuring method according to claim 10, wherein the light projection pattern irradiated in the light projection pattern irradiation step is a non-periodic light projection pattern using infrared light. 12. A labeling process for comparing a difference between pixel values of adjacent pixels in a distance image with a predetermined threshold value and assigning the same label to pixels having a smaller difference value than the threshold value. The three-dimensional shape measuring method according to claim 10, further comprising: a step of detecting a small area provided with a different label and surrounded by the same label area as a mismatching area. 13. The replacement step includes the step of comparing the distance image based on the texture image and the corresponding pixel value of the distance image based on the light projection pattern to determine the reliability of the distance image based on the texture image. The three-dimensional shape measuring method according to claim 10, wherein the distance image based on the confirmed texture image is applied as replacement data for the distance image based on the light projection pattern. 14. A three-dimensional shape measuring method for measuring a three-dimensional shape of a measurement object using images taken of the measurement object from different directions, wherein the light projection pattern is formed by using light having a wavelength different from visible light. A light-projection pattern irradiation step of irradiating the object to be measured, a light-projection pattern image illuminated with the light-projection pattern, and a light projection synchronized and separated by image separation means into two images of a texture image observed by visible light An image in which the pattern image and the texture image are separately captured by the reference camera, and the projection pattern image and the texture image separately captured by the reference camera from a different direction from the reference camera and at a timing synchronized with the reference camera. A capturing step; and a projection pattern image captured by the reference camera and the reference camera. A plurality of distance image parallel generation steps of generating a distance image based on a texture image captured by the reference camera and the reference camera in parallel as a second distance image, using the distance image based on the first distance image, The number generated in the generation step
A mismatching region detecting step of detecting a mismatching region in at least one of the distance images of the one distance image and the second distance image, and a mismatching region in one of the distance images detected by the mismatching region detecting step. A replacement step of replacing the distance image of the partial area including the other distance image with the other distance image. 15. The three-dimensional shape measuring method according to claim 14, wherein the light projection pattern irradiated in the light projection pattern irradiation step is a non-periodic light projection pattern by infrared light. 16. A labeling process for comparing a difference between pixel values of adjacent pixels in a distance image with a predetermined threshold value and assigning the same label to pixels having a smaller difference value than the threshold value. The three-dimensional shape measuring method according to claim 14, further comprising: a step of detecting a small area provided with a different label surrounded by the same label area as a mismatching area. 17. The method according to claim 1, wherein the replacing step includes comparing a distance image based on the texture image with a corresponding pixel value of the distance image based on the projection pattern, and performing a reliability determination on the distance image based on the texture image. The three-dimensional shape measuring method according to claim 14, wherein the distance image based on the confirmed texture image is applied as replacement data for the distance image based on the light projection pattern. 18. A three-dimensional modeling apparatus for measuring a three-dimensional shape of a measurement object using images taken of the measurement object from different directions and performing three-dimensional modeling, wherein a light having a wavelength different from visible light is used. A light projection pattern irradiating unit that irradiates the measurement target with a light projection pattern; a light projection pattern image irradiated with the light projection pattern by the light projection pattern irradiation unit; and a texture image observed by visible light. A reference camera that separately captures the projected light pattern image and the texture image separated and synchronized by the image separating means, a projected light pattern image irradiated with the projected light pattern by the light emitting pattern irradiating means, and observed by visible light. The light projection pattern image and the texture image which are synchronized by separating the two images of the texture image A reference camera that captures from a direction different from the reference camera and at a timing synchronized with the reference camera, a distance image based on the light emission pattern image captured by the reference camera and the reference camera, and the reference camera and reference A distance image generating means for generating a distance image based on a texture image captured by a camera; and a mismatching area detecting means for detecting a mismatching area in the distance image based on the light projection pattern image generated by the distance image generating means. A replacement unit that replaces a distance image of a partial region including a mismatched region detected by the mismatched region detection unit with a distance image based on a texture image; and based on a light projection pattern image replaced and generated by the replacement unit. The combination of the distance image based on the distance image and the texture image Distance image, and three-dimensional modeling apparatus characterized by having a three-dimensional shape data generating means for generating three-dimensional shape data based on the take incorporated texture image by the reference camera. 19. A three-dimensional modeling method for measuring a three-dimensional shape of the measurement object using images taken of the measurement object from different directions and performing three-dimensional modeling, wherein light having a wavelength different from visible light is used. A light emitting pattern irradiating step of irradiating the light emitting pattern to the object to be measured; and a light emitting pattern image irradiated with the light emitting pattern and a texture image observed by visible light are separated by an image separating unit. The projection pattern image and the texture image synchronized with the reference camera are separately captured by the reference camera, and the projection pattern image and the texture image synchronized by the reference camera from a different direction from the reference camera and at a timing synchronized with the reference camera. An image capturing step for capturing individually, and capturing by the reference camera and the reference camera. A first distance image generating step of generating a distance image based on the inserted light projection pattern image, and a mismatch matching detecting a mismatching area in the distance image based on the light projection pattern image generated in the first distance image generating step An area detecting step, a second distance image generating step of generating a distance image of a partial area including the mismatching area detected by the mismatching area detecting step based on the texture image, and a generating step in the first distance image generating step A replacement step of replacing a partial area including a mismatched area of a distance image based on the projected light projection pattern image with a distance image based on a texture image of the partial area generated in the second distance image generation step; Floodlight pattern generated by replacing the step And a three-dimensional shape data generating step of generating three-dimensional shape data based on a synthesized distance image of a distance image based on the base image and a distance image based on the texture image, and a three-dimensional shape data based on the texture image captured by the reference camera. A three-dimensional modeling method characterized in that: 20. A three-dimensional modeling method for measuring a three-dimensional shape of an object to be measured using images taken of the object from different directions and performing three-dimensional modeling, wherein a light having a wavelength different from visible light is used. A light emitting pattern irradiating step of irradiating the light emitting pattern to the object to be measured; and a light emitting pattern image irradiated with the light emitting pattern and a texture image observed by visible light are separated by an image separating unit. The projection pattern image and the texture image synchronized with the reference camera are separately captured by the reference camera, and the projection pattern image and the texture image synchronized by the reference camera from a different direction from the reference camera and at a timing synchronized with the reference camera. An image capturing step for capturing individually, and capturing by the reference camera and the reference camera. A plurality of distance image parallel generations in which a distance image based on a textured image captured by the reference camera and the reference camera is generated in parallel as a second distance image, using a distance image based on the projected light projection pattern image as a first distance image. And the step of generating the plurality of distance image parallel generation steps.
A mismatching region detecting step of detecting a mismatching region in at least one of the distance images of the one distance image and the second distance image, and a mismatching region in one of the distance images detected by the mismatching region detecting step. A replacement step of replacing the distance image of the partial area including the other distance image, a composite distance image of a distance image based on the light projection pattern image generated and replaced by the replacement step and a distance image based on the texture image, and the reference A three-dimensional shape data generating step of generating three-dimensional shape data based on a texture image captured by a camera. 21. A computer program that causes a computer system to execute a process for executing a three-dimensional shape measurement for measuring a three-dimensional shape of the measurement target using images obtained by photographing the measurement target from different directions. A computer program program, the computer program comprises: a projection pattern irradiation step of irradiating the measurement target with a projection pattern using light having a wavelength different from visible light; and The projected light pattern image and the texture image are separated by an image separating means, and the synchronized light projection pattern image and the texture image are separately captured by a reference camera. By the reference camera from a different direction from the camera and at the same timing as the reference camera An image capturing step of separately capturing a synchronized light projection pattern image and a texture image; a first distance image generating step of generating a distance image based on the light projection pattern image captured by the reference camera and the reference camera; A mismatching area detecting step of detecting a mismatching area in the distance image based on the light projection pattern image generated in the first distance image generating step, and a partial area including the mismatching area detected by the mismatching area detecting step. A second distance image generating step of generating a distance image based on the texture image; and a partial area including a mismatching area of the distance image based on the light projection pattern image generated in the first distance image generating step. The part generated in the two-distance image generation step Program providing medium characterized by having a substitution step of replacing the distance image based on the texture image of the region.