JP2008275392A - Three-dimensional shape measurement method and system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a three-dimensional shape measurement method and system provided with a projection pattern capable of measuring the shape of the object by a single taking image and also capable of synchronous measurement from a plurality of directions. <P>SOLUTION: The three-dimensional shape measurement system is provided with the control device 5, the imaging camera 2, and the projector 1. The control device 5 forms projection patterns with magnitude of ≥2 pixels and a plurality of small regions set with respectively recognizable characteristic color distribution and makes the projector 1 project on the objective matter 3, and makes imaging camera 2 take image of the objective matter 3 on which the projection pattern is projected, and based on the color distribution of an arbitrary small regions, the corresponding relations to the small regions of the projection patterns are obtained. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a three-dimensional shape measuring method and apparatus for measuring a three-dimensional shape of a target object.

  Many of the existing three-dimensional measuring apparatuses use the principle of triangulation. FIG. 9 is a diagram for explaining the principle of general triangulation. In other words, a corresponding point “seeing the same point” is obtained between a plurality of camera images, or “projected on a pixel in a camera image” between a camera image and a pattern projector (such as a projector or a laser transmitter). If it is possible to obtain a corresponding point “what direction is the projected point from the light source”, the three-dimensional position of the object surface can be obtained as the intersection of the two projection lines.

  Thus, it is a well-known fact that finding corresponding points from a plurality of camera images and pattern projectors plays an important role in three-dimensional measurement, and even existing methods solve such problems. Various ideas are made for the purpose. As an example, a method for switching patterns will be described here.

  FIG. 10 is a diagram for explaining a conventional method for switching a large number of projection patterns. Using a projector as a projector, an image is taken while changing the brightness in the projection pattern partially, and each pixel in the image is changed (for example, “white → black → black → white →...”). Change). By devising how to change the pattern, you can easily identify the position in the pattern that is changing in the same way as the change of a pixel in the image. It is possible to ask.

In this way, under the precondition that the measurement target is a stationary object, a method of performing multiple shootings while switching the projection pattern projected from the projector (a method of moving the projection pattern is also a part of the projection pattern switching) Widely known.
For example, Patent Document 2 is a kind of shape measurement technique using a color stripe pattern, and instead of using a plurality of images, a color pattern and a color camera are separated for each color component and used virtually. A technique for acquiring a plurality of black and white patterns and reducing the trouble of pattern switching is shown in FIG.

  Patent Document 3 discloses a kind of measurement technique called a so-called optical cutting method that scans a laser, but the fundamental principle is similar to the pattern switching method described above. Using the time information of “the moment when the light hits”, the correspondence between the projection direction of the laser transmitter and the pixels in the image is obtained. Patent Document 4 discloses a method called a phase shift method, which is a kind of pattern switching method, and reduces the pattern switching effort by using a different pattern for each color component as in the technique of Patent Document 2. The method is shown.

In these methods, it is possible to find a corresponding pixel on the projector side from time-to-time changes in the value of the same pixel viewed from the camera, and measurement is possible by performing triangulation from the found corresponding points. That is, in the conventional pattern switching method, it can be said that the temporal change of the pattern is used as the ID of the pixel in the pattern.
On the other hand, as a shape measurement method for measuring a three-dimensional shape of an object in a non-contact manner without projecting a pattern by projecting a specific pattern light, a method using a striped pattern or a random point group is also known. Yes.

  The method of shooting the target object on which the pattern is projected with multiple cameras and finding the corresponding points between multiple images is called stereo matching, and the method using striped pattern `` Rapid shape acquisition using color structured light and multi-pass dynamic programming (L. Zhang, B. Curless, and SM Seitz, Proc. Symposium on 3D Data Processing Visualization and Transmission, 2002) and the method using random point cloud “Logistics Robot Vision Using Random Dot Pattern Projection Stereo” "Development of the system" (Manabu Hashimoto, Kazuhiko Sumi, Tetsuji Hashita, Shinichi Kuroda, Norio Kodaira, Shotaro Iwata, Journal of the Robotics Society of Japan, Vol. 17, No.1, pp.48-49, 1999.) Has been proposed.

In addition to such a method based on the corresponding points between the camera and the camera, there is a method using a striped pattern or a random point group in order to obtain the corresponding points between the camera and the projector.
For example, Patent Document 1 discloses a three-dimensional shape measurement technique for obtaining corresponding points between a camera and a projector using a color stripe pattern.
In the method that does not switch these patterns, local changes in brightness and color observed in the image are used as feature amounts, and combinations of locations with similar feature amounts are combined between multiple cameras or between a camera and a projector. Measurement is possible by searching and performing triangulation from the found corresponding points.

As can be seen from the above description, if a unique ID (identification information) can be embedded for each pixel of the projection pattern by some method, the corresponding point is obtained by the ID extracted from each pixel in the captured image. That is, it is possible to perform three-dimensional shape measurement by triangulation. FIG. 11 is a diagram illustrating an example in which ID (identification information) is set for each pixel in the projection pattern.
JP 2003-42735 A Japanese Patent Laid-Open No. 9-21620 JP 2004-20536 A JP 2001-330417 A

In the conventional method of switching projection patterns, the switching method is devised to apply a switching method that guarantees that the same value does not change in different parts. Therefore, in principle, erroneous recognition of corresponding points does not occur. However, in principle, since multiple imaging is required, it takes time to acquire data, and the shape of an object that is deformed or moved cannot be measured.
In addition, measurement methods using striped patterns or random point clouds without switching patterns can also measure object shapes that are deformed or moved, but uncertain factors such as object surface shape, reflection characteristics, and shooting environment Depending on the situation, different parts may be photographed with similar characteristics, and there is a known danger of misrecognizing corresponding points.

That is, in the conventional method in which the pattern is not switched, there is no guarantee that a certain part in the captured image has a unique characteristic different from the other part.
In addition, in the conventional method, only one direction is assumed for pattern projection and shooting regardless of whether or not the pattern is switched. For example, it is possible to simultaneously measure with a plurality of devices from around the target object. It wasn't.

  In view of the above problems, the present invention is capable of measuring the shape of a target object in a single shooting operation, and capable of simultaneous measurement from a plurality of directions. An object is to provide a measurement method and apparatus.

In order to achieve the above purpose, it is necessary to make each part of the projection pattern of the projector have a unique characteristic so that one part on the surface of the object to be photographed and another part are not photographed with the same characteristic. is there.
In addition, it is necessary to give unique features to all portions of the projection patterns of all projectors so that a projection pattern from one projector and a projection pattern from another projector can be discriminated.

  Therefore, the present invention provides a small area of two or more pixels in the resolution of the projection pattern projected from the projector, sets characteristic color distributions that can be individually identified for each small area, By creating and projecting a projection pattern in which a small area is two-dimensionally distributed at intervals of 1 pixel or more, a projected pattern that is reflected and observed on the object surface is observed as a similar pattern between different parts. This makes it possible to determine the projector that projects the pattern.

The specific solution is as follows.
(1) In the three-dimensional shape measurement method, a projection pattern composed of a plurality of small areas having a size of 2 pixels or more and set with a distinctive color distribution that can be distinguished from each other is projected onto the object. Is captured, and a correspondence relationship with the small area of the projection pattern is obtained based on a color distribution of an arbitrary small area in the captured image.
(2) The three-dimensional shape measurement method according to (1) is characterized in that the projection pattern is a pattern that is discretely distributed on a two-dimensional plane with an interval of one pixel or more around the small region. To do.

(3) The three-dimensional shape measuring device includes a control device, an imaging camera, and a projector. The control device has a size of two or more pixels and sets a plurality of characteristic color distributions that can be distinguished from each other. A projection pattern composed of a small area is created and projected onto the object by the projector, the object on which the projection pattern is projected is imaged by the imaging camera, and the color distribution of an arbitrary small area in the captured image The correspondence relationship with the small area of the projection pattern is obtained based on the above.
(4) The three-dimensional shape measuring apparatus according to (3), wherein the projection pattern is a pattern that is discretely distributed on a two-dimensional plane with an interval of one pixel or more around the small region. To do.

The present invention can measure the shape of a target object based on captured image data obtained by one shooting. For this reason, it is possible to measure the shape of an object that is deformed or moved without taking time for data acquisition.
In addition, the shape of the entire circumference of the target object can be measured by using a plurality of projectors and a plurality of imaging cameras at the same time.

  In the present invention, the color information of each pixel can be used as ID information for specifying the position. In addition, since the unit of information to be handled is not a pixel unit as in the past, but a small area of 2 pixels or more, abundant color information of 2 pixels or more may be used as information for specifying an ID. it can. In addition, since the small area is composed of a plurality of pixels of two or more pixels, ID information with high identification ability can be configured by increasing the number of pixels.

The present invention is characterized in a method for obtaining an extracted small area on a captured image corresponding to a small area on a projection pattern and a means therefor in a three-dimensional shape measuring method and a measuring apparatus therefor.
Specifically, in order to accurately obtain the correspondence between the camera image and the projection pattern in one shooting, the projection pattern projected from the projector is provided with a plurality of small regions having a size of 2 pixels or more in the resolution. A characteristic color distribution that can be individually identified for each small area is set, and these various small areas are two-dimensionally discretely distributed with a space of one pixel or more around them. Create a pattern and project it onto the object to be measured. Thereby, it is possible to prevent the projection patterns reflected and reflected on the object surface from being observed as different patterns at different portions, and to determine the projector that projects the pattern.

For example, the small region includes a square block such as 2 (vertical) × 2 (horizontal) pixels, a rectangular block such as 2 × 1 pixels, a block having an arbitrary shape such as a circular (approximate) block, and an arbitrary area.
The coordinate data to be processed in units of small areas includes the center position of the small area (for example, the position where the corners of four pixels are in contact with each other in a 2 × 2 pixel block), the position of the pixels constituting the small area, Etc. can be set arbitrarily. Hereinafter, the coordinate data to be processed in units of small areas will be described as the center position of the small area.

The intervals provided around the small regions are appropriately formed depending on the arrangement pattern in which a plurality of small regions are arranged, the shape of the small regions, and the like.
As shown in FIGS. 4 and 5, the color distribution means a distribution of RGB values in a small area in the projection pattern and in the captured image. From this, the color distribution of the small area in the projection pattern is determined from the distribution of RGB values in the small area observed in the captured image.

In FIG. 4, since the image in the small region in the observation (captured) image displays data for the first and second pixels from the top as R components, the corresponding small pixels in the projection pattern are displayed. As an area pattern, 8 pixels in the first and second rows from the top have R component values (value 1 in binary display), and 8 elements in the third and fourth rows from the top are R elements. The ID of the R component is specified by displaying as no component value (same value 0). That is, the ID of the R component is
1st from left, 1st from left, 2nd from left, 3rd from left, 4th from left,
1st from left, 2nd from left, 3rd from left, 4th from left
3rd from the top, 1st from the left, 2nd from the left, 3rd from the left, 4th from the left,
4th row from the top, 1st from the left, 2nd from the left, 3rd from the left, 4th from the left,
In this order, [1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0] is obtained.

In the same way, the G component is [0, 0, 1, 1, 0, 0, 1, 1, 0, 0, 1, 1, 0, 0, 1, 1]. Similarly, the B component is [0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1].
In the same way in FIG. 5, the R component is [1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0], and the G component is [1. 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0], the B component is [0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1].
As color information, R (red), G (green), B (blue), C (cyan), M (magenta), Y (yellow), W (white), or non-displayed black may be used alone or in combination. Used in.

FIG. 1 is an explanatory diagram for explaining a first embodiment of the present invention.
A three-dimensional shape measurement apparatus that executes the three-dimensional shape measurement method of the present invention includes a projector 1 that projects a projection pattern, an imaging camera 2, and a control device 5 that controls these.
The projector 1 projects a projection pattern on the outer surface of the measurement target object 3. The projection patterns are arranged so as to be discretely provided at intervals from each other in accordance with the selected pattern of the small regions 4 composed of 4 × 4 pixel blocks. Each small region 4 has color distribution data (for example, R, G, B) so as to constitute an ID that can be independently identified.

The imaging camera 2 images the projection pattern on the measurement target object, that is, the small area 4 at a time.
The control device 5 controls the projector 1 and the imaging camera 2 to associate the small area on the projection pattern of the present invention with the extracted small area on the captured image, and corresponds to the coordinate data of the small area on the projection pattern. An association method for executing a flow for obtaining coordinate data of a small area on the captured image, and further, if necessary, a flow for obtaining the three-dimensional coordinate data of the measurement target object from the associated coordinate data Software for executing the three-dimensional position calculation method is provided.

The control device is preferably configured by a computer, but there is no problem as long as it has an arithmetic function and an I / O function.
If the resolution of the small area is set high, for example, it is possible to make each small area into a two-dimensional barcode pattern. (As is clear from the general use, it is very easy to identify two-dimensional barcode patterns.) However, since the resolution of the entire projection pattern is limited by the performance of the projector, When the resolution of the small area is increased, the number of small areas that can be set in the projection pattern is reduced, and there is a trade-off that the measurement data sampling becomes rough. In addition, there is a trade-off that when the area of the small region is increased in order to set the resolution of the small region high, the measurement accuracy and stability are reduced due to the influence of local shape deformation of the target object.

  As described above, the size and arrangement method of the small areas and how to set the pattern in each small area depend on the measurement target and the mounting. For example, in the embodiment by the inventor, a small area of 4 pixels × 4 pixels is provided, and each small area has a small color pattern by a combination of values in which RGB color information is set in binary. The variation of the small area in this embodiment is about several tens, but it works effectively when used together with the constraint condition (epipolar constraint) of the corresponding point search range by the three-dimensional arrangement of the camera and the projector.

FIG. 2 is an example of a projection pattern in the embodiment of the present invention.
In FIG. 2, small areas of 4 pixels × 4 pixels are arranged with an interval of 1 to 3 pixels, and 1200 small areas exist discretely in the entire projection pattern. In FIG. 2, the small areas are shown in black. Actually, however, each small area is set to one of several tens of RGB combination patterns as described above.

FIG. 3 shows a small area detected in a captured image obtained by projecting the projection pattern of FIG. 2 on a foot while walking and photographing the reflected light (that is, the pattern projected on the surface of the foot) with a camera. is there.
Since the projection pattern is composed of discrete small areas, discrete small areas are observed in the image. After detecting the small area in the image, the color distribution in each detected small area is examined. Since a characteristic color distribution that can be identified individually is set for each small area in the projection pattern, the color distribution of the small area detected in the image is also characteristic, and the corresponding point search between the camera and the camera is performed. In addition, or in the search for corresponding points between the camera and the projector, erroneous recognition can be greatly reduced.

(Extended example)
As described above, the present invention greatly reduces the misrecognition of corresponding points in an image captured by a camera, and is projected from a plurality of projectors onto a target object surface as shown in FIGS. Even if the projection patterns are mixed, the projector can distinguish which point is the projected point.
In FIG. 6, a projection pattern unique to each projector is projected onto the object 3 to be measured by the projectors 1 </ b> A and 1 </ b> B, in this case, the foot, and the projected image is captured by the imaging camera. By detecting the RGB pattern, the projection pattern unique to each projector is determined.

  As described above, when a pattern is projected simultaneously by a plurality of projectors, for example, in the case of two projectors, all the small areas included in the projection pattern of the projector 1B are included in all the small areas included in the projection pattern of the projector 1A. And have a different color distribution. By sorting the types of small areas for each projector, as shown in FIG. 6, by examining the distribution of RGB values in each small area from the projection patterns from a plurality of projectors observed in the image, The projector that projected the region can be identified.

(Processing flowchart of a method for associating a small area on the projection pattern with an extracted small area on the captured image)
FIG. 8 is a flowchart for explaining a method for associating a small area on the projection pattern of the present invention with an extracted small area on the captured image.
In FIG. 8, the block size of the small area is selected in advance according to the outer shape of the measurement target object, the required measurement accuracy, etc., and the content information of the color distribution that becomes the ID of the small area is selected in advance. These selection items can be automatically obtained. For example, the outer shape of the measurement target object is imaged, the block size of the small area is selected according to the degree of unevenness on the surface of the object, and the content information of the color distribution that becomes the ID of the small area may be selected. it can.

The program is started when the selection items are set in the corresponding program of the measuring device. Hereinafter, the procedure of the program is shown by a flowchart. Step (procedure) is abbreviated as “S”.
(1) Projecting a small area projection pattern (S1):
A projection pattern (see FIG. 2) that determines the shape, order, and arrangement pattern of the small regions is projected onto the measurement target object by the projector (see FIG. 3).
(2) Imaging (S2):
The measurement target object is imaged only once by the imaging camera according to the projection of the projector in S1 (see FIG. 3).

(3) Extract a small area from the captured image and detect the color distribution (S3):
A small area is appropriately extracted from the captured image, and the color distribution of the small area is detected.
(4) Did the small area in the projection pattern order correspond to the extracted small area of the captured image? (S4):
Was the small area in the projection pattern order associated with the extracted small area of the captured image? If YES, proceed to step S3. If NO, return to step S3.

(5) Have all the necessary small areas on the projection pattern been associated with the small areas on the captured image? (S5):
Have all or the necessary small areas on the projection pattern been associated with the small areas on the captured image? If YES, proceed to step S3. If NO, return to step S3.
The small area on the projection pattern to be associated can be set to all or a necessary range (a range set as appropriate) on the projection pattern.
By executing this association flowchart, all or a necessary range of association data on the projection pattern is obtained.

In FIG. 8, the three-dimensional position data of the surface of the measurement target object can also be obtained by executing the step 6 of calculating the three-dimensional position indicated by the dotted line after the association flowchart. Step 6 is as follows.
(6) Three-dimensional position calculation (S6):
Based on the coordinate data of the small area on the projection pattern obtained by executing the above association flowchart and the coordinate data of the small area on the captured image corresponding thereto, means for calculating a known three-dimensional shape, The three-dimensional coordinate data of the outer surface of the measurement target object is obtained.

(Three-dimensional shape calculation means)
The method for obtaining the three-dimensional coordinates on the surface of the object to be measured from the two-dimensional coordinate data on the projection pattern and the two-dimensional coordinate data on the captured image, which can be associated as described above, is a well-known matter in this field. The description is omitted here.

It is a block diagram of the three-dimensional shape measuring apparatus of this invention. It is an example of the projection pattern of this invention. It is the example which imaged the projection pattern of this invention projected on the leg | foot. The color distribution example 1 of this invention is shown. The color distribution 2 of this invention is shown. It is a figure explaining the example which provides multiple sets of the projector and imaging camera of this invention. It is a figure explaining the discrimination example at the time of providing multiple sets of the projector and imaging camera of this invention. It is a flowchart which shows the process sequence of this invention. It is explanatory drawing of the conventional triangulation. It is a figure which shows the example which switches the conventional projector pattern in large quantities. It is a figure explaining the example which sets ID for every pixel.

Explanation of symbols

1, 1A, 1B Projector 2, 2A, 2B Camera 3 Object to be measured 4 Small area 5 Control device

Claims (4)

Projecting a projection pattern consisting of a plurality of small regions having a size of 2 pixels or more and having a characteristic color distribution that can be distinguished from each other onto the object, and imaging the object on which the projection pattern is projected A three-dimensional shape measurement method characterized in that a correspondence relationship with the small area of the projection pattern is obtained based on a color distribution of an arbitrary small area in the captured image. 2. The three-dimensional shape measuring method according to claim 1, wherein the projection pattern is a pattern distributed discretely on a two-dimensional plane with an interval of one pixel or more around the small region. It consists of a control device, an imaging camera, and a projector, and the control device creates a projection pattern consisting of a plurality of small areas having a characteristic color distribution that has a size of 2 pixels or more and that can be distinguished from each other. Then, the small area of the projection pattern is projected based on the color distribution of an arbitrary small area in the picked-up image. A three-dimensional shape measuring apparatus characterized by obtaining a correspondence relationship between 4. The three-dimensional shape measuring apparatus according to claim 3, wherein the projection pattern is a pattern distributed discretely on a two-dimensional plane with an interval of one pixel or more around the small region.

Images