JP2018041169A - Information processing device and control method and program thereof - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a device and method capable of measuring a three-dimensional shape with high accuracy while suppressing calculation cost.SOLUTION: An information processing device acquires a picked-up image with a pattern image projected, which is obtained by imaging by an imaging device while the pattern image is projected by a projection device, calculates mapping representing geometrical distortion from a local area of an area corresponding to a local area having a known shape of a first pattern image in a picked-up image including the projection of the first pattern image, and searches a correspondence point by comparing a target area of a second pattern image with a corresponding area obtained by converting a target area in the picked-up image including the projection of the second pattern image on the basis of the mapping.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for obtaining a three-dimensional shape of an object based on a captured image.

  Projecting pattern light from a projector onto a target object and observing the reflected light with a camera to acquire the correspondence between the projected coordinates and imaging coordinates on the target object surface is necessary for three-dimensional measurement using active stereo. Known as technology. A target object obtained by projecting an aperiodic pattern image (such as random dots) to one of the active stereo methods is imaged with a camera, and a corresponding pixel is searched between the pattern image and the captured image. There is a method of obtaining a correspondence relationship between imaging coordinates. In such a search for corresponding pixels, a pixel that satisfies a predetermined condition (minimum, minimum, etc.) between the pattern image and the captured image, such as a luminance difference between the pixel of interest and its neighboring pixels (hereinafter referred to as a window), Search (hereinafter, corresponding point search) is performed.

  In the corresponding point search, a corresponding point is generally searched using a square window. However, since the projected pattern image is deformed depending on the shape of the target object, the viewpoint position and orientation of the camera, the pattern image viewed from the captured image is distorted. In general search for corresponding points, such a distortion is not taken into account, so that there is a problem that the accuracy of searching for corresponding points is lowered. In addition, since the distortion between the pattern image and the captured image increases as the baseline length of the projector and the camera increases, the distortion of the projected pattern image becomes a more serious problem.

  In order to solve such a problem, as a method for searching for a corresponding point in consideration of distortion, Non-Patent Document 1 uses a method for simultaneously estimating not only the corresponding point but also a parameter corresponding to the normal of the target object. Yes. According to Patent Document 1, by calculating a value such as a luminance difference after distorting the window according to the normal parameter of the target object, the correspondence between the projection coordinates and the imaging coordinates can be obtained with high accuracy. .

Michael Bleyer et al., Patch Match Stereo-Stereo Matching with Slanted Support Windows, British Machine Vision Conference 2011.

  However, since the method of Non-Patent Document 1 simultaneously estimates not only the corresponding points but also the window distortion method, the number of parameters to be estimated increases and the search range becomes enormous. Therefore, there is a problem that calculation cost becomes high.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide an apparatus and method capable of measuring a three-dimensional shape with high accuracy while suppressing calculation cost.

In order to achieve the above object, an information processing apparatus according to an aspect of the present invention includes the following arrangement. That is,
An acquisition means for acquiring a captured image on which the pattern image is projected, obtained by imaging with the imaging means in a state in which the pattern image is projected by the projection means;
Calculation for calculating a mapping representing a geometric distortion from the local region of a region corresponding to the local region having a known shape of the first pattern image in a captured image including the projection of the first pattern image Means,
Search for corresponding points by comparing the attention area in the second pattern image with the corresponding area obtained by converting the attention area in the captured image including the projection of the second pattern image based on the mapping. And a search means.

  According to the present invention, even when a pattern image appears distorted on a captured image, a three-dimensional shape can be measured with high accuracy at a low calculation cost.

1 is a block diagram illustrating a configuration example of an information processing apparatus according to a first embodiment. The figure which shows the pattern image for window shape calculation used in 1st Embodiment, and the pattern image for corresponding point search. The figure which shows the example which projected the pattern image for window shape calculation used in 1st Embodiment on the sphere. The figure explaining the process which detects the square of the pattern image for window shape calculation used in 1st Embodiment from the captured image. The figure explaining the process which calculates the mapping parameter of the distortion calculation part of 1st Embodiment. The figure explaining the process of the corresponding | compatible search part of 1st Embodiment. 5 is a flowchart for explaining measurement processing by the information processing apparatus according to the first embodiment. The figure which shows the pattern image which mixed the pattern image for window shape calculation used in the modification 1, and the pattern image for corresponding point search. The figure which shows the pattern image for window shape calculation used in the modification 2. FIG. The block diagram which shows the structural example of the information processing apparatus of 2nd Embodiment. 9 is a flowchart for explaining measurement processing by the information processing apparatus according to the second embodiment. The block diagram which shows the hardware structural example of information processing apparatus.

  DESCRIPTION OF EMBODIMENTS Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

<First Embodiment>
The information processing apparatus described in the first embodiment is a three-dimensional measuring instrument that measures a three-dimensional shape of a target object by projecting pattern light onto the target object and observing the reflected light. Pattern light projection is the projection of a predetermined pattern image. In the first embodiment, there are two types of pattern images: a first pattern image for calculating the distortion of the pattern image on the captured image, and a second pattern image for determining the correspondence between the projected coordinates and the captured coordinates. Is used. Hereinafter, the first pattern image is referred to as a window shape calculation pattern image, and the second pattern image is referred to as a corresponding point search pattern image. The information processing apparatus according to the first embodiment detects distortion from a captured image captured in a state in which a window shape calculation pattern image is projected, and obtains a window and a corresponding point search pattern image in which the detected distortion is considered. High-precision three-dimensional coordinates are calculated by searching for corresponding points.

[Device configuration]
FIG. 1 is a block diagram illustrating a configuration example of a three-dimensional measurement apparatus 1000 including the information processing apparatus 100 according to the first embodiment. The projection device 1 projects the window shape calculation pattern image and the corresponding point search pattern image onto the target object. These pattern lights are reflected by the surface of the target object and imaged by the imaging device 2. An image (captured image) captured by the imaging apparatus 2 is sent to the information processing apparatus 100. The information processing apparatus 100 calculates the three-dimensional coordinates of the target object based on the captured image from the imaging apparatus 2. Further, the information processing apparatus 100 controls operations of the projection apparatus 1 and the imaging apparatus 2. The projection device 1 and the imaging device 2 are calibrated in advance to make their coordinate systems coincide with each other, and calibration data is stored in the information processing apparatus 100.

  The information processing apparatus 100 includes a projection control unit 101, an image input unit 102, a distortion calculation unit 103, a correspondence search unit 104, and a coordinate calculation unit 105. The projection control unit 101 controls the projection apparatus 1 to project the window shape calculation pattern image and the corresponding point search pattern image in order, and outputs a control signal to the image input unit 102. The image input unit 102 controls the imaging device 2 in accordance with the timing of the control signal input from the projection control unit 101, and receives captured images that are captured in a state where each pattern image is projected. Further, the image input unit 102 outputs the captured image obtained in a state where the window shape calculation pattern image is projected to the distortion calculation unit 103, and the imaging obtained in the state where the corresponding point search pattern image is projected. The image is output to the correspondence search unit 104.

  The distortion calculation unit 103 compares the pattern image for window shape calculation and the captured image input from the image input unit 102, and calculates a distortion parameter indicating how each pixel in the captured image is mapped (distortion). Calculate and output to the correspondence search unit 104. The correspondence search unit 104 generates a window in consideration of distortion in the captured image input from the image input unit 102 based on the distortion parameter input from the distortion calculation unit 103, and generates a corresponding point search pattern image and the captured image. Search for corresponding points between them. The correspondence search unit 104 outputs the obtained correspondence between the projection coordinates of the projection device 1 and the imaging coordinates of the imaging device 2 to the coordinate calculation unit 105. The coordinate calculation unit 105 performs the three-dimensional surface of the target object based on the correspondence relationship between the projection coordinates and the image coordinates input from the correspondence search unit 104 and the calibration data of the projection apparatus 1 and the imaging apparatus 2 obtained in advance. Calculate the coordinates.

  FIG. 12 is a diagram illustrating a hardware configuration example of the information processing apparatus 100 that implements each functional unit illustrated in FIG. 1. In FIG. 12, a CPU 161 includes one or more computers (processors), and executes various processes of the information processing apparatus 100 by executing programs stored in the ROM 162 or the RAM 163. The ROM 162 is a read-only nonvolatile memory, and the RAM 163 is a volatile memory that can be written at any time. The RAM 163 also functions as a work memory for the CPU 161. The secondary storage device 164 is composed of, for example, a hard disk. The secondary storage device 164 stores a program 171, a pattern image 172, and calibration data 173. The program 171 includes a program for causing the CPU 161 to execute processing described later with reference to the flowchart of FIG. The pattern image 172 includes, for example, a window shape calculation pattern image and a corresponding point search pattern image. The calibration data is data for causing the projection apparatus 1 and the imaging apparatus 2 to match each other's coordinate systems. The display unit 165 performs various displays under the control of the CPU 161. A keyboard 166 and a pointing device 167 are instruction input units for receiving instructions from the user. The interface 168 connects the projection device 1 and the imaging device 2 to the information processing device 100. Each configuration described above is connected to the bus 169. Each functional unit shown in FIG. 1 may be realized by the CPU 161 executing a program, or may be realized by dedicated hardware.

[Measurement processing]
The measurement process by the information processing apparatus 100 according to the first embodiment will be described with reference to the flowchart of FIG.

  When the three-dimensional measuring apparatus 1000 is activated, an initialization process is executed in step S11. In the initialization process, the projection device 1 and the imaging device 2 are activated, the calibration data of the projection device 1 and the imaging device 2 is read into the work memory, the window shape calculation pattern image and the corresponding point search pattern image are stored in the work memory. Includes processing to read.

  In step S <b> 12, the projection control unit 101 causes the projection apparatus 1 to project a window shape calculation pattern image and sends a control signal to the image input unit 102. The window shape calculation pattern image used in the present embodiment is a pattern in which squares are regularly arranged as shown in FIG. In FIG. 2A, the white portion 11 in the pattern image indicates that light is projected, and the black portion 12 indicates that light is not projected. In response to receiving the control signal from the projection control unit 101, the image input unit 102 causes the imaging device 2 to perform imaging and inputs a captured image. The captured image input in this way is an image captured in a state in which the window shape calculation pattern image is projected. The image input unit 102 outputs the captured image input from the imaging device 2 to the distortion calculation unit 103.

  In step S <b> 13, the projection control unit 101 causes the projection device 1 to project the corresponding point search pattern image and sends a control signal to the image input unit 102. The corresponding point search pattern image used in this embodiment is, for example, an aperiodic image in which dots are randomly arranged as shown in FIG. The corresponding point search pattern image is preferably an image in which two or more local regions having the same characteristics do not appear in one search direction. In response to receiving the control signal from the projection control unit 101, the image input unit 102 causes the imaging device 2 to capture an image. In this way, the image input unit 102 inputs a captured image captured in a state where the corresponding point search pattern image is projected onto the target object from the imaging device 2 and outputs the captured image to the correspondence search unit 104.

  In step S14, the distortion calculation unit 103 detects a region corresponding to a square in the window shape calculation pattern image from the captured image input from the image input unit 102, and how the detected region is compared with the square. It is parameterized whether it is done (distortion). The distortion indicates that a local area (a square area in this example) in the window shape calculation pattern image is distorted into a parallelogram or the like in the captured image. For example, when the window shape calculation pattern image shown in FIG. 2A is projected onto a sphere by the projection device 1 and imaged by the imaging device 2, a captured image as shown in FIG. 3B is obtained. When this captured image is enlarged, it can be seen that the pattern image is distorted as shown in FIG.

  In order to parameterize the geometric distortion of the region corresponding to the local region of the window shape calculation pattern image in such a captured image, the distortion calculation unit 103 first starts from the captured image (FIG. 4A). A white line in the window shape calculation pattern image is detected. For example, the distortion calculation unit 103 determines whether or not the pixel is a line pixel (candidate pixel) according to the luminance of the captured image, and sets a set of candidate pixels that are substantially continuous in the vertical and horizontal directions as vertical and horizontal lines. Detect (line 21 in FIG. 4B). Then, the distortion calculation unit 103 sets an area surrounded by two adjacent vertical lines and two horizontal lines to an area corresponding to a square area in the window shape calculation pattern image (area 23 in FIG. 4C). ) To detect. For example, the distortion calculation unit 103 obtains a point included in both the vertical line and the horizontal line as an intersection of the lines (intersection 22 in FIG. 4B), and the intersections adjacent in the vertical direction and the horizontal direction are the same vertical. Judge whether it is on a line or a horizontal line. The distortion calculation unit 103 detects a region surrounded by intersections determined to have adjacent intersections on the same vertical and horizontal lines as a region corresponding to a square in the window shape calculation pattern image.

Next, the distortion calculation unit 103 parameterizes the distortion of all the areas in the captured image corresponding to the square in the window shape calculation pattern image. There are various methods for parameterization, and any expression may be used. For example, there is a method of parameterizing as a two-dimensional linear transformation. When the coordinates before conversion are (u, v) and the coordinates after conversion are (x, y), the two-dimensional linear conversion is expressed by the following equation (1).

  In Equation 1, f represents a mapping, and a, b, c, and d are parameters indicating linear transformation expressing f. These parameters are obtained from the correspondence between the square in the window shape calculation pattern image and the area detected from the captured image. If one point ((0, 0) in FIG. 5) is used as a reference, the corresponding relationship of the points or line segments ((0, 1) and (x1, y1), (1, 1) and (x2, in FIG. 5) y2), (1,0) and (x3, y3)) are obtained. A parameter may be obtained analytically from this correspondence, or an optimum parameter may be obtained by a method such as a least square method. It is assumed that the pattern image is distorted with the parameters obtained in this manner for the pixels in the detected area.

  By performing a series of processes in all regions in the captured image, the mapping f is determined for each pixel of the captured image. An area in the captured image corresponding to the square in the window shape calculation pattern image may or may not be detected. In such an area, the mapping f may be used as a unit matrix, or mapping in neighboring pixels. Interpolation may be performed by taking a linear sum of f. The distortion calculation unit 103 outputs the obtained distortion parameter (f for each pixel) to the correspondence search unit 104.

In step S15, the correspondence search unit 104 generates a window based on the distortion parameter input from the distortion calculation unit 103, and a corresponding point between the corresponding point search pattern image and the captured image input from the image input unit 102. Perform a search. In the corresponding point search, the cost based on the luminance difference between the target pixel and its neighboring pixels (for example, eight pixels around the target pixel) between the corresponding point search pattern image and the captured image satisfies the condition (minimum, minimum) Etc.) Search for points in the epipolar direction and find corresponding points. In general, the cost is calculated using a square window as a region of interest, but in this embodiment, cost calculation is performed for the pixels on which the pattern image is projected in consideration of the distortion f of the pattern image, and corresponding points are searched. To do. Corresponding point search pattern image brightness I, captured image brightness J, target pixel p, target pixel p neighboring pixel q, target pixel p and neighboring pixel q set Wp, corresponding target pixel p Let r be the pixel and fr be the mapping at the corresponding pixel r. In the captured image including the projection of the corresponding point search pattern image, the corresponding region is arranged around the corresponding pixel r by converting the positions of the surrounding neighboring pixels based on the mapping fr calculated for the corresponding pixel r. Area. The cost function C for calculating the cost between the corresponding area and the attention area is set as follows, for example.

  FIG. 6 shows an example of corresponding point search in consideration of how the pattern image is distorted. In the corresponding point search pattern image shown in FIG. 6A, the white circle 31 is the target pixel p, the white frame 32 is the target region (window) in which the target pixel p and its neighboring pixels are formed, and the white line 33 is the epipolar line. . A white circle 34 in the captured image shown in FIG. 6B is the corresponding pixel r. The white frame 35 is a corresponding region (window) in consideration of the distortion indicated by the mapping (fr) in the corresponding pixel r, and is formed by the corresponding pixel r and its neighboring pixels. In the corresponding point search, the attention pixel p (white circle 31) is moved along the epipolar line (white line 33), and the attention area and the corresponding area are compared at each position of the attention pixel p. The attention pixel p that satisfies the predetermined condition (minimum, minimum, etc.) for the attention area and the corresponding area calculated by the cost function of Equation 2 is obtained as the corresponding point. By generating a window in consideration of distortion on the captured image, the reference pixel may take a sub-pixel value. In such a case, if the luminance is calculated by bilinear interpolation, the cost is calculated. Good. By performing this process on all pixels on the captured image, corresponding points (pairs of p and r) are obtained by the number of pixels. Correspondences between all the obtained projection coordinates and imaging coordinates (a pair of p and r corresponding to the number of pixels) are output to the coordinate calculation unit 105.

  In step S <b> 16, the coordinate calculation unit 105 performs triangulation from the correspondence relationship between the projection coordinates and the imaging coordinates input from the correspondence search unit 104 and the calibration data of the projection apparatus 1 and the imaging apparatus 2 obtained in advance. In this way, three-dimensional coordinates ((X, Y, Z) coordinates corresponding to the number of corresponding points) of the surface of the target object are calculated, and the process ends.

  Note that the order of the steps does not necessarily have to be the order shown in the flowchart of FIG. 7, and may be changed as appropriate or may be processed in parallel.

  As described above, according to the first embodiment, the window is generated in consideration of how the projection pattern image is distorted in the captured image, and the corresponding point search is performed, so that the three-dimensional coordinates are obtained with high accuracy. Can be calculated.

[Modification 1]
In the above embodiment, the window shape calculation pattern image is projected in step S12, the corresponding point search pattern image is projected in step S13, and the captured images including the projection of these pattern images are obtained separately. It is not limited. For example, as shown in FIG. 8, a pattern image (FIG. 8C) obtained by mixing a window shape calculation pattern image (FIG. 8A) and a corresponding point search pattern image (FIG. 8B) is projected. Alternatively, this may be imaged and used for obtaining parameters indicating distortion and searching for corresponding points. Alternatively, the pattern images may be projected at the same time by changing the wavelength of light for projecting each pattern image and using an imaging device corresponding to the wavelengths. For example, a color image is captured in a state where a window shape calculation pattern is projected with light of a first wavelength (for example, red) and a corresponding point search pattern is projected with light of a second wavelength (for example, green). Do. A parameter indicating distortion is calculated by detecting a red line from the obtained captured image, and a green point is detected as a search target. As described above, according to the configuration of the first modification, measurement can be performed with a single projection and imaging, so that even a moving target object can be calculated with high accuracy.

[Modification 2]
Although FIG. 2 is shown as an example of the pattern image projected in step S12 or step S13, the pattern image is not limited to these. In particular, the window shape calculation pattern image does not need to be regularly arranged with known geometric shapes (squares) as shown in FIG. 2, and it is sufficient that there are at least one known geometric shape. For example, as shown in FIG. 9, pattern images in which equilateral triangles 41 and squares 42 are arranged as different types of known geometric shapes may be used. However, in order to obtain the distortion parameter for each pixel, it is desirable that the known shapes exist evenly in the window shape calculation pattern image.

In order to obtain the distortion parameter, it is necessary to detect a known geometric shape (hereinafter, known shape is used) from the captured image. For example, to detect a known shape,
A method for detecting hand edges by applying an edge detection filter such as a Sobel filter to a captured image, and detecting edges that are connected and making a round as a known shape,
A method of detecting a known shape by taking a matching using image features such as LBP (Local Binary Pattern) and SIFT (Scale-Invariant Feature Transform) using a known shape as a template.

  In addition, by allowing different types of different known shapes to exist for each type in the divided regions in the window shape calculation pattern image, the correspondence search unit 104 can determine which part of the pattern image each pixel has along with the distortion parameter of each pixel. You can see if it corresponds. For example, in the case of FIG. 9, the equilateral triangle 41 in the captured image corresponds to the upper part of the pattern image, and the square 42 corresponds to the lower part of the pattern image. Therefore, the correspondence search unit 104 can grasp that the pixel in the equilateral triangle 41 exists above the captured image, and the pixel in the square 42 exists below the captured image. As a result, the correspondence search unit 104 can limit the search range shown in FIG. 6A to the upper half of the epipolar line, for example, and can calculate the three-dimensional coordinates with high speed and high accuracy. it can.

[Modification 3]
In step S12 and step S13, the pattern image to be projected need not be limited to binary, and may be a pattern image having gradation or color tone. For example, as the window shape calculation pattern, an image in which known geometric shapes with different gradations and color tones are arranged can be used. What is necessary is just to detect several known geometric shapes from which a gradation and a color tone differ as a characteristic in the brightness | luminance, hue, etc. in a captured image. By doing in this way, as in Modification 2, it is possible to know which part in the pattern image corresponds to the distortion parameter of each pixel, so that the search range of the correspondence search unit 104 can be limited. Therefore, the three-dimensional coordinates can be calculated with high speed and high accuracy.

[Modification 4]
In step S15, the distortion parameter obtained in step S14 is used as an initial value, and the corresponding point and the window distortion method may be estimated simultaneously as in Non-Patent Document 1 (the parameter corresponding to fr in Equation 2 is also estimated simultaneously. To do). That is, the correspondence search unit 104 updates the mapping fr according to the update of the corresponding pixel r, using the mapping calculated by the distortion calculation unit 103 as an initial value. At this time, as in the first embodiment, the mapping may be searched by full search in the region of the corresponding point search pattern image corresponding to the local region of the window calculation pattern image, or Graph Cut, Belief The mapping may be searched using an algorithm such as Propagation or PatchMatch. By doing so, a distortion parameter that is correct to some extent can be set as an initial value, so that the search cost of mapping can be suppressed. Therefore, the three-dimensional coordinates can be calculated with high speed and high accuracy.

<Second Embodiment>
In the first embodiment, one projection device and one imaging device are configured. In the second embodiment, pattern light is projected onto a target object, and the reflected light is observed by two imaging devices. A configuration for measuring the three-dimensional shape of the target object will be described. The information processing apparatus according to the second embodiment generates a window in consideration of the distortion calculated from the window shape calculation pattern image, and searches for corresponding points between the captured images obtained from the two imaging apparatuses. Calculate accurate 3D coordinates. By using two imaging devices, it is possible to suppress a decrease in accuracy due to optical influences such as the projected pattern image being blurred or the pattern image being destroyed due to mutual reflection or the like.

[Device configuration]
FIG. 10 is a block diagram illustrating a configuration example of a three-dimensional measurement apparatus 1000 including the information processing apparatus 100 according to the second embodiment. This embodiment differs from the first embodiment in that one imaging device is added and the first imaging device 2 and the second imaging device 3 are provided. The first imaging device 2 and the third imaging device 3 perform imaging from different positions in a state where the pattern image is projected by the projection device 1. Note that the projection device 1, the first imaging device 2, and the second imaging device 3 have been calibrated in advance to match the coordinate systems, and calibration data is stored in the information processing apparatus 100. . Note that the hardware configuration of the information processing apparatus 100 is the same as that of the first embodiment (FIG. 12).

  The information processing apparatus 100 includes a projection control unit 101, an image input unit 102, a distortion calculation unit 103, a correspondence search unit 104, and a coordinate calculation unit 105. The projection control unit 101 controls the projection device 1 to project a pattern image obtained by mixing the window shape calculation pattern image and the corresponding point search pattern image (hereinafter referred to as a mixed pattern image), and to control the image input unit 102. Output a signal. An example of the mixed pattern image is shown in FIG. As in the first embodiment, the window shape calculation pattern image (FIG. 8 (a)) and the corresponding point search pattern image (FIG. 8 (b)) are projected individually and sequentially. Also good. The image input unit 102 controls the first imaging device 2 and the second imaging device 3 in accordance with the timing of the control signal input from the projection control unit 101 to execute imaging, and mix patterns from each. A captured image on which the image is projected is received. The image input unit 102 outputs the two captured images on which the mixed pattern image is projected to the distortion calculation unit 103.

  Based on the window shape calculation pattern image included in the mixed pattern image and the two captured images input from the image input unit 102, the distortion calculation unit 103 determines how each pixel in each captured image is displayed. A distortion parameter indicating whether mapping is performed is calculated. The distortion calculation unit 103 outputs the calculated distortion parameters of the two captured images to the correspondence search unit 104. The correspondence search unit 104 generates a window in consideration of distortion based on the distortion parameter input from the distortion calculation unit 103 and searches for corresponding points between the two captured images input from the image input unit 102. . The correspondence search unit 104 outputs the relationship between the imaging coordinates of the first imaging device 2 and the imaging coordinates of the second imaging device 3 obtained by the corresponding point search to the coordinate calculation unit 105. The coordinate calculation unit 105 calculates three-dimensional coordinates from the correspondence relationship between the imaging coordinates input from the correspondence search unit 104 and the calibration data of the first imaging device 2 and the second imaging device 3 obtained in advance.

[Measurement processing]
Measurement processing by the information processing apparatus 100 according to the second embodiment will be described with reference to the flowchart of FIG.

  When the three-dimensional measuring apparatus 2000 is activated, an initialization process is executed in step S21. In the initialization process, the start-up process of the projection device 1, the first imaging device 2 and the second imaging device 3, the calibration data of the projection device 1, the first imaging device 2, and the second imaging device 3 are stored. A process for reading into the working memory, a process for reading a mixed pattern image, and the like are included.

  In step S <b> 22, the projection control unit 101 causes the projection apparatus 1 to project the mixed pattern image and sends a control signal to the image input unit 102. Upon receiving the control signal, the image input unit 102 causes the first imaging device 2 and the second imaging device 3 to capture an image in which the mixed pattern image is projected on the target object. The image input unit 102 outputs the two input captured images to the distortion calculation unit 103.

  In step S23, the distortion calculation unit 103 determines how the square area in the window shape calculation pattern image is mapped to the captured image for each of the two captured images input from the image input unit 102. Calculate the distortion parameters shown. The process for obtaining the distortion parameter is the same as in the first embodiment (S14). The obtained distortion parameters (f for each pixel) and two captured images are output to the correspondence search unit 104.

In step S24, the correspondence search unit 104 generates a window in each of the two captured images based on the distortion parameter input from the distortion calculation unit 103, and searches for corresponding points between the two captured images. . As in the first embodiment, the cost calculation considering the distortion f of the pattern image is performed on the pixel on which the corresponding point search pattern image is projected, and the corresponding point is searched. The brightness of the captured image of the first imaging device 2 is I, the brightness of the captured image of the second imaging device 3 is J, the target pixel is p, the mapping at the target pixel p is fp, and the neighboring pixels of the target pixel p are q ( For example, the set of the pixel of interest p and the neighboring pixel q is Wp. In addition, a pixel corresponding to the target pixel p is r, and a mapping at the corresponding pixel r is fr. For example, the cost function C is calculated by the following equation 3.

  Corresponding point search processing using a cost function is the same as in the first embodiment. That is, the correspondence search unit 104 moves the pixel of interest along the epipolar line, and obtains the pixel of interest p with the cost function of Equation 3 satisfying a predetermined condition (minimum, minimum, etc.). As described above, when the corresponding points are searched for the two captured images, even if the pattern images are broken due to blurring or mutual reflection, the captured images look the same, and therefore the corresponding points are preferably searched. be able to. The correspondence search unit 104 outputs a correspondence relationship (a pair of p and r corresponding to the number of pixels) between all obtained imaging coordinates to the coordinate calculation unit 105.

  In step S <b> 25, the coordinate calculation unit 105 calculates a triangle from the correspondence between the imaging coordinates input from the correspondence search unit 104 and the calibration data obtained in advance for the first imaging device 2 and the second imaging device 3. Three-dimensional coordinates ((X, Y, Z) coordinates corresponding to the number of corresponding points) are calculated by surveying. Then, this process ends.

  As described above, according to the second embodiment, a window is generated in consideration of how the projection pattern image is distorted in each captured image, and a corresponding point search is performed. In addition, according to the second embodiment, by using two imaging devices, it is possible to suppress a decrease in accuracy due to optical influences such as blurring and mutual reflection. Therefore, the three-dimensional coordinates are calculated with high accuracy. be able to.

[Modification 5]
In step S24, not only the correspondence between the first imaging device 2 and the second imaging device 3, but also the projection device and the first imaging device 2, and the projection device and the second imaging device as in the first embodiment. The correspondence relationship of the devices 3 may be obtained. By doing in this way, the part which cannot measure a three-dimensional coordinate by shielding etc. can be reduced.

[Modification 6]
In the second embodiment, one projection device and two imaging devices are used. However, the number of projection devices and imaging devices is not limited to this, and any number of projection devices and imaging devices may be provided. Good. In step S23, the projection device and the imaging device may be paired to determine the distortion. In step S24, the cost of each pair may be calculated to obtain the correspondence, or all costs may be integrated to obtain the correspondence as in Multi-view Stereo. By doing in this way, the part which cannot be measured by shielding etc. can be reduced and the whole target object can be measured more reliably.

<Effect of embodiment>
According to the first embodiment, it is possible to calculate a three-dimensional coordinate with high accuracy by generating a window in consideration of how the projection pattern image is distorted in the captured image and performing a corresponding point search. it can.
According to the second embodiment, in addition to the effects of the first embodiment, by using two cameras, a reduction in accuracy due to optical influences such as blurring and mutual reflection is suppressed, so that three-dimensional Coordinates can be calculated with high accuracy.

<Definition>
As described in the first embodiment and the first, second, and third modifications, the pattern image that the projection control unit 101 has can generate any window shape that takes distortion into consideration and perform corresponding point search. It may be a pattern image.
Further, as described in the first embodiment and the fourth modification, the correspondence search unit 104 may use any algorithm as long as it can obtain corresponding points.

  Further, as described in the second embodiment and the sixth modification, the projection device 1 and the imaging device 2 do not have to be limited to one each, and may be a plurality.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

1000: three-dimensional measurement device, 1: projection device, 2: imaging device, 100: information processing device, 101: projection control unit, 102: image input unit, 103: distortion calculation unit, 104: correspondence search unit, 105: coordinates Calculation unit

Claims (16)

An acquisition means for acquiring a captured image on which the pattern image is projected, obtained by imaging with the imaging means in a state in which the pattern image is projected by the projection means;
Calculation for calculating a mapping representing a geometric distortion from the local region of a region corresponding to the local region having a known shape of the first pattern image in a captured image including the projection of the first pattern image Means,
Search for corresponding points by comparing the attention area in the second pattern image with the corresponding area obtained by converting the attention area in the captured image including the projection of the second pattern image based on the mapping. An information processing apparatus comprising: a search unit that performs the search. An acquisition means for acquiring captured images captured by the first imaging means and the second imaging means from different positions in a state in which the pattern image is projected by the projection means;
For each of the first and second captured images including the projection of the first pattern image acquired from the first and second imaging means, a local area whose shape is known in the first pattern image Calculating means for calculating a map representing the geometric distortion of the corresponding region;
Of the third and fourth captured images including the projection of the second pattern image acquired from the first and second imaging means, the attention area in the third captured image, and the fourth captured image Search means for searching for corresponding points by comparison with corresponding regions in an image,
The corresponding area is set by converting the attention area of the third captured image based on the map calculated for the first captured image and the map calculated for the second captured image. An information processing apparatus characterized by the above.   The information processing apparatus according to claim 1, wherein the search unit searches for the attention area along an epipolar line corresponding to a point of the corresponding area. The calculation means calculates a mapping for each pixel of the captured image,
The region of interest is formed by a pixel of interest and neighboring pixels around it,
4. The corresponding region is a region arranged around a corresponding pixel by converting the position of the surrounding neighboring pixels based on a mapping calculated for the corresponding pixel. The information processing apparatus according to any one of claims.   The information processing apparatus according to claim 4, wherein the search unit updates the map according to the update of the corresponding pixel, with the map calculated by the calculation unit as an initial value.   The acquisition unit acquires a captured image including the projection of the first pattern image and a captured image including the projection of the second pattern image as separate captured images. The information processing apparatus according to any one of the above.   The said acquisition means acquires the picked-up image containing projection of the pattern image which mixed the said 1st pattern image and the said 2nd pattern image, The any one of Claim 1 thru | or 5 characterized by the above-mentioned. Information processing device. The first pattern image includes at least one known geometry;
The information processing apparatus according to claim 1, wherein the second pattern image has an aperiodic pattern.   The information processing apparatus according to claim 1, wherein the first pattern image is an image in which known geometric shapes are arranged in different gradations or color tones.   The information processing apparatus according to claim 1, wherein the first pattern image is a pattern image in which known geometric shapes are regularly arranged.   The information according to any one of claims 1 to 10, further comprising coordinate calculation means for calculating a three-dimensional coordinate of the surface of the target object based on the corresponding points searched by the search means. Processing equipment.   The information processing apparatus according to claim 1, further comprising a projecting unit.   The information processing apparatus according to claim 1, further comprising an imaging unit. A method for controlling an information processing apparatus,
An acquisition step of acquiring a captured image on which the pattern image is projected, obtained by imaging by the imaging unit in a state where the pattern image is projected by the projection unit;
Calculation for calculating a mapping representing a geometric distortion from the local region of a region corresponding to the local region having a known shape of the first pattern image in a captured image including the projection of the first pattern image Process,
Search for corresponding points by comparing the attention area in the second pattern image with the corresponding area obtained by converting the attention area in the captured image including the projection of the second pattern image based on the mapping. A method for controlling the information processing apparatus. A method for controlling an information processing apparatus,
An acquisition step of acquiring captured images captured by the first imaging unit and the second imaging unit from different positions in a state in which the pattern image is projected by the projection unit;
For each of the first and second captured images including the projection of the first pattern image acquired from the first and second imaging means, a local area whose shape is known in the first pattern image A calculation step for calculating a map representing the geometric distortion of the corresponding region;
Of the third and fourth captured images including the projection of the second pattern image acquired from the first and second imaging means, the attention area in the third captured image, and the fourth captured image A search step for searching for a corresponding point by comparing with a corresponding region in an image,
The corresponding area is set by converting the attention area of the third captured image based on the map calculated for the first captured image and the map calculated for the second captured image. An information processing apparatus characterized by the above.   A program for causing a computer to execute each step of the control method of the information processing apparatus according to claim 14 or 15.