JP6867766B2 - Information processing device and its control method, program - Google Patents

Description

The present invention relates to a technique for obtaining a three-dimensional shape of an object based on a captured image.

The technology to acquire the correspondence between the projected coordinates and the imaging coordinates on the surface of the target object by projecting the pattern light from the projector onto the target object and observing the reflected light with the camera is necessary for three-dimensional measurement by active stereo. Known as technology. One of the active stereo methods is to capture a target object on which an aperiodic pattern image (random dots, etc.) is projected with a camera, and search for the corresponding pixel between the pattern image and the captured image to obtain the projected coordinates. There is a method of finding the correspondence between the imaging coordinates. In such a search for corresponding pixels, a pixel whose value such as the brightness difference between the pixel of interest and its neighboring pixels (hereinafter, window) satisfies a predetermined condition (minimum, minimum, etc.) between the pattern image and the captured image is selected. A search (hereinafter referred to as a corresponding point search) is performed.

In the correspondence point search, it is common to search for the correspondence point using a square window. However, since the projected pattern image is deformed depending on the shape of the target object, the viewpoint position and orientation of the camera, and the like, the pattern image seen from the captured image is distorted. Since such distortion is not taken into consideration in the general correspondence point search, there is a problem that the accuracy of the correspondence point search becomes low. Further, the longer the baseline length of the projector and the camera, the greater the distortion between the pattern image and the captured image, so that the distortion of the projected pattern image becomes a more serious problem.

In order to solve such a problem, as a method of searching for a corresponding point in consideration of distortion, Non-Patent Document 1 uses a method of simultaneously estimating not only the corresponding point but also the parameter corresponding to the normal of the target object. There is. According to Patent Part 1, the correspondence between the projected coordinates and the imaging coordinates can be obtained with high accuracy by calculating the value such as the brightness difference after distorting the window according to the normal parameter of the target object. ..

Michael Bleyer et al., Patch Match Stereo --Stereo Matching with Slanted Support Windows, British Machine Vision Conference 2011.

However, in the method of Non-Patent Document 1, not only the corresponding points but also the distortion of the window is estimated at the same time, so that the number of parameters to be estimated increases and the search range becomes enormous. Therefore, there is a problem that the calculation cost becomes high.

The present invention has been made in view of the above problems, and an object of the present invention is to provide an apparatus and a method capable of measuring a three-dimensional shape with high accuracy while suppressing a calculation cost.

The information processing apparatus according to one aspect of the present invention for achieving the above object has the following configuration. That is,
In the captured image in which the first pattern image is captured a state of being projected to have at least one known geometry, the area corresponding to the geometry of the first pattern image, geometry from the geometry A calculation means for calculating a mapping representing a geometrical distortion, and
A region of interest consists of neighboring pixels surrounding the target pixel in the second pattern image by the second pattern image is converted based on the region in the captured image obtained by capturing an image of a state of being projected onto the mapping A search means for searching for a corresponding point between the second pattern image and the captured image by comparison with the obtained corresponding area is provided .
The corresponding region is arranged around the corresponding pixel in the captured image obtained by capturing the projected state of the second pattern image, with the positions of the surrounding neighboring pixels converted for the corresponding pixel based on the mapping. area der Ru.

According to the present invention, even when the pattern image looks distorted on the captured image, the three-dimensional shape can be measured with high accuracy at a low calculation cost.

The block diagram which shows the structural example of the information processing apparatus of 1st Embodiment. The figure which shows the pattern image for window shape calculation and the pattern image for correspondence point search used in 1st Embodiment. The figure which shows the example which projected the pattern image for window shape calculation used in 1st Embodiment on a sphere. The figure explaining the process of detecting the square of the pattern image for window shape calculation used in 1st Embodiment from the captured image. The figure explaining the process of calculating the mapping parameter of the distortion calculation part of 1st Embodiment. The figure explaining the process of the correspondence search part of 1st Embodiment. The flowchart explaining the measurement process by the information processing apparatus of 1st Embodiment. The figure which shows the pattern image which mixed the pattern image for window shape calculation used in the modification 1 and the pattern image for search for corresponding points. The figure which shows the pattern image for window shape calculation used in the modification 2. The block diagram which shows the structural example of the information processing apparatus of 2nd Embodiment. The flowchart explaining the measurement process by the information processing apparatus of the 2nd Embodiment. The block diagram which shows the hardware configuration example of an information processing apparatus.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

<First Embodiment>
The information processing device described in the first embodiment is a three-dimensional measuring instrument that measures a three-dimensional shape of a target object by projecting pattern light onto the target object and observing the reflected light. The projection of pattern light is to project a predetermined pattern image. In the first embodiment, there are two types of pattern images, a first pattern image for calculating the distortion of the pattern image on the captured image and a second pattern image for obtaining the correspondence between the projected coordinates and the captured coordinates. Is used. Hereinafter, the first pattern image is referred to as a window shape calculation pattern image, and the second pattern image is referred to as a corresponding point search pattern image. The information processing apparatus of the first embodiment detects distortion from the captured image captured in a state where the pattern image for window shape calculation is projected, and displays the window and the pattern image for searching for corresponding points in consideration of the detected distortion. Highly accurate three-dimensional coordinates are calculated by performing a corresponding point search using this.

[Device configuration]
FIG. 1 is a block diagram showing a configuration example of a three-dimensional measuring device 1000 including the information processing device 100 of the first embodiment. The projection device 1 projects the window shape calculation pattern image and the corresponding point search pattern image onto the target object. These pattern lights are reflected on the surface of the target object and are imaged by the image pickup apparatus 2. The image (captured image) captured by the image pickup device 2 is sent to the information processing device 100. The information processing device 100 calculates the three-dimensional coordinates of the target object based on the captured image from the image pickup device 2. Further, the information processing device 100 controls the operations of the projection device 1 and the image pickup device 2. The projection device 1 and the image pickup device 2 have been calibrated in advance to match their coordinate systems, and the calibration data is stored in the information processing device 100.

The information processing device 100 includes a projection control unit 101, an image input unit 102, a distortion calculation unit 103, a correspondence search unit 104, and a coordinate calculation unit 105. The projection control unit 101 controls the projection device 1, projects the window shape calculation pattern image and the corresponding point search pattern image in order, and outputs a control signal to the image input unit 102. The image input unit 102 controls the image pickup device 2 in accordance with the timing of the control signal input from the projection control unit 101, and receives the captured image captured in the state in which each pattern image is projected. Further, the image input unit 102 outputs the captured image obtained in the state where the window shape calculation pattern image is projected to the distortion calculation unit 103, and the imaging obtained in the state where the corresponding point search pattern image is projected. The image is output to the corresponding search unit 104.

The distortion calculation unit 103 compares the pattern image for window shape calculation with the captured image input from the image input unit 102, and sets a distortion parameter indicating how each pixel in the captured image is mapped (distortion). It is calculated and output to the corresponding search unit 104. The correspondence search unit 104 generates a window in consideration of distortion in the captured image input from the image input unit 102 based on the distortion parameter input from the distortion calculation unit 103, and combines the corresponding point search pattern image and the captured image. Search for corresponding points between. The correspondence search unit 104 outputs to the coordinate calculation unit 105 the correspondence relationship between the obtained projection coordinates of the projection device 1 and the imaging coordinates of the imaging device 2. The coordinate calculation unit 105 is three-dimensional on the surface of the target object based on the correspondence relationship between the projected coordinates and the image coordinates input from the correspondence search unit 104 and the calibration data of the projection device 1 and the image pickup device 2 obtained in advance. Calculate the coordinates.

FIG. 12 is a diagram showing a hardware configuration example of the information processing apparatus 100 that realizes each functional unit shown in FIG. In FIG. 12, the CPU 161 includes one or a plurality of computers (processors), and executes various processes of the information processing apparatus 100 by executing a program stored in the ROM 162 or the RAM 163. The ROM 162 is a read-only non-volatile memory, and the RAM 163 is a volatile memory that can be written at any time. The RAM 163 also functions as a working memory of the CPU 161. The secondary storage device 164 is composed of, for example, a hard disk or the like. The program 171 and the pattern image 172 and the calibration data 173 are stored in the secondary storage device 164. The program 171 includes, for example, a program for causing the CPU 161 to execute a process described later according to the flowchart of FIG. 7. The pattern image 172 includes, for example, a pattern image for calculating the window shape and a pattern image for searching for a corresponding point. The calibration data is data for the projection device 1 and the image pickup device 2 to match each other's coordinate systems. The display unit 165 performs various displays under the control of the CPU 161. The keyboard 166 and the pointing device 167 are instruction input units for receiving instructions from the user. The interface 168 connects the projection device 1 and the imaging measure 2 with the information processing device 100. Each of the above configurations is connected to the bus 169. Each functional unit shown in FIG. 1 may be realized by the CPU 161 executing a program, or may be realized by dedicated hardware.

[Measurement processing]
The measurement process by the information processing apparatus 100 of the first embodiment will be described with reference to the flowchart of FIG. 7.

When the three-dimensional measuring device 1000 is started, the initialization process is executed in step S11. In the initialization process, the projection device 1 and the image pickup device 2 are activated, the calibration data of the projection device 1 and the image pickup device 2 are read into the working memory, and the window shape calculation pattern image and the corresponding point search pattern image are stored in the work memory. The process of reading is included.

In step S12, the projection control unit 101 projects the window shape calculation pattern image on the projection device 1 and sends a control signal to the image input unit 102. As shown in FIG. 2A, the window shape calculation pattern image used in the present embodiment is a pattern in which squares are regularly arranged. In FIG. 2A, the white portion 11 in the pattern image indicates that light is projected, and the black portion 12 indicates that light is not projected. In response to the reception of the control signal from the projection control unit 101, the image input unit 102 causes the image pickup device 2 to perform imaging and inputs the captured image. The captured image input in this way is an image captured in a state where the pattern image for calculating the window shape is projected. The image input unit 102 outputs the captured image input from the image pickup device 2 to the distortion calculation unit 103.

In step S13, the projection control unit 101 projects the pattern image for searching for the corresponding point on the projection device 1 and sends a control signal to the image input unit 102. The pattern image for searching for corresponding points used in the present embodiment is, for example, an aperiodic image in which dots are randomly arranged, as shown in FIG. 2 (b). It is desirable that the pattern image for searching for corresponding points is an image in which two or more local regions having the same characteristics do not appear in one search direction. In response to the reception of the control signal from the projection control unit 101, the image input unit 102 causes the image pickup device 2 to capture an image. In this way, the image input unit 102 inputs the captured image captured in the state where the corresponding point search pattern image is projected on the target object from the image pickup device 2, and outputs the captured image to the corresponding search unit 104.

In step S14, the distortion calculation unit 103 detects a region corresponding to a square in the pattern image for window shape calculation from the captured image input from the image input unit 102, and how the detected region is mapped as compared with the square. Parameterize whether it is done (distortion). Distortion means that the local region (square region in this example) in the pattern image for calculating the window shape is distorted into a parallelogram or the like in the captured image. For example, when the window shape calculation pattern image shown in FIG. 2 (a) is projected onto a sphere by the projection device 1 and imaged by the image pickup device 2, an captured image as shown in FIG. 3 (b) is obtained. When this captured image is magnified, it can be seen that the pattern image is distorted as shown in FIG. 3A.

In order to parameterize the geometric distortion of the region corresponding to the local region of the window shape calculation pattern image in the captured image, the distortion calculation unit 103 first starts with the captured image (FIG. 4A). Detects white lines in the window shape calculation pattern image. For example, the distortion calculation unit 103 determines the binary value of whether or not the pixels are on the line (candidate pixels) according to the brightness of the captured image, and sets a set of candidate pixels that are substantially continuous in the vertical and horizontal directions as vertical lines and horizontal lines. Detect (line 21 in FIG. 4B). Then, the strain calculation unit 103 sets a region surrounded by two adjacent vertical lines and two horizontal lines as a region corresponding to a square region in the pattern image for window shape calculation (region 23 in FIG. 4C). ). For example, the strain calculation unit 103 obtains a point included in both the vertical line and the horizontal line as the intersection of the lines (intersection 22 in FIG. 4B), and the intersections adjacent to each other in the vertical and horizontal directions are the same in the vertical direction. Determine if it is on a line or horizontal line. The strain calculation unit 103 detects a region surrounded by intersections determined to have adjacent intersections on the same vertical line and horizontal line as a region corresponding to a square in the window shape calculation pattern image.

Next, the distortion calculation unit 103 parameterizes the distortion of all the regions in the captured image corresponding to the squares in the window shape calculation pattern image. There are various methods for parameterizing, and any expression may be used. For example, there is a method of parameterizing as a two-dimensional linear transformation. Assuming that the coordinates before the conversion are (u, v) and the coordinates after the conversion are (x, y), the two-dimensional linear conversion is expressed by the following equation 1.

In Equation 1, f is a parameter representing a mapping, and a, b, c, and d are parameters indicating a linear transformation representing f. These parameters are obtained from the correspondence between the square in the pattern image for window shape calculation and the area detected from the captured image. If one point ((0,0) in FIG. 5) is used as a reference, the correspondence between points or line segments ((0,1) and (x1, y1) in FIG. 5), (1,1) and (x2, y2), (1,0) and (x3, y3)) are obtained. The parameters may be obtained analytically from this correspondence, or the optimum parameters may be obtained by a method such as the least squares method. It is assumed that the pattern image of the pixels in the detected area is distorted by the parameters obtained in this way.

By performing a series of processes in all regions in the captured image, the mapping f is determined for each pixel of the captured image. A region in the captured image corresponding to the square in the pattern image for window shape calculation may not be detected or may not be detected. In such a region, the mapping f may be used as an identity matrix, or a mapping in neighboring pixels may occur. Interpolation may be performed by taking a linear sum of f or the like. The distortion calculation unit 103 outputs the obtained distortion parameter (f for each pixel) to the corresponding search unit 104.

In step S15, the correspondence search unit 104 generates a window based on the distortion parameter input from the distortion calculation unit 103, and the correspondence point between the corresponding point search pattern image and the captured image input from the image input unit 102. Perform a search. In the corresponding point search, the cost based on the brightness difference between the corresponding point search pattern image and the captured image in the pixel of interest and its neighboring pixels (for example, 8 pixels around the pixel of interest) is satisfied (minimum, minimum). Etc.) Search for points in the epipolar direction and find the corresponding points. Generally, the cost is calculated with the square windows as the areas of interest, but in the present embodiment, the cost is calculated in consideration of the distortion f of the pattern image for the pixel on which the pattern image is projected, and the corresponding point is searched for. To do. The brightness of the pattern image for searching for corresponding points corresponds to I, the brightness of the captured image corresponds to J, the pixel of interest corresponds to p, the nearby pixel of the pixel of interest p corresponds to q, the set of the pixel of interest p and the neighboring pixel q corresponds to Wp, and the pixel of interest p. Let r be the pixel and fr be the mapping on the corresponding pixel r. In the captured image including the projection of the pattern image for searching for the corresponding point, the corresponding area is arranged around the corresponding pixel r by converting the positions of the surrounding neighboring pixels based on the mapping fr calculated for the corresponding pixel r. Area. The cost function C for calculating the cost between the corresponding area and the area of interest is set as follows, for example.

FIG. 6 shows an example of a corresponding point search in consideration of how the pattern image is distorted. The white circle 31 in the corresponding point search pattern image shown in FIG. 6A is the attention pixel p, the white frame 32 is the attention area (window) in which the attention pixel p and its neighboring pixels are formed, and the white line 33 is the epipolar line. .. The white circle 34 in the captured image shown in FIG. 6B is the corresponding pixel r. The white frame 35 is a corresponding region (window) in consideration of the distortion indicated by the mapping (fr) in the corresponding pixel r, and is formed by the corresponding pixel r and its neighboring pixels. In the corresponding point search, the pixel of interest p (white circle 31) is moved along the epipolar line (white line 33), and the region of interest and the corresponding region are compared at each position of the pixel of interest p. For the region of interest and the corresponding region, the pixel p of interest whose cost calculated by the cost function of Equation 2 satisfies a predetermined condition (minimum, minimum, etc.) is obtained as a corresponding point. By generating a window that takes distortion into consideration on the captured image, the reference pixel may take a sub-pixel value. In such a case, if the brightness is calculated by bilinear interpolation or the like, the cost can be calculated. Good. By performing this process on all the pixels on the captured image, corresponding points (pairs of p and r) can be obtained for the number of pixels. The correspondence between all the obtained projected coordinates and the imaging coordinates (pairs of p and r for the number of pixels) is output to the coordinate calculation unit 105.

In step S16, the coordinate calculation unit 105 performs triangulation based on the correspondence between the projected coordinates and the imaging coordinates input from the corresponding search unit 104 and the calibration data of the projection device 1 and the imaging device 2 obtained in advance. In this way, the three-dimensional coordinates ((X, Y, Z) coordinates of the corresponding points) of the surface of the target object are calculated, and the process is completed.

The order of the steps does not necessarily have to be the order shown in the flowchart of FIG. 7, and may be appropriately replaced or parallel processing may be performed.

As described above, according to the first embodiment, the windows are generated in consideration of how the projection pattern image is distorted in the captured image, and the corresponding point search is performed to obtain the three-dimensional coordinates with high accuracy. Can be calculated in.

[Modification 1]
In the above embodiment, the window shape calculation pattern image is projected in step S12, the corresponding point search pattern image is projected in Tep S13, and the captured images including the projection of these pattern images are separately acquired. It is not limited. For example, as shown in FIG. 8, a pattern image (FIG. 8 (c)) in which a window shape calculation pattern image (FIG. 8 (a)) and a corresponding point search pattern image (FIG. 8 (b)) are mixed is projected. , This may be imaged and used for acquiring a parameter indicating distortion and searching for a corresponding point. Alternatively, by changing the wavelength of the light for projecting each pattern image and using an imaging device corresponding to those wavelengths, these pattern images may be projected at the same time. For example, a color image is captured with the window shape calculation pattern projected by the light of the first wavelength (for example, red) and the corresponding point search pattern projected by the light of the second wavelength (for example, green). Do. A parameter indicating distortion is calculated by detecting a red line from the obtained captured image, and a green point is detected and used as a search target. As described above, according to the configuration of the modified example 1, since the measurement can be performed by one projection and the imaging, the three-dimensional coordinates can be calculated with high accuracy even for a moving target object.

[Modification 2]
FIG. 2 is shown as an example of the pattern image projected in step S12 and step S13, but the pattern image is not limited to these. In particular, in the pattern image for calculating the window shape, it is not necessary for the known geometric shapes (squares) to be regularly arranged as shown in FIG. 2, and it is sufficient that at least one known geometric shape is present. For example, as shown in FIG. 9, a pattern image in which equilateral triangles 41 and squares 42 are arranged side by side as different types of known geometric shapes may be used. However, since the distortion parameter is obtained for each pixel, it is desirable that the known shapes are evenly present in the pattern image for calculating the window shape.

In addition, in order to obtain the distortion parameter, it is necessary to detect a known geometric shape (hereinafter, both known shapes) from the captured image. For example, to detect a known shape,
-A method of applying an edge detection filter such as the Sobel filter to the captured image to detect the hand edge, and detecting the connected edge that goes around as a known shape.
-A method of detecting a known shape by matching using an image feature amount such as LBP (Local Binary Pattern) or SIFT (Scale-Invariant Feature Transform) using a known shape as a template can be mentioned.

By allowing a plurality of different types of known shapes to exist in the divided region of the window shape calculation pattern image for each type, the corresponding search unit 104 can use the distortion parameter of each pixel as well as approximately which part of the pattern image each pixel is. Can be grasped whether it corresponds to. For example, in the case of FIG. 9, the equilateral triangle 41 in the captured image corresponds to the upper part of the pattern image, and the square 42 corresponds to the lower part of the pattern image. Therefore, the corresponding search unit 104 can grasp that the pixels in the equilateral triangle 41 are present in the upper part of the captured image, and the pixels in the square 42 are present in the lower part of the captured image. As a result, the corresponding search unit 104 can limit the search range shown in FIG. 6A to the upper half of the epipolar line, for example, and can calculate the three-dimensional coordinates at high speed and with high accuracy. it can.

[Modification 3]
In step S12 and step S13, the pattern image to be projected does not have to be limited to binary, and may be a pattern image having gradation and color tone. For example, as a window shape calculation pattern, an image in which known geometric shapes having different gradations and color tones are arranged can be used. A plurality of known geometric shapes having different gradations and color tones may be detected as features such as brightness and hue in the captured image. By doing so, it is possible to limit the search range of the corresponding search unit 104 because it is possible to know which part of the pattern image corresponds to along with the distortion parameter of each pixel, as in the modified example 2. Therefore, the three-dimensional coordinates can be calculated at high speed and with high accuracy.

[Modification example 4]
In step S15, the strain parameter obtained in step S14 may be used as the initial value, and the corresponding point and the distortion of the window may be estimated at the same time as in Non-Patent Document 1 (the parameter corresponding to fr in Equation 2 is also estimated at the same time. To do). That is, the correspondence search unit 104 updates the mapping fr in response to the update of the corresponding pixel r, with the mapping calculated by the distortion calculation unit 103 as the initial value. At this time, as in the first embodiment, the mapping may be searched by the entire search in the area of the corresponding point search pattern image corresponding to the local area of the window calculation pattern image, or Graph Cut or Belief. The mapping may be searched using an algorithm such as Propagation or PatchMatch. By doing so, the distortion parameter that is correct to some extent can be set as the initial value, so that the search cost of the map can be suppressed. Therefore, the three-dimensional coordinates can be calculated at high speed and with high accuracy.

<Second embodiment>
In the first embodiment, the projection device and the image pickup device are configured one by one, but in the second embodiment, the pattern light is projected on the target object and the reflected light is observed by the two image pickup devices. The configuration for measuring the three-dimensional shape of the target object will be described in. The information processing apparatus of the second embodiment generates a window considering the distortion calculated from the pattern image for calculating the window shape, and searches for a corresponding point between the captured images obtained from the two imaging devices. Calculate accurate 3D coordinates. By using two image pickup devices, it is possible to suppress a decrease in accuracy due to optical influences such as the projected pattern image being blurred and the pattern image being distorted due to mutual reflection or the like.

[Device configuration]
FIG. 10 is a block diagram showing a configuration example of a three-dimensional measuring device 1000 including the information processing device 100 of the second embodiment. The first embodiment is different from the first embodiment in that one imaging device is added and the first imaging device 2 and the second imaging device 3 are provided. The first imaging device 2 and the second imaging device 3 perform imaging from different positions in a state where the pattern image is projected by the projection device 1. The projection device 1, the first image pickup device 2, and the second image pickup device 3 have been calibrated in advance to match their coordinate systems, and the calibration data is stored in the information processing device 100. .. The hardware configuration of the information processing device 100 is the same as that of the first embodiment (FIG. 12).

The information processing device 100 includes a projection control unit 101, an image input unit 102, a distortion calculation unit 103, a correspondence search unit 104, and a coordinate calculation unit 105. The projection control unit 101 controls the projection device 1 to project a pattern image (hereinafter referred to as a mixed pattern image) in which a pattern image for window shape calculation and a pattern image for searching corresponding points are mixed, and is controlled by the image input unit 102. Output a signal. An example of the mixed pattern image is shown in FIG. 8 (c). As in the first embodiment, the window shape calculation pattern image (FIG. 8 (a)) and the corresponding point search pattern image (FIG. 8 (b)) are sequentially projected and imaged individually. May be good. The image input unit 102 controls the first image pickup device 2 and the second image pickup device 3 in accordance with the timing of the control signal input from the projection control unit 101 to execute the image pickup, and mix patterns from each of them. Receive the captured image on which the image is projected. The image input unit 102 outputs the two captured images on which the mixed pattern image is projected to the distortion calculation unit 103.

The distortion calculation unit 103 determines how each pixel in each captured image is based on the window shape calculation pattern image included in the mixed pattern image and the two captured images input from the image input unit 102. Calculate the distortion parameter that indicates whether the image is mapped. The distortion calculation unit 103 outputs the calculated distortion parameters of the two captured images to the corresponding search unit 104. The correspondence search unit 104 generates a window considering the distortion based on the distortion parameter input from the distortion calculation unit 103, and performs a correspondence point search between the two captured images input from the image input unit 102. .. The correspondence search unit 104 outputs to the coordinate calculation unit 105 the relationship between the imaging coordinates of the first imaging device 2 and the imaging coordinates of the second imaging device 3 obtained by the corresponding point search. The coordinate calculation unit 105 calculates the three-dimensional coordinates from the correspondence relationship of the image pickup coordinates input from the correspondence search unit 104 and the calibration data of the first image pickup device 2 and the second image pickup device 3 obtained in advance.

[Measurement processing]
The measurement process by the information processing apparatus 100 of the second embodiment will be described with reference to the flowchart of FIG.

When the three-dimensional measuring device 2000 is started, the initialization process is executed in step S21. In the initialization process, the projection device 1, the activation process of the first image pickup device 2 and the second image pickup device 3, the calibration data of the projection device 1, the first image pickup device 2, and the second image pickup device 3 are used. It includes a process of reading into the working memory, a process of reading a mixed pattern image, and the like.

In step S22, the projection control unit 101 projects the mixed pattern image on the projection device 1 and sends a control signal to the image input unit 102. The image input unit 102 that has received the control signal causes the first image pickup device 2 and the second image pickup device 3 to take an image in which the mixed pattern image is projected on the target object. The image input unit 102 outputs the two input captured images to the distortion calculation unit 103.

In step S23, the distortion calculation unit 103 determines how the square region in the window shape calculation pattern image is mapped to the captured image for each of the two captured images input from the image input unit 102. Calculate the indicated distortion parameters. The process of obtaining the strain parameter is the same as that of the first embodiment (S14). Each of the obtained distortion parameters (f for each pixel) and two captured images are output to the corresponding search unit 104.

In step S24, the correspondence search unit 104 creates a window in each of the two captured images based on the distortion parameter input from the distortion calculation unit 103, and searches for a corresponding point between the two captured images. .. Similar to the first embodiment, the cost is calculated in consideration of the distortion f of the pattern image for the pixel on which the pattern image for searching the corresponding point is projected, and the corresponding point is searched. The brightness of the image captured by the first imaging device 2 is I, the brightness of the image captured by the second imaging device 3 is J, the pixel of interest is p, the mapping of the pixel of interest p is fp, and the pixels in the vicinity of the pixel of interest p are q ( For example, 8 pixels around the pixel of interest p), and the set of the pixel of interest p and the neighboring pixel q is Wp. Further, the pixel corresponding to the pixel of interest p is r, and the mapping on the corresponding pixel r is fr. For example, the cost function C is calculated by the following equation 3.

The corresponding point search process using the cost function is the same as that of the first embodiment. That is, the correspondence search unit 104 moves the attention pixel along the epipolar line, and obtains the attention pixel p (minimum, minimum, etc.) in which the cost function of Equation 3 satisfies a predetermined condition (minimum, minimum, etc.). In this way, when the corresponding points are searched for the two captured images, even if the pattern images are distorted due to blurring or mutual reflection, the captured images look the same, so the corresponding point search is preferably performed. be able to. The correspondence search unit 104 outputs the correspondence relationship (pairs of p and r for the number of pixels) between all the obtained imaging coordinates to the coordinate calculation unit 105.

In step S25, the coordinate calculation unit 105 triangulates from the correspondence relationship between the imaging coordinates input from the correspondence search unit 104 and the pre-obtained calibration data of the first imaging device 2 and the second imaging device 3. Three-dimensional coordinates ((X, Y, Z) coordinates for the number of corresponding points) are calculated by surveying. Then, this process is terminated.

As described above, according to the second embodiment, a window is generated in consideration of how the projection pattern image is distorted in each captured image, and a corresponding point search is performed. Further, according to the second embodiment, by using two imaging devices, it is possible to suppress a decrease in accuracy due to optical influences such as blurring and mutual reflection, so that the three-dimensional coordinates are calculated with high accuracy. be able to.

[Modification 5]
In step S24, not only the correspondence between the first image pickup device 2 and the second image pickup device 3, but also the projection device and the first image pickup device 2, and the projection device and the second image pickup as in the first embodiment. The correspondence relationship of the device 3 may be obtained. By doing so, it is possible to reduce the portion where the three-dimensional coordinates cannot be measured due to shielding or the like.

[Modification 6]
In the second embodiment, the number of projection devices is one and the number of image pickup devices is two, but the number of projection devices and image pickup devices is not limited to this, and the number of projection devices and image pickup devices is not limited to this. Good. In step S23, the distortion may be obtained by determining a pair of the projection device and the image pickup device. Further, in step S24, the cost of each pair may be calculated to obtain the correspondence relationship, or all the costs may be integrated to obtain the correspondence relationship as in Multi-view Stereo. By doing so, it is possible to reduce the portion that cannot be measured due to shielding or the like, and to measure the entire target object more reliably.

<Effect of embodiment>
According to the first embodiment, it is possible to calculate the three-dimensional coordinates with high accuracy by generating a window in consideration of how the projection pattern image is distorted in the captured image and performing a corresponding point search. it can.
According to the second embodiment, in addition to the effect of the first embodiment, by using two cameras, it is possible to suppress a decrease in accuracy due to optical influences such as blurring and mutual reflection, thereby making it three-dimensional. The coordinates can be calculated with high accuracy.

<Definition>
As for the pattern image possessed by the projection control unit 101, as described in the first embodiment and the modified examples 1, 2 and 3, what kind of pattern image can be obtained if a window shape considering distortion can be generated and a corresponding point search can be performed. It may be a pattern image.
Further, as described in the first embodiment and the modification 4, the correspondence search unit 104 may use any algorithm as long as the correspondence point can be obtained.

Further, the projection device 1 and the image pickup device 2 do not have to be limited to one as described in the second embodiment and the modification 6, and may be a plurality of each.

(Other Examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It is also possible to realize the processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

1000: 3D measuring device, 1: Projection device, 2: Imaging device, 100: Information processing device, 101: Projection control unit, 102: Image input unit, 103: Distortion calculation unit, 104: Correspondence search unit, 105: Coordinates Calculation unit

Claims (13)

In the captured image in which the first pattern image is captured a state of being projected to have at least one known geometry, the area corresponding to the geometry of the first pattern image, geometry from the geometry A calculation means for calculating a mapping representing a geometrical distortion, and
A region of interest consists of neighboring pixels surrounding the target pixel in the second pattern image by the second pattern image is converted based on the region in the captured image obtained by capturing an image of a state of being projected onto the mapping A search means for searching for a corresponding point between the second pattern image and the captured image by comparison with the obtained corresponding area is provided .
The corresponding region is arranged around the corresponding pixels in the captured image obtained by capturing the projected state of the second pattern image, with the positions of the surrounding neighboring pixels converted for the corresponding pixels based on the mapping. the information processing apparatus according to claim region der Rukoto was. The information processing apparatus according to claim 1, wherein the search means searches the region of interest along an epipolar line corresponding to a point in the corresponding region. The information processing apparatus according to claim 1 , wherein the search means updates the map according to the update of the corresponding pixel, with the map calculated by the calculation means as an initial value. A captured image captured a state where the first pattern image is projected, the captured image and the second pattern image is captured a state of being projected, further comprising an acquisition means for acquiring as a separate captured images The information processing apparatus according to any one of claims 1 to 3, which is characterized. Any of claims 1 to 4 , further comprising an acquisition means for acquiring an image captured by capturing a state in which a pattern image obtained by mixing the first pattern image and the second pattern image is projected. The information processing apparatus according to item 1. The information processing apparatus according to any one of claims 1 to 5 , wherein the second pattern image has an aperiodic pattern. The information processing apparatus according to any one of claims 1 to 6 , wherein the first pattern image is an image in which known geometric shapes are arranged, which have different gradations or color tones. The information processing apparatus according to any one of claims 1 to 7 , wherein the first pattern image is a pattern image in which the geometric shapes are regularly arranged. The information according to any one of claims 1 to 8 , further comprising a coordinate calculation means for calculating three-dimensional coordinates of the surface of the target object based on the corresponding point searched by the search means. Processing equipment. The information processing apparatus according to any one of claims 1 to 9 , further comprising a projection means. The information processing apparatus according to any one of claims 1 to 10 , further comprising an imaging means. It is a control method for information processing equipment.
In the captured image in which the first pattern image is captured a state of being projected to have at least one known geometry, the area corresponding to the geometry of the first pattern image, geometry from the geometry The process of calculating a mapping that represents a geometrical distortion,
A region of interest consists of neighboring pixels surrounding the target pixel in the second pattern image by the second pattern image is converted based on the region in the captured image obtained by capturing an image of a state of being projected onto the mapping the corresponding area obtained by the comparison, have a, a search step of searching corresponding points between the captured image and the second pattern image,
The corresponding region is arranged around the corresponding pixels in the captured image obtained by capturing the projected state of the second pattern image, with the positions of the surrounding neighboring pixels converted for the corresponding pixels based on the mapping. a method of controlling an information processing apparatus characterized by region der Rukoto was. A program for causing a computer to execute each step of the control method of the information processing apparatus according to claim 12.

Images