JP2018063161A - Information processing apparatus, method for controlling information processing apparatus, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To measure a three-dimensional image in an accurate and safe manner irrespective of the texture of a target object.SOLUTION: An information processing apparatus comprises: an image acquisition part that acquires a first image obtained by picking up an image of a target object in which a pattern image is projected, and a plurality of second images obtained by picking up images of the target object from a plurality of view points; a first similarity acquisition part that acquires first similarity on the basis of comparison between a local area of the first image and a local area of the pattern image; a second similarity acquisition part that acquires second similarity on the basis of comparison between local areas of the plurality of second images; and a determination part that determines three-dimensional coordinates of the surface of the target object on the basis of the first similarity and second similarity.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing apparatus, a control method for the information processing apparatus, and a program.

  Methods for acquiring the three-dimensional shape of the target object are roughly classified into an active method and a passive method. The active method is a method of performing three-dimensional measurement by projecting pattern light onto a target object from a projector and observing the reflected light with a camera. On the other hand, the passive method is a method of performing three-dimensional measurement by imaging a target object with a camera from a plurality of viewpoints. In these measurement methods, a similarity between local regions of an image is calculated, and a three-dimensional shape is obtained by searching for a three-dimensional coordinate that maximizes the similarity. However, both the active method and the passive method have a problem that the measurement accuracy deteriorates depending on the texture of the target object. In the active method, correct three-dimensional coordinates can be calculated when there is no texture of the target object, but it is difficult to calculate correct three-dimensional coordinates when the texture of the target object is complicated. On the other hand, in the passive method, correct three-dimensional coordinates can be calculated when the texture of the target object is complex, but it is difficult to calculate correct three-dimensional coordinates when there is no texture of the target object or when it is periodic.

  On the other hand, Patent Document 1 discloses a technique that is combined with not only a passive three-dimensional measurement technique but also another three-dimensional measurement technique such as an active method.

JP2015-49200A

  However, the technique described in Patent Literature 1 has a problem in that the measurement accuracy is not constant depending on the texture of the target object because the measurement accuracy differs between the passive three-dimensional measurement method and other three-dimensional measurement methods. .

  The present invention has been made in view of the above problems, and an object of the present invention is to provide a technique for stably measuring a three-dimensional shape with high accuracy regardless of the texture of a target object.

An information processing apparatus according to the present invention that achieves the above object is as follows.
Image acquisition means for acquiring a first image obtained by imaging a target object on which a pattern image is projected, and a plurality of second images obtained by imaging the target object from a plurality of viewpoints;
First similarity acquisition means for acquiring a first similarity based on a comparison between the local region of the first image and the local region of the pattern image;
Second similarity acquisition means for acquiring a second similarity based on a comparison between local regions in each of the plurality of second images;
Determining means for determining a three-dimensional coordinate of the surface of the target object based on the first similarity and the second similarity;
It is characterized by providing.

  According to the present invention, a three-dimensional shape can be measured stably and with high accuracy regardless of the texture of the target object.

1 is a block diagram illustrating a configuration example of an information processing apparatus according to a first embodiment. 5 is a flowchart for explaining measurement processing by the information processing apparatus according to the first embodiment. The figure explaining the process of the 1st similarity acquisition part of 1st Embodiment. The figure explaining the process of the 2nd similarity acquisition part of 1st Embodiment. The block diagram which shows the structural example of the information processing apparatus of 2nd Embodiment. 9 is a flowchart for explaining measurement processing by the information processing apparatus according to the second embodiment. The figure explaining the process of the reliability acquisition part of 2nd Embodiment. The block diagram which shows the structural example of the information processing apparatus of 3rd Embodiment. 10 is a flowchart for explaining measurement processing by the information processing apparatus according to the third embodiment.

  Hereinafter, embodiments will be described with reference to the drawings. The configurations shown in the following embodiments are merely examples, and the present invention is not limited to the illustrated configurations.

(First embodiment)
The information processing apparatus described in the first embodiment is a three-dimensional measurement apparatus that acquires a three-dimensional shape of a target object using a combination of active and passive three-dimensional measurement techniques. A three-dimensional point is determined based on a similarity obtained by integrating a similarity obtained by an active method using a random dot pattern and a similarity obtained by a binocular stereo passive method. When there is no texture of the target object, a correct three-dimensional coordinate is obtained by the similarity of the active method, and when the texture of the target object is complicated, the correct three-dimensional coordinate is obtained by the similarity of the passive method. As a result, stable and highly accurate three-dimensional coordinates can be calculated regardless of the texture of the target object.

[Device configuration]
With reference to the block diagram of FIG. 1, an example of a three-dimensional measurement apparatus 1000 including the information processing apparatus 1100 according to the first embodiment is shown. The near-infrared projection device 1 projects a random dot pattern onto a measurement target object using a near-infrared light source. The pattern light is reflected by the surface of the measurement target object and is imaged by the near-infrared imaging device 2. At the same time, the target object is imaged by the visible light imaging device 3 and the visible light imaging device 4. Images captured by the near-infrared imaging device 2, the visible light imaging device 3, and the visible light imaging device 4 are sent to the information processing device 1100, and the information processing device 1100 calculates the three-dimensional coordinates of the target object. Further, the information processing apparatus 1100 controls the operations of the near-infrared imaging device 2, the visible light imaging device 3, and the visible light imaging device 4. Further, the near-infrared projection device 1, the near-infrared imaging device 2, the visible light imaging device 3, and the visible light imaging device 4 are preliminarily calibrated so as to know their positional relationship, and the calibration data is external. Stored in memory. In addition, in order to make the accuracy of the three-dimensional point calculated | required from each of an active system and a passive system correspond, the base line length between the near-infrared projection apparatus 1 and the near-infrared imaging device 2, the visible light imaging device 3, and the visible light imaging device The lengths of the baseline lengths between the four (between imaging devices) are substantially the same. Further, the resolution and focal length of the near-infrared imaging device 2, the visible light imaging device 3, and the visible light imaging device 4 are substantially the same.

  The information processing apparatus 1100 includes a projection pattern information holding unit 101, an image acquisition unit 102, a first similarity acquisition unit 103, a second similarity acquisition unit 104, and a three-dimensional coordinate determination unit 105. The projection pattern information holding unit 101 holds a luminance image of a random dot pattern to be projected.

  The image acquisition unit 102 controls the near-infrared imaging device 2, the visible light imaging device 3, and the visible light imaging device 4 to capture images, and acquires each captured image. Then, the image captured by the near-infrared imaging device 2 (first image: image for the active method) is output to the first similarity acquisition unit 103. Further, the image picked up by the visible light imaging device 3 and the visible light imaging device 4 (second image: image for passive method) is output to the second similarity acquisition unit 104.

  The first similarity acquisition unit 103 calculates and acquires a first similarity that indicates how similar the luminance distribution in the local region between the first image and the random dot pattern image is. Then, the calculated first similarity is output to the three-dimensional coordinate determination unit 105.

  The second similarity acquisition unit 104 calculates and acquires a second similarity that indicates how similar the luminance distribution in the local region between two second images captured from a plurality of viewpoints is. Then, the calculated second similarity is output to the three-dimensional coordinate determination unit 105.

  The three-dimensional coordinate determination unit 105 obtains the similarity considering both the active method and the passive method by integrating the first similarity and the second similarity, and the measurement target based on the integrated similarity Determine the three-dimensional coordinates of the object.

[Measurement processing]
Next, the procedure of the measurement process performed by the information processing apparatus 1100 according to the first embodiment will be described with reference to the flowchart of FIG.

(Step S11)
In step S11, when the three-dimensional measurement apparatus 1000 is activated, an initialization process is executed. In the initialization process, the near-infrared projection device 1, the near-infrared imaging device 2, the visible light imaging device 3, and the visible light imaging device 4 are activated, and the near-infrared projection device 1 starts from the projection pattern information holding unit 101. A process of projecting the read random dot pattern image is included. Also included is processing for reading calibration data from an external memory.

(Step S12)
In step S12, the image acquisition unit 102 controls the near-infrared imaging device 2, the visible light imaging device 3, and the visible light imaging device 4 to capture an image. At this time, since the imaging wavelength is divided into near-infrared light and visible light, the target object onto which the random dot pattern is projected is copied to the near-infrared imaging device 2, and the visible-light imaging device 3, and The visible light imaging device 4 captures a target object illuminated only by ambient light. The image acquisition unit 102 outputs the first image captured by the near-infrared imaging device 2 to the first similarity acquisition unit 103, and the first image captured by the visible light imaging device 3 and the visible light imaging device 4. The second image is output to the second similarity acquisition unit 104.

(Step S13)
In step S <b> 13, the first similarity acquisition unit 103 obtains the similarity between the local region of the first image input from the image acquisition unit 102 and the local region of the random dot pattern image. Hereinafter, the process for obtaining the similarity will be described with reference to FIG. In FIG. 3, reference numeral 11 denotes a first image, p1 denotes a pixel of interest in the first image, v1 denotes a viewpoint of the near-infrared imaging device 2, and e1 denotes a light ray direction passing through p1. Q0, q1, and q2 are three-dimensional points on e1, 12 is a random dot pattern image, and p2 is a pixel of interest in the random dot pattern image.

  First, a random dot pattern image necessary for calculating the similarity is read from the projection pattern information holding unit 101. Next, a luminance difference between the first image 11 and the random dot pattern image 12 in a local region (pixels of interest (p1 and p2) and pixels in the vicinity thereof (for example, 8 pixels around the pixel of interest)) The first similarity is calculated by obtaining the reciprocal of the absolute value (SAD: Sum of Absolute Difference). When calculating the similarity, a three-dimensional point (q: q0, q1, q2,...) Is taken along the light ray direction e1 passing through p1 obtained from the calibration data with reference to the viewpoint v1 of the near-infrared imaging device 2. Move to calculate the similarity at each q. Since each three-dimensional point q is projected on the epipolar line on the random dot pattern image 12, the target pixel p2 corresponding to the three-dimensional point q is known from the calibration data, and the target pixel p1 and the target pixel p2 and their vicinity The reciprocal of SAD with the pixels in the area is determined.

  By performing such processing for all the target pixels p1, the first similarity is obtained for each pixel of the first image. The obtained first similarity is output to the three-dimensional coordinate determination unit 105.

(Step S14)
In step S <b> 14, the second similarity acquisition unit 104 obtains the similarity between the local regions of the two second images input from the image acquisition unit 102. Hereinafter, the process for obtaining the similarity will be described with reference to FIG. 4 and 3 that are the same as those in FIG. In FIG. 4, 13 indicates an image captured by the visible light imaging device 3, and 14 indicates an image captured by the visible light imaging device 4. Further, p3 represents a pixel of interest in an image captured by the visible light imaging device 3, and p4 represents a pixel of interest in an image captured by the visible light imaging device 4, respectively.

  Next, the similarity in the pixel of interest (p3 and p4) between the image 13 captured by the visible light imaging device 3 and the image 14 captured by the visible light imaging device 4 is calculated in the same manner as in step S14. . Since each three-dimensional point q is projected on the epipolar line on each image 13, 14, the target pixels p3 and p4 corresponding to the three-dimensional point q are known from the calibration data, and the target pixel p3 and the target pixel in the target pixel p1 are found. The reciprocal of SAD between p4 and the pixels in the vicinity thereof is obtained.

  By performing such processing for all the target pixels p1, the second similarity is obtained for each pixel of the first image. The obtained second similarity is output to the three-dimensional coordinate determination unit 105.

(Step S15)
In step S <b> 15, the 3D coordinate determination unit 105 uses the first similarity input from the first similarity acquisition unit 103 and the second similarity input from the second similarity acquisition unit 104 to perform tertiary processing. Determine the original coordinates.

  First, in each pixel of the first image, the first similarity and the second similarity are integrated by the following formula.

  Where p is a pixel in the first image, q is a three-dimensional point, C (p, q) is the similarity after integration, A (p, q) is the first similarity, B (p, q ) Represents the second similarity. Then, the position where the integrated similarity C (p, q) obtained in this way is maximized is determined as the correct three-dimensional point q. By performing such processing at each pixel of the first image, three-dimensional coordinates (X, Y, Z) corresponding to the number of pixels are calculated, and the processing is terminated.

  With the configuration described above, the 3D point is determined based on the similarity obtained by integrating the similarities obtained from the active method and the passive method, so that the 3D coordinates are stable and high regardless of the texture of the target object. It can be calculated with accuracy.

(Second Embodiment)
As in the first embodiment, the information processing apparatus described in the second embodiment is a three-dimensional measurement apparatus that acquires the three-dimensional shape of a target object using a combination of active and passive three-dimensional measurement techniques. is there. However, the second embodiment is different in that the second embodiment further includes a reliability acquisition unit that calculates the reliability of the similarity based on each similarity. By determining the three-dimensional point with a higher priority on the similarity having a higher corresponding reliability among the similarities, it is possible to calculate more stable and highly accurate three-dimensional coordinates.

[Device configuration]
With reference to the block diagram of FIG. 5, an example of a three-dimensional measurement apparatus 2000 including the information processing apparatus 2100 of the second embodiment is shown. In addition, about the apparatus substantially the same as 1st Embodiment, a same sign is attached | subjected and description is abbreviate | omitted.

  The information processing apparatus 2100 includes a projection pattern information holding unit 201, an image acquisition unit 202, a first similarity acquisition unit 203, a second similarity acquisition unit 204, a reliability acquisition unit 205, and a three-dimensional coordinate determination unit 206. Note that the projection pattern information holding unit 201 and the image acquisition unit 202 are substantially the same as those in the first embodiment, and a description thereof will be omitted.

  The first similarity acquisition unit 203 calculates a first similarity indicating how similar the luminance distribution in the local region between the first image and the random dot pattern image is. The calculated first similarity is output to the reliability acquisition unit 205.

  The second similarity acquisition unit 204 calculates a second similarity indicating how similar the luminance distribution in the local region between the two second images is. The calculated second similarity is output to the reliability acquisition unit 205.

  The reliability acquisition unit 205 calculates a first reliability and a second reliability that indicate how reliable each similarity is from the first similarity and the second similarity. The calculated first reliability and second reliability, and the first similarity and second similarity are output to the three-dimensional coordinate determination unit 206.

  The three-dimensional coordinate determination unit 206 integrates the first reliability and the second reliability, and the first similarity and the second similarity, so that both the active method and the passive method are considered. Find the degree and determine the final 3D coordinates.

[Measurement processing]
Next, with reference to a flowchart of FIG. 6, a measurement process procedure performed by the information processing apparatus 2100 according to the second embodiment will be described. In addition, since step S21, step S22, step S23, and step S24 are substantially the same as each corresponding step of 1st Embodiment, description is abbreviate | omitted.

(Step S25)
In step S <b> 25, the reliability acquisition unit 205 calculates the reliability of each similarity input from the first similarity acquisition unit 203 and the second similarity acquisition unit 204. The reliability is defined as the maximum peak degree (kurtosis) taken by the similarity. It is determined that the steeper peak (the higher the kurtosis), the more reliable the three-dimensional coordinate of the peak position, and the gentler the peak (the lower the kurtosis), the more unreliable the three-dimensional coordinate of the peak position. For example, in the case of similarity as shown in FIG. 7, if the maximum value of similarity is s1, and the second largest value is s2, s1 / s2−1 may be used as the reliability.

  By performing such processing with respect to the first similarity and the second similarity, the first reliability and the second reliability are obtained for each pixel of the first image. The first reliability and the second reliability, and the first similarity and the second similarity are output to the three-dimensional coordinate determination unit 206.

(Step S26)
In step S <b> 26, the three-dimensional coordinate determination unit 206 uses the first reliability and the second reliability, and the first similarity and the second similarity input from the reliability acquisition unit 205 to perform tertiary processing. Determine the original coordinates.

  First, in each pixel of the first image, the first similarity and the second similarity are integrated by the following formula.

  Where p is an arbitrary pixel of the first image, q is a three-dimensional point, C (p, q) is the similarity after integration, A (p, q) is the first similarity, B (p, q ) Represents the second similarity, α represents the first reliability, and β represents the second reliability. Then, the position where the integrated similarity C (p, q) obtained in this way is maximized is determined as the correct three-dimensional point q. By performing this process on each pixel of the first image, three-dimensional coordinates (X, Y, Z) corresponding to the number of pixels are calculated, and the process ends.

  With the configuration described above, the three-dimensional point is determined based on the similarity obtained from each of the active method and the passive method, and the reliability obtained from the similarity, so that the texture of the target object can be more stably and accurately obtained. Regardless, the three-dimensional coordinates can be calculated.

[Modification of Second Embodiment]
In step S25, the calculation of the reliability need not be limited to the peak degree (kurtosis) of the similarity. For example, you may set by the distance of the three-dimensional point calculated | required by each of an active system and a passive system independently. If the distance between the peak positions of the similarity obtained from each is within a threshold value (for example, 1), the same reliability is set. If the distance is far from the threshold value (for example, 1), the distance from the image as shown in the third embodiment It is only necessary to determine which one is more reliable depending on the required reliability. Alternatively, the closer the distance, the lower the reliability of the active method and the higher the reliability of the passive method, and the higher the distance, the higher the reliability of the active method and the lower the reliability of the passive method. .

(Third embodiment)
As in the second embodiment, the information processing apparatus described in the third embodiment is a three-dimensional measurement apparatus that acquires a three-dimensional shape of a target object using a combination of active and passive three-dimensional measurement techniques. is there. However, although the reliability acquisition unit of the second embodiment calculates the reliability based on the similarity, the reliability acquisition unit of the third embodiment calculates the reliability based on the captured image. Is different. Since the reliability is determined based on the captured image, it is not necessary to calculate the similarity in the entire search range, and stable and highly accurate three-dimensional coordinates can be calculated at higher speed regardless of the texture of the target object.

[Device configuration]
With reference to the block diagram of FIG. 8, an example of a three-dimensional measurement apparatus 3000 including the information processing apparatus 3100 of the third embodiment is shown. In addition, about the apparatus substantially the same as 2nd Embodiment, the same code | symbol is attached | subjected and description is abbreviate | omitted.

  The information processing apparatus 3100 includes a projection pattern information holding unit 301, an image acquisition unit 302, a first similarity acquisition unit 303, a second similarity acquisition unit 304, a reliability acquisition unit 305, and a three-dimensional coordinate determination unit 306. Note that the projection pattern information holding unit 301 is substantially the same as in the second embodiment, and a description thereof is omitted.

  As in the second embodiment, the image acquisition unit 302 controls the near-infrared imaging device 2, the visible light imaging device 3, and the visible light imaging device 4 to capture images, and acquires each captured image. Then, the image captured by the near-infrared imaging device 2 (first image: image for the active method) is output to the first similarity acquisition unit 303 and the reliability acquisition unit 305. In addition, the image captured by the visible light imaging device 3 and the visible light imaging device 4 (second image: an image for a passive method) is output to the second similarity acquisition unit 304 and the reliability acquisition unit 305. .

  The first similarity acquisition unit 303 calculates and acquires a first similarity indicating how similar the luminance distribution in the local region between the first image and the random dot pattern image is. The calculated first similarity is output to the three-dimensional coordinate determination unit 306.

  The second similarity acquisition unit 304 calculates and acquires a second similarity indicating how similar the luminance distribution in the local region between the two second images is. The calculated second similarity is output to the three-dimensional coordinate determination unit 306.

  The reliability acquisition unit 305 calculates and acquires the first reliability and the second reliability that indicate how reliable the similarity calculated from the first image and the second image is. The calculated first reliability and second reliability are output to the three-dimensional coordinate determination unit 306.

  The three-dimensional coordinate determination unit 306 integrates the first reliability and the second reliability, and the first similarity and the second similarity, so that both the active method and the passive method are considered. Find the degree and search for and determine plausible 3D coordinates.

[Measurement processing]
Subsequently, a measurement process performed by the information processing apparatus 3100 according to the third embodiment will be described with reference to a flowchart of FIG. Note that step S31 and step S32 are substantially the same as the corresponding steps of the second embodiment, and a description thereof will be omitted.

(Step S33)
In step S <b> 33, the reliability acquisition unit 305 calculates the reliability of each similarity acquired from the first image and the second image input from the image acquisition unit 302. The reliability is defined by the height of the spatial frequency in the pixel of interest and the pixels in the vicinity region (for example, 8 pixels around the pixel of interest). The pixel of interest and its neighboring region are Fourier transformed, and the higher the spatial frequency having a certain amplitude or higher, the higher the reliability is set. In the active method, accurate three-dimensional coordinates can be calculated if the projected pattern is clearly blurred and blurred, and in the passive method, accurate three-dimensional coordinates can be calculated if there is a gradient feature. Therefore, it can be determined that a reliable similarity is calculated in the case of both textures including a high spatial frequency.

  By performing such processing with respect to each pixel of the first image and the second image, the first reliability in each pixel of the first image and the second in each pixel of the second image. Reliability is required. The obtained first reliability and second reliability are output to the three-dimensional coordinate determination unit 306.

(Step S34)
In step S34, the three-dimensional coordinate determination unit 306 gives an appropriate three-dimensional coordinate as an initial value for each pixel of the first image. Hereinafter, the process of giving the initial value of the three-dimensional coordinates will be described with reference to FIG. Since the description of each symbol is the same as in the first embodiment, the description is omitted.

  Specifically, at each pixel of the first image, one three-dimensional point is randomly set along the e1 direction obtained from the calibration data. The set random three-dimensional points are output to the first similarity acquisition unit 303 and the second similarity acquisition unit 304.

(Step S35)
In step S35, the first similarity acquisition unit 303 projects the three-dimensional coordinates input from the three-dimensional coordinate determination unit 306 onto the first image and the random dot pattern image input from the image acquisition unit 302, The similarity between local areas in the projected pixel is obtained. Hereinafter, the process for obtaining the similarity will be described with reference to FIG. Since the description of each symbol is the same as in the first embodiment, the description is omitted.

  First, a random dot pattern image necessary for calculating the similarity is read from the projection pattern information holding unit 301. Next, three-dimensional coordinates randomly set based on the calibration data are projected, and the target pixels p1 and p2 are calculated. Finally, as in the first embodiment, the reciprocal of SAD is obtained.

  By performing such processing for all three-dimensional coordinates, the first similarity and the target pixel p1 are obtained for each three-dimensional point. The obtained first similarity and the target pixel p1 are output to the three-dimensional coordinate determination unit 306.

(Step S36)
In step S 36, the second similarity acquisition unit 304 projects the three-dimensional coordinates input from the three-dimensional coordinate determination unit 306 onto the two second images input from the image acquisition unit 302. The similarity between local areas in a pixel is obtained. Hereinafter, the process for obtaining the similarity will be described with reference to FIG. Since the description of each symbol is the same as in the first embodiment, the description is omitted.

  First, the three-dimensional coordinates set at random based on the calibration data are projected, and the target pixels p3 and p4 are calculated. Then, as in the first embodiment, the reciprocal of SAD is obtained.

  By performing such processing in all three-dimensional coordinates, the second similarity and the target pixel p3 are obtained for each three-dimensional point. The obtained second similarity and the target pixel p3 are output to the three-dimensional coordinate determination unit 306.

(Step S37)
In step S <b> 37, the three-dimensional coordinate determination unit 306 includes the first similarity input from the first similarity acquisition unit 303, the target pixel p <b> 1, and the second similarity input from the second similarity acquisition unit 304. The three-dimensional coordinate is evaluated using the target pixel p3 and the first reliability and the second reliability input from the reliability acquisition unit 305, and the three-dimensional coordinate is updated or determined.

  First, in each three-dimensional coordinate, the first reliability in the target pixel p1 and the second reliability in the target pixel p3 are obtained from the target pixel p1 and the target pixel p3. Next, the first similarity and the second similarity in each three-dimensional coordinate are integrated based on Expression 2 as in the second embodiment. Then, using the integrated similarity C (p, q) as an evaluation value, the three-dimensional coordinates are updated or determined by the hill-climbing method. The current integrated similarity and the integrated similarity in the vicinity of the current three-dimensional coordinate are compared and evaluated, and updated to a three-dimensional coordinate having a higher similarity. By repeating the update of the three-dimensional coordinates and the evaluation of the integrated similarity until there is no change in the similarity, it is possible to obtain a three-dimensional coordinate with the maximum (or maximum) integrated similarity. When there is no change in the integrated similarity, the search is finished and the three-dimensional coordinates are determined. When there is a change in the integrated similarity, the three-dimensional coordinates are updated, and the updated three-dimensional coordinates are obtained as the first similarity. To the unit 303 and the second similarity acquisition unit 304.

  By performing such processing for each three-dimensional coordinate, three-dimensional coordinates (X, Y, Z) corresponding to the number of pixels are calculated for each pixel of the first image, and the processing ends.

  With the configuration described above, the three-dimensional points are determined based on the similarity obtained from the active method and the passive method and the reliability obtained from each image, so that all the images can be processed at high speed regardless of the texture of the target object. The peripheral shape can be calculated stably and with high accuracy.

[Modification of Third Embodiment]
In step S34, the calculation of the reliability need not be limited to the spatial frequency. For example, in the active method, a correct three-dimensional coordinate can be obtained if the projection pattern is captured with high luminance. Therefore, in the first image, the higher the luminance value, the higher the reliability, and the lower the pixel, the lower the reliability. It may be set. On the other hand, in the passive method, since correct three-dimensional coordinates can be obtained if there is a gradient feature, the reliability of pixels with a strong gradient strength in the second image may be set high, and the reliability of weak pixels may be set low. .

(Modification common to each embodiment)
[Modification 1]
In the first embodiment, the second embodiment, and the third embodiment, the wavelengths of light used in the active method and the passive method are divided into near-infrared light and visible light, respectively, but this is necessary. There is no. Visible light may be used for the active method, and near infrared light may be used for the passive method, or light having a wavelength other than near infrared light or visible light may be used. Furthermore, instead of dividing the wavelength of the light, it may be possible to take an image of the state in which the pattern is projected and the state in which the pattern is not projected separately.

  Further, the imaging devices used in the active method and the passive method may be shared. That is, one projection device and two imaging devices may be used. At this time, the state in which the pattern is projected and the state in which the pattern is not projected may be imaged separately, or only the state in which the pattern is projected may be imaged. In either case, the same effect can be obtained if the similarity is calculated in the same manner as in the first embodiment, the second embodiment, and the third embodiment. By doing in this way, since a required apparatus can be reduced, the cost of an apparatus can be held down.

[Modification 2]
In step S11, step S21, and step S31, the pattern image to be projected and the pattern image held by the projection pattern information holding unit are random dot patterns. However, the present invention is not limited to this. Any pattern may be used as long as a pattern similar to a certain local area in the pattern image is not found in another area in the search direction (epipolar direction). For example, when epipolar lines are aligned with the horizontal direction of the image (when rectified), a dot pattern with randomness only in the horizontal direction may be used, or instead of dots, line segments, circles, etc. It is also possible to have a pattern in which are arranged randomly. Further, the pattern image need not be limited to binary, and may be a pattern having gradation and color tone.

  Moreover, it is not necessary to hold | maintain as a pattern image. For example, only the pixel position where the dot of the random dot exists may be held, or a random dot pattern image may be generated according to the probability of randomly hitting a dot as a parameter.

[Modification 3]
In step S12, step S22, and step S32, the image is acquired by capturing the target object in the middle of the measurement process. However, the present invention is not limited to this. An image captured in advance may be read from the memory, or may be downloaded from a server on the network.

[Modification 4]
In step S13, step S14, step S23, step S24, and step S34, the active imaging device is used as the reference viewpoint. However, the present invention is not limited to this. Another projection device or imaging device may be used as the reference viewpoint, or a virtually set viewpoint may be used as the reference viewpoint. Further, a three-dimensional shape may be expressed on a voxel basis without setting a reference viewpoint.

[Modification 5]
In step S13, step S14, step S23, step S24, and step S34, a three-dimensional point is projected on each image to determine a target pixel for calculating the similarity and a pixel in the vicinity region (local region). Although the pixel is determined, it is not necessary to be limited to this. For example, if the stereo image has been rectified in advance, the pixel of interest may be represented by a parallax instead of a 3D point, or a 3D plane or a 3D curved surface instead of a 3D point may be projected on each image. By doing so, the local region may be determined.

[Modification 6]
In step S13, step S14, step S23, step S24, step S35, and step S36, SAD is set as the similarity, but it is not necessary to be limited to this. Any index can be used as long as it can express the similarity, such as SSD (Sum of Squared Difference), NCC (Normalizes Cross-Correlation), and ZNCC (Zero-mean Normalized Cross-Correlation). Furthermore, a value that is scaled by taking an index or logarithm for these indices may be used.

  In addition, the pixels in the vicinity region for calculating the similarity need not be limited to the eight pixels around the target pixel, and any number of the surrounding pixels may be used. There may be 0 surrounding pixels or 24 surrounding pixels.

[Modification 7]
In step S15, step S26, and step S37, the calculation method of the integrated similarity need not be limited to Expression 1 and Expression 2.

  It may be represented by multiplication of similarity as in the above equation (3), or may be represented by logarithmic sum as in equation (4).

[Modification 8]
In step S15, step S26, and step S37, the three-dimensional coordinate determination method is not limited to the full search or the hill-climbing method. You may determine using algorithms, such as Graph Cut, Belief Propagation, annealing method, PatchMatch.

<Effect of embodiment>
According to the first embodiment, the three-dimensional point is determined based on the similarity obtained by integrating the similarities obtained from the active method and the passive method, so that the three-dimensional coordinates can be stabilized regardless of the texture of the target object. Can be calculated with high accuracy.

  According to the second embodiment, by determining the three-dimensional point based on the similarity obtained from the active method and the passive method and the reliability obtained from the similarity, the target object can be more stably and accurately obtained. Three-dimensional coordinates can be calculated regardless of the texture.

  According to the third embodiment, the three-dimensional point is determined based on the similarity obtained from each of the active method and the passive method and the reliability obtained from each image. Therefore, the entire peripheral shape can be calculated stably and with high accuracy.

<Definition>
The pattern image held by the projection pattern information holding unit in the present invention is a pattern in which a pattern similar to a certain local area in the pattern image cannot be found in another area in the search direction, as described in Modification 2. Any pattern is acceptable. For example, if the search direction is the horizontal direction of the image, a pattern in which dots are randomly arranged only in the horizontal direction may be used, or a pattern in which simple geometric shapes such as line segments and circles are arranged instead of dots. Good. Further, the pattern image need not be limited to binary, and may be a pattern having gradation and color tone. Moreover, it is not necessary to hold | maintain as a pattern image. For example, only the pixel position where the dot of the random dot exists may be held, or a random dot pattern image may be generated according to the probability of randomly hitting a dot as a parameter.

  In addition, as described in the third modification, the image acquisition unit according to the present invention can acquire the first image that is an image for an active method and the second image that is an image for a passive method. Such a method may be used. For example, an image may be captured every time three-dimensional measurement is performed, or an image captured in advance may be read from the memory.

  In addition, as described in the sixth modification, the first similarity acquisition unit and the second similarity acquisition unit according to the present invention are any indexes as long as they are indexes representing the similarity between local regions of an image. May be used. For example, it may be SAD, SSD, NCC, ZNCC, or the like.

  In addition, as described in the second embodiment and the third embodiment, the reliability acquisition unit according to the present invention may use any value as long as the value represents the reliability of similarity. Good. For example, the higher the peak degree (kurtosis) of each degree of similarity, the higher it may be, or it may be determined by the distance of three-dimensional coordinates obtained from each method. Also, the higher the spatial frequency in the local region of each image, the higher it may be, or it may be determined by the brightness value or the height of the gradient strength.

  In addition, as described in the sixth modification, the three-dimensional coordinate determination unit according to the present invention can express any similarity as long as the integrated similarity considering both the first similarity and the second similarity can be expressed. You may take the degree integration method. For example, it may be a weighted linear sum, a multiplication, or a weighted logarithmic sum. Furthermore, as described in the modified example 7, any algorithm may be used as long as it can search for a three-dimensional coordinate having the maximum similarity (or maximum). For example, full search may be used, hill climbing method, Graph Cut, Belief Propagation, annealing method, PatchMatch, etc.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  1000: three-dimensional measuring device, 1: near infrared projection device, 2: near infrared imaging device, 3: visible light imaging device, 4: visible light imaging device, 1100: information processing device, 101: projection pattern information holding unit , 102: image acquisition unit, 103: first similarity acquisition unit, 104: second similarity acquisition unit, 105: three-dimensional coordinate determination unit

Claims (17)

Image acquisition means for acquiring a first image obtained by imaging a target object on which a pattern image is projected, and a plurality of second images obtained by imaging the target object from a plurality of viewpoints;
First similarity acquisition means for acquiring a first similarity based on a comparison between the local region of the first image and the local region of the pattern image;
Second similarity acquisition means for acquiring a second similarity based on a comparison between local regions in each of the plurality of second images;
Determining means for determining a three-dimensional coordinate of the surface of the target object based on the first similarity and the second similarity;
An information processing apparatus comprising: A reliability obtaining unit for obtaining a first reliability representing the reliability of the first similarity and a second reliability representing the reliability of the second similarity;
The determining means determines a three-dimensional coordinate of the surface of the target object based on a similarity having a higher corresponding degree of reliability among the first similarity and the second similarity. The information processing apparatus according to claim 1.   The determining means has a maximum similarity obtained from a sum of the first similarity and the second similarity weighted based on the first reliability and the second reliability, respectively. The information processing apparatus according to claim 2, wherein the three-dimensional coordinate is determined as a three-dimensional coordinate of the surface of the target object.   The reliability acquisition means calculates the first reliability higher as the kurtosis of the maximum value of the first similarity is higher, and increases the kurtosis of the maximum value of the second similarity. The information processing apparatus according to claim 2, wherein the second reliability is calculated to be high.   When the distance between the three-dimensional coordinate at which the first similarity is the maximum and the three-dimensional coordinate at which the second similarity is the maximum is smaller than a threshold, The information processing apparatus according to claim 2, wherein the second reliability is calculated as the same value.   The reliability acquisition means calculates the first reliability lower as the distance between the three-dimensional coordinate at which the first similarity is maximum and the three-dimensional coordinate at which the second similarity is maximum is smaller. The information processing apparatus according to claim 2, wherein the second reliability is calculated to be high.   The reliability acquisition means calculates the first reliability higher as the spatial frequency in the local region of the first image is higher, and increases the spatial frequency in the local region of the second image as the first. The information processing apparatus according to claim 2, wherein the second reliability is calculated to be high.   8. The method according to claim 2, wherein the reliability acquisition unit calculates the first reliability higher as a luminance value in the local region of the first image is higher. Information processing device.   The said reliability acquisition means calculates said 2nd reliability so that the gradient intensity | strength in the said local area | region of said 2nd image is strong, The any one of Claim 2 thru | or 8 characterized by the above-mentioned. Information processing device.   The information processing apparatus according to claim 1, wherein the second image is an image obtained by capturing the target object on which the pattern image is not projected.   2. The local region is a neighborhood region of each pixel obtained by projecting arbitrary three-dimensional coordinates on the pattern image, the first image, and the second image. The information processing apparatus according to any one of 1 to 10. A projection device and a first imaging device used to capture the first image;
A plurality of second imaging devices used to capture the plurality of second images;
The information processing apparatus according to claim 1, further comprising:   The information processing according to claim 12, wherein a baseline length between the projection device and the first imaging device and a baseline length between the plurality of second imaging devices are substantially the same. apparatus.   The information processing apparatus according to claim 12 or 13, wherein the first imaging apparatus and the plurality of second imaging apparatuses have substantially the same resolution and focal length.   The information processing apparatus according to claim 1, further comprising a projection pattern information holding unit that holds information of the pattern image. A method for controlling an information processing apparatus,
An image acquisition step of acquiring a first image obtained by imaging a target object on which a pattern image is projected, and a plurality of second images obtained by imaging the target object from a plurality of viewpoints;
A first similarity acquisition step of acquiring a first similarity based on a comparison between the local region of the first image and the local region of the pattern image;
A second similarity acquisition step of acquiring a second similarity based on a comparison between local regions in each of the plurality of second images;
A determination step of determining a three-dimensional coordinate of the surface of the target object based on the first similarity and the second similarity;
A method for controlling an information processing apparatus, comprising:   The program for functioning a computer as each means of the information processing apparatus of any one of Claims 1 thru | or 15.

Images