JP2019184568A - Parallax detection device, distance detection device, robot device, parallax detection method, distance detection method and program - Google Patents

Abstract

To provide a parallax detection device that can reduce calculation errors upon calculating an amount of parallax using images with periodicity.SOLUTION: A parallax detection device comprises: acquisition means that acquires a pair of images having parallax; correlation calculation means that sets a reference image on one image of the pair of images, and calculates a correlation value of the pair of images on the basis of the reference image; and parallax calculation means that calculates an amount of parallax of the pair of images using the correlation value. The correlation calculation means is configured to: set a first reference image on the one image of the pair of images; calculate a first correlation value on the basis of the first reference image; set a second reference image at a position different from that of the first reference image in a prescribed direction on the one image of the pair of images; and calculate a second correlation value on the basis of the second reference image. The parallax calculation means is configured to calculate the amount of parallax using the first correlation value and the second correlation value.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a parallax detection device, a distance detection device, a robot device, a parallax detection method, a distance detection method, and a program.

  There has been proposed a method of acquiring a captured image of a subject and calculating distance information with respect to the subject from the captured image. As an example of such a method, there is a method of acquiring distance information by acquiring image sets including images from different viewpoints, obtaining a parallax amount from a correlation value (also referred to as similarity) between the images.

  Specifically, first, an image signal of a partial region including the target pixel in one image of the image set is extracted as a reference image. Next, the image signal of the partial area in the other image is extracted as a reference image. Thereafter, the correlation value between the reference image and the reference image at each position is calculated (correlation calculation) while changing the position on the image from which the reference image is extracted. The parallax amount at the target pixel can be calculated by obtaining the position where the correlation value is highest in the correlation value between the calculated standard image and the reference image at each position. Thereafter, the distance information of the subject can be calculated by converting the parallax amount into distance information by a known method.

  However, if there is a region with a poor pattern in the captured image, the contrast of the image signal decreases, the amount of parallax cannot be calculated by correlation calculation, and distance information cannot be calculated (ranging). On the other hand, Patent Document 1 proposes a technique that enables distance measurement of such an area by acquiring a captured image using pattern light projection.

Japanese Patent No. 5803305

  However, when distance measurement is performed using an image captured by projecting pattern light, an area where the pixel value of the captured image changes greatly (a boundary between a bright area and a dark area of the projection pattern) and an end of the reference image If the amount of parallax is calculated in a state where the two overlap, an error occurs in the calculated amount of parallax. In this case, due to an error in the amount of parallax, an error also occurs in the distance information obtained by converting the amount of parallax. In particular, this error becomes prominent when distance measurement is performed by projecting pattern light having periodicity such as a line pattern in which high luminance regions and low luminance regions are alternately arranged. Note that this parallax amount calculation error may occur in the same manner when the parallax amount is obtained using a captured image of a subject having a periodic pattern.

  Therefore, the present invention provides a parallax detection device, a distance detection device, a robot device, a parallax detection method, a distance detection method, and a program capable of reducing a calculation error when calculating a parallax amount using an image having periodicity. I will provide a.

  The parallax detection device according to an embodiment of the present invention includes an acquisition unit configured to acquire an image set having parallax, a reference image set on one image of the image set, and the image set based on the reference image. A correlation calculating unit that calculates a correlation value; and a parallax calculating unit that calculates a parallax amount of the image set using the correlation value, wherein the correlation calculating unit includes a first unit on one image of the image set. A reference image is set, a first correlation value is calculated based on the first reference image, and a position different from the position of the first reference image in a predetermined direction on one image of the image set A second reference image is set to the second reference image, a second correlation value is calculated based on the second reference image, and the parallax calculation means uses the first correlation value and the second correlation value. The parallax amount is calculated.

  A parallax detection device according to another embodiment of the present invention includes a projection unit that projects pattern light onto a subject, an acquisition unit that acquires an image set having parallax, and a reference image on one image of the image set. And a calculation unit for calculating a correlation value of the image set based on the reference image, and a parallax calculation unit for calculating a parallax amount of the image set using the correlation value. The first pattern light and the second pattern light having patterns whose positions are shifted with respect to each other in a predetermined direction are projected, and the correlation calculation unit projects the first pattern light to obtain the first pattern light. A first correlation value is calculated based on the second image set, a second correlation value is calculated based on the second image set obtained by projecting the second pattern light, and the parallax calculating unit includes: Using the first correlation value and the second correlation value It calculates the amount of parallax.

  Furthermore, a parallax detection device according to still another embodiment of the present invention includes a projection unit that projects pattern light onto a subject, an acquisition unit that acquires an image group having parallax, and an image on one image of the image group. A correlation calculation unit that sets a reference image and calculates a correlation value of the image set based on the reference image; and a parallax calculation unit that calculates a parallax amount of the image set using the correlation value, The light includes a first sub-pattern light and a second sub-pattern light, and the first sub-pattern light and the second sub-pattern light have a first direction and a second direction perpendicular to the first direction. The reference light includes an image of a region where the first sub-pattern light and the second sub-pattern light are projected.

  According to the present invention, it is possible to reduce a calculation error when calculating a parallax amount using an image having periodicity.

1 shows a schematic configuration example of a distance detection device according to a first embodiment. 1 illustrates a schematic configuration example of pixels included in an image sensor according to Embodiment 1. It is a figure for demonstrating the distance detection method which concerns on Example 1. FIG. An example of the measurement result which concerns on Example 1 is shown. It is a figure for demonstrating the distance detection method which concerns on Example 1. FIG. It is a figure for demonstrating the distance detection method which concerns on Example 1. FIG. It is a figure for demonstrating the distance detection method which concerns on the modification of Example 1. FIG. The flow of the distance detection method which concerns on the modification of Example 1 is shown. 3 shows a schematic configuration example of a robot apparatus according to a second embodiment. The schematic structural example of the distance detection apparatus which concerns on Example 3, and the flow of the distance detection method are shown. An example of the pattern light which concerns on Example 4 is shown. It is a figure for demonstrating the distance detection method which concerns on Example 4. FIG. An example of the measurement result which concerns on Example 4 is shown. It is a figure for demonstrating the distance detection method which concerns on Example 4. FIG. It is a figure for demonstrating the distance detection method which concerns on Example 4. FIG. 10 shows a schematic configuration example of a distance detection device according to Embodiment 5. It is a figure for demonstrating the distance detection method which concerns on Example 5. FIG. 10 shows a schematic configuration example of a distance detection apparatus according to Embodiment 6. It is a figure for demonstrating the distance detection method which concerns on Example 6. FIG. An example of the measurement result which concerns on Example 6 is shown. It is a figure for demonstrating the distance detection method which concerns on Example 6. FIG. It is a figure for demonstrating the distance detection method which concerns on Example 6. FIG. It is a figure for demonstrating the distance detection method which concerns on the modification of Example 6. FIG.

  Hereinafter, exemplary embodiments for carrying out the present invention will be described in detail with reference to the drawings. However, dimensions, materials, shapes, and relative positions of components described in the following embodiments are arbitrary and can be changed according to the configuration of the apparatus to which the present invention is applied or various conditions. Also, in the drawings, the same reference numerals are used between the drawings to indicate the same or functionally similar elements.

Example 1
Hereinafter, a distance detection apparatus according to Embodiment 1 of the present invention will be described with reference to FIGS. FIG. 1 shows a schematic configuration example of a distance detection apparatus according to the present embodiment. FIG. 1A shows a schematic configuration example of a distance detection apparatus 100 using pattern light projection together. FIG. 1B shows an example of the projected pattern light.

(Device configuration)
The distance detection device 100 according to the present embodiment is provided with a projection device 101 and a photographing device 103. The projection device 101 projects pattern light onto the subject 102, and the photographing device 103 photographs the subject 102 using the return light of the pattern light from the subject 102 to obtain a photographed image. The projection apparatus 101 and the imaging apparatus 103 are connected to the control unit 108, and the control unit 108 performs control such as synchronization of the projection apparatus 101 and the imaging apparatus 103. Note that the projection apparatus 101 can be fixed to the imaging apparatus 103 by an arbitrary method, and may be fixed to the imaging apparatus 103 so as to be detachable.

  The projection apparatus 101 is provided with a light source (not shown), an imaging optical system, and a pattern mask in which a pattern is formed on ground glass or a metal plate as an example of a pattern forming unit. As the light source, an LED (Light Emitting Diode) or the like can be used. Note that when only these components are arranged in the projection apparatus 101, the apparatus cost can be reduced and the apparatus can be downsized. In the present embodiment, a line pattern 109 shown in FIG. 1B is projected as an example of pattern light projected by the projection apparatus 101.

  The imaging apparatus 103 is provided with an imaging optical system 104, an image sensor 105, an arithmetic processing unit 106, and a main body memory 107. Note that the photographing apparatus 103 may be provided with a mount or the like for fixing the projection apparatus 101.

  The imaging optical system 104 has a function of forming an image of the subject 102 on the imaging element 105 that is an imaging surface. The imaging optical system 104 is provided with a plurality of lens groups (not shown), stops, and the like, and the imaging optical system 104 has an exit pupil at a position away from the image sensor 105 by a predetermined distance. Here, in FIG. 1A, the optical axis 140 of the imaging optical system 104 is indicated by a one-dot chain line, and the optical axis 140 is parallel to the z-axis. The x axis and the y axis are perpendicular to each other and are perpendicular to the optical axis 140 and the z axis.

  The image sensor 105 is provided with a substrate 204 and a plurality of pixels. Here, FIGS. 2A and 2B show schematic configuration examples of the pixels included in the image sensor 105. FIG. FIG. 2A is a cross-sectional view of pixels arranged in the image sensor 105.

  Each pixel is provided with a microlens 201, a color filter 202, and photoelectric conversion units 203A and 203B. The image sensor 105 is provided with a spectral characteristic of RGB (Red, Green, Blue: red, green, blue) corresponding to the wavelength band to be detected for each pixel by the color filter 202, and each pixel has a well-known public image (not shown). They are arranged on the xy plane by a color arrangement pattern. On the substrate 204 of the image sensor 105, photoelectric conversion units 203A and 203B having sensitivity in the wavelength band to be detected are formed for each pixel. Each pixel is provided with a wiring (not shown), and each pixel can send an output signal (image signal) to the arithmetic processing unit 106 via the wiring.

  FIG. 2B shows the exit pupil 130 of the imaging optical system 104 viewed from the intersection (center image height) between the optical axis 140 and the image sensor 105. The first light beam that has passed through the first pupil region 210 and the second light beam that has passed through the second pupil region 220 are incident on the photoelectric conversion unit 203A and the photoelectric conversion unit 203B, respectively. To do. The photoelectric conversion unit 203A and the photoelectric conversion unit 203B in each pixel can generate image signals corresponding to the A image and the B image, respectively, by photoelectrically converting the incident light flux. The generated image signal is transmitted to the arithmetic processing unit 106 which is an example of an arithmetic unit, and the arithmetic processing unit 106 generates an A image and a B image based on the received image signal. The arithmetic processing unit 106 performs a distance detection process using the A image and the B image, calculates a distance value, and stores the calculated distance value in the main body memory 107. Further, the arithmetic processing unit 106 stores an image obtained by adding the A image and the B image in the main body memory 107 as image information, and can use it in the subsequent processing. The arithmetic processing unit 106 can also store the A image and the B image itself in the main body memory 107.

  FIG. 2B shows the centroid position of the first pupil region 210 (first centroid position 211) and the centroid position of the second pupil region 220 (second centroid position 221). . In the present embodiment, the first barycentric position 211 is decentered (moved) along the first axis 200 from the center of the exit pupil 130. On the other hand, the second center of gravity position 221 is eccentric (moved) along the first axis 200 in the direction opposite to the first center of gravity position 211. A direction connecting the first barycentric position 211 and the second barycentric position 221 is referred to as a pupil division direction. In addition, the distance between the centroids of the first centroid position 211 and the second centroid position 221 is the baseline length 230.

  The positions of the A image and the B image are shifted in the same direction as the pupil division direction (in the present embodiment, the x-axis direction) by defocusing. The relative positional shift amount between the images, that is, the parallax amount between the A image and the B image is an amount corresponding to the defocus amount. Therefore, this parallax amount can be acquired by a method described later, and the parallax amount can be converted into a defocus amount by a known conversion method. Further, the defocus amount can be converted into distance information by a known conversion method.

  The arithmetic processing unit 106 includes a correlation calculation unit 161, a parallax calculation unit 162, and a distance calculation unit 163. The correlation calculation unit 161 sets an image of a partial area including a pixel (target pixel) for calculating a distance as a reference image on the A image, sets a reference image for the B image, and sets the position of the reference image in a predetermined direction. The correlation value between the standard image and the reference image is calculated while moving the image.

  The parallax calculation unit 162 calculates the parallax amount of the image set including the A image and the B image using the correlation value calculated by the correlation calculation unit 161. The distance calculation unit 163 calculates the distance to the subject 102 using the parallax amount calculated by the parallax calculation unit 162.

  The control unit 108 may be configured using a general-purpose computer, or may be configured as a computer dedicated to the distance detection device 100. Each component of the arithmetic processing unit 106 can be configured by a software module that is executed on an arithmetic device such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). Similarly, each component of the arithmetic processing unit 106 may be configured by a circuit that performs a specific function such as an ASIC. Furthermore, the control unit 108 that performs synchronization control of the projection device 101 and the imaging device 103 may be realized in the arithmetic processing unit 106 of the imaging device 103. The main body memory 107 may be configured by any known memory such as a RAM or a ROM.

(Distance detection process flow)
Next, a flow of distance detection processing according to the present embodiment will be described with reference to FIGS. FIG. 3A is a flowchart of distance detection processing according to the present embodiment, and FIGS. 3B and 3C are diagrams for explaining the correlation calculation performed by the correlation calculation unit 161. When the distance detection process according to the present embodiment is started, the process proceeds to step S301.

  In step S301 “acquire pattern light projection image”, image capturing is performed using the image capturing apparatus 103 while pattern light is being projected from the projection apparatus 101 to the subject 102, and the captured image is stored in the main body memory 107. Specifically, first, light having a line pattern 109 is generated by a spatial light modulator (not shown) which is an example of a pattern control unit provided inside the projection apparatus 101 and is irradiated onto the surface of the subject 102. . Thereafter, photographing is performed by the photographing apparatus 103 in this state, an image set including an A image and a B image having parallax is generated and acquired, and the acquired image set is stored in the main body memory 107. At this time, the control unit 108 controls the operations and timings of the projection apparatus 101 and the imaging apparatus 103 so that the imaging apparatus 103 is exposed in a state where pattern light projection is performed.

  Here, when the subject 102 having a poor pattern (also referred to as a texture) is photographed with only ambient light, the contrast and S / N ratio of the A image and the B image are lowered, and distance calculation (measurement) using correlation calculation is performed. The accuracy of distance calculation is reduced. On the other hand, the accuracy of distance calculation can be improved by irradiating and projecting pattern light from the projection apparatus 101 onto the subject 102 and shooting with the texture superimposed on the surface of the subject 102.

  The processing from step S302 to step S305 is performed by the arithmetic processing unit 106. Here, FIGS. 3B and 3C are diagrams for explaining the positional relationship between the standard image and the reference image set in step S302 and step S303. FIG. 3B shows an A image 310A, and FIG. 3C shows a B image 310B.

  In step S302 “correlation calculation 1”, the correlation calculation unit 161 of the calculation processing unit 106 calculates the first correlation value of the A image 310A and the B image 310B. Specifically, the correlation calculation unit 161 first extracts a partial area including the target pixel 320 and its neighboring pixels from the A image 310 </ b> A and sets it as the first reference image 311. Next, the correlation calculation unit 161 extracts a region having the same area (image size) as that of the first reference image 311 on the B image 310B and sets it as the reference image 313. Thereafter, the correlation calculation unit 161 moves the position where the reference image 313 is extracted on the B image 310B in the same x-axis direction as the pupil division direction, and the reference image 313 and the first reference image at each movement amount (each position). A correlation value with 311 is calculated. Thereby, the correlation calculation unit 161 calculates a first correlation value including a data string of correlation values corresponding to each movement amount. Note that the correlation calculation unit 161 can set the reference image 313 as an image having the same vertical and horizontal dimensions as the vertical and horizontal dimensions of the first base image 311.

  A direction in which the reference image 313 is moved and the correlation calculation is performed is referred to as a parallax calculation direction. By setting the parallax calculation direction to the same direction as the pupil division direction, it is possible to correctly calculate the parallax amount generated according to the subject distance between the A image 310A and the B image 310B. In addition, as a calculation method of a correlation value, general calculation methods, such as SAD (Sum of Absolute Difference) and SSD (Sum of Squared Difference), can be used.

  Next, in step S303 “correlation calculation 2”, the correlation calculation unit 161 calculates the second correlation value of the A image 310A and the B image 310B. Specifically, the correlation calculation unit 161 is a region having the same area (image size) as the first reference image 311 on the A image 310A and is located at a different position in the pupil division direction (x-axis direction). An area is extracted and set as the second reference image 312. Next, the correlation calculation unit 161 extracts a region having the same area (image size) as that of the second standard image 312 on the B image 310B and sets it as the reference image 313. Thereafter, similarly to step S302, the correlation calculation unit 161 moves the position of the reference image 313 in the parallax calculation direction, and calculates a correlation value between the reference image 313 and the second reference image 312 at each movement amount. Thereby, the correlation calculation part 161 calculates the 2nd correlation value which consists of a data sequence of the correlation value with respect to each moving amount | distance. Note that the correlation calculation unit 161 can set the second reference image 312 as an image having the same vertical and horizontal dimensions as the vertical and horizontal dimensions of the first reference image 311.

  In the correlation calculation 2, the reference image 313 for the second standard image 312 can be set under the same conditions as the setting conditions for the reference image 313 for the first standard image 311. For example, it is assumed that the reference image 313 is set to be an image at the position of the B image 310B corresponding to the position of the first standard image 311 in the A image 310A. In this case, in the correlation calculation 2, the reference image 313 may be set to be an image at the position of the B image 310B corresponding to the position of the second standard image 312 in the A image 310A. The correspondence relationship between the position on the A image 310A and the position on the B image 310B may be specified by a known method, for example, based on the structure of the pixels that acquire the image signals constituting each image. Can.

  Further, the amount of movement of the reference image 313 in the correlation calculation 2 can be made substantially the same as the amount of movement of the reference image 313 in the correlation calculation 1. For example, when the movement amount of the reference image 313 in the correlation calculation 1 is changed from −M to + M, the correlation calculation unit 161 can also change the movement amount of the reference image 313 in the correlation calculation 2 from −M to + M.

  In step S304 “calculation of parallax”, the parallax calculation unit 162 of the arithmetic processing unit 106 calculates the amount of parallax using the first correlation value and the second correlation value obtained in steps S302 and S303. Specifically, the parallax calculation unit 162 adds or averages the correlation values of the first correlation value and the second correlation value for each movement amount, and calculates a third correlation value. At this time, the parallax calculation unit 162 calculates the third correlation from the data string of each correlation value obtained by adding or averaging the correlation values of the corresponding movement amounts in the first correlation value and the second correlation value. A value can be calculated. For example, the parallax calculation unit 162 adds or averages the first correlation value and the second correlation value of the movement amount −M of the reference image, and calculates a third correlation value corresponding to the movement amount −M. Can do.

  Thereafter, the parallax calculation unit 162 calculates the amount of parallax using the third correlation value by a known arbitrary method. For example, the data amount of the correlation value corresponding to the movement amount that provides the highest correlation among the third correlation values and the movement amount in the vicinity thereof is extracted, and the highest correlation is obtained by any known interpolation method. The amount of parallax can be calculated by estimating the amount of movement with subpixel accuracy.

  In step S305 “distance value calculation”, the distance calculation unit 163 of the arithmetic processing unit 106 converts the parallax amount into the defocus amount or the subject distance by any known method. Conversion from the amount of parallax to the amount of defocus can be performed using a geometric relationship using the baseline length. Further, the conversion from the defocus amount to the subject distance can be performed using the imaging relationship of the imaging optical system 104. Alternatively, the parallax amount may be converted into a defocus amount or a subject distance by multiplying the parallax amount by a predetermined conversion coefficient. By such a method, the distance calculation unit 163 can calculate the distance to the subject 102 using the amount of parallax at the target pixel 320.

  As described above, in the distance detection method according to the present embodiment, the first reference image 311 and the second reference image 312 are set at different positions in the pupil division direction (parallax calculation direction), and the reference image set for each is set. The correlation value with 313 is calculated, and the amount of parallax is calculated from both correlation values. According to this processing, it is possible to reduce the measurement error of the parallax amount that occurs in relation to the luminance distribution of the projection pattern and the position of the reference image, thereby reducing the distance measurement error, and performing highly accurate distance measurement. Can do.

  Here, an example of the processing result of the distance detection method according to the present embodiment will be described with reference to FIGS. FIG. 4A is an image obtained by projecting pattern light using a projection device 101 onto a flat plate arranged in parallel with the imaging device 103 at a known distance and imaging the imaging device 103. FIG. 4B shows a result of an error when calculating the amount of parallax at each position on the flat plate. The horizontal axis in FIG. 4B represents the movement amount (pixel position), and the vertical axis represents the parallax amount calculation error. In FIG. 4B, the parallax amount calculation error 401 calculated by the conventional process is indicated by a broken line, and the parallax amount calculation error 402 calculated by the process according to the present embodiment is indicated by a solid line. It can be seen that the error approaches 0 and the calculation error of the parallax amount is reduced by the method of the present embodiment as compared with the prior art.

  Next, with reference to FIGS. 5A to 5E, the reason why an error occurs in the calculation of the parallax amount in the conventional processing and the calculation error of the parallax amount is reduced in the processing according to the present embodiment will be described. In the following description, it is assumed that the A image and the B image have the same shading and have no parallax. 5A to 5C are diagrams for explaining the reason for the occurrence of an error.

  FIG. 5A is a diagram illustrating a positional relationship between the A image 501 having a line pattern in which bright regions and dark regions appear alternately, the reference image 502, and the reference image 503. The reference image 502 includes image edges 504 and 505 (boundary portions) where the bright area and the dark area of the A image 501 are switched inside the reference image 502. FIG. 5B shows correlation values calculated by moving the reference image set with respect to the standard image 502 and performing a correlation calculation between the standard image 502 and the reference image. Here, the lower the correlation value, the higher the correlation.

  Correlation values C0, Cp, and Cm are correlation values when the position of the reference image is moved by 0, +1, and −1 pixel, respectively. Here, since it is assumed that the A image and the B image have no parallax, when the movement amount is 0, the two images coincide with each other, and the correlation value C0 is a low value. When the reference image is moved by +1 pixel or −1 pixel, a difference (difference) occurs between the base image 502 and the reference image due to the image edges 504 and 505, and thus the correlation values Cp and Cm increase from the correlation value C0. . At this time, since the absolute values of the movement amounts of the reference images corresponding to the correlation values Cp and Cm are the same, the difference between the images caused by the image edges 504 and 505 on the line pattern is also the same amount. Therefore, the correlation value Cp and the correlation value Cm are the same value. The correlation value is interpolated to obtain a correlation curve 510, and the movement amount (parallax amount) that gives the highest correlation value is calculated, whereby the parallax amount 511 is obtained, and the correct value (parallax amount 0) is obtained.

  On the other hand, in the reference image 503, the image edge 504 and the right end of the reference image 503 overlap. FIG. 5C shows each correlation value calculated by moving the reference image set with respect to the standard image 503 and performing a correlation calculation between the standard image 503 and the reference image. When the movement amount is 0, both images match and the correlation value C0 is a low value. When the movement amount is +1 pixel, the correlation value Cp increases from the correlation value C0 as in the case of the reference image 502. On the other hand, when the movement amount is −1 pixel, the difference between the base image and the reference image is caused only by the image edge 505, and therefore, the increase amount of the correlation value Cm from the correlation value C0 is very small. For this reason, the correlation value in this case is asymmetrical on the + side and the − side of the movement amount of the reference image, and the parallax amount 513 obtained from the correlation curve 512 obtained by interpolating this is different from the correct value (parallax amount 0). An error occurs. This is a calculation error of the amount of parallax generated in relation to the luminance distribution of the projection pattern and the position of the reference image.

  Next, as in the present embodiment, a second reference image having a position different from that of the first reference image in the parallax calculation direction is set, and calculation is performed using each of the first reference image and the second reference image. The reason why the parallax amount calculation error is reduced by calculating the parallax amount from the correlation value will be described. FIG. 5D is a diagram showing the positions of the A image 501, the first reference image 503, and the second reference image 506. FIG. 5E shows correlation values calculated using the first reference image 503 and the second reference image 506.

  The first reference image 503 is assumed to be a reference image in which the image edge 504 and the right end of the reference image overlap as described above. At this time, the first correlation values Cm1, C01, Cp1 calculated from the first reference image 503 are the same as the correlation values Cm, C0, Cp shown in FIG.

  Next, the second reference image 506 is set so that the left end of the second reference image 506 overlaps the image edge 504. At this time, the positional relationship between the second reference image 506 and the image edge 504 is a relationship obtained by inverting the positional relationship between the first reference image 503 and the image edge 504. Therefore, as shown in FIG. 5E, the second correlation values Cm2, C02, and Cp2 obtained using the second reference image 506 are the first correlation obtained using the first reference image 503. The values Cm1, C01, and Cp1 are inverted.

  When these correlation values are added and averaged for each position (movement amount) of the reference image, the correlation values become the third correlation values Cm3, C03, and Cp3. The third correlation value is symmetrical because the asymmetry on the + side and − side of the movement amount of the reference image is canceled. In this case, the parallax amount 515 obtained from the correlation curve 514 obtained by interpolating the correlation value becomes a correct value (parallax amount 0), and an appropriate parallax amount can be calculated. In the process according to the present embodiment, the calculation error of the parallax amount can be reduced based on such a principle, and the distance can be measured with high accuracy based on the appropriately calculated parallax amount.

  Here, in the distance detection method according to the present embodiment, the calculation error of the parallax amount is further reduced by making the positions of the first reference image and the second reference image different by an appropriate amount in the parallax calculation direction. Can do. FIG. 6 is a diagram illustrating an appropriate position of each reference image.

  The first reference image 520 and the first reference image 521 are reference images in which the right edge of the reference image overlaps the image edge 504 or the image edge 505 of the A image 501. Consider a case where an error that occurs in a correlation value when these first reference images 520 and 521 are used is canceled with the correlation value calculated with the second reference image.

  In this case, since the error of the correlation value due to the image edge at the right end of the first reference image is canceled, the second error of the correlation value due to the image edge at the left end of the second reference image is generated. The reference image may be set. Therefore, the second reference image may be set so that the left end of the second reference image overlaps the image edge included in the A image 501 in the vicinity of the first reference images 520 and 521.

For example, as shown in FIG. 6, the second reference image can be set like any one of the second reference images 522, 523, 524, 525, 526, 527. At this time, when the position difference in the x-axis direction between the first reference image 520 or the first reference image 521 and each of the second reference images 522, 523, 524, 525, 526, 527 is obtained, The difference of can be expressed by the following formula (1a) or (1b).

  Here, W is the width of the first reference image and the second reference image in the x direction (parallax calculation direction), P is the period of the projected pattern on the captured image, and H is the high brightness area on the captured image of the projected pattern. , N represents an arbitrary integer.

  In the distance detection method according to the present embodiment, the positions of the first reference image and the second reference image are most appropriately changed from each other in the parallax calculation direction by the amount indicated by Expression (1a) or Expression (1b). The calculation error of the parallax amount can be reduced, and highly accurate distance measurement can be performed.

  Note that the position of the second reference image in the direction perpendicular to the parallax calculation direction (y-axis direction) may be set to a position different from that of the first reference image. Further, the positions of the first reference image and the second reference image in the x-axis direction or the y-axis direction can be as close as possible to each other. When the first reference image and the second reference image are closer to each other, both reference images can be set in a region where the distance between the subjects is substantially the same, and the parallax amount calculation error can be reduced more appropriately.

  Further, an image acquired by the distance detection device 100 has an image height dependency due to illumination unevenness of the projection device 101, aberration of the imaging optical system 104, and the like. Also from this viewpoint, the positions of the first reference image and the second reference image in the x-axis direction or the y-axis direction can be provided close to each other. Preferably, if the diagonal length of the captured image is 1, the first reference image and the second reference image should be close to each other so that the distance is within 0.1, and more preferably the distance It is better to make it close to within 0.05. Thus, by setting the positions of the first and second reference images, it is possible to more appropriately reduce the parallax amount calculation error.

  As described above, the distance detection device according to the present embodiment includes the projection device 101, the imaging device 103, and the arithmetic processing unit 106 including the correlation calculation unit 161, the parallax calculation unit 162, and the distance calculation unit 163. . The projection device 101 projects pattern light onto the subject 102. The imaging apparatus 103 acquires an image set having parallax using the pattern light projected from the projection apparatus 101. The correlation calculation unit 161 sets a reference image on one image of the acquired image set, and calculates a correlation value of the image set based on the reference image. More specifically, the correlation calculation unit 161 sets a reference image on the other image in the image set, and calculates a correlation value between the reference image and the reference image while moving the position of the reference image in a predetermined direction. . The parallax calculation unit 162 calculates the parallax amount of the image set using the correlation value calculated by the correlation calculation unit 161. The distance calculation unit 163 calculates the distance to the subject 102 using the parallax amount calculated by the parallax calculation unit 162.

  More specifically, the correlation calculation unit 161 sets a first reference image on one image of the image set, and calculates a first correlation value based on the first reference image. Further, the correlation calculation unit 161 sets the second reference image at a position different from the position of the first reference image in the parallax calculation direction on one image of the image set, and based on the second reference image. A second correlation value is calculated. Then, the parallax calculation unit 162 calculates the amount of parallax using the first correlation value and the second correlation value.

  Here, the correlation calculation unit 161 sets the first reference image and the second reference image according to the formula (1a) or the formula (1b). More specifically, the correlation calculation unit 161 determines the first reference image and the second reference image as the width of the first reference image in the parallax calculation direction, or the width and the period of the pattern light on the captured image. Are set at different positions in the parallax calculation direction by the difference between the two. Alternatively, the correlation calculation unit 161 converts the first reference image and the second reference image into a parallax by a difference between the width in the parallax calculation direction of the first reference image and the width of the high-luminance area of the pattern light on the captured image. Set different positions in the calculation direction. Alternatively, the correlation calculation unit 161 converts the first reference image and the second reference image from the width in the parallax calculation direction of the first reference image to the width of the high-brightness region of the pattern light and the cycle of the pattern light. Are set at different positions in the parallax calculation direction by the amount obtained by subtracting.

  Further, the parallax calculation unit 162 calculates the amount of parallax from the correlation value obtained by adding or averaging the first correlation value and the second correlation value.

  With such a configuration, the distance detection apparatus 100 can reduce parallax amount calculation errors in distance detection using a captured image on which pattern light is projected. Therefore, the distance detection apparatus 100 can acquire highly accurate distance information of the subject 102 based on the amount of parallax with a reduced calculation error.

  In the distance detection processing flow according to the present embodiment, the correlation calculation 2 is performed after the correlation calculation 1. However, after setting the first reference image in the correlation calculation 1, the correlation calculation 1 and the correlation calculation 2 are performed. You may process in parallel.

  Further, the method for calculating the amount of parallax in the present embodiment is not limited to the method in step S304 described above. For example, the parallax calculation unit 162 calculates the first parallax amount from the first correlation value, calculates the second parallax amount from the second correlation value, and adds and averages these parallax amounts. An appropriate amount of parallax may be calculated. In this case, the correlation calculation unit 161 does not have to obtain the third correlation value using the first and second correlation values.

  Further, as shown in FIG. 7A, in the A image 310A, the first reference image 701 and the second reference image 702 are decentered in opposite directions in the parallax calculation direction with the target pixel 320 as the center. May be. Also in this case, since the average distance information of the partial area centered on the target pixel 320 is calculated using the first reference image 701 and the second reference image 702, the position and distance information of the target pixel 320 are calculated. Can be reduced. Further, as described above, by appropriately setting the positional relationship between the first reference image 701 and the second reference image 702 in the parallax calculation direction, it is possible to more appropriately reduce the parallax amount calculation error.

  In the present embodiment, the method of setting the two reference images of the first reference image and the second reference image and calculating the parallax amount has been described. However, a larger number of reference images may be set around the first reference image, and the amount of parallax may be calculated using the correlation value calculated for each reference image.

  For example, as shown in FIG. 7B, the correlation calculation unit 161 sets a first reference image 701, a second reference image 702, and a third reference image 703, and uses each reference image to set the first reference image 701, the second reference image 702, and the third reference image 703. One correlation value, a second correlation value, and a third correlation value are calculated. Thereafter, the parallax calculation unit 162 may calculate the parallax amount from the correlation value obtained by adding or averaging the correlation values as described above. Further, the parallax calculation unit 162 calculates the amount of parallax from the correlation value obtained by adding the first correlation value and the second correlation value, and obtains it by adding the first correlation value and the third correlation value. The final amount of parallax may be calculated by calculating the amount of parallax based on the correlation value and averaging the parallax amounts.

  Even when the first reference image and the second reference image are set for the same image edge, the first correlation value is affected by the variation in the brightness of the projection pattern, the noise added to the captured image, the influence of aberration, and the like. And the second correlation value may not be completely symmetric. On the other hand, by setting more reference images and calculating the amount of parallax using the correlation value obtained from each reference image, these effects are reduced and the calculation error of the amount of parallax is further reduced. be able to. Therefore, distance measurement can be performed appropriately and with high accuracy by performing the processing.

  Furthermore, in the present embodiment, the case where the distance to the subject 102 is calculated for one target pixel has been described. In contrast, FIG. 8 shows a flow of a distance detection method for efficiently calculating distances (distance images) at a plurality of pixels on the A image. When the distance detection method is started, the process proceeds to step S801.

  In step S <b> 801 “acquire pattern light projection image”, as in step S <b> 301, the image capturing apparatus 103 captures an image while the projection apparatus 101 projects pattern light onto the subject 102, and the captured image is stored in the main body memory 107.

  Next, in step S802 “Calculate correlation value of each pixel”, the correlation calculation unit 161 calculates a correlation value for each pixel on the A image. Specifically, the reference image is set by extracting a partial region including the target pixel and its neighboring pixels on the A image. Next, a reference image is set on the B image, the position where the reference image is extracted is moved in the parallax calculation direction, and the correlation value between the reference image and the standard image at each movement amount is calculated, thereby correlating with each movement amount. Calculate the value. By performing this calculation by setting each pixel on the A image as a target pixel, a correlation value at each pixel is calculated.

  In step S803 “calculation of parallax amount of each pixel”, the parallax calculation unit 162 calculates the parallax amount of each pixel on the A image. Thereafter, the parallax calculation unit 162 selects a pixel that differs from the target pixel on the A image by a predetermined position in the parallax calculation direction. At this time, the parallax calculation unit 162 selects a pixel corresponding to the target pixel in the second reference image set so as to be different from the first reference image including the target pixel only at an appropriate position as described above. Then, the parallax calculation unit 162 selects the correlation value calculated in step S802 for the target pixel and the selected pixel, and calculates the parallax amount using the selected both correlation values. The amount of parallax can be calculated by the same method as in step S304.

  In step S804 “calculation of distance value of each pixel”, the distance calculation unit 163 calculates the distance value of each pixel on the A image. Specifically, the distance calculation unit 163 converts the parallax amount calculated in each pixel in step S803 into a defocus amount or a subject distance using a known method, similarly to step S305.

  By such a flow, compared with the case where a plurality of reference images are set for each pixel of interest and the correlation value is calculated, the number of correlation operations performed in duplicate can be reduced. A distance (distance image) in the pixel can be calculated.

  As an alternative example, after the correlation calculation unit 161 calculates the correlation value of each pixel in step S802, the parallax calculation unit 162 calculates a temporary parallax amount of each pixel using the correlation value. Next, in step S803, the parallax calculation unit 162 uses the parallax amount of each pixel calculated in step S802 to add and average the parallax amount of the pixel that differs from the target pixel by the appropriate position described above and the parallax amount of the target pixel. By doing so, the final amount of parallax may be calculated. Even in this case, the distances of a plurality of pixels can be calculated efficiently. The provisional parallax amount may be calculated in step S803.

  Note that the projection pattern irradiated onto the subject 102 by the projection apparatus 101 is preferably a line pattern in which a high luminance region and a low luminance region extend in a direction perpendicular to the parallax calculation direction. When the projection pattern is tilted obliquely with respect to the direction perpendicular to the parallax calculation direction, the spatial frequency component in the parallax calculation direction (pupil division direction) included in the captured image decreases, so the accuracy of the correlation calculation decreases, and the amount of parallax The calculation accuracy of is reduced.

  Therefore, the angle formed by the parallax calculation direction and the direction in which the bright area of the projection pattern extends is preferably 60 degrees or more, and more preferably 80 degrees or more. By projecting a pattern in which a high brightness area (illumination area) extends in a direction closer to the direction perpendicular to the parallax calculation direction, and calculating the parallax amount by the method described in this embodiment, more accurate distance measurement is performed. Can do.

  Further, in this embodiment, the second reference image is set in the vicinity of the first reference image, and the parallax amount is calculated using the correlation value calculated in both reference images. It is desirable that they are close to each other. Therefore, the projection pattern is preferably a periodic pattern in which the luminance distribution is periodically repeated. In this case, if the period pattern is complete, an area shifted by one period may be erroneously detected when calculating the amount of parallax by correlation calculation. Therefore, this erroneous detection can be avoided by limiting the range in which the amount of parallax is searched (the amount of movement for moving the reference image) to be smaller than the cycle of the pattern.

  Further, the projection pattern does not need to be a pattern in which the same luminance distribution is repeated. The projection pattern may be a pattern in which substantially the same luminance distribution is periodically repeated. For example, the width of the bright region in the parallax calculation direction may be different for each line. Moreover, the pattern which has the dispersion | variation in a brightness | luminance in a bright area | region or a dark area | region may be sufficient. By appropriately changing the luminance, it is possible to avoid erroneous detection of a region shifted by one cycle.

  Furthermore, the imaging device that acquires a parallax image may be configured by a stereo camera that includes two or more optical systems and imaging elements corresponding thereto. In this case, the design freedom of the base line length is improved, and the ranging resolution is improved. Further, the distance detection device may be configured with the photographing device and the projection device as separate devices.

  Note that when the control device is configured using a CPU (Central Processing Unit) or the like provided in the imaging device, the device can be downsized.

  Further, the projection apparatus 101 may use an LD (Laser Diode) as a light source. Further, a reflective LCOS (Liquid Crystal On Silicon), a transmissive LCOS, or a DMD (Digital Micromirror Device) may be used as the pattern forming means. When these are used, the cycle of the projection pattern can be changed as appropriate according to the size and distance of the subject, and more accurate distance measurement can be performed according to the situation.

  Furthermore, by making the light from the light source of the projection device 101 white in which the wavelength of the light includes the entire visible light range, the effect of correcting the reflectance can be obtained regardless of the spectral reflectance of the subject 102. Further, from the viewpoint of light utilization efficiency with respect to energy used, the light source of the projection apparatus 101 can be configured with three colors of RGB, and the wavelength of light from the light source can be matched with the color filter transmission band of the imaging apparatus 103.

  When the wavelength of the light source of the projection apparatus 101 is in the IR (Infrared) region, it is possible to perform imaging using the imaging apparatus 103 including the color filter having the corresponding transmission band and light receiving sensitivity and the imaging element 105. In this case, a viewing image can be taken simultaneously using the RGB band. In particular, when the IR wavelength band is between 800 nm and 1100 nm, Si can be used for the photoelectric conversion unit of the image sensor, and by changing the arrangement of the color filters, RGB image and IR measurement can be performed by one image sensor. A distance image can be acquired.

  In the present embodiment, an example in which the distance to the subject 102 is calculated has been described. However, the parallax amount calculation method according to the present embodiment can also be applied to a parallax detection device that detects a parallax amount. In this case, step S305 “distance value calculation” may be omitted in FIG. For example, the parallax detection device can perform processing such as cutting out a subject near the in-focus position from an image based on the parallax amount. If this parallax amount detection device is configured to output the parallax amount as it is without converting it into a distance in the distance detection device 100 in this embodiment, the other configuration may be the same as the distance detection device 100.

  The distance detection method according to the present embodiment may be realized as a computer program in addition to being applied to the distance detection device. Such a computer program causes a computer (processor) to execute a predetermined process in order to calculate a distance or a parallax amount. The program is installed in a computer of an imaging apparatus such as a digital camera equipped with a distance detection apparatus, a parallax detection apparatus, or any one thereof. In this case, the installed program is executed by a computer, thereby realizing the above-described function, and highly accurate distance detection and parallax amount detection can be performed.

(Example 2)
With reference to FIG. 9, a robot apparatus such as a robot apparatus for production work according to the second embodiment will be described. The robot apparatus 900 according to this embodiment includes a gantry 901, a robot arm 902 that is an articulated robot arm, a robot hand 903, a control device 905, and a distance detection device 100. In addition, since the distance detection apparatus 100 according to the present embodiment is the same apparatus as the distance detection apparatus 100 according to the first embodiment, the same reference numerals are given and description thereof is omitted.

  The robot arm 902 is installed on a mount 901, and a robot hand 903 is attached to the tip of the robot arm 902. The robot hand 903 can grip a work (industrial part) 904 and assemble the gripped work 904 on another part.

  The distance detection device 100 is fixed to the tip of the robot arm 902 so that the workpiece 904 is within the imaging range. The method for fixing the distance detection device 100 may be any method and may be configured to be detachable. The distance detection device 100 transmits image information and distance information acquired through shooting to the control device 905.

  The control device 905 controls the robot arm 902, the robot hand 903, the distance detection device 100, and the like. The control device 905 is provided with a calculation unit 951 and a control unit 952.

  The calculation unit 951 estimates the position and orientation of the workpiece 904 and calculates the driving amount of the robot arm 902 and the robot hand 903 based on the distance information and image information sent from the distance detection device 100. The control unit 952 controls the driving of the robot arm 902 and the robot hand 903 based on the timing at which a distance detection command is transmitted to the distance detection device 100 and the calculation result of the calculation unit 951.

  Note that the control device 905 may be configured by an arbitrary computer, and each component of the control device 905 can be configured by a software module executed on an arithmetic device such as a CPU or MPU. Similarly, each component of the control device 905 may be configured by a circuit that performs a specific function such as an ASIC.

  In the production process using the robot apparatus 900, the workpiece 904 placed on the gantry 901 is gripped by the robot hand 903. Therefore, the control unit 952 transmits a movement command to the robot arm 902 via the serial communication path, and controls the robot arm 902 and the robot hand 903 so that the robot hand 903 moves near the workpiece 904. .

  Since the position and orientation of the work 904 is indefinite, the robot apparatus 900 captures the work 904 by the distance detection apparatus 100 and acquires image information and distance information before the work 904 is gripped by the robot hand 903. The calculation unit 951 of the control device 905 calculates the position and orientation information of the workpiece 904 based on the image information and the distance information, and estimates the position and orientation of the workpiece 904. Further, the calculation unit 951 calculates the movement amount of the robot arm 902 based on the calculated position and orientation information of the workpiece 904. The calculation unit 951 transmits the calculated movement amount data of the robot arm 902 to the control unit 952.

  The control unit 952 transmits a command to the robot arm 902 so as to move by the amount of movement received from the calculation unit 951. As a result, the robot arm 902 moves to a position suitable for gripping the workpiece 904. When the movement of the robot arm 902 is completed, the control unit 952 transmits a command to the robot hand 903 to close the hand. The robot hand 903 closes the hand and grips the workpiece 904 in response to a command from the control unit 952.

  The control unit 952 transmits a command to move the robot arm 902 to a predetermined position and to open the robot hand 903 after the movement in order to assemble the workpiece 904 gripped by the robot hand 903 to a main body component (not shown). By repeatedly performing these operations, the assembly operation of the workpiece 904 by the robot apparatus 900 is performed.

  A general work 904 often has no pattern on the surface. Therefore, in the robot apparatus 900, pattern light is projected from the projection apparatus 101 of the distance detection apparatus 100, and imaging is performed in a state where the texture is superimposed on the surface of the workpiece 904. Thereby, highly accurate ranging can be performed.

  Further, the distance detection device 100 sets the first reference image and the second reference image appropriately as in the first embodiment, calculates the amount of parallax from the correlation value calculated using these reference images, By obtaining the distance, the distance information of the workpiece 904 can be acquired with higher accuracy. Thereby, in the robot apparatus 900, the estimation accuracy of the position and orientation of the workpiece 904 is improved, the accuracy of position control of the robot arm 902 and the robot hand 903 is improved, and a more accurate assembly operation can be performed.

  As described above, the robot apparatus 900 according to the present embodiment includes the distance detection apparatus 100, the robot arm 902, the robot hand 903 provided on the robot arm 902, and the control apparatus that controls the robot arm 902 and the robot hand 903. 905. Here, the distance detection apparatus 100 acquires distance information including the distance to the workpiece 904 and image information of the workpiece 904. The control device 905 estimates the position and orientation of the workpiece 904 using the distance information and the image information, and controls the robot arm 902 and the robot hand 903 based on the estimated position and orientation. With such a configuration, the robot apparatus 900 can reduce the parallax amount calculation error, acquire the distance information of the workpiece 904 with higher accuracy, improve the position and orientation estimation accuracy of the workpiece 904, and increase the accuracy of the workpiece 904. Assembly work can be performed.

  In this embodiment, the example in which the distance detection device 100 is fixed to the robot arm 902 is described. However, the distance detection device 100 may be provided at a position away from the robot arm 902. In this case, the distance detection device 100 may be installed so that the workpiece 904 falls within the imaging range. In addition, the arithmetic processing unit 106 may not be provided in the distance detection device 100 and may be provided in the control device 905. Further, the processing performed by the arithmetic processing unit 106 may be performed by the arithmetic unit 951 in the control device 905.

  The distance between the distance detection apparatus 100 and the workpiece 904 changes according to the positions of the robot arm 902 and the robot hand 903. At this time, if distance detection is performed without changing the projection pattern of the projection apparatus 101, the pattern period on the captured image changes according to the distance between the distance detection apparatus 100 and the workpiece 904. For this reason, the optimal positional relationship between the first reference image and the second reference image changes, and the effect of reducing the parallax amount calculation error may be weakened.

  Therefore, the correlation calculation unit 161 may analyze an image acquired by photographing the projection pattern and set the position of the second reference image based on the analysis result. Specifically, the correlation calculation unit 161 analyzes an image set acquired by photographing a projection pattern, and calculates and evaluates a period of variation in pixel values indicating a pattern period on the image. Next, the correlation calculation unit 161 determines a position where the second reference image is set based on the width of the first reference image and the pattern period in the parallax calculation direction. At this time, the correlation calculation unit 161 can determine the position of the second reference image according to the above-described equation (1a) or equation (1b). By such processing, the correlation calculation unit 161 can appropriately set the position of the second reference image according to the distance between the distance detection device 100 and the workpiece 904.

  Further, as the distance between the workpiece 904 and the robot hand 903 is closer, it is required to acquire distance information with higher in-plane resolution and to estimate the position and orientation of the workpiece 904 with higher accuracy. Here, the distance of the workpiece 904 can be roughly calculated from the size of the workpiece 904 on the captured image. In addition, since the distance information of the workpiece 904 by the distance detection device 100 and the control of the robot arm 902 by the control device 905 are sequentially performed in time series, based on the distance information acquired one cycle before these processing, You can also know the distance roughly.

  Therefore, the correlation calculation unit 161 or the arithmetic processing unit 106 makes the size of the first reference image and the second reference image smaller as the distance of the work 904 is closer based on the approximate distance information of the work 904. In addition, the size of these images may be set. In this case, the correlation calculation unit 161 can determine the position of the second reference image from the period of the projection pattern on the image and the size of the first reference image.

  Further, the projection pattern of the projection apparatus 101 may be changed by the control unit 108 so that the cycle of the projection pattern becomes finer as the distance between the workpieces 904 is shorter. Through such processing, the second reference image can be appropriately set according to the distance between the distance detection device 100 and the workpiece 904.

  By appropriately setting the second reference image by these methods, the robot apparatus 900 can reduce the parallax amount calculation error by the distance detection apparatus 100 and calculate the distance of the workpiece 904 with high accuracy. be able to. For this reason, in the robot apparatus 900, the estimation accuracy of the position and orientation of the workpiece 904 is improved, the accuracy of the position control of the robot arm 902 and the robot hand 903 is improved, and more accurate assembly work can be performed.

(Example 3)
With reference to FIGS. 10A and 10B, a distance detection apparatus according to Embodiment 3 will be described. Although the projection apparatus 101 is provided in the distance detection apparatus 100 according to the first embodiment, the configuration of the distance detection apparatus is not limited to this. In the distance detection device 110 according to the present embodiment, the pattern of the subject 112 is analyzed and the distance to the subject 112 is detected without providing the projection device 101. Such a distance detection device 110 is particularly effective when distance detection is performed on a subject 112 whose pattern changes periodically.

  The distance detection device 110 according to the present embodiment is the same as the distance detection device 100 according to the first embodiment except that the projection device 101 is not provided and the determination unit 164 is provided in the arithmetic processing unit 116. It has the composition of. Therefore, constituent elements other than the arithmetic processing unit 116 and the determination unit 164 are denoted by the same reference numerals as those in the first embodiment, and description thereof is omitted. Hereinafter, the difference between the distance detection device 110 according to the present embodiment and the distance detection device 100 according to the first embodiment will be mainly described.

  FIG. 10A shows a schematic configuration example of the distance detection device 110 according to the present embodiment. The imaging apparatus 103 is provided with an imaging optical system 104, an image sensor 105, an arithmetic processing unit 116, and a main body memory 107. The imaging optical system 104, the image sensor 105, and the main body memory 107 have the same configuration as that of the first embodiment.

  In the arithmetic processing unit 116, a determination unit 164 is provided in addition to the correlation calculation unit 161, the parallax calculation unit 162, and the distance calculation unit 163. The correlation calculation unit 161, the parallax calculation unit 162, and the distance calculation unit 163 have the same configuration as in the first embodiment. The determination unit 164 determines whether the acquired captured image of the subject 112 has periodicity, and transmits the determination result to the correlation calculation unit 161. The correlation calculation unit 161 performs a correlation calculation based on the determination result received from the determination unit 164.

  With reference to FIG.10 (b), the flow of the distance detection method which concerns on a present Example is demonstrated. When the distance detection method is started, the process proceeds to step S1001.

  In step S1001 “image acquisition”, the subject 112 is photographed by the photographing device 103, an image set including an A image and a B image having parallax is generated and obtained, and the obtained image set is stored in the main body memory 107.

  Processing of each subsequent step is performed by the arithmetic processing unit 116. In step S1002 “correlation calculation 1”, the correlation calculation unit 161 sets a first reference image and calculates a first correlation value.

  Next, in step S1003 “periodicity determination processing”, the determination unit 164 determines whether the pixel value of the captured image has periodicity in the parallax calculation direction. For example, the periodicity is determined by extracting a partial area image of the A image and performing a correlation operation between the extracted image and another partial area image on the A image. When the region where the correlation is high appears periodically, the determination unit 164 can determine that the captured image has periodicity. The determination unit 164 uses the first correlation value obtained by the correlation calculation unit 161 to periodically increase the correlation value between the first reference image and the reference image with respect to the movement amount of the reference image. It may be determined whether or not. In this case, the determination unit 164 can determine that the captured image has periodicity if the correlation value periodically increases. If it is determined in step S1003 that the captured image has periodicity, the process proceeds to step S1004.

  In step S1004 “correlation calculation 2”, the correlation calculation unit 161 sets a second reference image and calculates a second correlation value. The method for setting the second reference image and the method for calculating the second correlation value are the same as in the first embodiment.

  In step S1005 “calculation of parallax amount”, the parallax calculation unit 162 calculates the parallax amount from the first correlation value and the second correlation value obtained in steps S1002 and S1004. The method for calculating the amount of parallax is the same as in the first embodiment. In this embodiment, since pattern light is not projected, the image edge becomes the pattern edge of the subject 112.

  In step S1006 “distance value calculation”, the distance calculation unit 163 converts the parallax amount calculated in step S1005 into a defocus amount or a subject distance by a known method, as in the first embodiment.

  On the other hand, if it is determined in step S1003 that the captured image does not have periodicity, the process proceeds to step S1007 “parallax amount calculation 2”. In step S1007, the parallax calculation unit 162 calculates the amount of parallax from the first correlation value calculated in step S1002. A method similar to that described above can be used as a method of calculating the amount of parallax from the first correlation value. After calculating the parallax amount in step S1007, the distance calculation unit 163 calculates a distance value based on the calculated parallax amount in step S1006.

  By such processing, the distance detection device 110 according to the present embodiment can reduce the parallax amount calculation error for the subject 112 whose pattern changes periodically for the same reason as in the first embodiment, and can achieve high accuracy. Distance detection can be performed.

  As described above, the distance detection device 110 according to the present embodiment includes the determination unit 164 that determines whether any image in the image set acquired by the imaging device 103 has periodicity in the parallax calculation direction. When it is determined that any image in the image set has periodicity in the parallax calculation direction, the correlation calculation unit 161 calculates the first correlation value and the second correlation value, and the parallax calculation unit 162 The amount of parallax is calculated using the first correlation value and the second correlation value. As a result, the distance detection device 110 can reduce parallax amount calculation errors and perform highly accurate distance measurement based on the periodically changing pattern of the subject 112 even when pattern light is not projected. be able to.

  Note that the distance detection apparatus according to the present embodiment may be applied to the robot apparatus as in the second embodiment. Even in this case, the distance of the workpiece can be calculated with high accuracy. Therefore, by improving the estimation accuracy of the position and orientation of the workpiece, it is possible to improve the accuracy of position control of the robot arm and robot hand, and to perform more accurate assembly work.

  Further, the parallax detection method in the distance detection method according to the present embodiment may be applied to a parallax detection device for outputting the detected amount of parallax, as in the first embodiment. Also in this case, it is possible to reduce the detection error of the parallax amount for the subject whose pattern changes periodically.

Example 4
In Example 1, reference images are set at different locations in the parallax calculation direction in an image obtained by photographing light of one projection pattern, and a correlation value with a reference image is calculated for each reference image to detect a distance. On the other hand, in the distance detection device according to the fourth embodiment, the reference image is set at the same location in the two image sets obtained by photographing the light of the two projection patterns shifted in the parallax calculation direction, and the reference image is related to each reference image. The correlation value is calculated and the distance is detected.

  Hereinafter, the distance detection apparatus according to the present embodiment will be described with reference to FIGS. 11 (a) to 15 (b). Since the configuration of the distance detection device according to the present embodiment is the same as the configuration of the distance detection device 100 according to the first embodiment, the same reference numerals as those of the first embodiment are used and description thereof is omitted. Hereinafter, the difference between the distance detection device according to the present embodiment and the distance detection device 100 according to the first embodiment will be mainly described.

  FIG. 11A and FIG. 11B show pattern light to be projected in this embodiment. The pattern light 1101 that is the first pattern light has a period 1103 in which a high luminance region and a low luminance region are alternately repeated in the x-axis direction. The pattern light 1101 has a line pattern in which each luminance region extends in the y-axis direction. The pattern light 1102 as the second pattern light has a periodic luminance distribution in the same x-axis direction as the pattern light 1101 and a line pattern in which each luminance region extends in the y-axis direction. The line pattern of the pattern light 1102 is shifted in position in the x-axis direction from the line pattern of the pattern light 1101.

  Next, a ranging calculation flow according to the present embodiment will be described with reference to FIGS. FIG. 12A is a flowchart of distance detection processing according to the present embodiment, and FIGS. 12B and 12C are diagrams for explaining the correlation calculation performed by the correlation calculation unit 161. When the distance detection process according to the present embodiment is started, the process proceeds to step S1201.

  In step S1201 “pattern light projection image acquisition 1”, the patterning light 1101 is projected from the projection apparatus 101 onto the subject 102, and the photographing is performed using the photographing apparatus 103, and the photographed image is stored in the main body memory 107. Since the pattern light projection method is the same as that of the first embodiment, description thereof is omitted.

  In step S1202 “pattern light projection image acquisition 2”, photographing is performed while the pattern light 1102 is projected onto the subject 102, and the photographed image is stored in the main body memory 107. Since the pattern light projection method is the same as that of the first embodiment, the description thereof is omitted.

  The processing from step S1203 to step S1206 is performed by the arithmetic processing unit 106. Here, FIGS. 12B and 12C are diagrams for explaining the positional relationship between the standard image and the reference image set in steps S1203 and S1204, respectively. FIG. 12B shows the A image 1210A and the B image 1210B acquired in step S1201, and FIG. 12C shows the A image 1220A and the B image 1220B acquired in step S1202. Hereinafter, a pixel for which distance calculation is performed at the same position on the A image 1210A and the A image 1220A will be described as a target pixel 1230.

  In step S1203 “correlation calculation 1”, the correlation calculation unit 161 calculates a first correlation value using the image set acquired in step S1201. Specifically, the correlation calculation unit 161 extracts a partial region including the target pixel 1230 and its neighboring pixels on the A image 1210A, and sets it as the first reference image 1211. Next, the correlation calculation unit 161 extracts a region having the same area (image size) as that of the first reference image 1211 on the B image 1210B and sets it as a reference image 1212. Thereafter, the correlation calculation unit 161 moves the position where the reference image 1212 is extracted on the B image 1210B in the same x-axis direction as the pupil division direction, and the reference image 1212 and the first reference image at each movement amount (each position). Correlation value with 1211 is calculated. Thereby, the correlation calculation unit 161 calculates a first correlation value including a data string of correlation values corresponding to each movement amount.

  The correlation value calculation method may be the same as that in step S302 according to the first embodiment. In addition, the correlation calculation unit 161 can set the reference image 1212 as an image having the same vertical and horizontal dimensions as the vertical and horizontal dimensions of the first standard image 1211.

  Next, in step S1204 “correlation calculation 2”, the correlation calculation unit 161 calculates a second correlation value using the image set acquired in step S1202. Specifically, the correlation calculation unit 161 extracts a partial region including the target pixel 1230 and its neighboring pixels on the A image 1220A, and sets it as the second reference image 1221. Next, the correlation calculation unit 161 extracts a region having the same area as that of the second reference image 1221 on the B image 1220B and sets it as the reference image 1222. Thereafter, as in step S1203, the correlation calculation unit 161 moves the position of the reference image 1222 in the parallax calculation direction, and calculates a correlation value with the second reference image 1221 so that the correlation corresponding to each movement amount is obtained. A second correlation value composed of a data string of values is calculated. The setting conditions for the reference images 1212 and 1222 may be the same as those in the first embodiment.

  In step S1205 “parallax amount calculation”, the parallax calculation unit 162 uses the first correlation value and the second correlation value obtained in steps S1203 and S1204 in the same manner as in step S304 according to the first embodiment. Is calculated. More specifically, the parallax calculation unit 162 adds or averages the correlation values of the movement amounts of the first correlation value and the second correlation value, respectively, calculates a third correlation value, The amount of parallax is calculated based on the correlation value. Also, in step S1206 “distance value calculation”, the distance calculation unit 163 converts the parallax amount into the defocus amount or the subject distance by a known method, as in step S305 according to the first embodiment, and the distance to the subject 102. Is calculated.

  As described above, in the distance detection method according to this embodiment, the first reference image 1211 and the second reference set are respectively obtained for the first image set and the second image set having different line patterns in the pupil division direction (parallax calculation direction). An image 1221 is set. Thereafter, the correlation value between the first reference image 1211 and the reference image 1212 set for the first reference image 1211, and the reference set for the second reference image 1221 and the second reference image 1221. A correlation value with the image 1222 is calculated, and a parallax amount is calculated from both correlation values. According to this process, the first correlation value and the second correlation value are calculated from each image set, and the parallax amount is calculated from both correlation values, so that the brightness distribution of the projection pattern and the position of the reference image are obtained. A related measurement error can be reduced, and highly accurate distance measurement can be performed.

  Here, an example of the processing result of the distance detection method according to the present embodiment will be described with reference to FIG. FIG. 13 shows the parallax at each position on the flat plate, which is projected by using the projection device 101 to project pattern light onto a flat plate arranged in parallel with the photographing device 103 at a known distance. It is a graph which shows the result of having calculated quantity. The horizontal axis in FIG. 13 represents the movement amount (pixel position), and the vertical axis represents the parallax amount calculation error. In FIG. 13, as a conventional process, a parallax amount calculation error 1301 calculated from an image set obtained by projecting only the first pattern light is indicated by a broken line, and the parallax amount calculated by the method according to the present embodiment Is calculated by a solid line. It can be seen that the error amount is close to 0 and the parallax amount calculation error is reduced by the method according to the present embodiment as compared with the conventional case. Therefore, according to the method according to the present embodiment, the distance can be measured with high accuracy.

  Next, with reference to FIGS. 14A to 14F, the reason why an error occurs in the calculation of the parallax amount in the conventional process and the calculation error of the parallax amount is reduced in the process according to the present embodiment will be described. In the following description, it is assumed that the A image and the B image have the same shading and have no parallax. FIGS. 14A to 14C are diagrams for explaining the reason for the error. The reason for the occurrence of the error is the same as that described in the first embodiment, and therefore the description is simplified.

  FIG. 14A is a diagram showing a positional relationship between an A image 1401 having a line pattern in which bright areas and dark areas alternately appear, a reference image 1402 and a reference image 1403. The reference image 1402 includes an image edge 1404 (boundary portion) in which the bright area and the dark area of the A image 1401 are switched inside the reference image 1402. FIG. 14B shows each correlation value calculated by moving the reference image set with respect to the standard image 1402 and performing a correlation calculation between the standard image 1402 and the reference image.

  Correlation values C0, Cp, and Cm are correlation values when the position of the reference image is moved by 0, +1, and −1 pixel, respectively. In this case, as described in the first embodiment, since the absolute value of the movement amount of the reference image corresponding to the correlation values Cp and Cm is the same, the difference between images caused by the image edge 1404 on the line pattern is also the same. It becomes quantity. Therefore, the correlation value Cp and the correlation value Cm are the same value. The correlation value is interpolated to obtain a correlation curve 1410, and the amount of parallax 1411 is obtained by calculating the movement amount (parallax amount) that is the highest correlation value, and the correct value (parallax amount 0) is obtained.

  On the other hand, in the reference image 1403, the image edge 1404 and the right end of the reference image 1403 overlap. FIG. 14C shows each correlation value calculated by moving the reference image set with respect to the standard image 1403 and performing correlation calculation between the standard image 1403 and the reference image. As described in the first embodiment, the correlation value in this case is asymmetrical on the + side and the − side of the movement amount of the reference image, and the parallax amount 1413 obtained from the correlation curve 1412 obtained by interpolating this is a correct value (parallax amount). 0) and an error occurs. This is a calculation error of the amount of parallax generated in relation to the luminance distribution of the projection pattern and the position of the reference image.

  Next, as in this embodiment, the first correlation value and the second correlation value are obtained from the respective image sets obtained by projecting the first pattern light and the second pattern that are displaced in the parallax calculation direction. The reason why the aforementioned error is reduced by calculating and calculating the parallax amount from both correlation values will be described. FIG. 14D is a diagram showing the positions of the A image 1401 and the first reference image 1403 acquired by projecting the first pattern light. FIG. 14E is a diagram showing the positions of the A image 1405 and the second reference image 1406 acquired by projecting the second pattern light. FIG. 14F shows the correlation value calculated using each reference image.

  As the first reference image 1403, a reference image in which the image edge 1404 and the right end of the reference image overlap is assumed as described above. At this time, the first correlation values Cm1, C01, Cp1 calculated from the first reference image 1403 are the same as the correlation values Cm, C0, Cp shown in FIG.

  Next, the second pattern light is projected so that the left edge of the second reference image 1406 set at the same position as the first reference image 1403 and the image edge 1407 overlap. At this time, the positional relationship between the second reference image 1406 and the image edge 1407 is a relationship obtained by inverting the positional relationship between the first reference image 1403 and the image edge 1404. Therefore, as shown in FIG. 14F, the second correlation values Cm2, C02, Cp2 obtained using the second reference image 1406 are obtained by inverting the first correlation values Cm1, C01, Cp1. It becomes.

  When these correlation values are added and averaged for each position (movement amount) of the reference image, the correlation values become the third correlation values Cm3, C03, and Cp3. The third correlation value is symmetrical because the asymmetry on the + side and − side of the movement amount of the reference image is canceled. In this case, the parallax amount 1415 obtained from the correlation curve 1414 obtained by interpolating the correlation value is a correct parallax amount (parallax amount 0), and an appropriate parallax amount can be calculated. In the process according to the present embodiment, the calculation error of the parallax amount can be reduced based on such a principle, and the distance can be measured with high accuracy based on the appropriately calculated parallax amount.

  In the present embodiment, the calculation error of the parallax amount can be further reduced by making the positions of the first pattern light and the second pattern different by an appropriate amount in the parallax calculation direction. FIGS. 15A and 15B are diagrams for explaining appropriate positions of the first pattern light and the second pattern light for each reference image.

  The first reference image 1403 and the second reference image 1406 are reference images set at the same positions on the images 1401 and 1405 obtained using the first pattern light and the second pattern light, respectively. The right edge of the first reference image 1403 and the image edge 1404 due to the first pattern light overlap.

  Here, a case is considered where an error occurring in the correlation value when the first reference image 1403 is used is canceled with the correlation value calculated using the second reference image 1406. In this case, since the error of the correlation value due to the image edge 1404 at the right end of the first reference image 1403 is canceled, the error of the correlation value due to the image edge is generated with respect to the left end of the second reference image 1406. In addition, the second pattern light may be set. The setting of the pattern light may be performed by the arithmetic processing unit 106 or the control unit 108.

  FIG. 15A shows a case where the left end of the second reference image 1406 overlaps with the image edge 1407 by the second pattern light. In this case, an error occurring in the first correlation value calculated using the first reference image 1403 can be canceled with the second correlation value calculated using the second reference image 1406.

  FIG. 15B shows a case where the left end of the second reference image 1406 and another image edge 1508 due to the second pattern light overlap. Also in this case, an error occurring in the first correlation value calculated using the first reference image 1403 can be canceled with the second correlation value calculated using the second reference image 1406.

  If the position difference in the parallax calculation direction (x-axis direction) between the first pattern light and the second pattern light in these cases is obtained, it can be expressed by the above-described equation (1a) or equation (1b). . Note that W, P, H, and n in the equation can be the same parameters as those described in the first embodiment.

  In the distance detection method according to the present embodiment, the positions of the first pattern light and the second pattern light are different from each other in the parallax calculation direction by the amount indicated by the formula (1a) or the formula (1b). In addition, the parallax amount calculation error can be reduced, and highly accurate distance measurement can be performed. The position of the pattern light can be changed by the control unit 108 by controlling / changing the position of the pattern forming unit such as the pattern mask with respect to the light source in the projection apparatus 101. Further, according to the control of the control unit 108, the position of the pattern light is adjusted by other known arbitrary methods such as controlling the spatial light modulator in the projection apparatus 101 or switching between a plurality of pattern forming means. It may be changed.

  As described above, the projection apparatus 101 according to the present embodiment projects the first pattern light and the second pattern light having patterns whose positions are shifted from each other in the parallax calculation direction. The correlation calculation unit 161 calculates a first correlation value based on the first image set obtained by projecting the first pattern light, and the second image set obtained by projecting the second pattern light. The second correlation value is calculated based on The correlation calculation unit 161 uses the reference image set at the same position on one image of the first image set and one image of the second image set, and uses the first correlation value and the second correlation value. Value. The parallax calculation unit 162 calculates the amount of parallax using the first correlation value and the second correlation value.

  Here, the first pattern light and the second pattern light are periodic lights in which the high luminance region and the low luminance region are alternately repeated in the parallax calculation direction, and the second direction is perpendicular to the parallax calculation direction. Have a line pattern in which a high luminance region and a low luminance region extend. In particular, in the present embodiment, the first pattern light and the second pattern light are light obtained by shifting pattern lights having the same luminance distribution in the parallax calculation direction with respect to each other.

  Further, the projection apparatus 101 projects the first pattern light and the second pattern light whose positions are set according to the formula (1a) or the formula (1b). More specifically, the arithmetic processing unit 106 sets the first pattern in the visual calculation direction by the width of the reference image in the parallax calculation direction or the difference between the width and the period of the first pattern light in the captured image. The positions of the light and the second pattern light are shifted and set. Alternatively, the arithmetic processing unit 106 may match the first pattern light and the second pattern light in the visual calculation direction by the difference between the width of the parallax calculation direction of the reference image and the width of the high brightness area of the first pattern light in the captured image. The position of the pattern light is shifted and set. Alternatively, the arithmetic processing unit 106 sets the positions of the light of the first pattern light and the second pattern in the parallax calculation direction by an amount obtained by subtracting the period of the first pattern light in the captured image from the difference. . The projection apparatus 101 outputs the first and second pattern lights based on the set positions so that the positions of the pattern of the first pattern light and the pattern of the second pattern light in the parallax calculation direction are different. Project.

  With such a configuration, the distance detection apparatus according to the present embodiment can reduce a parallax amount calculation error in distance detection using a captured image on which pattern light is projected. Therefore, the distance detection device can acquire highly accurate distance information of the subject 102 based on the amount of parallax with reduced calculation error.

  Note that the method of calculating the amount of parallax in the present embodiment is not limited to the method described in step S304 as in the first embodiment. For example, the parallax calculation unit 162 calculates the first parallax amount from the first correlation value, calculates the second parallax amount from the second correlation value, and adds and averages these parallax amounts. An appropriate amount of parallax may be calculated. Also in this case, it is possible to reduce parallax amount calculation errors and perform highly accurate distance measurement. In this case, the correlation calculation unit 161 does not have to obtain the third correlation value using the first and second correlation values.

  In this embodiment, in steps S1201 and S1202, images based on the first pattern light and the second pattern light are acquired, and then in steps S1203 and S1204, correlation is performed using the images based on the respective pattern lights. An example of calculating the value is shown. However, the timing for calculating the correlation value is not limited to this. For example, an image associated with the first pattern light may be acquired to calculate the first correlation value, and then an image associated with the second pattern light may be acquired to calculate the second correlation value. In this case, the same effect as described above can be obtained.

  Moreover, it is preferable that there is an image edge based on each pattern light in the vicinity of both ends of the reference image regardless of the position where the reference image is set on the captured image. Therefore, like the first embodiment, the projection pattern is preferably a periodic pattern in which the luminance distribution is periodically repeated in the x-axis direction. In this case, if the period pattern is complete, an area shifted by one period may be erroneously detected when calculating the amount of parallax by correlation calculation. Therefore, this erroneous detection can be avoided by limiting the range in which the amount of parallax is searched (the amount of movement for moving the reference image) to be smaller than the cycle of the pattern.

  Further, the projection pattern does not need to be a pattern in which the same luminance distribution is repeated. The projection pattern may be a pattern in which substantially the same luminance distribution is periodically repeated. For example, the width of the bright region in the parallax calculation direction may be different for each line. Moreover, the pattern which has the dispersion | variation in a brightness | luminance in a bright area | region or a dark area | region may be sufficient. By appropriately changing the luminance, it is possible to avoid erroneously detecting a region shifted by one period, and to achieve the effect of reducing the above-described parallax amount calculation error.

  In addition, the distance detection apparatus according to the present embodiment can be applied to a production work robot apparatus as in the second embodiment. An example of this case will be briefly described with reference to FIG. Note that the configuration of the robot apparatus in this case may be the same as the configuration of the robot apparatus according to the second embodiment, and thus the description thereof is omitted using the same reference numerals as those in FIG.

  In such a robot apparatus, as described in the present embodiment, the first pattern light and the second pattern light shifted in the parallax calculation direction are projected onto the work 904, and an image set based on each pattern light is generated. Take an image. Thereafter, the first correlation value and the second correlation value are calculated from each image group, and the distance information of the workpiece 904 can be obtained with higher accuracy by obtaining the distance. Thereby, in the robot apparatus 900, the estimation accuracy of the position and orientation of the workpiece 904 is improved, the accuracy of position control of the robot arm 902 and the robot hand 903 is improved, and a more accurate assembly operation can be performed.

  Note that the distance between the distance detection device and the workpiece 904 changes according to the positions of the robot arm 902 and the robot hand 903. At this time, if distance measurement is performed without changing the projection pattern of the projection apparatus 101, the pattern period on the captured image changes according to the distance. Therefore, the optimal positional relationship between the first pattern light and the second pattern light may change, and the effect of reducing the parallax amount calculation error may be weakened.

  Therefore, the arithmetic processing unit 106 may analyze the image acquired by projecting the first pattern light, and determine the amount of positional deviation of the line pattern in the second pattern light based on the analysis result. Specifically, the arithmetic processing unit 106 analyzes an image acquired by projecting the first pattern light, and changes the pixel value indicating the period of the first pattern light in the parallax calculation direction on the acquired image. Calculate and evaluate the period. Next, the arithmetic processing unit 106 determines the amount of positional deviation in the parallax calculation direction of the second pattern light based on the width (size) of the first reference image in the parallax calculation direction and the pattern period.

  In addition, the arithmetic processing unit 106 analyzes the positional deviation amounts of the first pattern light and the second pattern light from the images obtained by projecting the respective pattern lights, and accordingly, the first and second patterns in the parallax calculation direction are analyzed. The width of the second reference image may be determined. Specifically, the arithmetic processing unit 106 uses the first pattern light in the parallax calculation direction based on at least one image of each image group obtained by projecting the first pattern light and the second pattern light, respectively. And the amount of positional deviation of the second pattern light is calculated. Next, the arithmetic processing unit 106 determines the widths of the first and second reference images based on the amount of positional deviation between the first pattern light and the second pattern light.

  Through the processing as described above, the arithmetic processing unit 106 detects the positional deviation amount between the first pattern light and the second pattern light and the first and second reference images according to the distance between the distance detection device and the workpiece 904. Can be set appropriately. Note that when determining the amount of misalignment and the width of the reference image, these can be determined according to the above-described equation (1a) or equation (1b). Further, the processing by the arithmetic processing unit 106 may be performed by the control unit 108 or may be performed by the control device 905.

  Further, as the distance between the workpiece 904 and the robot hand 903 is closer, it is required to acquire distance information with higher in-plane resolution and to estimate the position and orientation of the workpiece 904 with higher accuracy. Here, the distance of the workpiece 904 can be roughly calculated from the size of the workpiece 904 on the captured image. In addition, since acquisition of distance information of the workpiece 904 by the distance detection device and control of the robot arm 902 by the control device 905 are sequentially performed in time series, the distance of the workpiece 904 is based on the distance information acquired one cycle before these processes. Can also be known roughly.

  Therefore, the arithmetic processing unit 106 may set the size of the first reference image to be smaller as the distance of the workpiece 904 is closer, based on the schematic distance information of the workpiece 904. In this case, the arithmetic processing unit 106 can determine the positional deviation amount of the second pattern light from the period of the first pattern light on the image and the size of the first reference image.

  Further, the arithmetic processing unit 106 may change the projection pattern so that the cycle of the projection pattern becomes finer as the distance is shorter. By such processing, the arithmetic processing unit 106 can appropriately set the positional deviation amount between the first pattern light and the second pattern light according to the distance between the distance detection device and the workpiece 904. Instead of the arithmetic processing unit 106, the control unit 108 or the control device 905 may analyze the image and determine the amount of pattern light misregistration. Further, the position of the pattern light may be controlled by the control unit 108.

  By appropriately setting the first pattern light, the second pattern light, and the reference image by these methods, the robot apparatus 900 can reduce the parallax amount calculation error by the distance detection apparatus, and the workpiece 904 can be reduced. Can be calculated with high accuracy. For this reason, in the robot apparatus 900, the estimation accuracy of the position and orientation of the workpiece 904 is improved, the accuracy of the position control of the robot arm 902 and the robot hand 903 is improved, and more accurate assembly work can be performed.

(Example 5)
In the distance detection device according to the fourth embodiment, a reference image is set at the same place in two images obtained by photographing light of two projection patterns shifted in the parallax calculation direction, and a correlation value with the reference image is calculated for each reference image. The distance was detected. On the other hand, in the distance detection device according to the fifth embodiment, a plurality of pattern lights having different wavelength bands are projected, and an image set using each pattern light is acquired to detect the distance.

  Hereinafter, the distance detecting device according to the present embodiment will be described with reference to FIGS. 16 (a) to 17 (c). FIG. 16A shows a schematic configuration of a distance detection apparatus 1600 according to the present embodiment. In the distance detection device 1600 according to the present embodiment, the same reference numerals are used for the same configurations as those of the distance detection device 100 according to the first embodiment, and description thereof is omitted. Hereinafter, the distance detection apparatus 1600 according to the present embodiment will be described focusing on the difference from the distance detection apparatus according to the fourth embodiment.

  The distance detection device 1600 according to the present embodiment is provided with a projection device 1610 and an imaging device 1603. The imaging device 1603 is provided with an imaging optical system 104, an image sensor 1620, an arithmetic processing unit 106, and a main body memory 107. Further, the projection device 1610 and the imaging device 1603 are connected to the control unit 108, and the control unit 108 performs control such as synchronization of the projection device 1610 and the imaging device 1603.

  The projection device 1610 is configured to project the pattern light 1611 and the pattern light 1612. FIGS. 16B and 16C show pattern lights 1611 and 1612 which are examples of pattern light projected in the present embodiment, respectively. Pattern light 1611 which is the first pattern light has a line pattern, and a high luminance region and a low luminance region are alternately repeated in the x-axis direction, and has a period 1613. The pattern light 1612 that is the second pattern light has a line pattern having a periodic luminance distribution in the same x-axis direction as the pattern light 1611. Further, the line pattern of the pattern light 1612 is shifted in position in the x-axis direction from the line pattern of the pattern light 1611.

  Here, the pattern light 1611 and the pattern light 1612 according to the present embodiment are lights having different wavelength bands. Therefore, the projection apparatus 1610 is provided with two projection optical systems including, for example, a light source, an imaging optical system, and a pattern forming unit. The two projection optical systems include a light source having a different wavelength band and a pattern mask having a different pattern. Note that a projection optical system including the same pattern mask and configured to change the position of the pattern mask may be used.

  FIG. 16D shows a part of the image sensor 1620. A plurality of pixels 1621, 1622, and 1623 are arranged on the image sensor 1620, and color filters having different transmission wavelength bands are arranged on these pixels. In this embodiment, the color filter on the pixel 1621 is configured to transmit the wavelength band of the pattern light 1611 and the color filter on the pixel 1622 is configured to transmit the light in the wavelength band of the pattern light 1612. By using such an image sensor, a pixel having a color filter corresponding to the wavelength of each pattern light can receive the pattern light of the corresponding wavelength, and an image set based on each pattern light is separated. And can be obtained. Note that the arrangement of the plurality of pixels 1621, the plurality of pixels 1622, and the plurality of pixels 1623 in the imaging element 1620 is not limited to the arrangement illustrated in FIG. 16D, and may be arbitrarily changed according to a desired configuration. Good.

  Next, a ranging calculation flow according to the present embodiment will be described with reference to FIGS. FIG. 17A is a flowchart of the distance detection processing according to the present embodiment, and FIGS. 17B and 17C are diagrams for explaining the correlation calculation performed by the correlation calculation unit 161. When the distance detection process according to the present embodiment is started, the process proceeds to step S1701.

  In step S1701 “acquire pattern light projection image”, the patterning light 1611 and 1612 are projected onto the subject 102 from the projection device 1610, and photographing is performed using the photographing device 1603, and the photographed image is stored in the main body memory 107. . The image capturing apparatus 1603 can perform image capturing in this state to acquire the image set 1710 using the pixel 1621 and acquire the image set 1720 using the pixel 1622 and store the image set 1720 in the main body memory 107. Note that the pattern light projection method is the same as in the first and fourth embodiments, and a description thereof will be omitted. In the following description, it is assumed that the image set 1710 includes an A image 1710A and a B image 1710B, and the image set 1720 includes an A image 1720A and a B image 1720B.

  The processing from step S1702 to step S1705 is performed by the arithmetic processing unit 106. Here, FIGS. 17B and 17C are diagrams illustrating the positional relationship between the standard image and the reference image set in steps S1702 and S1703. FIG. 17B shows an A image 1710A and a B image 1710B acquired using the pixel 1621 in step S1701, and FIG. 17C shows an A image 1720A and a B image 1720B acquired using the pixel 1622. It is shown. Hereinafter, a pixel for which distance calculation is performed at the same position on the A image 1710A and the A image 1720A will be described as a target pixel 1730.

  In step S1702 “correlation calculation 1”, the correlation calculation unit 161 calculates a correlation value using the image set 1710 acquired in step S1701. Specifically, the correlation calculation unit 161 extracts a partial region including the target pixel 1730 and its neighboring pixels on the A image 1710A, and sets it as the first reference image 1711. Next, the correlation calculation unit 161 extracts a region having the same area as that of the first reference image 1711 on the B image 1710B and sets it as a reference image 1712. After that, the correlation calculation unit 161 calculates the first correlation value using the first standard image 1711 and the reference image 1712 as in step S1203 according to the fourth embodiment.

  The correlation value calculation method may be the same as that in step S302 according to the first embodiment. In addition, the correlation calculation unit 161 can set the reference image 1712 as an image having the same vertical and horizontal dimensions as the vertical and horizontal dimensions of the first standard image 1711.

  Next, in step S1703 “correlation calculation 2”, the correlation calculation unit 161 calculates a correlation value using the image set 1720 acquired in step S1701. Specifically, the correlation calculation unit 161 extracts a partial region including the target pixel 1730 and its neighboring pixels on the A image 1720A and sets it as the second reference image 1721. Next, the correlation calculation unit 161 extracts a region having the same area as the second reference image 1721 on the B image 1720B and sets it as a reference image 1722. Thereafter, the correlation calculation unit 161 calculates the second correlation value using the second reference image 1721 and the reference image 1722 as in step S1204 according to the fourth embodiment. Note that the correlation calculation unit 161 can set the second reference image 1721 at the same position as the first reference image 1711 as in the fourth embodiment.

  In step S1704 “calculation of parallax”, the parallax calculation unit 162 uses the first correlation value and the second correlation value obtained in steps S1703 and S1704 as in step S304 according to the first embodiment. Is calculated. More specifically, the parallax calculation unit 162 adds or averages the correlation values of the movement amounts of the first correlation value and the second correlation value, respectively, calculates a third correlation value, The amount of parallax is calculated based on the correlation value. Also, in step S1705 “distance value calculation”, the distance calculation unit 163 converts the parallax amount into the defocus amount or the subject distance by a known method, as in step S305 according to the first embodiment, and the distance to the subject 102. Is calculated.

  As described above, in the distance detection method according to the present embodiment, the projection device 1610 projects the first pattern light and the second pattern light having the line patterns shifted in the parallax calculation direction and having different wavelength bands. Also, the imaging device 1603 acquires the image set based on the first pattern light and the image set based on the second pattern light separately. The correlation calculation unit 161 sets a first reference image 1711 and a second reference image 1721 for each set of images. After that, the correlation calculation unit 161 sets the first correlation value with the reference image 1712 set corresponding to the first reference image 1711 and the reference image 1722 set corresponding to the second reference image 1721. The second correlation value is calculated. The parallax calculation unit 162 calculates the amount of parallax from both correlation values. According to this process, the first correlation value and the second correlation value are calculated from each image set, and the parallax amount is calculated from both correlation values, so that the brightness distribution of the projection pattern and the position of the reference image are obtained. A related measurement error can be reduced, and highly accurate distance measurement can be performed.

  In particular, in the present embodiment, the first pattern light and the second pattern light having different wavelength bands are simultaneously projected, and an image set based on each pattern light is acquired by the image sensor 1620. For this reason, distance measurement can be performed by one pattern projection, and high-speed measurement can be performed.

  Note that the wavelength bands of the first pattern light and the second pattern light are preferably separated from each other. For example, the first pattern light can be a blue or ultraviolet wavelength band, and the second pattern light can be a red or infrared wavelength band. By separating the wavelength bands of the two pattern lights, it becomes easy to separate and acquire an image set by the two pattern lights with a commonly used image sensor.

  Further, in this embodiment, by arranging a color filter that does not transmit light in the wavelength bands of the pattern light 1611 and the pattern light 1612 on the pixel 1623, it is possible to acquire a subject image without a pattern light pattern. By using such an image, it is possible to specify which position on the subject 102 the distance information obtained by the above-described method is measured. These image information and distance information can be used to detect the position and orientation of the subject 102. It can also be used to determine which of the plurality of types of objects the photographed subject 102 is. Note that these processes such as detection of the position and orientation of the subject 102 using the image information and the distance information may be performed by the arithmetic processing unit 106 and the control unit 108.

(Example 6)
In Example 1, reference images are set at different locations in the parallax calculation direction in an image obtained by photographing light of one projection pattern, and a correlation value with a reference image is calculated for each reference image to detect a distance. On the other hand, in the distance detection device according to the sixth embodiment, a reference image is set in an image obtained by capturing light of a projection pattern including two sub patterns shifted in the parallax calculation direction, and a correlation value between the reference image and the reference image is set. To detect the distance.

  Hereinafter, the distance detection apparatus according to the present embodiment will be described with reference to FIGS. Since the configuration of the distance detection device according to the present embodiment is the same as the configuration of the distance detection device 100 according to the first embodiment, the same reference numerals as those of the first embodiment are used and description thereof is omitted. Hereinafter, the difference between the distance detection device according to the present embodiment and the distance detection device 100 according to the first embodiment will be mainly described.

  An example of pattern light to be projected in this embodiment is shown in FIG. The pattern light 1800 includes a pattern light 1801 (first sub pattern light) and a pattern light 1802 (second sub pattern light). Both pattern lights 1801 and 1802 have the same luminance distribution in the x-axis direction (first direction), and their positions in the x-axis direction are shifted from each other. Further, both pattern lights 1801 and 1802 are located at different positions in the y-axis direction (second direction). In this embodiment, both pattern lights 1801 and 1802 are located alternately in the y-axis direction. . The luminance distribution in the x-axis direction of each pattern light has a cycle 1803 in which a high luminance region and a low luminance region are alternately repeated. Both pattern lights 1801 and 1802 have the same length 1804 in the y-axis direction.

  Next, with reference to FIGS. 19A to 19C, the ranging calculation flow according to the present embodiment will be described. FIG. 19A is a flowchart of distance detection processing according to the present embodiment, and FIGS. 19B and 19C are diagrams for explaining the correlation calculation performed by the correlation calculation unit 161. When the distance detection process according to the present embodiment is started, the process proceeds to step S1901.

  In step S1901, “acquire pattern light projection image”, the patterning light 1800 is projected from the projection apparatus 101 onto the subject 102, and the photographing is performed using the photographing apparatus 103, and the photographed image is stored in the main body memory 107. Since the pattern light projection method is the same as that of the first embodiment, description thereof is omitted.

  The processing from step S1902 to step S1904 is performed by the arithmetic processing unit 106. Here, FIGS. 19B and 19C are diagrams for explaining the positional relationship between the standard image and the reference image set in step S1902. FIG. 19B shows an A image 1910A and a B image 1910B acquired in step S1901. Hereinafter, a pixel for which distance calculation is performed on the A image 1910A will be described as a target pixel 1920.

  In step S1902 “correlation calculation”, the correlation calculation unit 161 calculates a first correlation value using the image set acquired in step S1901. Specifically, the correlation calculation unit 161 extracts a partial region including the pixel of interest 1920 for which distance calculation is performed on the A image 1910 </ b> A and its neighboring pixels as a reference image 1911. FIG. 19C is a diagram for explaining a region of the reference image. The correlation calculation unit 161 sets the reference image 1911 so as to include an area where the pattern light 1801 and the pattern light 1802 are projected.

  Next, the correlation calculation unit 161 extracts a region having the same area (image size) as that of the base image 1911 on the B image 1910B and sets it as a reference image 1912. After that, the correlation calculation unit 161 moves the position where the reference image 1912 is extracted on the B image 1910B in the same x-axis direction as the pupil division direction, and the reference image 1912 and the standard image 1911 at each movement amount (each position). A correlation value is calculated. Thereby, the correlation calculation part 161 calculates the correlation value which consists of a data string of the correlation value with respect to each moving amount | distance.

  The correlation value calculation method may be the same as that in step S302 according to the first embodiment. Further, the correlation calculation unit 161 can set the reference image 1912 as an image having the same vertical and horizontal dimensions as the vertical and horizontal dimensions of the standard image 1911.

  Next, in step S1903 “parallax amount calculation”, the parallax calculation unit 162 calculates the parallax amount using the correlation value obtained in step S1902 by any known method. For example, a data sequence of correlation values corresponding to the movement amount that provides the highest correlation among the correlation values and the movement amount in the vicinity thereof is extracted, and the movement amount that provides the highest correlation is obtained by any known interpolation method. By estimating with sub-pixel accuracy, the amount of parallax can be calculated.

  In step S1904 “distance value calculation”, the distance calculation unit 163 calculates the distance to the subject 102 by converting the parallax amount into the defocus amount or the subject distance by a known method, as in step S305 according to the first embodiment. To do.

  As described above, in the distance detection method according to the present embodiment, the reference image 1911 is set so as to include a region where the pattern light 1801 and the pattern light 1802 that are displaced in the pupil division direction (parallax calculation direction) are projected. Thereafter, a correlation value between the standard image 1911 and the reference image 1912 is calculated, and a parallax amount is calculated from the correlation value, thereby reducing a ranging error that occurs in relation to the luminance distribution of the projection pattern and the position of the standard image. And high-precision ranging can be performed.

  Here, an example of the processing result of the distance detection method according to the present embodiment will be described with reference to FIG. FIG. 20 shows a pattern light projected onto a flat plate arranged in parallel with the imaging device 103 at a known distance using the projection device 101, and taken by the imaging device 103, and the parallax at each position on the flat plate. It is a graph which shows the result of having calculated quantity. The horizontal axis in FIG. 20 represents the movement amount (pixel position), and the vertical axis represents the parallax amount calculation error. In FIG. 20, as a conventional process, a parallax amount calculation error 2001 calculated by projecting a line pattern in which a bright part and a dark part extend uniformly in the y-axis direction is indicated by a broken line, and the pattern according to the present embodiment is A calculation error 2002 of the parallax amount calculated by projection is indicated by a solid line. It can be seen that the error amount is close to 0 and the parallax amount calculation error is reduced by the method of the present embodiment as compared with the prior art. Therefore, according to the processing according to the present embodiment, distance measurement can be performed with high accuracy.

  Next, with reference to FIGS. 21A to 21F, the reason why an error occurs in the calculation of the parallax amount in the conventional processing and the calculation error of the parallax amount is reduced in the processing according to the present embodiment will be described. In the following description, it is assumed that the A image and the B image have the same shading and have no parallax. FIGS. 21A to 21C are diagrams for explaining the reason why an error occurs when a parallax amount is calculated by projecting a line pattern as a conventional method. The reason for the occurrence of the error is the same as that described in the first embodiment, and therefore the description is simplified.

  FIG. 21A is a diagram showing the positional relationship between an A image 2101 having a line pattern in which bright regions and dark regions appear alternately, a reference image 2102, and a reference image 2103. The reference image 2102 includes image edges 2104, 2105, 2106 (boundary portions) in which the bright area and dark area of the A image 2101 are switched inside the reference image 2102. FIG. 21B shows correlation values calculated by moving a reference image set with respect to the standard image 2102 and performing a correlation calculation between the standard image 2102 and the reference image.

  Correlation values C0, Cp, and Cm are correlation values when the position of the reference image is moved by 0, +1, and −1 pixel, respectively. In this case, as described in the first embodiment, since the absolute value of the movement amount of the reference image corresponding to the correlation values Cp and Cm is the same, the image between the images caused by the image edges 2104, 2105, and 2106 on the line pattern The difference is the same amount. Therefore, the correlation value Cp and the correlation value Cm are the same value. By interpolating this correlation value, a correlation curve 2110 is obtained, and a movement amount (parallax amount) having the highest correlation value is calculated, whereby the parallax amount 2111 is obtained, and a correct value (parallax amount 0) is obtained.

  On the other hand, in the reference image 2103, the image edge 2104 and the right end of the reference image 2103 overlap. FIG. 21C shows each correlation value calculated by moving the reference image set with respect to the standard image 2103 and performing a correlation calculation between the standard image 2103 and the reference image. In this case, when the movement amount is +1 pixel, the correlation value Cp increases from the correlation value C0 as in the case of the reference image 2102. On the other hand, when the movement amount is −1 pixel, the difference between the base image and the reference image is caused only by the image edges 2105 and 2106, and therefore, the increase amount of the correlation value Cm from the correlation value C0 is very small. . Therefore, the correlation value is asymmetrical on the + side and the − side of the movement amount of the reference image, and the parallax amount 2113 obtained by the correlation curve 2112 interpolated with this is different from the correct value (parallax amount 0), and the error is appear. This is a calculation error of the amount of parallax generated in relation to the luminance distribution of the projection pattern and the position of the reference image.

  Next, as in this embodiment, the pattern light 1800 including the pattern lights 1801 and 1802 shifted in the parallax calculation direction is projected and photographed, and the distance measurement calculation is performed using the reference image including both the pattern lights 1801 and 1802. The reason why the error is reduced by performing the above will be described. FIG. 21D is a diagram showing an A image 2100 and a reference image 2120 acquired by projecting the pattern light 1800 according to the present embodiment. FIG. 21 (e) shows a correlation calculated by dividing the reference image 2120 into a first partial image 2121 and a second partial image 2122 for simple explanation of the principle. It is the figure which showed the value.

  As the first partial image 2121, a partial image in which the image edge 2123 caused by the pattern light 1801 and the right end of the partial image overlap is assumed. At this time, the first correlation values Cm1, C01, Cp1 calculated from the first partial image 2121 are the same as the correlation values Cm, C0, Cp shown in FIG.

  Next, a partial image different from the first partial image 2121 is set as the second partial image 2122 so that the left end of the partial image overlaps the image edge 2124 by the pattern light 1802. At this time, the positional relationship between the second partial image 2122 and the image edge 2124 is a relationship obtained by inverting the positional relationship between the first partial image 2121 and the image edge 2123. Therefore, as shown in FIG. 21E, the second correlation values Cm2, C02, and Cp2 obtained using the second partial image 2122 are obtained by inverting the first correlation values Cm1, C01, and Cp1. It becomes.

  When these correlation values are added and averaged for each position (movement amount) of the reference image, the correlation values become the third correlation values Cm3, C03, and Cp3. The third correlation value is symmetrical because the asymmetry on the + side and − side of the movement amount of the reference image is canceled. In this case, the parallax amount 2115 obtained from the correlation curve 2114 obtained by interpolating the correlation value becomes a correct value (parallax amount 0), and an appropriate parallax amount can be calculated. In the process according to the present embodiment, the calculation error of the parallax amount can be reduced based on such a principle, and the distance can be measured with high accuracy based on the appropriately calculated parallax amount.

  In the present embodiment, the parallax amount calculation error can be further reduced by making the pattern light 1801 and the pattern light 1802 different by an appropriate amount in the x-axis direction (parallax calculation direction). 22A and 22B are diagrams for explaining appropriate positions of the first pattern light, the second pattern light, and the reference image included in the projected pattern light.

  The reference image 2230 is a reference image set to images 2210 and 2220 acquired using different pattern lights including the first pattern light and the second pattern light. 22A and 22B, the right edge of the reference image 2230 and the image edge by the first pattern light overlap.

  Here, consider a case where the error of the correlation value based on the image edge of the first pattern light is canceled using the second pattern light. In this case, for example, in order to cancel the error of the correlation value due to the image edge of the first pattern light at the right end of the reference image, the correlation value due to the image edge of the second pattern light with respect to the left end of the reference image. The second pattern light may be set so that this error occurs. When canceling the correlation value error caused by the image edge of the first pattern light at the left end of the reference image, an error of the correlation value caused by the image edge of the second pattern light occurs at the right end of the reference image. In this way, the second pattern light may be set. The setting of the pattern light may be performed by the arithmetic processing unit 106 or may be performed by the control unit 108.

  FIG. 22A shows an image 2210 obtained by projecting pattern light including the first pattern light 2211 and the second pattern light 2212. In the image 2210, the right end of the reference image 2230 overlaps with the image edge 2213 of the first pattern light 2211, and the left end of the reference image overlaps with the image edge 2214 of the second pattern light 2212. In this case, the error of the first correlation value generated based on the image edge 2213 of the first pattern light 2211 is used as the second correlation value calculated based on the image edge 2214 of the second pattern light 2212. Can be canceled.

  FIG. 22B shows an image 2220 obtained by projecting pattern light including the first pattern light 2221 and the second pattern light 2222. In the image 2220, the right end of the reference image 2230 overlaps with the image edge 2223 of the first pattern light 2221, and the left end of the reference image 2230 overlaps with the image edge 2224 of the second pattern light 2222. In this case, the error of the first correlation value generated based on the image edge 2223 of the first pattern light 2221 is used as the second correlation value calculated based on the image edge 2224 of the second pattern light 2222. Can be canceled.

  If the position difference in the parallax calculation direction (x-axis direction) between the first pattern light and the second pattern light in these cases is obtained, it can be expressed by the above-described equation (1a) or equation (1b). . Note that W, P, H, and n in the equation can be the same parameters as those described in the first embodiment.

  In the distance detection method according to the present embodiment, the positions of the first pattern light and the second pattern light are different from each other in the parallax calculation direction by the amount indicated by the formula (1a) or the formula (1b). In addition, the parallax amount calculation error can be reduced, and highly accurate distance measurement can be performed. Note that the position of the pattern light is changed by the control unit 108 by controlling / changing the position of a pattern forming unit such as a pattern mask for forming each sub-pattern light with respect to the light source in the projection apparatus 101. be able to. Further, according to the control of the control unit 108, the position of the pattern light is adjusted by other known arbitrary methods such as controlling the spatial light modulator in the projection apparatus 101 or switching between a plurality of pattern forming means. It may be changed.

  Note that the length of the first pattern light and the second pattern light in the direction perpendicular to the parallax calculation direction (y-axis direction) is set to be shorter than the length of the reference image in the same direction. Thereby, the projection area of both pattern lights is included in the reference image, and the calculation error of the parallax amount can be reduced by using such a reference image. In particular, the length of the first pattern light and the second pattern light in the direction perpendicular to the parallax calculation direction can be set to an even-numbered length of the length of the reference image in the same direction. In this case, the projection area of both pattern lights is uniformly included in the reference image, and the effect of reducing the parallax amount calculation error can be most obtained.

  In addition, the first pattern light and the second pattern light can be alternately projected without a gap in the y-axis direction. In this case, since the number of areas in which the luminance changes due to the first pattern light and the second pattern light in the reference image increases, it is possible to appropriately reduce the parallax amount calculation error and calculate the parallax amount with high accuracy. be able to.

  In the present embodiment, the pattern light is described as including two sub-pattern lights, but the number of sub-pattern lights included in the pattern light is not limited to two. For example, as shown in FIG. 23, pattern light 2300 includes pattern light 2301 (first sub-pattern light), pattern light 2302 (second sub-pattern light), and pattern light 2303 (third sub-pattern light). ) May be included.

  The pattern lights 2301, 2302, and 2303 have the same luminance distribution in the x-axis direction (parallax calculation direction, first direction), and the positions in the x-axis direction are shifted from each other. The pattern lights 2301, 2302, and 2303 are projected at different positions in the y-axis direction (second direction), and more specifically, repeatedly in order. The luminance distribution in the x-axis direction of each of the pattern lights 2301, 2302, and 2303 has the same cycle in which the high luminance region and the low luminance region are alternately repeated.

  Even with such pattern light, an image edge caused by any one of the pattern lights is applied to both ends of the reference image, so that the above-described effect of reducing the parallax amount calculation error can be achieved. Even if the number of sub-pattern lights is three or more, the same effect can be obtained.

  As described above, in the distance detection apparatus according to the present embodiment, the projected pattern light includes the first sub pattern light and the second sub pattern light. The first sub-pattern light and the second sub-pattern light are pattern lights whose positions are shifted from each other in the parallax calculation direction (first direction) and the second direction perpendicular to the first direction. Further, the first sub-pattern light and the second sub-pattern light are periodic lights in which a high-luminance region and a low-luminance region are alternately repeated in the first direction. In this embodiment, the same luminance distribution is used. Are light obtained by shifting the pattern light having s in the first and second directions with respect to each other. Further, the reference image includes an image of a region where the first sub pattern light and the second sub pattern light are projected.

  Further, the projection apparatus 101 projects the first sub-pattern light and the second sub-pattern light whose positions are set according to the formula (1a) or the formula (1b). Specifically, the arithmetic processing unit 106 calculates the visual amount by the width of the reference image in the first direction or the difference between the width and the period of the first sub-pattern light in the first direction in the captured image. The positions of the first sub pattern light and the second sub pattern light are shifted in the direction. Alternatively, the arithmetic processing unit 106 performs the first calculation in the visual calculation direction by the difference between the width in the first direction of the reference image and the width of the high-luminance area in the first direction of the first sub-pattern light in the captured image. The positions of the first sub-pattern light and the second sub-pattern light are shifted and set. Alternatively, the arithmetic processing unit 106 is equivalent to the first sub pattern light and the second sub pattern in the visual calculation direction by an amount obtained by subtracting the period of the first sub pattern light in the captured image from the difference from the difference. Set by shifting the position of the pattern light. Based on the set position, the projection device 101 projects the pattern light so that the first sub-pattern light and the second sub-pattern light have different positions in the first direction.

  With such a configuration, the distance detection apparatus according to the present embodiment can reduce a parallax amount calculation error in distance detection using a captured image on which pattern light is projected. Therefore, the distance detection device can acquire highly accurate distance information of the subject 102 based on the amount of parallax with reduced calculation error.

  In this embodiment, the correlation value is calculated by dividing the inside of the reference image, calculating the correlation value for each partial image, and averaging. However, the calculation method of the correlation value is not limited to this. For example, the correlation value may be calculated for each row in the reference image, and the correlation value of each row may be calculated and averaged, or the entire region of the reference image may be calculated. The correlation value may be calculated using Note that the calculation for the correlation values of the partial images and rows is not limited to the averaging, but may be addition.

  In the present embodiment, the case where the reference image 2120 is set so that the upper half includes the projection area of the pattern light 1801 and the lower half includes the projection area of the pattern light 1802 has been described. However, the reference image setting method is not limited to this. FIG. 21F shows another setting example of the reference image. For example, like the reference image 2130 shown in FIG. 21F, the reference image may be set so that the projection areas of the pattern lights 1801 and 1802 are included in a plurality of areas in the y direction in the reference image. . In other words, the length of the reference image in the y direction may be set to be a length that is an integral multiple of the period of the pattern lights 1801 and 1802 in the y direction.

  By calculating the correlation value by these methods and calculating the amount of parallax, the calculation error of the amount of parallax can be reduced, and the distance can be measured with high accuracy based on the appropriately calculated amount of parallax. .

  It should be noted that no matter where the reference image is set on the photographed image, it is preferable that there are image edges based on the respective pattern lights at both ends of the reference image. Therefore, the projection pattern is preferably a periodic pattern in which the luminance distribution is periodically repeated in the X-axis direction. In this case, if the period pattern is complete, an area shifted by one period may be erroneously detected when calculating the amount of parallax by correlation calculation. Therefore, this erroneous detection can be avoided by limiting the range in which the amount of parallax is searched (the amount of displacement by which the reference image is moved) to be smaller than the pattern period.

  Further, the projection pattern does not need to be a pattern in which the same luminance distribution is repeated. The projection pattern may be a pattern in which substantially the same luminance distribution is periodically repeated. For example, the width of the bright region in the parallax calculation direction may be different for each line. Moreover, the pattern which has the dispersion | variation in a brightness | luminance in a bright area | region or a dark area | region may be sufficient. By appropriately changing the luminance, it is possible to avoid erroneously detecting a region shifted by one period, and to achieve the effect of reducing the above-described parallax amount calculation error.

  In addition, the distance detection apparatus according to the present embodiment can be applied to a production work robot apparatus as in the second embodiment. An example of this case will be briefly described with reference to FIG. Note that the configuration of the robot apparatus in this case may be the same as the configuration of the robot apparatus according to the second embodiment, and thus the description thereof is omitted using the same reference numerals as those in FIG.

  In such a robot apparatus, as described in the present embodiment, the pattern light 1800 including the first pattern light 1801 and the second pattern light 1802 is projected onto the work 904, and an image set based on the pattern light 1800 is captured. To do. Thereafter, the correlation value is calculated using the reference image including the projection areas of both pattern lights 1801 and 1802, and the distance information is obtained, whereby the distance information of the workpiece 904 can be acquired with higher accuracy. Thereby, in the robot apparatus 900, the estimation accuracy of the position and orientation of the workpiece 904 is improved, the accuracy of position control of the robot arm 902 and the robot hand 903 is improved, and a more accurate assembly operation can be performed.

  Note that the distance between the distance detection device and the workpiece 904 changes according to the positions of the robot arm 902 and the robot hand 903. At this time, if distance measurement is performed without changing the projection pattern of the projection apparatus 101, the size of the pattern on the captured image changes according to the distance. Therefore, in this case, the position of the image edge by the pattern light 1801 and 1802 and the ratio of the pattern light 1801 and 1802 included in the reference image change, and the effect of reducing the detection error of the parallax amount may be weakened. is there.

  Accordingly, the arithmetic processing unit 106 analyzes an image acquired by photographing the pattern light 1800, and the amount of positional deviation in the x-axis direction, the periodic interval in the x-axis direction, or the length in the y-axis direction of the pattern light 1801 and 1802. Can be calculated and evaluated. Next, the arithmetic processing unit 106 can determine the size of the reference image according to these sizes.

  In addition, the arithmetic processing unit 106 determines the positional deviation amount and the period interval in the x-axis direction of each pattern light 1801 and 1802 and the length in the y-axis direction from these sizes calculated from the captured image and the size of the reference image. May be. In this case, the projection apparatus 101 can be controlled by the control unit 108, and the pattern light corrected based on the parameter determined by the arithmetic processing unit 106 can be projected to perform distance measurement.

  According to such processing, distance measurement can be performed with an optimum reference image and projection pattern light according to the distance between the distance detection device and the workpiece 904, and highly accurate distance measurement can be performed. Note that the processing by the arithmetic processing unit 106 may be performed by the control unit 108 or may be performed by the control device 905.

  Further, as the distance between the workpiece 904 and the robot hand 903 is closer, it is required to acquire distance information with higher in-plane resolution and to estimate the position and orientation of the workpiece 904 with higher accuracy. Here, the distance of the workpiece 904 can be roughly calculated from the size of the workpiece 904 on the captured image. In addition, since acquisition of distance information of the workpiece 904 by the distance detection device and control of the robot arm 902 by the control device 905 are sequentially performed in time series, the distance of the workpiece 904 is based on the distance information acquired one cycle before these processes. You can also get an overview.

  Therefore, the arithmetic processing unit 106 is set so that the positional deviation amount and the period interval in the x-axis direction of each pattern light 1801, 1802 and the length in the y-axis direction are reduced based on the schematic distance information of the workpiece 904. May be. Further, as the distance is shorter, the arithmetic processing unit 106 may set the size of the reference image in the x-axis direction and the y-axis direction to be smaller based on the periodic intervals of the pattern lights 1801 and 1802. According to such processing, distance measurement can be performed with the optimum reference image and projection pattern light according to the distance between the distance detection device and the workpiece 904, and highly accurate distance measurement can be performed. Note that the processing by the arithmetic processing unit 106 may be performed by the control unit 108 or may be performed by the control device 905.

  According to these methods, each pattern light and the reference image can be appropriately set, and the robot apparatus 900 can reduce the parallax amount calculation error by the distance detection apparatus, and the distance of the workpiece 904 can be set with high accuracy. Can be calculated. For this reason, in the robot apparatus 900, the estimation accuracy of the position and orientation of the workpiece 904 is improved, the accuracy of the position control of the robot arm 902 and the robot hand 903 is improved, and more accurate assembly work can be performed.

(Example 7)
In the first embodiment, the first reference image and the second reference image are set at different positions in the parallax calculation direction on the captured image obtained by projecting the pattern light, and the correlation value between the reference image and the reference image is set. The parallax amount was calculated from both correlation values. In the fourth embodiment, the first pattern light and the second pattern light that are displaced in the parallax calculation direction are projected to obtain an image set based on each pattern light, and the first correlation value is obtained from each image set. The second correlation value was calculated, and the parallax amount was calculated from both correlation values. Further, in the sixth embodiment, pattern light including a plurality of sub-pattern lights shifted in the parallax calculation direction is projected, and a reference image is set so as to include an area where both sub-pattern lights are projected on the captured image, The correlation value was calculated using the reference image to calculate the amount of parallax.

  Here, the technique according to the first embodiment is a first technique, the technique according to the sixth embodiment is a second technique, and the technique according to the fourth embodiment is a third technique. In the first method and the second method, distance measurement can be performed by one pattern projection, and high-speed distance measurement can be performed. On the other hand, in the third method, the reference image is set in a narrow region including the target pixel, and the distance information of the object around the target pixel is obtained by performing the distance measurement by adjusting the position of the plurality of pattern lights. Can be mixed, and the distance of the target pixel can be measured with high accuracy.

  Therefore, in the seventh embodiment, a case where these first to third methods are used by switching according to the distance measurement condition will be described. The configuration of the distance detection device and the robot device according to the present embodiment is the same as the configuration of the distance detection device according to the first embodiment and the configuration of the robot device according to the second embodiment. The description will be omitted using reference numerals.

  For example, in the robot apparatus 900 described in the second embodiment, when the distance between the robot hand 903 and the workpiece 904 is long, the robot hand 903 is required to be brought close to the vicinity of the workpiece 904 quickly. In such a case, it is preferable to perform distance measurement at high speed using the first method or the second method.

  Further, when the distance between the robot hand 903 and the workpiece 904 is short, it is required to align the robot hand 903 and the workpiece 904 with high accuracy. In such a case, it is preferable to measure the position of the workpiece with high accuracy using the third method.

  For this reason, in the robot apparatus according to the present embodiment, when the distance between the distance detection apparatus and the workpiece 904 is longer than the predetermined distance, the arithmetic processing unit 106 performs the first method or implementation described in the first embodiment. Ranging is performed using the second method described in Example 6. Further, when the distance between the distance detection device and the workpiece 904 is equal to or less than a predetermined distance (shorter than the predetermined distance), the arithmetic processing unit 106 uses the third method described in the fourth embodiment to measure the distance. I do.

  Here, the distance of the workpiece 904 can be roughly calculated from the size of the workpiece 904 on the captured image. In addition, since acquisition of distance information of the workpiece 904 by the distance detection device and control of the robot arm 902 by the control device 905 are sequentially performed in time series, the workpiece is based on the distance information acquired one cycle before (before) these processes. It is also possible to know the approximate distance of 904. Therefore, the arithmetic processing unit 106 switches the method used for detecting the distance (second distance) between the distance detection device and the workpiece 904 based on the approximate distance (first distance) of the workpiece 904. You can go. Note that the arithmetic processing unit 106 sets in advance among the first to third methods when the method used for distance measurement is switched based on the distance detected over time, or when the distance of the workpiece 904 is detected for the first time. The distance of the workpiece 904 may be detected by using the method described above.

  In the distance detection apparatus according to the present embodiment, distance measurement can be performed by switching the distance measurement method based on the approximate distance between the distance detection apparatus and the subject. Therefore, more appropriate distance measurement can be performed according to the positional relationship between the distance detection device and the subject, the situation, and the like.

  Note that the above-described processing by the arithmetic processing unit 106 may be performed by the control unit 108 or the control device 905. Further, the subject whose distance is detected by the distance detection apparatus according to the present embodiment is not limited to the work 904, and may be any subject.

  The distance detection apparatuses according to the first embodiment and the third to seventh embodiments described above are not limited to the configuration applied to the robot apparatus, and may be applied to an imaging apparatus such as a camera or an endoscope.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  Although the present invention has been described with reference to the embodiments, the present invention is not limited to the above embodiments. Inventions modified within the scope not departing from the spirit of the present invention and inventions equivalent to the present invention are also included in the present invention. Moreover, each above-mentioned Example and modification can be combined suitably in the range which is not contrary to the meaning of this invention.

100: distance detection device, 101: projection device, 102: subject, 103: imaging device, 161: correlation calculation unit, 162: parallax calculation unit

Claims (41)

Acquisition means for acquiring an image set having parallax;
Correlation calculating means for setting a reference image on one image of the image set and calculating a correlation value of the image set based on the reference image;
Parallax calculating means for calculating the parallax amount of the image set using the correlation value;
With
The correlation calculating means includes
Setting a first reference image on one image of the image set, and calculating a first correlation value based on the first reference image;
A second reference image is set at a position different from the position of the first reference image in a predetermined direction on one image of the image set, and a second correlation value is set based on the second reference image. To calculate
The parallax calculation unit is a parallax detection device that calculates the parallax amount using the first correlation value and the second correlation value.   The parallax detection device according to claim 1, wherein the acquisition unit acquires an image set having parallax obtained by imaging a subject on which pattern light is projected.   The parallax detection device according to claim 2, wherein the pattern light has a line pattern in which a high luminance region and a low luminance region extend in a direction perpendicular to the predetermined direction.   The correlation calculating means calculates the first reference image and the second reference image, the width of the first reference image in the predetermined direction, and the width and the image on at least one image of the image set. 4. The parallax detection device according to claim 2, wherein the parallax detection device is set at a different position in the predetermined direction by one of the differences from the pattern light period.   The correlation calculating means determines the first reference image and the second reference image as the width of the first reference image in the predetermined direction and the height of the pattern light on at least one image of the image set. 4. The parallax detection according to claim 2, wherein the position is set to a different position in the predetermined direction by either one of a difference from a width of a luminance region and an amount obtained by subtracting a period of the pattern light from the difference. apparatus.   The correlation calculation unit calculates a fluctuation period of pixel values based on the pattern light from the image set, and determines the position of the second reference image based on the calculated fluctuation period. 3. The parallax detection device according to 3.   The parallax detection device according to any one of claims 2 to 6, further comprising projection means for projecting the pattern light onto a subject. Determination means for determining whether any image of the image set has periodicity in the predetermined direction;
The parallax detection device according to claim 1, wherein, when any image in the image set is determined to have the periodicity, the correlation calculation unit calculates the first and second correlation values.   9. The correlation calculation unit according to claim 1, wherein the correlation calculation unit sets the first reference image and the second reference image at different positions in a direction perpendicular to the predetermined direction. Parallax detection device.   The parallax detection device according to any one of claims 1 to 9, wherein the first reference image and the second reference image have the same image size. Projection means for projecting pattern light onto the subject;
Acquisition means for acquiring an image set having parallax;
A correlation calculating means for setting a reference image on one image of the image set and calculating a correlation value of the image set based on the reference image;
Parallax calculating means for calculating the parallax amount of the image set using the correlation value;
With
The projecting means projects first pattern light and second pattern light having patterns whose positions are shifted from each other in a predetermined direction,
The correlation calculating means includes
Calculating a first correlation value based on a first image set obtained by projecting the first pattern light;
Calculating a second correlation value based on a second image set obtained by projecting the second pattern light;
The parallax calculation unit is a parallax detection device that calculates the parallax amount using the first correlation value and the second correlation value.   The correlation calculation means uses the reference image set at the same position on one image of the first image set and one image of the second image set, and uses the first correlation value and the second correlation value. The parallax detection device according to claim 11, wherein the correlation value is calculated.   The first pattern light and the second pattern light are periodic lights in which a high luminance region and a low luminance region are alternately repeated in the predetermined direction, and a second light that is perpendicular to the predetermined direction. The parallax detection device according to claim 11 or 12, comprising a line pattern in which the high luminance region and the low luminance region extend in a direction.   The parallax detection device according to claim 11, wherein the first pattern light and the second pattern light are light obtained by shifting pattern lights having the same luminance distribution in the predetermined direction with respect to each other.   The projection unit is configured to obtain a difference between a width of the reference image in the predetermined direction and a width of the high-intensity region of the first pattern light in at least one image of the first image set, and at least the difference from the difference. Positions of the pattern of the first pattern light and the pattern of the second pattern light in the predetermined direction by either one of the amounts obtained by subtracting the period of the first pattern light in one image The parallax detection device according to claim 11, wherein the second pattern light is projected so that the two are different from each other.   The projecting means may be one of a width of the reference image in the predetermined direction and a difference between the width and the period of the first pattern light in at least one image of the first image set. The second pattern light is projected so that a pattern of the first pattern light and a pattern of the second pattern light have different positions in the predetermined direction. Parallax detection device. Evaluation means for evaluating a period in the predetermined direction of the first pattern light in at least one image of the first image set;
The parallax detection device according to claim 13, wherein the projection unit projects the second pattern light at a position determined based on the evaluated period.   Based on at least one image of the first image set and at least one image of the second image set, a positional shift amount of the first pattern light and the second pattern light in the predetermined direction is determined. The parallax detection device according to any one of claims 11 to 14, further comprising calculation means for calculating and determining a length of the reference image in the predetermined direction based on the amount of positional deviation. The projection means projects the first pattern light and the second pattern light having different wavelength bands,
The parallax detection according to any one of claims 11 to 18, wherein the acquisition unit acquires the image set based on the first pattern light and the image set based on the second pattern light separately. apparatus.   The parallax calculation unit calculates the amount of parallax using a correlation value calculated by either addition or addition averaging of the first correlation value and the second correlation value. The parallax detection device according to one item. The parallax calculation means includes
Calculating a first parallax amount from the first correlation value;
A second amount of parallax is calculated from the second correlation value;
The parallax detection device according to any one of claims 1 to 19, wherein a parallax amount of the image set is calculated using a parallax amount obtained by averaging the first parallax amount and the second parallax amount. The correlation calculating unit sets a reference image on the other image in the image set, and calculates a correlation value between the reference image and the reference image while moving the position of the reference image in the predetermined direction. The parallax detection device according to any one of claims 1 to 21. Projection means for projecting pattern light onto the subject;
Acquisition means for acquiring an image set having parallax;
A correlation calculating means for setting a reference image on one image of the image set and calculating a correlation value of the image set based on the reference image;
Parallax calculating means for calculating the parallax amount of the image set using the correlation value;
With
The pattern light includes a first sub-pattern light and a second sub-pattern light,
The first sub pattern light and the second sub pattern light are pattern lights whose positions are shifted from each other in a first direction and a second direction perpendicular to the first direction,
The parallax detection device, wherein the reference image includes an image of a region where the first sub-pattern light and the second sub-pattern light are projected.   The parallax according to claim 23, wherein the first sub-pattern light and the second sub-pattern light are periodic lights in which a high-luminance region and a low-luminance region are alternately repeated in the first direction. Detection device.   The parallax according to claim 24, wherein the first sub-pattern light and the second sub-pattern light are light obtained by shifting pattern lights having the same luminance distribution in the first and second directions with respect to each other. Detection device.   The projection means includes a difference between the width of the reference image in the first direction and the width of the high-luminance area in the first direction of the first sub-pattern light in at least one image of the image set; and The first sub-pattern light and the second sub-pattern are equal to one of the difference obtained by subtracting the period in the first direction of the first sub-pattern light in the at least one image from the difference. The parallax detection device according to claim 24 or 25, wherein the pattern light is projected so that the position of the light in the first direction is different.   The projection means includes a width of the reference image in the first direction, and a difference between the width and a period of the first sub-pattern light in the first direction in at least one image of the image set. 26. The pattern light is projected so that the position of the first sub-pattern light and the second sub-pattern light in the first direction is different from each other by any one amount. Parallax detection device.   The parallax detection device according to any one of claims 23 to 27, wherein the first sub-pattern light and the second sub-pattern light are light having the same period in the second direction.   The length of the reference image in the second direction is a length that is an integral multiple of the period of the first sub-pattern light and the second sub-pattern light in the second direction. The parallax detection device described.   The period of the first sub-pattern light in the first direction in at least one image of the image set, or the first sub-pattern light and the second sub-pattern light in the at least one image. And further comprising evaluation means for evaluating a difference in position in one direction and determining the length of the first direction and the second direction of the reference image based on the period or the difference in position. Item 26. The parallax detection device according to any one of Items 23 to 25. The period of the first sub-pattern light in the first direction in at least one image of the image set, or the first sub-pattern light and the second sub-pattern light in the at least one image. An evaluation means for evaluating the difference in position in the direction of 1;
The projecting means may determine the first sub-pattern light and the second sub-light based on the evaluated period or position difference and the lengths of the reference image in the first direction and the second direction. The parallax detection device according to any one of claims 23 to 25, which projects pattern light in which the period and position of the pattern light are corrected.   The correlation calculating unit sets a reference image on the other image of the image set, and calculates a correlation value between the reference image and the reference image while moving the position of the reference image in the first direction. The parallax detection device according to any one of claims 23 to 31. A parallax detection device according to any one of claims 1 to 32;
A robot arm,
A robot hand provided on the robot arm;
A control device for controlling the robot arm and the robot hand;
A robot apparatus comprising The parallax detection device includes:
A distance calculating means for calculating a distance to the subject based on the parallax amount;
Obtain distance information including the distance to the workpiece and image information of the workpiece,
The controller is
Estimating the position and orientation of the workpiece using the distance information and the image information;
The robot apparatus according to claim 33, wherein the robot arm and the robot hand are controlled based on the position and orientation. A parallax detection device according to any one of claims 1 to 32;
Distance calculating means for detecting the distance to the subject using the amount of parallax detected by the parallax detection device;
A distance detection device comprising: Obtaining a set of images having parallax;
Setting a reference image on one image of the image set, and calculating a correlation value of the image set based on the reference image;
Calculating a parallax amount of the image set using the correlation value;
Including
Calculating the correlation value is
Setting a first reference image on one image of the image set, and calculating a first correlation value based on the first reference image;
A second reference image is set at a position different from the position of the first reference image in a predetermined direction on one image of the image set, and a second correlation value is set based on the second reference image. Calculating
Including
Calculating the amount of parallax includes calculating the amount of parallax using the first correlation value and the second correlation value. Projecting pattern light onto the subject;
Obtaining a set of images having parallax;
Setting a reference image on one image of the image set, and calculating a correlation value of the image set based on the reference image;
Calculating a parallax amount of the image set using the correlation value;
Including
The projecting includes projecting a first pattern light and a second pattern light having patterns whose positions are shifted from each other in a predetermined direction,
Calculating the correlation value is
Calculating a first correlation value based on a first image set obtained by projecting the first pattern light;
Calculating a second correlation value based on a second set of images obtained by projecting the second pattern light;
Including
Calculating the amount of parallax includes calculating the amount of parallax using the first correlation value and the second correlation value. Projecting pattern light onto the subject;
Obtaining a set of images having parallax;
Setting a reference image on one image of the image set, and calculating a correlation value of the image set based on the reference image;
Calculating a parallax amount of the image set using the correlation value;
Including
The pattern light includes a first sub-pattern light and a second sub-pattern light,
The first sub-pattern light and the second sub-pattern light are pattern lights whose positions are shifted from each other in a first direction and a second direction perpendicular to the first direction,
The parallax detection method, wherein the reference image includes an image of an area where the first sub-pattern light and the second sub-pattern light are projected. Each step of the parallax detection method according to any one of claims 36 to 38;
Detecting the distance to the subject using the amount of parallax detected by the parallax detection method;
A distance detection method including: A distance detection method using a distance detection device,
The first of the subject and the distance detection device is based on either the size of the subject in the image photographed by the distance detection device or the distance between the subject and the distance detection device previously detected for the subject. To get the distance of
The second distance between the subject and the distance detection device is detected based on the amount of parallax detected by the parallax detection method according to claim 36 or 38 when the first distance is longer than a predetermined distance. And
Detecting the second distance based on the amount of parallax detected by the parallax detection method according to claim 37 when the first distance is equal to or less than the predetermined distance;
A distance detection method using a distance detection device.   41. A program that, when executed by a processor, causes the processor to execute the steps of the method according to any one of claims 36 to 40.