JP2019184587A - Parallax detection device, parallax detection method, and parallax detection device control program - Google Patents

Abstract

To solve the problem in which it is difficult to set a condition for obtaining a prescribed calculation speed and range-finding accuracy regarding a technique of contracting a pair of raw images acquired by using pattern projections, and calculating an amount of parallax.SOLUTION: A parallax detection device of the present invention comprises: acquisition means that photographs a measurement object having pattern light projected, and acquires a pair of raw images having parallax; contraction means that contracts the pair of raw images; and calculation means that calculates a parallax map serving a distribution of an amount of parallax having a prescribed search range in each area and a search window therein and corresponding to each area of the pair of raw images from the pair of raw images to be obtained from the contraction means. In order to obtain the parallax map having a prescribed number of data, the calculation means is configured to deem a hierarchy of the parallax map to be calculated using the pair of images of the smallest magnification to be obtained from the contraction means as the lowest layer, and a prescribed search range upon calculating the parallax map in each hierarchy is determined on the basis of the parallax map below one hierarchy.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a parallax calculation device, a parallax calculation method, and a control program for a parallax calculation device.

  A method for acquiring a captured image and calculating distance information from the acquired captured image has been proposed. For example, there is a method of acquiring a plurality of images from different viewpoints, obtaining a parallax amount from a correlation value between the images, and acquiring distance information. Specifically, a signal of a partial area called a search window (hereinafter also simply referred to as a window) is extracted from each of the image sets having parallax. Then, the parallax amount is calculated by calculating the correlation value while changing the position of the search window. At this time, when there is a region with a poor pattern (hereinafter also referred to as a texture) in the captured image, the correlation accuracy is lowered due to the low contrast of the signal, and the ranging accuracy is lowered. On the other hand, there is a method of acquiring a captured image using pattern light projection and reducing the influence of a decrease in distance measurement accuracy even in an originally poor pattern area.

  Japanese Patent Application Laid-Open No. 2004-228561 proposes a method of calculating a parallax amount by projecting striped pattern light onto an object and using two images (image sets) captured from different positions.

JP2015-137933A

  However, in the distance measuring technique using the projection of pattern light as in Patent Document 1, sufficient studies on speeding up processing and reducing the load have not been made.

  The present invention has been made in view of the above problems, and in stereo distance measurement using pattern projection, a parallax calculation apparatus, a parallax calculation method, and a parallax calculation apparatus capable of calculating a parallax amount using hierarchical processing An object is to provide a control program.

  In order to achieve the above object, a parallax calculation device as an example of the present invention includes an acquisition unit that captures a measurement target on which pattern light is projected and acquires an original image set having parallax, and the original image set. Reduction means for reducing, and calculation means for calculating a disparity map that is a distribution of the amount of parallax corresponding to each area of the image set with a predetermined search range and search window in each area from the image set obtained from the reduction means. In order to obtain a parallax map having a predetermined number of data, the calculation means includes a plurality of parallax map hierarchies calculated using the image set with the smallest magnification obtained from the reduction means, The parallax map is calculated in each layer, and the predetermined search range when calculating the parallax map in each layer is determined based on the parallax map one layer below

  According to the present invention, in stereo ranging using pattern projection, the processing speed can be increased or the processing load can be reduced by calculating the amount of parallax using hierarchical processing.

It is a mimetic diagram showing an example of a distance measuring device concerning a 1st embodiment. It is a schematic diagram which shows an example of the imaging device which concerns on 1st Embodiment. A is a cross-sectional view illustrating an example of a pixel according to the first embodiment, and B is a schematic diagram illustrating an exit pupil according to the first embodiment. An example of the projection pattern which concerns on 1st Embodiment is shown. 1 shows an example of a projection apparatus according to a first embodiment. An example of the flow of the distance detection method which concerns on 1st Embodiment is shown. A is a schematic diagram showing an example of an image captured by projecting pattern light, and B is an example of a graph showing the amplitude and period of a signal of a first pixel and a signal of a second pixel. A is an example of a graph showing a spatial frequency and sampling pixels, and B is an example of a graph showing a signal sampled after being reduced by a magnification m1. A is an example of a graph showing a spatial frequency and a sampling pixel, and B is an example of a graph showing a signal sampled after being reduced by a magnification m2. It is a schematic diagram which shows the other example of the imaging device which concerns on 1st Embodiment. It is a schematic diagram which shows an example of the robot which concerns on 2nd Embodiment. An example of the flow of the distance detection method which concerns on 2nd Embodiment is shown. An example of the image acquired in provisional imaging is shown. It is a schematic diagram which shows an example of the user interface which concerns on 2nd Embodiment.

  Exemplary embodiments of the present invention will be described in detail with reference to the drawings. The same reference numbers have been used to refer to the same configuration. The present invention is not limited to the contents described in each embodiment. Moreover, you may combine each embodiment suitably.

(First embodiment)
In the present embodiment, a method using hierarchical processing is proposed for the purpose of increasing the speed of stereo ranging using pattern projection or reducing the load. In the hierarchical processing, the calculation load is reduced and the processing speed is increased by calculating the amount of parallax using a reduced image created by thinning out a captured image at a fixed pixel interval, for example. Further, in the calculation of the amount of parallax in the original image with a large number of pixels, the search range is limited based on the amount of parallax obtained in the lower hierarchy, thereby reducing the computation load and speeding up the processing.

  The magnification (size of the reduced image) when generating a reduced image is preferably determined according to a predetermined calculation speed and a calculation accuracy of predetermined distance information. In the pattern projection image captured by projecting pattern light, the magnification affects the pattern. In other words, if the combination of the magnification and the spatial frequency of pattern projection is not suitable, the texture S / N ratio of the texture due to the pattern decreases in each image of the generated image set. Thereby, the calculation accuracy of parallax, that is, the distance measurement accuracy decreases. Therefore, in this embodiment, in stereo ranging using pattern projection, various processing conditions are used to reduce the processing load by using hierarchical processing and further suppress the decrease in ranging accuracy (parallax amount and phase difference calculation accuracy). suggest.

(Device configuration and pattern imaging unit)
FIG. 1 shows a distance measuring apparatus as an example of a parallax calculating apparatus including a pattern imaging unit 100 that acquires an original image set by projecting pattern light onto a distance measurement target. The pattern imaging unit 100 as an example of an acquisition unit includes a projection device 101 (projection unit) that projects pattern light and an imaging device 103 (imaging unit) that captures an original image set having parallax. The projection device 101 projects pattern light onto a subject 102 that is a measurement target. Then, the imaging device 103 captures the original image set. The projection apparatus 101 and the imaging apparatus 103 are connected to a computer 104 (calculation unit) that performs control such as synchronization and calculates a parallax amount from an image set obtained by reducing an original image set. In the present embodiment, the pattern imaging unit 100 including the projection device 101 and the imaging device 103 and the computer 104 are provided as separate devices, but may be integrated. Further, the projection apparatus 101 and the computer 104 may be integrated, and the imaging apparatus 103 may be a separate apparatus, and the apparatus configuration is not limited to the above-described configuration. When the imaging device 103 and the computer 104 are integrated, the arithmetic processing unit 204 of the imaging device 103 illustrated in FIG. 2 may also serve as the configuration / function of the computer 104.

  The computer 104 includes a resizing unit 1040, a correlation calculation unit 1041, a parallax amount calculation unit 1042, a distance calculation unit 1043, a memory 1044, and a control unit 1045. A resizing unit 1040 as an example of a reducing unit reduces an original image set having parallax input from the imaging device 103 at a predetermined magnification and outputs the reduced image to the correlation calculation unit 1041. The correlation calculation unit 1041 calculates a correlation value map of the image set using, for example, SAD (Sum of Absolute Difference) while relatively moving the image set having the parallax input from the resizing unit 1040. The parallax amount calculation unit 1042 as an example of a calculation unit determines, for example, a correlation value that provides the highest correlation from the correlation value map obtained by the correlation calculation unit 1041, and calculates the parallax amount (image shift amount) of the image set. To do. The distance calculation unit 1043 converts the parallax amount (image shift amount) obtained by the parallax amount calculation unit 1042 into a defocus amount using a conversion coefficient corresponding to the current state of the optical system, and further calculates the distance to the subject. The subject distance shown is calculated. The computer 104 uses at least one of a parallax amount, a defocus amount, and a subject distance as a calculation result based on an original image (signal) set sequentially input from the imaging device 103 or an original image set temporarily stored in the memory 1044. Store in memory 1044 or output to outside. In the present embodiment, the parallax amount, the defocus amount, and the subject distance described above are combined, and the depth information and the information indicating the position or positional relationship of the subject in the image in the depth direction (depth direction) correspond to the image. I will call it. The control unit 1045 sends an instruction to each unit in the computer 104 to control each unit.

  FIG. 2 shows the configuration of the imaging device 103. The imaging device 103 includes a camera body 201, an imaging lens 202, an imaging element 203, a calculation processing unit 204 (calculation unit), and a main body memory 205. In the present embodiment, each pixel of the image sensor 203 is divided in at least one direction so that the image capturing apparatus 103 can acquire the original image set. One pixel of the image sensor 203 includes a microlens 311, a filter 312, and photoelectric conversion units 310 </ b> A and 310 </ b> B as illustrated in a cross-sectional view in FIG. 3A. The image sensor 203 is provided with a spectral characteristic corresponding to the wavelength band detected by the filter 312 for each pixel, and has a predetermined color arrangement pattern (for example, a Bayer-array using RGB (Red, Green, Blue) color filters) on the xy plane. Is arranged. On the substrate 313, photoelectric conversion units 310A and 310B having sensitivity in the wavelength band to be detected are formed. The wavelength band to be detected is not limited to visible light, but it is assumed that the wavelength band of light projected by the projection apparatus 101 is included in the wavelength band in which at least some of the photoelectric conversion units can receive light with high sensitivity. Each pixel includes a wiring (not shown).

  A first pupil region 321A and a second pupil region 321B shown in FIG. 3B are different regions of the exit pupil 320 of the imaging lens 202, respectively. The first light beam that has passed through the first pupil region 321A is incident on the photoelectric conversion unit 310A, and the second light beam that has passed through the second pupil region 321B is incident on the photoelectric conversion unit 310B. A first signal is acquired from the first light flux acquired by the photoelectric conversion unit 310A, which is the first pixel in the image sensor 203. Similarly, the second signal is acquired from the second light flux acquired by the photoelectric conversion unit 310B that is the second pixel. Then, the arithmetic processing unit 204 (FIG. 2) forms an A image from the first signal and forms a B image from the second signal. The formed A image and B image are stored in the main body memory 205, and the computer 104 (FIG. 1) performs a distance measurement calculation process using the A image and the B image to calculate a parallax map and a distance map.

  The ranging calculation process is performed by a known method. For example, the computer 104 calculates a correlation value map, obtains a parallax map from the correlation value map, and converts the parallax map into a distance map. Pattern light is projected onto the subject 102 from the projection device 101, and the imaging device 103 captures an image with the texture superimposed on the surface of the subject 102. Thereby, the calculation of the correlation value peak, and thus the accuracy of the distance measurement calculation is improved.

  FIG. 4 shows a pattern 401 of pattern light to be projected. The pattern 401 is not limited, but a pattern 401 having a fixed period with respect to the parallax division direction 402 is desirable so that the spatial frequency of the pattern in the acquired original image set can be easily controlled. In addition, there are few restrictions on the direction 403 perpendicular to the parallax division direction 402. However, since the acquired original image set and the correlation value can be easily smoothed, a similar repeated pattern, that is, a vertical stripe pattern having a constant period in the parallax dividing direction 402 such as the pattern 401 is desirable.

  A projection apparatus 101 illustrated in FIG. 5 includes a body 501, a light source 502, an adjustment lens 503, a spatial modulator 504, and a projection lens 505. The light source 502 has a white LED, and the light emitted from the light source 502 is imaged on the spatial modulator 504 by the adjustment lens 503. Thereafter, the spatial modulator 504 forms pattern light, and the pattern light is imaged on the surface of the subject 102 by the projection lens 505. Spatial modulator 504 is formed of DMD (Digital Mirror Device). The projection apparatus 101 is not limited to this configuration. A diffuser plate may be introduced to make the light quantity distribution uniform, or various projection optical systems such as Koehler illumination or an optical system that forms a light source image on the entrance pupil may be applied.

(Basic processing flow and hierarchical processing)
A basic processing flow for acquiring an original image set and calculating a distance map, and a hierarchical process using the reduced image set will be described. In the present embodiment, in order to obtain a parallax map having a predetermined number of data, processing using an image set that is reduced from the original image set by reducing the magnification by the reduction process is set as the lowest layer. The magnification for generating the reduced image of the lowermost layer is preferably set to a magnification that can suppress a decrease in distance measurement accuracy. Note that the magnification of the lowermost reduced image will be described later in detail. Then, the parallax map is sequentially calculated in a plurality of hierarchies where the magnification of the image set is increased step by step (not reduced so much). At this time, the search range or search window in each layer is determined by the limited search range or search window size based on the calculation result of one layer below, thereby greatly increasing the processing load and processing speed for calculating the parallax map. To reduce.

  FIG. 6 shows a basic processing flow according to the first embodiment. In step S601, the projection apparatus 101 projects pattern light onto the subject 102, and the imaging apparatus 103 acquires an original image set of the subject 102 onto which the pattern light is projected (pattern imaging step). In step S <b> 602, the resizing unit 1040 of the computer 104 reduces the original image set by the magnification m, and creates an image set of the lowermost reduced image. Here, the magnification m corresponds to the ratio of the number of pixels read out when forming an image set. Accordingly, the magnification m of the reduced image created by thinning out by outputting every a pixels with respect to the number of pixels of the original image group acquired in step S601 can be expressed by the following equation (1).

m = 1 / a (1)
Therefore, a reduced image with a magnification factor m can be obtained by outputting every a pixels from the pixels of the original image set acquired in step S601.

  Next, a correlation value map (correlation value distribution corresponding to each region of the image set) is calculated by the known calculation method as described above using the image set obtained by reducing the magnification m in step S603. To do. At this time, since the correlation value map is calculated using the image set reduced by thinning out the number of pixels, the in-plane resolution (sampling number, data number) is compared with the case where the correlation value map is calculated from the conventional original image set. Decrease. On the other hand, since the search range of the correlation value map and the number of pixels in the search window used for calculation are smaller than when calculating using the original image set, the calculation speed is improved. In step S604, a parallax map between the A and B images of the image set is calculated from the obtained correlation value map (calculation step).

  In step S605, it is determined whether or not the parallax map has a desired output resolution (number of data, number of pixels) or more. If not, the process proceeds to step S606. Proceed. The desired output resolution can be arbitrarily set, and the setting method is not particularly limited. When the desired output resolution cannot be obtained, the calculator 104 calculates a parallax map from another image set obtained by reducing the original image set at a larger magnification. That is, in step S606, according to the desired output resolution at the time of calculating the distance map, another image set is thinned out by outputting every b pixels smaller than a with respect to the number of pixels of the original image set. An image set is created and used for correlation calculation on one layer (a> b). Therefore, the magnification of the other image set is larger than the magnification of the previously created image set, and the image set has a higher output resolution than the previously created image set.

  Next, in step S607, based on the disparity map previously obtained in step S604, a search range in the correlation calculation on one layer is set, and the correlation value map is generated using the other image set generated in step S606. Is calculated. In the present embodiment, based on the amount of parallax previously obtained in the corresponding region, a pixel shift amount (for example, ± 10 Pixel in a certain hierarchy) that relatively shifts the pixels of the image set in the search range, that is, the correlation calculation, It is set to the value of the parallax amount or a value in the vicinity thereof.

  In addition, the search window size (for example, 4 lines in the vertical direction and 20 pixels in the horizontal direction in a certain hierarchy) that is the size of the area to be subjected to correlation calculation in the image set is the magnification (a, b, etc.) or resolution of each hierarchy. Is set according to That is, the size (resolution, number of pixels) of the search window is set so that the correlation calculation target regions (subjects to be processed) in each layer are substantially the same.

  Furthermore, according to the reliability of the correlation value obtained earlier in the lower layer, the region with high reliability has high reliability of the calculated amount of parallax (there is no need to increase the size to absorb the calculation error), The search window size may be set smaller than the size converted from the magnification. Here, the reliability of the correlation value is used to measure the slope of the curve plotting the image shift amount obtained by the correlation calculation and the correlation value, or whether the image signal itself is easily correlated and has a large edge component. Alternatively, the edge integral value in the target region of the image set or the SN ratio of the image signal may be used.

  Further, the search window size may be set to a constant value (for example, four rows in the vertical direction and 20 pixels in the horizontal direction in all layers) regardless of the magnification of each layer. The resolution of the image increases as the hierarchy increases, but the in-plane resolution of the correlation value map is improved because the search window size is a constant value. In addition, since the correlation value is calculated in the vicinity of the correct parallax amount based on the calculated parallax amount in the lower layer, the search window size is not changed according to the hierarchy but is set to a constant value (target area of the correlation calculation) Correlation value calculation errors due to changes in the subject can be avoided.

  With the above processing, the processing load and processing speed for the calculation processing of the amount of parallax can be significantly reduced as compared to the case where a uniform large search range and search window are set for the entire image when the amount of parallax is unknown. it can.

  Thereafter, in step S608, a parallax map (a distribution of parallax amounts corresponding to each region of the image set) is calculated. As described above, since the parallax map of the image set obtained in step S604 is reflected in the setting of the search range and search window in the next other image set, the setting of the search range and search window is unnecessarily large. Don't be. Thereby, the calculation speed is improved as compared with the case where the calculation of the image group obtained by the reduction is not performed. In step S609, it is determined whether or not the parallax map is equal to or higher than the desired output resolution. repeat. That is, an image set having a higher magnification than the other image sets is created by reducing the original image set, a correlation value map is calculated using the created new image set, and then a parallax map is calculated.

  If a parallax map with a desired output resolution or higher is obtained (YES in step S609), a distance map (distance corresponding to each area of the image set) from the parallax map based on the optical system information of the camera in step S610. Value distribution). If the resolution of the image set created by the magnification m (image set created by thinning out a pixels) is equal to or higher than the desired output resolution (YES in step S605), a distance map is generated using the parallax map of the image set. calculate. In this case, without proceeding to step S606, the process proceeds to step S610 to calculate the distance map.

(Variable determination method and relationship between magnification m and spatial frequency f)
Subsequently, in the calculation of the parallax map according to the first embodiment, the magnification m for reducing the original image set and the pattern light are projected in the lowest layer hierarchical processing for reducing the original image set to the smallest in step S602. The relationship with the spatial frequency f in the original image obtained by imaging will be described.

  FIG. 7A shows an original image 701 obtained by projecting the pattern 401 onto the subject 102 with the configuration shown in FIG. In order to simplify the description here, the image signal of the section 703 corresponding to the subject 102 on which the pattern light is projected on the line segment 702 in the original image 701 (one image of the original image set) is extracted in FIG. 7B. Show. The image signal in the section 703 includes a first signal 711 (solid line) and a second signal 712 (broken line) corresponding to a certain period of the pattern 401. Here, the first signal 711 is a signal of an A image acquired by the photoelectric conversion unit 310A that is the first pixel. The second signal 712 is a B image signal acquired by the photoelectric conversion unit 310B, which is the second pixel. The first signal 711 and the second signal 712 have a phase shift 713 corresponding to the amount of parallax.

  8A and 8B, the relationship between the magnification m and the reference frequency f (hereinafter also referred to as the spatial frequency f) that is the pattern spatial frequency on the measurement target in the acquired original image set will be described. FIG. 8A shows a partial section of the first signal 711 and the second signal 712 shown in FIG. 7B and shows the relationship with the imaging pixel 801 in one dimension. The first signal 711 has a reference frequency f (= 1 / T) corresponding to the period T indicated by the section 802 on the acquired original image (the same applies to the second signal 712). On the other hand, the number of pixels a1 corresponding to the section 803 is thinned out, and the image set is created by using only the sampling pixel 804 indicated by a thick frame in FIG. The image signal is shown in FIG. 8B.

  An image signal of an image sampled by reducing the first signal 711 at the magnification m1 is a signal 811 indicated by a solid line. An image signal of an image sampled by reducing the second signal 712 at the magnification m1 is a signal 812 indicated by a broken line. At this time, the sampling pixel 804 is set in a relationship of m1 = 2f so as to satisfy the sampling theorem. As shown in FIG. 8B, the signal 811 corresponding to the first signal 711 retains the original signal shape, whereas the signal 812 corresponding to the second signal 712 loses the original signal shape. .

  This is because the first signal 711 and the second signal 712 have a phase shift due to the amount of parallax, so that even if the sampling theorem is applied to one signal, the shape of the other signal is not saved or restored. is there. If an appropriate magnification is not set for the original image set acquired by projecting the pattern light onto the subject 102 in this way, the contrast of the signal shape (texture) of the A image or B image of the reduced image set is set. It will decline. As a result, the calculation accuracy of the correlation value map, and hence the calculation accuracy of the parallax map and the distance map, is reduced.

  It is when the phase shift is ± π / 4 that the shape of the other signal disappears most with respect to the minimum sampling that keeps the shape of one signal to the maximum. Therefore, if the magnification (sampling pixel) is set so as to satisfy the relationship of m ≧ 4f, both the A signal and the B signal can be sampled with the minimum number of pixels while leaving the signal shape. FIGS. 9A and 9B show a state of sampling with the minimum number of pixels.

  As in FIG. 8A, in FIG. 9A, the first signal 711 has a spatial frequency f (= 1 / T) corresponding to the period T indicated by the section 802 on the acquired original image (the second signal 712 is also the same). ). On the other hand, the number of pixels a2 corresponding to the section 903 is thinned out, and only the sampling pixel 904 indicated by a thick frame in FIG. 9A is created, that is, the image sampled after being reduced by the magnification m2 (= 1 / a2) The image signal is shown in FIG. 9B. An image signal of an image sampled by reducing the first signal 711 at the magnification m2 is a signal 911 indicated by a solid line. An image signal of an image sampled by reducing the second signal 712 at the magnification m2 is a signal 912 indicated by a broken line. Here, the relationship between the magnification m2 and the spatial frequency f is m2 = 4f. At this time, even in a reduced image set, the signal 911 corresponding to the A image and the signal 912 corresponding to the B image both reflect the signal shape of the original image signal. Therefore, the parallax map can be calculated without reducing the calculation accuracy of the correlation value map. In order to obtain the effect of the first embodiment, the relationship between the magnification m determined by the sampling pixel interval and the spatial frequency f should satisfy the following formula (2).

      m ≧ 4f (2)

  However, the above equation (2) is established regardless of the value of the magnification m if the spatial frequency f is reduced as much as possible, that is, if the period T of the pattern light pattern 401 to be projected is increased as much as possible. On the other hand, a period T that is unnecessarily large for the subject 102 for which the distance map is calculated is not suitable from the viewpoint of in-plane resolution for calculating the distance map. Therefore, the pattern imaging unit 100 including the projection device 101 and the imaging device 103 controls the spatial frequency f on the acquired original image set so as to satisfy the relationship of the magnification m = 4f with respect to the magnification m. Here, the magnification m is determined by a predetermined spatial resolution and calculation speed, and the spatial frequency f is a spatial frequency on an acquired image set of pattern light that provides the maximum in-plane resolution. The spatial frequency f may be calculated by pattern detection in sequentially acquired image sets, and corresponds when the optical conditions of the projection apparatus 101 and the imaging apparatus 103, the positional relationship with the subject, and the like are known in advance. The spatial frequency f may be stored in the memory 1044 in advance.

  Here, the setting accuracy of the spatial frequency f will be described. An error superimposed by the optical system of the projection device 101 and the imaging device 103 of the pattern imaging unit 100 and an error caused by the relative distance between the devices including the subject 102 are superimposed. Then, the spatial frequency f on the acquired original image has an error of ± 25%. Therefore, the relationship between the magnification m and the reference frequency f may satisfy the following formula (3).

      5f ≧ m ≧ 3f (3)

  When the relationship between the reference frequency f, which is the spatial frequency of the pattern 401 of the subject 102 on the acquired image, and the magnification m determined by the number of thinned pixels satisfies Expression (3), the parallax map is reduced while maintaining the calculation accuracy. A correlation value map can be calculated from the image set formed in this way. That is, the effect of setting the magnification m satisfying the predetermined calculation speed and resolution without reducing the distance measurement accuracy is obtained. For example, the computer 104 sets a projection pattern or an imaging condition so as to satisfy the above formula (3) (setting step). That is, the computer 104 sets the generation pattern or pattern mask in the spatial modulator 504 in the projection apparatus 101 or the magnification of the projection lens 505 and the angle of view or magnification of the image acquired by the imaging apparatus 103. Alternatively, the computer 104 sets the hierarchical processing magnification m so as to satisfy the above formula (3) (setting step).

(Other configurations)
The imaging device 103 that acquires the original image set may be a stereo camera having two or more optical systems and imaging elements corresponding thereto. By using a stereo camera, the design flexibility of the baseline length is improved and the ranging resolution is improved. In addition, the pattern imaging unit 100 according to the first embodiment may be a single device in which the projection device 101 is mounted on the imaging device 103. Since this apparatus is in a state in which the positional relationship between the imaging apparatus 103 and the projection apparatus 101 is fixed, it is possible to improve environmental resistance and facilitate setting of the angle of view or magnification condition. Specifically, as illustrated in FIG. 10, the imaging apparatus according to the first embodiment includes a projection apparatus 1001 mounted on the upper part of the camera body 201 and controls each other's parameters and synchronization using the control apparatus 1010. The imaging device 1000 may be used.

  By using the computer 104 as a CPU (Central Processing Unit) provided in the imaging apparatus 1000, the imaging apparatus 1000 can be reduced in size. The spatial modulator 504 serving as a pattern control unit may be any one of a reflective LCOS (Liquid Crystal On Silicon), a transmissive LCOS, and a DMD (Digital Micromirror or Device). By using these spatial modulators 504, a pattern with a large number of pixels can be generated at a high response speed. Thereby, the in-plane resolution can be improved, the pattern can be switched at high speed, and the image acquisition speed can be improved.

  The pattern control unit may project the pattern by putting a pattern mask in which a pattern is formed in advance, such as ground glass or a metal plate, into and out of the optical path of the projection apparatuses 101 and 1001. Thereby, reduction of apparatus cost and size reduction of an apparatus are realizable. Furthermore, the light source in the projection apparatuses 101 and 1001 may be any of a heat source such as an LD (Laser Diode), an LED (Light Emitting Diode), and a halogen lamp. In view of the size of the device and the subject, the device can be reduced in size and cost by designing an appropriate light amount and light source size.

  The light sources of the projection apparatuses 101 and 1001 can emit white light whose wavelength includes the entire visible light range. Thereby, in the method of the first embodiment, the effect of the first embodiment can be obtained regardless of the spectral reflectance of the subject. Further, the light sources of the projection apparatuses 101 and 1001 may emit light of three colors of R, G, and B. Thereby, the color filter transmission band of the imaging device and the wavelength of light can be matched, and the light utilization efficiency with respect to the energy used can be increased.

  Further, the wavelength of the light source of the projectors 101 and 1001 is an IR (Infrared) region, and an object is captured using an imaging device including an imaging element in which a color filter having a transmission band corresponding to the IR region and a light receiving sensitivity is arranged. You may image. Thereby, an image for viewing using the RGB band can be taken simultaneously with the distance measurement. In particular, when the IR wavelength band is between 800 nm and 1100 nm, Si can be used for the photoelectric conversion portion. Thereby, by changing the arrangement of the color filters, a viewing image in the RGB band and a ranging image in the IR band can be acquired by one image sensor.

  The reduced image may be generated by averaging the pixels corresponding to the number of thinned pixels. Thereby, although the identification performance of the pattern and texture concerning ranging accuracy falls, environmental resistance improves by smoothing. The calculation method of the correlation value map may be any of SSD (Sum of Squared Difference), SAD (Sum of Absolute Difference), and POC (Phase Only Correlation), but is not limited to these methods.

  The first embodiment includes a computer program in addition to the imaging apparatus. The computer program of the first embodiment causes a computer to execute a predetermined process for calculating a distance map or a parallax map. The program of the first embodiment is installed in a computer of an imaging apparatus such as a digital camera equipped with a distance detection apparatus, a parallax amount detection apparatus, or any one thereof. When the installed program is executed by the computer, the above function is realized, and the distance map and the parallax map can be detected at high speed and with high accuracy. As an example, this program is a control program for the imaging apparatus 1000 including a pattern imaging unit 100 that acquires an original image set and a calculator 104 that calculates a parallax map from an image set obtained by reducing the original image set. . The control program causes the computer to execute a calculation step and a setting step. In this calculation step, a parallax map is calculated from an image set obtained by reducing the original image set. In the setting step, either the pattern imaging unit 100 or the computer 104 is set so that the relationship between the magnification m, which is a magnification at the time of reduction, and the reference frequency f on the original image satisfies the above equation (3). Set the conditions.

(Second Embodiment)
(High-speed measurement mode, FA inspection device)
With reference to FIGS. 11 to 13, a FA (Factory Automation) inspection apparatus using the imaging apparatus 1100 according to the second embodiment and a speed setting mode will be described.

  FIG. 11 shows a robot 2000 of the FA inspection apparatus according to the second embodiment. The robot 2000 includes an imaging device 1100, a work stage 1101, a robot arm 1102, a robot hand 1103, and a control device 1105 (calculation unit). Based on the depth information acquired using the imaging device 1100, the control device 1105 controls the robot arm 1102 and the robot hand 1103 to grip the work 1104 (measurement target) that is the main subject. FIG. 12 shows a flow related to the calculation of depth information in the gripping operation.

  In step S1201, various conditions such as a measurement speed, a reference frequency, a region of interest, the number of pixels to be thinned out, a magnification, a projection condition, and an imaging condition are set. For example, the user sets the measurement speed. The measurement speed may be an actual speed or may be selected by a user from a preset stage such as “high speed / medium speed / low speed”. In accordance with the measurement speed, provisional imaging is performed using the imaging device 1100, the magnification m, the pattern light to be projected (projection pattern) or the imaging conditions of the imaging device 1100 are set, and the reference frequency f is set.

  A captured image 1301 acquired by provisional imaging of the workpiece 1104 is shown in FIG. In step S <b> 1202, the control device 1105 sets an area in which the workpiece 1104 is captured as a region of interest: ROI (Region Of Interest) 1302. Next, the control device 1105 calculates a magnification corresponding to the measurement speed set in step S1201. Specifically, when the “high speed” mode is set, the control device 1105 calculates the magnification m_ROI so that pixels are thinned out every 10 pixels from the image of the ROI 1302. At this time, the control device 1105 causes the projection device (not shown) so that the relationship between the magnification m_ROI and the spatial frequency f of the captured image 1301 (pattern light superimposed on the workpiece 1104) satisfies the following formula (4). Alternatively, the conditions of the imaging apparatus 1100 are set (setting step). Alternatively, the control device 1105 sets the magnification m_ROI so as to satisfy the following formula (4) (setting step).

      5f ≧ m_ROI ≧ 3f (4)

  For example, when the conditions of the imaging device 1100 are fixed, the control device 1105 changes the conditions of the spatial modulator 504 in the projection device so that the spatial frequency f of the captured image 1301 satisfies the relationship of the above formula (4). Control. Thereafter, the processing from step S1202 to step S1210 is performed, and the control device 1105 calculates a distance map in step S1211. Note that steps S1202 to S1210 are the same as steps S601 to S609 described in the first embodiment, and a description thereof will be omitted. Based on the distance map calculated in step S <b> 1211, the control device 1105 controls the robot arm 1102 and the robot hand 1103 to grip the workpiece 1104.

  As described above, the imaging apparatus according to the second embodiment includes a speed setting mode in which a parallax map is calculated at a predetermined speed by setting a region of interest of a captured image to be measured. Thus, the distance map can be calculated at a predetermined calculation speed by setting the ROI for the main subject of interest, setting the magnification according to the purpose, and reducing the scale. Further, the control device 1105 receives the depth information (depth map) not in the form of the distance value (distance map) but in the form of the parallax amount (parallax map) or the defocus amount (defocus map), and receives the robot arm 1102 and the robot. The hand 1103 may be controllable. In this case, the distance map calculation step of S1206 is not necessary.

  In step S1201, the user may directly set the number of pixels to be thinned out or the magnification. Thereby, since a user can finely adjust according to the objective, usability can be improved. In step S1201, the user may set the number of pixels to be thinned out or the magnification in advance for each of the preset speed items. Thereby, usability can be improved.

  The provisional imaging in step S1201 can be omitted by storing in advance the shape of the target workpiece and the imaging angle of view used for measurement in the memory. The imaging apparatus of the second embodiment may be equipped with an automatic determination mode that recognizes a subject from a captured image to be measured and automatically determines various variables. For example, the control device 1105 may automatically set the magnification and the reference frequency as needed based on the imaged workpiece and pre-registered shape information by using various image recognition techniques. Thereby, usability can be improved. The ROI may be set manually by the user, or the control device 1105 may automatically set the ROI when the target workpiece is recognized from various image recognitions.

  The reference frequency f can be the largest number of components in the spatial frequency distribution in the original image set. For example, by analyzing the frequency of an image group (original image group) obtained by provisional imaging, the region having the most spatial frequency components can be set as the ROI, and the spatial frequency of the ROI can be set as the reference frequency f. Thereby, the ROI and the reference frequency f can be automatically set. In addition, the reference frequency f can be a spatial frequency on the focal plane in the original image set. For example, the in-focus position of the image set (original image set) at the time of provisional imaging is calculated by contrast analysis or provisional distance value measurement, and the in-focus position area is set to ROI. The ROI and the reference frequency f can be automatically set by setting the spatial frequency in the focus position area to the reference frequency f. Furthermore, the reference frequency f can be the highest spatial frequency in the original image set. For example, the region having the highest spatial frequency component is set as the ROI by performing frequency analysis on an image obtained by provisional imaging. By setting the spatial frequency of the region to the reference frequency f, the ROI and the reference frequency f can be automatically set.

  In order to satisfy the measurement conditions according to the second embodiment, the robot 2000 may include a user interface that assists the user in setting projection conditions, imaging conditions, and the like. This user interface can facilitate the operation. FIG. 14 shows an example of a user interface for setting the ROI. The robot 2000 includes a display element 1401 that is a touch panel that displays a captured image. On the display element 1401, the user touches the measurement range 1402 to set the ROI. At this time, the display element 1401 displays, as a grid 1403, a projection pattern having a spatial frequency that satisfies the above formula (4) with respect to a magnification corresponding to a preset measurement speed or the number of pixels to be thinned out. Based on this display, the user changes the pattern shape (projection pattern) by the projection apparatus or the optical conditions for projection or imaging.

(Third embodiment)
(High accuracy mode)
Subsequently, a high-accuracy mode mounted on the imaging apparatus according to the third embodiment will be described. In setting the magnification for generating the reduced image, the reduced image with the highest in-plane resolution is created when the number of pixels to be thinned out a = 2. At this time, the magnification M = 1 / a is 0.5 at the maximum. The imaging apparatus according to the third embodiment is equipped with a high accuracy mode in which the magnification M is the largest value. The magnification M when sampling is performed every two pixels coincides with the Nyquist frequency f_nyq determined by the image sensor, that is, the pixel interval of the image sensor.

  Therefore, in the high accuracy mode of the third embodiment, the reference frequency f satisfies the above formula (3) after satisfying the magnification M = f_nyq. That is, when the sampling pixels are set every two pixels as the high accuracy mode, the reference frequency f on the acquired original image set is set so as to satisfy the equation (5). For example, the projection condition or the imaging condition of the pattern imaging unit is set so as to satisfy Expression (5).

(1/3) * f_nyq ≧ f ≧ (1/5) * f_nyq (5)
By setting the reference frequency f in this way, it is possible to suppress deterioration of the signal shape of the original image set even in the most accurate reduced image (even when sampling pixels are set every two pixels). As a result, the parallax map can be calculated with high accuracy, and thus the distance can be measured with high accuracy.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

DESCRIPTION OF SYMBOLS 100 Acquisition means 102 Measuring object 1040 Reduction means 1042 Calculation means

Claims (13)

An acquisition unit that captures an image of a measurement target onto which pattern light is projected and acquires an original image set having parallax;
Reduction means for reducing the original image set;
Calculating means for calculating a disparity map that is a distribution of the amount of disparity corresponding to each area of the image set with a predetermined search range and search window in each area from the image set obtained from the reducing means;
In order to obtain a parallax map having a predetermined number of data, the calculation means sets the parallax map hierarchy calculated using the image set with the smallest magnification obtained from the reduction means as the lowest hierarchy, The parallax calculation device, wherein the predetermined search range when calculating the parallax map of each hierarchy is determined based on the parallax map of one hierarchy below.   A magnification m, which is a magnification for obtaining the lowermost parallax map when the reduction means reduces the original image set, and a reference frequency on the original image set of the pattern light projected on the measurement object The parallax calculation apparatus according to claim 1, wherein f satisfies a predetermined condition. The relationship between the magnification m and the reference frequency f is as follows: 5f ≧ m ≧ 3f
The parallax calculation device according to claim 2, wherein:   The parallax calculation apparatus according to claim 2, wherein the magnification m is set so as to satisfy the condition.   5. The parallax calculation device according to claim 2, wherein a projection pattern of the pattern light or an imaging condition by the acquisition unit is set so as to satisfy the condition. 6.   5. The parallax calculation apparatus according to claim 2, wherein the reference frequency f is a maximum number of components in a spatial frequency distribution in the original image set. 6.   5. The parallax calculation device according to claim 2, wherein the reference frequency f is a spatial frequency on a focal plane in the original image set.   The parallax calculation device according to any one of claims 2 to 4, wherein the reference frequency f is a highest spatial frequency in the original image set.   9. A speed setting mode in which a parallax map is calculated at a predetermined speed by the calculating means by setting a region of interest in a captured image corresponding to the measurement object is mounted. The disparity calculating apparatus according to any one of the above. A high-accuracy mode in which the reduction means reduces the original image set with the largest magnification is mounted, and with respect to the Nyquist frequency f_nyq determined by the image sensor, the magnification and the pattern projected onto the measurement target The reference frequency f of the light on the original image set satisfies the following condition (1/3) * f_nyq ≧ f ≧ (1/5) * f_nyq
The parallax calculation device according to claim 1, wherein at least one condition of the acquisition unit and the calculation unit is set so as to satisfy the above. From the acquisition unit that captures the measurement target on which the pattern light is projected and acquire the parallax original image set, the reduction unit that reduces the original image set, and the image set obtained from the reduction unit, in each region A parallax calculation method using a parallax calculation device comprising: a parallax map that calculates a parallax map corresponding to each area of the image set with a predetermined search range and search window;
In order to obtain a parallax map having a predetermined number of data, the calculation means sets the parallax map hierarchy calculated using the image set with the smallest magnification obtained from the reduction means as the lowest hierarchy, Calculating a parallax map;
The parallax calculation method, wherein the predetermined search range when calculating a parallax map for each layer is determined based on a parallax map one layer below. From the acquisition unit that captures the measurement target on which the pattern light is projected and acquire the parallax original image set, the reduction unit that reduces the original image set, and the image set obtained from the reduction unit, in each region A control program for a parallax calculation device comprising: a calculation unit that calculates a parallax map that is a distribution of a parallax amount corresponding to each region of the image set with a predetermined search range and a search window;
In order to obtain a parallax map having a predetermined number of data, the calculation means uses a parallax map hierarchy calculated using the image set with the smallest magnification obtained from the reduction means as the lowest layer, and each of the plurality of hierarchies Performing a step of calculating a parallax map;
The control program for a parallax calculation device, wherein the predetermined search range when calculating a parallax map in each hierarchy is determined based on a parallax map one hierarchy below.   A computer-readable storage medium storing the program according to claim 12.

Images