JP2019120590A - Parallax value calculation device, parallax value calculation method and program - Google Patents

Abstract

To provide a technology that improves calculation accuracy of a parallax value among a plurality of images.SOLUTION: A parallax value calculation device acquires a first image and a second image imaged by two cameras, generates a first enlarged image and a second enlarged image by performing enlargement processing in a predetermined parallax direction to each of the first image and the second image, calculates a first census feature quantity and a second census feature quantity by comparing a pixel value with a peripheral pixel for each first pixel of the first enlarged image and for each second pixel of the second enlarged image, calculates a difference degree for each of the second pixels using the first census feature quantity and the second census feature quantity for each first pixel, acquires a parallax value corresponding to the smallest difference degree as the parallax value of a pixel of a parallax image corresponding to the first enlarged image for each first pixel, and calculates a parallax value for each pixel of the first image by performing reduction processing of the parallax image.SELECTED DRAWING: Figure 14

Description

  The present disclosure relates to a disparity value calculating device, a disparity value calculating method, and a program.

  In recent years, in a development field such as a safe driving support system and an automatic driving system of a car, a surveillance camera system for detecting a suspicious person or a robot, a technology for calculating a distance from a camera to a subject using an image captured by the camera Is being considered. As a technique for calculating the distance, there is a stereo distance measuring technique for calculating the distance from images taken by a plurality of cameras. In captured images of a plurality of cameras, the direction in which the same object is viewed is different, and such difference in direction is called parallax. In stereo ranging technology, triangulation technology is used to calculate the distance to the subject based on the parallax. The accuracy of the parallax has a greater effect on the error of the distance measurement result of the subject as the subject is farther. Therefore, techniques for determining the parallax between a plurality of images are being studied. For example, Patent Document 1 discloses a technique for calculating parallax in two images captured by a stereo camera. In Patent Document 1, in order to associate pixels of two images, a census conversion of pixel values and a correlation operation of census-converted pixel values are used. Non-Patent Document 1 discloses a disparity value calculation method called Semi-Global Maching (SGM).

Patent No. 4168125 gazette

Heiko Hirschmuller, "Stereo Processing by Semiglobal Matching and Mutual Information", IEEE, IEEE Transactions on Pattern Analysis Machine Intelligence, February 2008, Vol. 39, No. 2

  The techniques of Patent Document 1 and Non-Patent Document 1 may have low accuracy in calculation of disparity values with sub-pixel accuracy.

  Thus, the present disclosure provides a parallax value calculation device, a parallax value calculation method, and a program that improve the calculation accuracy of parallax values between a plurality of images.

  A parallax value calculation device according to an aspect of the present disclosure includes a processor and a memory, the processor including: (a) a first image captured by a first camera disposed at a first position; The second image captured by the second camera arranged at position 2 is acquired from the memory, and (b) the first image is enlarged by a predetermined parallax direction. Generating a first enlarged image including a plurality of first pixels by increasing the number of pixels of the first image, and enlarging the second image in a predetermined parallax direction Processing to generate a second magnified image including a plurality of second pixels by increasing the number of pixels of the second image; and (c) in the first magnified image, the plurality of first images. The first census feature by comparing pixel values with surrounding pixels for each pixel of And calculating a second census feature value by comparing pixel values with surrounding pixels for each of the plurality of second pixels in the second enlarged image, and (d) calculating the plurality of second plurality of The first census feature amount and the second census feature amount of the at least one second pixel are used for each of the one pixel, and the at the position of each of the at least one second pixel Calculating the degree of difference corresponding to the parallax value for the first pixel, and (e) selecting the parallax value corresponding to the smallest degree of difference using the degree of difference for each of the plurality of first pixels Acquiring a parallax value for each pixel in the parallax image corresponding to the first enlarged image, and (f) performing a reduction process on the parallax value for each pixel of the parallax image to obtain the second image Of the first image against To calculate the disparity value of Motogoto.

  A parallax value calculation method according to an aspect of the present disclosure includes: (a) a first image captured by a first camera disposed at a first position, and a second camera disposed at a second position (B) increasing the number of pixels of the first image by performing enlargement processing in a predetermined parallax direction on the first image (b). Generating a first enlarged image including a plurality of first pixels, and increasing the number of pixels of the second image by performing enlargement processing on the second image in a predetermined parallax direction Thereby generating a second enlarged image including a plurality of second pixels, and (c) comparing pixel values with surrounding pixels for each of the plurality of first pixels in the first enlarged image Calculates a first census feature amount, and the plurality of second pixels in the second enlarged image A second census feature value is calculated by comparing pixel values with surrounding pixels, and (d) the first census feature value and the at least one first census feature value for each of the plurality of first pixels. The difference degree corresponding to the parallax value with respect to the first pixel at each position of the at least one second pixel is calculated using the second census feature value of two pixels, and (e) the above The parallax value for each pixel in the parallax image corresponding to the first enlarged image is acquired by selecting the parallax value corresponding to the smallest degree of difference using the degree of difference for each of the plurality of first pixels And (f) calculating a parallax value for each pixel of the first image with respect to the second image by performing reduction processing on the parallax value for each pixel of the parallax image, and processing (a) ~ At least one of processing (f) It is performed by at least one processor.

  A program according to an aspect of the present disclosure includes: (a) a first image captured by a first camera disposed at a first position, and a second image disposed at a second position; A second image, and (b) enlarging the number of pixels of the first image by enlarging the first image in a predetermined parallax direction, A first magnified image including a first pixel is generated, and the number of pixels of the second image is increased by performing enlargement processing on the second image in a predetermined parallax direction. Generating a second enlarged image including a plurality of second pixels; and (c) comparing the pixel values with surrounding pixels for each of the plurality of first pixels in the first enlarged image. Calculating a census feature amount of 1 for each of the plurality of second pixels in the second enlarged image; A second census feature value is calculated by comparing pixel values with surrounding pixels, and (d) the first census feature value and at least one of the second census feature values for each of the plurality of first pixels. The difference degree corresponding to the parallax value with respect to the first pixel at the position of each of the at least one second pixel is calculated using the second census feature value of the pixel, and (e) the plurality of pixels The parallax value for each pixel in the parallax image corresponding to the first enlarged image is acquired by selecting the parallax value corresponding to the smallest degree of difference using the degree of difference for each of the first pixels, (F) causing a computer to calculate a parallax value for each pixel of the first image with respect to the second image by performing reduction processing on the parallax value for each pixel of the parallax image .

  Note that the above-described comprehensive or specific aspect may be realized by a recording medium such as a system, an apparatus, a method, an integrated circuit, a computer program, or a computer readable recording disk, and the system, an apparatus, a method, an integrated circuit , And may be realized by any combination of computer program and recording medium. The computer readable recording medium includes, for example, a non-volatile recording medium such as a CD-ROM (Compact Disc-Read Only Memory).

  According to the parallax value calculation device and the like of the present disclosure, it is possible to improve the calculation accuracy of parallax values between a plurality of images.

FIG. 1 is a diagram showing an image processing system disclosed in Patent Document 1. As shown in FIG. FIG. 2A is a diagram for explaining the conformal fitting method disclosed in Patent Document 1. FIG. 2B is a view for explaining the conformal fitting method disclosed in Patent Document 1. FIG. 3 is a diagram schematically showing the flow of disparity value calculation processing according to the prior art. FIG. 4 is a diagram schematically showing the flow of the prior art Semi-Global Maching process. FIG. 5 is a CG image used for verification of disparity value calculation processing using SGM. FIG. 6 is a diagram showing the calculation result of the parallax value by SGM using the CG image of FIG. 5 and an image obtained by translating the CG pixels in the horizontal direction. FIG. 7 shows a photographed image used for verification of disparity value calculation processing using SGM. FIG. 8 is a view showing the calculation result of the parallax value by SGM using the photographed image of FIG. 7 and an image obtained by translating the photographed pixels in the horizontal direction. FIG. 9 shows a photographed image used for verification of disparity value calculation processing using SGM. FIG. 10 is a view showing the calculation result of the parallax value by SGM using the photographed image of FIG. 9 and an image obtained by translating the photographed pixels in the horizontal direction. FIG. 11 shows a photographed image used for verification of disparity value calculation processing using SGM. FIG. 12 is a diagram showing the calculation result of the parallax value by SGM using the photographed image of FIG. 11 and the image obtained by translating the photographed pixels in the horizontal direction. FIG. 13 is a diagram showing a hypothetical example of the relationship between the correct parallax value and the calculated parallax value. FIG. 14 is a block diagram showing an example of a functional configuration of a distance measuring system including the disparity value calculating device according to Embodiment 1. FIG. 15 is a schematic perspective view showing an arrangement example of cameras of the imaging unit. FIG. 16 is a schematic front view showing an example of the baseline length of the camera of FIG. FIG. 17 is a schematic perspective view showing another arrangement example of the cameras of the imaging unit. FIG. 18A is a diagram illustrating an example of peripheral pixels in census conversion. FIG. 18B is a diagram illustrating another example of peripheral pixels in census conversion. FIG. 19A is a diagram illustrating an example of the census conversion of the peripheral pixels in FIG. 18A. FIG. 19B is a diagram illustrating an example of the census conversion of the peripheral pixels in FIG. 18B. FIG. 20 is a diagram illustrating an example of a search range of pixels at the time of calculation of the degree of difference. FIG. 21 is a diagram showing another example of the search range of the pixel at the time of calculation of the degree of difference. FIG. 22 is a diagram showing an example of the relationship between the degree of difference calculated for one target pixel and the parallax value of the reference pixel corresponding to the degree of difference. FIG. 23 is a flowchart showing an example of the operation of the disparity value calculating device according to the first embodiment. FIG. 24 is a block diagram showing an example of a functional configuration of a disparity value calculating device according to Embodiment 2. FIG. 25 is a flowchart illustrating an example of the operation of the disparity value calculating device according to the second embodiment. FIG. 26 is a block diagram showing an example of a functional configuration of a disparity value calculating device according to Embodiment 3. FIG. 27 is a flowchart illustrating an example of the operation of the disparity value calculating device according to the third embodiment. FIG. 28 is a block diagram illustrating an example of a functional configuration of a disparity value calculating device according to a modification of the third embodiment. FIG. 29 is a block diagram showing an example of a functional configuration of a disparity value calculating device according to Embodiment 4. FIG. 30 is a block diagram showing an example of a functional configuration of a disparity value calculating device according to a modification of the fourth embodiment. FIG. 31 is a block diagram illustrating an example of a functional configuration of the disparity value calculating device according to the fifth embodiment. FIG. 32 is a diagram showing an example of a plurality of scanning directions from the pixel of interest at the time of cost calculation. FIG. 33 is a diagram showing an example of the scanning direction of a pixel on a straight line passing the pixel of interest at the time of cost calculation. FIG. 34 is a diagram showing an example of the cost of one pixel on a scanning line for cost calculation. FIG. 35 is a flowchart illustrating an example of the operation of the disparity value calculating device according to the fifth embodiment. FIG. 36 is a block diagram showing an example of a functional configuration of a disparity value calculating device according to Variation 1 of Embodiment 5. FIG. 37 is a block diagram showing an example of a functional configuration of a disparity value calculating device according to Variation 2 of Embodiment 5. FIG. 38 is a block diagram showing an example of a functional configuration of a disparity value calculating device according to Variation 3 of Embodiment 5. FIG. 39 is a block diagram showing an example of a functional configuration of a disparity value calculating device according to Variation 4 of Embodiment 5. FIG. 40 is a block diagram showing an example of a functional configuration of a disparity value calculating device according to Embodiment 6. FIG. 41 is a diagram showing an example of a plurality of window areas of the pixel of interest. FIG. 42 is a block diagram showing an example of a functional configuration of a disparity value calculating device according to a modification of the sixth embodiment. FIG. 43 is a block diagram illustrating an example of a functional configuration of a disparity value calculating device according to a seventh embodiment. FIG. 44 is a diagram showing an example of the relationship between the inter-bit difference between the pixel of interest and the peripheral pixels and the weight based on the luminance difference. FIG. 45 is a block diagram showing an example of a functional configuration of a disparity value calculating device according to a modification of the seventh embodiment.

[Findings by the inventor]
The present inventors examined a technique for calculating a parallax value between images with sub-pixel accuracy finer than the pixel unit of a digital image in order to improve distance measurement accuracy using a digital image. Note that the unit of disparity value is one pixel, that is, a pixel, and the sub-pixel indicates less than one pixel. The present inventors examined the techniques disclosed in Patent Document 1 and Non-Patent Document 1 as described below. For example, in the prior art disclosed in Patent Document 1, pixels of stereo images that are a plurality of images captured by a plurality of cameras or compound-eye cameras are subjected to census conversion, and feature values after census conversion for each pixel A census feature value is calculated. Furthermore, the degree of difference of the census feature value of each pixel between stereo images is calculated using the Hamming distance. For the pixel of interest in one of the images, the pixel whose Hamming distance to the pixel of interest is the smallest is extracted from the other image, and the extracted pixel is determined as the pixel corresponding to the pixel of interest. Furthermore, for the pixel corresponding to the pixel of interest in the other image and the neighboring pixels, one pixel is used by using a method called equiangular fitting based on the Hamming distance to the pixel of interest and the positional relationship between the pixels. The disparity value is calculated with sub-pixel accuracy less than.

  For example, FIG. 1 shows an image processing system disclosed in Patent Document 1. This system generates a parallax image 290 from original image data 200 and 201 captured by the left and right cameras 21 and 22. The parallax image 290 is a set of selected optimal parallaxes for individual image elements in the original image. In order to obtain this disparity image 290, the image data 200 and 201 are transformed, correlated between them, and checked for errors and reliability.

  For example, left camera 21 and right camera 22 capture scene 10, and a frame grabber and digitizer provide image data to a reconfigurable image processing system. The left image data 200 and the right image data 201 in the form of individual pixel elements and their respective luminances are mapped to the left luminance image 210 and the right luminance image 211, respectively. Each of these luminance images is the width of X and the height of Y, that is, an X × Y luminance image. Non-parametric local transforms 215 and 216, such as census transform or rank transform, are applied to each of these intensity images. Transform 215 is applied to left luminance image 210, as indicated by arrow 218, to produce transformed vector left image 220. Similarly, transform 216 is applied to the right intensity image 211, as indicated by arrow 219, to produce a transformed vector right image 221. These transformations apply to substantially all of the image elements of the two luminance images in the neighborhood or window of the image elements in each of the two luminance images. Thus, the size of the window and the position of the reference image element determine which image element at the edge of the intensity image is to be ignored in the conversion calculation. These neglected image elements are not used as reference image elements, but the image elements may still be used in the calculation of transformation vectors of other reference image elements.

  The image processing system of FIG. 1 further includes a correlation sum process 225. The correlation summation process 225 is one step in determining the correspondence between the left image 220 and the right image 221. The correlation sum process 225 operates on the transform vector inside the correlation window for the left image 220 and the transform vector inside the correlation window of the same size in the right image 221 and is indicated by a single arrow 226 Thus, the correlation sum matrix 230 is generated. In generating this correlation sum matrix 230, either the left or right image 220 or 221 is used as a reference image and the windows in the other image are shifted. If the right image 221 is treated as a reference image, the correlation sum matrix 230 shows how each right image element of the right image 221 in the correlation window is correlated, or of the left image element of the left image 220. It contains data indicating how it corresponds to the left image element in the correlation window for each shift or disparity from the right image element. By definition, data indicating the correlation or correspondence of a particular left image element with various shifts or disparities of the right image element is also included in the correlation sum matrix 230. The disparity based correlation sum and correlation sum matrix 230, the optimal disparity indicated by the arrow 231, may be selected for each right image element and stored in the external index (also called "index") array 270. The final disparity image 290 is then determined at the external index array 270 as indicated by the arrow 271. In the case of stereo vision, the parallax is the horizontal offset between the window of transformed image 1 and the window of transformed image 2.

  Although not included in FIG. 1, Patent Document 1 discloses that parallax with sub-pixel accuracy is derived by an equiangular fitting method (also referred to as “equiangular straight line fitting method”). In this conformal fitting method, the value of the correlation sum for each of the parallax value giving the minimum value of the correlation sum and the parallax values before and after it, for example, the parallax value of ± 1 for the parallax value giving the minimum value of the correlation sum , And the relationship between the “correlation total value” and the disparity value, the relationship of the V-shaped function is applied.

  For example, FIGS. 2A and 2B illustrate the conformal fitting method disclosed in Patent Document 1. In FIG. 2A, the relationship between the correlation sum value and the disparity value is shown, and the correlation sum value is the minimum value Y2 at disparity value 2. In this case, in addition to the point A where the correlation sum is the minimum value Y2 and the parallax value is 2, the point B with the parallax value 1 and the correlation sum Y1 and the point C with the parallax value 3 and the correlation sum Y3 Used. Furthermore, as shown in FIG. 2B, a V-shaped function passing through these three points is fitted. At this time, the absolute values of the slopes of the two lines of the V-shaped function are the absolute value of the slope of the difference of the correlation sum with respect to the difference of the parallax value between the points A and B, and the points A and C And the absolute value of the slope of the difference of the correlation sum with respect to the difference of the disparity value, whichever is larger. Specifically, in FIG. 2A, the absolute value of the difference (Y3-Y2) of the correlation total value, which is the slope between point A and point C, is the slope between point A and point B. It is larger than the absolute value of the difference (Y2-Y1) of the correlation sum. Therefore, the absolute value of (Y3-Y2) is adopted as the slope of the V-shaped function. In a V-shaped function, the absolute value of the slope of the line on either side of the V-shaped vertex is the same. That is, the angles 1 and 2 of the line on both sides and the horizontal line are the same. Thus, a straight line passing through the point B and having a negative slope as the absolute value of the slope, and a straight line passing through the points A and C and having a positive slope as the absolute value of the slope are formed. By obtaining the intersection point D of these two straight lines, the parallax value can be obtained with sub-pixel accuracy of an interval of less than one pixel. For example, in the example of FIG. 2B, the disparity value is 1.8.

  The determination of disparity image 290 in FIG. 1 may include three optional confidence / error detection tests: object operation, left / right consistency check, and mode filter. The subject operation determines if the luminance image is associated with high level confidence based on the texture of the captured scene. In this way, object operations associated with image elements of the scene that are uniform patterns (textures) have lower confidence than scenes in which the textures are changing. The object operation is applied to only one of the luminance images, or to either the left or right luminance image. However, in other aspects, object operations applied to both intensity images may be covered. In FIG. 1, the subject operation 235 is applied to the right intensity image 2 as indicated by arrow 236, and for each image element inside the object window, as shown by arrow 237, total sliding disparity (SSD) An array 240 is generated. Application of the threshold operation 241 generates a final target result array 250 as a target result. The subject result includes data that reflects whether a particular image element passes the confidence threshold established in the image processing system. Based on the data in the target result array 250, a parallax image 290 may be determined in relation to the external index array 270.

  Left-right consistency check is a form of error detection. This examination determines that the image element in the left image selected as the optimal image element by the image element in the right image also selects the same image element in the right image as the optimal image element and Check. Left-right consistency check 245 is applied to correlation sum array 230, as indicated by arrow 246, compared to external index array 270, as indicated by arrow 276, and LR result array, as indicated by arrow 247. Generate 260. The LR result array 260 includes data indicating image elements that pass the left-right consistency check 245. The LR result array 260 is used to generate a parallax image 290, as indicated by the arrow 261 in cooperation with the external index array 270.

  The third confidence / error detection test is a mode filter. The mode filter determines whether the selected optimal disparity has a high degree of consistency by selecting the optimal disparity based on population analysis. Thereby, if the selected optimal disparities in the external index array 270 do not show a high degree of consistency, then these optimal disparities are invalidated. Mode filter 275 operates on external index array 270, as indicated by arrow 276, and generates mode filter external index (also referred to as "index") array 280, as indicated by arrow 277. The mode filter external index array 280 includes data indicating whether a particular image element has selected a disparity that has passed the disparity consistency check. Data and mode filter external index array 280 may be used to cooperate with external index array 270 to generate disparity image 290, as indicated by arrow 281.

  Whether to perform these three confidence / error detection checks is optional. In one aspect, all three of these tests may be performed in determining the parallax image 290, but in other aspects these tests may not be included at all. In yet another aspect, a combination of these tests may be included.

  For example, the parallax value calculation process of the prior art as described in Patent Document 1 can be schematically shown as shown in FIG. FIG. 3 is a diagram schematically showing the flow of disparity value calculation processing according to the prior art. In the input images I1 and I2 that are stereo images, each pixel is subjected to census conversion in step S1501, and census feature values are obtained. Furthermore, in step S1502, the correlation of census feature values for parallax values between pixels (also referred to as “difference degree”) between the input images I1 and I2 is calculated for each pixel of the input images I1 and I2. It is calculated. Here, the degree of difference between census feature quantities is calculated by the Hamming distance.

  In steps S1503a and S1503b, for each pixel of the reference image which is one of the input images I1 and I2, respectively, the parallax value and the degree of difference of each pixel within the preset search range in the other image (ie Hamming From the distance), the parallax value giving the smallest difference is selected. Thereby, the parallax value of the pixel precision in each pixel of a reference | standard image is obtained. In step S1503a, the above process is performed on the input image I1 as a reference image, and in step S1503b on the input image I2 as a reference image.

  In steps S1504a and S1504b, the parallax values of the subpixel accuracy at each pixel in each of the reference image I1 and the reference image I2 are calculated by the equiangular fitting method as described with reference to FIGS. 2A and 2B. Specifically, for each pixel of the reference image, the disparity value with sub-pixel accuracy is calculated using the minimum difference degree, the disparity value giving it, and the disparity values before and after the disparity value and the difference degree thereof. . In step S1505, the consistency between corresponding pixels is checked for the parallax value in the case of the reference image I1 and the parallax value in the case of the reference image I2. Specifically, it is checked whether the difference between the parallax values in both directions between the corresponding pixels of the reference image I1 and the reference image I2 is less than a predetermined threshold. Furthermore, the parallax value which comprises the difference more than a threshold value is invalidated, and the parallax estimation result about the pixel corresponding to the said parallax value is not output.

  Also, disparity value calculation processing called Semi-Global Maching (hereinafter also referred to as “SGM”) as described in Non-Patent Document 1 can be schematically shown as shown in FIG. In addition, FIG. 4 is a figure which shows typically the flow of the Semi-Global Machining process of a prior art. In step S1601, the input images I1 and I2 and the initial parallax image are reduced. The initial parallax image is an image set in advance, and each pixel has a parallax value between two images in the pixel as a pixel value. In image reduction, the number of pixels is reduced. In step S1602, the degree of difference is calculated for each parallax value for each pixel between the input images I1 and I2. In step S1602, the degree of difference (Hamming distance) between census feature amounts using mutual information or the same census feature amount as in Patent Document 1 can be used to calculate the degree of difference between input images I1 and I2. In step S1603, the cost for the distribution of disparity values on a straight line in the eight or sixteen directions passing through the pixel of interest, that is, the center, is calculated for each pixel of interest in the reference image. In step S1604, with respect to the cost of distribution of parallax on a straight line in 8 or 16 directions passing through the pixel of interest, a parallax value that provides the minimum cost in each direction is selected. In other words, costs are consolidated.

  In steps S1605a and S1605b, among the parallax values that provide the minimum cost in each direction, the parallax value in the direction that provides the minimum cost is selected. At this time, in step S1605a, a parallax value using the input image I1 as a reference image is selected, and in step S1605b, a parallax value using the input image I2 as a reference image is selected. In step S1606, the consistency between the parallax value with the input image I1 as the reference image and the parallax value with the input image I2 as the reference image is checked. In step S1607, the parallax image in which each pixel has the parallax value subjected to the consistency check in step S1606 as a pixel value is enlarged. In this enlargement, both a process of increasing the number of pixels by spatial enlargement of the parallax image and a process of increasing the parallax value (pixel value of parallax image) along with the spatial enlargement are performed. For example, when the spatial magnification is doubled, the parallax value is also converted to double. In step S1608, it is determined whether or not to repeat the processing of steps S1601 to S1607.

  In FIG. 4, when four-step repetitive processing is performed as an example, the parallax value calculation processing of steps S1601 to S1608 with a scale reduced to 1⁄8 in the first processing with respect to the input images I1 and I2 Is calculated, and the parallax value is calculated with a scale reduced by a factor of 4 in the second process, by a scale reduced by a factor of 2 in the third process, and in the same magnification scale in the fourth process. It is processed. Therefore, in step S1608, if the number of repetitions is less than four, the process is continued, and if the number of repetitions is four, the process ends.

  The following differences exist between the SGM and the parallax value calculation process of Patent Document 1. In Patent Document 1, the parallax value search is performed in a local process (local process), and at the time of parallax value calculation, the relationship between the parallax value in the pixel of interest and the parallax values in surrounding pixels is considered. Absent. In SGM, when a cost is calculated, a penalty is given to the spatial change of the parallax value, so that the parallax value in the pixel of interest easily takes the same value as the parallax value of the pixels around it. According to Patent Document 1, streaking artifacts may be generated in parallax images, that is, streak-like artifacts due to differences in parallax values between lines such as edges and boundaries, but SGM reduces streaking artifacts. Can.

  However, in the conventional parallax value calculation processing of Patent Document 1 and Non-patent Document 1, the degree of difference (that is, the Hamming distance), or the cost based on the degree of difference is specifically around the parallax value of integer pixel accuracy at which the cost is minimum. The accuracy of the obtained parallax value is low when the parallax value of the sub-pixel accuracy is calculated using the difference degree or the cost corresponding to the parallax values before and after the parallax value (the parallax value ± 1 pixel parallax value) That is, there is a problem that the error is large.

  The inventors considered the cause of the occurrence of the above problem as follows. In the parallax value calculation by the cost function defined by luminance difference such as SAD (Sum of Absolute Difference: sum of absolute value of difference of luminance value) and SSD (Sum of Squared Difference: sum of square of difference of luminance value) It is known that disparity values with sub-pixel accuracy can be calculated by performing equiangular fitting or parabola fitting around the minimum value of the cost function at integer pixel accuracy. This utilizes the fact that the spatial distribution of the cost function defined by the luminance difference can be well approximated by a linear or quadratic equation near its minimum value. This means that if, for example, the distribution of pixel values of 8 bits is regarded as a continuous amount, the difference between pixel values between corresponding points of two images can also be regarded as a continuous amount. It means that it can be well approximated by the following Taylor expansion.

  On the other hand, the Hamming distance between census feature quantities is the difference between the magnitude relation of the brightness between the pixel of interest and its peripheral pixels, which is formed into a bit string. In the vicinity of such a minimum value of the Hamming distance, there is no guarantee that the Hamming distance behaves linearly, that is, linearly or quadratically with respect to positional variation of the order of sub-pixels. In other words, it is appropriate to consider that there is no regularity in the variation of the Hamming distance at one pixel intervals, that is, the position where the reversal of the magnitude relation of brightness occurs within one pixel interval. As for the phenomenon without such regularity, if the number of reference pixels that are peripheral pixels of the pixel of interest in the census conversion is increased, the large number law will be effective, and it is expected that the property will be smooth. However, it is considered that the number of reference pixels of, for example, about 8 pixels or 64 pixels to be referred to in the census conversion is not sufficient for securing the calculation accuracy of the parallax value of the sub-pixel accuracy. Furthermore, in the vicinity of the minimum value of the Hamming distance, only a few bits in the 8-bit or 64-bit bit string have different bits, and in the vicinity of the minimum value of the Hamming distance, the large number law is the least effective.

  The present inventors verified disparity value calculation processing using SGM as described in Non-Patent Document 1 by actually calculating disparity values as shown in FIGS. 5 to 13. FIG. 5 shows a CG (Computer Graphic) image used for verification, and FIGS. 7, 9 and 11 show a photographed image used for verification. FIG. 6 shows the calculation result of the parallax value by SGM using the CG image of FIG. 5 and an image obtained by translating the CG pixels in the horizontal direction. FIG. 8 shows the calculation result of the parallax value by SGM using the photographed image of FIG. 7 and the image obtained by translating the photographed pixels in the horizontal direction. FIG. 10 shows the calculation result of the parallax value by SGM using the photographed image of FIG. 9 and an image obtained by translating the photographed pixels in the horizontal direction. FIG. 12 shows the calculation result of the parallax value by SGM using the photographed image of FIG. 11 and the image obtained by translating the photographed pixels in the horizontal direction.

  For each image, 12 cases or 16 cases of horizontal translation are performed, and the breakdown is 6 cases by the amount of movement at 0.1 pixel intervals from 1.0 pixel to 1.5 pixels, or 1 .10 cases by the movement amount at 0.1 pixel intervals from 0 pixels to 1.9 pixels, and 6 cases by the movement amount at 0.1 pixel intervals from 5.0 pixels to 5.5 pixels It is. The reference image, which is an image after parallel movement, is generated so that the pixel coordinates correspond to each other before and after movement by interpolating the pixels of the image after movement by the movement amount. The pixel coordinates are two-dimensional coordinates on the image, and are coordinates in units of pixels of the image. Then, the parallax value of the pixels of the reference image corresponding to each pixel of the reference image is calculated from the reference image and the reference image using the image before movement as the reference image. For example, when the calculated parallax value is "+1.0" pixel or "-1.0" pixel, specifically from the pixel of the reference image at the same position as the pixel of the reference image in pixel coordinates in the horizontal direction. The pixel at the position moved by one pixel in the x-axis positive direction or the x-axis negative direction of the pixel coordinates shows the same subject as the pixel of the reference image.

  In the above-described calculation of the parallax value, as described above, after the calculation of the degree of difference in parallax values with integer accuracy in SGM for each pixel of the reference image, the parallax value of the pixel giving the minimum degree of difference The parallax value is calculated with sub-pixel accuracy by the conformal fitting method using the parallax value and the dissimilarity at pixels adjacent to ± 1 pixel. In addition, the same result is obtained even when the parallax value is calculated with sub-pixel accuracy by the equiangular fitting method near the minimum value of the difference degree after the calculation of the difference degree with the parallax value with integer precision with Hamming distance. It can be easily analogized to become.

  In each of FIG. 6, FIG. 8, FIG. 10 and FIG. 12, the left column shows the correct parallax value which is the above horizontal movement amount, and the middle column shows the calculated parallax value of each pixel from the left column movement amount. The mean value (also referred to as "mean") of the difference is shown, and the right column is the root mean square ("RMS (Root mean square)" difference between the movement amount of the left column and the calculated parallax value of each pixel Also referred to as From these figures, compared with the case where the correct parallax value is in the integer pixel unit of “1.0” and “5.0”, the correct parallax value in the case of sub-pixel units other than the above is the parallax value of It can be seen that the estimation error tends to be large.

  In addition, when the calculation accuracy of the parallax value in the integer pixel unit is high and the error of the parallax value in the sub pixel unit becomes a random error, only the parallax value of the integer pixel accuracy is calculated and the parallax value in the sub pixel unit is calculated. Suppose you do not. In this case, the relationship between the correct parallax value and the calculated parallax value can be assumed as shown in FIG. That is, it is assumed that the disparity value in the sub-pixel unit is included in the hatched area in FIG. 13 hatched, and the distribution of the error in the disparity value in the sub-pixel unit becomes uniform within ± 0.5 pixel range it can. In this case, the RMS value of the disparity value error in units of sub-pixels can be obtained by the following equation 1.

  On the other hand, in FIG. 6, for the CG image of FIG. 5, the RMS value of the error of all parallax values (also referred to as the total error RMS value, specifically, 1.0 to 1.9 and 5.0 to 5.9). The RMS value in the case of the error evaluation at intervals of 0.1 pixel, where the interval from 1.6 to 1.9 and 5.6 to 5.9 is 1.1 to 1. RMS values for up to 1.4 and 5.1 to 5.4 are used in a folded form for 1.5 and 5.5)) with the integer pixel precision shown in equation 1 above. The RMS value of the error at the time of parallax value calculation is exceeded. Further, in FIGS. 8, 10 and 12, although the RMS value of the error of all parallax values for the photographed image is less than the RMS value of the error shown in the above equation 1, any RMS value is Qualitatively, the correct disparity value increases as it deviates from the integer value, indicating that the disparity value is not calculated with sub-pixel accuracy.

  As a result of examining the problems of the prior art as described in Patent Document 1 and Non-patent Document 1 as described above, the present inventors have improved the calculation accuracy of the parallax value of the subpixel accuracy between a plurality of images. In order to do this, the following technologies were devised.

  A parallax value calculation device according to an aspect of the present disclosure includes a processor and a memory, the processor including: (a) a first image captured by a first camera disposed at a first position; The second image captured by the second camera arranged at position 2 is acquired from the memory, and (b) the first image is enlarged by a predetermined parallax direction. Generating a first enlarged image including a plurality of first pixels by increasing the number of pixels of the first image, and enlarging the second image in a predetermined parallax direction Processing to generate a second magnified image including a plurality of second pixels by increasing the number of pixels of the second image; and (c) in the first magnified image, the plurality of first images. The first census feature by comparing pixel values with surrounding pixels for each pixel of And calculating a second census feature value by comparing pixel values with surrounding pixels for each of the plurality of second pixels in the second enlarged image, and (d) calculating the plurality of second plurality of The first census feature amount and the second census feature amount of the at least one second pixel are used for each of the one pixel, and the at the position of each of the at least one second pixel Calculating the degree of difference corresponding to the parallax value for the first pixel, and (e) selecting the parallax value corresponding to the smallest degree of difference using the degree of difference for each of the plurality of first pixels Acquiring a parallax value for each pixel in the parallax image corresponding to the first enlarged image, and (f) performing a reduction process on the parallax value for each pixel of the parallax image to obtain the second image Of the first image against To calculate the disparity value of Motogoto.

  According to the above aspect, the parallax value calculation device calculates the parallax value of each pixel of the parallax image corresponding to each pixel of the first enlarged image, using the first enlarged image and the second enlarged image. Furthermore, the parallax value calculation device calculates the parallax value for each pixel of the first image with respect to the second image by performing a reduction process on the parallax value of the parallax image. The disparity values as described above may be less than one pixel accurate, ie sub-pixel accurate. For example, even if the parallax value of the parallax image has integer precision, the sub-pixel precision can be obtained by reducing such a parallax value. Further, since the first enlarged image and the second enlarged image are images enlarged in the same direction, the calculation of the parallax value by the reduction process is simple. Therefore, the disparity value calculating device can easily calculate the disparity value between a plurality of images with high accuracy.

  In the parallax value calculation device according to an aspect of the present disclosure, the processor may calculate a Hamming distance between the first census feature amount and the second census feature amount as the degree of difference.

  According to the above aspect, the calculation of the degree of difference using the Hamming distance between census feature quantities is simple. Therefore, the disparity value calculating device can easily calculate the degree of difference.

  In the parallax value calculation device according to an aspect of the present disclosure, the processor may be configured to: (g) the first calculation target of the parallax value calculation in the first enlarged image between the processing (d) and the processing (e). For each of the evaluation pixels which are the first pixels located in a predetermined direction from the pixel of interest which is a pixel of the second evaluation pixel adjacent to the first evaluation pixel in the predetermined direction among the evaluation pixels For the parallax value of each of the second pixels in the search range for calculating the degree of difference with respect to the corresponding pixel at the position corresponding to the first evaluation pixel in the second enlarged image. A first evaluation value of each of the second pixels in the search range is calculated based on the difference degree between the first evaluation pixel and each of the second pixels in the search range, Each of the second pixels in the search range And calculating a second evaluation value for the parallax value of each of the second pixels in the search range, based on the first evaluation value of the first and second parallax values for each of the second pixels in the search range. The second evaluation value of all corresponding evaluation pixels is added to calculate an addition evaluation value, and in processing (e), the parallax value corresponding to the minimum addition evaluation value is used as the parallax of the pixel of interest You may decide on the value.

  According to the above aspect, the second evaluation value is an index that reflects the change in the parallax value of the pixels around the pixel of interest in the degree of difference. Since the parallax value calculation device determines the parallax value that minimizes the sum of the second evaluation values as the parallax value of the pixel of interest, the parallax value of the pixel of interest is the same value as the parallax value of the pixels around it. It becomes easy to take. Therefore, the parallax value calculation apparatus can calculate the parallax value with high accuracy by reducing the streaking artifact in which the streak-like artifact is generated due to the parallax value being different between the lines such as the edge of the subject and the boundary.

  In the parallax value calculation device according to one aspect of the present disclosure, in the processing (c), the target pixel as a comparison region that is a region of surrounding pixels to be compared when calculating a census feature amount of the target pixel. And determining the plurality of comparison regions having different positions with respect to the target pixel, and for each of the comparison regions, the first census feature value of the plurality of first pixels and the plurality of second pixels. The second census feature amount is calculated, and in the processing (d), the degree of difference is calculated for each of the comparison regions, and in the processing (e), the difference is the largest among the degrees of difference of all the comparison regions. The parallax value for each pixel in the parallax image corresponding to the first enlarged image may be acquired by selecting the parallax value corresponding to the small difference degree.

  According to the above aspect, the disparity value calculating device determines, as the disparity value of the pixel of interest, the disparity value corresponding to the smallest difference degree among the differences of the pixels of interest related to the plurality of comparison areas. Such a parallax value reduces an error of the parallax value caused by the position of the comparison area with respect to the target pixel. Therefore, the parallax value calculating device determines the position of the boundary due to the difference in the parallax value in the parallax image, and the position of the outline of the subject for the pixels located in the vicinity of the boundary between the foreground or the background where the boundary of occlusion or the parallax value changes. Can be precisely aligned.

  In the parallax value calculation device according to one aspect of the present disclosure, in the processing (d), for the first pixel and the second pixel for which the degree of difference is calculated, the processor determines the first pixel A difference between a luminance value that is a pixel value of a surrounding pixel and a luminance value that is a pixel value of the surrounding pixel of the second pixel, a luminance gradient of the surrounding pixel of the first pixel, and the second A weight based on at least one of the difference between the pixel of the pixel and the brightness gradient of the surrounding pixels may be added to the degree of difference.

  According to the above aspect, the weight based on the difference between the luminance value of the pixel around the first pixel and the luminance value of the pixel around the second pixel also corresponds to the luminance gradient of the pixel around the first pixel and The weight based on the difference with the luminance gradient of the pixels around the 2 pixels also spatially weights the inside of the comparison area even if the arrangement of the comparison area with respect to the target pixel is fixed. Thereby, the difference degree can reflect the change of the brightness value and / or the change of the brightness gradient. Therefore, the parallax value calculation apparatus can use only a single comparison area for the pixel of interest for pixels located in the vicinity of the boundary between the occlusion boundary or the boundary between the foreground and the background where the parallax value changes. The position of the boundary due to the difference in value and the position of the contour of the subject can be accurately aligned.

  A parallax value calculation method according to an aspect of the present disclosure includes: (a) a first image captured by a first camera disposed at a first position, and a second camera disposed at a second position (B) increasing the number of pixels of the first image by performing enlargement processing in a predetermined parallax direction on the first image (b). Generating a first enlarged image including a plurality of first pixels, and increasing the number of pixels of the second image by performing enlargement processing on the second image in a predetermined parallax direction Thereby generating a second enlarged image including a plurality of second pixels, and (c) comparing pixel values with surrounding pixels for each of the plurality of first pixels in the first enlarged image Calculates a first census feature amount, and the plurality of second pixels in the second enlarged image A second census feature value is calculated by comparing pixel values with surrounding pixels, and (d) the first census feature value and the at least one first census feature value for each of the plurality of first pixels. The difference degree corresponding to the parallax value with respect to the first pixel at each position of the at least one second pixel is calculated using the second census feature value of two pixels, and (e) the above The parallax value for each pixel in the parallax image corresponding to the first enlarged image is acquired by selecting the parallax value corresponding to the smallest degree of difference using the degree of difference for each of the plurality of first pixels And (f) calculating a parallax value for each pixel of the first image with respect to the second image by performing reduction processing on the parallax value for each pixel of the parallax image, and processing (a) ~ At least one of processing (f) It is performed by at least one processor. According to the above aspect, similar effects to the parallax value calculation device according to one aspect of the present disclosure can be obtained.

  In the parallax value calculation method according to an aspect of the present disclosure, a Hamming distance between the first census feature amount and the second census feature amount may be calculated as the degree of difference.

  In the parallax value calculation method according to an aspect of the present disclosure, in the process (d) and the process (e), (g) the first pixel for which a parallax value is to be calculated in the first enlarged image. For each of the evaluation pixels that are the first pixels located in a predetermined direction from the pixel of interest, among the evaluation pixels, the parallax value of the second evaluation pixel adjacent to the first evaluation pixel in the predetermined direction and The search based on a comparison of the corresponding pixel at the position corresponding to the first evaluation pixel in the second enlarged image with the parallax value of each of the second pixels in the search range for calculating the degree of difference; A first evaluation value of each of the second pixels in the range is calculated, and a degree of difference between the first evaluation pixel and each of the second pixels in the search range; A first evaluation value of each of the second pixels and Based on the second evaluation value for the parallax value of each of the second pixels in the search range, a second evaluation value is calculated, and all evaluation pixels corresponding to the parallax values of the second pixels in the search range are calculated. The second evaluation value may be added to calculate an addition evaluation value, and in processing (e), the parallax value corresponding to the minimum addition evaluation value may be determined as the parallax value of the pixel of interest.

  In the parallax value calculation method according to an aspect of the present disclosure, the target pixel is included as a comparison region that is a region of surrounding pixels to be compared when calculating a census feature value of the target pixel in the processing (c) The plurality of comparison areas having different positions with respect to the target pixel are determined, and the first census feature value of the plurality of first pixels and the second of the plurality of second pixels are determined for each of the comparison areas. In the process (d), the degree of difference is calculated for each of the comparison areas, and in the process (e), the smallest degree of difference among the degrees of difference in all the comparison areas is calculated. The parallax value for each pixel in the parallax image corresponding to the first enlarged image may be acquired by selecting the corresponding parallax value.

  In the parallax value calculation method according to an aspect of the present disclosure, in the processing (d), with respect to the first pixel and the second pixel for which the degree of difference is calculated, pixels of the surrounding pixels of the first pixel. A difference between a luminance value which is a pixel value and a luminance value which is a pixel value of the surrounding pixels of the second pixel, a luminance gradient of the surrounding pixels of the first pixel, and the luminance of the second pixel A weight based on at least one of the difference with the brightness gradient of surrounding pixels may be added to the degree of difference.

  A program according to an aspect of the present disclosure includes: (a) a first image captured by a first camera disposed at a first position, and a second image disposed at a second position; A second image, and (b) enlarging the number of pixels of the first image by enlarging the first image in a predetermined parallax direction, A first magnified image including a first pixel is generated, and the number of pixels of the second image is increased by performing enlargement processing on the second image in a predetermined parallax direction. Generating a second enlarged image including a plurality of second pixels; and (c) comparing the pixel values with surrounding pixels for each of the plurality of first pixels in the first enlarged image. Calculating a census feature amount of 1 for each of the plurality of second pixels in the second enlarged image; A second census feature value is calculated by comparing pixel values with surrounding pixels, and (d) the first census feature value and at least one of the second census feature values for each of the plurality of first pixels. The difference degree corresponding to the parallax value with respect to the first pixel at the position of each of the at least one second pixel is calculated using the second census feature value of the pixel, and (e) the plurality of pixels The parallax value for each pixel in the parallax image corresponding to the first enlarged image is acquired by selecting the parallax value corresponding to the smallest degree of difference using the degree of difference for each of the first pixels, (F) causing a computer to calculate a parallax value for each pixel of the first image with respect to the second image by performing reduction processing on the parallax value for each pixel of the parallax image . According to the above aspect, similar effects to the parallax value calculation device according to one aspect of the present disclosure can be obtained.

  Note that the above-described comprehensive or specific aspect may be realized by a recording medium such as a system, an apparatus, a method, an integrated circuit, a computer program, or a computer readable recording disk, and the system, an apparatus, a method, an integrated circuit , And may be realized by any combination of computer program and recording medium. The computer readable recording medium includes, for example, a non-volatile recording medium such as a CD-ROM. A device may also be comprised of one or more devices. When the device is configured by two or more devices, the two or more devices may be disposed in one device or may be separately disposed in two or more separate devices. In the present specification and claims, "device" may mean not only one device but also a system consisting of a plurality of devices.

  Hereinafter, the parallax value calculation device according to the present disclosure will be specifically described with reference to the drawings. Note that all the embodiments described below show general or specific examples. Numerical values, shapes, components, arrangement positions and connection forms of components, steps (steps), order of steps, and the like shown in the following embodiments are merely examples, and are not intended to limit the present disclosure. Further, among the components in the following embodiments, components not described in the independent claim indicating the highest concept are described as arbitrary components. Further, each drawing is a schematic view, and is not necessarily illustrated exactly. Further, in the drawings, substantially the same components are denoted by the same reference numerals, and overlapping descriptions may be omitted or simplified.

First Embodiment
A ranging system 1 including the parallax value calculation device 100 according to the first embodiment will be described. The distance measuring system 1 calculates the distance between the subject in the captured image and the plurality of cameras from the plurality of captured images acquired by the plurality of cameras, that is, the position of the subject in the captured image is three-dimensionally Ranging to The disparity value calculating device 100 calculates disparity values between a plurality of captured images used for distance measurement. In addition, the object with which the parallax value calculation device 100 is provided is not limited to the distance measurement system 1, and any object may be used as long as the parallax value between a plurality of captured images is used.

  FIG. 14 shows a block diagram of an example of a functional configuration of the distance measuring system 1 including the disparity value calculating device 100 according to the first embodiment. As shown in FIG. 14, the distance measurement system 1 includes an imaging unit 10, a storage unit 20, and an image processing unit 30. The imaging unit 10, the storage unit 20, and the image processing unit 30 may be mounted on one device, or may be separately mounted on a plurality of devices. In the latter case, the plurality of devices may exchange information via wired communication or wireless communication. Any existing wired communication may be applied to the wired communication. Any existing wireless communication may be applied to the wireless communication. For example, a wireless LAN (Local Area Network) such as Wi-Fi (registered trademark) (Wireless Fidelity) may be applied to the wireless communication, and a short distance such as Bluetooth (registered trademark) or ZigBee (registered trademark) Wireless communication may be applied.

  The imaging unit 10 captures and acquires an image, and outputs the image to the storage unit 20. The imaging unit 10 includes at least two cameras, and in the present embodiment, includes two cameras 11 a and 11 b arranged at different optical axis centers. In the present embodiment, the cameras 11a and 11b are cameras that capture digital images whose pixel values of the respective pixels are luminance values as moving images and / or still images, but are cameras that capture any digital image. It is also good. The cameras 11a and 11b may constitute separate cameras, or may be integrated to constitute a compound eye camera. The number of cameras is not limited to two, and may be any number of two or more. In the present embodiment, the cameras 11a and 11b simultaneously perform imaging.

  For example, FIG. 15 is a schematic perspective view showing an arrangement example of the cameras 11 a and 11 b of the imaging unit 10. Furthermore, FIG. 16 shows an example of the base lengths of the cameras 11a and 11b of FIG. 15 in a schematic front view. In the example of FIG. 15, the cameras 11a and 11b are arranged side by side in the horizontal direction. In this case, the base line BL connecting the centers of the lenses of the cameras 11a and 11b extends in the horizontal direction. Furthermore, the optical axis centers OAa and OAb of the cameras 11a and 11b are horizontally separated from each other, horizontally aligned and parallel, and perpendicular to the direction of the base line BL. In this case, as shown in FIG. 16, the base line length L which is the distance between the optical axis centers OAa and OAb of the cameras 11a and 11b is larger than zero. Furthermore, the cameras 11a and 11b are oriented in the same direction D, for example, in the example of FIG. 15, the optical axis centers OAa and OAb extend horizontally and are parallel to the direction D. Furthermore, the horizontal scanning direction of the cameras 11a and 11b is parallel to the direction of the base line BL. The horizontal scanning direction is the horizontal direction among the vertical and horizontal alignment directions of the plurality of imaging elements of the camera, and is the same as the direction of the base line BL. The plurality of imaging elements are, for example, planarly arranged in a grid shape in two directions of a horizontal direction which is a horizontal direction and a vertical direction which is a vertical direction perpendicular thereto. Here, the camera 11a is an example of a first camera disposed at a first position, and the camera 11b is an example of a second camera disposed at a second position.

  The arrangement of the cameras 11a and 11b is not limited to the arrangement in the horizontal direction as shown in FIG. 15. For example, as shown in FIG. 17, the arrangement may be the arrangement in the vertical direction, that is, the vertical direction. FIG. 17 is a schematic perspective view showing another arrangement example of the cameras 11 a and 11 b of the imaging unit 10. In this case, the base line BL extends in the vertical direction. Optical axis centers OAa and OAb of the cameras 11a and 11b are vertically separated from each other, aligned in parallel in the vertical direction, and parallel to the direction of the base line BL. Furthermore, the cameras 11a and 11b are oriented in the same direction D, for example, in the example of FIG. 17, the optical axis centers OAa and OAb extend horizontally and are parallel to the direction D. Furthermore, the horizontal scanning direction of the cameras 11a and 11b is perpendicular to the direction of the base line BL.

  The cameras 11a and 11b as described above constitute a stereo camera capable of picking up the same subject, and the images picked up by such cameras 11a and 11b constitute stereo images.

  In the present specification and claims, "parallel" and "perpendicular" can mean not only the cases that are completely parallel and perpendicular, respectively, but also the cases that have differences with respect to completely parallel and perpendicular. The difference may be, for example, an angular difference of less than 10 °.

  The storage unit 20 enables storage and retrieval of various information. For example, the storage unit 20 stores the captured image acquired by the imaging unit 10. The storage unit 20 may store a program for operating the imaging unit 10 and / or the image processing unit 30. These programs may be stored in a memory (not shown) that the imaging unit 10 and the image processing unit 30 have. The storage unit 20 is realized by, for example, a storage device such as a read-only memory (ROM), a random access memory (RAM), a semiconductor memory such as a flash memory, a hard disk drive, or a solid state drive (SSD). Here, the storage unit 20 is an example of a memory.

  The image processing unit 30 includes a parallax value calculation device 100, a distance calculation unit 31, and an output unit 32. Furthermore, the parallax value calculation device 100 includes a horizontal enlargement unit 101, a census conversion unit 102, a dissimilarity calculation unit 103, a parallax selection unit 104, and a horizontal reduction unit 105. Each component of the image processing unit 30 and the parallax value calculation device 100 described above is a computer system including a processor such as a central processing unit (CPU) or a digital signal processor (DSP), and a memory such as a RAM and a ROM. Not shown). Some or all of the functions of each component may be achieved by the CPU or DSP executing a program recorded in the ROM using the RAM as a working memory. Also, some or all of the functions of each component may be achieved by dedicated hardware circuits such as electronic circuits or integrated circuits. Some or all of the functions of each component may be configured by a combination of the above software functions and hardware circuits. The program may be provided as an application through communication via a communication network such as the Internet, communication according to a mobile communication standard, another wireless network, a wired network, broadcast, or the like.

  The parallax value calculation device 100 calculates the distance between various positions on the pickup image, specifically, the parallax between the pickup images in each pixel, using the corresponding pickup images acquired by the cameras 11a and 11b, and calculates the distance. Output to section 31. An example of the corresponding captured image is an image captured at the same time by the cameras 11a and 11b. The distance calculation unit 31 uses the triangulation technique based on the parallax between the images taken by the cameras 11a and 11b to be taken out at various positions on the taken image, specifically, the pixels in each pixel. The distance between the subject and the cameras 11 a and 11 b is calculated and output to the output unit 32. The distance calculation unit 31 calculates the distance between the subject and the base line BL of the cameras 11a and 11b. The distance calculation unit 31 may generate a distance image in which the pixel value of each pixel is the distance from the optical center (OAa or OAb in FIGS. 15 and 17) of the reference camera (camera 11a or camera 11b). The output unit 32 associates the distance of each pixel of the captured image calculated by the distance calculation unit 31 and the distance information such as the distance image with the captured image, and stores the storage unit 20 and the device including the distance measuring system 1 Output to For example, when the ranging system 1 is mounted on a vehicle, the output unit 32 may output distance information to an ECU (Electronic Control Unit) of the vehicle.

  The horizontal enlargement unit 101 of the parallax value calculation device 100 sets the captured images I1 and I2 of the cameras 11a and 11b, which are input images, and the initial parallax image Dinit in the horizontal parallax direction, which is a predetermined parallax search direction. Expand, that is, expand horizontally. The parallax search direction is an arrangement direction of pixels in a search range described later. The captured images I1 and I2 are horizontally enlarged to enlarged images I1h and I2h, respectively. The initial parallax image Dinit is an image including the initial value of the parallax value calculated in advance based on the camera parameters of the cameras 11a and 11b as the pixel value of each pixel. The initial value of such a parallax value may be the same for all pixels. The parallax value of all the pixels of the initial parallax image Dinit may be zero. Examples of camera parameters are the position and orientation of the camera, as well as the focal length and center position of the optical axis. The parallax search direction is a direction in which pixels are compared between the images of the cameras 11a and 11b when calculating parallax values, and in the present embodiment, it is a one-dimensional search direction in the horizontal direction. In this case, when the parallax value is calculated based on the image of the camera 11a, with respect to the pixel of interest which is one pixel of the image, in the image of the camera 11b, the reference pixels in the same line as the pixel of interest are in the horizontal direction. That is, the search is sequentially performed in the line direction and compared with the target pixel. The same applies to the case where not the image of the camera 11a but the image of the camera 11b is used as a reference. Here, the captured image I1 is an example of a first image, the captured image I2 is an example of a second image, and the parallax search direction is an example of a predetermined parallax direction.

  In the horizontal enlargement process, the number of pixels in the horizontal direction is increased, but the number of pixels in the vertical direction does not change. That is, the number of pixel columns increases and the number of pixel rows is constant. Horizontal enlargement processing is performed by interpolation enlargement. Although a plurality of interpolation expansion methods are known, in the present embodiment, it is preferable to perform bilinear interpolation rather than bicubic interpolation which is known to have a high restoration image quality. The reason is that the overshoot component superimposed in the bicubic interpolation may become an error factor of the difference calculation using the census conversion result described later.

  The census conversion unit 102 performs census conversion on the enlarged images I1h and I2h. Census conversion is processing of converting the distribution of the pixel values of peripheral pixels around the pixel of interest, that is, the luminance value, into a bit string based on the magnitude relationship between the pixel of interest and the peripheral pixels around it. Specifically, for example, when the pixel value of the target pixel is larger than the pixel values of the peripheral pixels, the census conversion value of the peripheral pixels is 1, and the pixel value of the target pixel is less than or equal to the pixel values of the peripheral pixels. The census conversion value of the peripheral pixels is zero. The census conversion unit 102 performs census conversion on all the pixels of each of the enlarged images I1h and I2h.

  For example, FIGS. 18A and 18B show an example of peripheral pixels in census conversion. 19A and 19B respectively show an example of the census conversion of the peripheral pixels in FIGS. 18A and 18B. In FIGS. 18A, 18B, 19A, and 19B, ctr represents a target pixel. FIGS. 18A and 19A are examples in which eight pixels in the vicinity of the pixel of interest are peripheral pixels, and FIGS. 18B and 19B are examples in which 64 pixels in the vicinity of the pixel of interest are peripheral pixels. In the examples of FIGS. 19A and 19B, the distribution of pixel values of peripheral pixels around the pixel of interest can be represented by a bit string of 8 bits or 64 bits by census conversion, respectively. Specifically, FIG. 19A and FIG. 19B respectively show an example of pixel values of peripheral pixels and a pixel of interest before census conversion, an example of census conversion values of peripheral pixels, and an example of a bit string consisting of census conversion values. .

  For example, in FIG. 18A, FIG. 18B, FIG. 19A and FIG. 19B, the x axis of pixel coordinates is defined in the horizontal direction in the drawing, the y axis of pixel coordinates is defined in the vertical direction of the horizontal direction, Direction The x-axis positive direction is defined as the right direction, and the y-axis positive direction is defined as the vertically downward direction. The x-axis extends in a direction along the base line BL of the cameras 11a and 11b. The bit string includes, for example, census transformed values of peripheral pixels with y coordinate minimum and x coordinate minimum as the first element, that is, bits, and further, census transformed values of peripheral pixels in the positive direction of x axis with the same y coordinate are sequentially It is included as an element, and then, while the y-coordinate is incremented one by one, census transformation values of peripheral pixels in the positive direction of the x-axis whose y-coordinates are the same are sequentially included as an element. For example, in the example of FIG. 19A, the bit string is indicated by (00011011). In this way, bit strings are generated for all the pixels of each of the enlarged images I1h and I2h. The number and arrangement of peripheral pixels need not be limited to the numbers and arrangements shown in FIGS. 18A and 18B, and may be other numbers and arrangements. Here, the above bit string composed of the census conversion value constitutes census feature value.

  The degree-of-difference calculation unit 103 calculates the degree of difference in census feature value between the enlarged images I1h and I2h for pixels in a predetermined search range. In the present embodiment, the Hamming distance is used as the degree of difference between census feature quantities. Here, the search range will be described. The search range includes at least one pixel. As shown in FIG. 15, in the present embodiment, the camera 11a is disposed on the left side of the camera 11b toward the direction D, which is the imaging direction, to constitute a left camera, and the camera 11b constitutes a right camera. Do.

A search range in the case where the magnified image I1h corresponding to the captured image I1 of the camera 11a is a reference image and the magnified image I2h corresponding to the captured image I2 of the camera 11b is a reference image will be described. In this case, as shown in FIG. 20, for example, all the pixels in the search range on the reference image I2h (hereinafter referred to as “reference pixels”) on the reference image I1h (hereinafter also referred to as “focused pixels”) The degree of difference between census feature quantities is determined between each of I 2 hp _{1 to I 2} hp _n . FIG. 20 is a diagram illustrating an example of a search range of pixels in the calculation of the degree of difference. The reference pixels I2hp _{1 to} I2hp _n form a row of pixels having the same y coordinate as the pixel of interest I1hp, and the row of pixels forms a search range. Such a search range is a one-dimensional search range in pixel coordinates, and is all rows of one pixel in the reference image I2h.

Alternatively, the pixel of interest I1hp is included in the census feature value with each of k + 1 (k is a positive integer) reference pixels including the corresponding pixel I2hp _m having the same pixel coordinates as the pixel of interest I1hp on the reference image I2h. The degree of difference may be determined. Although the pixel coordinates of the target pixel I1hp and the pixel coordinates of the corresponding pixel I2hp _m differ depending on the parallax value at the target pixel I1hp, in the present embodiment, an initial parallax indicating the initial value of the parallax value of each pixel The parallax value of each pixel in the image is zero. For this reason, the pixel coordinates of the focused pixel I1hp and the pixel coordinates of the corresponding pixel I2hp _m are the same. Such reference pixels, the corresponding pixel I2hp _m and y-coordinates are the same and the corresponding pixel I2hp _m k + 1 pixels in the x-axis negative direction from, that is, from the corresponding pixel I2hp _m pixels I2hp _{m-k k} + 1 single It is a pixel and constitutes a search range. Such a search range is a one-dimensional search range, and in a row of pixels including the corresponding pixel I2hp _m of the reference image I2h, in the range of k + 1 pixels from the corresponding pixel I2hp _m toward the camera 11a along the base line BL. is there. In the following embodiment, a case in which k + 1 pixels are set as a search range will be described.

The parallax value of the corresponding pixel I2hp _m with respect to the focused pixel I1hp is 0 pixel, and the parallax value of the pixel I2hp _{m-k with} respect to the focused pixel I1hp is −k pixels. For this reason, the search range is also a range in which the parallax value in the horizontal direction with respect to the target pixel I1hp is 0 pixel to −k pixel. In addition, although the said search range is a range whose parallax value is zero or less, it is not limited to this, You may also include the range whose parallax value is more than zero. In the present embodiment, when the parallax value is a positive value, it indicates parallax in the x-axis positive direction, and when it is a negative value, it indicates parallax in the x-axis negative direction.

Similarly, a search range in the case where the enlarged image I2h is a reference image and the enlarged image I1h is a reference image will be described. In this case, as shown in FIG. 21, the degree of difference in census feature value between the target pixel I2hp on the reference image I2h and each of all the reference pixels I1hp _{1 to} I1hp _n within the search range on the reference image I1h. Is required. FIG. 21 is a diagram showing another example of the search range of the pixel at the time of calculation of the degree of difference. Such a search range is all of one row of pixels in the reference image I1h, and the y coordinate of this row is the same as the target pixel I2hp.

Alternatively, for the pixel of interest I2hp, the degree of difference in census feature value may be determined between each of k + 1 reference pixels including the corresponding pixel I1hp _m on the reference image I1h. Such a reference pixel has the same y coordinate as the corresponding pixel I1hp _m and k + 1 pixels in the positive direction along the x axis from the corresponding pixel I1hp _m , that is, k + 1 pixels from the corresponding pixel I1hp _m to the pixel I1hp _{m + k} Yes, configure the search range. Such a search range is a range of k + 1 pixels from the corresponding pixel I1hp _m toward the camera 11b along the base line BL in the row of pixels including the corresponding pixel I1hp _m of the reference image I1h. Further, the search range is also a range in which the parallax value in the horizontal direction with respect to the target pixel I2hp is 0 pixel to k pixels. The above search range is a range in which the disparity value is 0 or more, but is not limited to this, and may include a range in which the disparity value is less than 0.

  Moreover, the difference degree using Hamming distance is calculated | required as follows. For example, when the census feature value of the target pixel of the reference image is bit string (00011011) and the census feature value of reference pixel of the reference image is bit string (10110001), the Hamming distance between these two pixels is 1 + 0 + 1 + 0 + 1 + 0 + 1 + 0. It is = 4, and a difference degree is "4". The degree of difference is the Hamming distance, and the number of bits of the same digit is different between bit strings which are census features of two pixels. That is, in the calculation of the Hamming distance, when the bit numbers of the same digit are the same between two bit strings, the inter-bit difference is “0”, and when the bit numbers of the same digit are different, the inter-bit difference is “1”. And the sum of all inter-bit differences is the Hamming distance.

  As described above, the dissimilarity calculation unit 103 uses the census feature of the pixel of interest and the census feature of at least one reference pixel of the reference image for each pixel of interest of the reference image to perform at least one reference pixel. The difference degree corresponding to the parallax value with respect to the pixel of interest at the position of is calculated.

  The parallax selection unit 104 selects the smallest degree of difference from among the degrees of difference of reference pixels in the corresponding search range for the pixel of interest, and specifies the parallax value of the reference pixel corresponding to the degree of difference. Then, the disparity selection unit 104 determines the identified disparity value as the disparity value of the pixel of interest. The parallax selection unit 104 sequentially determines all pixels in the reference image as the target pixel, thereby determining the parallax value of each pixel in the reference image. Thereby, the parallax selection unit 104 generates a parallax image in which the pixel value of each pixel is a parallax value. The parallax image is an image in which a pixel value including a luminance value in each pixel of the reference image is replaced with a parallax value.

  Specifically, the determination of the parallax value of the pixel of interest is processed as described below. For example, FIG. 22 illustrates an example of the relationship between the degree of difference calculated for one target pixel and the parallax value of the reference pixel corresponding to the degree of difference. As described above, the degree of difference is calculated at an interval of one pixel within a search range determined by the minimum value and the maximum value of the preset parallax values. Then, the parallax selection unit 104 selects the smallest degree of difference from the degrees of difference within the search range, and determines the parallax value corresponding to the degree of difference as the parallax value of the pixel of interest.

  The horizontal reduction unit 105 reduces the parallax value of each pixel of the reference image determined by the parallax selection unit 104 at a horizontal reduction ratio corresponding to the horizontal enlargement ratio in the horizontal enlargement unit 101 to obtain a scale before horizontal enlargement. Convert to disparity value. For example, when the horizontal enlargement ratio in the horizontal enlargement unit 101 is twice, the horizontal reduction unit 105 performs horizontal reduction processing at a horizontal reduction ratio of 1⁄2. For example, when the horizontal enlargement ratio in the horizontal enlargement unit 101 is four, the horizontal reduction unit 105 performs horizontal reduction processing at a horizontal reduction ratio of one fourth.

  Specifically, the horizontal reduction unit 105 reduces the parallax image generated by the parallax selection unit 104 at the above-described horizontal reduction ratio. For example, when the horizontal reduction ratio is 1⁄2, the horizontal reduction unit 105 averages the pixel values of two pixels aligned in the horizontal direction, that is, the x-axis direction, selects one of the pixel values of two pixels, etc. By combining the two pixels, the parallax image is reduced so that the number of pixels is halved. Furthermore, the horizontal reduction unit 105 halves the pixel value of each pixel after integration, that is, the parallax value. Thereby, the horizontal reduction unit 105 generates a half parallax image in which the parallax image is reduced by half as a reduced parallax image. Each pixel of such a 1/2 parallax image includes a parallax value with half pixel accuracy.

  Therefore, in the case of 2 × horizontal enlargement by the horizontal enlargement unit 101 and 1⁄2 horizontal reduction by the horizontal reduction unit 105, it is possible to calculate a parallax value with 1⁄2 pixel accuracy. In the case of 1⁄4 horizontal enlargement and horizontal 1⁄4 horizontal reduction by the horizontal reduction unit 105, it is possible to calculate parallax values with 1⁄4 pixel accuracy.

  In the present embodiment, it is assumed that the arrangement of the left and right cameras 11a and 11b is appropriate. That is, as shown in FIG. 15, the optical axis centers OAa and OAb of the cameras 11a and 11b are perpendicular to the direction of the base line BL, and the optical axis centers OAa and OAb are parallel to each other and the cameras 11a and 11b. It is assumed that the horizontal scanning direction and the base line BL direction are parallel and that there is no image distortion in the cameras 11a and 11b. The corresponding point in the reference image corresponding to the point of interest in the reference image exists in the horizontal line in the x-axis direction in which the y coordinate of the pixel coordinates of the point of interest and the y coordinate of the pixel coordinates are the same. The imaging of stereo images by the arrangement of the cameras 11a and 11b is called parallel imaging. When the arrangement of the cameras 11a and 11b deviates from the arrangement of parallel imaging, or distortion of the captured image is so large that it can not be ignored, distortion correction and image distortion are performed before or after horizontal enlargement is performed by the horizontal enlargement unit 101. By performing the parallelization process, the image may be corrected so as to be an image obtained by parallel imaging without distortion. Further, as shown in FIG. 17, when the cameras 11a and 11b are arranged apart in the vertical direction, the corresponding point in the reference image corresponding to the focused point in the reference image is the x coordinate of the pixel coordinates of the focused point. And the x-coordinates of the pixel coordinates are in the same vertical column in the y-axis direction.

  Furthermore, the operation of disparity value calculation apparatus 100 according to the first embodiment will be described. FIG. 23 is a flowchart showing an example of the operation of the disparity value calculating device 100 according to Embodiment 1. First, in step S101, the horizontal enlargement unit 101 obtains, from the storage unit 20, captured images I1 and I2 captured by the cameras 11a and 11b, respectively. The captured images I1 and I2 are images that capture the same subject, and are images captured by the cameras 11a and 11b at the same time. That is, the captured images I1 and I2 are stereo images.

  Next, in step S102, the horizontal enlargement unit 101 enlarges the captured images I1 and I2 by enlarging each of the captured images I1 and I2 in the x-axis direction of the pixel coordinates that is the parallax search direction, that is, in the horizontal direction. Generate I1h and I2h. The horizontal enlargement unit 101 performs horizontal enlargement processing by increasing the number of pixels by interpolation enlargement. In the present embodiment, bilinear interpolation is used as a method of interpolation enlargement.

  Next, in step S103, the census conversion unit 102 performs census conversion on each of the enlarged images I1h and I2h. The census conversion unit 102 performs census conversion on all the pixels of each of the enlarged images I1h and I2h, and calculates census feature quantities of the respective pixels. For example, as shown in FIGS. 18A, 18B, 19A, and 19B, the census feature value is a bit string corresponding to the peripheral pixels of the pixel of interest subjected to census conversion, and is also referred to as a “sensor vector”.

  Next, in step S104, the degree-of-difference calculation unit 103 calculates the degree of difference with respect to the other reference image of the enlarged images I1h and I2h for each pixel of one of the enlarged images I1h and I2h. In the present embodiment, the degree-of-difference calculation unit 103 determines the enlarged image I1h corresponding to the camera 11a located on the left side as a reference image and determines the enlarged image I2h as a reference image as shown in FIG. It is not limited to this. The determination of the reference image and the reference image may be performed in accordance with a predetermined rule, or may be performed in accordance with a command from the user via an input device (not shown). The degree-of-difference calculation unit 103 determines, for example, the census feature value between the reference pixels in the search range corresponding to the pixel of interest in the reference image I2h and the pixel of interest for the pixel of interest in the reference image I1h. Calculate the degree of difference of The dissimilarity calculation unit 103 calculates the dissimilarity using all pixels of the reference image I1h as a pixel of interest. In the present embodiment, the degree of difference is the Hamming distance between census feature quantities of the pixel of interest and the reference pixel. For example, when the number of reference pixels in the search range is k + 1, k + 1 differences are calculated for one target pixel.

Next, in step S105, the parallax selection unit 104 determines, for each pixel of the reference image I1h, the parallax value of the pixel based on the calculated degree of difference. For example, as illustrated in FIG. 22, the parallax selection unit 104 extracts the minimum difference degree from the difference degree corresponding to the target pixel of the reference image I1h, and generates the parallax between the reference pixel corresponding to the minimum difference degree and the target pixel. Identify the value. The parallax value can be calculated, for example, from the position of the reference pixel with respect to the corresponding pixel I2hp _m of the reference image I2h corresponding to the target pixel I1hp as shown in FIG. The parallax selection unit 104 determines the parallax value of all the pixels in the reference image I1h by sequentially scanning all the pixels as a target pixel in the reference image I1h. Thereby, the parallax selection unit 104 generates a parallax image of the reference image I1h.

  Next, in step S106, the horizontal reduction unit 105 generates a reduced parallax image by reducing the parallax image at the horizontal reduction ratio according to the horizontal enlargement ratio in step S102. At this time, the horizontal reduction unit 105 generates a reduced parallax image having the same number of pixels as the captured image I1 by integrating a plurality of pixels adjacent in the horizontal direction, that is, in the x-axis direction in the parallax image. When integrating a plurality of pixels, the horizontal reduction unit 105 reduces the parallax value, which is the pixel value, at the horizontal reduction rate. Thus, the reduced parallax image includes, as pixel values, parallax values at the scale before horizontal enlargement. For example, when the horizontal enlargement ratio in step S102 is double, the horizontal reduction ratio is half, and when the horizontal enlargement ratio is four, the horizontal reduction ratio is quarter. The reduced parallax image includes parallax values with half pixel accuracy in the former case, and includes parallax values with quarter pixel accuracy in the latter case.

  If the arrangement of the left and right cameras 11a and 11b is not appropriate, distortion correction and parallelization processing of an image may be performed before step S102 or step S103. Also, at that time, considering the movement of the image center (the position in the image of the optical axis centers OAa and OAb in FIGS. 15 and 17) accompanying the parallelization, the initial value of the parallax value in the initial parallax image is a value other than 0. It may be set to

  As described above, the parallax value calculation device 100 according to the first embodiment acquires the captured image I1 captured by the camera 11a and the captured image I2 captured by the camera 11b from the storage unit 20, and captures the captured image I1. On the other hand, the magnified image I1h is generated by increasing the number of pixels by the enlargement process in the predetermined parallax direction, and the enlargement process in the predetermined parallax direction is performed on the captured image I2. An enlarged image I2h is generated by increasing the number of pixels. Furthermore, the parallax value calculation device 100 calculates a first census feature amount for each of a plurality of first pixels of the enlarged image I1h, and a second census feature amount for each of a plurality of second pixels of the enlarged image I2h. The position of each of at least one second pixel is calculated using the first census feature and the second census feature of each at least one second pixel for each of the plurality of first pixels. The degree of difference corresponding to the disparity value at is calculated. The parallax value calculation device 100 selects, for each of the plurality of first pixels, the parallax value corresponding to the smallest degree of difference using the degree of difference, whereby the parallax for each pixel in the parallax image corresponding to the enlarged image I1 h The parallax value of each pixel of the captured image I1 with respect to the captured image I2 is calculated by obtaining a value and performing reduction processing on the disparity value of each pixel of the parallax image.

  According to the above configuration, the parallax value calculation device 100 calculates the parallax value of each pixel of the parallax image corresponding to each pixel of the enlarged image I1h, which is the reference image, using the enlarged images I1h and I2h with integer accuracy. Since the magnified images I1h and I2h are magnified in the same direction, the calculation of the parallax value is easy. The parallax value obtained by performing the reduction process on the parallax image as described above has an accuracy of less than one pixel, that is, sub-pixel accuracy. Therefore, the disparity value calculating device 100 can calculate disparity values between a plurality of images with high accuracy. More specifically, the disparity value calculating device 100 can calculate sub-pixel accuracy disparity values in the vicinity of disparity values of integer pixel accuracy at which the degree of difference which is the Hamming distance is minimum. The same applies to the case where the enlarged image I2h is a reference image.

  Further, in the parallax value calculation device 100 according to the first embodiment, the degree of difference is the Hamming distance between the first census feature amount and the second census feature amount, so that the calculation of the degree of difference is easy. . Therefore, the disparity value calculating device 100 can calculate disparity values between a plurality of images with high accuracy and easily.

Second Embodiment
The disparity value calculating device 100 according to the first embodiment determines one of the enlarged images I1h and I2h corresponding to the cameras 11a and 11b as a reference image, determines the other as a reference image, and sets disparity values corresponding to pixels of the reference image. Calculated. The disparity value calculating device 200 according to Embodiment 2 determines the enlarged images I1h and I2h as reference images, and checks the consistency of disparity values by comparing the parallax values corresponding to the reference images. That is, the parallax value calculation device 200 according to Embodiment 2 calculates bidirectional parallax values in order to enhance the reliability of the parallax value calculation results, and checks the consistency based on the comparison of the parallax value calculation results. By doing this, the calculation accuracy of the parallax value is improved. In Embodiment 2, about the component similar to Embodiment 1, the same referential mark as Embodiment 1 is attached | subjected, and the description is abbreviate | omitted. In the following, differences from the first embodiment will be mainly described, and description of the same points as the first embodiment will be omitted.

  FIG. 24 shows a block diagram of an example of a functional configuration of the disparity value calculating device 200 according to Embodiment 2. As shown in FIG. 24, the disparity value calculating device 200 includes a horizontal enlargement unit 101, a census conversion unit 102, a dissimilarity calculation unit 103, a first disparity selection unit 204a, a second disparity selection unit 204b, and L. / R check unit (also referred to as "left / right check unit") 206 and a horizontal reduction unit 105.

  The dissimilarity calculating unit 103 calculates the dissimilarity of each pixel of the reference image using each of the enlarged images I1h and I2h corresponding to the captured images I1 and I2 of the cameras 11a and 11b as a reference image. Specifically, the degree-of-difference calculation unit 103 searches for each pixel in the search range of the enlarged image I2h which is the first reference image with respect to each pixel in the enlarged image I1h which is the first reference image, that is, searching The degree of difference of the census feature value is calculated for each parallax value in the range. Similarly, with respect to each pixel in the enlarged image I2h which is the second reference image, the difference degree calculation unit 103 determines the difference in census feature value with respect to each pixel in the search range of the enlarged image I1h which is the second reference image. Calculate the degree.

  The first disparity selection unit 204a extracts the smallest difference degree from among the difference degrees of the pixels for the difference degree calculated for each pixel of the enlarged image I1h, and gives the minimum difference degree of the enlarged image I2h Identify the disparity value of the pixel. Then, the first parallax selection unit 204a generates a parallax image of the enlarged image I1h as a parallax image of the first reference image.

  The second parallax selection unit 204b extracts the smallest difference degree from among the difference degrees of the pixels for the difference degree calculated for each pixel of the enlarged image I2h, and gives the minimum difference degree of the enlarged image I1h Identify the disparity value of the pixel. Then, the second parallax selection unit 204b generates a parallax image of the enlarged image I2h as a parallax image of the second reference image.

  The L / R check unit 206 checks consistency between the parallax image of the first reference image and the parallax image of the second reference image. That is, the L / R check unit 206 checks the consistency between the parallax image of the enlarged image I1h and the parallax image of the enlarged image I2h. The L / R check unit 206 uses the following Equation 2 to check the consistency.

  In Equation 2 above, d1 is the parallax value of the target pixel of the enlarged image I1h. x 'indicates the x coordinate of the pixel in the enlarged image I1h after horizontal enlargement, and (x', y) is the pixel coordinate of the target pixel. x '+ d1 (x', y) indicates the x coordinate of a parallax corresponding pixel which is a pixel of the enlarged image I2h at a position corresponding to the parallax value of the focused pixel of the enlarged image I1h. d2 is a parallax value of the parallax corresponding pixel of the enlarged image I2h. x ′ + d1 (x ′, y) + d2 (x ′ + d1 (x ′, y), y) is the x coordinate of the pixel of the enlarged image I1h at a position corresponding to the parallax value of the parallax corresponding pixel of the enlarged image I2h Show. Such equation 2 represents the difference between the parallax value based on the enlarged image I1 h and the parallax value based on the enlarged image I2 h. When the value of Equation 2 is equal to or less than a predetermined threshold, the reliability of the parallax value of the pixel of interest of enlarged image I1h is high, and the parallax value based on enlarged image I1h and the parallax value based on enlarged image I2h It can be regarded as consistent with. If the value of Equation 2 is larger than a predetermined threshold, the reliability of the parallax value of the target pixel of the enlarged image I1h is low, and it can be considered that the above-mentioned consistency is not present. The threshold is predetermined according to the degree of consistency required. An example of the threshold is one pixel.

  The L / R check unit 206 determines and outputs the parallax value d1 as the parallax value of the pixel of interest (x ', y) of the enlarged image I1h as it is, when the value of the equation 2 is equal to or less than a predetermined threshold. On the other hand, when the value of Equation 2 above is larger than the threshold value, the L / R check unit 206 detects the disparity value of the pixel of interest (x ′, y) of the enlarged image I1h, for example, the disparity value included in the search range Change to a specific value other than and output. This particular value is a value indicating that the disparity values are not consistent. For example, as shown in FIG. 20, when the search range is from 0 pixel to −k pixel, the above specific value may be a value less than −k pixel. As described above, the L / R check unit 206 generates a new parallax image by maintaining or changing the parallax value of each pixel in the parallax image of the enlarged image I1h based on the presence or absence of consistency.

  The horizontal reduction unit 105 reduces the new parallax image generated by the L / R check unit 206 at a horizontal reduction ratio according to the horizontal enlargement ratio in the horizontal enlargement unit 101, and the same number of pixels as the captured image before horizontal enlargement. To generate a reduced parallax image of Specifically, the horizontal reduction unit 105 reduces a new parallax image generated from the parallax image of the enlarged image I1h. When integrating a plurality of pixels, the horizontal reduction unit 105 reduces the parallax value of the pixel having a consistent parallax value at the above horizontal reduction ratio, and determines the pixel value of the pixel after integration. The parallax value of the pixel after such integration indicates the parallax value at the scale before horizontal enlargement. On the other hand, the horizontal reduction unit 105 does not reduce the above specific value indicating that the parallax value is not consistent with respect to the parallax value of the pixel that is not consistent in parallax value, and does not reduce the pixel of the pixel after integration. Determine the value. Accordingly, in the reduced parallax image output by the horizontal reduction unit 105, the pixel having consistency of parallax values includes the calculated parallax value of sub-pixel accuracy as a pixel value, and the pixel having inconsistency of parallax values is: An exception value indicating consistency or low reliability is included as the pixel value.

  The operation of the disparity value calculating device 200 will be described with reference to FIG. FIG. 25 is a flowchart showing an example of the operation of the disparity value calculating device 200 according to Embodiment 2. The disparity value calculating device 200 performs the processes of steps S101 to S103 as in the first embodiment.

  Next, in step S201, the dissimilarity calculating unit 103 calculates the dissimilarity to the enlarged image I2h, which is the first reference image, for each pixel of the enlarged image I1h, using the enlarged image I1h as a first reference image. Furthermore, the dissimilarity calculation unit 103 calculates the dissimilarity to the magnified image I1h, which is the second reference image, for each pixel of the magnified image I2h, using the magnified image I2h as a second reference image.

  Next, in step S202, the first parallax selection unit 204a determines the parallax value of the pixel based on the calculated degree of difference for each pixel of the enlarged image I1h that is the first reference image. The first parallax selection unit 204a generates a parallax image of the enlarged image I1h, that is, the first reference image by replacing the luminance value of each pixel in the enlarged image I1h with a parallax value.

  Next, in step S203, the second parallax selection unit 204b determines the parallax value of the pixel based on the calculated degree of difference for each pixel of the enlarged image I2h that is the second reference image. The second parallax selection unit 204b generates a parallax image of the enlarged image I2h, that is, the second reference image by replacing the luminance value of each pixel in the enlarged image I2h with a parallax value. The order of steps S202 and S203 may be reversed.

  Next, in step S204, the L / R check unit 206 checks, for each pixel of the parallax image of the enlarged image I1h which is the first reference image, the consistency of the parallax value with the parallax image of the enlarged image I2h. . The L / R check unit 206 maintains the parallax value of each pixel in the parallax image of the enlarged image I1h when there is consistency, and changes the parallax value to an exceptional parallax value when there is no consistency. Generate a new parallax image.

  Next, in step S205, the horizontal reduction unit 105 generates a reduced parallax image by reducing the new parallax image generated in step S204 at the horizontal reduction ratio according to the horizontal enlargement ratio in step S102. When integrating a plurality of pixels, the horizontal reduction unit 105 reduces the parallax value of the pixel having a consistent parallax value at the above horizontal reduction ratio, and determines the pixel value of the pixel after integration. The horizontal reduction unit 105 determines the exceptional parallax value set for the pixel having no consistency of parallax value as the pixel value of the pixel after integration without reducing the parallax value.

  According to the disparity value calculating device 200 according to the second embodiment as described above, the same effect as that of the first embodiment can be obtained. Furthermore, the parallax value calculation device 200 sets the parallax value of each pixel of the enlarged image I1h when the enlarged image I1h of the captured image I1 of the camera 11a is a reference image and the enlarged image I2h of the captured image I2 of the camera 11b is a reference image. Calculate Furthermore, the parallax value calculation device 200 calculates the parallax value of each pixel of the enlarged image I2h when the enlarged image I2h is a reference image and the enlarged image I1h is a reference image. The disparity value calculating device 200 determines the consistency of the disparity value based on the difference between the disparity value based on the enlarged image I1 h and the disparity value based on the enlarged image I2 h, and the pixels having the consistent disparity value are determined. Apply inconsistent values of disparity to values that indicate low consistency and apply to pixels. Then, the disparity value calculating device 200 generates a disparity image in which it is easy to determine whether the disparity values of the pixels are consistent. Therefore, the disparity value calculating device 200 can calculate disparity values between a plurality of images with high accuracy and reliability.

  In the present embodiment, L / R check section 206 selects each of the parallax images of enlarged image I1h according to the presence or absence of the consistency of the parallax value between the parallax image of enlarged image I1h and the parallax image of enlarged image I2h. Although the parallax value of the pixel is changed to generate a new parallax image, the present invention is not limited to this. The L / R check unit 206 may change the parallax value of each pixel of the parallax image of the enlarged image I2h according to the presence or absence of the consistency, and generate a new parallax image. In this case, the horizontal reduction unit 105 reduces a new parallax image generated from the parallax image of the enlarged image I2h.

  Alternatively, the L / R check unit 206 determines, for each pixel of the parallax image of the enlarged image I1h, whether the parallax value with the parallax image of the enlarged image I2h is consistent or not, and the parallax image of the enlarged image I2h The presence or absence of the consistency of the parallax value with the parallax image of the enlarged image I1 h may be determined for each of the pixels. Furthermore, the L / R check unit 206 may compare the presence or absence of the consistency of the parallax value between corresponding two pixels of the parallax image of the enlarged image I1h and the parallax image of the enlarged image I2h. In this case, when both of the corresponding two pixels have the consistency of the parallax value, the L / R check unit 206 adopts the parallax value of each pixel as it is, and both of the corresponding two pixels When the parallax value does not have consistency, the parallax value of each of the pixels is changed to the specific value. When one of the two corresponding pixels has the consistency of parallax values and the other does not have the consistency of parallax values, the L / R check unit 206 sets the parallax values of the respective pixels as: A disparity value of pixels having consistency may be adopted, or the above specific value may be adopted.

Third Embodiment
A disparity value calculating device 300 according to Embodiment 3 will be described. The disparity value calculating device 300 according to the third embodiment further performs processing of reducing and enlarging an image, as compared with the disparity value calculating device 200 according to the second embodiment. Specifically, the parallax value calculation device 200 according to the second embodiment only performs enlargement and reduction of the image in the horizontal direction, but the parallax value calculation device 300 according to the third embodiment Make further reductions and enlargements. In Embodiment 3, about the component similar to Embodiment 1 or 2, the same referential mark as Embodiment 1 and 2 is attached | subjected, and the description is abbreviate | omitted. In the following, differences from Embodiments 1 and 2 will be mainly described, and descriptions of the same points as Embodiments 1 or 2 will be omitted.

  FIG. 26 shows a block diagram of an example of a functional configuration of disparity value calculation apparatus 300 according to Embodiment 3. As shown in FIG. 26, the parallax value calculation device 300 includes a reduction unit 307, a horizontal enlargement unit 101, a census conversion unit 102, a difference degree calculation unit 103, a first parallax selection unit 204a, and a second parallax selection. It includes a unit 204b, an L / R check unit 206, a horizontal reduction unit 105, an enlargement unit 308, and an end determination unit 309.

  The reduction unit 307 reduces the captured images I1 and I2 of the cameras 11a and 11b at a predetermined reduction ratio, and outputs the reduced images to the horizontal enlargement unit 101. The reduction unit 307 acquires an initial parallax image from the storage unit 20 or acquires a parallax image from the enlargement unit 308, reduces the parallax image at a predetermined reduction ratio, and outputs the reduced parallax image to the horizontal enlargement unit 101. The reduction unit 307 performs normal isotropic reduction processing on the captured images I1 and I2, that is, the same reduction processing in the x-axis direction and the y-axis direction which are the horizontal direction and the vertical direction. For example, when the reduction ratio is 1⁄2, the number of pixels in the x-axis direction is reduced to 1⁄2 and the number of pixels in the y-axis direction is reduced to 1⁄2 in each of the captured images I1 and I2 The image is transformed by interpolation reduction. In the present embodiment, bilinear interpolation is used as a method of interpolation reduction.

  The reduction unit 307 reduces the parallax value (also referred to as “gain down”), which is a pixel value, in accordance with the reduction ratio, in addition to the normal reduction processing, for the parallax image. This is because, for example, when the width of the parallax image is reduced to one half in the horizontal direction at a reduction ratio of one half, the parallax value of each pixel is also half the size before reduction according to this. In order to

  The reduction process by the reduction unit 307 is performed a plurality of times according to the number of repetition processes described later. For example, in the case where the number of iterations is three, an example of the reduction ratio is the first reduction ratio, the second reduction ratio, and the third equal reduction ratio (ie It is an output of the input signal as it is. The reduction unit 307 reduces the initial parallax image in the first iterative process, and reduces the parallax image acquired from the enlargement unit 409 in the second and subsequent iterative processes. In the initial parallax image, the parallax value which is the pixel value of all the pixels is zero.

  The enlargement unit 308 performs enlargement processing on the reduced parallax image output by the horizontal reduction unit 105 at an enlargement ratio corresponding to the reduction ratio in the reduction unit 307, and outputs the enlarged parallax image to the reduction unit 307. The enlargement unit 308 enlarges the parallax value, which is a pixel value, according to the enlargement ratio, in addition to the normal enlargement processing, for the reduced parallax image. For example, when the reduction ratio is half, the enlargement unit 308 performs normal interpolation enlargement processing and enlargement processing of the parallax value of each pixel at a magnification ratio of 2 with respect to the reduction parallax image. The reduced parallax image is converted by interpolation enlargement into an image in which the number of pixels in the x-axis direction is doubled and the number of pixels in the y-axis direction is doubled. In the present embodiment, bilinear interpolation is used as a method of interpolation enlargement.

  The end determination unit 309 determines whether to continue or end the calculation of the parallax value based on the reduction ratio in the reduction unit 307 for calculating the parallax image. The end determination unit 309 determines continuation of the calculation of the disparity value if the reduction ratio is less than 1, that is, less than equal magnification, and determines the end of the calculation of the disparity value if equal magnification, and determines the disparity Output an image.

  The operation of the disparity value calculation device 300 will be described with reference to FIG. FIG. 27 is a flowchart illustrating an example of the operation of the disparity value calculating device 300 according to the third embodiment. First, in step S301, the reduction unit 307 obtains, from the storage unit 20, captured images I1 and I2 which are stereo images captured by the cameras 11a and 11b, and initial parallax images Ii1 and Ii2. The initial parallax images Ii1 and Ii2 are parallax images set in advance corresponding to the cameras 11a and 11b, respectively, and the parallax values of all pixels of the initial parallax images Ii1 and Ii2 are zero.

  Next, in step S302, the reduction unit 307 generates reduced images I1s and I2s by performing normal reduction processing of the captured images I1 and I2 at a predetermined reduction ratio S. The reduction unit 307 outputs the reduced images I1s and I2s to the horizontal enlargement unit 101. Further, the reduction unit 307 reduces the initial parallax images Ii1 and Ii2 or the parallax images Id1 and Id2 acquired from the enlargement unit 308 at the same reduction ratio as the captured images I1 and I2, respectively, and reduces the reduced parallax images Id1s and Id2s. Generate At this time, the reduction unit 307 reduces the parallax value of each pixel of each parallax image at the reduction rate. The reduction unit 307 outputs the reduced parallax images Id1s and Id2s to the horizontal enlargement unit 101. The parallax images Id1 and Id2 are parallax images of the captured images I1 and I2, respectively.

  The number of repetitions of the series of processes in steps S301 to S307 is set in advance, and the reduction ratio at each time is set in advance and stored in the storage unit 20. The reduction rate increases as the number of repetitions increases. In the present embodiment, the number of times of repeated processing is set three times, and the reduction ratio at the time of the first, second and third processing is set to one fourth, one second, and one time, respectively. ing. The reduction unit 307 counts the number of times of reduction processing, and reduces the captured images I1 and I2 at a reduction ratio corresponding to the number of repetition processing, with the count number as the number of repetition processing, and the initial parallax images Ii1 and Ii2 Alternatively, the parallax images Id1 and Id2 are reduced. The reduction unit 307 reduces the initial parallax images Ii1 and Ii2 in the first iterative process, and reduces the parallax images Id1 and Id2 in the second and subsequent iterative processes. When the reduction ratio is equal, the reduction unit 307 outputs the captured images I1 and I2 and the parallax images Id1 and Id2 as they are without reducing them. The parallax value of each pixel of the initial parallax images Ii1 and Ii2 is zero.

  Next, in step S303, the horizontal enlargement unit 101 generates enlarged images I1sh and I2sh by enlarging the reduced images I1s and I2s, respectively, at a predetermined horizontal enlargement ratio H. Further, the horizontal enlargement unit 101 generates the enlarged parallax images Id1sh and Id2sh by enlarging the reduced parallax images Id1s and Id2s at the horizontal enlargement ratio H, respectively. At this time, the horizontal enlargement unit 101 also enlarges the parallax value of each pixel of the reduced parallax images Id1s and Id2s at the horizontal enlargement ratio H. The horizontal enlargement unit 101 outputs the enlarged images I1sh and I2sh to the census conversion unit 102, and outputs the enlarged parallax images Id1sh and Id2sh to the dissimilarity calculation unit 103.

  The number of pixels in the x-axis direction of each of the enlarged images I1sh and I2sh is H / S times the number of pixels in the x-axis direction of each of the captured images I1 and I2. The number of pixels in the y-axis direction of each of the enlarged images I1sh and I2sh is 1 / S times the number of pixels in the y-axis direction of each of the captured images I1 and I2. The number of pixels in the x-axis direction and the number of pixels in the y-axis direction of the enlarged parallax images Id1sh and Id2sh are the same as those of the enlarged images I1sh and I2sh, and the initial parallax images Ii1 and Ii2 or the parallax images Id1 and Id2 in the x-axis direction H / S times the number of pixels, and 1 / S times the number of pixels in the y-axis direction. The parallax value of each pixel of each of the enlarged parallax images Id1sh and Id2sh is H / S times the parallax value of each pixel of each of the initial parallax images Ii1 and Ii2 or each of the parallax images Id1 and Id2.

  Next, in step S304, the census conversion unit 102 performs census conversion on each of the enlarged images I1sh and I2sh, as in step S103 in the first embodiment.

  Next, in step S305, the degree-of-difference calculation unit 103 sets the degree of difference with respect to the enlarged image I2sh, which is the first reference image, to the enlarged image, using the enlarged image I1sh as the first reference image, as in step S201 of the second embodiment. Calculated for each pixel of I1sh. Furthermore, the dissimilarity calculation unit 103 calculates the dissimilarity to the magnified image I1sh, which is the second reference image, for each pixel of the magnified image I2sh, using the magnified image I2sh as a second reference image.

  In the first iteration, the degree-of-difference calculation unit 103 calculates the degree of difference according to a predetermined search range. Specifically, the degree-of-difference calculation unit 103 determines the search range based on the preset parallax search range, the reduction ratio, and the horizontal enlargement ratio at the same scale as the captured image. For example, if the reduction ratio in the first iteration is 1⁄4, the horizontal enlargement ratio is 2 ×, and the parallax search range at the same scale as the captured image is 64 pixels in the x-axis direction, the search range Is determined to be ± 64 pixels × 1⁄4 × 2 = ± 32 pixels in the x-axis direction with the corresponding pixel of the reference image corresponding to the target pixel of the reference image as the center. When it is known that the positional relationship between the reference image and the reference image, that is, which of the captured images I1 and I2 of the cameras 11a and 11b corresponds to, the search range is from the corresponding pixel to the x-axis positive direction. It can be one of the pixel or the x-axis negative direction pixel. Specifically, when the reference image corresponds to the captured image I1, this corresponds to the case where k = 32 in FIG. 20, and the search range is a range from -32 pixels to 0 pixels with respect to the corresponding pixel. When the reference image corresponds to the captured image I2, this corresponds to the case where k = 32 in FIG. 21, and the search range is the range from 0 pixel to 32 pixels for the corresponding pixel.

  In the second and subsequent iterative processes, the dissimilarity calculating unit 103 calculates the dissimilarity using the range in the vicinity of the parallax value obtained by the immediately preceding iterative process as the search range. Specifically, based on the enlarged parallax images Id1sh and Id2sh obtained by the previous repetition processing, the difference degree calculation unit 103 corresponds the pixel of the position corresponding to the parallax value of the pixel of interest in the reference image to the correspondence of the pixel of interest The pixel is determined, and the vicinity range in the x-axis direction of the corresponding pixel is determined as the search range. For example, the dissimilarity calculating unit 103 determines a range of ± 2 pixels as a search range in the x-axis direction with the corresponding pixel as a center. Such a search range may be smaller than the first repeated processing.

  The reduction rate in step S302 increases as the number of times the process is repeated increases. For this reason, as the iterative process progresses, the search shifts from a search with a coarse pixel density to a search with a dense pixel density. That is, the coarse and dense search is performed by repetitive processing. In the first iterative process, the dissimilarity computing unit 103 uses the coarse search for the low-resolution image with a small capacity in all the search, and then uses the image with a higher resolution according to the increase in the number of iterations. Performs a gradually dense search only around the previous search results. Thereby, the calculation amount of the whole parallax value calculation reduces while the calculation accuracy of the parallax value is maintained high.

  Next, in steps S202 to S205, the disparity value calculating device 300 performs the same process as that of the second embodiment based on the degree of difference between each pixel of the enlarged images I1sh and I2sh calculated in step S305.

  Next, in step S306, the end determination unit 309 determines whether to continue the process of calculating the disparity value. If the end determination unit 309 continues the parallax value calculation process (Yes in step S306), the process advances to step S307. If the parallax value calculation process is not continued (No in step S306), the process ends. Specifically, the end determination unit 309 acquires the reduction ratio in step S302 from the reduction unit 307, and acquires the parallax image generated in step S205 from the L / R check unit 206. Then, when the reduction ratio is less than 1, the end determination unit 309 continues the calculation process of the parallax value, and outputs the acquired parallax image to the enlargement unit 308. When the reduction ratio is equal, the end determination unit 309 ends the calculation processing, and outputs the acquired parallax image to the distance calculation unit 31, the storage unit 20, and the like. When the reduction ratio is equal, the scale of the parallax image acquired from the L / R check unit 206 is the same as that of the captured images I1 and I2.

  Next, in step S307, the enlargement unit 308 performs normal enlargement processing and enlargement processing of the parallax value of each pixel at the enlargement ratio corresponding to the reduction ratio in step S302 with respect to the parallax image acquired from the end determination unit 309. And output to the reduction unit 307. In the present embodiment, the enlargement ratios at the first, second and third processings correspond to reduction ratios of 1/4, 1/2 and 1 ×, 4 ×, 2 × and 1 ×, respectively. It is. When the reduction ratio is equal, as described above in step S306, the enlargement unit 308 does not acquire a parallax image.

  In step S302 subsequent to step S307, the reduction unit 307 reduces the captured images I1 and I2 and the parallax images Id1 and Id2 at a reduction ratio larger than that of the previous repetition process, and reduces the reduced images I1sa and I2sa and the reduced parallax. Generate images Id1sa and Id2sa. In step S303, the horizontal enlargement unit 101 horizontally enlarges the reduced images I1sa and I2sa and the reduced parallax images Id1sa and Id2sa to generate enlarged images I1sah and I2sah and enlarged parallax images Id1sah and Id2sah. In step S304, the census conversion unit 102 performs census conversion on the magnified images I1sah and I2sah. In step S305, when the enlarged image I1sah is a reference image, the dissimilarity calculating unit 103 acquires the parallax value of the pixel of interest of the enlarged image I1sah based on the enlarged parallax image Id1sah. Furthermore, the dissimilarity calculation unit 103 determines the pixel at the position where the parallax value is added to the same position as the focused pixel on the enlarged image I2sah which is the reference image, as the corresponding pixel of the focused pixel. To be the search range. The dissimilarity calculating unit 103 calculates the dissimilarity of the target pixel based on the search range. In addition, even when the enlarged image I2sah is a reference image, the difference degree calculation unit 103 processes the same as the above based on the enlarged parallax image Id2sah. In addition, the parallax value calculation device 300 repeats the processes of steps S202 to S307 as in the previous repeated process.

  According to the disparity value calculating device 300 according to the third embodiment as described above, the same effect as that of the second embodiment can be obtained. Furthermore, the parallax value calculation device 300 repeatedly performs calculation of the parallax value while changing the captured image so as to increase the reduction ratio. In the iterative process, the parallax value calculation device 300 limits the search range for computing the degree of difference using the parallax image obtained in the previous iterative process. In the first iteration of the process, the disparity value calculating device 300 performs the rough search on the small and low-resolution image by performing the entire search by calculating the degree of difference and the disparity value, and then increasing the number of times of the iterative process. Accordingly, the degree of difference and the parallax value are calculated by performing a dense search on the high resolution image which is greatly reduced gradually in a limited search range obtained from the immediately preceding parallax image. As a result, the disparity value calculating device 300 can reduce the overall calculation amount of disparity value calculation while maintaining the disparity value calculation accuracy to the sub-pixel accuracy.

  The disparity value calculating apparatus 300 according to the third embodiment has a configuration in which the reducing unit 307, the enlarging unit 308, and the end determining unit 309 are added to the disparity value calculating apparatus 200 according to the second embodiment. Like the parallax value calculation apparatus 300a according to the modification of the third embodiment shown in FIG. 28, the parallax value calculation apparatus 100 according to the first embodiment includes the reduction unit 307, the enlargement unit 308, and the end determination unit 309. The configuration may be added. FIG. 28 is a block diagram showing an example of a functional configuration of a disparity value calculating device 300a according to a modification of the third embodiment.

Fourth Embodiment
A disparity value calculating device 400 according to Embodiment 4 will be described. The parallax value calculation device 300 according to Embodiment 3 performs isotropic enlargement by the horizontal enlargement unit 101 on the captured image after isotropic reduction by the reduction unit 307, and further by the L / R check unit 206. After the consistency check, horizontal reduction by the horizontal reduction unit 105 and isotropic enlargement by the enlargement unit 308 are performed on the parallax image. The disparity value calculating device 400 according to the fourth embodiment does not individually perform reduction and enlargement processing by the reduction unit 307 and the horizontal enlargement unit 101 and reduction and enlargement processing by the horizontal reduction unit 105 and the enlargement unit 308, respectively. Perform at one time in the combined process. In Embodiment 4, about the component similar to Embodiment 1-3, the referential mark same as Embodiment 1-3 is attached | subjected, and the description is abbreviate | omitted. In the following, differences from the first to third embodiments will be mainly described, and the description of the same points as the first to third embodiments will be omitted.

  FIG. 29 shows a block diagram of an example of a functional configuration of disparity value calculation apparatus 400 according to Embodiment 4. As shown in FIG. 29, the parallax value calculation device 400 includes an anisotropy reduction unit 410, a census conversion unit 102, a dissimilarity calculation unit 103, a first parallax selection unit 204a, and a second parallax selection unit 204b. , L / R check unit 206, end determination unit 309, and anisotropic enlargement unit 411. In the disparity value calculating device 400, in the disparity value calculating device 300 according to the third embodiment, the reducing unit 307 and the horizontal enlarging unit 101 are changed to the anisotropic reducing unit 410, and the horizontal reducing unit 105 and the enlarging unit 308 are anisotropic. The configuration is changed to the sex expansion unit 411.

  The anisotropic reduction unit 410 performs processing combining the reduction processing of the reduction unit 307 and the horizontal enlargement processing of the horizontal enlargement unit 101. For example, as exemplified in the third embodiment, the reduction rates of the first, second and third times in the three repetition processes are respectively one fourth, one half and one half, and repetition of each time When the horizontal enlargement ratio in the process is twice, the anisotropic reduction unit 410 performs the reduction process at the following change rates. That is, the change rate in the first repeated process is 1/2 in the horizontal direction and 1/4 in the vertical direction, and the change rate in the second repeated process is equal to 1 in the horizontal direction and 2 in the vertical direction. The rate of change in the third repetition processing is twice in the horizontal direction and equal in the vertical direction. As described above, the anisotropic reduction unit 410 performs reduction processing of a captured image at a change rate obtained by combining the reduction ratio in the reduction unit 307 and the horizontal enlargement ratio in the horizontal enlargement unit 101. In addition, the anisotropy reduction unit 410 reduces the parallax value of each pixel of the parallax image at a horizontal change rate, which is also the search direction of the parallax value in the present embodiment. Specifically, a half change rate is applied in the first repeated process, an equal change rate is applied in the second repeated process, and a double change in the third repeated process. Rates apply.

  The anisotropic enlargement unit 411 performs processing combining the horizontal reduction processing of the horizontal reduction unit 105 and the enlargement processing of the enlargement unit 308. That is, the anisotropic enlargement unit 411 converts the scale of the parallax image into the same scale as the captured image by performing the enlargement process on the parallax image at a change rate corresponding to the change rate of the anisotropic reduction unit 410. For example, in the case of processing in the same manner as the example of the third embodiment described above, the rate of change in the first repetitive processing in the anisotropic expansion portion 411 is twice in the horizontal direction and four times in the vertical direction. The rate of change in the repetition process is 1 × in the horizontal direction and 2 × in the vertical direction, and the rate of change in the third repetition process is 1⁄2 in the horizontal direction and 1 × in the vertical direction. In addition, the anisotropic enlargement unit 411 enlarges the parallax value of each pixel of the parallax image at a change rate in the horizontal direction in the present embodiment. Specifically, a double change rate is applied in the first repeated process, an equal change rate is applied in the second repeated process, and a half change in the third repeated process. Rates apply.

  According to the disparity value calculating device 400 according to the fourth embodiment as described above, the same effect as that of the third embodiment can be obtained. Furthermore, since the parallax value calculation device 400 performs the process of combining the reduction and horizontal enlargement processes of the captured image at one time and performs the process of combining the horizontal reduction and enlargement processes of parallax images at one time, It can be reduced.

  The disparity value calculating device 400 according to the fourth embodiment is configured based on the disparity value calculating device 300 according to the third embodiment, but a disparity value calculating device according to a modification of the fourth embodiment shown in FIG. Like 400a, the structure based on the parallax value calculation apparatus 300a which concerns on the modification of Embodiment 3 may be sufficient. Such a parallax value calculation device 400a includes an anisotropy reduction unit 410, a census conversion unit 102, a difference degree calculation unit 103, a parallax selection unit 104, an end determination unit 309, and an anisotropy enlargement unit 411. including. FIG. 30 is a block diagram showing an example of a functional configuration of a disparity value calculating device 400a according to a modification of the fourth embodiment.

Fifth Embodiment
A disparity value calculating device 500 according to Embodiment 5 will be described. The disparity value calculating device 500 according to the fifth embodiment is configured to weight the dissimilarity in the disparity value calculating device 100 according to the first embodiment, and to determine the disparity value based on the weighted dissimilarity. In Embodiment 5, about the component similar to Embodiment 1-4, the referential mark same as Embodiment 1-4 is attached | subjected, and the description is abbreviate | omitted. In the following, differences from the first to fourth embodiments will be mainly described, and the description of the same points as the first to fourth embodiments will be omitted.

  FIG. 31 shows a block diagram of an example of a functional configuration of the disparity value calculating device 500 according to the fifth embodiment. As shown in FIG. 31, the parallax value calculation device 500 includes a horizontal enlargement unit 101, a census conversion unit 102, a dissimilarity calculation unit 103, a cost calculation unit 512, a cost aggregation unit 513, and a parallax selection unit 104. , And a horizontal reduction unit 105. The disparity value calculating device 500 has a configuration in which a cost calculating unit 512 and a cost aggregating unit 513 are added to the disparity value calculating device 100 according to the first embodiment. The horizontal enlargement unit 101, the census conversion unit 102, and the horizontal reduction unit 105 are the same as in the first embodiment.

  The cost calculation unit 512 calculates, for the pixel of interest in the reference image, the cost relating to the distribution of parallax values in pixels along the respective scanning directions from the pixel of interest. Specifically, the cost calculation unit 512 calculates the cost relating to the distribution of disparity values on each of a plurality of straight lines passing the pixel of interest. An example of a plurality of straight lines is a radial half line starting from the pixel of interest. In the present embodiment, the plurality of straight lines are semi-lines extending equidistantly in the direction of 8 or 16 from the pixel of interest, and the eight semi-lines constitute 4 straight lines passing through the pixel of interest. The straight line constitutes eight straight lines passing through the pixel of interest. Here, the scanning direction, that is, the direction of the half line is an example of a predetermined direction.

  The cost aggregation unit 513 sets a plurality of costs for the distribution of parallax values on a plurality of half straight lines extending from the pixel of interest calculated by the cost calculation unit 512 with respect to each parallax value in the search range for difference degree calculation. Add and consolidate costs in half straight line.

  The parallax selection unit 104 determines, as the parallax value of the pixel of interest, the parallax value that provides the minimum cost for the cost of each parallax value in the pixel of interest aggregated by the cost aggregation unit 513.

  Here, the details of the cost calculation process and the cost aggregation process will be described. In the case where the magnified image I1h is a reference image and the magnified image I2h is a reference image, a processing example regarding the cost of the distribution of disparity values on the half line in the x-axis negative direction among the eight directions is described. As illustrated in FIG. 32, the cost calculation unit 512 sets, on the enlarged image I1h, half straight lines L1 to L8 in eight directions around the pixel of interest I1hpa of the enlarged image I1h. Then, the cost calculation unit 512 calculates the cost of the distribution of parallax values of the pixels on the respective half straight lines L1 to L8. FIG. 32 is a diagram showing an example of a plurality of scanning directions from the pixel of interest at the time of cost calculation.

As shown in FIG. 33, for each pixel on the half line L1 in the positive direction of the x-axis, the cost calculation unit 512 determines the search range corresponding to the pixel and the dissimilarity corresponding to the parallax value within the search range. , To calculate the cost. Cost calculation unit 512, in order from the edge of the enlarged image I1h the target pixel I1hpa, i.e., in order from the pixels I1hp ₁ is a x-coordinate of the pixel coordinates 1 to the target pixel I1hpa, each pixel on the half line L1 Calculate the cost for FIG. 33 is a diagram showing an example of the scanning direction of a pixel on a straight line passing the pixel of interest at the time of cost calculation.

For example, FIG. 34 shows an example of the cost for the pixel I1hp _i (i is a positive integer) between the pixel I1hp ₁ and the target pixel I1hpa. As shown in FIG. 34, the cost calculation unit 512, the pixel I1hp _i, to obtain the difference degree for each disparity value in the search range calculated by the difference calculation unit 103. FIG. 34 is a diagram showing an example of the cost of one pixel on the scanning line for cost calculation. In this example, the search range is from 0 pixel to −k pixel. Furthermore, the cost calculation unit 512 determines a penalty value of the disparity value based on the difference between each disparity value in the search range and the disparity value of the adjacent pixel I1hp _i-1 . If the disparity value in the search range is the same as the disparity value of the pixel I1hp _i-1 , the penalty value is P1, and if the disparity value in the search range is different from the disparity value of the pixel I1hp _i-1 , the penalty The value is P2. In the present embodiment, P2 is larger than P1. Pixel I1hp _i-1, to the pixel I1hp _i, a pixel adjacent in the x-axis positive direction, a pixel that is costing the immediately preceding pixel I1hp _i along the half line L1. The cost calculation unit 512 calculates, for each disparity value in the search range, a cost that is a value obtained by adding a penalty value to the difference degree. Thus, obtained is the cost of each disparity value in the search range for pixels I1hp _i. Here, the pixel I1hp _i on the half straight line L1 is an example of a first evaluation pixel, the pixel I1 hp _i-1 is an example of a second evaluation pixel, and the penalty value is a value of the first evaluation value. The cost is an example of the second evaluation value.

The cost calculation unit 512 calculates the cost of each parallax value within the search range for all the pixels on the half line L1. Note that the pixel I1hp _1, since the adjacent pixels do not exist, there is no penalty value, difference of each disparity value is cost. Further, the cost calculation unit 512 calculates the cost of each parallax value within the search range for all the pixels on the other half straight lines L2 to L8 for each of the other half straight lines L2 to L8. Therefore, the cost of each parallax value of 0 pixel to -k pixel is calculated for all the pixels on the half straight lines L1 to L8. The cost aggregation unit 513 calculates the sum of costs of the same parallax value for all the pixels on the half straight lines L1 to L8. Then, the parallax selection unit 104 determines the parallax value having the smallest sum among the sums of costs of parallax values from 0 pixel to -k pixels as the parallax value of the pixel of interest I1hpa. The cost calculation unit 512 and the cost aggregation unit 513 determine the parallax value of each pixel by performing the above-described process with all the pixels of the enlarged image I1h as the pixel of interest. Here, the sum of costs is an example of the addition evaluation value.

As described above, in the enlarged image I1h as the first enlarged image, for example, the cost calculation unit 512 evaluates the evaluation pixel positioned in a predetermined direction from the target pixel I1hpa as the first pixel for which the parallax value is to be calculated. for pixel I1hp _i respectively as the disparity value of the pixel I1hp _i-1 adjacent to each other in a predetermined direction in the pixel I1hp _i, for the corresponding pixel position corresponding to the pixel I1hp _i in the enlarged image I2h as a second enlarged image The penalty value as the first evaluation value of each of the pixels in the search range is calculated based on comparison with the parallax value of each of the pixels in the search range for calculating the difference degree. Furthermore, based on the degree of difference between the pixel I1hp _i and each of the pixels within the search range and the penalty value of each of the pixels within the search range, the cost calculation unit 512 determines the disparity value for each pixel within the search range. Calculate the cost as the second evaluation value. The cost aggregation unit 513 adds the costs of all the evaluation pixels corresponding to the parallax value of each pixel within the search range, and calculates the sum of the costs as the addition evaluation value. The disparity selection unit 104 determines the disparity value corresponding to the sum of the minimum costs as the disparity value of the pixel of interest I1hpa.

  The operation of the disparity value calculating device 500 will be described with reference to FIG. FIG. 35 is a flowchart illustrating an example of the operation of the disparity value calculating device 500 according to the fifth embodiment. Similar to the first embodiment, the parallax value calculation device 500 acquires the captured images I1 and I2 from the storage unit 20 by performing the processes of steps S101 to S104, and enlarges the images I1h and I2h of the captured images I1 and I2. The census feature value of each pixel is calculated. Furthermore, the parallax value calculation device 500 calculates the degree of difference with respect to each pixel of the search range of the enlarged image I2h for each pixel of the enlarged image I1h, using the enlarged image I1h as a reference image.

  Next, in step S501, the cost calculation unit 512 calculates, for the pixel of interest in the enlarged image I1h that is the reference image, the cost of the distribution of parallax values on each of a plurality of straight lines passing the pixel of interest. The cost calculation unit 512 determines, for each pixel on each straight line, each parallax value within the search range based on the difference between the parallax value within the search range for calculating the degree of difference and the parallax value of the pixel adjacent to the pixel. The cost which is a value obtained by weighting the degree of difference corresponding to. The cost calculation unit 512 calculates the cost with all the pixels of the enlarged image I1h as the target pixel.

  Next, in step S502, the cost aggregating unit 513 adds the costs calculated for each of the pixels on the plurality of straight lines for the target pixel in the enlarged image I1h, and adds them together for each parallax value in the search range, that is, , Add all of the costs of the same disparity value. Thereby, the sum of costs is calculated for each disparity value in the search range. The cost aggregation unit 513 aggregates the cost of each pixel, with all the pixels of the enlarged image I1h as the target pixel.

  Next, in step S503, the parallax selection unit 104 sets, as the parallax value of the pixel of interest, the parallax value giving the sum of the minimum cost among the sum of costs for each parallax value in the search range for the pixel of interest in the enlarged image I1h. decide. The parallax selection unit 104 determines the parallax value of each pixel with all the pixels of the enlarged image I1h as the pixel of interest, and generates a parallax image of the enlarged image I1h.

  Next, in step S106, the parallax value calculation device 500 generates a reduced parallax image in which the parallax image of the enlarged image I1h is horizontally reduced, as in the first embodiment.

  According to the disparity value calculating device 500 according to the fifth embodiment as described above, the same effect as that of the first embodiment can be obtained. In addition, the parallax value calculation device 500 calculates the parallax value of the pixel of interest based on the cost in which the change in parallax value of the pixels around the pixel of interest of the reference image is reflected in the degree of difference. In this way, the parallax value calculation device 500 makes it easy for the parallax value at the pixel of interest to take the same value as the parallax value of the pixels around it, and causes streaks due to differences in parallax values between lines such as edges and boundaries of the object. Can reduce streaking artifacts that occur. Thus, the disparity value calculating device 500 can calculate disparity values with high accuracy. Although the parallax value calculation device 500 uses the enlarged image I1h as the reference image in the above description, the enlarged image I2h may be used as the reference image.

  The disparity value calculating device 500 according to the fifth embodiment has a configuration based on the disparity value calculating device 100 according to the first embodiment, but has a configuration based on the second to fourth embodiments and the modification thereof. May be For example, a disparity value calculating device 500a according to the first modification of the fifth embodiment based on the second embodiment has a configuration as shown in FIG. Further, a disparity value calculating device 500b according to the second modification of the fifth embodiment based on the third embodiment has a configuration as shown in FIG. A disparity value calculating device 500c according to the third modification of the fifth embodiment based on the fourth embodiment has a configuration as shown in FIG. In the first to third modifications, the cost calculation unit 512 acquires the degree of difference from the degree of difference calculation unit 103, and the cost aggregation unit 513 transmits the aggregated costs to the first parallax selection unit 204a and the second parallax selection unit 204b. Output. Further, a disparity value calculating device 500d according to the fourth modification of the fifth embodiment based on the modification of the fourth embodiment has a configuration as shown in FIG. In the fourth modification, the cost calculation unit 512 acquires the difference degree from the difference degree calculation unit 103, and the cost aggregation unit 513 outputs the aggregated cost to the disparity selection unit 104.

Sixth Embodiment
A disparity value calculating device 600 according to Embodiment 6 will be described. In the parallax value calculation device 100 according to Embodiment 1, the area of the peripheral pixel with respect to the pixel of interest is constant, for example, as shown in FIGS. 18A and 18B at the time of census conversion. The disparity value calculating device 600 according to the sixth embodiment calculates the census feature value of the pixel of interest based on various regions of peripheral pixels with respect to the pixel of interest. In the present specification, in the census conversion, a region of a peripheral pixel referred to a target pixel is also referred to as a “window region”. In Embodiment 6, about the component similar to Embodiment 1-5, the referential mark same as Embodiment 1-5 is attached | subjected, and the description is abbreviate | omitted. In the following, differences from the first to fifth embodiments will be mainly described, and the description of the same points as the first to fifth embodiments will be omitted. Here, the window area is an example of the comparison area.

  In the case where the window area crosses an occlusion area where the subject in front can not see the subject behind, or an area where the parallax changes significantly, in the parallax image area where the parallax value is calculated based on such a window area There are cases where the outline of the subject such as the boundary of the foreground and background in the captured image and the boundary line of the parallax image formed by the difference in parallax value do not match. In the present embodiment, as in the first to fifth embodiments, in the parallax value calculation device 600, the pixel of interest is located at a position shifted from the center other than the normal window region where the pixel of interest is located at the center of the window region. Use at least one window area. The parallax value calculation device 600 generates a parallax image in which the boundary of the difference in parallax value is more consistent with the contour of the actual subject by determining the parallax value that minimizes the degree of difference calculated for the plurality of window regions. .

  FIG. 40 shows a block diagram of an example of a functional configuration of disparity value calculation apparatus 600 according to Embodiment 6. As shown in FIG. 40, the parallax value calculation device 600 includes the horizontal enlargement unit 101, the census conversion unit 602, the dissimilarity calculation unit 603, the parallax selection unit 604, and the horizontal reduction unit as in the first embodiment. And 105. The configurations of the horizontal enlargement unit 101 and the horizontal reduction unit 105 are the same as in the first embodiment.

  The census conversion unit 602 sets a plurality of window areas for the pixel of interest of the reference image. The census conversion unit 602 performs census conversion between the pixel of interest and peripheral pixels in the window area for each window area to calculate a census feature value. The census conversion unit 602 calculates census feature values for each of a plurality of window regions for all pixels of each of the enlarged images I1h and I2h.

  For example, in the present embodiment, as shown in FIG. 41, the census conversion unit 602 sets nine window regions W1 to W9 for one pixel of interest ctr. FIG. 41 is a diagram showing an example of a plurality of window areas of the pixel of interest. The window area W5 is a normal window area in which the target pixel ctr is located at the center of the window area. In the window regions W1 to W4 and W6 to W9, the pixel of interest ctr is located other than at the center of the window region. The census conversion unit 602 calculates nine census feature quantities by performing census conversion on the pixel of interest ctr with the peripheral pixels of each of the window regions W1 to W9.

  The dissimilarity calculating unit 603 calculates the dissimilarity of each pixel of the reference image to the reference image for each window area. When the dissimilarity calculating unit 603 calculates the dissimilarity of the reference pixel in the reference image, each pixel of the reference image is calculated using the same window region as the window region used for calculating the census feature amount of the target pixel. The degree of difference is calculated between the census feature amount of and the census feature amount of the target pixel. That is, in the calculation of the degree of difference, the positional relationship between the window region used for calculation of the census feature amount and the target pixel and the reference pixel is the same between the reference image and the reference image. That is, for example, when the window region W1 is used to calculate the census feature amount of the reference image, the census feature amount of the reference image used to calculate the degree of difference of the target pixel of the reference image uses the window region W1. It is calculated. Therefore, in the case of the example of FIG. 41, the dissimilarity calculation unit 603 calculates the dissimilarity corresponding to each of the window regions W1 to W9 with respect to one focused pixel of the reference image.

  Based on the degree of difference of each pixel of the reference image for each window area calculated by the degree of difference calculation unit 603, the parallax selection unit 604 determines the smallest difference among the degrees of difference of the target pixel of the reference image for each window area. The degree is extracted, and the disparity value giving the minimum degree of difference is extracted. Furthermore, the disparity selection unit 604 extracts the minimum difference degree from among the difference degrees of disparity values extracted in all the window regions, and sets the disparity value giving the minimum difference degree to the disparity value of the pixel of interest. Decide on. That is, the parallax selection unit 604 extracts the smallest degree of difference among the degrees of difference of the target pixels in all the window regions, and determines the parallax value giving the minimum degree of difference as the parallax value of the target pixel. Furthermore, the parallax selection unit 604 determines the parallax value of the pixel with all pixels of the reference image as the target pixel, and generates a parallax image.

  According to the disparity value calculating device 600 according to the sixth embodiment as described above, the same effect as that of the first embodiment can be obtained. Also, the parallax value calculation device 600 is based on the degree of difference using the census feature value calculated using the window area that minimizes the degree of difference among the plurality of window areas where the position of the pixel of interest with respect to the window area is different. Calculate the disparity value. Such a parallax value reduces an error of the parallax value caused by the position of the window area with respect to the target pixel. Therefore, the parallax value calculation device 600 detects the position of the boundary due to the difference in parallax value in the parallax image and the position of the outline of the subject for the pixels located in the vicinity of the boundary between the occlusion boundary or the foreground and background where the parallax value changes. And can be precisely aligned.

  The position of the window area with respect to the pixel of interest is not limited to the example shown in FIG. 41, and the pixel of interest may be located at any position in the window area. Further, in the example of FIG. 41, the shape of the plurality of window regions is rectangular, but may be any shape.

  In addition, although the parallax value calculation device 600 according to the sixth embodiment is configured based on the parallax value calculation device 100 according to the first embodiment, the configuration is based on the second to fifth embodiments and the modification thereof. It may be.

  For example, a disparity value calculating device 600a according to a modification of the sixth embodiment based on the fifth embodiment has a configuration as shown in FIG. In the parallax value calculation device 600a according to the modification, as in the sixth embodiment described above, the census conversion unit 602 generates census characteristics for each of a plurality of window regions for all pixels of each of the enlarged images I1h and I2h. Calculate the quantity. The dissimilarity calculating unit 603 calculates the dissimilarity of each pixel of the reference image to the reference image for each window area.

  The cost calculation unit 512 calculates the cost of the pixel of interest with respect to the distribution of parallax values on each of a plurality of straight lines passing the pixel of interest, with all pixels in the reference image as pixels of interest for each window region. The cost calculation unit 512 determines, for each pixel on each straight line, each parallax value within the search range based on the difference between the parallax value within the search range for calculating the degree of difference and the parallax value of the pixel adjacent to the pixel. Is calculated as the cost of the pixel of interest. The cost aggregation unit 513 adds, for each window area, the cost calculated for each of the pixels on the plurality of straight lines passing the pixel of interest for each disparity value in the search range, that is, the sum of costs for each disparity value By calculating, the cost of the pixel of interest is consolidated. The parallax selection unit 604 extracts the sum of the minimum cost from the sum of the costs for each parallax value of the search range in all the window areas for the pixel of interest, and provides the window area and the parallax value giving the sum of the minimum cost. Are identified, and the identified disparity value is determined as the disparity value of the pixel of interest. Furthermore, the parallax selection unit 604 determines the parallax value of the pixel with all pixels of the reference image as the target pixel, and generates a parallax image.

Seventh Embodiment
A disparity value calculating device 700 according to Embodiment 7 will be described. The parallax value calculation device 600 according to the sixth embodiment sets a plurality of pixels set as one pixel of interest in order to make the boundary line due to the difference in parallax value in the parallax image coincide with the contour of the subject such as the boundary of foreground and background. The parallax value of the pixel of interest is determined by calculating the degree of difference or the cost for the window region of When the parallax value calculation device 700 according to the seventh embodiment calculates the Hamming distance, which is the degree of difference between the focused pixel of the reference image and the reference pixel of the reference image, the parallax value calculation device 700 The difference between the bits is weighted by the luminance difference which is the difference in pixel value between corresponding pixels in the census conversion. In Embodiment 7, about the component similar to Embodiment 1-6, the referential mark same as Embodiment 1-6 is attached | subjected, and the description is abbreviate | omitted. In the following, differences from the first to sixth embodiments will be mainly described, and the description of the same points as the first to sixth embodiments will be omitted.

  FIG. 43 shows a block diagram of an example of a functional configuration of disparity value calculation apparatus 700 according to Embodiment 7. As shown in FIG. 43, the parallax value calculation device 700 includes the horizontal enlargement unit 101, the census conversion unit 102, the dissimilarity calculation unit 703, the parallax selection unit 104, and the horizontal reduction unit as in the first embodiment. And 105. The horizontal enlargement unit 101, the census conversion unit 102, the parallax selection unit 104, and the horizontal reduction unit 105 are the same as in the first embodiment.

  When calculating the Hamming distance which is the degree of difference according to the parallax value between the census feature amounts of the enlarged images I1h and I2h, the degree of difference calculation unit 703 responds to the luminance difference which is the difference in pixel value between corresponding pixels. Weight. Specifically, when the dissimilarity calculation unit 703 calculates the dissimilarity of the target pixel of the reference image of the enlarged images I1h and I2h, the bit string of the census feature amount of the target pixel and the census of the reference pixel of the reference image The inter-bit difference which is the difference between the bit numbers of the same digit from the bit string of the feature amount is referred to in the census conversion of the pixel of interest and in the census conversion of the reference pixel It is weighted by the difference between the corresponding pixel and the luminance value of the pixel. Here, the weight according to the luminance difference between corresponding pixels is small when the luminance difference between corresponding pixels is large, and is increased when the luminance difference between corresponding pixels is small. Thereby, the inter-bit difference in the Hamming distance when the luminance difference is small between corresponding pixels is mainly reflected in the Hamming distance, and the inter-bit difference between the Hamming distances when the luminance difference is large between the corresponding pixels is the Hamming distance Less impact on.

  For example, FIG. 44 shows an example of the relationship between the inter-bit difference between the pixel of interest and the reference pixel and the weight based on the luminance difference. FIG. 44 shows the case where eight peripheral pixels are referred to during census conversion. In FIG. 44, the degree-of-difference calculating unit 703 performs processing using the enlarged image I1h as a reference image and the enlarged image I2h as a reference image. The degree-of-difference calculation unit 703 acquires, from the enlarged image I1h, the luminance values of peripheral pixels around the pixel of interest ctr in the reference image. The peripheral pixels are eight pixels referred to in the census conversion of the target pixel ctr. Furthermore, in the reference image, the degree-of-difference calculation unit 703 obtains, from the enlarged image I2h, luminance values of peripheral pixels around the reference pixel ref. The reference pixel ref is a pixel within a search range for which the degree of difference with the pixel of interest ctr is to be calculated, and the peripheral pixels of the reference pixel ref are eight pixels referred to in the census conversion of the reference pixel ref. is there.

  The degree-of-difference calculation unit 703 calculates corresponding peripheral pixels which are peripheral pixels at the same position with respect to the window area between the window area A1 of the peripheral pixel of the target pixel ctr and the window area A2 of the peripheral pixel of the reference pixel ref. The difference between the brightness values of (i.e., the absolute value of the difference between the brightness values) is calculated. Furthermore, the degree-of-difference calculation unit 703 calculates weights based on differences in luminance values for each of the eight sets of corresponding peripheral pixels. In the present embodiment, the degree-of-difference calculation unit 703 calculates the reciprocal of the difference in luminance value as a weight. The weight is not limited to this, and may be small as long as the difference is large, such as the square of the reciprocal of the difference in luminance value, and large as the difference is small. For example, the degree-of-difference calculation unit 703 determines that the weight “0 .. 6” is based on the difference in luminance value between the peripheral pixel of the luminance value “85” in the window region A1 and the peripheral pixel of the luminance value “120” in the window region A2. Calculate “03”. The degree-of-difference calculating unit 703 calculates weights for each of eight peripheral pixels in the window area A1.

  Further, the dissimilarity calculation unit 703 calculates the inter-bit difference between the bit strings of the census feature amount from the census feature amount of the target pixel and the census feature amount of the reference pixel. The degree-of-difference calculation unit 703 compares the census conversion value for the pixel of interest in each peripheral pixel in the window area A1 with the census conversion value for the reference pixel in each peripheral pixel in the window area A2, and makes the same for the window area. The inter-bit difference is calculated based on whether the census conversion value is the same between the corresponding pixels of the position. For example, the dissimilarity calculation unit 703 calculates the inter-bit difference “1” between the census converted value “1” of the upper left corner in the window area A1 and the census converted value “0” of the upper left corner in the window area A2. Do. The dissimilarity calculating unit 703 calculates inter-bit differences for each of the eight peripheral pixels in the window area A1.

  Furthermore, the dissimilarity calculation unit 703 determines the weighted after-weighting bit difference in each peripheral pixel of the window region A1 from the weight of each peripheral pixel in the window region A1 and the difference between the bits of each peripheral pixel in the window region A1. Calculate the inter-bit difference. The degree-of-difference calculating unit 703 calculates the difference between the weighting bits by multiplying the weight corresponding to the same peripheral pixel in the window area A1 by the difference between the bits. For example, the dissimilarity calculating unit 703 calculates the weighting bit difference “0.03” by multiplying the weight “0.03” of the peripheral pixels in the upper left corner of the window area A1 and the bit difference “1”. Do.

  Furthermore, the dissimilarity calculating unit 703 calculates the sum of the weighting bit difference of each of the peripheral pixels of the pixel of interest ctr as the difference of the pixel of interest ctr. In the example of FIG. 44, the dissimilarity calculating unit 703 calculates the sum “0.03 + 0.09 + 0.04 = 0.16” of the differences between the weighting bits of the respective peripheral pixels of the window region A1 as the dissimilarity of the pixel of interest ctr. Do.

  The disparity selection unit 104 extracts, for the pixel of interest, the smallest difference degree among the difference degrees calculated by the difference degree calculation unit 703 between each of the reference pixels within the search range in the reference image, The disparity value corresponding to the extracted degree of difference is determined as the disparity value of the pixel of interest.

  According to the disparity value calculating device 700 according to the seventh embodiment as described above, the same effect as that of the first embodiment can be obtained. In addition, the disparity value calculating device 700 compares the degree of difference between the focused pixel of the reference image and the reference pixel of the reference image with the luminance value of the peripheral pixel referred to in the census conversion of the focused pixel and the census conversion of the reference pixel. Weighting is performed based on the difference from the luminance value of the peripheral pixel to be referred to. This weighting is increased as the difference in luminance value decreases, and reduced as the difference in luminance value increases. The component of the difference degree where the difference in luminance value is small is mainly reflected in the difference degree, and the component of the difference degree where the difference in luminance value is large has a small influence on the difference degree. Such weighting can, for example, spatially weight the inside of the window area even if the arrangement of the window area with respect to the target pixel is only the center of the window area, and adaptively change the barycentric position with respect to the window area Show what you can do. That is, the same effect as if the window area was moved adaptively with respect to the pixel of interest can be obtained. Thereby, the difference degree can reflect the change of the pixel value. Therefore, the parallax value calculation device 700 can use only a single window area for the pixel of interest for pixels located in the vicinity of the boundary between the boundary of the occlusion or the boundary between the foreground and the background where the parallax value changes. The position of the boundary due to the difference in parallax value and the position of the contour of the object can be accurately aligned.

  The weighting for the inter-bit difference is not limited to the luminance difference of the corresponding pixel between the comparison region of the target pixel and the reference pixel, but may be the difference of the respective luminance gradients in the corresponding pixel. Alternatively, it may be a brightness gradient at the corresponding pixel. Furthermore, the weighting may be a combination of at least two of the luminance difference between corresponding pixels, the difference in luminance gradient at the corresponding pixels, and the luminance gradient of the target pixel or the corresponding pixels. Here, the weighting is set to be heavier as the luminance difference, the luminance gradient difference, or the luminance gradient at the target pixel or the corresponding pixel is larger, and to be lighter as it is smaller. Such weighting can also achieve the same effect as the weighting using the luminance difference between corresponding pixels. The brightness gradient may be a gradient in the x-axis direction or a gradient in the y-axis direction in image coordinates, or may be a gradient considering both the gradient in the x-axis direction and the y-axis direction. It may be a gradient in any other direction.

  As described above, the parallax value calculation device 700 calculates the luminance values of pixels around the pixel of interest and the luminance values of pixels around the reference pixel for the pixel of interest of the standard image for which the degree of difference is calculated and the reference pixel of the reference image. Based on at least one of the difference between the brightness gradient of the pixel around the pixel of interest and the brightness gradient of the pixel around the reference pixel and the brightness gradient of the pixel of interest, the reference pixel, or a pixel around it A weight may be added to the dissimilarity.

  In addition, the parallax value calculation device 700 according to the seventh embodiment is configured based on the parallax value calculation device 100 according to the first embodiment, but the configuration is based on the second to sixth embodiments and the modification thereof. It may be.

  For example, a disparity value calculating device 700a according to a modification of the seventh embodiment based on the fifth embodiment has a configuration as shown in FIG. In the disparity value calculating device 700a according to the modified example, as in the seventh embodiment, the dissimilarity calculating unit 703 calculates weighted dissimilarities for all pixels of the enlarged image I1h that is a reference image.

  The cost calculation unit 512 calculates the cost of the pixel of interest regarding the distribution of parallax values on each of a plurality of straight lines passing the pixel of interest, with all pixels in the reference image as the pixel of interest. The cost calculation unit 512 determines, for each pixel on each straight line, each parallax value within the search range based on the difference between the parallax value within the search range for calculating the degree of difference and the parallax value of the pixel adjacent to the pixel. Is calculated as the cost of the pixel of interest. The cost aggregation unit 513 adds the cost calculated for each of the pixels on the plurality of straight lines passing the pixel of interest for each parallax value in the search range, that is, calculates the sum of costs for each parallax value, Consolidate the cost of the pixel of interest. The disparity selection unit 104 extracts the sum of the minimum cost from the sum of the costs for each disparity value in the search range for the pixel of interest, identifies the disparity value giving the sum of the minimum cost, and identifies the identified disparity value Is determined as the parallax value of the pixel of interest. Furthermore, the parallax selection unit 104 determines the parallax value of the pixel with all the pixels of the reference image as the target pixel, and generates a parallax image.

  In addition, although the parallax value calculation device 700a according to the modification performs weighting on the degree of difference, the present invention is not limited thereto, and the same weighting may be performed on each pixel at the time of cost calculation or cost consolidation. . Similar effects can be obtained in such a case as well.

Specifically, when the cost of the target pixel I1hp _i shown in FIG. 34, the value of the penalty values P1 and P2, the luminance difference between the target pixel and the corresponding pixel, the difference in luminance gradient between the target pixel and the corresponding pixel, or , Are weighted according to these combinations. To further simplify, the penalty values P1 and P2 may be weighted according to the luminance gradient at the pixel of interest or the corresponding pixel. Here, when the x-coordinate value of the target pixel I1hp _i in the reference image and xb, when the parallax value 0, the corresponding pixel is in a reference image, a pixel to be x coordinate xb-0 = xb. Similarly, the corresponding pixel is a pixel having xb-1 as the x coordinate in the reference image in the case of parallax value −1, and a pixel having xb−k as the x coordinate in the reference image in the case of parallax value −k. It is. Incidentally, the y-coordinate of any of the corresponding pixel in the reference image is also equal to the y coordinate of the pixel of interest I1hp _i in the reference image.

  The weighting is set to be heavier as the luminance difference, the luminance gradient difference, or the luminance gradient at the target pixel or the corresponding pixel is larger, and to be lighter as it is smaller. The brightness gradient may be a gradient in the x-axis direction or a gradient in the y-axis direction in image coordinates, or may be a gradient considering both the gradient in the x-axis direction and the y-axis direction. It may be a gradient in any other direction.

As described above, the disparity value calculating device determines whether the difference in luminance value between the pixel of interest I1hp _i as the first evaluation pixel and the pixel within the search range of the corresponding pixel, the pixel of interest I1 hp _i and the above search range the difference in brightness gradient between the pixels, as well, at least one the basis weight of the luminance gradient of the reference pixel in the target pixel I1hp _i or the search range, is added to the penalty value as a first evaluation value May be

[Others]
As mentioned above, although the parallax value calculation apparatus which concerns on one or several aspect was demonstrated based on embodiment and a modification, this indication is not limited to these embodiment and a modification. As long as it does not deviate from the spirit of the present disclosure, various modifications that those skilled in the art may think of are applied to the embodiment and the modifications, and an embodiment configured by combining components in different embodiments and modifications is also one or It may be included within the scope of a plurality of aspects.

  Although the parallax value calculation device according to the embodiment and the modification calculates the parallax value between the images captured by the two cameras 11a and 11b, the present invention is not limited to this. The disparity value calculating device may calculate disparity values between captured images of three or more cameras. In this case, the parallax value calculation device may calculate, for each pair of two or more cameras among two or more cameras, the parallax value between the two captured images of the pair.

  Further, as described above, the technology of the present disclosure may be realized by a recording medium such as a system, an apparatus, a method, an integrated circuit, a computer program, or a computer readable recording disk, and the system, an apparatus, a method, an integrated circuit. , And may be realized by any combination of computer program and recording medium. The computer readable recording medium includes, for example, a non-volatile recording medium such as a CD-ROM.

  For example, each processing unit included in the disparity value calculating device according to the above-described embodiment and modification is typically realized as an LSI (Large Scale Integration: large scale integrated circuit) which is an integrated circuit. These may be individually made into one chip, or may be made into one chip so as to include some or all.

  Further, the circuit integration is not limited to LSI's, and implementation using dedicated circuitry or general purpose processors is also possible. A field programmable gate array (FPGA) that can be programmed after LSI fabrication, or a reconfigurable processor that can reconfigure connection and setting of circuit cells inside the LSI may be used.

  In the above embodiment and modification, each component may be configured by dedicated hardware or implemented by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a processor such as a CPU reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory.

  In addition, part or all of the above components may be composed of a removable integrated circuit (IC) card or a single module. The IC card or module is a computer system including a microprocessor, a ROM, a RAM, and the like. The IC card or module may include the above LSI or system LSI. The IC card or module achieves its function by the microprocessor operating according to the computer program. These IC cards and modules may be tamper resistant.

  The parallax value calculation method of the present disclosure may be realized by a processor such as a micro processing unit (MPU) and a CPU, a circuit such as an LSI, an IC card, or a single module.

  Furthermore, the technology of the present disclosure may be realized by a software program or a digital signal consisting of a software program, and may be a non-transitory computer readable recording medium in which the program is recorded. Further, it goes without saying that the program can be distributed via a transmission medium such as the Internet.

  In addition, all numbers such as ordinal numbers and quantities used above are illustrated to specifically describe the technology of the present disclosure, and the present disclosure is not limited to the illustrated numbers. Also, the connection relationship between components is illustrated to specifically describe the technology of the present disclosure, and the connection relationship that implements the function of the present disclosure is not limited thereto.

  Also, division of functional blocks in the block diagram is an example, and a plurality of functional blocks may be realized as one functional block, one functional block may be divided into a plurality, or some functions may be transferred to another functional block. May be Also, a single piece of hardware or software may process the functions of a plurality of functional blocks having similar functions in parallel or in time division.

  The technique of the present disclosure is useful for a technique of calculating disparity values of subpixel accuracy between captured images of a camera. The technology of the present disclosure is useful for any technology that uses a parallax value between captured images of a camera, such as a distance sensor using a camera mounted on a vehicle, a ship, an aircraft, a robot or the like.

20 storage unit (memory)
100, 200, 300, 300a, 400, 400a, 500, 500a, 500b, 500c, 500d, 600, 600a, 700, 700a Parallax value calculation device 101 Horizontal enlargement unit 102, 602 Census conversion unit 103, 603, 703 Degree of difference Calculation unit 104, 604 Parallax selection unit 105, 205 Horizontal reduction unit 204a First parallax selection unit 204b Second parallax selection unit 206 L / R check unit 307 Reduction unit 308 Enlargement unit 309 End determination unit 410 Anisotropy reduction unit 411 Directionality expansion unit 512 Cost calculation unit 513 Cost consolidation unit

Claims (11)

Processor and memory,
The processor is
(A) From the memory, a first image taken by a first camera disposed at a first position and a second image taken by a second camera disposed at a second position Acquired,
(B) A first enlargement including a plurality of first pixels by increasing the number of pixels of the first image by enlarging the first image in a predetermined parallax direction. A second image including a plurality of second pixels by generating an image and enlarging the number of pixels of the second image by performing enlargement processing in a predetermined parallax direction on the second image; Generate a magnified image of the
(C) In the first enlarged image, a first census feature value is calculated for each of the plurality of first pixels by comparing pixel values with surrounding pixels, and in the second enlarged image, A second census feature value is calculated for each of the plurality of second pixels by comparing pixel values with surrounding pixels.
(D) For each of the plurality of first pixels, using the first census feature amount and the second census feature amount of at least one second pixel, the at least one second census feature amount Calculating a degree of difference corresponding to the parallax value with respect to the first pixel at each pixel position;
(E) For each of the plurality of first pixels, by selecting the parallax value corresponding to the smallest degree of difference using the degree of difference, each pixel in the parallax image corresponding to the first enlarged image Get the disparity value,
(F) A parallax value calculation device that calculates a parallax value for each pixel of the first image with respect to the second image by performing reduction processing on the parallax value for each pixel of the parallax image. The parallax value calculation device according to claim 1, wherein the processor calculates a Hamming distance between the first census feature amount and the second census feature amount as the degree of difference. The processor is
Between the treatment (d) and the treatment (e)
(G) In the first enlarged image, from the pixel of interest, which is the first pixel for which the parallax value is calculated, for each of the evaluation pixels, which are the first pixels positioned in a predetermined direction,
Among the evaluation pixels, the parallax value of the second evaluation pixel adjacent to the first evaluation pixel in the predetermined direction, and the corresponding pixel at the position corresponding to the first evaluation pixel in the second enlarged image The first evaluation value of each of the second pixels in the search range is calculated based on comparison with the parallax value of each of the second pixels in the search range for calculating the degree of difference,
The search range based on the degree of difference between the first evaluation pixel and each of the second pixels in the search range, and the first evaluation value of each of the second pixels in the search range. Calculating a second evaluation value for the parallax value of each of the second pixels in
The second evaluation value of all the evaluation pixels corresponding to the parallax value of each of the second pixels in the search range is added to calculate an addition evaluation value.
The parallax value calculation device according to claim 1, wherein in the processing (e), the parallax value corresponding to the minimum addition evaluation value is determined as the parallax value of the target pixel. The processor is
In the processing (c), a plurality of comparison areas including the object pixel and having different positions with respect to the object pixel are determined as comparison areas that are areas of surrounding pixels to be compared when calculating the census feature of the object pixel. And
Calculating, for each of the comparison regions, the first census feature of the plurality of first pixels and the second census feature of the plurality of second pixels;
In the processing (d), the degree of difference is calculated for each of the comparison regions,
In the process (e), the parallax value for each pixel in the parallax image corresponding to the first enlarged image is selected by selecting the parallax value corresponding to the smallest difference degree among the difference degrees of all the comparison areas. The parallax value calculation apparatus as described in any one of Claims 1-3. The processor is
In the processing (d), with respect to the first pixel and the second pixel for which the degree of difference is calculated, a luminance value which is a pixel value of the pixel around the first pixel and a value of the second pixel At least one of a difference from a luminance value which is a pixel value of the peripheral pixel, and a difference between a luminance gradient of the peripheral pixel of the first pixel and a luminance gradient of the peripheral pixel of the second pixel The disparity value calculating device according to any one of claims 1 to 4, wherein a weight based on is added to the degree of difference. (A) acquiring a first image captured by a first camera disposed at a first position, and a second image captured by a second camera disposed at a second position;
(B) A first enlargement including a plurality of first pixels by increasing the number of pixels of the first image by enlarging the first image in a predetermined parallax direction. A second image including a plurality of second pixels by generating an image and enlarging the number of pixels of the second image by performing enlargement processing in a predetermined parallax direction on the second image; Generate a magnified image of the
(C) In the first enlarged image, a first census feature value is calculated for each of the plurality of first pixels by comparing pixel values with surrounding pixels, and in the second enlarged image, A second census feature value is calculated for each of the plurality of second pixels by comparing pixel values with surrounding pixels.
(D) For each of the plurality of first pixels, using the first census feature amount and the second census feature amount of at least one second pixel, the at least one second census feature amount Calculating a degree of difference corresponding to the parallax value with respect to the first pixel at each pixel position;
(E) For each of the plurality of first pixels, by selecting the parallax value corresponding to the smallest degree of difference using the degree of difference, each pixel in the parallax image corresponding to the first enlarged image Get the disparity value,
(F) A parallax value for each pixel of the first image with respect to the second image is calculated by performing a reduction process on the parallax value for each pixel of the parallax image;
At least one of the processing (a) to the processing (f) is executed by at least one processor. The parallax value calculation method according to claim 6, wherein a Hamming distance between the first census feature amount and the second census feature amount is calculated as the degree of difference. Between the treatment (d) and the treatment (e)
(G) In the first enlarged image, from the pixel of interest, which is the first pixel for which the parallax value is calculated, for each of the evaluation pixels, which are the first pixels positioned in a predetermined direction,
Among the evaluation pixels, the parallax value of the second evaluation pixel adjacent to the first evaluation pixel in the predetermined direction, and the corresponding pixel at the position corresponding to the first evaluation pixel in the second enlarged image The first evaluation value of each of the second pixels in the search range is calculated based on comparison with the parallax value of each of the second pixels in the search range for calculating the degree of difference,
The search range based on the degree of difference between the first evaluation pixel and each of the second pixels in the search range, and the first evaluation value of each of the second pixels in the search range. Calculating a second evaluation value for the parallax value of each of the second pixels in
The second evaluation value of all the evaluation pixels corresponding to the parallax value of each of the second pixels in the search range is added to calculate an addition evaluation value.
The parallax value calculation method according to claim 6, wherein in the processing (e), the parallax value corresponding to the minimum addition evaluation value is determined as the parallax value of the target pixel. In the processing (c), a plurality of comparison areas including the object pixel and having different positions with respect to the object pixel are determined as comparison areas that are areas of surrounding pixels to be compared when calculating the census feature of the object pixel. And
Calculating, for each of the comparison regions, the first census feature of the plurality of first pixels and the second census feature of the plurality of second pixels;
In the processing (d), the degree of difference is calculated for each of the comparison regions,
In the process (e), the parallax value for each pixel in the parallax image corresponding to the first enlarged image is selected by selecting the parallax value corresponding to the smallest difference degree among the difference degrees of all the comparison areas. The parallax value calculation method according to any one of claims 6 to 8. In the processing (d), with respect to the first pixel and the second pixel for which the degree of difference is calculated, a luminance value which is a pixel value of the pixel around the first pixel and a value of the second pixel At least one of a difference from a luminance value which is a pixel value of the peripheral pixel, and a difference between a luminance gradient of the peripheral pixel of the first pixel and a luminance gradient of the peripheral pixel of the second pixel The disparity value calculation method according to any one of claims 6 to 9, wherein a weight based on is added to the degree of difference. (A) acquiring a first image captured by a first camera disposed at a first position, and a second image captured by a second camera disposed at a second position,
(B) A first enlargement including a plurality of first pixels by increasing the number of pixels of the first image by enlarging the first image in a predetermined parallax direction. A second image including a plurality of second pixels by generating an image and enlarging the number of pixels of the second image by performing enlargement processing in a predetermined parallax direction on the second image; Generate a magnified image of the
(C) In the first enlarged image, a first census feature value is calculated for each of the plurality of first pixels by comparing pixel values with surrounding pixels, and in the second enlarged image, A second census feature value is calculated for each of the plurality of second pixels by comparing pixel values with surrounding pixels.
(D) For each of the plurality of first pixels, using the first census feature amount and the second census feature amount of at least one second pixel, the at least one second census feature amount Calculating a degree of difference corresponding to the parallax value with respect to the first pixel at each pixel position;
(E) For each of the plurality of first pixels, by selecting the parallax value corresponding to the smallest degree of difference using the degree of difference, each pixel in the parallax image corresponding to the first enlarged image Get the disparity value,
(F) A computer is executed to calculate a parallax value for each pixel of the first image with respect to the second image by performing reduction processing on the parallax value for each pixel of the parallax image. program.