JP2016070774A - Parallax value derivation device, moving body, robot, parallax value production method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve the problem in which as a method of obtaining a cost value, an SAD method enables an excellent cost value in a part where texture in an image is strong to be obtained, but does not enable the excellent cost value in a part where the texture in the image is weak, while a Census method enables a relatively excellent cost value to be obtained even in the part where the texture in the image is weak, but in the part where the texture in the image is strong, the Census method does not enable the excellent cost value to be obtained in comparison with the SAD method, and thus, a more correct cost value cannot be derived.SOLUTION: A first cost value C1 calculated by a cost calculation unit 310a, and a second cost value C2 calculated by a cost calculation unit 310b are selected using a reliability level (a prescribed reference) indicated by a comparison determination function CostSelval, which in turn exerts an effect that the more correct cost value can be derived.SELECTED DRAWING: Figure 12

Description

  The present invention relates to an invention for deriving a parallax value indicating parallax with respect to an object from a plurality of images obtained by imaging the object.

  2. Description of the Related Art Conventionally, a distance measuring method is known in which a parallax with respect to an object is derived from a stereo camera by a stereo image method, and a distance from the stereo camera to the object is measured based on a triangulation principle based on a parallax value indicating the parallax. . With this distance measuring method, for example, the distance between automobiles and the distance between automobiles and obstacles are measured, which is useful for preventing collision of automobiles.

  Further, as a method for obtaining the parallax value, stereo matching processing is used. This stereo matching processing is performed by comparing a plurality of corresponding pixels in a comparison image obtained by the other camera with respect to a reference pixel of interest in a reference image obtained by one of the two cameras of the stereo camera. The parallax value between the reference image and the comparison image is detected by obtaining the position of the corresponding pixel that is the pixel with the most similar image signal while sequentially shifting the candidates. In general, by comparing the luminance values of image signals obtained by two cameras, the cost value (Cost: “dissimilarity” in this case) of the luminance value to be compared for each shift amount is the lowest. The position is determined (see Patent Document 1).

  As a method for obtaining the cost value as described above, a SAD method for obtaining a cost value from a sum of luminance differences (SAD: Sum of Absolute Difference) with respect to this region by cutting out a predetermined region including the corresponding pixel from the image to be compared, or the region described above. There is a Census method for obtaining a cost value from the result of binarizing the pixels.

  However, for example, in the SAD method, a good cost value can be obtained in a portion where the texture is strong in the image, but a good cost value cannot be obtained in a portion where the texture is weak in the image. In contrast, with the Census method, a relatively good cost value can be obtained even in a portion where the texture in the image is weak, but a good cost value is obtained in a portion where the texture in the image is strong compared to the SAD method. Absent. Thereby, the subject that a more exact cost value cannot be derived arises.

  In order to solve the above-described problem, the invention according to claim 1 is based on a reference image obtained by imaging a predetermined object at a first position and a comparative image obtained by imaging the object at a second position. A parallax value deriving device for deriving a parallax value indicating parallax with respect to the object, based on a luminance value of a reference region in the reference image and each luminance value of a candidate for a corresponding region in the comparison image First calculation means for calculating a first dissimilarity for each corresponding area candidate with respect to the reference area; a luminance value of the reference area in the reference image; and a corresponding area candidate in the comparison image. A second calculating unit that calculates a second dissimilarity different from the first dissimilarity for each candidate of the corresponding region with respect to the reference region based on each luminance value; and a candidate for the corresponding region A first curve that is a collection of first dissimilarities for each of the first curves and the corresponding region A specific curve is determined based on the reliability indicating whether each of the first curve and the second curve is reliable among the second curves that are the second set of dissimilarities for each candidate. Selecting means for selecting; combining means for combining each dissimilarity in the specific curve with respect to one reference area; and dissimilarity in the specific curve with respect to another reference area; and Derivation means for deriving the parallax value based on the position in the reference image and the position in the comparison image of the corresponding region where the dissimilarity after synthesis by the synthesis means is minimized This is a parallax value deriving device.

  As described above, according to the present invention, the first curve, which is a first dissimilarity group for each corresponding area candidate, and the second dissimilarity group for each corresponding area candidate. Among the second curves, a specific curve is selected based on the reliability indicating whether each of the first curve and the second curve is reliable. Thereby, there exists an effect that a more exact cost value can be derived.

It is explanatory drawing of the principle which derives | leads-out the distance from an imaging device to an object. (A) is a reference image, (b) is a high-density parallax image for (a), and (c) is a conceptual diagram showing an edge parallax image for (a). (A) is a conceptual diagram showing reference pixels in the reference image, and (b) is a conceptual diagram when calculating shift amounts while sequentially shifting corresponding pixel candidates in the comparative image with respect to the reference pixels in (a). . It is a graph which shows the cost value for every shift amount. It is a conceptual diagram for deriving a synthesis cost value. It is a graph which shows the synthetic | combination cost value for every parallax value. (A) is a schematic diagram showing the side of a car carrying a parallax value derivation device concerning one embodiment of the present invention, and (b) is a schematic diagram showing the front of a car. It is a general-view figure of a parallax value derivation device. It is a hardware block diagram of the whole parallax value derivation | leading-out apparatus. It is explanatory drawing of a Census system. (A) is a conceptual diagram which shows the high-density parallax image by the SAD system at the time of backlight, (b) is a conceptual diagram which shows the high-density parallax image by the Census system at the time of backlight. It is a hardware block diagram of the principal part of a parallax value derivation device. It is the graph which showed the cost curve by a SAD system, and the cost curve by a Census system. It is explanatory drawing at the time of comparing and determining a cost curve. (A) is a graph showing a cost curve when using the SAD method at normal times and a cost curve when using the Census method, and (b) is using a cost curve and Census method when using the SAD method during backlighting. It is the graph which showed the cost curve at the time of doing. It is the flowchart which showed the process of the parallax value production method of embodiment.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[Outline of Ranging Method Using SGM Method]
First, an outline of a distance measuring method using an SGM (Semi-Global Matching) method will be described with reference to FIGS. Since the SGM method is disclosed in non-patent literature (Accurate and Efficient Stereo Processing by Semi-Global Matching and Mutual Information), an outline will be described below.

<Principles of ranging>
With reference to FIG. 1, the principle of deriving a parallax with respect to an object from a stereo camera by the stereo image method and measuring the distance from the stereo camera to the object with a parallax value indicating the parallax will be described. FIG. 1 is an explanatory diagram of the principle of deriving the distance from the imaging device to the object. Further, in the following, in order to simplify the description, the description will be made on a pixel-by-pixel basis, not on a predetermined region composed of a plurality of pixels.

(Parallax value calculation)
First, the images captured by the imaging device 10a and the imaging device 10b shown in FIG. 1 are referred to as a reference image Ia and a comparative image Ib, respectively. In FIG. 1, it is assumed that the imaging device 10a and the imaging device 10b are installed in parallel equiposition. In FIG. 1, the point S on the object E in the three-dimensional space is mapped to a position on the same horizontal line of the imaging device 10a and the imaging device 10b. That is, the S point in each image is imaged at a point Sa (x, y) in the reference image Ia and a point Sb (X, y) in the comparative image Ib. At this time, the parallax value Δ is expressed as (Equation 1) using Sa (x, y) at the coordinates on the imaging device 10a and Sb (X, y) at the coordinates on the imaging device 10b. .

Δ = X−x (Formula 1)
Here, in the case as shown in FIG. 1, the distance between the point Sa (x, y) in the reference image Ia and the intersection of the perpendicular drawn from the imaging lens 11a on the imaging surface is set to Δa, and the comparison image Ib When the distance between the point Sb (X, y) and the intersection of the perpendicular line taken from the imaging lens 11b on the imaging surface is Δb, the parallax value Δ = Δa + Δb.

(Distance calculation)
Further, by using the parallax value Δ, the distance Z between the imaging devices 10a and 10b and the object E can be derived. Specifically, the distance Z is a distance from a plane including the focal position of the imaging lens 11a and the focal position of the imaging lens 11b to the specific point S on the object E. As shown in FIG. 1, using the focal length f of the imaging lens 11a and the imaging lens 11b, the baseline length B that is the length between the imaging lens 11a and the imaging lens 11b, and the parallax value Δ, (Equation 2 ) To calculate the distance Z.

Z = (B × f) / Δ (Formula 2)
According to this (Equation 2), the larger the parallax value Δ, the smaller the distance Z, and the smaller the parallax value Δ, the larger the distance Z.

<SGM method>
Next, a distance measuring method using the SGM method will be described with reference to FIGS. 2A is a reference image, FIG. 2B is a conceptual diagram showing a high-density parallax image for (a), and FIG. 2C is a conceptual diagram showing an edge parallax image for (a). Here, the reference image is an image in which an object is indicated by luminance. The high-density parallax image is an image derived from the reference image by the SGM method, and is an image showing the parallax value at each coordinate of the reference image. The edge parallax image is an image derived from the block matching method, and is an image showing the parallax value of only a portion having a relatively strong texture such as an edge portion of the reference image.

  The SGM method is a method for appropriately deriving the parallax value even for an object with a weak texture. Based on the reference image shown in FIG. This is a method for deriving a density parallax image. When the block matching method is used, the edge parallax image shown in FIG. 2C is derived based on the reference image shown in FIG. As can be seen by comparing the inside of the broken-line ellipse in FIGS. 2B and 2C, the high-density parallax image can represent detailed information such as a road having a weak texture compared to the edge parallax image. More detailed distance measurement can be performed.

  This SGM method does not calculate the cost value that is dissimilarity and immediately derives the parallax value, but after calculating the cost value, it further calculates the synthesis cost value that is the synthetic dissimilarity. This is a method of deriving a parallax value and finally deriving a parallax image (here, a high-density parallax image) indicating the parallax value in almost all pixels.

  In the case of the block matching method, the cost value is calculated in the same way as the SGM method. However, as in the SGM method, a comparatively strong texture such as an edge portion is calculated without calculating a synthesis cost value. Only the parallax value is derived.

(Calculation of cost value)
First, a method for calculating the cost value C (p, d) will be described with reference to FIGS. 3 and 4. 3A is a conceptual diagram showing reference pixels in the reference image, and FIG. 3B is a shift amount (shifting) while sequentially shifting corresponding pixel candidates in the comparison image with respect to the reference pixels in FIG. It is a conceptual diagram at the time of calculating (shift amount). FIG. 4 is a graph showing the cost value for each shift amount. Here, the corresponding pixel is a pixel in the comparison image that is most similar to the reference pixel in the reference image. In the following description, C (p, d) represents C (x, y, d).

  As shown in FIG. 3A, a predetermined reference pixel p (x, y) in the reference image and an epipolar line (Epipolar Line) in the comparison image for the reference pixel p (x, y). On the basis of each luminance value with a plurality of corresponding pixel candidates q (x + d, y), the cost value C (p, d) of each corresponding pixel candidate q (x + d, y) with respect to the reference pixel p (x, y). ) Is calculated. d is the shift amount (shift amount) between the reference pixel p and the corresponding pixel candidate q, and in this embodiment, the shift amount in pixel units is represented. That is, in FIG. 3, the corresponding pixel candidate q (x + d, y) is sequentially shifted by one pixel within a predetermined range (for example, 0 <d <25). And a cost value C (p, d), which is a dissimilarity between the luminance values of the reference pixel p (x, y).

  The cost value C (p, d) calculated in this way can be represented by a graph of a cost curve CL, which is a collection of cost values C for each shift amount d, as shown in FIG. In FIG. 4, the cost value C is 0 (zero) when the shift amount d = 5, 12, 19 and cannot be obtained. Thus, in the case of an object having a weak texture, it is difficult to obtain the minimum value of the cost value C.

(Computation cost value calculation)
Next, a method for calculating the synthesis cost value Ls (p, d) will be described with reference to FIGS. 5 and 6. FIG. 5 is a conceptual diagram for deriving a synthesis cost value. FIG. 6 is a graph of a synthesis cost curve showing a synthesis cost value for each parallax value. The calculation method of the synthesis cost value in the present embodiment is not only the calculation of the cost value C (p, d), but also the cost value when pixels around the predetermined reference pixel p (x, y) are used as the reference pixels. The combined cost value Ls (p, d) is calculated by consolidating the cost value C (p, d) at the reference pixel p (x, y).

  Here, the method of calculating the synthesis cost value will be described in more detail. In order to calculate the combined cost value Ls (p, d), it is first necessary to calculate the route cost value Lr (p, d). (Expression 3) is an expression for calculating the route cost value Lr (p, d), and (Expression 4) is an expression for calculating the combined cost value Ls.

Lr (p, d) = C (p, d) + min {(Lr (pr, d), Lr (pr, d-1) + P1, Lr (pr, d + 1) + P1, Lrmin (p) -R) + p2} (Formula 3)
Here, r indicates a direction vector in the aggregation direction, and has two components in the x direction and the y direction. min {} is a function for obtaining the minimum value. Lrmin (p−r) indicates the minimum value of Lr (p−r, d) when the shift amount d is changed in the coordinates obtained by shifting p by one pixel in the r direction. Note that Lr is applied recursively as shown in (Equation 3). P1 and P2 are fixed parameters determined in advance by experiments, and are parameters that make it easy for the parallax values Δ of adjacent reference pixels on the path to be continuous. For example, P1 = 48 and P2 = 96.

  Further, as shown in (Equation 3), Lr (p, d) is the cost value C of the reference pixel p (x, y), and each of the pixels in the r direction shown in FIG. It is obtained by adding the minimum value of the path cost value Lr of the pixel. Thus, in order to obtain Lr for each pixel in the r direction, first, Lr is obtained from the pixel at the extreme end in the r direction of the reference image p (x, y), and Lr is obtained along the r direction.

Then, as shown in FIG. 5, Lr ₀ , Lr ₄₅ , Lr ₉₀ , Lr ₁₃₅ , Lr ₁₈₀ , Lr ₂₂₅ , Lr ₂₇₀ , and Lr _{315 are} obtained in the eight directions, and finally based on (Expression 4) Thus, the synthesis cost value Ls is obtained.

  The composite cost value Ls (p, d) calculated in this way is a graph of the composite cost curve in which the composite cost value Ls (p, d) is shown for each shift amount d as shown in FIG. Can be represented by In FIG. 6, the combined cost value Ls is calculated as a parallax value Δ = 3 because the minimum value is obtained when the shift amount d = 3. In the above description, the number of r is assumed to be 8. However, the present invention is not limited to this. For example, the eight directions may be further divided into two to be divided into sixteen directions and three into 24 directions.

[Specific Description of this Embodiment]
Hereinafter, this embodiment will be specifically described with reference to the drawings. Here, the parallax value deriving device 1 mounted on an automobile will be described.

<< Configuration of Embodiment >>
First, the overall configuration of the present embodiment will be described with reference to FIGS.

<Appearance configuration>
The appearance configuration of the parallax value deriving device 1 according to the present embodiment will be described with reference to FIGS. 7 and 8. FIG. 7A is a schematic diagram showing the side of an automobile equipped with the parallax value deriving device according to one embodiment of the present invention, and FIG. 7B is a schematic diagram showing the front of the automobile. FIG. 8 is an overview of the parallax value deriving device.

  As shown in FIGS. 7A and 7B, the parallax value deriving device 1 of the present embodiment includes an imaging device 10a and an imaging device 10b, and the imaging device 10a and the imaging device 10b are those of an automobile. It is installed so that it can capture a scene in front of the traveling direction.

  As shown in FIG. 8, the parallax value deriving device 1 includes a main body 2 and a pair of cylindrical imaging devices 10a and 10b provided to the main body 2. Yes.

<Overall hardware configuration>
Next, the overall hardware configuration of the parallax value deriving device 1 will be described with reference to FIG. FIG. 9 is an overall hardware configuration diagram of the parallax value deriving device.

  As shown in FIG. 9, the parallax value deriving device 1 includes an imaging device 10a, an imaging device 10b, a signal conversion device 20a, a signal conversion device 20b, and an image processing device 30.

  Among these, the imaging device 10a captures a front scene and generates an analog signal representing an image, and includes an imaging lens 11a, a diaphragm 12a, and an image sensor 13a. The imaging lens 11a is an optical element that forms an object image by refracting light passing through the imaging lens 11a. The diaphragm 12a adjusts the amount of light input to the image sensor 13a described later by blocking part of the light that has passed through the imaging lens 11a. The image sensor 13a is a semiconductor element that converts light input from the imaging lens 11a and the aperture 12a into an electrical analog image signal, and is realized by a CCD (Charge Coupled Devices) or a CMOS (Complementary Metal Oxide Semiconductor). The Since the imaging device 10b has the same configuration as the imaging device 10a, description of the imaging device 10b is omitted. Further, the imaging lens 11a and the imaging lens 11b are installed so that their lens surfaces are in the same plane.

  The signal converter 20a converts an analog signal representing a captured image into image data in a digital format, and includes a CDS (Correlated Double Sampling) 21a, an AGC (Auto Gain Control) 22a, and an ADC (Analog Digital Converter). ) 23a and a frame memory 24a. The CDS 21a removes noise from the analog image signal converted by the image sensor 13a by correlated double sampling. The AGC 22a performs gain control for controlling the intensity of the analog image signal from which noise has been removed by the CDS 21a. The ADC 23a converts an analog image signal whose gain is controlled by the AGC 22a into digital image data. The frame memory 24a stores the image data converted by the ADC 23a.

  Similarly, the signal conversion device 20b acquires image data from the analog image signal converted by the imaging device 10b, and includes a CDS 21b, an AGC 22b, an ADC 23b, and a frame memory 24b. Note that the CDS 21b, AGC 22b, ADC 23b, and frame memory 24b have the same configuration as the CDS 21a, AGC 22a, ADC 23a, and frame memory 24a, respectively, and thus description thereof is omitted.

  Further, the image processing device 30 is a device for processing the image data converted by the signal conversion device 20a and the signal conversion device 20b. The image processing apparatus 30 includes an FPGA (Field Programmable Gate Array) 31, a CPU (Central Processing Unit) 32, a ROM (Read Only Memory) 33, a RAM (Random Access Memory) 34, an I / F (Interface) 35, and each of the above-mentioned As shown in FIG. 9, a bus line 39 such as an address bus or a data bus is provided for electrically connecting the components 31 to 35 as shown in FIG.

  Among these, the FPGA 31 is an integrated circuit, and here performs a process of calculating the parallax value Δ in the image represented by the image data. The CPU 32 controls each function of the parallax value deriving device 1. The ROM 33 stores an image processing program executed by the CPU 32 to control each function of the parallax value deriving device. The RAM 34 is used as a work area for the CPU 32. The I / F 35 is an interface for communicating with an external controller or the like, and can be connected to a CAN (Controller Area Network) of an automobile, for example.

  Note that the image processing program may be distributed in a computer-readable recording medium as an installable or executable file. This recording medium is a CD-ROM, an SD card, or the like.

<Explanation of cost calculation method>
Here, the SAD method and the Census method will be briefly described as an example of the cost calculation method with reference to FIGS. FIG. 10 is an explanatory diagram of the Census method. FIG. 11A is a conceptual diagram showing a high-density parallax image by the SAD method at the time of backlighting, and FIG. 11B is a conceptual diagram showing a high-density parallax image by the Census method at the time of backlighting.

  Further, (Expression 5) is an expression for calculating a cost value (an example of the first cost value) by the SAD method, and (Expression 6) is an cost value (an example of the second cost value) by the Census method. ). The SAD method is generally used, and a predetermined area including a corresponding pixel is cut out from the comparison image, and a cost value is calculated from the sum of luminance differences (SAD) with respect to the predetermined area. On the other hand, in the Census method, the cost value of the corresponding pixel is calculated from the result of binarizing the peripheral pixels excluding the corresponding pixel (center pixel) in the predetermined area. For example, as shown in FIG. 10A, in the binary mode, among the eight neighboring pixels in the predetermined region, a pixel that is less than a predetermined threshold (in this case, the luminance value 80) is “1”. By setting “0” to pixels that are equal to or greater than the predetermined threshold, the larger one of “1” and “0” is set as the cost value of the corresponding pixel (center pixel). As shown in FIG. 10B, in the case of the ternary mode, among the eight peripheral pixels in the predetermined area, the pixel exceeding the first threshold (in this case, the luminance value 83) is “10”. By setting “01” for pixels less than the second threshold value (in this case, luminance value 80) and “00” for other pixels, the largest number among “10”, “01”, and “00” The value is the cost value of the corresponding pixel.

Here, i and j are pixel positions in the predetermined area, I is the luminance value of the pixel of the reference image, and T is the luminance value of the corresponding pixel of the comparison image.

  As described above, the SAD method and the Census method differ in their particular scenes. For example, when the scene is backlit, if the SAD method is used, a lot of noise is generated as shown in FIG. 11A, but if the Census method is used, it is shown in FIG. 11B. As shown, the generation of noise can be suppressed. On the other hand, when the scene is normal, the SAD method can suppress noise more than the Census method.

<Hardware configuration of main parts>
Next, the hardware configuration of the main part of the parallax value deriving device 1 will be described with reference to FIGS. 12 to 15. FIG. 12 is a hardware configuration diagram of the main part of the parallax value deriving device.

  As shown in FIG. 12, the FPGA 31 in FIG. 9 includes a cost calculation unit 310a, a cost calculation unit 310b, a normalization unit 320a, a normalization unit 320b, a selection unit 330, a cost synthesis unit 340, and a parallax value derivation unit. 350. These are part of the circuit of the FPGA 31, but the same processing may be performed by executing the program. The cost calculation unit 310a is an example of a first calculation unit, and the cost calculation unit 310b is an example of a second calculation unit.

  Among these, as shown in FIG. 3, the cost calculation unit 310a uses the luminance value of the reference pixel p (x, y) in the reference image (see FIG. 3A) and the comparison image (FIG. 3). In (b)), by shifting on the epipolar line EL based on the reference pixel p (x, y), each luminance value of the corresponding pixel candidate q (x + d, y) specified for each shift amount d is obtained. Based on the SAD method, the cost value C1 of the candidate pixel q (x + d, y) corresponding to the reference pixel p (x, y) is calculated. On the other hand, as shown in FIG. 3, the cost calculation unit 310b uses the luminance value of the reference pixel p (x, y) in the reference image (see FIG. 3A) and the comparison image (see FIG. 3). 3 (b)), each luminance value of the corresponding pixel candidate q (x + d, y) specified for each shift amount d by shifting on the epipolar line EL based on the reference pixel p (x, y) The cost value C2 of the candidate pixel q (x + d, y) corresponding to the reference pixel p (x, y) is calculated by using the Census method.

  Further, the normalization unit 320a normalizes the cost value C1 calculated by the cost calculation unit 310a based on the following (formula 7). On the other hand, the normalization unit 320b normalizes the cost value C2 calculated by the cost calculation unit 310b based on (Equation 8).

Normalized SAD value = (SAD_max-SAD_min) / SAD_max (Formula 7)
Normalized Census value = (Census_max-Census_min) / Census_max (Equation 8)
FIG. 13 is a graph showing a cost curve based on the SAD method and a cost curve based on the Census method. As shown in FIG. 13, the cost value based on the SAD method and the cost value based on the Census method are different in units (scales), and thus need to be normalized in order to compare the respective cost curves. Note that even if the FPGA 31 includes either one of the normalization unit 320a and the normalization unit 320b, it is possible to compare the respective cost curves.

  The selection unit 330 selects either one of the cost curves CL1 and CL2 after the cost curves CL1 and CL2 are normalized based on a predetermined reference corresponding to the strength of the texture for each pixel of the reference image. To do. Here, the predetermined reference will be described in detail with reference to FIGS. 14 and 15. FIG. 14 is an explanatory diagram for comparing and determining cost curves. FIG. 15A is a graph showing a cost curve when using the SAD method during normal time and a cost curve when using the Census method, and FIG. 15B is a cost curve when using the SAD method during backlighting. It is the graph which showed the cost curve at the time of using a Census system.

  The selection unit 330 uses a predetermined reliability (predetermined criterion) indicated by the comparison determination function CostSelval as shown in (Equation 9), and selects a cost curve having a smaller CostSelval value. .

Here, min1 indicates a cost value of the minimum extreme value min1 when an arbitrary cost curve CL0 is described as shown in FIG. s _A and s _B indicate slope values of the first-order approximation curves fd1 and fd2 of the cost curve CL0 with the minimum extreme value (minimum value) min1 as a base point. shp _mim1a indicates the degree of spread of the quadratic approximate curve sd of the cost curve CL0 having the minimum extreme value (minimum value) min1 as the apex during a specified period (for example, between five slide amounts). d _m1m2 represents the difference between the cost value of the smallest extreme value and the cost value of the second smallest extreme value. A secondary approximate curve may be used instead of the primary approximate curves fd1 and fd2, and a primary approximate curve may be used instead of the secondary approximate curve sd. That is, (w ₂ (s _A + s _B )) and (w ₁ shp _min1a ) indicate the extent of the approximate curve having the minimum value of the cost value C as a vertex. Both quadratic curves are included.

Further, w _{1 to} w ₄ are adjustment weight values, respectively. At least one of w _{1 to} w ₄ can be zero.

For example, when w ₂ = w ₃ = w ₄ = 0, only w ₁ min1 remains, and CostSelval is a criterion in which the smallest extreme value (minimum value) of both cost curves CL1 and CL2 is selected. It becomes.

Of w ₁ to w _4, the case of _{w 1 = w 3 = w 4} = 0, w 2 / (| s A | + | s B |)) only remains, CostSelval is both cost curve CL1, CL2 Among them, the narrower one of the approximate curves with the minimum value of the cost value C as a vertex is a criterion for selection.

Of w ₁ to w _4, the case of _{w 1 = w 2 = w 4} = 0, w 3 / (shp min1a) only remains, CostSelval, of both cost curve CL1, CL2, the minimum value of the cost value C The narrower the curve of the approximate curve with vertices at is the criterion for selection.

Of w ₁ to w _4, the case of _{w 1 = w 2 = w 3} = 0, the remainder only _{w 4 / (d m1m2))} , CostSelval , of both cost curve CL1, CL2, minimum extreme value (minimum Value) and the second smallest extrema are the criteria for selection.

  Subsequently, the cost composition unit 340 uses the cost value C in the specific cost curve selected by the selection unit 330, and the corresponding pixel candidate q (x + d, y) for one reference pixel p (x, y). And the cost value C of the corresponding pixel candidate q ′ (x ′ + d, y ′) for the other reference pixel p ′ (x ′, y ′) for each shift amount d, The synthesis cost value Ls is output. In this synthesis process, after calculating the route cost value Lr from the cost value C based on (Equation 3), the route cost value Lr in each direction is further added based on (Equation 4). This is a process for calculating the combined cost value Ls.

  Furthermore, the parallax value deriving unit 350 corresponds to the position (x, y) in the reference image of one reference pixel p (x, y) and the corresponding pixel q that minimizes the combined cost value Ls after combining by the cost combining unit 340. The parallax value Δ is derived based on the position (x + Δ, y) in the comparison image of (x + Δ, y), and a parallax image indicating the parallax value at each pixel is output.

<Process or Operation of Embodiment>
Next, the processing or operation of this embodiment will be described with reference to FIG. FIG. 16 is a flowchart illustrating processing of the parallax value production method according to the embodiment.

  First, the imaging device 10a illustrated in FIG. 9 captures the object E and generates analog image data (step S1-1). Similarly, the imaging device 10a captures an object and generates analog image data (step S1-2).

  Next, the signal converter 20a converts the analog image data into digital image data (step S2-1). Similarly, the signal conversion device 20b converts analog image data into digital image data (step S2-2).

  Next, the signal conversion device 20a outputs the converted digital image data as reference image data to the FPGA 31 of the image processing device 30 (step S3-1). A conceptual diagram of this reference image is shown in FIG. Similarly, the signal conversion device 20b outputs the converted digital image data as comparison image data to the FPGA 31 of the image processing device 30 (step S3-2). Although this comparative image is an image captured by the imaging device 10b, since there is no extreme difference from the reference image shown in FIG. 2A, a conceptual diagram of the comparative image is omitted without being shown.

  Next, the cost calculation unit 310a shown in FIG. 12 uses the SAD method, based on the data of the reference image and the data of the comparison image, as shown in the graph of FIG. A cost value C1 for each is calculated (step S4-1). Similarly, the cost calculation unit 310b uses the Census method to calculate the cost value C2 for each shift amount d as shown in the graph of FIG. 13 based on the reference image data and the comparison image data. (Step S4-2).

  Next, the normalization unit 320a normalizes the cost value C1 calculated by the cost calculation unit 310a (step S5-1). Similarly, the normalization unit 320b normalizes the cost C2 calculated by the cost calculation unit 310b (step S5-2).

  Next, the selection unit 330 includes a cost curve CL1 (an example of a first curve) that is a collection of cost values C1 output by the normalization unit 320a and a collection of cost values C2 output by the normalization unit 320b. Among a certain cost curve CL2 (an example of a second curve), a specific cost curve CL (an example of a specific curve) is selected using the comparison determination function shown in (Equation 9) (step S6). .

  Next, the cost synthesizing unit 340 synthesizes each shift amount d based on each cost value C in the specific cost curve CL selected by the selecting unit 330, for example, as shown in the graph of FIG. The cost value Ls is output (step S7). In the case of FIG. 6, since the shift amount that minimizes the combined cost value Ls is d = 3, the parallax value deriving unit 350 derives the parallax value Δ = 3 (step S8).

  Thereafter, a high-density parallax image is transmitted from the I / F 35 illustrated in FIG. 9, and the distance between the imaging devices 10 a and 10 b and the object E by the CPU in the device outside the parallax value deriving device 1 (not illustrated). Z is calculated.

<Main effects of the embodiment>
As described above, according to the present embodiment, the first cost value C1 calculated by the cost calculation unit 310a and the second cost value C2 calculated by the cost calculation unit 310b are represented by the comparison determination function CostSelval. Since the selected reliability is used for the selection, a more accurate cost value can be derived.

<< Supplement of Embodiment >>
In the above embodiment, the cost calculation unit calculates the cost value using the SAD method or the Census method, but the present invention is not limited to this method. For example, another method such as an SSD (Sum of Squared Difference) method or an NCC (Normalized Cross-Correlation) method may be used. In this case, among the two arbitrary methods, the selection unit 330 is set so as to select a cost curve in consideration of a good scene.

  Moreover, in the said embodiment, although the cost value C is shown as dissimilarity, you may show as similarity. In this case, the selection unit 330 uses a predetermined reliability indicated by the comparison determination function CostSelval as shown in (Equation 9), and replaces mim1 of CostSelval with max1 (maximum extreme value ( Maximum value)) is used. In this case, the parallax value Δ that maximizes the combined cost value Ls is derived. Moreover, it is good also as a coincidence degree including both dissimilarity and similarity.

  In the above embodiment, for simplification of description, the description has been given in units of one pixel. However, the present invention is not limited to this, and processing may be performed in units of a predetermined area including a plurality of pixels. In this case, the predetermined area including the reference pixel is indicated as the reference area, and the predetermined area including the corresponding pixel is expressed as the corresponding area. The reference area includes the case of only the reference pixel, and the corresponding area includes the case of only the corresponding pixel.

  Furthermore, although the said embodiment demonstrated the parallax value derivation | leading-out apparatus 1 mounted in a motor vehicle, it does not restrict to this. For example, it may be mounted not only on an automobile as an example of a vehicle but also on a vehicle such as a motorcycle, a bicycle, a wheelchair, and an agricultural cultivator. In addition to a vehicle as an example of a moving body, a moving body such as a robot may be used.

  Furthermore, the robot may be an apparatus such as an industrial robot fixedly installed in FA (Factory Automation) as well as a moving body. Further, the device to be fixedly installed may be not only a robot but also a security monitoring camera.

  In the above embodiment, the external CPU calculates the distance Z. However, the present invention is not limited to this, and the CPU 32 of the image processing apparatus 30 may calculate the distance Z.

DESCRIPTION OF SYMBOLS 1 Parallax value derivation | leading-out apparatus 2 Main part 10a Imaging device 10b Imaging device 20a Signal conversion device 20b Signal conversion device 30 Image processing device 310a Cost calculation part (an example of 1st calculation means)
310b Cost calculation unit (an example of second calculation means)
320a normalization unit (an example of first normalization means)
320b normalization unit (an example of second normalization means)
330 selection unit (example of selection means)
340 Cost composition part (an example of composition means)
350 Parallax value deriving unit (an example of deriving means)

JP 2012-181142 A

Claims (18)

Parallax value derivation for deriving a parallax value indicating parallax for the object based on a reference image obtained by imaging a predetermined object at a first position and a comparison image obtained by imaging the object at a second position A device,
A first dissimilarity is calculated for each candidate of the corresponding region with respect to the reference region based on the luminance value of the reference region in the reference image and each luminance value of the candidate of the corresponding region in the comparison image. First calculating means;
Based on the luminance value of the reference area in the reference image and the respective luminance values of the corresponding area candidates in the comparison image, the first dissimilarity for each corresponding area candidate with respect to the reference area Second calculating means for calculating different second dissimilarities;
Of the first curve that is a set of first dissimilarities for each candidate for the corresponding region and the second curve that is a set of second dissimilarities for each candidate for the corresponding region, the first curve Selecting means for selecting a particular curve based on a confidence level indicating whether each of the second curve and the second curve is reliable;
Combining means for combining each dissimilarity in the specific curve with respect to one reference region and each dissimilarity in the specific curve with respect to another reference region;
Derivation means for deriving the parallax value based on the position of the one reference area in the reference image and the position of the corresponding area in the comparison image where the dissimilarity after synthesis by the synthesis means is minimized;
A parallax value deriving device characterized by comprising:   The parallax value deriving device according to claim 1, wherein the selection unit selects one of the first curve and the second curve having a smaller minimum value of each dissimilarity.   2. The selection unit according to claim 1, wherein the selection unit selects one of the first curve and the second curve that has a narrower spread of an approximate curve having a minimum value of each dissimilarity as a vertex. Parallax value deriving device. The first calculation means is based on the luminance value of the reference region in the reference image and each luminance value of the corresponding region candidate specified for each shift amount by shifting in a predetermined direction in the comparison image. , Calculating a first dissimilarity for each candidate of the corresponding region with respect to the reference region,
The second calculation means is based on the brightness value of the reference area in the reference image and each brightness value of the corresponding area candidate specified for each shift amount by shifting in a predetermined direction in the comparison image. Calculating a second dissimilarity for each candidate of the corresponding region with respect to the reference region;
The selection means selects one of the first curve and the second curve that has a narrower spread of an approximate curve having the minimum value of each dissimilarity as a vertex between the predetermined slide amounts. The parallax value deriving device according to claim 3, wherein   The selection means selects, from the first curve and the second curve, the one having the largest difference between the smallest extreme value of each dissimilarity and the second smallest extreme value. Item 2. The parallax value deriving device according to Item 1. Parallax value derivation for deriving a parallax value indicating parallax for the object based on a reference image obtained by imaging a predetermined object at a first position and a comparison image obtained by imaging the object at a second position A device,
Based on the luminance value of the reference area in the reference image and each luminance value of the corresponding area candidate in the comparison image, a first similarity is calculated for each corresponding area candidate with respect to the reference area. 1 calculating means;
Based on the brightness value of the reference area in the reference image and the brightness values of the corresponding area candidates in the comparison image, the first similarity is different for each corresponding area candidate with respect to the reference area. Second calculating means for calculating a second similarity,
Of the first curve that is a first similarity group for each corresponding area candidate and the second curve that is a second similarity group for each corresponding area candidate, the first curve Selection means for selecting a particular curve based on a confidence level indicating whether each of the curve and the second curve is reliable;
Combining means for combining each similarity in the specific curve with respect to one reference region and each similarity in the specific curve with respect to another reference region;
Derivation means for deriving the parallax value based on the position of the one reference area in the reference image and the position of the corresponding area in the comparison image in which the similarity after synthesis by the synthesis means is maximized;
A parallax value deriving device characterized by comprising:   The parallax value deriving device according to claim 6, wherein the selection unit selects a larger one of the maximum values of the similarities among the first curve and the second curve.   7. The selection unit according to claim 6, wherein the selection unit selects one of the first curve and the second curve that has a narrower spread of an approximate curve having a maximum value of each similarity as a vertex. Disparity value deriving device. The first calculation means is based on the luminance value of the reference region in the reference image and each luminance value of the corresponding region candidate specified for each shift amount by shifting in a predetermined direction in the comparison image. , Calculating a first similarity for each candidate of the corresponding region with respect to the reference region,
The second calculation means is based on the brightness value of the reference area in the reference image and each brightness value of the corresponding area candidate specified for each shift amount by shifting in a predetermined direction in the comparison image. , Calculating a second similarity for each candidate for the corresponding region with respect to the reference region,
The selection means selects one of the first curve and the second curve that has a narrower spread of an approximate curve having the maximum value of each similarity as a vertex between the predetermined slide amounts. The parallax value deriving device according to claim 8.   The selection means selects one of the first curve and the second curve having a larger difference between the maximum extreme value and the second largest extreme value of each similarity. 7. The parallax value deriving device according to 6.   A moving body comprising the parallax value deriving device according to claim 1.   The moving body according to claim 11, wherein the moving body is a vehicle or a robot.   A robot comprising the parallax value deriving device according to any one of claims 1 to 10.   The robot according to claim 13, wherein the robot is an industrial robot fixedly installed. Parallax value production for deriving a parallax value indicating parallax for the object based on a reference image obtained by imaging a predetermined object at a first position and a comparison image obtained by imaging the object at a second position A method,
A first dissimilarity is calculated for each candidate of the corresponding region with respect to the reference region based on the luminance value of the reference region in the reference image and each luminance value of the candidate of the corresponding region in the comparison image. A first calculation step;
Based on the luminance value of the reference area in the reference image and the respective luminance values of the corresponding area candidates in the comparison image, the first dissimilarity for each corresponding area candidate with respect to the reference area A second calculation step of calculating a different second dissimilarity;
Of the first curve that is a set of first dissimilarities for each candidate for the corresponding region and the second curve that is a set of second dissimilarities for each candidate for the corresponding region, the first curve A selection step of selecting a particular curve based on a confidence level indicating whether each of the first curve and the second curve is reliable;
Combining each dissimilarity in the specific curve with respect to one reference region and each dissimilarity in the specific curve with respect to another reference region;
A derivation step of deriving the parallax value based on the position of the one reference area in the reference image and the position of the corresponding area in the comparison image where the dissimilarity after synthesis by the synthesis means is minimized;
Including a parallax value. Parallax value production for deriving a parallax value indicating parallax for the object based on a reference image obtained by imaging a predetermined object at a first position and a comparison image obtained by imaging the object at a second position A method,
Based on the luminance value of the reference area in the reference image and each luminance value of the corresponding area candidate in the comparison image, a first similarity is calculated for each corresponding area candidate with respect to the reference area. 1 calculation step;
Based on the brightness value of the reference area in the reference image and the brightness values of the corresponding area candidates in the comparison image, the first similarity is different for each corresponding area candidate with respect to the reference area. A second calculating step for calculating a second similarity;
Of the first curve, which is a collection of first similarities for each candidate of the corresponding region, and the second curve, which is a collection of second similarities for each candidate of the corresponding region, the first curve And a selection step of selecting a particular curve based on a confidence indicating whether each of the second curves is reliable;
Combining each similarity in the specific curve with respect to one reference region and each similarity in the specific curve with respect to another reference region;
A derivation step of deriving the parallax value based on the position of the one reference region in the reference image and the position of the corresponding region in the comparison image where the similarity after the combining by the combining unit is maximized;
The parallax value production method characterized by including.   A program causing a computer to execute the method according to claim 15.   A program causing a computer to execute the method according to claim 16.