JP6459482B2 - Parallax value deriving apparatus, device control system, moving body, robot, parallax value deriving method, and program - Google Patents

Description

  The present invention relates to a parallax value derivation device, a device control system, a moving body, a robot, a parallax value derivation method, and a program.

  In recent years, in-vehicle systems that prevent collision of automobiles by measuring the distance between automobiles or the distance between automobiles and obstacles have been generally used. As a method of measuring such a distance, there is a method using a stereo matching process using a triangulation principle using a stereo camera having two cameras.

  Stereo matching processing is to obtain the parallax by matching the corresponding pixels between the reference image captured by one camera and the comparison image captured by the other camera. This is a process of calculating the distance to the contained object. By calculating the distance to the object by the stereo matching process using such a stereo camera, various recognition processes and brake control and steering control for collision prevention can be performed.

  In the stereo matching process as described above, in order to evaluate the similarity between images, a region is cut out from the images to be compared, and the sum of luminance differences (SAD: Sum of Absolute Difference) and the square sum of the luminance differences are compared. There are block matching methods for obtaining (SSD: Sum of Squared Difference), normalized cross-correlation (ZNCC), and the like. As a technique using stereo matching processing by such a block matching method, a technique has been proposed in which only one pair of stereo cameras achieves both a wide viewing angle and high ranging accuracy in a specific region within the field of view ( Patent Document 1).

  However, the technique described in Patent Document 1 is a matching process between the reference region of the reference image and the candidate region of the comparison image, and the same size regardless of the amount of shift between the reference region and the candidate region. Matching processing is performed in blocks (areas). Therefore, in the reference image (or comparative image), the matching process is performed on the close subject and the distant subject using the same block, so that the accuracy of the parallax value may be reduced.

  The present invention has been made in view of the above, and can provide a parallax value deriving device, a device control system, a moving body, and a robot that can derive a parallax value with high accuracy regardless of whether the subject is a near subject or a far subject. An object is to provide a parallax value derivation method and a program.

  In order to solve the above-described problems and achieve the object, the present invention provides a reference image obtained by the first image pickup unit picking up an image of a subject, and a second position different from the position of the first image pickup unit. A parallax value deriving device that derives a parallax value indicating parallax with respect to the subject based on a comparison image obtained by imaging the subject by the imaging unit, the first reference region of the reference image, and the comparison A plurality of candidate areas that are identified by shifting by a predetermined shift amount from an area corresponding to the first reference area in the image and that are candidates for the corresponding area in the comparison image corresponding to the first reference area; Filtering means for performing a filtering process with a low-pass filter having a size corresponding to the shift amount, and the filtering process performed by the filtering means. The calculation means for calculating the degree of coincidence between the reference area and each of the plurality of candidate areas subjected to the filter processing, and the parallax value between the first reference area and the corresponding area based on the degree of coincidence. And a derivation means for deriving.

  According to the present invention, it is possible to derive a highly accurate parallax value regardless of whether the subject is a near subject or a far subject.

FIG. 1 is a diagram for explaining the principle of deriving the distance from an imaging device to an object. FIG. 2 is a conceptual diagram illustrating a reference image, a high-density parallax image, and an edge parallax image. FIG. 3 is an explanatory diagram for obtaining a corresponding pixel in a comparison image corresponding to a reference pixel in the reference image. FIG. 4 is a graph showing an example of the relationship between the shift amount and the cost value. FIG. 5 is a conceptual diagram for deriving the synthesis cost. FIG. 6 is a graph showing an example of the relationship between the shift amount and the synthesis cost value. FIG. 7 is a diagram illustrating an example in which the device control system according to the present embodiment is mounted on a vehicle. FIG. 8 is a diagram illustrating an example of the appearance of the parallax value deriving device according to the present embodiment. FIG. 9 is a diagram illustrating an example of a hardware configuration of the disparity value deriving device according to the present embodiment. FIG. 10 is a diagram illustrating an example of a block configuration of the disparity value deriving device according to the present embodiment. FIG. 11 is a diagram for explaining the operation of switching the low-pass filter when obtaining the corresponding pixel of the comparison image corresponding to the reference pixel of the reference image. FIG. 12 is a graph showing an example of the relationship between the shift amount and the combined cost value in the present embodiment. FIG. 13 is a diagram for explaining subpixel estimation by parabolic fitting. FIG. 14 is a diagram for explaining subpixel estimation by the method of least squares. FIG. 15 is a conceptual diagram illustrating an example of a high-density parallax image obtained by the stereo matching process of the present embodiment. FIG. 16 is a flowchart illustrating an example of the stereo matching processing operation of the disparity value deriving device according to the present embodiment.

[Outline of Ranging Method Using SGM Method]
First, an outline of a distance measuring method using an SGM (Semi-Global Matching) method will be described with reference to FIGS.

(Principles of ranging)
FIG. 1 is a diagram for explaining the principle of deriving the distance from an imaging device to an object. With reference to FIG. 1, the principle of deriving a parallax with respect to an object from the stereo camera by stereo matching processing and measuring the distance from the stereo camera to the object with the parallax value indicating the parallax will be described. In the following, in order to simplify the description, an example of matching in units of pixels will be described instead of matching a predetermined region composed of a plurality of pixels.

  The imaging system shown in FIG. 1 has an imaging device 510a and an imaging device 510b arranged in parallel equiposition. The imaging devices 510a and 510b respectively include imaging lenses 511a and 511b that refract incident light and form an image of an object on an image sensor that is a solid-state imaging device. The images captured by the imaging device 510a and the imaging device 510b are referred to as a comparison image Ia and a reference image Ib, respectively. In FIG. 1, the point S on the object E in the three-dimensional space is mapped to a position on a straight line parallel to a straight line connecting the imaging lens 511a and the imaging lens 511b in each of the comparison image Ia and the reference image Ib. Here, the point S mapped to each image is a point Sa (x, y) in the comparative image Ia and a point Sb (X, y) in the reference image Ib. At this time, the parallax value dp is expressed by the following (Equation 1) using Sa (x, y) at the coordinates on the comparison image Ia and the point Sb (X, y) at the coordinates on the reference image Ib. expressed.

  dp = X−x (Formula 1)

  In FIG. 1, the distance between the point Sa (x, y) in the comparative image Ia and the intersection of the perpendicular line taken from the imaging lens 511a on the imaging surface is Δa, and the point Sb (X, y) in the reference image Ib is If the distance from the intersection of the perpendicular line taken from the imaging lens 511b on the imaging surface is Δb, the parallax value dp can also be expressed as dp = Δa + Δb.

  Next, a distance Z between the imaging devices 510a and 510b and the object E is derived by using the parallax value dp. Here, the distance Z is a distance from a straight line connecting the focal position of the imaging lens 511a and the focal position of the imaging lens 511b to the point S on the object E. As shown in FIG. 1, using the focal length f of the imaging lens 511a and the imaging lens 511b, the baseline length B that is the length between the imaging lens 511a and the imaging lens 511b, and the parallax value dp, According to 2), the distance Z can be calculated.

  Z = (B × f) / dp (Formula 2)

  From this (Equation 2), it can be seen that the larger the parallax value dp, the smaller the distance Z, and the smaller the parallax value dp, the larger the distance Z.

(SGM method)
Next, a distance measuring method using the SGM method will be described with reference to FIGS.

  FIG. 2 is a conceptual diagram illustrating a reference image, a high-density parallax image, and an edge parallax image. 2A shows a reference image, FIG. 2B shows a high-density parallax image obtained using the reference image shown in FIG. 2A, and FIG. FIG. 3 is a conceptual diagram showing an edge parallax image obtained using the reference image shown in FIG. Here, the reference image corresponds to the reference image Ib shown in FIG. 1 and is an image in which the captured subject is represented by a luminance value. Further, the high-density parallax image indicates an image in which each pixel of the reference image is represented by a parallax value corresponding to each pixel in the reference image derived by the SGM method. The edge parallax image indicates an image in which each pixel of the reference image is represented by a parallax value corresponding to each pixel in the reference image derived by the block matching method. However, as will be described later, the disparity value that can be derived by the block matching method is a relatively textured portion such as an edge portion in the reference image, and when the disparity value cannot be derived such as a weak texture portion, for example, An image is configured with a parallax value of “0”.

  The SGM method is a method for appropriately deriving a parallax value even for a portion having a weak texture in an image. Based on the reference image shown in FIG. 2A, the high-density parallax shown in FIG. This is a method for deriving an image. The block matching method is a method for deriving the edge parallax image shown in FIG. 2C based on the reference image shown in FIG. According to the SGM method, as can be seen by comparing the inside of the dashed ellipse in FIGS. 2B and 2C, the high-density parallax image is detailed even on roads where the texture is weaker than the edge parallax image. Since the distance based on the parallax value can be expressed, more detailed distance measurement can be performed.

  In the SGM method, the cost value as the degree of coincidence on the comparison image with respect to the reference image is not calculated and the parallax value is not derived immediately, but the parallax value is derived by calculating the combined cost value after calculating the cost value. It is a method to do. The SGM method finally derives a parallax image (here, a high-density parallax image) represented by parallax values corresponding to almost all pixels in the reference image.

  In the case of the block matching method, the cost value is calculated in the same way as the SGM method. However, as in the SGM method, a comparatively strong texture such as an edge portion is calculated without calculating a synthesis cost value. Only the parallax value is derived.

<Calculation of cost value>
FIG. 3 is an explanatory diagram for obtaining a corresponding pixel in a comparison image corresponding to a reference pixel in the reference image. FIG. 4 is a graph showing an example of the relationship between the shift amount and the cost value. A method for calculating the cost value C (p, d) will be described with reference to FIGS. 3 and 4. In the following description, C (p, d) represents C (x, y, d).

  3A is a conceptual diagram showing reference pixels in the reference image, and FIG. 3B is a corresponding pixel candidate in the comparison image corresponding to the reference pixel shown in FIG. It is a conceptual diagram at the time of calculating a cost value while shifting (shifting) sequentially. Here, the corresponding pixel indicates a pixel in the comparison image that is most similar to the reference pixel in the reference image. The cost value is an evaluation value that represents the degree of coincidence of each pixel in the comparison image with respect to the reference pixel in the reference image. That is, the cost value (and synthesis cost value) shown below indicates that the smaller the value is, the more similar the pixel in the comparative image is to the reference pixel.

  As shown in FIG. 3A, a candidate that is a candidate for a reference pixel p (x, y) in the reference image Ib and a corresponding pixel on the epipolar line EL in the comparison image Ia with respect to the reference pixel p (x, y). Based on each luminance value of the pixel q (x + d, y), the cost value C (p, d) of the candidate pixel q (x + d, y) corresponding to the reference pixel p (x, y) is calculated. d is a shift amount (shift amount) between the reference pixel p and the candidate pixel q, and the shift amount d is shifted in units of pixels. That is, in FIG. 3, the candidate pixel q (x + d, y) and the reference pixel p are sequentially shifted by one pixel within a predetermined range (for example, 0 <d <25). A cost value C (p, d) that is the degree of coincidence of the luminance value with (x, y) is calculated.

  As described above, since the imaging devices 510a and 510b are arranged in parallel equiposition, the comparison image Ia and the reference image Ib are also in parallel equivalence relations. Therefore, the corresponding pixel in the comparison image Ia corresponding to the reference pixel p in the reference image Ib exists on the epipolar line EL shown as a horizontal line in FIG. 3, and the corresponding pixel in the comparison image Ia is In order to obtain it, the pixel on the epipolar line EL of the comparison image Ia may be searched.

  The cost value C (p, d) calculated in this way is represented by the graph shown in FIG. 4 in relation to the shift amount d. In the example of FIG. 4, the cost value C is “0” when the shift amount d = 5, 12, and 19, and thus the minimum value cannot be obtained. For example, when there is a portion where the texture is weak in the image, it is difficult to obtain the minimum value of the cost value C in this way.

<Calculation of synthetic cost value>
FIG. 5 is a conceptual diagram for deriving the synthesis cost. FIG. 6 is a graph showing an example of the relationship between the shift amount and the synthesis cost value. A method for calculating the combined cost value Ls (p, d) will be described with reference to FIGS. 5 and 6.

  The calculation of the combined cost value is not only the calculation of the cost value C (p, d), but also the cost value when the pixels around the reference pixel p (x, y) are used as the reference pixel is the reference pixel p (x, d The combined cost value Ls (p, d) is calculated by concentrating on the cost value C (p, d) in y). In order to calculate the combined cost value Ls (p, d), first, the route cost value Lr (p, d) is calculated. The route cost value Lr (p, d) is calculated by the following (Formula 3).

Lr (p, d) = C (p, d) + min (Lr (p−r, k) + P (d, k))
(Formula 3)
(P = 0 (when d = k))
P = P1 (when | d−k | = 1),
P = P2 (> P1) (when | dk |> 1))

  As shown in (Formula 3), the route cost value Lr is obtained recursively. Here, r indicates a direction vector in the aggregation direction, and has two components in the x direction and the y direction. min () is a function for obtaining the minimum value. Lr (p−r, k) is the respective path when the shift amount is changed (the shift amount in this case is k) for the pixel having the coordinate shifted by one pixel in the r direction from the coordinate of the reference pixel p. The cost value Lr is shown. Then, based on the relationship between the shift amount d of the route cost value Lr (p, d) and the shift amount k, the value P (d, k) as shown in (1) to (3) below. Lr (p−r, k) + P (d, k) is calculated.

(1) When d = k, P = 0. That is, Lr (p−r, k) + P (d, k) = Lr (p−r, k).
(2) When | d−k | = 1, P = P1. That is, Lr (p−r, k) + P (d, k) = Lr (p−r, k) + P1.
(3) When | d−k |> 1, P = P2 (> P1). That is, Lr (p−r, k) + P (d, k) = Lr (p−r, k) + P2.

  Then, min (Lr (p−r, k) + P (d, k)) is Lr (p−r) calculated in the above (1) to (3) when k is changed to various values. , K) + P (d, k) is a value obtained by extracting the minimum value. In other words, the value P1 or the value P2 (> P1) when the pixel located at the coordinate position of the reference pixel p in the comparison image Ia moves away from the pixel shifted by the shift amount k from the pixel (pr) adjacent in the r direction. ) Is added so that the shift amount d for the pixels away from the coordinates of the reference pixel p in the comparison image Ia is not affected by the discontinuous path cost value Lr. Further, the value P1 and the value P2 are fixed parameters determined in advance by experiments, and are parameters such that the parallax values of the reference pixels adjacent on the path are likely to be continuous. For example, P1 = 48 and P2 = 96. Thus, in order to obtain the path cost value Lr for each pixel in the r direction in the comparison image Ia, first, the path cost value Lr from the extreme end pixel in the r direction from the coordinates of the reference pixel p (x, y). And the route cost value Lr is obtained along the r direction.

Then, as shown in FIG. 5, Lr ₀ , Lr ₄₅ , Lr which are route cost values Lr in eight directions (r ₀ , r ₄₅ , r ₉₀ , r ₁₃₅ , r ₁₈₀ , r ₂₂₅ , r ₂₇₀ and r ₃₁₅ ). ₉₀ , Lr ₁₃₅ , Lr ₁₈₀ , Lr ₂₂₅ , Lr ₂₇₀ and Lr ₃₁₅ are obtained, and finally, a combined cost value Ls (p, d) is obtained based on the following (Equation 4).

  Ls (p, d) = ΣLr (Formula 4)

  As described above, the calculated combined cost value Ls (p, d) can be represented by the graph shown in FIG. 6 in relation to the shift amount d. In the example of FIG. 6, the combined cost value Ls is derived as a parallax value dp = 3 because the minimum value is obtained when the shift amount d = 3. In the above description, the number in the r direction is eight, but the present invention is not limited to this. For example, the eight directions may be further divided into two to be 16 directions, or may be divided into three to be 24 directions. Or it is good also as what calculates | requires the path | route cost value Lr in any of eight directions, and calculates the synthetic | combination cost value Ls.

[Specific Description of this Embodiment]
Hereinafter, this embodiment will be described in detail with reference to FIGS. Here, a case where the parallax value deriving device 1 that performs stereo matching processing is mounted on an automobile will be described as an example.

(Schematic configuration of a vehicle equipped with a parallax value deriving device)
FIG. 7 is a diagram illustrating an example in which the device control system according to the present embodiment is mounted on a vehicle. A vehicle 100 equipped with the device control system 50 according to the present embodiment will be described with reference to FIG. 7A, FIG. 7A is a side view of the vehicle 100 on which the device control system 50 is mounted, and FIG. 7B is a front view of the vehicle 100.

  As shown in FIG. 7, a vehicle 100 that is an automobile includes a device control system 50. The device control system 50 includes a parallax value deriving device 1, a control device 5, a steering wheel 6, and a brake pedal 7 that are installed in a cabin that is a room space of the vehicle 100.

  The parallax value deriving device 1 has an imaging function for imaging the traveling direction of the vehicle 100 and is installed, for example, in the vicinity of the rearview mirror inside the front window of the vehicle 100. The parallax value deriving device 1 includes a main body 2, an imaging unit 10a fixed to the main unit 2, and an imaging unit 10b, as will be described in detail later. The imaging units 10a and 10b (first imaging unit and second imaging unit) are fixed to the main body 2 so that a subject in the traveling direction of the vehicle 100 can be imaged.

  The control device 5 executes various vehicle controls based on distance information from the parallax value deriving device 1 to the subject obtained based on the image data of the parallax images received from the parallax value deriving device 1. As an example of vehicle control, the control device 5 controls the steering system (control target) including the steering wheel 6 based on the image data of the parallax image received from the parallax value deriving device 1 to avoid an obstacle. Alternatively, brake control or the like for controlling the brake pedal 7 (control target) to decelerate and stop the vehicle 100 is executed.

  As in the device control system 50 including the parallax value deriving device 1 and the control device 5 as described above, vehicle control such as steering control or brake control is executed, so that the driving safety of the vehicle 100 can be improved. it can.

  As described above, the parallax value deriving device 1 images the front of the vehicle 100, but is not limited thereto. That is, the parallax value deriving device 1 may be installed so as to image the rear or side of the vehicle 100. In this case, the parallax value deriving device 1 can detect the position of a subsequent vehicle behind the vehicle 100 or another vehicle that translates laterally. And the control apparatus 5 can detect the danger at the time of the lane change of the vehicle 100, or at the time of lane merge, etc., and can perform the above-mentioned vehicle control. In addition, when the control device 5 determines that there is a risk of a collision based on the parallax image of the obstacle behind the vehicle 100 detected by the parallax value deriving device 1 in the back operation when the vehicle 100 is parked or the like. In addition, the vehicle control described above can be executed.

(Configuration of parallax value deriving device)
FIG. 8 is a diagram illustrating an example of the appearance of the parallax value deriving device according to the present embodiment. As shown in FIG. 8, the parallax value deriving device 1 includes the main body 2, the imaging unit 10a fixed to the main body 2, and the imaging unit 10b as described above. The imaging units 10 a and 10 b are configured by a pair of cylindrical cameras arranged in parallel equiposition with respect to the main body 2. For convenience of explanation, the imaging unit 10a illustrated in FIG. 8 is referred to as a “left” camera, and the imaging unit 10b is referred to as a “right” camera.

<Hardware configuration of parallax value deriving device>
FIG. 9 is a diagram illustrating an example of a hardware configuration of the disparity value deriving device according to the present embodiment. A hardware configuration of the parallax value deriving device 1 will be described with reference to FIG.

  As illustrated in FIG. 9, the parallax value deriving device 1 includes a main body 2, an imaging unit 10 a, an imaging unit 10 b, a signal conversion unit 20 a, a signal conversion unit 20 b, and an image processing unit 30. .

  The main body 2 constitutes the casing of the parallax value deriving device 1, and fixes and supports the imaging units 10a and 10b, and incorporates the signal conversion units 20a and 20b and the image processing unit 30.

  The imaging unit 10a is a processing unit that captures an image of a front subject and generates analog image data. The imaging unit 10a includes an imaging lens 11a, a diaphragm 12a, and an image sensor 13a.

  The imaging lens 11a is an optical element that refracts incident light to form an image of an object on the image sensor 13a. The diaphragm 12a is a member that adjusts the amount of light imaged on the image sensor 13a by blocking part of the light that has passed through the imaging lens 11a. The image sensor 13a is a solid-state imaging device that converts light that has entered the imaging lens 11a and passed through the aperture 12a into electrical analog image data. The image sensor 13a is realized by, for example, a CCD (Charge Coupled Devices) or a CMOS (Complementary Metal Oxide Semiconductor).

  The imaging unit 10b is a processing unit that captures an image of a front subject and generates analog image data. The imaging unit 10b includes an imaging lens 11b, a diaphragm 12b, and an image sensor 13b. The functions of the imaging lens 11b, the diaphragm 12b, and the image sensor 13b are the same as the functions of the imaging lens 11a, the diaphragm 12a, and the image sensor 13a, respectively. Further, the imaging lens 11a and the imaging lens 11b are installed so that the respective lens surfaces are on the same plane so that the left and right cameras are imaged under the same conditions.

  The signal conversion unit 20a is a processing unit that converts analog image data generated by the imaging unit 10a into digital image data. The signal converter 20a includes a CDS (Correlated Double Sampling) 21a, an AGC (Auto Gain Control) 22a, an ADC (Analog Digital Converter) 23a, and a frame memory 24a.

  The CDS 21a removes noise from the analog image data generated by the image sensor 13a using correlated double sampling, a horizontal differential filter, a vertical smoothing filter, and the like. The AGC 22a performs gain control for controlling the intensity of analog image data from which noise has been removed by the CDS 21a. The ADC 23a converts analog image data whose gain is controlled by the AGC 22a into digital image data. The frame memory 24a stores the image data converted by the ADC 23a.

  The signal conversion unit 20b is a processing unit that converts analog image data generated by the imaging unit 10b into digital image data. The signal conversion unit 20b includes a CDS 21b, an AGC 22b, an ADC 23b, and a frame memory 24b. The functions of the CDS 21b, AGC 22b, ADC 23b, and frame memory 24b are the same as the functions of the CDS 21a, AGC 22a, ADC 23a, and frame memory 24a, respectively.

  The image processing unit 30 is a device that performs image processing on the image data converted by the signal conversion unit 20a and the signal conversion unit 20b. The image processing unit 30 includes an FPGA (Field Programmable Gate Array) 31, a CPU (Central Processing Unit) 32, a ROM (Read Only Memory) 33, a RAM (Random Access Memory) 34, and a communication I / F (Interface). 35 and a bus 39.

  The FPGA 31 is an integrated circuit and performs processing for deriving a parallax value dp in an image based on image data. The CPU 32 controls each function of the parallax value deriving device 1. The ROM 33 stores an image processing program that the CPU 32 executes to control each function of the parallax value deriving device 1. The RAM 34 is used as a work area for the CPU 32. The communication I / F 35 is an interface for communicating with an external controller or the like, and is connected to the control device 5 shown in FIG.

  Note that the communication I / F 35 is connected to a display device such as a CAN (Controller Area Network) of a car, a liquid crystal display that displays a captured image or a parallax image, or an organic EL (Electro-Luminescence) display. Also good. Further, the image processing unit 30 includes the FPGA 31 as an integrated circuit for deriving the parallax value dp. However, the image processing unit 30 is not limited thereto, and may be an integrated circuit such as an ASIC (Application Specific Integrated Circuit). .

  As shown in FIG. 9, the bus 39 is an address bus, a data bus, or the like that is connected so that the FPGA 31, the CPU 32, the ROM 33, the RAM 34, and the communication I / F 35 can communicate with each other.

  Note that the above-described image processing program may be distributed in a computer-readable recording medium in an installable or executable format file. This recording medium is a CD-ROM or an SD card.

<Block Configuration of Disparity Value Deriving Device and Operations of Each Block>
FIG. 10 is a diagram illustrating an example of a block configuration of the disparity value deriving device according to the present embodiment. FIG. 11 is a diagram for explaining the operation of switching the low-pass filter when obtaining the corresponding pixel of the comparison image corresponding to the reference pixel of the reference image. FIG. 12 is a graph showing an example of the relationship between the shift amount and the combined cost value in the present embodiment. FIG. 13 is a diagram for explaining subpixel estimation by parabolic fitting. FIG. 14 is a diagram for explaining subpixel estimation by the method of least squares. FIG. 15 is a conceptual diagram illustrating an example of a high-density parallax image obtained by the stereo matching process of the present embodiment. The block configuration of the main part of the parallax value deriving device 1 and the operation of each block will be described with reference to FIGS.

  As shown in FIG. 10, the parallax value deriving device 1 includes an image acquisition unit 110, a conversion unit 210, a cost calculation unit 310, a cost synthesis unit 320 (synthesis unit), and a subpixel estimation unit 330 (derivation unit). And a storage unit 340 (storage means) and a generation unit 350.

  The image acquisition unit 110 is a processing unit that captures an object in front by two left and right cameras, generates image data in analog format, and obtains two luminance images that are images based on each image data. The image acquisition unit 110 is realized by the imaging unit 10a and the imaging unit 10b illustrated in FIG.

  The conversion unit 210 removes noise from the image data of the two luminance images obtained by the image acquisition unit 110, converts the image data into digital format image data, and outputs the image data. Here, image data captured by the right camera (imaging unit 10 b) of the image acquisition unit 110 among the image data of the two luminance images output by the conversion unit 210 (hereinafter simply referred to as “luminance image”). The image data of the reference image Ib (hereinafter simply referred to as “reference image Ib”), and the image data captured by the left camera (imaging unit 10a) is the image data of the comparison image Ia (hereinafter simply referred to as “comparison image Ia”). ). That is, the conversion unit 210 outputs the reference image Ib and the comparison image Ia based on the two luminance images output from the image acquisition unit 110. The converter 210 is realized by the signal converters 20a and 20b shown in FIG.

  The storage unit 340 uses the cost calculation unit 310 to calculate the cost value C (p, d) of the candidate pixel q (x + d, y) corresponding to the reference pixel p (x, y) in the comparison image Ia. Further, low-pass filter data (hereinafter simply referred to as a low-pass filter) (for example, low-pass filters F1 and F2 shown in FIG. 11 described later) stored in the reference pixel p and the candidate pixel q are stored. The storage unit 340 stores a low-pass filter determined by the shift amount d between the reference pixel p and the candidate pixel q. Further, the storage unit 340 stores threshold value data for the shift amount d, which is a criterion for determining which low-pass filter the cost calculation unit 310 uses according to the shift amount d. For example, it is assumed that the storage unit 340 stores threshold values Th1 and Th2 (> Th1) determined in advance based on experiments or the like. The threshold values Th1 and Th2 will be described later.

  The cost calculation unit 310 calculates the reference pixel on the epipolar line in the comparison image Ia based on the luminance value of the reference pixel p (x, y) (first reference region) in the reference image Ib and the reference pixel p (x, y). Each candidate pixel is determined based on each luminance value of the candidate pixel q (x + d, y), which is a candidate for the corresponding pixel, specified by shifting from the pixel corresponding to the position of p (x, y) by the shift amount d. A cost value C (p, d) of q (x + d, y) is calculated. The cost calculation unit 310 includes a filter unit 310a (filtering unit) and a calculation unit 310b (calculation unit). The cost calculation unit 310 is realized by the FPGA 31 illustrated in FIG.

  First, the filter unit 310a reads and acquires the threshold values Th1 and Th2 from the storage unit 340. When the shift amount d of the candidate pixel q (x + d, y) of the comparison image Ia to be compared with the reference pixel p (x, y) of the reference image Ib is less than the threshold Th1, the filter unit 310a determines the reference pixel p and the candidate. The subject corresponding to the pixel q is determined to be far away. In this case, the filter unit 310a reads a low-pass filter having a 1 × 1 dimension and a filter coefficient “1” from the storage unit 340, and performs filtering processing on the reference pixel p and the candidate pixel q using the read low-pass filter ( Convolution calculation (convolution calculation)) is performed. Specifically, in this case, the filter unit 310a calculates the product of the luminance value of the reference pixel p and the candidate pixel q and the filter coefficient “1” of the low-pass filter, that is, the luminance of the reference pixel p and the candidate pixel q. The values themselves are obtained as the luminance values of the reference pixel p and the candidate pixel q after the filter processing, respectively. Note that in this case, the filtering process by the filter unit 310a may determine the luminance values of the reference pixel p and the candidate pixel q, and thus the 1 × 1-dimensional filtering process may not be performed.

  Further, as shown in FIG. 11A, the filter unit 310a has a shift amount d1 of the candidate pixel q (x + d1, y) of the comparison image Ia to be compared with the reference pixel p (x, y) of the reference image Ib. If the threshold value Th1 is greater than or equal to the threshold value Th2, the subject corresponding to the reference pixel p and the candidate pixel q is determined to be at a medium distance. In this case, the filter unit 310a reads the low-pass filter F1 having 1 × 3 dimensions and the filter coefficient “1/3” from the storage unit 340, and reads the low-pass filter F1 read out for the reference pixel p and the candidate pixel q. Filter processing (convolution calculation (convolution calculation)) is performed. Specifically, in this case, the filter unit 310a performs a convolution operation with the reference pixel p and the three luminance values of the left and right pixels of the reference pixel p and the low-pass filter F1, and calculates the reference pixel p after the filtering process. Find the luminance value. This means that the average value of the three luminance values of the reference pixel p and the left and right pixels of the reference pixel p is obtained. Similarly, the filter unit 310a performs a convolution operation with the three luminance values of the candidate pixel q and the left and right luminance values of the candidate pixel q and the low-pass filter F1, and the luminance value of the candidate pixel q after the filtering process. Ask for. This means that the average value of the three luminance values of the candidate pixel q and the left and right pixels of the candidate pixel q is obtained.

  Further, as shown in FIG. 11B, the filter unit 310a has a shift amount d2 of the candidate pixel q (x + d2, y) of the comparison image Ia to be compared with the reference pixel p (x, y) of the reference image Ib. If the threshold Th2 is equal to or greater than the threshold Th2, it is determined that the subjects corresponding to the reference pixel p and the candidate pixel q are close. In this case, the filter unit 310a reads the low-pass filter F2 having a 1 × 5 dimension and the filter coefficient “1/5” from the storage unit 340, and the low-pass filter F2 that has been read out with respect to the reference pixel p and the candidate pixel q. Filter processing (convolution calculation (convolution calculation)) is performed. Specifically, in this case, the filter unit 310a performs a convolution operation with the reference pixel p and the five luminance values of the left and right pixels of the reference pixel p and the low-pass filter F2, and after the filter processing, The luminance value of the reference pixel p is obtained. This means that the average value of the five luminance values of the reference pixel p and the two pixels on the left and right of the reference pixel p is obtained. Similarly, the filter unit 310a performs a convolution operation between the five luminance values of the left and right luminance values of the candidate pixel q and the candidate pixel q and the low-pass filter F2, and the candidate pixel q after the filtering process Find the brightness value of. This means that the average value of the five luminance values of the candidate pixel q and each of the left and right pixels of the candidate pixel q is obtained.

  Note that the number of threshold values and the dimensions of the low-pass filter described above are examples, and the range of the shift amount d may be determined by other threshold values, and the low-pass filter corresponding to the range of each shift amount d may be applied. The low-pass filter is a one-dimensional low-pass filter, but is not limited to this, and may be an m × n-dimensional (m, n ≧ 1) low-pass filter. However, the calculation load of the filter processing can be reduced by making the low-pass filter one-dimensional as described above.

  The calculation unit 310b calculates the cost value C of the candidate pixel q based on the luminance values of the reference pixel p and the candidate pixel q after the filter processing by the filter unit 310a using the low-pass filter. As the cost value C calculated by the calculation unit 310b, for example, SAD (Sum of Absolute Differences), SSD (Sum of Squared Differences), or the like can be used. For example, when SAD is used as the cost value C calculated by the calculation unit 310b, the pixel included in this block is assumed when, for example, a 3 × 3 region is assumed as a block, centering on each of the reference pixel p and the candidate pixel q. It is necessary to perform the above-described filtering process by the filter unit 310a for each. And the graph which shows the relationship between the shift amount d and the cost value C calculated by the calculation part 310b is a graph shown in the above-mentioned FIG. 4, for example. In this way, the cost calculation unit 310 does not use the luminance values of the pixels constituting the reference image Ib and the comparison image Ia, but uses a low-pass filter having a size corresponding to the shift amount d for the luminance value. A cost value C is calculated based on the luminance value after performing the filtering process. Specifically, if the subject is the same, the image is small when the subject is far from the parallax value deriving device 1 and is large when the subject is close, so when the shift amount d is small (the subject is far), When a low-pass filter with a small size is used and the shift amount d is large (the subject is close), a low-pass filter with a large size is used. Thus, since the size of the low-pass filter to be applied is changed according to the shift amount d, the accuracy of the derived parallax value can be improved.

  The cost synthesis unit 320 uses the cost calculation unit 310 to calculate the cost value C of the pixel in the comparison image Ia for the reference pixel when the pixel around the reference pixel p (x, y) in the reference image Ib is the reference pixel. The synthesized cost value Ls (p, d) (synthetic matching degree) of the candidate pixel q (x + d, y) is calculated by aggregating the calculated cost values C (p + d, y) of the candidate pixel q (x + d, y). To do. The cost synthesis unit 320 is realized by the FPGA 31 illustrated in FIG.

Specifically, in order to calculate the combined cost value Ls, the cost combining unit 320 first calculates a route cost value Lr (p, d) in a predetermined r direction according to the above (Equation 3). In this case, Lr (p−r, k) in (Expression 3) is obtained when the shift amount is changed with respect to the pixel at the coordinate shifted by one pixel in the r direction from the coordinate of the reference pixel p (the shift amount in this case is k)) is shown. Next, as shown in FIG. 5, the cost synthesis unit 320 stores Lr ₀ , Lr ₄₅ , Lr ₉₀ , Lr ₁₃₅ , Lr ₁₈₀ , Lr ₂₂₅ , Lr ₂₇₀ and Lr ₃₁₅ that are eight-way route cost values Lr. Finally, the combined cost value Ls (p, d) is calculated based on the above (Formula 4). And the graph which shows the relationship between the shift amount d and the synthetic | combination cost value Ls calculated by the cost synthetic | combination part 320 is a graph shown in FIG. As shown in FIG. 12, the synthesis cost value Ls becomes the minimum value when the shift amount d = 3.

  The sub-pixel estimation unit 330 calculates the shift amount d corresponding to the minimum value (extreme value) of the combined cost value Ls of the pixels in the comparison image Ia for the reference pixel in the reference image Ib calculated by the cost combining unit 320, and Based on the combined cost value Ls in the adjacent shift amount d, subpixel estimation is performed. The sub-pixel estimation unit 330 is realized by the FPGA 31 illustrated in FIG. The graph of the synthesis cost value Ls shown in FIG. 13 is a graph of the synthesis cost value Ls with respect to the shift amount d in pixel units. Therefore, the minimum value of the synthesis cost value Ls in the graph shown in FIG. 13 is the synthesis cost value Ls at the shift amount d = 3 divided in units of pixels. That is, in the graph of the combined cost value Ls with respect to the shift amount d divided in units of pixels as shown in FIG. 13, only a value in units of pixels can be derived as the parallax value dp. Here, the sub-pixel estimation is to derive the parallax value dp by estimating the parallax value dp in a unit smaller than the pixel (hereinafter referred to as sub-pixel unit) instead of the pixel unit value.

  First, a case where the subpixel estimation unit 330 performs subpixel estimation by parabolic fitting will be described with reference to FIG. The sub-pixel estimation unit 330 obtains the value of the shift amount d that minimizes the synthesis cost value Ls in the graph of the synthesis cost value Ls calculated by the cost synthesis unit 320 (see FIG. 12). In the example of FIG. 12, the combined cost value Ls is minimum when the shift amount d = 3. Next, the sub-pixel estimation unit 330 obtains a shift amount d adjacent to the shift amount d = 3. Specifically, the shift amount d = 2,4. Next, in the graph of the shift amount d and the combined cost value Ls shown in FIG. 12, the sub-pixel estimation unit 330 passes through three points where the shift amount d = 2, 3, and 4 as shown in FIG. A convex quadratic curve is obtained. Then, the subpixel estimation unit 330 estimates that the shift amount d in subpixel units corresponding to the minimum value of the quadratic curve is the parallax value dp.

  Next, a case where the subpixel estimation unit 330 performs subpixel estimation by the least square method will be described with reference to FIG. The sub-pixel estimation unit 330 obtains the value of the shift amount d that minimizes the synthesis cost value Ls in the graph of the synthesis cost value Ls calculated by the cost synthesis unit 320 (see FIG. 12). In the example of FIG. 12, the combined cost value Ls is minimum when the shift amount d = 3. Next, the sub-pixel estimation unit 330 obtains four shift amounts d in the vicinity of the shift amount d = 3. Specifically, the shift amount d = 1, 2, 4, 5. Next, as shown in FIG. 14, in the graph of the shift amount d and the combined cost value Ls shown in FIG. 12, the sub-pixel estimation unit 330 uses the least square method to shift the amount d = 1, 2, 3, 4, and so on. A convex quadratic curve passing through the vicinity of 5 points, which is 5, is obtained. Then, the subpixel estimation unit 330 estimates that the shift amount d in subpixel units corresponding to the minimum value of the quadratic curve is the parallax value dp.

  The subpixel estimation unit 330 estimates and derives the parallax value dp by either subpixel estimation by parabolic fitting shown in FIG. 13 or subpixel estimation by the least square method shown in FIG. As a result, the parallax value dp can be derived in units of sub-pixels, which is a unit smaller than the pixel, so that it is possible to derive a precise and dense parallax value dp.

  The sub-pixel estimation is not limited to the above-described parabolic fitting or the least-square method, and the sub-pixel estimation may be performed by other methods. For example, the sub-pixel estimation unit 330 uses the three points shown in FIG. 13 to perform sub-pixel estimation by equiangular straight line fitting that obtains a conformal straight line passing through the three points instead of the quadratic curve and estimates the parallax value dp. It may be executed.

  Further, in the sub-pixel estimation by the least square method, the quadratic curve is obtained using five points on the graph shown in FIG. 14, but the present invention is not limited to this, and 2 using a different number of points. A quadratic curve may be obtained.

  Further, the present invention is not limited to calculating the parallax value dp in units of subpixels by subpixel estimation by the subpixel estimation unit 330. The parallax value dp in units of pixels is calculated without performing the subpixel estimation. It may be a thing. In this case, the sub-pixel estimation unit 330 calculates the shift amount d corresponding to the minimum value of the combined cost value Ls of the pixels in the comparison image Ia for the reference pixel in the reference image Ib calculated by the cost combining unit 320 as the parallax value dp. And it is sufficient.

  The generation unit 350 is an image in which the luminance value of each pixel of the reference image Ib is represented by the parallax value dp corresponding to the pixel based on the sub-pixel unit parallax value dp derived by the sub-pixel estimation unit 330. A parallax image Ip (high-density parallax image) is generated. The generation unit 350 is realized by the FPGA 31 illustrated in FIG. 15A shows an example of the comparison image Ia, FIG. 15B shows an example of the reference image Ib, and FIG. 15C shows the parallax generated by the generation unit 350. A schematic diagram of an image Ip is shown.

  In addition, although the cost calculation part 310, the cost synthetic | combination part 320, the subpixel estimation part 330, and the production | generation part 350 were implement | achieved by FPGA31, ie, implement | achieved by the hardware circuit, it is not limited to this. That is, at least one of the cost calculation unit 310, the cost synthesis unit 320, the subpixel estimation unit 330, and the generation unit 350 may be realized by the CPU 32 executing a program that is software. Moreover, the cost calculation part 310, the cost synthetic | combination part 320, the subpixel estimation part 330, and the production | generation part 350 are the block configurations of a function, Comprising: It is not limited to such a structure. For example, a plurality of functional units illustrated as independent functional units in FIG. 10 may be configured as one functional unit. On the other hand, the function of one functional unit in FIG. 10 may be divided into a plurality of parts and configured as a plurality of functional parts.

(Image processing operation of parallax value deriving device)
FIG. 16 is a flowchart illustrating an example of the stereo matching processing operation of the disparity value deriving device according to the present embodiment. The flow of the image processing operation based on the stereo matching process (SGM method) of the parallax value deriving device 1 will be described with reference to FIG.

<Step S1-1>
The image acquisition unit 110 of the parallax value deriving device 1 images a front subject with the left camera (imaging unit 10a), generates analog image data, and generates a luminance image that is an image based on the image data. obtain. As a result, image data to be subjected to subsequent image processing is obtained. And it transfers to step S2-1.

<Step S1-2>
The image acquisition unit 110 of the parallax value deriving device 1 images a front subject with the right camera (imaging unit 10b), generates analog image data, and generates a luminance image that is an image based on the image data. obtain. As a result, image data to be subjected to subsequent image processing is obtained. Then, the process proceeds to step S2-2.

<Step S2-1>
The conversion unit 210 of the parallax value deriving device 1 removes noise from analog image data obtained by imaging by the imaging unit 10a and converts the image data into digital image data. In this way, by converting the image data into digital image data, image processing for each pixel can be performed on an image based on the image data. And it transfers to step S3-1.

<Step S2-2>
The conversion unit 210 of the parallax value deriving device 1 removes noise from analog image data obtained by imaging by the imaging unit 10b and converts the analog image data into digital image data. In this way, by converting the image data into digital image data, image processing for each pixel can be performed on an image based on the image data. Then, the process proceeds to step S3-2.

<Step S3-1>
The converter 210 outputs an image based on the digital image data converted in step S2-1 as the comparison image Ia in the stereo matching process. Thus, an image to be compared for obtaining a parallax value in the stereo matching process is obtained. Then, the process proceeds to step S4.

<Step S3-2>
The converter 210 outputs an image based on the digital image data converted in step S2-2 as the reference image Ib in the stereo matching process. Thus, an image serving as a reference for obtaining the parallax value in the stereo matching process is obtained. Then, the process proceeds to step S4.

<Step S4>
The filter unit 310a of the cost calculation unit 310 of the parallax value deriving device 1 determines that the shift amount d of the candidate pixel q (x + d, y) of the comparison image Ia to be compared with the reference pixel p (x, y) of the reference image Ib is a threshold value. It is determined whether it is less than Th1, a threshold value Th1 or more and less than a threshold value Th2, or a threshold value Th2 or more.

  When the filter unit 310a determines that the shift amount d is less than the threshold value Th1, the filter unit 310a reads a low-pass filter having 1 × 1 dimension and a filter coefficient of “1” from the storage unit 340. Then, the filter unit 310a performs filter processing (convolution calculation) using the read low-pass filter on the reference pixel p and the candidate pixel q, and obtains the luminance values of the reference pixel p and the candidate pixel q after the filter processing.

  Further, when the filter unit 310a determines that the shift amount d is greater than or equal to the threshold value Th1 and less than the threshold value Th2, the filter unit 310a reads the low-pass filter F1 having 1 × 3 dimensions and a filter coefficient of “1/3” from the storage unit 340. Then, the filter unit 310a performs a filtering process (convolution calculation) on the reference pixel p and the candidate pixel q using the read low-pass filter F1, and obtains luminance values of the reference pixel p and the candidate pixel q after the filtering process. .

  Further, when the filter unit 310a determines that the shift amount d is equal to or greater than the threshold value Th2, the filter unit 310a reads the low-pass filter F2 having 1 × 5 dimensions and a filter coefficient of “1/5” from the storage unit 340. Then, the filter unit 310a performs a filtering process (convolution calculation) on the reference pixel p and the candidate pixel q using the read low-pass filter F2, and obtains luminance values of the reference pixel p and the candidate pixel q after the filtering process. .

  The calculation unit 310b of the cost calculation unit 310 of the parallax value derivation device 1 uses the candidate pixel q based on the luminance values of the reference pixel p and the candidate pixel q after the filter processing by the filter unit 310a using the low-pass filter. The cost value C is calculated.

  In this way, the cost calculation unit 310 does not use the luminance values of the pixels constituting the reference image Ib and the comparison image Ia, but uses a low-pass filter having a size corresponding to the shift amount d for the luminance value. A cost value C is calculated based on the luminance value after performing the filtering process. Then, the process proceeds to step S5.

<Step S5>
The cost composition unit 320 of the parallax value deriving device 1 calculates the cost value C of the pixel in the comparison image Ia for the reference pixel when the pixel around the reference pixel p (x, y) in the reference image Ib is used as the reference pixel. Then, the cost value C (p, d) of the candidate pixel q (x + d, y) calculated by the cost calculation unit 310 is aggregated to obtain the combined cost value Ls (p, d) of the candidate pixel q (x + d, y). calculate. Thus, the cost synthesis unit 320 calculates the synthesis cost value Ls using the cost value C calculated by the cost calculation unit 310 using the low-pass filter. Then, the process proceeds to step S6.

<Step S6>
The sub-pixel estimation unit 330 of the parallax value deriving device 1 calculates the shift amount d corresponding to the minimum value of the combined cost value Ls of the pixels in the comparison image Ia for the reference pixel in the reference image Ib calculated by the cost combining unit 320. Based on the combined cost value Ls in the adjacent shift amount d, subpixel estimation is performed. The sub-pixel estimation unit 330 has the parallax value dp as the shift amount d in sub-pixel units corresponding to the minimum value of the approximate curve obtained by sub-pixel estimation (the downwardly convex quadratic curve in FIGS. 13 and 14). Estimated. The estimated parallax value dp is a value indicating the distance to the portion of the subject corresponding to the reference pixel p in the comparison image Ia. Then, the process proceeds to step S7.

<Step S7>
The generation unit 350 of the parallax value deriving device 1 determines the luminance value of each pixel of the reference image Ib based on the sub-pixel unit parallax value dp derived by the sub-pixel estimation unit 330, and the parallax value dp corresponding to the pixel. A parallax image Ip (high-density parallax image) that is an image represented by

  Thereafter, the image data of the parallax image Ip is output via the communication I / F 35 illustrated in FIG. 9, and is based on the parallax image Ip by a device outside the parallax value deriving device 1 (for example, the control device 5). A distance between the imaging units 10a and 10b and the object is calculated. The calculated distance is used for controlling the brake or steering of the vehicle together with the result of the object recognition process performed in a separate process.

(Main effects of this embodiment)
As described above, the cost calculation unit 310 of the parallax value deriving device 1 according to the present embodiment does not use the luminance values of the pixels constituting the reference image Ib and the comparison image Ia, but instead uses the luminance values. Thus, the cost value C is calculated on the basis of the luminance value after the filter processing by the low-pass filter corresponding to the shift amount d. Specifically, if the subject is the same, the image is small when the subject is far from the parallax value deriving device 1 and is large when the subject is close, so when the shift amount d is small (the subject is far), When a low-pass filter with a small size is used and the shift amount d is large (the subject is close), a low-pass filter with a large size is used. As a result, the size of the low-pass filter to be applied is changed according to the magnitude of the shift amount d, so that the accuracy of the derived parallax value can be improved.

  Further, in the parallax value deriving device 1, the sub-pixel estimation unit 330 can derive the parallax value dp in units of sub-pixels, which is a unit smaller than the pixel, and therefore derives a high-precision and dense parallax value dp. And a more accurate parallax image can be obtained.

  In the above-described embodiment, the cost synthesis unit 320 calculates the synthesis cost value Ls using the cost value C calculated by the cost calculation unit 310 using the low-pass filter, and based on the synthesis cost value Ls. Although the parallax value dp is obtained, the present invention is not limited to this. That is, the parallax value dp may be obtained based on the cost value C calculated by the cost synthesis unit 320 using a low-pass filter. This can also be expected to improve the accuracy of the derived parallax value. However, the cost value C of the pixel in the comparison image Ia for the reference pixel when the pixel around the reference pixel p is set as the reference pixel by the cost composition unit 320 is calculated using the candidate pixel q calculated by the cost calculation unit 310. When the synthesis cost value Ls (p, d) of the candidate pixel q is calculated by being aggregated into the cost value C (p, d), it can be expected that the accuracy of the parallax value is cumulatively improved.

  In the above-described embodiment, the cost value C as the degree of coincidence is an evaluation value that represents the degree of dissimilarity, but may be an evaluation value that represents the degree of similarity. In this case, the shift amount d at which the combined cost value Ls is maximum (extreme value) in subpixel units is the parallax value dp.

  Further, in the above-described embodiment, for simplification of description, description has been made on matching in units of pixels. However, the present invention is not limited to this, and matching may be performed in units of a predetermined region including a plurality of pixels. . In this case, the predetermined area including the reference pixel is indicated as the reference area, the predetermined area including the corresponding pixel is indicated as the corresponding area, and the candidate pixel of the corresponding pixel is indicated as the candidate area. The reference area includes the case of only the reference pixel, the corresponding area includes the case of only the corresponding pixel, and the candidate area includes the case of only the candidate pixel.

  In the above-described embodiment, the device (for example, the control device 5) outside the parallax value deriving device 1 calculates the distance Z based on the parallax image (parallax value). However, the present invention is not limited to this. Instead, the CPU 32 of the image processing unit 30 may calculate the distance Z.

  Moreover, although the above-mentioned embodiment demonstrated the parallax value derivation | leading-out apparatus 1 mounted in the motor vehicle as the vehicle 100, it is not limited to this. For example, it may be mounted on a vehicle such as a motorcycle, bicycle, wheelchair, or agricultural cultivator as an example of another vehicle. In addition to a vehicle as an example of a moving body, a moving body such as a robot may be used.

  Furthermore, the robot may be an apparatus such as an industrial robot that is fixedly installed in an FA (Factory Automation) as well as a moving body. Further, the device to be fixedly installed may be not only a robot but also a surveillance camera for security.

  When at least one of the conversion unit 210, the filter unit 310a, the calculation unit 310b, the cost synthesis unit 320, the subpixel estimation unit 330, and the generation unit 350 of the parallax value derivation device 1 is realized by executing a program, The program is provided by being incorporated in advance in a ROM or the like. The program executed by the disparity value deriving device 1 according to the present embodiment is an installable or executable file and can be read by a computer such as a CD-ROM, flexible disk (FD), CD-R, or DVD. The recording medium may be recorded and provided. Further, the program executed by the disparity value deriving device 1 according to the present embodiment may be configured to be provided by being stored on a computer connected to a network such as the Internet and downloaded via the network. The program executed by the disparity value deriving device 1 according to the present embodiment may be configured to be provided or distributed via a network such as the Internet. The program executed by the parallax value deriving device 1 according to the present embodiment has a module configuration including at least one of the above-described functional units. As actual hardware, the CPU 32 downloads the program from the ROM 33 described above. By reading and executing, each functional unit described above is loaded on the main storage device (RAM 34 or the like) and generated.

DESCRIPTION OF SYMBOLS 1 Parallax value derivation | leading-out apparatus 2 Main body 5 Control apparatus 6 Steering wheel 7 Brake pedal 10a, 10b Imaging part 11a, 11b Imaging lens 12a, 12b Aperture 13a, 13b Image sensor 20a, 20b Signal conversion part 21a, 21b CDS
22a, 22b AGC
23a, 23b ADC
24a, 24b Frame memory 30 Image processing unit 31 FPGA
32 CPU
33 ROM
34 RAM
35 Communication I / F
39 bus 50 device control system 100 vehicle 110 image acquisition unit 210 conversion unit 310 cost calculation unit 310a filter unit 310b calculation unit 320 cost synthesis unit 330 subpixel estimation unit 340 storage unit 350 generation unit 510a, 510b imaging device 511a, 511b imaging lens B Baseline length C Cost value d, d1, d2 Shift amount dp Parallax value E Object EL Epipolar line f Focal length F1, F2 Low-pass filter Ia Comparison image Ib Reference image Ip Disparity image Lr Path cost value Ls Composite cost value S, Sa, Sb point Th1, Th2 threshold Z distance

JP 2012-198077 A

Claims (13)

Based on a reference image obtained by imaging the subject by the first imaging means and a comparative image obtained by imaging the subject by the second imaging means at a position different from the position of the first imaging means. A parallax value deriving device for deriving a parallax value indicating parallax with respect to the subject,
The first reference area of the reference image and the comparison image corresponding to the first reference area identified by shifting by a predetermined shift amount from an area corresponding to the first reference area in the comparison image. Filtering means for performing a filtering process with a low-pass filter having a size corresponding to the shift amount with respect to a plurality of candidate areas that are candidates for corresponding areas;
Calculating means for calculating a degree of coincidence between the first reference area on which the filtering process has been performed by the filtering means and each of the plurality of candidate areas on which the filtering process has been performed;
Derivation means for deriving the parallax value between the first reference area and the corresponding area based on the degree of coincidence;
A parallax value deriving device comprising: The filtering means performs a filtering process by the low-pass filter on the luminance value of the first reference region and the luminance values of the plurality of candidate regions,
The calculation means is configured to calculate the luminance value of each of the plurality of candidate areas based on the luminance value of the first reference area on which the filtering process has been performed and the luminance value of the plurality of candidate areas on which the filtering process has been performed. Calculate the degree of match,
The parallax value deriving device according to claim 1, wherein the deriving unit derives the parallax value based on the shift amount corresponding to the extreme value of the value based on the degree of coincidence of the candidate regions in the comparison image.   3. The filtering unit according to claim 1, wherein the filtering unit performs the filter process with the low-pass filter having a smaller size as the shift amount is smaller, and performs the filter process with the low-pass filter having a larger size as the shift amount is larger. The parallax value deriving device described.   The parallax value deriving device according to claim 1, wherein the low-pass filter is a one-dimensional filter.   The parallax value deriving device according to claim 1, wherein the filtering unit does not perform the filtering process when the shift amount is less than a predetermined value. A synthesizing unit that aggregates the matching degrees in the comparison image for the second reference area around the first reference area into the matching degrees of the plurality of candidate areas and obtains a synthesis matching degree for each of the plurality of candidate areas And more,
The parallax value deriving device according to claim 1, wherein the deriving unit derives the parallax value between the first reference area and the corresponding area based on the composite matching degree.   The parallax value deriving device according to any one of claims 1 to 6, further comprising storage means for storing the low-pass filter in advance.   The deriving means calculates the parallax value by sub-pixel estimation based on a plurality of adjacent shift amounts including the shift amount corresponding to the extreme value of the value based on the degree of coincidence of each candidate region in the comparison image. The parallax value deriving device according to claim 1, wherein the parallax value deriving device is derived. The parallax value deriving device according to any one of claims 1 to 8,
A control device that controls a control target based on distance information from the parallax value deriving device to a subject that is obtained from the parallax value derived by the parallax value deriving device;
Equipment control system equipped with.   A moving body comprising the device control system according to claim 9.   A robot comprising the device control system according to claim 9. Based on a reference image obtained by imaging the subject by the first imaging means and a comparative image obtained by imaging the subject by the second imaging means at a position different from the position of the first imaging means. A parallax value derivation method for deriving a parallax value indicating parallax with respect to the subject,
The first reference area of the reference image and the comparison image corresponding to the first reference area identified by shifting by a predetermined shift amount from an area corresponding to the first reference area in the comparison image. A filtering step of performing a filtering process with a low-pass filter having a size corresponding to the shift amount with respect to a plurality of candidate areas that are candidates for corresponding areas;
A calculation step of calculating a degree of coincidence between the first reference area subjected to the filtering process and each of the plurality of candidate areas subjected to the filtering process;
A derivation step for deriving the parallax value between the first reference area and the corresponding area based on the degree of coincidence;
A method for deriving a disparity value. Based on a reference image obtained by imaging the subject by the first imaging means and a comparative image obtained by imaging the subject by the second imaging means at a position different from the position of the first imaging means. , A program for deriving a parallax value indicating parallax with respect to the subject,
The first reference area of the reference image and the comparison image corresponding to the first reference area identified by shifting by a predetermined shift amount from an area corresponding to the first reference area in the comparison image. Filtering means for performing a filtering process with a low-pass filter having a size corresponding to the shift amount with respect to a plurality of candidate areas that are candidates for corresponding areas;
Calculating means for calculating a degree of coincidence between the first reference area on which the filtering process has been performed by the filtering means and each of the plurality of candidate areas on which the filtering process has been performed;
Derivation means for deriving the parallax value between the first reference area and the corresponding area based on the degree of coincidence;
A program to make a computer realize.