JP6337504B2 - Image processing apparatus, moving body, robot, device control method and program - Google Patents

Description

  The present invention relates to an image processing apparatus, a moving body, a robot, a device control method, and a program.

  In recent years, in-vehicle systems that prevent collision of automobiles by measuring the distance between automobiles or the distance between automobiles and obstacles have been generally used. Such an in-vehicle system, for example, monitors the front of an automobile, measures the distance from a pedestrian, another vehicle, an obstacle, etc., and may collide with various recognition processes or other vehicles. Performs steering control for controlling the steering to avoid the collision, or braking control for controlling the brake to avoid the collision.

  As a method for measuring such a distance, a stereo camera is used and a stereo matching process using the principle of triangulation is used. Stereo matching processing refers to finding the parallax by matching the corresponding pixels between a reference image captured by one camera and a comparison image captured by the other camera of the two cameras of the stereo camera. In this process, the distance between the stereo camera and the object included in the image is calculated from the parallax.

  As a stereo matching process as described above, when the correlation value (evaluation value) of a predetermined area (block) of a pair of images captured by two cameras is calculated, and the position of the predetermined area with the highest correlation can be identified In addition, there is a block matching process for calculating a shift amount between pixels in the predetermined area as a parallax. As a technique using such a block matching process, a technique has been proposed in which the processing speed is increased by using a texture characteristic value using edge information of an image in the block matching process (Patent Document 1). reference).

  However, although the block matching process can calculate parallax relatively quickly and with high accuracy, it is difficult to extract features as an image in a portion where the texture that is not an edge portion is weak in the image. In some cases, high parallax cannot be obtained. On the other hand, parallax values can be obtained by taking into consideration the evaluation value of the entire image, not local information, as in block matching, and the parallax can be accurately calculated even in areas with weak textures such as road surfaces. A process using an SGM (Semi-Global Matching) method, which is a dense algorithm, has been proposed (see Non-Patent Document 1).

  However, processing using a dense algorithm such as the SGM method (hereinafter referred to as dense processing) requires more computation than block matching processing, requires a large number of processing steps, requires many resources such as hardware, and takes a long processing time. Take it. Therefore, it is difficult to obtain the parallax value in real time, and there is a problem that the performance of recognition processing may be affected.

  The present invention has been made in view of the above, and provides an image processing apparatus, a moving body, a robot, a device control method, and a program that suppress the influence on the performance of recognition processing even when dense processing is used. Objective.

In order to solve the above-described problems and achieve the object, the present invention is obtained by the reference image obtained by the first imaging unit imaging the subject and the second imaging unit imaging the subject. Acquisition means for acquiring a comparison image; dense processing means for performing a density process based on the reference image and the comparison image to derive a first parallax value; and a block matching process based on the reference image and the comparison image And a matching processing means for deriving a second parallax value and detecting whether each pixel constituting the reference image is a pixel corresponding to an edge portion or a pixel corresponding to a non-edge portion, and is information of a detection result Detection means for outputting detection information, and when the detection information indicates that the pixel of the reference image corresponding to the detection information is a pixel corresponding to a non-edge portion, When the parallax value is output and indicates that the pixel of the reference image corresponding to the detection information is a pixel corresponding to the edge portion, the selection unit that outputs the second parallax value corresponding to the pixel and the selection unit outputs Generating means for generating a parallax image based on the first parallax value and the second parallax value .

  According to the present invention, the influence on the performance of the recognition process can be suppressed even if the dense process is used.

FIG. 1 is a diagram for explaining the principle of deriving the distance from an imaging device to an object. FIG. 2 is a conceptual diagram illustrating a reference image, a dense parallax image, and an edge parallax image. FIG. 3 is an explanatory diagram for obtaining a corresponding pixel in a comparison image corresponding to a reference pixel in the reference image. FIG. 4 is a graph showing the relationship between the shift amount and the cost value. FIG. 5 is a conceptual diagram for deriving the synthesis cost. FIG. 6 is a graph showing the relationship between the shift amount and the synthesis cost value. FIG. 7 is a diagram illustrating an example in which the device control system according to the first embodiment is mounted on a vehicle. FIG. 8 is a diagram illustrating an example of the appearance of the parallax value deriving device according to the first embodiment. FIG. 9 is a diagram illustrating an example of a hardware configuration of the disparity value deriving device according to the first embodiment. FIG. 10 is a diagram illustrating an example of a block configuration of the disparity value deriving device according to the first embodiment. FIG. 11 is a diagram illustrating an example of an image captured by the parallax value deriving device according to the first embodiment. FIG. 12 is a diagram illustrating a scanning operation of dense processing for the pixels of the reference image in the first embodiment. FIG. 13 is a graph showing the relationship between the shift amount and the synthesis cost value in the first embodiment. FIG. 14 is a diagram for explaining subpixel estimation by parabolic fitting. FIG. 15 is a diagram for explaining subpixel estimation by the method of least squares. FIG. 16 is a conceptual diagram showing a dense parallax image using sub-pixel estimation. FIG. 17 is a diagram illustrating an example of an operation flow of stereo matching processing by dense processing of the disparity value deriving device according to the first embodiment. FIG. 18 is a diagram illustrating an example of an operation flow of stereo matching processing by block matching processing of the disparity value deriving device according to the first embodiment. FIG. 19 is a diagram illustrating a timing chart of the process of the device control system according to the first embodiment. FIG. 20 is a diagram illustrating a timing chart of processing of the device control system according to the second embodiment. FIG. 21 is a diagram illustrating a timing chart of processing of the device control system according to the third embodiment. FIG. 22 is a diagram illustrating a scanning operation of dense processing for the pixels of the reference image in the fourth embodiment. FIG. 23 is a diagram illustrating a timing chart of processing of the device control system according to the fourth embodiment. FIG. 24 is a diagram illustrating a scanning operation of dense processing for the pixels of the reference image in the fifth embodiment. FIG. 25 is a diagram illustrating an example of a block configuration of the disparity value deriving device according to the sixth embodiment. FIG. 26 is a diagram illustrating an example of a block configuration of a disparity value deriving device according to the seventh embodiment. FIG. 27 is a diagram illustrating an example of an operation for detecting a non-edge portion.

[Outline of Ranging Method Using SGM Method]
First, an outline of a distance measuring method using the SGM method will be described with reference to FIGS.

(Principles of ranging)
FIG. 1 is a diagram for explaining the principle of deriving the distance from an imaging device to an object. With reference to FIG. 1, the principle of deriving a parallax with respect to an object from the stereo camera by stereo matching processing and measuring the distance from the stereo camera to the object with the parallax value indicating the parallax will be described. In the following, in order to simplify the description, an example of matching in units of pixels instead of matching of a predetermined area (block) will be described.

  The imaging system (stereo camera) illustrated in FIG. 1 includes an imaging device 510a and an imaging device 510b arranged in parallel equiposition. The imaging devices 510a and 510b respectively include imaging lenses 511a and 511b that refract incident light and form an image of an object on an image sensor (not shown) that is a solid-state imaging device. The images captured by the imaging device 510a and the imaging device 510b are referred to as a comparison image Ia and a reference image Ib, respectively. In FIG. 1, the point S on the object E in the three-dimensional space is mapped to a position on a straight line parallel to a straight line connecting the imaging lens 511a and the imaging lens 511b in each of the comparison image Ia and the reference image Ib. Here, the point S mapped to each image is a point Sa (x, y) in the comparative image Ia and a point Sb (X, y) in the reference image Ib. At this time, the parallax value dp is expressed by the following (Equation 1) using the point Sa (x, y) at the coordinates on the comparison image Ia and the point Sb (X, y) at the coordinates on the reference image Ib. It is expressed in

  dp = X−x (Formula 1)

  In FIG. 1, the distance between the point Sa (x, y) in the comparative image Ia and the intersection of the perpendicular line taken from the imaging lens 511a on the imaging surface is Δa, and the point Sb (X, y) in the reference image Ib is The parallax value dp can also be expressed as dp = Δa + Δb, where Δb is the distance between the imaging lens 511b and the intersection of the perpendiculars on the imaging surface.

  Next, a distance Z between the imaging devices 510a and 510b and the object E is derived by using the parallax value dp. Here, the distance Z is a distance from a straight line connecting the focal position of the imaging lens 511a and the focal position of the imaging lens 511b to the point S on the object E. As shown in FIG. 1, using the focal length f of the imaging lens 511a and the imaging lens 511b, the baseline length B that is the length between the imaging lens 511a and the imaging lens 511b, and the parallax value dp, According to 2), the distance Z can be calculated.

  Z = (B × f) / dp (Formula 2)

  From this (Equation 2), it can be seen that the larger the parallax value dp, the smaller the distance Z, and the smaller the parallax value dp, the larger the distance Z.

(SGM method)
Next, a distance measuring method using the SGM method will be described with reference to FIGS.

  FIG. 2 is a conceptual diagram illustrating a reference image, a dense parallax image, and an edge parallax image. 2A shows a reference image, FIG. 2B shows a dense parallax image with respect to the reference image shown in FIG. 2A, and FIG. 2C shows FIG. 2A. It is a conceptual diagram which shows the edge parallax image with respect to the reference | standard image shown in FIG. Here, the reference image corresponds to the reference image Ib shown in FIG. 1 and is an image in which the captured subject is represented by a luminance value. The dense parallax image is an image in which each pixel of the reference image is represented by a parallax value corresponding to each pixel in the reference image derived by the SGM method. The edge parallax image indicates an image in which each pixel of the reference image is represented by a parallax value corresponding to each pixel in the reference image derived by the edge detection method. However, as will be described later, the disparity value that can be derived by the edge detection method is a relatively textured portion such as an edge portion in the reference image, and when the disparity value cannot be derived such as a weak texture portion, An image is configured with a parallax value of “0”.

  The SGM method is a method for appropriately deriving a parallax value even for a portion having a weak texture in an image. Based on the reference image shown in FIG. 2A, the dense parallax image shown in FIG. Is a method of deriving. Note that the edge detection method is a method for deriving the edge parallax image shown in FIG. 2C based on the reference image shown in FIG. According to the SGM method, the dense parallax image is a detailed parallax even on roads and the like where the texture is weaker than the edge parallax image, as can be seen by comparing the dashed ellipses in FIGS. 2B and 2C. Since the distance based on the value can be expressed, more detailed distance measurement can be performed.

  The SGM method is a method of deriving a parallax value by calculating a cost value after calculating a cost value on a comparative image with respect to a reference image, and then calculating a combined cost value after calculating the cost value. The SGM method finally derives a parallax image (here, a dense parallax image) represented by parallax values corresponding to almost all pixels in the reference image.

  In the case of the edge detection method, the cost value is calculated in the same way as the SGM method. However, as in the SGM method, a comparatively strong texture such as an edge portion is calculated without calculating a synthesis cost value. Only the parallax value is calculated.

<Calculation of cost value>
FIG. 3 is an explanatory diagram for obtaining a corresponding pixel in a comparison image corresponding to a reference pixel in the reference image. FIG. 4 is a graph showing the relationship between the shift amount and the cost value. A method for calculating the cost value C (p, d) will be described with reference to FIGS. 3 and 4. 3A is a conceptual diagram showing reference pixels in the reference image, and FIG. 3B is a corresponding pixel candidate in the comparison image corresponding to the reference pixel shown in FIG. It is a conceptual diagram at the time of calculating the cost value of a candidate pixel, shifting the candidate pixel to become sequentially (shifting). Here, the corresponding pixel indicates a pixel in the comparison image that is most similar to the reference pixel in the reference image. The cost value is an evaluation value that represents the degree of dissimilarity of each candidate pixel in the comparison image with respect to the reference pixel in the reference image. That is, the cost value (and synthesis cost value) shown below indicates that the smaller the value, the more similar the candidate pixel in the comparative image is to the reference pixel.

  As shown in FIG. 3A, a predetermined reference pixel p (x, y) in the reference image Ib and a corresponding pixel candidate on the epipolar line EL in the comparison image Ia with respect to the reference pixel p (x, y). Based on each luminance value of a candidate pixel q (x + d, y), a cost value C (p, d) of a candidate pixel q (x + d, y) corresponding to the reference pixel p (x, y) is calculated. . d is a shift amount (shift amount) between the reference pixel p and the candidate pixel q, and the shift amount d is shifted in units of pixels. That is, in FIG. 3, the candidate pixel q (x + d, y) and the reference pixel p are sequentially shifted by one pixel within a predetermined range (for example, 0 <d <25). A cost value C (p, d), which is the dissimilarity of the luminance value with (x, y), is calculated.

  As described above, since the imaging devices 510a and 510b are arranged in parallel equiposition, the comparison image Ia and the reference image Ib are also in parallel equivalence relations. Therefore, the corresponding pixel in the comparison image Ia corresponding to the reference pixel p in the reference image Ib exists on the epipolar line EL shown in FIG. 3, and in order to obtain the corresponding pixel in the comparison image Ia, the comparison image Ia The pixel on the epipolar line EL may be searched.

  The cost value C (p, d) calculated in this way is represented by the graph shown in FIG. 4 in relation to the shift amount d in pixel units. In the example of FIG. 4, the cost value C is “0” when the shift amount d = 5, 12, and 19, and thus the minimum value cannot be obtained. For example, when there is a portion where the texture is weak in the image, it is difficult to obtain the minimum value of the cost value C in this way.

<Calculation of synthetic cost value>
FIG. 5 is a conceptual diagram for deriving the synthesis cost. FIG. 6 is a graph showing the relationship between the shift amount and the synthesis cost value. A method for calculating the combined cost value Ls (p, d) will be described with reference to FIGS. 5 and 6.

  The calculation of the synthesis cost value is peculiar to the SGM method, and not only the calculation of the cost value C (p, d) but also the cost when the peripheral pixels of the reference pixel p (x, y) are used as the reference pixels. The values are aggregated into the cost value C (p, d) at the reference pixel p (x, y), and the combined cost value Ls (p, d) is calculated. In order to calculate the combined cost value Ls (p, d), first, the route cost value Lr (p, d) is calculated. The route cost value Lr (p, d) is calculated by the following (Formula 3).

Lr (p, d) = C (p, d) + min (Lr (p−r, k) + P (d, k))
(Formula 3)
(P = 0 (when d = k))
P = P1 (when | d−k | = 1),
P = P2 (> P1) (when | dk |> 1))

  As shown in (Formula 3), the route cost value Lr is obtained recursively. Here, r indicates a direction vector in the aggregation direction, and has two components in the x direction and the y direction. min () is a function for obtaining the minimum value. Lr (p−r, k) is the respective path when the shift amount is changed (the shift amount in this case is k) for the pixel having the coordinate shifted by one pixel in the r direction from the coordinate of the reference pixel p. The cost value Lr is shown. Based on the relationship between d, which is the amount of shift of the route cost value Lr (p, d) in the route cost value Lr (pr, k), and the shift amount k, the following (1) The value P (d, k) is obtained as in (3), and Lr (pr, k) + P (d, k) is calculated.

(1) When d = k, P = 0. That is, Lr (p−r, k) + P (d, k) = Lr (p−r, k).
(2) When | d−k | = 1, P = P1. That is, Lr (p−r, k) + P (d, k) = Lr (p−r, k) + P1.
(3) When | d−k |> 1, P = P2 (> P1). That is, Lr (p−r, k) + P (d, k) = Lr (p−r, k) + P2.

  Min (Lr (p−r, k) + P (d, k)) is the smallest of Lr (p−r, k) + P (d, k) calculated in the above (1) to (3). It becomes the value which extracted the value of. That is, the value P1 or the value P1 is exceeded when the pixel located at the coordinate position of the reference pixel p in the comparison image Ia is away from the pixel shifted by the shift amount d from the pixel (p−r) adjacent in the r direction. By adding the large value P2, it is prevented from being excessively influenced by the path cost value Lr of the pixels away from the coordinates of the reference pixel p in the comparison image Ia. Further, the value P1 and the value P2 are fixed parameters determined in advance by experiments, and are parameters such that the parallax values of the reference pixels adjacent on the path are likely to be continuous. For example, P1 = 48 and P2 = 96. In this way, in order to obtain the path cost value Lr for each pixel in the r direction in the comparison image Ia, first, the path cost value Lr from the extreme end pixel in the r direction from the coordinates of the reference image p (x, y). And the route cost value Lr is obtained along the r direction.

Then, as shown in FIG. 5, Lr ₀ , Lr ₄₅ , Lr which are route cost values Lr in eight directions (r ₀ , r ₄₅ , r ₉₀ , r ₁₃₅ , r ₁₈₀ , r ₂₂₅ , r ₂₇₀ and r ₃₁₅ ). ₉₀ , Lr ₁₃₅ , Lr ₁₈₀ , Lr ₂₂₅ , Lr ₂₇₀ and Lr ₃₁₅ are obtained, and finally, a combined cost value Ls (p, d) is obtained based on the following (Equation 4).

  Ls (p, d) = ΣLr (Formula 4)

  As described above, the calculated combined cost value Ls (p, d) can be represented by the graph shown in FIG. 6 in relation to the shift amount d. In the example of FIG. 6, the combined cost value Ls is derived as a parallax value dp = 3 because the minimum value is obtained when the shift amount d = 3. In the above description, the number in the r direction is eight, but the present invention is not limited to this. For example, the eight directions may be further divided into two to be 16 directions, or may be divided into three to be 24 directions. Or it is good also as what calculates | requires the path | route cost value Lr in any of eight directions, and calculates the synthetic | combination cost value Ls.

  Hereinafter, embodiments of an image processing apparatus, a moving body, a robot, a device control method, and a program according to the present invention will be described in detail with reference to the drawings. Here, a case where the parallax value deriving device 1 that performs stereo matching processing is mounted on an automobile will be described as an example. In addition, the present invention is not limited by the following embodiments, and constituent elements in the following embodiments are easily conceivable by those skilled in the art, substantially the same, and so-called equivalent ranges. Is included. Furthermore, various omissions, substitutions, and changes of the components can be made without departing from the scope of the following embodiments.

[First Embodiment]
(Schematic configuration of a vehicle equipped with a parallax value deriving device)
FIG. 7 is a diagram illustrating an example in which the device control system according to the first embodiment is mounted on a vehicle. A vehicle 100 equipped with the device control system 50 according to the present embodiment will be described with reference to FIG. 7A, FIG. 7A is a side view of the vehicle 100 on which the device control system 50 is mounted, and FIG. 7B is a front view of the vehicle 100.

  As shown in FIG. 7, a vehicle 100 that is an automobile includes a device control system 50. The device control system 50 includes a parallax value deriving device 1, a control device 5, a steering wheel 6, and a brake pedal 7 that are installed in a passenger compartment that is a living room space. Note that both the device control system 50 and the parallax value deriving device 1 function as “image processing devices”.

  The parallax value deriving device 1 has an imaging function for imaging the traveling direction of the vehicle 100 and is installed, for example, in the vicinity of the rearview mirror inside the front window of the vehicle 100. The parallax value deriving device 1 includes a main body 2, an imaging unit 10 a (first imaging unit) fixed to the main body 2, and an imaging unit 10 b (second imaging unit), details of which will be described later. The imaging units 10 a and 10 b are fixed to the main body 2 so that a subject in the traveling direction of the vehicle 100 can be imaged.

  The control device 5 executes various recognition processes and various vehicle controls based on the distance information from the parallax value deriving device 1 to the subject obtained based on the image data of the parallax image received from the parallax value deriving device 1. As an example of the recognition process, the control device 5 recognizes a person, a sign, a vehicle, and other obstacles (second subject) whose state change is large, or a texture such as a road surface is weak. A road surface recognition process for recognizing a portion (first subject) whose state change is small is executed. As an example of vehicle control, the control device 5 controls the steering system (control target) including the steering wheel 6 based on the image data of the parallax image received from the parallax value deriving device 1 to avoid an obstacle. Alternatively, brake control or the like for controlling the brake pedal 7 (control target) to decelerate and stop the vehicle 100 is executed. Note that a subject having a large change in state indicates, for example, a subject having a large degree of positional variation in an image, a subject having a strong texture, or the like, such as a person, a sign, or a vehicle.

  By performing recognition processing such as object recognition processing and road surface recognition processing, and vehicle control such as steering control or brake control as in the device control system 50 including the parallax value deriving device 1 and the control device 5 described above. The safety of driving the vehicle 100 can be improved.

  As described above, the parallax value deriving device 1 images the front of the vehicle 100, but is not limited thereto. That is, the parallax value deriving device 1 may be installed so as to image the rear or side of the vehicle 100. In this case, the parallax value deriving device 1 can detect the distance of a subsequent vehicle behind the vehicle 100 or another vehicle that translates laterally. And the control apparatus 5 can detect the danger at the time of the lane change of the vehicle 100, or at the time of lane merge, etc., and can perform the above-mentioned vehicle control. In addition, when the control device 5 determines that there is a risk of a collision based on the parallax image of the obstacle behind the vehicle 100 detected by the parallax value deriving device 1 in the back operation when the vehicle 100 is parked or the like. In addition, the vehicle control described above can be executed.

(Configuration of parallax value deriving device)
FIG. 8 is a diagram illustrating an example of the appearance of the parallax value deriving device according to the first embodiment. As shown in FIG. 8, the parallax value deriving device 1 includes the main body 2, the imaging unit 10a fixed to the main body 2, and the imaging unit 10b as described above. The imaging units 10 a and 10 b are configured by a pair of cylindrical cameras arranged in parallel equiposition with respect to the main body 2. For convenience of explanation, the imaging unit 10a illustrated in FIG. 8 is referred to as a “left” camera, and the imaging unit 10b is referred to as a “right” camera.

<Hardware configuration of parallax value deriving device>
FIG. 9 is a diagram illustrating an example of a hardware configuration of the disparity value deriving device according to the present embodiment. A hardware configuration of the parallax value deriving device 1 will be described with reference to FIG.

  As illustrated in FIG. 9, the parallax value deriving device 1 includes a main body 2, an imaging unit 10 a, an imaging unit 10 b, a signal conversion unit 20 a, a signal conversion unit 20 b, and an image processing unit 30. .

  The main body 2 constitutes the casing of the parallax value deriving device 1, and fixes and supports the imaging units 10a and 10b, and incorporates the signal conversion units 20a and 20b and the image processing unit 30.

  The imaging unit 10a is a processing unit that captures an image of a front subject and generates analog image data. The imaging unit 10a includes an imaging lens 11a, a diaphragm 12a, and an image sensor 13a.

  The imaging lens 11a is an optical element that refracts incident light to form an image of an object on the image sensor 13a. The diaphragm 12a is a member that adjusts the amount of light imaged on the image sensor 13a by blocking part of the light that has passed through the imaging lens 11a. The image sensor 13a is a solid-state imaging device that converts light that has entered the imaging lens 11a and passed through the aperture 12a into electrical analog image data. The image sensor 13a is realized by, for example, a CCD (Charge Coupled Devices) or a CMOS (Complementary Metal Oxide Semiconductor).

  The imaging unit 10b is a processing unit that captures an image of a front subject and generates analog image data. The imaging unit 10b includes an imaging lens 11b, a diaphragm 12b, and an image sensor 13b. The functions of the imaging lens 11b, the diaphragm 12b, and the image sensor 13b are the same as the functions of the imaging lens 11a, the diaphragm 12a, and the image sensor 13a, respectively. In addition, the imaging lens 11a and the imaging lens 11b are installed such that their lens surfaces are in the same plane.

  The signal conversion unit 20a is a processing unit that converts analog image data generated by the imaging unit 10a into digital image data. The signal converter 20a includes a CDS (Correlated Double Sampling) 21a, an AGC (Auto Gain Control) 22a, an ADC (Analog Digital Converter) 23a, and a frame memory 24a.

  The CDS 21a removes noise from the analog image data generated by the image sensor 13a using correlated double sampling, a horizontal differential filter, a vertical smoothing filter, and the like. The AGC 22a performs gain control for controlling the intensity of analog image data from which noise has been removed by the CDS 21a. The ADC 23a converts analog image data whose gain is controlled by the AGC 22a into digital image data. The frame memory 24a stores the image data converted by the ADC 23a.

  The signal conversion unit 20b is a processing unit that converts analog image data generated by the imaging unit 10b into digital image data. The signal conversion unit 20b includes a CDS 21b, an AGC 22b, an ADC 23b, and a frame memory 24b. The functions of the CDS 21b, AGC 22b, ADC 23b, and frame memory 24b are the same as the functions of the CDS 21a, AGC 22a, ADC 23a, and frame memory 24a, respectively.

  The image processing unit 30 is a device that performs image processing on the image data converted by the signal conversion unit 20a and the signal conversion unit 20b. The image processing unit 30 includes an FPGA (Field Programmable Gate Array) 31, a CPU (Central Processing Unit) 32, a ROM (Read Only Memory) 33, a RAM (Random Access Memory) 34, and a communication I / F (Interface). 35, a CANC (CAN Controller) 36, and a bus 39.

  The FPGA 31 is an integrated circuit, and here performs processing for deriving a parallax value in an image based on image data. The CPU 32 controls each function of the parallax value deriving device 1. The ROM 33 stores an image processing program that the CPU 32 executes to control each function of the parallax value deriving device 1. The RAM 34 is used as a work area for the CPU 32. The communication I / F 35 is an interface for communicating with an external controller or the like, and is connected to, for example, an image display device 3 as shown in FIG. The CANC 36 communicates with other control devices (ECU (Electronic Control Unit)) and the like based on a CAN (Controller Area Network) protocol, and is connected to the control device 5 as shown in FIG. 7, for example. The CANC 36 is not limited to performing communication using the CAN protocol, and may perform communication using the FlexRay protocol, for example. As shown in FIG. 9, the bus 39 is an address bus and a data bus that electrically connect the FPGA 31, the CPU 32, the ROM 33, the RAM 34, the communication I / F 35, and the CANC 36.

  The FPGA 31 is not limited to being an FPGA, and may be configured by, for example, an ASIC (Application Specific Integrated Circuit) or a DSP (Digital Signal Processor), or a combination thereof.

  The image display device 3 shown in FIG. 9 is a display device such as a liquid crystal display, an organic EL (Electro-Luminescence) display, or a HUD (Head Up Display). The image display device 3 displays an image based on the image data converted by the signal conversion units 20a and 20b, a parallax image represented by a parallax value derived by the FPGA 31, and the like. As a method of displaying the parallax image, for example, there is a method of displaying in a different color according to each parallax value (or distance information based thereon) on the parallax image.

<Block Configuration of Disparity Value Deriving Device and Operations of Each Block>
FIG. 10 is a diagram illustrating an example of a block configuration of the disparity value deriving device according to the first embodiment. FIG. 11 is a diagram illustrating an example of an image captured by the parallax value deriving device according to the first embodiment. FIG. 12 is a diagram illustrating a scanning operation of dense processing for the pixels of the reference image in the first embodiment. FIG. 13 is a graph showing the relationship between the shift amount and the synthesis cost value in the first embodiment. FIG. 14 is a diagram for explaining subpixel estimation by parabolic fitting. FIG. 15 is a diagram for explaining subpixel estimation by the method of least squares. FIG. 16 is a conceptual diagram showing a dense parallax image using sub-pixel estimation. With reference to FIGS. 10 to 16, the block configuration of the main part of the parallax value deriving device 1 and the operation of each block will be described.

  As shown in FIG. 10, the parallax value derivation device 1 includes an image acquisition unit 110 (acquisition unit), a filter unit 210, a correction unit 310, a first storage unit 350, and a parallax value calculation unit 300 (dense processing unit). , A matching processing unit), a second storage unit 360, and a parallax image generation unit 370 (generation unit). A recognition processing unit 410 is connected to the output side of the parallax image generation unit 370.

  The image acquisition unit 110 is a processing unit that captures an object in front by two left and right cameras, generates image data in analog format, and obtains two luminance images that are images based on each image data. An example of the luminance image captured by the image acquisition unit 110 is shown in FIG. FIG. 11 shows an example of the reference image Ib of the two luminance images. The reference image Ib shown in FIG. 11 includes, for example, an image portion Img1 that is a person image and an image portion that is a vehicle image. Img2, an image portion Img3 that is an image of the road surface, and the like. The image acquisition unit 110 is realized by the imaging unit 10a and the imaging unit 10b illustrated in FIG.

  The filter unit 210 removes noise from the image data of the two luminance images obtained by the image acquisition unit 110, converts the image data into digital image data, and outputs the image data. Here, among the image data of the two luminance images output from the filter unit 210 (hereinafter simply referred to as “luminance image”), the reference image Ib captured by the right camera (imaging unit 10b) of the image acquisition unit 110. Image data (hereinafter, simply referred to as “reference image Ib”) and image data of the comparison image Ia (hereinafter simply referred to as “comparison image Ia”) captured by the left camera (imaging unit 10a). That is, the filter unit 210 outputs the reference image Ib and the comparison image Ia based on the two luminance images output from the image acquisition unit 110. The filter unit 210 is realized by the signal conversion units 20a and 20b illustrated in FIG.

  The correction unit 310 performs an optical shift and a geometric shift between the imaging unit 10a and the imaging unit 10b with respect to the two images (the comparison image Ia and the reference image Ib) input from the filter unit 210. The corrected comparison image Ia and the reference image Ib are output. The correction unit 310 is realized by the FPGA 31 illustrated in FIG.

  The first storage unit 350 is a storage unit that temporarily stores the comparison image Ia and the reference image Ib output from the correction unit 310. The first storage unit 350 is realized by the RAM 34 shown in FIG. The first storage unit 350 may be realized by an external storage device such as an HDD (Hard Disk Drive) not shown in FIG.

  The parallax value calculation unit 300 is a processing unit that performs a dense process and a block matching process, which are stereo matching processes, on the comparison image Ia and the reference image Ib. The parallax value calculation unit 300 includes a first cost calculation unit 320 (first calculation unit), a cost synthesis unit 330 (synthesis unit), a first subpixel estimation unit 340 (first derivation unit), and a second cost calculation. A unit 321 (second calculation unit) and a second sub-pixel estimation unit 341 (second derivation unit). The disparity value calculation unit 300 is realized by the FPGA 31 illustrated in FIG. 9, and in the present embodiment, the above-described processing units included in the disparity value calculation unit 300 are realized by a common hardware circuit (integrated circuit). It shall be. Of the parallax value calculation unit 300, the first cost calculation unit 320, the cost synthesis unit 330, and the first sub-pixel estimation unit 340 perform a dense process on the comparison image Ia and the reference image Ib by operations described later. In addition, in the parallax value calculation unit 300, the second cost calculation unit 321 and the second subpixel estimation unit 341 perform block matching processing on the comparison image Ia and the reference image Ib by operations described later.

  The first cost calculation unit 320 performs the luminance value of the reference pixel p (x, y) (first reference region) in the reference image Ib and the epipolar line in the comparison image Ia based on the reference pixel p (x, y). Each brightness of candidate pixel q (x + d, y) (first candidate region), which is a candidate for the corresponding pixel, specified by shifting by the shift amount d from the pixel corresponding to the position of the reference pixel p (x, y) Based on the value, the first cost value C1 (p, d) of each candidate pixel q (x + d, y) is calculated. As the first cost value C1 calculated by the first cost calculation unit 320, for example, SAD (Sum of Absolute Differences), SSD (Sum of Squared Differences), or the like can be used. As described above, the first cost calculation unit 320 calculates the first cost value C1 of each candidate pixel q in the comparison image Ia for each reference pixel p in the reference image Ib. In this case, as shown in FIG. 12, the first cost calculation unit 320 moves the reference pixel p in the reference image Ib from the upper left pixel one pixel at a time in the horizontal direction and in the vertical direction in the manner of raster scanning. While moving, the first cost value C1 in the comparison image Ia corresponding to each reference pixel p is calculated. An example of a graph showing the relationship between the shift amount d and the first cost value C1 calculated by the first cost calculation unit 320 is the cost value C is changed to the first cost value C1 in the graph shown in FIG. It is a replacement.

The cost synthesizing unit 330 uses the pixel around the reference pixel p (x, y) in the reference image Ib as the reference pixel, and the first cost value of the pixel in the comparison image Ia for the reference pixel (second reference region). C1 is aggregated into the first cost value C1 (p, d) of the candidate pixel q (x + d, y) calculated by the first cost calculation unit 320, and the combined cost value Ls of the candidate pixel q (x + d, y) (P, d) is calculated. The cost synthesis unit 330 calculates a synthesis cost value Ls (p, d) based on the shift amount d that is shifted in units of pixels. In order to calculate the combined cost value Ls, the cost combining unit 330 first calculates a route cost value Lr (p, d) in a predetermined r direction according to the above (Equation 3). Next, as shown in FIGS. 5 and 12, the cost composition unit 330 performs Lr ₀ , Lr ₄₅ , Lr ₉₀ , Lr ₁₃₅ , Lr ₁₈₀ , Lr ₂₂₅ , Lr ₂₇₀ and eight-way route cost values Lr. Lr ₃₁₅ is calculated, and finally, the combined cost value Ls (p, d) is calculated based on the above (Formula 4). And the graph which shows the relationship between the shift amount d of a pixel unit and the synthetic | combination cost value Ls calculated by the cost synthetic | combination part 330 is a graph shown in FIG. As shown in FIG. 13, the combined cost value Ls becomes the minimum value when the shift amount d = 3.

  The first sub-pixel estimation unit 340 calculates the shift amount corresponding to the minimum value (first extreme value) of the combined cost value Ls of the pixels in the comparison image Ia for the reference pixel in the reference image Ib calculated by the cost combining unit 330. Sub-pixel estimation is executed based on the combined cost value Ls at d and the shift amount d adjacent thereto. The graph of the synthesis cost value Ls shown in FIG. 13 is a graph of the synthesis cost value Ls with respect to the shift amount d in pixel units as described above. The minimum value of the synthesis cost value Ls in the graph shown in FIG. 13 is the synthesis cost value Ls at the shift amount d = 3 in pixel units. That is, in the graph of the combined cost value Ls with respect to the shift amount d in pixel units as shown in FIG. 13, it is only possible to derive the value in pixel units as the parallax value. Here, sub-pixel estimation is to estimate and derive a parallax value not in units of pixels but in units smaller than pixels (hereinafter referred to as sub-pixel units) as the first parallax value dp1.

  First, a case where the first subpixel estimation unit 340 performs subpixel estimation by parabolic fitting will be described with reference to FIG. The first sub-pixel estimation unit 340 obtains the value of the shift amount d that minimizes the synthesis cost value Ls in the graph of the synthesis cost value Ls calculated by the cost synthesis unit 330 (FIG. 13). In the example of FIG. 13, the combined cost value Ls is minimum when the shift amount d = 3. Next, the first sub-pixel estimation unit 340 obtains a shift amount d adjacent to the shift amount d = 3. Specifically, the shift amount d = 2,4. Next, in the graph of the shift amount d and the combined cost value Ls shown in FIG. 13, the first subpixel estimation unit 340 calculates three points where the shift amount d = 2, 3 and 4 as shown in FIG. 14. A convex quadratic curve passing through is obtained. Then, the first subpixel estimation unit 340 estimates that the shift amount d in subpixel units corresponding to the minimum value of the quadratic curve is the first parallax value dp1.

  Next, a case where the first subpixel estimation unit 340 performs subpixel estimation by the least square method will be described with reference to FIG. The first sub-pixel estimation unit 340 obtains the value of the shift amount d that minimizes the synthesis cost value Ls in the graph of the synthesis cost value Ls calculated by the cost synthesis unit 330 (FIG. 13). In the example of FIG. 13, the combined cost value Ls is minimum when the shift amount d = 3. Next, the first sub-pixel estimation unit 340 obtains four shift amounts d in the vicinity of the shift amount d = 3. Specifically, the shift amount d = 1, 2, 4, 5. Next, in the graph of the shift amount d and the combined cost value Ls shown in FIG. 13, the first subpixel estimation unit 340 has a shift amount d = 1 to 5 by the least square method as shown in FIG. 15. A downward convex quadratic curve passing through the vicinity of the point is obtained. Then, the first subpixel estimation unit 340 estimates that the shift amount d in subpixel units corresponding to the minimum value of the quadratic curve is the first parallax value dp1.

  The first subpixel estimation unit 340 estimates and derives the first parallax value dp1 by either subpixel estimation by parabolic fitting shown in FIG. 14 or subpixel estimation by the least square method shown in FIG.

  As described above, the synthesis cost value Ls is calculated by the cost synthesis unit 330, and further, the first subpixel estimation unit 340 performs subpixel estimation on the graph of the synthesis cost value Ls. Thereby, even when there is a portion where the texture is weak in the luminance image, the parallax value can be accurately derived, and the parallax value is estimated in units of sub-pixels, so that a dense parallax value can be derived.

  The sub-pixel estimation is not limited to the above-described parabolic fitting or the least-square method, and may be sub-pixel estimation using other methods. For example, the first subpixel estimation unit 340 uses the three points shown in FIG. 14 to perform subpixel estimation by equiangular straight line fitting that estimates a disparity value by obtaining an equiangular straight line that passes through three points instead of a quadratic curve. It is good also as what performs.

  Further, in the subpixel estimation by the least square method, the quadratic curve is obtained using the five points on the graph shown in FIG. 15, but the present invention is not limited to this. A quadratic curve may be obtained.

  The first subpixel estimation unit 340 is not limited to calculating the first parallax value dp1 in subpixel units by subpixel estimation. The first subpixel estimation is not performed, and the first in pixel units is not performed. The parallax value dp1 may be calculated. In this case, the first sub-pixel estimation unit 340 calculates the shift amount d corresponding to the minimum value of the combined cost value Ls of the pixels in the comparison image Ia for the reference pixel in the reference image Ib calculated by the cost combining unit 330. One parallax value dp1 may be used.

  The second cost calculation unit 321 compares the luminance value of a predetermined area (block) (third reference area) including the reference pixel p (x, y) in the reference image Ib and the reference pixel p (x, y). A candidate pixel q (x + d, y), which is a candidate for the corresponding pixel, is identified by shifting by a shift amount d from the pixel corresponding to the position of the reference pixel p (x, y) on the epipolar line in the image Ia. Based on each luminance value of the block (second candidate region), a second cost value C2 (p, d) of the block including the candidate pixel q (x + d, y) is calculated. As the second cost value C2 calculated by the second cost calculation unit 321, for example, SAD (Sum of Absolute Differences), SSD (Sum of Squared Differences), or the like can be used. As described above, the second cost calculation unit 321 calculates the second cost value C2 of the block including each candidate pixel q in the comparison image Ia for each block including the reference pixel p in the reference image Ib. In this case, similarly to the first cost calculation unit 320, the second cost calculation unit 321 sets the reference pixel p in the horizontal direction from the upper left pixel in the reference image Ib as shown in FIG. The second cost value C2 in the comparison image Ia corresponding to the block including each reference pixel p is calculated while moving pixel by pixel and in the vertical direction.

  If the block including the reference pixel p in the reference image Ib is a weak texture portion, a graph showing the relationship between the shift amount d and the second cost value C2 calculated by the second cost calculation unit 321. For example, a graph in which the cost value C is replaced with the second cost value C2 in the graph shown in FIG. 4 described above is obtained. In this case, in the graph shown in FIG. 4, since the shift amount d = 5, 12, 19 is an approximate value as the minimum value of the second cost value C2, the minimum value of the second cost value C2 is obtained. Thus, it is difficult to obtain the corresponding pixel in the comparison image Ia corresponding to the reference pixel. However, when the block including the reference pixel p in the reference image Ib includes an edge portion, that is, a portion having a strong texture, the shift amount d and the second cost value C2 calculated by the second cost calculation unit 321 are used. For example, a graph in which the combined cost value Ls is replaced with the second cost value C2 in the graph shown in FIG. 13 described above is obtained. In this case, in the graph shown in FIG. 13, when the shift amount d = 3, it can be recognized that the second cost value C2 is the minimum value.

  The second subpixel estimation unit 341 calculates the minimum value (second value) of the second cost value C2 of the block of pixels in the comparison image Ia for the block including the reference pixel in the reference image Ib, calculated by the second cost calculation unit 321. Subpixel estimation is executed based on the shift amount d corresponding to the extreme value) and the second cost value C2 in the shift amount d adjacent thereto. When the graph shown in FIG. 13 is a graph showing the relationship between the shift amount d and the second cost value C2, as described above, this is a graph of the second cost value C2 with respect to the shift amount d in pixel units. In this case, the minimum value of the second cost value C2 in the graph is the second cost value C2 at the shift amount d = 3 in pixel units. That is, in the graph of the second cost value C2 with respect to the shift amount d in pixel units as shown in FIG. 13, it is only possible to derive a value in pixel units as the parallax value. Here, as described above with respect to the operation of the first sub-pixel estimation unit 340, the second sub-pixel estimation unit 341 uses the sub-pixel estimation so that the disparity value is not a pixel unit value but a sub-pixel unit disparity smaller than a pixel. The value is estimated and derived as the second parallax value dp2. As described above with respect to the operation of the first subpixel estimation unit 340, the second subpixel estimation unit 341 estimates and derives the second parallax value dp2 by either parabolic fitting or subpixel estimation by the least square method. .

  As described above, the second cost calculation unit 321 calculates the second cost value C2, and the second subpixel estimation unit 341 further performs subpixel estimation on the graph of the second cost value C2. As a result, the parallax value can be accurately derived in the luminance image including the edge portion, that is, the portion having a strong texture, and the parallax value is estimated in sub-pixel units, so that a dense parallax value is derived. Can do.

  In addition, the second disparity estimation unit 341 is not limited to calculating the second disparity value dp2 in subpixel units by subpixel estimation. The parallax value dp2 may be calculated. In this case, the second sub-pixel estimation unit 341 calculates the minimum value of the second cost value C2 in the block of pixels of the comparison image Ia for the block including the reference pixel in the reference image Ib, calculated by the second cost calculation unit 321. The shift amount d corresponding to the second parallax value dp2 may be used.

  The second storage unit 360 is a storage unit that stores the first parallax value dp1 derived by the first subpixel estimation unit 340 and the second parallax value dp2 derived by the second subpixel estimation unit 341. The second storage unit 360 is realized by the RAM 34 shown in FIG. The second storage unit 360 may be realized by an external storage device such as an HDD (not shown in FIG. 9).

  The parallax image generation unit 370 stores the luminance value of each pixel of the reference image Ib by the first parallax value dp1 stored in the second storage unit 360 and derived by the dense processing, and the first parallax value dp1 corresponding to the pixel. A dense parallax image Ip1 (first parallax image) is generated. In addition, the parallax image generation unit 370 stores the luminance of each pixel of the reference image Ib according to the second parallax value dp2 stored in the second storage unit 360 and derived by the block matching process. An edge parallax image Ip2 (second parallax image) that is an image represented by the value dp2 is generated. The parallax image generation unit 370 is realized by the FPGA 31 illustrated in FIG. 16A shows an example of the comparison image Ia, FIG. 16B shows an example of the reference image Ib, and FIG. 16C is generated by the parallax image generation unit 370. A schematic diagram of the dense parallax image Ip1 is shown.

  Based on the dense parallax image Ip1 generated by the parallax image generation unit 370, the recognition processing unit 410 executes a road surface recognition process for recognizing a weak texture portion such as a road surface as a recognition process. In addition, the recognition processing unit 410 executes object recognition processing for recognizing a person, a sign, a vehicle, and other obstacles as recognition processing based on the edge parallax image Ip2 generated by the parallax image generation unit 370. The recognition processing unit 410 is realized by the image display device 3 shown in FIG. 9 or the control device 5 shown in FIG. Note that the recognition processing unit 410 is not limited to being realized in the image display device 3 or the control device 5 that is an external device of the parallax value deriving device 1. That is, the recognition processing by the recognition processing unit 410 may be executed by the disparity value deriving device 1, and in this case, the recognition processing unit 410 may be realized by the FPGA 31 illustrated in FIG.

(Density processing operation of parallax value deriving device)
FIG. 17 is a diagram illustrating an example of an operation flow of stereo matching processing by dense processing of the disparity value deriving device according to the first embodiment. The flow of the image processing operation based on the dense processing of the disparity value deriving device 1 will be described with reference to FIG.

<Step S1-1>
The image acquisition unit 110 of the parallax value deriving device 1 images a front subject with the left camera (imaging unit 10a), generates analog image data, and generates a luminance image that is an image based on the image data. obtain. Then, the process proceeds to step S2-1.

<Step S1-2>
The image acquisition unit 110 of the parallax value deriving device 1 images a front subject with the right camera (imaging unit 10b), generates analog image data, and generates a luminance image that is an image based on the image data. obtain. Then, the process proceeds to step S2-2.

<Step S2-1>
The filter unit 210 of the parallax value deriving device 1 removes noise from analog image data obtained by imaging by the imaging unit 10a and converts the image data into digital image data. Then, the process proceeds to step S3-1.

<Step S2-2>
The filter unit 210 of the parallax value deriving device 1 removes noise from analog image data obtained by imaging by the imaging unit 10b and converts the analog image data into digital image data. Then, the process proceeds to step S3-2.

<Step S3-1>
The filter unit 210 outputs an image based on the digital image data converted in step S2-1 as the comparison image Ia in the stereo matching process. Then, the process proceeds to step S4.

<Step S3-2>
The filter unit 210 outputs an image based on the digital image data converted in step S2-2 as the reference image Ib in the stereo matching process. Then, the process proceeds to step S4.

<Step S4>
The correction unit 310 of the parallax value deriving device 1 has an optical shift between the image capturing unit 10a and the image capturing unit 10b, and a geometry with respect to the two images (the comparison image Ia and the reference image Ib) input from the filter unit 210. The geometrical deviation is corrected, and the corrected comparison image Ia and reference image Ib are output. The first storage unit 350 of the parallax value derivation device 1 temporarily stores the comparison image Ia and the reference image Ib output from the correction unit 310. The first cost calculation unit 320 of the parallax value deriving device 1 inputs the comparison image Ia and the reference image Ib output from the correction unit 310 or the comparison image Ia and the reference image Ib stored in the first storage unit 350. . The first cost calculation unit 320 uses the reference pixel p (x, y) on the epipolar line in the comparison image Ia based on the luminance value of the reference pixel p (x, y) in the reference image Ib and the reference pixel p (x, y). Each candidate pixel q (x + d, y) is determined based on each luminance value of the candidate pixel q (x + d, y), which is a candidate for the corresponding pixel, specified by shifting from the pixel corresponding to the position y) by the shift amount d. The first cost value C1 (p, d) of y) is calculated. Then, the process proceeds to step S5.

<Step S5>
The cost composition unit 330 of the parallax value deriving device 1 calculates the cost value C of the pixel in the comparison image Ia for the reference pixel when the pixel around the reference pixel p (x, y) in the reference image Ib is used as the reference pixel. The cost value C (p, d) of the candidate pixel q (x + d, y) calculated by the first cost calculation unit 320 is aggregated, and the combined cost value Ls (p, d) of the candidate pixel q (x + d, y) ) Is calculated. In order to calculate the synthesis cost value Ls, the cost synthesis unit 330 calculates the comparison image Ia and the reference image for Lr ₀ , Lr ₄₅ and Lr ₉₀ among the route cost values Lr (p, d) in a predetermined r direction. Calculation is possible even if the entire Ib (that is, the entire pixel value to be configured) is not stored in the first storage unit 350. That is, the cost composition unit 330 calculates the first cost calculated by the first cost calculation unit 320 using the directly input pixel value among the pixel values constituting the comparison image Ia and the reference image Ib output from the correction unit 310. Based on the value C1, Lr ₀ , Lr ₄₅ and Lr ₉₀ can be calculated. On the other hand, the cost combining unit 330 calculates Lr ₁₃₅ , Lr ₁₈₀ , Lr ₂₂₅ , Lr ₂₇₀ and Lr ₃₁₅ among predetermined route cost values Lr (p, d) in the r direction in order to calculate the combined cost value Ls. Cannot be calculated unless the entire comparison image Ia and the reference image Ib (that is, the entire pixel value) is stored in the first storage unit 350. That is, the cost composition unit 330 outputs the first cost calculated by the first cost calculation unit 320 with the pixel values that form the comparison image Ia and the reference image Ib that are output from the correction unit 310 and stored in the first storage unit 350. Based on the value C1, Lr ₁₃₅ , Lr ₁₈₀ , Lr ₂₂₅ , Lr ₂₇₀ and Lr ₃₁₅ are calculated. Then, the process proceeds to step S6.

<Step S6>
The first sub-pixel estimation unit 340 of the parallax value deriving device 1 calculates the shift amount corresponding to the minimum value of the combined cost value Ls of the pixels in the comparison image Ia for the reference pixel in the reference image Ib, calculated by the cost combining unit 330. Sub-pixel estimation is executed based on the combined cost value Ls at d and the shift amount d adjacent thereto. The first sub-pixel estimation unit 340 determines that the shift amount d in sub-pixel units corresponding to the minimum value of the approximate curve obtained by the sub-pixel estimation (the downwardly convex quadratic curve in FIGS. 14 and 15) is the first parallax. Estimated to be the value dp1. Then, the process proceeds to step S7.

<Step S7>
The second storage unit 360 of the parallax value derivation device 1 stores the first parallax value dp1 derived by the first subpixel estimation unit 340. The parallax image generation unit 370 of the parallax value deriving device 1 stores the luminance value of each pixel of the reference image Ib corresponding to the pixel by the first parallax value dp1 stored in the second storage unit 360 and derived by the dense processing. A dense parallax image Ip1 that is an image represented by the first parallax value dp1 is generated.

  Thereafter, the image data of the dense parallax image Ip1 is output via the communication I / F 35 or the CANC 36, and the recognition processing unit 410 that is an apparatus external to the parallax value deriving apparatus 1 (for example, the image display apparatus 3 or the control apparatus 5). Thus, a recognition process such as a road surface recognition process based on the dense parallax image Ip1 is executed.

(Block matching processing operation of parallax value deriving device)
FIG. 18 is a diagram illustrating an example of an operation flow of stereo matching processing by block matching processing of the disparity value deriving device according to the first embodiment. The flow of the image processing operation based on the block matching process of the disparity value deriving device 1 will be described with reference to FIG.

<Step S11-1>
The image acquisition unit 110 of the parallax value deriving device 1 images a front subject with the left camera (imaging unit 10a), generates analog image data, and generates a luminance image that is an image based on the image data. obtain. Then, the process proceeds to step S12-1.

<Step S11-2>
The image acquisition unit 110 of the parallax value deriving device 1 images a front subject with the right camera (imaging unit 10b), generates analog image data, and generates a luminance image that is an image based on the image data. obtain. Then, the process proceeds to step S12-2.

<Step S12-1>
The filter unit 210 of the parallax value deriving device 1 removes noise from analog image data obtained by imaging by the imaging unit 10a and converts the image data into digital image data. Then, the process proceeds to step S13-1.

<Step S12-2>
The filter unit 210 of the parallax value deriving device 1 removes noise from analog image data obtained by imaging by the imaging unit 10b and converts the analog image data into digital image data. Then, the process proceeds to step S13-2.

<Step S13-1>
The filter unit 210 outputs an image based on the digital image data converted in step S12-1 as the comparison image Ia in the stereo matching process. Then, the process proceeds to step S14.

<Step S13-2>
The filter unit 210 outputs an image based on the digital image data converted in step S12-2 as the reference image Ib in the stereo matching process. Then, the process proceeds to step S14.

<Step S14>
The correction unit 310 of the parallax value deriving device 1 has an optical shift between the image capturing unit 10a and the image capturing unit 10b, and a geometry with respect to the two images (the comparison image Ia and the reference image Ib) input from the filter unit 210. The geometrical deviation is corrected, and the corrected comparison image Ia and reference image Ib are output to the second cost calculation unit 321. The second cost calculation unit 321 of the parallax value deriving device 1 compares the luminance value of a predetermined area (block) including the reference pixel p (x, y) in the reference image Ib and the reference pixel p (x, y). A candidate pixel q (x + d, y), which is a candidate for the corresponding pixel, is identified by shifting by a shift amount d from the pixel corresponding to the position of the reference pixel p (x, y) on the epipolar line in the image Ia. Based on each luminance value of the block, a second cost value C2 (p, d) of the block including the candidate pixel q (x + d, y) is calculated. Then, the process proceeds to step S15.

<Step S15>
The second subpixel estimation unit 341 of the disparity value deriving device 1 calculates the second cost value C2 of the block of the pixel in the comparison image Ia for the block including the reference pixel in the reference image Ib calculated by the second cost calculation unit 321. The sub-pixel estimation is executed based on the shift amount d corresponding to the minimum value of and the second cost value C2 in the shift amount d adjacent thereto. The second subpixel estimation unit 341 determines that the shift amount d in subpixel units corresponding to the minimum value of the approximate curve (downward convex secondary curve in FIGS. 14 and 15) obtained by the subpixel estimation is the second parallax. Estimated to be the value dp2. Then, the process proceeds to step S16.

<Step S16>
The second storage unit 360 of the parallax value derivation device 1 stores the second parallax value dp2 derived by the second subpixel estimation unit 341. The parallax image generation unit 370 of the parallax value deriving device 1 stores the luminance of each pixel of the reference image Ib according to the second parallax value dp2 stored in the second storage unit 360 and derived by the block matching process. An edge parallax image Ip2 that is an image represented by the second parallax value dp2 is generated.

  Thereafter, the image data of the edge parallax image Ip2 is output via the communication I / F 35 or the CANC 36, and the recognition processing unit 410 that is an apparatus outside the parallax value deriving apparatus 1 (for example, the image display apparatus 3 or the control apparatus 5). Thus, a recognition process such as an object recognition process based on the edge parallax image Ip2 is executed.

(Operation timing of device control system processing)
FIG. 19 is a diagram illustrating a timing chart of the process of the device control system according to the first embodiment. With reference to FIG. 19, the operation timing of processing of the device control system 50 according to the present embodiment will be described.

  As shown in FIG. 19, when the effective frame signal FV is in the on state, the image acquisition unit 110 obtains one luminance image, that is, one luminance image, respectively, by the imaging units 10a and 10b.

  The block matching processing by the second cost calculation unit 321 and the second subpixel estimation unit 341 of the parallax value calculation unit 300 is performed by the derivation of the second parallax value dp2 and the edge parallax even during the luminance image acquisition by the image acquisition unit 110. The image Ip2 can be generated (described as “parallax value calculation” in FIG. 19), and the load of the calculation process is lighter than that of the dense process. Accordingly, in the block matching process, every time one set of the comparison image Ia and the reference image Ib is acquired, the parallax value calculation and the recognition process based on the edge parallax image Ip2 by the recognition processing unit 410 (“object recognition” in FIG. 19). (Exemplary processing) can be executed.

On the other hand, the dense processing performed by the first cost calculation unit 320, the cost synthesis unit 330, and the first sub-pixel estimation unit 340 of the parallax value calculation unit 300 is first performed with the acquisition of the luminance image of the image acquisition unit 110, the comparison image Ia, Storage of the image Ib in the first storage unit 350 is started. Further, as described above, Lr ₀ , Lr _45, and Lr ₉₀ among the predetermined route cost values Lr in the r direction can be calculated even while the luminance image is acquired by the image acquisition unit 110. However, as described above, in the dense processing, the luminance image is acquired by the image acquisition unit 110 for Lr ₁₃₅ , Lr ₁₈₀ , Lr ₂₂₅ , Lr ₂₇₀ and Lr ₃₁₅ out of the predetermined route cost value Lr in the r direction. The calculation cannot be performed unless the comparison image Ia and the reference image Ib are stored in the first storage unit 350. Therefore, the dense processing has a larger calculation amount and a longer processing time than the block matching processing, and the storage of the comparison image Ia and the reference image Ib in the first storage unit 350 has been completed for some operations. It cannot be executed without it. Therefore, the derivation of the first parallax value dp1 by the parallax value calculation unit 300 and the generation of the dense parallax image Ip1 by the parallax image generation unit 370 (described as “parallax value calculation” in FIG. 19) are the parallax values in the block matching process. Processing time is longer than calculation.

  In the example of the time chart of FIG. 19, in the dense process, for example, the parallax value calculation can be executed once while one set of the comparison image Ia and the reference image Ib is acquired twice. The recognition processing based on the dense parallax image Ip1 by the recognition processing unit 410 (exemplified as “road surface recognition processing” in FIG. 19) can be executed once. However, every time a set of the comparison image Ia and the reference image Ib is acquired twice, the road surface state of the road is usually accompanied by a rapid change even if the road surface recognition process or the like by the recognition processing unit 410 is performed. Since it is not a thing, there is little influence on the performance of recognition processing. In addition to road surface recognition processing, subject recognition processing that is not accompanied by abrupt changes in the image may be executed based on the dense parallax image Ip1 generated by dense processing with a heavy processing load. There is little impact on the performance of the recognition process. In the example of the time chart of FIG. 19, the parallax value calculation is executed once while one set of the comparison image Ia and the reference image Ib is acquired twice in the dense process. As long as there is little influence on the performance, the parallax value calculation may be executed once while image acquisition is performed three or more times.

  As described above, in the disparity value deriving device 1 according to the present embodiment, the disparity value calculation unit 300 that performs the dense processing and the block matching processing is realized by common hardware (integrated circuit). And block matching processing are executed in parallel. In this case, since the block matching processing is less computationally expensive than the dense processing, every time one set of the comparison image Ia and the reference image Ib is acquired, the parallax value calculation and the recognition processing based on the edge parallax image Ip2 Can be executed. On the other hand, since dense processing is more computationally intensive than block matching processing, parallax value computation and dense parallax images are obtained while one set of comparison image Ia and reference image Ib is acquired multiple times. The recognition process based on Ip1 can be executed once. In this way, dense processing and block matching processing can be operated in parallel, and recognition processing for a subject that does not change easily (such as a road surface of a road) is processed by a dense parallax image. The recognition processing for a vehicle or the like is processed by an edge parallax image. As a result, even if dense processing with a heavy processing load is used, the influence on the performance of various recognition processing can be suppressed.

  Further, in the parallax value deriving device 1, the first subpixel estimation unit 340 and the second subpixel estimation unit 341 can derive the parallax value in units of subpixels, which is a unit smaller than a pixel, so that the accuracy is high. In addition, a dense parallax value can be derived, and a more accurate parallax image can be obtained.

  The correction unit 310, the first cost calculation unit 320, the cost synthesis unit 330, the first subpixel estimation unit 340, the second cost calculation unit 321, the second subpixel estimation unit 341, and the parallax image generation unit 370 are performed by the FPGA 31. Although it is assumed to be realized by a hardware circuit (integrated circuit), it is not limited to this. That is, at least one of the correction unit 310, the first cost calculation unit 320, the cost synthesis unit 330, the first subpixel estimation unit 340, the second cost calculation unit 321, the second subpixel estimation unit 341, and the parallax image generation unit 370. May be realized by the CPU 32 executing an image processing program that is software. The correction unit 310, the first cost calculation unit 320, the cost synthesis unit 330, the first subpixel estimation unit 340, the second cost calculation unit 321, the second subpixel estimation unit 341, and the parallax image generation unit 370 have functions. It is a block configuration conceptually and is not limited to such a configuration.

  Further, the above-described image processing program may be recorded in a computer-readable recording medium and distributed as a file in an installable format or an executable format. This recording medium is a CD-ROM or an SD card.

[Second Embodiment]
The disparity value deriving device and device control system according to the second embodiment will be described focusing on differences from the disparity value deriving device and device control system according to the first embodiment. The disparity value deriving device 1 and the device control system 50 according to the present embodiment are the same as the first embodiment with respect to the configuration and the contents of the dense process and the block matching process.

(Operation timing of device control system processing)
FIG. 20 is a diagram illustrating a timing chart of processing of the device control system according to the second embodiment. The operation timing of processing of the device control system 50 according to the present embodiment will be described with reference to FIG.

  As shown in FIG. 20, when the effective frame signal FV is in the on state, the image acquisition unit 110 obtains one luminance image by the imaging units 10 a and 10 b, that is, one luminance image.

  The device control system 50 according to the present embodiment executes only block matching processing, dense processing, and block matching processing every time the image acquisition unit 110 acquires a predetermined number of sets of luminance images (a predetermined number of pages). It is assumed that the execution of parallel processing with is switched. In other words, the device control system 50 always executes the block matching process, and the dense process repeats execution and stop of the process for each set of predetermined luminance images.

  As a result, a period during which both dense processing and block matching processing are not executed simultaneously is provided, so that the amount of calculation can be reduced. The operation of the device control system 50 according to the present embodiment is particularly effective when recognition processing based on dense parallax images such as road surface recognition processing is not required.

  In FIG. 20, the number of pages on which only block matching processing is executed and the number of pages on which parallel processing of dense processing and block matching processing is executed are described as being the same, but this is not limitative. Is not to be done. For example, as described above, in the case where recognition processing based on dense parallax images such as road surface recognition processing is not required, the number of pages on which only block matching processing is executed is calculated as parallel processing of dense processing and block matching processing. May be larger than the number of pages executed.

[Third Embodiment]
A parallax value deriving device and device control system according to a third embodiment will be described focusing on differences from the parallax value deriving device and device control system according to the first embodiment. The disparity value deriving device 1 and the device control system 50 according to the present embodiment are the same as the first embodiment with respect to the configuration and the contents of the dense process and the block matching process.

(Operation timing of device control system processing)
FIG. 21 is a diagram illustrating a timing chart of processing of the device control system according to the third embodiment. With reference to FIG. 21, the operation timing of processing of the device control system 50 according to the present embodiment will be described.

  As shown in FIG. 21, when the effective frame signal FV is in the on state, the image acquisition unit 110 obtains one luminance image by the imaging units 10a and 10b, that is, one luminance image.

  The device control system 50 according to the present embodiment performs only block matching processing and only dense processing every time the image acquisition unit 110 acquires a predetermined number of sets of luminance images (a predetermined number of pages). Are to be switched. Further, the recognition processing unit 410 executes both the object recognition process and the road surface recognition process based on the dense parallax image Ip1 based on the first parallax value dp1 derived by the dense process in a period in which only the dense process is executed.

  As a result, it is not necessary to execute the block matching process and the dense process in parallel, so that the scale of the hardware circuit can be reduced, and the calculation amount, cost, and power consumption can be reduced.

[Fourth Embodiment]
A parallax value deriving device and device control system according to a fourth embodiment will be described focusing on differences from the parallax value deriving device and device control system according to the first embodiment. The disparity value deriving device 1 and the device control system 50 according to the present embodiment are the same as those in the first embodiment with respect to the configuration and the contents of the block matching process.

(Density processing operation of parallax value deriving device)
FIG. 22 is a diagram illustrating a scanning operation of dense processing for the pixels of the reference image in the fourth embodiment. With reference to FIG. 22, the dense process of the disparity value deriving device 1 according to the present embodiment will be described.

  Like the reference image Ib shown in FIG. 11 described above, it is considered that subjects with a lot of changes such as people and vehicles are often scattered throughout the reference image Ib. Often present in the lower half of Ib. Therefore, the road surface recognition process is particularly necessary not in the entire dense parallax image Ip1 generated by the parallax image generation unit 370 in the first embodiment, but in the lower half of the dense parallax image Ip1. . Therefore, in the disparity value deriving device 1 according to the present embodiment, as shown in FIG. 22, the dense processing target is the lower half of the reference image Ib.

  Specifically, the first cost calculation unit 320 corresponds to the luminance value of the reference pixel p in the lower half of the reference image Ib and the position of the reference pixel p on the epipolar line in the comparison image Ia based on the reference pixel p. The first cost value C1 of each candidate pixel q is calculated based on each luminance value of the candidate pixel q that is a candidate for the corresponding pixel specified by shifting from the pixel to be shifted by the shift amount d. In addition, the cost composition unit 330 calculates the first cost value C1 of the pixel in the comparison image Ia for the reference pixel when the pixel around the reference pixel p in the lower half of the reference image Ib is the reference pixel. The combined cost value Ls of the candidate pixel q is calculated by consolidating the first cost value C1 of the candidate pixel q calculated by the cost calculation unit 320. The first subpixel estimation unit 340 also calculates the shift amount corresponding to the minimum value of the combined cost value Ls of the pixels in the comparison image Ia for the reference pixel in the lower half of the reference image Ib, calculated by the cost combining unit 330. Subpixel estimation is performed based on d and the combined cost value Ls in the shift amount d adjacent thereto, and a first parallax value dp1 in subpixel units is derived. Then, the parallax image generation unit 370 stores each of the lower half of the reference image Ib based on the first parallax value dp1 corresponding to the lower half of the reference image Ib that is stored in the second storage unit 360 and derived by the dense processing. A dense parallax image that is an image in which the luminance value of a pixel is represented by the first parallax value dp1 corresponding to the pixel is generated.

  As described above, since the object on which the dense process is executed is the lower half of the reference image Ib, the processing time of the dense process can be halved.

(Operation timing of device control system processing)
FIG. 23 is a diagram illustrating a timing chart of processing of the device control system according to the fourth embodiment. With reference to FIG. 23, the operation timing of processing of the device control system 50 according to the present embodiment will be described.

  As shown in FIG. 23, when the effective frame signal FV is in an on state, the image acquisition unit 110 obtains one luminance image by the imaging units 10a and 10b, that is, one luminance image.

  As described above, the processing time of the dense process can be halved by setting the target for performing the dense process to the lower half of the reference image Ib. Therefore, as shown in FIG. 23, the dense processing is performed by the parallax value calculation and the recognition processing unit 410 every time one set of the comparison image Ia and the reference image Ib is acquired, as in the block matching processing. The recognition process based on the parallax image (half the size of the dense parallax image Ip1) can be executed.

  As a result, it is possible to reduce the load of calculation processing by dense processing. In addition, each time a set of the comparison image Ia and the reference image Ib is acquired, the recognition process based on the dense parallax image can be executed, so that the performance of the recognition process can be ensured and the dense parallax can be ensured. It is possible to cope with a sudden change in a subject that has been subjected to recognition processing based on an image.

  Note that although the lower half of the reference image Ib is the target of the dense processing, the present invention is not limited to this, and may be a portion below a specific position in the vertical direction of the reference image Ib.

[Fifth Embodiment]
A parallax value deriving device and device control system according to a fifth embodiment will be described focusing on differences from the parallax value deriving device and device control system according to the first embodiment. The disparity value deriving device 1 and the device control system 50 according to the present embodiment are the same as those in the first embodiment with respect to the configuration and the contents of the block matching process.

(Density processing operation of parallax value deriving device)
FIG. 24 is a diagram illustrating a scanning operation of dense processing for the pixels of the reference image in the fifth embodiment. With reference to FIG. 24, the dense process of the disparity value deriving device 1 according to the present embodiment will be described.

  Like the reference image Ib shown in FIG. 11 described above, it is considered that subjects with a lot of changes such as people and vehicles are often scattered throughout the reference image Ib. Often present in the lower half of Ib. Therefore, it is not necessary for the cost composition unit 330 to aggregate the first cost values C1 from all eight directions as shown in FIG. 5, and in the present embodiment, as shown in FIG. 24, 0 degrees, 180 degrees, The first cost value C1 is aggregated from five directions (specific directions) of 225 degrees, 270 degrees, and 315 degrees.

Specifically, in order to calculate the synthesis cost value Ls, the cost synthesis unit 330 calculates the above-described five-way route cost values Lr (p, d), that is, Lr ₀ , Lr, according to the above (Equation 3). ₁₈₀ , Lr ₂₂₅ , Lr ₂₇₀ and Lr ₃₁₅ are calculated, and the sum thereof is taken as the combined cost value Ls. A dense parallax image is obtained by the operations of the first sub-pixel estimation unit 340 and the parallax image generation unit 370.

  As described above, since the calculation of the route cost value Lr by the cost composition unit 330 in the case of executing the dense process is reduced from eight directions to five directions, the processing time of the dense process can be halved. Thus, as in the fourth embodiment, the dense processing is performed every time one set of the comparison image Ia and the reference image Ib is acquired, as in the block matching processing. It is possible to execute recognition processing based on dense parallax images. In addition, each time a set of the comparison image Ia and the reference image Ib is acquired, the recognition process based on the dense parallax image can be executed, so that the performance of the recognition process can be ensured and the dense parallax can be ensured. It is possible to cope with a sudden change in a subject that has been subjected to recognition processing based on an image.

Note that the cost synthesis unit 330 calculates the route cost value Lr (p, d) in five directions, but is not limited to this, and may be three directions or four directions. As the three directions, for example, the cost composition unit 330 calculates Lr ₂₂₅ , Lr ₂₇₀ and Lr ₃₁₅ , and as the four directions, for example, the cost composition unit 330 uses Lr ₀ , Lr ₂₂₅ , Lr ₂₇₀ and Lr _315. May be calculated.

[Sixth Embodiment]
The disparity value deriving device and device control system according to the sixth embodiment will be described focusing on differences from the disparity value deriving device and device control system according to the first embodiment. The disparity value deriving device 1 and the device control system 50 according to the present embodiment are the same as the first embodiment with respect to the contents of the dense processing and the block matching processing.

<Block Configuration of Disparity Value Deriving Device>
FIG. 25 is a diagram illustrating an example of a block configuration of the disparity value deriving device according to the sixth embodiment. A block configuration of a main part of the parallax value deriving device 1 will be described with reference to FIG.

  As illustrated in FIG. 25, the parallax value derivation device 1 includes an image acquisition unit 110, a filter unit 210, a correction unit 310, a first storage unit 350, a parallax value calculation unit 300a (dense processing unit), and parallax. A value calculation unit 300b (matching processing means), a second storage unit 360, and a parallax image generation unit 370 are provided. A recognition processing unit 410 is connected to the output side of the parallax image generation unit 370.

  The parallax value calculation unit 300a includes a first cost calculation unit 320, a cost synthesis unit 330, and a first subpixel estimation unit 340, and performs a dense process on the comparison image Ia and the reference image Ib. . The parallax value calculation unit 300b includes a second cost calculation unit 321 and a second subpixel estimation unit 341, and executes block matching processing on the comparison image Ia and the reference image Ib. In other words, the disparity value deriving device 1 according to the present embodiment is replaced with the disparity value calculating unit 300 realized by the same hardware circuit (integrated circuit) in the disparity value deriving device 1 of the first embodiment. In addition, a disparity value calculation unit 300a and a disparity value calculation unit 300b, which are separate hardware circuits (integrated circuits), are provided. The functions of the respective processing units included in the parallax value deriving device 1 shown in FIG. 25 are the same as those in the first embodiment.

  As described above, the parallax value calculation unit 300a and the parallax value calculation unit 300b are configured by separate hardware circuits, so that dense processing and block matching processing can be executed in parallel. Can be executed independently by the wear circuit. As a result, both dense processing and block matching processing can be processed at a higher speed than in the first embodiment, and can be executed with a timing chart as shown in FIG. Therefore, each time a set of the comparison image Ia and the reference image Ib is acquired, the recognition process based on the dense parallax image can be executed, so that the performance of the recognition process can be ensured and the dense parallax can be ensured. It is possible to cope with a sudden change in a subject that has been subjected to recognition processing based on an image.

[Seventh Embodiment]
The disparity value deriving device and device control system according to the seventh embodiment will be described focusing on differences from the disparity value deriving device and device control system according to the sixth embodiment.

<Block Configuration of Disparity Value Deriving Device and Operations of Each Block>
FIG. 26 is a diagram illustrating an example of a block configuration of a disparity value deriving device according to the seventh embodiment. FIG. 27 is a diagram illustrating an example of an operation for detecting a non-edge portion. With reference to FIG. 26 and FIG. 27, the block configuration of the main part of the parallax value deriving device 1 and the operation of each block will be described.

  As illustrated in FIG. 26, the parallax value derivation device 1 includes an image acquisition unit 110, a filter unit 210, a correction unit 310, a first storage unit 350, a parallax value calculation unit 300a, and a parallax value calculation unit 300b. , A detection unit 380 (detection unit), a selection unit 390 (selection unit), a second storage unit 360a, and a parallax image generation unit 370a (generation unit). A recognition processing unit 410 (recognition processing means) is connected to the output side of the parallax image generation unit 370a.

  The parallax value calculation unit 300a includes a first cost calculation unit 320, a cost synthesis unit 330, and a first subpixel estimation unit 340, and performs a dense process on the comparison image Ia and the reference image Ib. . The parallax value calculation unit 300b includes a second cost calculation unit 321 and a second subpixel estimation unit 341, and executes block matching processing on the comparison image Ia and the reference image Ib. That is, the disparity value deriving device 1 according to the present embodiment is similar to the disparity value deriving device 1 according to the sixth embodiment, and the disparity value calculating unit 300a, which is a separate hardware circuit, and the disparity value calculating unit 300b.

  The detection unit 380 receives the reference image Ib output from the correction unit 310, detects whether each pixel constituting the reference image Ib is a pixel corresponding to an edge portion or a pixel corresponding to a non-edge portion, and detects it. Detection information as result information is passed to the selection unit 390. For example, the detection unit 380 detects whether the pixels constituting the reference image Ib are edge portions or non-edge portions as follows. As shown in FIG. 27A, it is assumed that the pixel of interest to be detected is “E”, and the adjacent pixels that are adjacent to the pixel of interest “E” are denoted by “A to D, F to I”. To do. The detection unit 380 calculates the absolute value of the difference between the pixel value of “E” that is the target pixel and the pixel values of “A to D and F to I” that are adjacent pixels. Next, the detection unit 380 determines whether each of the calculated eight absolute values is equal to or less than a predetermined threshold Th. Then, as illustrated in FIG. 27B, the detection unit 380 determines that “E” that is the target pixel is a non-edge portion when all of the eight absolute values are equal to or less than a predetermined threshold Th. judge. In addition, when the condition of FIG. 27B is not satisfied, the detection unit 380 determines that “E” as the pixel of interest is an edge portion.

  The selection unit 390 receives the first parallax value dp1 from the first subpixel estimation unit 340, the second parallax value dp2 from the second subpixel estimation unit 341, and the detection information from the detection unit 380. Then, based on the detection information indicating whether or not the specific pixel of the reference image Ib is a non-edge part, the selection unit 390 includes the first parallax value dp1 or the second parallax value dp2 corresponding to the specific pixel. Either one is output and stored in the second storage unit 360a. Specifically, when the detection information corresponding to the specific pixel of the reference image Ib is information indicating a non-edge portion, the selection unit 390 outputs the first parallax value dp1 corresponding to the specific pixel, The first parallax value dp1 associated with the specific pixel is stored in the second storage unit 360a. Further, when the detection information corresponding to the specific pixel of the reference image Ib is information indicating an edge portion, the selection unit 390 outputs the second parallax value dp2 corresponding to the specific pixel, The associated second parallax value dp2 is stored in the second storage unit 360a. The selection unit 390 is realized by the FPGA 31 illustrated in FIG.

  The second storage unit 360a stores the first parallax value dp1 and the second parallax value dp2 output from the selection unit 390 in association with the pixel of the reference image Ib. The second storage unit 360a is realized by the RAM 34 shown in FIG. The second storage unit 360a may be realized by an external storage device such as an HDD (not shown in FIG. 9).

  The parallax image generation unit 370a uses the first parallax value dp1 and the second parallax value dp2 stored in association with each pixel of the reference image Ib in the second storage unit 360a to calculate the luminance value of each pixel of the reference image Ib. A mixed parallax image Ip3 that is an image represented by the first parallax value dp1 or the second parallax value dp2 corresponding to the pixel is generated. That is, the mixed parallax image Ip3 is generated by mixing the first parallax value dp1 and the second parallax value dp2, and the pixel corresponding to the first parallax value dp1 corresponds to a non-edge portion in the reference image Ib. The pixel corresponding to the second parallax value dp2 corresponds to the pixel corresponding to the edge portion in the reference image Ib. The parallax image generation unit 370a is realized by the FPGA 31 illustrated in FIG.

  The recognition processing unit 410 is based on the mixed parallax image Ip3 generated by the parallax image generation unit 370a, and recognizes a road surface recognition process for recognizing a weak textured part such as a road surface of a road, a person, a sign, a vehicle, and other obstacles. Both object recognition processing and the like for recognizing an object are executed. The recognition processing unit 410 is realized by the image display device 3 shown in FIG. 9 or the control device 5 shown in FIG.

  The correction unit 310, the first cost calculation unit 320, the cost synthesis unit 330, the first subpixel estimation unit 340, the second cost calculation unit 321, the second subpixel estimation unit 341, the detection unit 380, the selection unit 390, and the parallax The image generation unit 370a is realized by the FPGA 31, that is, is realized by a hardware circuit, but is not limited thereto. That is, the correction unit 310, the first cost calculation unit 320, the cost synthesis unit 330, the first subpixel estimation unit 340, the second cost calculation unit 321, the second subpixel estimation unit 341, the detection unit 380, the selection unit 390, and the parallax At least one of the image generation units 370a may be realized by the CPU 32 executing a program that is software. In addition, the correction unit 310, the first cost calculation unit 320, the cost synthesis unit 330, the first subpixel estimation unit 340, the second cost calculation unit 321, the second subpixel estimation unit 341, the detection unit 380, the selection unit 390, and the parallax The image generation unit 370a has a conceptual block configuration of functions, and is not limited to such a configuration.

  As described above, the disparity value deriving device 1 according to the present embodiment detects whether or not each pixel of the reference image Ib, which is a luminance image, is a non-edge part. The first parallax value dp1 derived by the parallax value calculation unit 300a is used for a certain pixel, and the second parallax value dp2 derived by the parallax value calculation unit 300b is used for a pixel that is an edge unit. The mixed parallax image Ip3 is generated. And the recognition process part 410 shall perform various recognition processes using this mixed parallax image Ip3. As a result, the common mixed parallax image Ip3 is used both in the road surface recognition process for recognizing a weak textured part such as a road surface and the object recognition process for recognizing a person, a sign, a vehicle, and other obstacles. Can be processed. And since the parallax value which comprises mixed parallax image Ip3 becomes a parallax value suitable for any recognition process, the performance of a recognition process can be improved.

  In the above-described embodiment, the cost value C, the first cost value C1, and the second cost value C2 are evaluation values representing dissimilarities, but may be evaluation values representing similarity. In this case, the shift amount d that maximizes the combined cost value Ls in subpixel units is the first parallax value dp1.

  In the above-described embodiment, for the sake of simplification of description, the description has been made on matching in units of pixels. However, the present invention is not limited to this, and matching may be performed in units of predetermined regions (blocks). In this case, the predetermined area including the reference pixel is indicated as the reference area, the predetermined area including the corresponding pixel is indicated as the corresponding area, and the candidate pixel of the corresponding pixel is indicated as the candidate area. The reference area includes the case of only the reference pixel, the corresponding area includes the case of only the corresponding pixel, and the candidate area includes the case of only the candidate pixel.

  In the above-described embodiment, the device (for example, the image display device 3) outside the parallax value deriving device 1 calculates the distance Z based on the parallax image (parallax value). However, the present invention is not limited to this. Instead, the CPU 32 of the image processing unit 30 may calculate the distance Z.

  Moreover, although the above-mentioned embodiment demonstrated the parallax value derivation | leading-out apparatus 1 mounted in the motor vehicle as the vehicle 100, it is not limited to this. For example, it may be mounted on a vehicle such as a motorcycle, bicycle, wheelchair, or agricultural cultivator as an example of another vehicle. In addition to a vehicle as an example of a moving body, a moving body such as a robot may be used.

  Furthermore, the robot may be an apparatus such as an industrial robot that is fixedly installed in an FA (Factory Automation) as well as a moving body. Further, the device to be fixedly installed may be not only a robot but also a surveillance camera for security.

DESCRIPTION OF SYMBOLS 1 Parallax value derivation | leading-out apparatus 2 Main body 3 Image display apparatus 5 Control apparatus 6 Steering wheel 7 Brake pedal 10a, 10b Imaging part 11a, 11b Imaging lens 12a, 12b Aperture 13a, 13b Image sensor 20a, 20b Signal conversion part 21a, 21b CDS
22a, 22b AGC
23a, 23b ADC
24a, 24b Frame memory 30 Image processing unit 31 FPGA
32 CPU
33 ROM
34 RAM
35 Communication I / F
36 CANC
39 bus 50 device control system 100 vehicle 110 image acquisition unit 210 filter unit 300, 300a, 300b parallax value calculation unit 310 correction unit 320 first cost calculation unit 321 second cost calculation unit 330 cost synthesis unit 340 first subpixel estimation unit 341 Second sub-pixel estimation unit 350 First storage unit 360, 360a Second storage unit 370, 370a Parallax image generation unit 380 Detection unit 390 Selection unit 410 Recognition processing unit 510a, 510b Imaging device 511a, 511b Imaging lens B Base length C Cost value C1 first cost value C2 second cost value d shift amount dp parallax value dp1 first parallax value dp2 second parallax value E object EL epipolar line f focal length FV effective frame signal Ia comparison image Ib reference image Img1 to Img3 image Department Ip1 Dense Differential image Ip2 edge parallax image Ip3 mixture parallax image Lr path cost value Ls synthesis cost value S, Sa, Sb point Th threshold Z distance

JP 2012-018013 A

H. Hirschmuller, Accurate and Efficient Stereo Processing by Semi-Global Matching and Mutual Information, IEEE Conference on Computer Vision and Patter.

Claims (13)

An acquisition means for acquiring a reference image obtained by imaging the subject by the first imaging means, and a comparative image obtained by imaging the subject by the second imaging means;
Dense processing means for performing a dense process based on the reference image and the comparison image to derive a first parallax value;
Matching processing means for performing block matching processing based on the reference image and the comparison image to derive a second parallax value;
Detecting means for detecting whether each pixel constituting the reference image is a pixel corresponding to an edge portion or a pixel corresponding to a non-edge portion, and outputs detection information which is information of a detection result;
When the detection information indicates that the pixel of the reference image corresponding to the detection information is a pixel corresponding to the non-edge portion, the first parallax value corresponding to the pixel is output, and the detection information A selection unit that outputs the second parallax value corresponding to the pixel when the corresponding pixel of the reference image indicates a pixel corresponding to the edge portion;
Generating means for generating a parallax image based on the first parallax value and the second parallax value output by the selection means;
An image processing apparatus. Recognition processing means for executing recognition processing for the first subject using the first parallax value and executing recognition processing for the second subject having a greater change in state than the first subject using the second parallax value. The image processing apparatus according to claim 1, further comprising: The dense processing means is:
Shifting by a shift amount in units of pixels from the region corresponding to the position of the first reference region on the luminance value of the first reference region in the reference image and the epipolar line in the comparison image based on the first reference region. And the first cost value of each of the plurality of first candidate regions based on the luminance values of the plurality of first candidate regions that are candidates for the corresponding region of the comparison image corresponding to the first reference region. First calculating means for calculating
The first cost value in the comparison image for the second reference region around the first reference region is aggregated into the first cost value of the first candidate region, and the combined cost of each of the plurality of first candidate regions A synthesis means for calculating a value;
First derivation means for deriving the first parallax value based on the shift amount corresponding to a first extreme value among the combined cost values of the plurality of first candidate regions in the comparison image;
The image processing apparatus according to claim 1, further comprising: The matching processing means includes
Shifting by a shift amount in units of pixels from the region corresponding to the position of the third reference region on the luminance value of the third reference region in the reference image and the epipolar line in the comparison image based on the third reference region. And the second cost value of each of the plurality of second candidate regions based on the brightness values of the plurality of second candidate regions that are candidates for the corresponding region of the comparison image corresponding to the third reference region. Second calculating means for calculating
Second derivation means for deriving the second parallax value based on the shift amount corresponding to a second extreme value among the second cost values of the plurality of second candidate regions in the comparison image;
The image processing apparatus according to claim 1, further comprising: The first calculation means shifts the luminance value of the first reference region in the lower part from the predetermined position of the reference image and the pixel unit shift from the region corresponding to the position of the first reference region on the epipolar line. Calculating a first cost value for each of the plurality of first candidate regions based on the luminance values of the plurality of first candidate regions identified by shifting by an amount;
The synthesizing unit uses the first cost value in the comparison image for the second reference region around the first reference region in the lower portion from the predetermined position of the reference image as the first cost of the first candidate region. A composite cost value of each of the plurality of first candidate regions is aggregated into a value,
The first deriving unit derives the first parallax value corresponding to a lower portion from the predetermined position of the reference image,
The image processing apparatus according to claim 3, wherein the generation unit generates a parallax image based on the first parallax value corresponding to a lower portion from the predetermined position of the reference image.   The synthesizing unit aggregates the first cost value in the comparison image for the second reference region adjacent in a plurality of specific directions of the first reference region into the first cost value of the first candidate region, The image processing device according to claim 3, wherein a route cost value in each of the specific directions of each of the plurality of first candidate regions is calculated, and the combined cost value is calculated by summing the route cost values.   The image processing apparatus according to claim 1, wherein the dense processing unit and the matching processing unit are configured by separate integrated circuits. An acquisition means for acquiring a reference image obtained by imaging the subject by the first imaging means, and a comparative image obtained by imaging the subject by the second imaging means;
Dense processing means for performing a dense process based on the reference image and the comparison image to derive a first parallax value;
Matching processing means for performing block matching processing based on the reference image and the comparison image to derive a second parallax value;
Detecting means for detecting whether each pixel constituting the reference image is a pixel corresponding to an edge portion or a pixel corresponding to a non-edge portion, and outputs detection information which is information of a detection result;
When the detection information indicates that the pixel of the reference image corresponding to the detection information is a pixel corresponding to the non-edge portion, the first parallax value corresponding to the pixel is output, and the detection information A selection unit that outputs the second parallax value corresponding to the pixel when the corresponding pixel of the reference image indicates a pixel corresponding to the edge portion;
Generating means for generating a parallax image based on the first parallax value and the second parallax value output by the selection means;
With
The dense processing means and the matching processing means are image processing apparatuses configured by separate integrated circuits. An acquisition means for acquiring a reference image obtained by imaging the subject by the first imaging means, and a comparative image obtained by imaging the subject by the second imaging means;
Dense processing means for performing a dense process based on the reference image and the comparison image to derive a first parallax value;
Matching processing means for performing block matching processing based on the reference image and the comparison image to derive a second parallax value;
Detecting means for detecting whether each pixel constituting the reference image is a pixel corresponding to an edge portion or a pixel corresponding to a non-edge portion, and outputs detection information which is information of a detection result;
When the detection information indicates that the pixel of the reference image corresponding to the detection information is a pixel corresponding to the non-edge portion, the first parallax value corresponding to the pixel is output, and the detection information A selection unit that outputs the second parallax value corresponding to the pixel when the corresponding pixel of the reference image indicates a pixel corresponding to the edge portion;
Generating means for generating a parallax image based on the first parallax value and the second parallax value output by the selection means;
With
One of the dense processing means and the matching processing means is constituted by an integrated circuit, and the other is realized by executing a program. Mobile with an image processing apparatus according to any one of claims 1-9. Robot having an image processing apparatus according to any one of claims 1-9. An acquisition step of acquiring a reference image obtained by imaging the subject by the first imaging means and a comparative image obtained by imaging the subject by the second imaging means;
A dense processing step of performing a dense process based on the reference image and the comparison image to derive a first parallax value;
A matching process step of performing a block matching process based on the reference image and the comparison image to derive a second parallax value;
A detection step of detecting whether each pixel constituting the reference image is a pixel corresponding to an edge portion or a pixel corresponding to a non-edge portion, and outputting detection information which is information of a detection result;
When the detection information indicates that the pixel of the reference image corresponding to the detection information is a pixel corresponding to the non-edge portion, the first parallax value corresponding to the pixel is output, and the detection information A selection step of outputting the second parallax value corresponding to the pixel when the corresponding pixel of the reference image indicates that the pixel corresponds to the edge portion;
Generating a parallax image based on the output first parallax value and the second parallax value;
A device control method. An acquisition means for acquiring a reference image obtained by imaging the subject by the first imaging means, and a comparative image obtained by imaging the subject by the second imaging means;
Dense processing means for performing a dense process based on the reference image and the comparison image to derive a first parallax value;
Matching processing means for performing block matching processing based on the reference image and the comparison image to derive a second parallax value;
Detecting means for detecting whether each pixel constituting the reference image is a pixel corresponding to an edge portion or a pixel corresponding to a non-edge portion, and outputs detection information which is information of a detection result;
When the detection information indicates that the pixel of the reference image corresponding to the detection information is a pixel corresponding to the non-edge portion, the first parallax value corresponding to the pixel is output, and the detection information A selection unit that outputs the second parallax value corresponding to the pixel when the corresponding pixel of the reference image indicates a pixel corresponding to the edge portion;
Generating means for generating a parallax image based on the first parallax value and the second parallax value output by the selection means;
A program to make a computer realize.