WO2021245972A1 - Computation device and parallax calculation method - Google Patents

Abstract

This computation device comprises an input part into which a base image acquired by a first imaging part and a reference image acquired by a second imaging part are inputted, a target point setting part for setting a parallax calculation target pixel which is a target pixel for calculating parallax in the base image, a comparison point setting part for setting a comparison target pixel which is a pixel candidate corresponding to the parallax calculation target pixel in the reference image, a storage part for storing standard shape information which prescribes an estimation target region based on a predetermined pixel, a pre-correction base region specification part for specifying a pre-correction base region based on the parallax calculation target pixel from the standard shape information, a pre-correction comparison region specification part for specifying a pre-correction comparison region based on the comparison target pixel from the standard shape information, an estimation value generation part for calculating an estimation value for each pixel included in the pre-correction base region, the estimation value being an index of similarity to a corresponding pixel in the pre-correction comparison region, a correction part for correcting the standard shape information into corrected shape information from comparison between the estimation value and a threshold value, thereby maintaining or narrowing the estimation target region, and a parallax calculation part for calculating parallax of the parallax calculation target pixel from information of the base image in a post-correction base region specified on the basis of the parallax calculation target pixel from the corrected shape information and information of the reference image in a post-correction comparison region specified on the basis of the comparison target pixel from the corrected shape information.

Description

Arithmetic logic unit, parallax calculation method

The present invention relates to an arithmetic unit and a parallax calculation method.

A method of calculating the distance between the camera and the observation object using a stereo camera and performing recognition processing of the observation object is widely known. Patent Document 1 describes an image processing apparatus that performs image processing of a plurality of image data having different viewpoints, the acquisition means for acquiring the plurality of image data, and the plurality of image data acquired by the acquisition means. Among them, the reference image area is set in the image of the first image data which is the reference image, the reference image area is set in the image of the second image data which is the reference image, and the reference image area and the reference image area are set. The arithmetic processing means includes an arithmetic processing means for calculating the amount of misalignment of the plurality of image data by a correlation calculation with the image data, and the arithmetic processing means has a brightness change value of an adjacent pixel at a boundary portion in a discriminatory direction of the reference image region less than a threshold value. It is characterized in that the reference image region having a shape is determined, and the correlation calculation is performed between the determined reference image region and the reference image region having a shape corresponding to the shape of the reference image region. The image processing apparatus is disclosed.

Japanese Patent Application Laid-Open No. 2020-21126

In the invention described in Patent Document 1, there is room for improvement in the calculation accuracy of parallax.

The arithmetic unit according to the first aspect of the present invention is a target for calculating the difference between the reference image acquired by the first imaging unit, the input unit into which the reference image acquired by the second imaging unit is input, and the reference image. A target point setting unit that sets a target pixel for parallax calculation, which is a pixel, a comparison point setting unit that sets a comparison target pixel that is a candidate for a pixel in the reference image corresponding to the target pixel for parallax calculation, and a predetermined pixel as a reference. A storage unit that stores standard shape information that defines the evaluation target area to be evaluated, a pre-correction reference area specifying unit that specifies a pre-correction reference area based on the misalignment calculation target pixel based on the standard shape information, and the above. Based on the standard shape information, the pre-correction comparison area specifying unit that specifies the pre-correction comparison area based on the comparison target pixel, and the pre-correction comparison area for each pixel included in the pre-correction reference area. The evaluation target area is obtained by correcting the standard shape information to the corrected shape information based on the comparison between the evaluation value and the threshold value and the evaluation value generation unit that calculates the evaluation value which is an index similar to the corresponding pixel. With the correction unit for maintaining or narrowing, the information of the reference image in the corrected reference region specified based on the correction target pixel based on the correction shape information, and the comparison target pixel based on the correction shape information. It is provided with a parallax calculation unit that calculates the misalignment of the parallax calculation target pixel based on the information of the reference image in the specified corrected comparison region.
In the method for calculating the disparity according to the second aspect of the present invention, an evaluation target based on a reference image acquired by the first imaging unit, an input unit into which a reference image acquired by the second imaging unit is input, and a predetermined pixel is used as a reference. It is a parallax calculation method executed by a calculation device including a storage unit for storing standard shape information defining an area, and sets a misparity calculation target pixel which is a target pixel for calculating parallax in the reference image. To specify a comparison target pixel that is a pixel candidate in the reference image corresponding to the disparity calculation target pixel, and to specify a pre-correction reference region based on the disparity calculation target pixel based on the standard shape information. And, based on the standard shape information, the pre-correction comparison region based on the comparison target pixel is specified, and the pre-correction comparison region corresponds to each pixel included in the pre-correction reference region. Maintaining or narrowing the evaluation target area by calculating an evaluation value which is an index similar to a pixel and correcting the standard shape information to the corrected shape information based on the comparison between the evaluation value and the threshold value. And the information of the reference image in the corrected reference region specified based on the corrected shape information and the corrected reference pixel specified as a reference, and the corrected comparison specified based on the corrected shape information based on the comparison target pixel. It includes calculating the parallax of the misparity calculation target pixel based on the information of the reference image in the region.

According to the present invention, the calculation accuracy of parallax can be improved.

Functional block diagram of arithmetic unit Diagram explaining the definition of terms Detailed diagram of the functional configuration of the matching block generator The figure which shows an example of the processing of the 2nd generation part The figure which shows an example of the similarity judgment threshold generation function The figure which shows an example of standard shape information and correction shape information Flow chart showing the processing of the parallax generator A flowchart showing the details of step S306 in FIG. Flowchart showing the processing of the matching block generator The figure which shows the detail of the function which the arithmetic unit has in 1st Embodiment. Figure showing another example of standard shape information and corrected shape information Functional configuration diagram of the matching block generation unit in the second embodiment The figure which shows an example of the parallax invalidity determination part Functional configuration diagram of the matching block generation unit in the third embodiment Functional block diagram of arithmetic unit in 4th Embodiment A flowchart showing the processing of the parallax generation unit in the fourth embodiment. A flowchart showing the processing of the matching block generation unit in the fourth embodiment. A flowchart showing the processing of the matching block generation unit in the fourth embodiment. The figure which shows the detail of the function which the arithmetic unit has in 4th Embodiment

-First embodiment-
Hereinafter, the first embodiment of the arithmetic unit will be described with reference to FIGS. 1 to 10.

(Overview of functional configuration)
FIG. 1 is a functional configuration diagram of the arithmetic unit 1 according to the present invention. The arithmetic unit 1 is connected to the first image pickup unit 2 and the second image pickup unit 3. The arithmetic unit 1 has, as its functions, an input unit 10, an evaluation value generation unit 11, a matching block generation unit 12, a parallax generation unit 13, a recognition processing unit 4, a vehicle control unit 5, and a non-volatile storage. It has a storage unit 15 which is a device.

The input unit 10 is a communication interface corresponding to, for example, IEEE802.3. The input unit 10 acquires a photographed image obtained by being photographed by the first image pickup unit 2 and the second image pickup unit 3. Hereinafter, the captured image of the first imaging unit 2 is referred to as a reference image 100, and the captured image of the second imaging unit 3 is referred to as a reference image 101. The reference image 100 and the reference image 101 are convenient names, and both may be interchanged. The input unit 10 inputs the reference image 100 and the reference image 101 to the evaluation value generation unit 11, the matching block generation unit 12, and the parallax generation unit 13.

The evaluation value generation unit 11, the matching block generation unit 12, the parallax generation unit 13, the recognition processing unit 4, and the vehicle control unit 5 are, for example, a program in which a CPU, which is a central processing unit, is stored in a storage unit 15. Is realized by expanding it into a RAM, which is a readable / writable storage device, and executing the above. However, these are realized by FPGA (Field Programmable Gate Array), which is a rewritable logic circuit instead of the combination of CPU, storage unit 15, and RAM, and ASIC (Application Specific Integrated Circuit), which is an integrated circuit for specific applications. May be good. Further, instead of the combination of the CPU, the storage unit 15, and the RAM, these may be realized by a combination of different configurations, for example, a combination of the CPU, the storage unit 15, RAM and the FPGA. Note that the CPU and RAM are not shown in FIG.

In the evaluation value generation unit 11, the reference image 100 and the reference image 101 are input from the input unit 10, and the pre-correction reference area 921 and the pre-correction comparison area 923 are input from the matching block generation unit 12. The evaluation value generation unit 11 calculates the evaluation value 103 and outputs it to the matching block generation unit 12 as described later. The processing of the evaluation value generation unit 11 will be described in detail later.

In the matching block generation unit 12, the reference image 100 and the reference image 101 are input from the input unit 10, the search area information 106 is input from the parallax generation unit 13, and the evaluation value 108 is input from the evaluation value generation unit 11. The matching block generation unit 12 generates the pre-correction reference area 921 and the pre-correction comparison area 923 and outputs them to the evaluation value generation unit 11, and generates the correction shape information 912 and outputs it to the parallax generation unit 13. More specifically, the matching block generation unit 12 generates the pre-correction reference region 921 and the pre-correction comparison region 923 using the search region information 106. Further, the matching block generation unit 12 generates the correction shape information 912 using the reference image 101, the reference image 101, and the evaluation value 108.

In the parallax generation unit 13, the reference image 100 and the reference image 101 are input from the input unit 10, and the correction shape information 912 is input from the matching block generation unit 12. The parallax generation unit 13 generates the parallax information 105 and outputs it to the recognition processing unit 4.

The recognition processing unit 4 performs various recognition processes by inputting the parallax information 105. As an example of the recognition process in the recognition process unit 4, there is a three-dimensional object detection using parallax information. Further, as an example of the recognition target, there are position information, type information, motion information, and danger information of the subject. As an example of position information, there is a direction and a distance from the own vehicle. Examples of type information include pedestrians, adults, children, the elderly, animals, rockfalls, bicycles, peripheral vehicles, peripheral structures, and curbs. Examples of motion information include pedestrian and bicycle wobbling, jumping out, crossing, moving direction, moving speed, and moving locus. Examples of danger information include pedestrian jumping out, falling rocks, sudden stop, sudden deceleration, and abnormal operation of surrounding vehicles such as sudden steering. The recognition result generated by the recognition processing unit 4 is supplied to the vehicle control unit 5. The vehicle control unit 5 performs various vehicle controls based on the recognition result.

As an example of vehicle control performed by the vehicle control unit 5, brake control, steering wheel control, accelerator control, in-vehicle lamp control, horn generation, in-vehicle camera control, peripheral vehicles connected via a network, and remote center equipment There is information output about the observation object around the image pickup device. As a specific example, there is speed and brake control according to the parallax information 105 of an obstacle existing in front of the vehicle. In addition, instead of the parallax information, the distance information that can be generated based on the parallax information may be used.

Further, although not described in this embodiment, the vehicle control unit 5 may perform subject detection processing based on the image processing result using the reference image 100 or the reference image 101, or may be connected to the vehicle control unit 5. The displayed display device may be displayed with an image obtained through the first image pickup unit 2 or the second image pickup unit 3 or for the viewer to recognize, map information, congestion information, or the like. Information on the observation target detected based on the image processing result may be supplied to the information device that processes the traffic information of the above.

(Definition of terms)
FIG. 2 is a diagram illustrating definitions of terms in the present embodiment. The upper part of FIG. 2 shows that the third information is generated by combining the first information and the second information. The lower part of FIG. 2 is a diagram illustrating the first information and the third information.

The first information in FIG. 2 is the parallax calculation target pixel 901 and the comparison target pixel 902. Both the parallax calculation target pixel 901 and the comparison target pixel 902 are information for defining a region to be evaluated, that is, an evaluation target region. The parallax calculation target pixel 901 is information on the coordinates of the pixel for which the parallax is calculated in the reference image 100, more specifically, the pixel. In the present embodiment, the parallax calculation target pixel 901 has a width of 1 pixel in the X direction and a width of 1 pixel in the Y direction.

That is, the parallax calculation target pixel 901 in this embodiment is a certain pixel. The comparison target pixel 902 is a pixel set as a candidate for a pair with the parallax calculation target pixel 901 in the reference image 101, and more specifically, information on the coordinates of the pixel. Is. The number of pixels of the comparison target pixel 902 is the same as that of the parallax calculation target pixel 901. The comparison target pixel 902 exists inside the search range 930 set with reference to the parallax calculation target pixel 901.

The second information in FIG. 2 is standard shape information 911 and corrected shape information 912. The standard shape information 911 is information indicating a region relative to a reference pixel, for example, information such as "a region of ± 7 pixels in the X direction and ± 3 pixels in the Y direction centered on the reference pixel". Is. In the present embodiment, the standard shape information 911 is common to all the parallax calculation target pixels 901 and the comparison target pixels 902. The standard shape information 911 is preset and stored in the storage unit 15.

Each region represented by the standard shape information 911 and the corrected shape information 912 can be called a "matching block" because it is used for matching, that is, matching with the reference image 100 and the reference image 101. Further, the region represented by the standard shape information 911 is also referred to as a “matching block before correction”, and the region represented by the correction shape information 912 is also referred to as a “matching block after correction”.

The corrected shape information 912 is information that is calculated for each comparison target pixel 902 and indicates a region relative to the reference pixel. The corrected shape information 912 is, for example, information that "a region excluding a total of 4 pixels, one pixel each at the corner from a rectangular area of ± 7 pixels in the X direction and ± 3 pixels in the Y direction centered on the reference pixel". Is. The corrected shape information 912 is commonly used for the reference image 100 and the reference image 101. The area represented by the corrected shape information 912 is an area equal to or smaller than the area represented by the standard shape information 911. Details will be described later.

The third information in FIG. 2 is the pre-correction reference area 921, the post-correction reference area 922, the pre-correction comparison area 923, and the post-correction comparison area 924. All of these four pieces of information are information indicating the coordinates of a plurality of pixels in the reference image 100 or the reference image 101. The pre-correction reference area 921 is an area obtained by applying the standard shape information 911 with the parallax calculation target pixel 901 as a reference in the reference image 100. Since the standard shape information 911 is common to all pixels as described above, for example, if the parallax calculation target pixel 901 moves by one pixel in the X direction, the entire pre-correction reference region 921 moves by one pixel in the X direction and is compared. No matter how the pixel 902 changes, the pre-correction reference region 921 does not change.

The corrected reference area 922 is an area obtained by applying the corrected shape information 912 with the parallax calculation target pixel 901 as a reference in the reference image 100. Since the corrected shape information 912 exists as many as the number of combinations of the parallax calculation target pixel 901 and the comparison target pixel 902 as described above, the corrected reference region 922 is the same regardless of which of the parallax calculation target pixel 901 and the comparison target pixel 902 changes. May change. Since the standard shape information 911 and the corrected shape information 912 may be the same, in that case, the pre-correction reference area 921 and the post-correction reference area 922 are the same.

The pre-correction comparison area 923 is an area obtained by applying the standard shape information 911 with reference to the comparison target pixel 902 in the reference image 101. Since the standard shape information 911 is common to all pixels as described above, for example, if the comparison target pixel 902 moves by one pixel in the X direction, the entire pre-correction comparison area 923 moves by one pixel in the X direction.

The corrected comparison area 924 is an area obtained by applying the corrected shape information 912 with reference to the comparison target pixel 902 in the reference image 101. Since the corrected shape information 912 exists as many as the number of combinations of the parallax calculation target pixel 901 and the comparison target pixel 902 as described above, the corrected comparison area 924 is the same regardless of which of the parallax calculation target pixel 901 and the comparison target pixel 902 changes. May change.

(Details of function)
FIG. 3 is a detailed view of the functional configuration of the matching block generation unit 12. However, in FIG. 3, the operation of the evaluation value generation unit 11 will also be described. The matching block generation unit 12 includes a first generation unit 20, a second generation unit 21, and a threshold value generation unit 22.

The first generation unit 20 repeatedly generates the pre-correction reference area 921 and the pre-correction comparison area 923 based on the search area information 106 supplied from the parallax generation unit 13, and causes the second generation unit 21 and the evaluation value generation unit 11. Supply. The search area information 106 is information on the search range in the reference image 101 corresponding to the parallax calculation target pixel 901, the standard shape information 911, and the parallax calculation target pixel 901. That is, the first generation unit 20 generates the pre-correction reference region 921 from the parallax calculation target pixel 901 and the standard shape information 911, and from the search range and the standard shape information 911 in the reference image 101 corresponding to the parallax calculation target pixel 901. The pre-correction comparison area 923 is generated. For example, when the search range has a range of 100 pixels, 100 pre-correction comparison areas 923 are generated. Then, the first generation unit 20 generates, for example, 100 combinations of the pre-correction reference region 921 and the pre-correction comparison region 923 without changing the value of the pre-correction reference region 921, and outputs the combination.

The evaluation value generation unit 11 calculates the evaluation value 103 each time the combination of the pre-correction reference region 921 and the pre-correction comparison region 923 is received from the first generation unit 20, and outputs the evaluation value 103 to the threshold value generation unit 22 and the second generation unit 21. do. The evaluation value generation unit 11 acquires specific pixel information in the pre-correction reference region 921, for example, a luminance value, by combining the pre-correction reference region 921 and the reference image 100. Similarly, the evaluation value generation unit 11 acquires specific pixel information in the pre-correction comparison area 923, for example, a luminance value, by combining the pre-correction comparison area 923 and the reference image 101. Then, the evaluation value generation unit 11 calculates the evaluation value 103 for each pixel by using the information of the pixel of the pre-correction reference area 921 and the information of the pixel of the pre-correction comparison area 923.

As an example of the evaluation value 103, there is a brightness fluctuation amount of each pixel in the matching block of the reference image 100 and the pixel in the matching block of the corresponding reference image 101. As an example of a general brightness fluctuation amount, SAD (Sum of Absolute Difference), ZSAD (Zero means Sum of Absolute Difference), SSD (Sum of Squared Difference), NCC (Normalized Cross Correlation), ZNCC (Zero means Correlation), ZNCC (Zero means Correlation) )and so on.

Since the pre-correction reference area 921 and the pre-correction comparison area 923 are generated by the standard shape information 911 common to different reference points, the number of pixels of the pre-correction reference area 921 and the pre-correction comparison area 923 are the same. When the standard shape information 911 is an area of 15 pixels in the X direction and 7 pixels in the Y direction, the pre-correction reference area 921 and the pre-correction comparison area 923 both have 105 pixels, and 105 evaluation values 103 are calculated. To.

As another example of the evaluation value 103, there is a luminance fluctuation amount or a luminance fluctuation direction or a luminance fluctuation pattern with each pixel in the matching block of the reference image 100 and the reference image 101 and its peripheral pixels. As an example of the luminance fluctuation pattern, there is an increase / decrease pattern from the left with respect to three horizontal pixels including the left and right adjacent pixels of the target pixel. For example, when the left adjacent pixel value is 10, the target pixel value is 25, and the right adjacent pixel value is 80, the increase / decrease pattern is “increase → increase”.

In the second generation unit 21, the pre-correction reference region 921 and the pre-correction comparison region 923 are supplied from the first generation unit 20, the similarity determination threshold value 201 is supplied from the threshold value generation unit 22, and the evaluation value generation unit 11 supplies the evaluation value. 103 is supplied. The second generation unit 21 generates the correction shape information 912 by removing the pixels whose evaluation value 103 exceeds the similarity determination threshold value 201 (hereinafter referred to as invalid pixels) in the standard shape information 911, and causes the parallax generation unit 13 to generate the correction shape information 912. Supply. In the following, pixels other than the invalid pixels in the standard shape information 911 will be referred to as effective pixels. The corrected shape information 912 may be expressed as an effective pixel flag for each pixel. As an example of the effective pixel flag, the flag information is assigned to each pixel of the correction shape information 912, and is a flag information that is an effective pixel when the value is "1" and an invalid pixel when the value is "0".

FIG. 4 is a diagram showing an example of processing of the second generation unit 21. The upper part of FIG. 4 shows the luminance value in each pixel of the pre-correction reference area 921, and the middle part of FIG. 4 shows the luminance value in each pixel of the pre-correction comparison area 923. The shape of the pre-correction reference region 921 and the pre-correction comparison region 923 is standard shape information 911. In the example shown in FIG. 4, the evaluation value 103 is the absolute value of the difference in luminance, and the lower part of FIG. 4 shows the absolute value of the difference in the luminance value between the upper part of FIG. 4 and the middle part of FIG. The value of each pixel in the lower part of FIG. 4 is provided by the evaluation value generation unit 11 as the evaluation value 103.

When the similarity determination threshold value 201 calculated by the threshold value generation unit 22 is “50”, the second generation unit 21 determines that each of the four corners is an invalid pixel as shown in the lower part of FIG. 4, and is a valid pixel. A region having a cross shape at a certain center is calculated as the correction shape information 912.

The threshold value generation unit 22 calculates the similarity determination threshold value 201. The similarity determination threshold value 201 may be a fixed and specified predetermined threshold value. This threshold value may be a fixed threshold value, or although not described in this embodiment, a threshold value designated from the outside via the input unit may be used. As another example of the similarity determination threshold value 201, a function based on the evaluation value 103 of the parallax generation target pixel in the reference image 100 and the pixel in the reference image 101 corresponding to the pixel, for example, a similarity determination threshold value generation function is used. There is a method to use.

FIG. 5 is a diagram showing an example of the similarity determination threshold generation function 900. The horizontal axis of FIG. 5 is an evaluation value 103 (SV) of the parallax calculation target pixel 901 in the reference image 100 and the comparison target pixel 902 in the reference image 101. The vertical axis of FIG. 5 is the similarity determination threshold TH0. LMmax, LMmin, and OFFSET may be fixed values, or may be specified from the outside via the input unit, although not described in this embodiment. The similarity determination threshold generation function 900 is expressed by, for example, Equation 1 below.

Similarity determination threshold TH0 =
LMmin (SV≤LMmin)
K0 x SV + OFFSET (LMmin <SV≤LMmax)
LMmax (LMmax <SV) ... (Equation 1)

That is, Equation 1 shows that there are three cases: when SV is LMmin or less, when SV is larger than LMmin and LMmax or less, and when SV is larger than LMmax.

Another example of the similarity determination threshold value 201 is a function based on distance information. For example, various parameters (LMmin, LMmax, OFFSET, etc.) used in the function based on the fixed threshold value, the threshold value specified from the outside, and the evaluation value 103 may be changed based on the parallax information and the distance information obtained in advance.

FIG. 6 is a diagram showing an example of standard shape information 911 and corrected shape information 912. In this example, the size of the area indicated by the standard shape information 911 is 7 pixels in the vertical direction and 15 pixels in the horizontal direction, so that the reference area 921 before correction also has the same size. The parallax calculation target pixel 901 is arranged in the center of the pre-correction reference region 921. Further, the pixels shown by hatching in the figure are examples of invalid pixels. In the example shown in FIG. 6, the region indicated by the correction shape information 912 generated by the second generation unit 21 is a region in which invalid pixels are removed from the pre-correction reference region 921 and is a region based on the parallax calculation target pixel 901. be.

FIG. 7 is a flowchart showing the processing of the parallax generation unit 13. The parallax generation unit 13 executes the process shown in FIG. 7 each time the reference image 100 and the reference image 101 are received from the input unit 10. In step S301, the parallax generation unit 13 specifies the parallax calculation target pixel 901 from the reference image 100. In this step, among the pixels included in the reference image 100, one of the pixels for which the parallax has not been calculated yet is specified as the parallax calculation target pixel 901.

In the following step S302, the parallax generation unit 13 reads the standard shape information 911 from the storage unit 15. In the following step S303, the parallax generation unit 13 specifies the search range in the reference image 101. For example, in order to simplify the explanation, it is assumed that the search range in the reference image is fixed at ± 100 pixels in the X direction and ± 5 pixels in the Y direction centering on the parallax calculation target pixel 901 of the reference image. In this case, when the coordinates of the parallax calculation target pixel 901 are (300,300), the search range in the reference image 101 is specified as (200, 295) to (400, 305). That is, in this example, the number of pixels included in the search range is "1111", which is the product of 100x2 + 1 and 5x2 + 1. The search range in the reference image 101 may be predetermined, or different values may be set for each pixel of the reference image 100 by a method not described in the present embodiment.

In the following step S304, the parallax generation unit 13 transmits the search area information 106 to the matching block generation unit 12. In the above example, the standard shape information and the information of the search range in the reference image are transmitted to the matching block generation unit 12 to the effect that the coordinates of the parallax calculation target pixel 901 are (300,300). In the following step S305, the parallax generation unit 13 receives the correction shape information 912 for the entire area of the search area. For example, in the above example, the parallax generation unit 13 receives "1111" correction shape information 912s.

In the following step S306, the parallax generation unit 13 identifies the corrected comparison area 924 most similar to the corrected reference area 922 from the search area of the reference image 101. However, in this step, the corrected shape information 912 acquired in step S305 is used. The details of this step will be described later with reference to FIG.

In the following step S307, the parallax generation unit 13 calculates the distance of the parallax calculation target pixel from the arithmetic unit 1. For this calculation, for example, the value of the difference between the X coordinate value of the parallax calculation target pixel in the reference image 100 and the X coordinate value of the center of the block specified in step S306 in the reference image 101, and a look-up table created in advance. It is calculated using and. In the following step S308, the parallax generation unit 13 determines whether or not the distance has been calculated for all the pixels in the reference image. When the parallax generation unit 13 determines that the distance has been calculated for all the pixels in the reference image 100, the process shown in FIG. 7 is terminated, and when it is determined that there are pixels of the reference image 100 for which the distance has not been calculated, a step is taken. Return to S301.

FIG. 8 is a flowchart showing the details of step S306 in FIG. 7. At the time of starting the process of FIG. 8, the search area information is specified by the process up to step S303, and the corrected shape information 912 is obtained for the entire search area in step S305.

In step S321, which is the first process in FIG. 8, the parallax generation unit 13 specifies the comparison target pixel 902 in the reference image 101. The comparison target pixel 902 is any coordinate that is included in the range of coordinates specified by the information of the search range in the reference image included in the search area information and that has not been processed in steps S322 to S325.

In the following step S322, the parallax generation unit 13 reads the correction shape information 912 corresponding to the comparison target pixel 902. The target of reading in this step is included in the information obtained in step S305 of FIG. In the following step S323, the parallax generation unit 13 applies the corrected shape information 912 with the parallax calculation target pixel 901 as a reference in the reference image 100, and specifies the corrected reference region 922.

In the following step S324, the parallax generation unit 13 applies the corrected shape information 912 with reference to the comparison target pixel 902 in the reference image 101, and specifies the corrected comparison region 924. In the following step S315, the parallax generation unit 13 calculates the similarity between the luminance information included in the corrected reference region 922 specified in step S323 and the luminance information included in the corrected comparison region 924 specified in step S324. ..

In the following step S326, the parallax generation unit 13 determines whether or not the similarity is calculated for the entire region included in the range of coordinates specified by the information of the search range in the reference image. The parallax generation unit 13 proceeds to step S327 when it is determined that the similarity has been calculated for the entire region, and returns to step S321 when it is determined that there are pixels for which the similarity has not been calculated. In step S327, the parallax generation unit 13 identifies the corrected comparison area 924 most similar to the corrected reference area 922, for example, the corrected comparison area 924 having the smallest SAD (Sum of Abusolute Difference), and performs the process shown in FIG. finish.

FIG. 9 is a flowchart showing the processing of the matching block generation unit 12. The matching block generation unit 12 executes the process shown in FIG. 9 each time the search area information 106 is received from the parallax generation unit 13. However, in FIG. 9, it is described as the first step in order to clearly indicate the reception of the search area information 106.

The matching block generation unit 12 first receives the search area information 106 from the parallax generation unit 13 in step S331. In the following step S332, the matching block generation unit 12 generates one pre-correction reference area 921 and a plurality of pre-correction comparison areas 923 using the received search area information 106, and proceeds to step S333. In step S333, the matching block generation unit 12 transmits a set of pre-correction reference area 921 and pre-correction comparison area 923 to the evaluation value generation unit 11. However, the combination of the pre-correction reference area 921 and the pre-correction comparison area 923 transmitted in this step is a combination of the pre-correction reference area 921 and the pre-correction comparison area 923 that have not been transmitted yet.

In the following step S334, the matching block generation unit 12 receives the evaluation value 103 from the evaluation value generation unit 11. The evaluation value 103 received in this step is the same number as the number of pixels of the standard shape information 911, and corresponds to each pixel of the standard shape information 911, in other words, each pixel of the pre-correction reference region 921. Although it was the coordinate information that was transmitted to the evaluation value generation unit 11 in step S333, the evaluation value generation unit 11 also referred to the reference image 100 and the reference image 101 to acquire the luminance information and calculate the evaluation value 103. did. In the following step S335, the matching block generation unit 12 causes the threshold value generation unit 22 to generate the similarity determination threshold value 201.

In the following step S336, the second generation unit 21 compares each evaluation value 103 with the similarity determination threshold value 201. In the following step S337, the corrected shape information 912 is generated based on the comparison result in step S336 and transmitted to the parallax generation unit 13. In the following step S338, the matching block generation unit 12 determines whether or not all the pre-correction comparison regions 923 generated in step S332 have been processed. The matching block generation unit 12 ends the process shown in FIG. 9 when it is determined that all the uncorrected comparison areas 923 have been processed, and returns to step S333 when it is determined that the unprocessed uncorrected comparison area 923 exists. ..

(Details of function)
FIG. 10 is a diagram showing details of the functions of the arithmetic unit 1. FIG. 10 shows the details of the functional configuration diagram shown in FIG. 1, but the recognition processing unit 4 and the vehicle control unit 5 are not shown for convenience of illustration. Newly described in FIG. 10 are a target point setting unit 801 and a comparison point setting unit 802, a pre-correction reference area specifying unit 803, a pre-correction comparison area specifying unit 804, a correction unit 805, and a post-correction reference area specifying unit 806. The corrected comparison area specifying unit 807 and the parallax calculation unit 808.

The target point setting unit 801 sets the parallax calculation target pixel 901, which is the target pixel for which the parallax is calculated in the reference image 100. The target point setting unit 801 is included in the parallax generation unit 13. The comparison point setting unit 802 identifies the comparison target pixel 902 which is a pixel candidate in the reference image 101 corresponding to the parallax calculation target pixel 901. The comparison point setting unit 802 is included in the first generation unit 20 of the matching block generation unit 12.

The pre-correction reference area specifying unit 803 specifies the pre-correction reference area 921 based on the parallax calculation target pixel 901 based on the standard shape information 911. The pre-correction reference region specifying unit 803 is included in the first generation unit 20 of the matching block generation unit 12. The pre-correction comparison area specifying unit 804 specifies the pre-correction comparison area 923 with reference to the comparison target pixel 902 based on the standard shape information 911. The pre-correction comparison area specifying unit 804 is included in the first generation unit 20 of the matching block generation unit 12.

The correction unit 805 maintains or narrows the evaluation target area by correcting the standard shape information 911 to the correction shape information 912 based on the comparison between the evaluation value 103 and the similarity determination threshold value 201. The correction unit 805 is included in the second generation unit 21 of the matching block generation unit 12.

The corrected reference area specifying unit 806 specifies the corrected reference area 922 based on the parallax calculation target pixel 901 based on the corrected shape information 912. The corrected reference region specifying unit 806 is included in the parallax generation unit 13. The corrected comparison area specifying unit 807 specifies the corrected comparison area 924 based on the comparison target pixel 902 based on the corrected shape information 912. The corrected comparison area specifying unit 807 is included in the parallax generation unit 13.

The parallax calculation unit 808 specifies the information of the reference image 100 in the corrected reference region 922 specified with reference to the parallax calculation target pixel 901 based on the corrected shape information 912, and the comparison target pixel 902 with reference to the corrected shape information 912. The parallax of the parallax calculation target pixel 901 is calculated based on the information of the reference image 101 in the corrected comparison area 924.

According to the first embodiment described above, the following effects can be obtained.
(1) The arithmetic unit 1 is an object for calculating the difference between the reference image 100 acquired by the first imaging unit 2, the input unit 10 into which the reference image 101 acquired by the second imaging unit 3 is input, and the reference image 100. The target point setting unit 801 for setting the disparity calculation target pixel 901, which is a pixel of the above, and the comparison point setting unit 802 for specifying the comparison target pixel 902, which is a pixel candidate in the reference image 101 corresponding to the disparity calculation target pixel 901. A storage unit 15 that stores standard shape information 911 that defines an evaluation target area based on a predetermined pixel, and a correction that specifies a pre-correction reference area 921 that is based on the parallax calculation target pixel 901 based on the standard shape information 911. Each of the pre-correction area specifying unit 804 that specifies the pre-correction comparison area 923 based on the comparison target pixel 902 and the pre-correction reference area 921 based on the pre-reference area specifying unit 803 and the standard shape information 911. Based on the comparison between the evaluation value generation unit 11 that calculates the evaluation value 103, which is an index similar to the corresponding pixel in the pre-correction comparison area 923, and the evaluation value 103 and the similarity determination threshold 201, the standard is obtained. The correction unit 805 that maintains or narrows the evaluation target area by correcting the shape information 911 to the correction shape information 912, and the reference in the corrected reference area 922 that is specified based on the correction shape information 912 with reference to the parallax calculation target pixel 901. The parallax calculation unit 808 that calculates the misalignment of the parallax calculation target pixel 901 based on the information of the image 100 and the information of the reference image 101 in the corrected comparison region 924 specified with reference to the comparison target pixel 902 based on the correction shape information 912. And prepare. In this way, the arithmetic unit 1 corrects the evaluation target area according to the degree of similarity between the pixels of the reference image 100 and the reference image 101 before correction, so that the adverse effect of a small number of pixels having an extreme difference can be eliminated. The calculation accuracy of parallax can be improved.

(2) As shown in FIG. 5, the similarity determination threshold value 201 is based on the degree of similarity with the information of the reference image 100 in the pre-correction reference region 921 and the information of the reference image 101 in the pre-correction comparison region 923. It is determined. Therefore, a more appropriate threshold value can be set as compared with the case where the similarity determination threshold value 201 is set to a fixed value.

(3) The evaluation value generation unit 11 calculates an evaluation value for each pixel included in the pre-correction reference region 921 based on the difference in brightness from the corresponding pixel in the pre-correction comparison region 923. Therefore, the dissociation between the reference image 100 and the reference image 101 can be evaluated.

(Modification 1)
FIG. 11 is a diagram showing another example of the standard shape information 911 and the corrected shape information 912. In the example shown in FIG. 11, the size of the region indicated by the standard shape information 911 is 6 pixels in the vertical direction and 12 pixels in the horizontal direction, and the reference region 921 before correction also has the same size. Further, in the example shown in FIG. 11, the parallax calculation target pixel 901 has a size of 2 vertical pixels and 2 horizontal pixels. In the example shown in FIG. 11, since the invalid pixel does not exist, the corrected shape information 912 is the same as the standard shape information 911.

Further, the parallax calculation target pixel 901 may be one pixel as shown in the first embodiment, or may be a pixel block composed of a plurality of pixels as shown in FIG. When the parallax calculation target pixel 901 is composed of a plurality of pixels, any of the average value, the minimum value, the maximum value, and the intermediate value of the similarity of each pixel included in the parallax calculation target pixel 901 is adopted. You may.

(Modification 2)
The arithmetic unit 1 does not have to include the recognition processing unit 4 and the vehicle control unit 5. Further, the function of the arithmetic unit 1 may be realized by a plurality of hardware devices, for example, two or more electronic control devices.

(Modification 3)
The parallax generation unit 13 only needs to be able to specify the difference between the pixels in the baseline length direction and the corresponding pixels of the reference image 101 in the reference image 100, and does not have to calculate a specific distance. That is, the process of step S307 in FIG. 7 may be deleted.

-Second embodiment-
A second embodiment of the arithmetic unit will be described with reference to FIGS. 12 to 13. In the following description, the same components as those in the first embodiment are designated by the same reference numerals, and the differences will be mainly described. The points not particularly described are the same as those in the first embodiment. This embodiment is different from the first embodiment in that the matching block generation unit 12 mainly includes a parallax invalidity determination unit.

FIG. 12 is a functional configuration diagram of the matching block generation unit 12A in the second embodiment. The matching block generation unit 12A further includes a parallax invalidity determination unit 30 in addition to the configuration of the first embodiment.

The parallax invalidity determination unit 30 determines whether or not the total number of effective parallax existing in the correction shape information 912 is equal to or less than a predetermined number based on the correction shape information 912 supplied from the second generation unit 21. When the total number of effective parallax is determined to be less than or equal to a predetermined number, the parallax invalidity determination unit 30 outputs the parallax invalid information 300, which is information indicating that the parallax corresponding to the comparison target pixel 902 is invalid. When the parallax invalidity determination unit 30 determines that the total number of effective parallax is larger than a predetermined number, the parallax invalidity determination unit 30 outputs the corrected shape information 912 as it is.

The parallax generation unit 13 determines that the combination of the parallax calculation target pixel 901 and the comparison target pixel 902 corresponding to the parallax invalid information 300 is invalid, and excludes them from the candidates for calculating the parallax.

FIG. 13 is a diagram showing an example of the parallax invalidity determination unit 30. In this example, the size of the area indicated by the standard shape information 911 is 7 pixels in the vertical direction and 15 pixels in the horizontal direction, so that the reference area 921 before correction also has the same size. The parallax calculation target pixel 901 is arranged in the center of the pre-correction reference region 921. Further, the pixels indicated by hatching in the figure indicate invalid pixels. In the example shown in FIG. 13, the number of effective pixels is small, and for example, the effective pixels do not reach the threshold value of 20% or more of the standard shape information 911, so that the parallax invalidity determination unit 30 determines that the parallax is invalid.

According to the second embodiment described above, the following effects can be obtained.
(4) The arithmetic unit 1A includes a parallax invalidity determination unit 30 that changes the corrected shape information 912 to the parallax invalid information 300 when the number of pixels in the evaluation target area defined by the corrected shape information 912 is smaller than a predetermined threshold value. .. The parallax calculation unit 808 sets the parallax corresponding to the combination of the parallax calculation target pixel 901 and the comparison target pixel 902 corresponding to the parallax invalid information 300 as the invalid parallax. Therefore, it is possible to prevent the output of poor quality parallax information that occurs when the total number of effective pixels is small.

-Third embodiment-
A third embodiment of the arithmetic unit will be described with reference to FIG. In the following description, the same components as those in the first embodiment are designated by the same reference numerals, and the differences will be mainly described. The points not particularly described are the same as those in the first embodiment. This embodiment is different from the first embodiment in that the matching block generation unit 12 mainly includes a region determination unit.

FIG. 14 is a functional configuration diagram of the matching block generation unit 12B in the third embodiment. The matching block generation unit 12B further includes an area determination unit 40 in addition to the configuration of the first embodiment.

The area determination unit 40 detects an object by image processing for at least the pre-correction reference area 921 of the reference image 100. Then, the area determination unit 40 outputs the correction shape information 912 output by the second generation unit 21 as it is in the area where the same object as the area of the parallax calculation target pixel 901 exists, and the object different from the area of the parallax calculation target pixel 901. The area in which is present outputs foreign matter parallax invalid information 400, which is information indicating that parallax is invalid. For example, when the area determination unit 40 detects a road and a tire in the pre-correction reference area 921 and a road exists in the parallax calculation target pixel 901, the area determination unit 40 displays the foreign body parallax invalid information 400 in the tire area. Output. The object detection targeting the reference image 100 may be executed by another configuration included in the arithmetic unit 1, or may be executed by an apparatus different from the arithmetic unit 1.

The object detection by the area determination unit 40 may be easily realized by using, for example, a change in brightness. That is, when there is a change in brightness of a predetermined threshold value or more from the parallax calculation target pixel 901 toward the peripheral portion of the reference image 100, the pixel region from the pixel having the change of the predetermined value or more to the boundary of the corrected reference region 922 is a foreign object. The parallax invalid information 400 may be used.

The parallax generation unit 13 determines that the combination of the parallax calculation target pixel 901 and the comparison target pixel 902 corresponding to the foreign body parallax invalid information 400 is invalid, and excludes them from the candidates for calculating the parallax.

According to the third embodiment described above, the following effects can be obtained.
(5) The arithmetic unit 1B includes an area determination unit 40 that identifies the boundary of an object based on the information of the reference image 100 in the pre-correction reference area 921. The parallax calculation unit 808 calculates the parallax using the information of the pixels belonging to the same region as the parallax calculation target pixel 901 due to the boundary in the reference image 100. Therefore, the parallax can be calculated without explicitly using the information of the pixels that are different from the parallax calculation target pixel 901 and are expected to have different parallax.

(Modification 1 of the third embodiment)
The area determination unit 40 may recognize the object by using the information output by the sensor (not shown). For example, the area determination unit 40 may perform object detection using an output of a monocular camera, LiDAR (Light Detection and Ranging), an ultrasonic sensor, a millimeter wave radar, or the like.

-Fourth Embodiment-
A fourth embodiment of the arithmetic unit will be described with reference to FIGS. 15 to 19. In the following description, the same components as those in the first embodiment are designated by the same reference numerals, and the differences will be mainly described. The points not particularly described are the same as those in the first embodiment. The present embodiment is different from the first embodiment mainly in that the calculation is made more efficient. In the previous embodiments, the degree of similarity is not designated by a reference numeral, but in the present embodiment, the degree of similarity is designated by a reference numeral 107.

FIG. 15 is a functional configuration diagram of the arithmetic unit 1C according to the fourth embodiment. The operations of the matching block generation unit 12C and the parallax generation unit 13C are different from those of the first embodiment. Further, in the present embodiment, the matching block generation unit 12C transmits the similarity degree 107 to the parallax generation unit 13C instead of the correction shape information 912.

FIG. 16 is a flowchart showing the processing of the parallax generation unit 13C in the fourth embodiment. FIG. 16 replaces step S305 and step S306 in FIG. 7 with steps S305A and S306A as compared to FIG. 7 in the first embodiment. The points not particularly described are the same as those in FIG. 7.

In step S305A, the parallax generation unit 13C receives the similarity 107 for the entire search range from the matching block generation unit 12C. In the following step S306A, the parallax generation unit 13C uses the similarity 107 for the entire search range received in step S305A to make a corrected comparison area 924 most similar to the corrected reference area 922, for example, a corrected comparison with the minimum SAD. Identify region 924. Since the processes after step S307 are the same as those in FIG. 7, the description thereof will be omitted.

17 and 18 are flowcharts showing the processing of the matching block generation unit 12C in the fourth embodiment. 17 is the same as FIG. 9 in the first embodiment, and the processes up to step S335 are the same, and the processes after that are different. The points not particularly described are the same as those in FIG.

In step S351 executed after step S335, the matching block generation unit 12C initializes the variable sum to zero and proceeds to step S352. In step S352, the matching block generation unit 12C selects one of the evaluation values received in step S334. However, the evaluation values selected here are those that have not been selected so far. In the following step S353, the matching block generation unit 12C compares the evaluation value selected in step S352 with the threshold value generated in step S335. Then, the matching block generation unit 12C proceeds to step S354 in FIG. 18 via the circled A.

In step S354, the matching block generation unit 12C determines the result of comparison in step S353. The matching block generation unit 12C proceeds to step S355 when it is determined that the evaluation value is less than the threshold value, and proceeds to step S356 when it is determined that the evaluation value is equal to or more than the threshold value. In step S355, the matching block generation unit 12C adds the value of the evaluation value to the variable sum and proceeds to step S356. In step S356, the matching block generation unit 12C determines whether or not all the evaluation values received in step S334 have been evaluated. When the matching block generation unit 12C determines that all the evaluation values have been evaluated, the process proceeds to step S357, and when it is determined that there are evaluation values that have not been evaluated, the matching block generation unit 12C goes through the circled B to step 17 in FIG. Return to S352.

In step S357, the matching block generation unit 12C transmits the value of the variable sum as the similarity 107 to the parallax generation unit 13C. In the following step S338, the matching block generation unit 12C determines whether or not all the pre-correction comparison regions 923 have been processed. When the matching block generation unit 12C determines that all the uncorrected comparison areas 923 have been processed, the processing shown in FIG. 18 is terminated, and when it is determined that the unprocessed uncorrected comparison area 923 exists, the circled C Return to step S333 in FIG. 17 via. The above is the processing of the matching block generation unit 12C.

FIG. 19 is a diagram showing details of the function in the fourth embodiment, and corresponds to FIG. 10 in the first embodiment. In FIG. 19, the corrected reference area specifying unit 806 and the corrected comparison area specifying unit 807 are moved from the parallax generation unit 13 to the second generation unit 21 of the matching block generation unit 12C as compared with FIG. Since the second generation unit 21 compares the evaluation value 103 with the similarity determination threshold value 201 and adds it to the variable sum (S353 to S355), the generation of the corrected shape information 912, the corrected reference area 922, and the corrected comparison area 924. Can be considered to be collectively performing the generation of and the calculation of the similarity 107.

According to the fourth embodiment described above, the following effects can be obtained.
(6) The parallax generation unit 13C including the parallax calculation unit 808 calculates the parallax using the evaluation value 103 calculated by the evaluation value generation unit 11. Specifically, the parallax generation unit 13C acquires and uses the similarity 107 obtained by aggregating the evaluation values 103 calculated by the evaluation value generation unit 11 by the second generation unit 21 of the matching block generation unit 12C. Therefore, the parallax generation unit 13C does not need to specify the corrected comparison area 924 using the comparison target pixel 902 again, and does not need to refer to the pixel values of the reference image 100 and the reference image 101. Therefore, the entire arithmetic unit 1C does not need to be referred to. The arithmetic processing can be reduced. That is, as compared with the prior art, it is possible to perform high-speed processing using the same arithmetic resource, or to reduce the arithmetic resource and keep the processing speed the same.

(Modified example of the fourth embodiment)
In the fourth embodiment described above, the parallax generation unit 13C used the similarity 107 received from the matching block generation unit 12C as it is for the evaluation. However, the parallax generation unit 13C may process the similarity 107 received from the matching block generation unit 12C and use it for evaluation. For example, when the similarity 107 is a value of SAD, the value of ZSAD may be calculated using this and this value may be used for evaluation.

The present invention is not limited to the above-described embodiment, but includes various modifications. For example, the above-described embodiment has been described in detail in order to explain the present invention in an easy-to-understand manner, and is not necessarily limited to the one including all the described configurations. Further, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Further, it is possible to add / delete / replace a part of the configuration of each embodiment with another configuration.

Further, each of the above configurations may be partially or wholly configured by hardware or may be configured to be realized by executing a program on a processor. In addition, the control lines and information lines indicate those that are considered necessary for explanation, and do not necessarily indicate all the control lines and information lines in the product. In practice, it can be considered that almost all configurations are interconnected.

In each of the above-described embodiments and modifications, the configuration of the functional block is only an example. Several functional configurations shown as separate functional blocks may be integrally configured, or the configuration represented by one functional block diagram may be divided into two or more functions. Further, a configuration in which a part of the functions of each functional block is provided in another functional block may be provided.

The storage unit 15 in which the program of the arithmetic unit 1 is stored may be a rewritable storage device such as a flash memory. Further, the program may be read from another device via a medium that can be used by the input unit 10 of the arithmetic unit 1. Here, the medium refers to, for example, a storage medium that can be attached to and detached from an input / output interface, or a communication medium, that is, a network such as wired, wireless, or optical, or a carrier wave or digital signal that propagates in the network. In addition, some or all of the functions realized by the program may be realized by the hardware circuit or FPGA.

The above-mentioned embodiments and modifications may be combined. Although various embodiments and modifications have been described above, the present invention is not limited to these contents. Other aspects considered within the scope of the technical idea of the present invention are also included within the scope of the present invention.

1, 1A, 1B, 1C ... Calculation device 10 ... Input unit 11 ... Evaluation value generation unit 12, 12A, 12B, 12C ... Matching block generation unit 13, 13C ... Parallax generation unit 13C ... Parallax generation unit 15 ... Storage unit 20 ... 1st generation unit 21 ... 2nd generation unit 22 ... threshold generation unit 30 ... parallax invalidity determination unit 40 ... area determination unit 100 ... reference image 101 ... reference image 103 ... evaluation value 106 ... search area information 107 ... similarity 201 ... similarity Degree determination threshold 300 ... Parallax invalid information 400 ... Foreign object parallax invalid information 801 ... Target point setting unit 802 ... Comparison point setting unit 803 ... Pre-correction reference area specifying unit 804 ... Pre-correction comparison area specifying unit 805 ... Correction unit 806 ... After correction Reference area specifying unit 807 ... After correction Comparison area specifying unit 808 ... Parallax calculation unit 900 ... Parallax determination threshold generation function 901 ... Parallax calculation target pixel 902 ... Comparison target pixel 911 ... Standard shape information 912 ... Corrected shape information 921 ... Before correction Reference area 922 ... Reference area after correction 923 ... Comparison area before correction 924 ... Comparison area after correction 930 ... Search range

Claims (7)

1. An input unit into which a reference image acquired by the first imaging unit and a reference image acquired by the second imaging unit are input, and
   A target point setting unit for setting a parallax calculation target pixel, which is a target pixel for which parallax is calculated in the reference image, and a target point setting unit.
   A comparison point setting unit that sets a comparison target pixel that is a pixel candidate in the reference image corresponding to the parallax calculation target pixel, and a comparison point setting unit.
   A storage unit that stores standard shape information that defines an evaluation target area based on a predetermined pixel, and a storage unit.
   A pre-correction reference area specifying unit that specifies a pre-correction reference area based on the parallax calculation target pixel based on the standard shape information, and a pre-correction reference area specifying unit.
   A pre-correction comparison area specifying unit that specifies a pre-correction comparison area based on the comparison target pixel based on the standard shape information, and a pre-correction comparison area specifying unit.
   An evaluation value generation unit that calculates an evaluation value that is an index similar to the corresponding pixel in the pre-correction comparison area for each pixel included in the pre-correction reference area.
   A correction unit that maintains or narrows the evaluation target area by correcting the standard shape information to the correction shape information based on the comparison between the evaluation value and the threshold value.
   In the information of the reference image in the corrected reference region specified with the parallax calculation target pixel as a reference based on the corrected shape information, and in the corrected comparison region specified with the comparison target pixel as a reference based on the corrected shape information. A calculation device including a parallax calculation unit that calculates the parallax of the pixel for which the parallax is calculated based on the information of the reference image.
2. In the arithmetic unit according to claim 1,
   The threshold value is an arithmetic unit determined based on the degree of similarity between the information of the reference image in the pre-correction reference region and the information of the reference image in the pre-correction comparison region.
3. In the arithmetic unit according to claim 1,
   The evaluation value generation unit is an arithmetic unit that calculates the evaluation value based on the difference in brightness between each pixel included in the pre-correction reference region and the corresponding pixel in the pre-correction comparison region.
4. In the arithmetic unit according to claim 1,
   Further provided with a parallax invalidity determining unit that changes the corrected shape information to parallax invalid information when the number of pixels in the evaluation target area defined by the corrected shape information is smaller than a predetermined threshold value.
   The parallax calculation unit is an arithmetic unit that uses parallax corresponding to a combination of the parallax calculation target pixel corresponding to the parallax invalid information and the comparison target pixel as an invalid parallax.
5. In the arithmetic unit according to claim 1,
   Further, a region determination unit for specifying the boundary of the object based on the information of the reference image in the pre-correction reference region is provided.
   The parallax calculation unit is an arithmetic unit that calculates parallax using information of pixels belonging to the same region as the pixel to be calculated for parallax due to the boundary in the reference image.
6. In the arithmetic unit according to claim 1,
   The parallax calculation unit is an arithmetic unit that calculates the parallax using the evaluation value calculated by the evaluation value generation unit.
7. An input unit for inputting a reference image acquired by the first imaging unit and a reference image acquired by the second imaging unit, and a storage unit for storing standard shape information defining an evaluation target area based on a predetermined pixel. It is a parallax calculation method executed by an arithmetic unit including.
   Setting the parallax calculation target pixel, which is the target pixel for which the parallax is calculated in the reference image,
   Identifying a comparison target pixel that is a pixel candidate in the reference image corresponding to the parallax calculation target pixel, and
   Based on the standard shape information, specifying the pre-correction reference region based on the parallax calculation target pixel, and
   Based on the standard shape information, specifying the pre-correction comparison area based on the comparison target pixel, and
   For each pixel included in the pre-correction reference region, an evaluation value which is an index similar to the corresponding pixel in the pre-correction comparison region is calculated.
   By correcting the standard shape information to the corrected shape information based on the comparison between the evaluation value and the threshold value, the evaluation target area can be maintained or narrowed.
   In the information of the reference image in the corrected reference region specified with the parallax calculation target pixel as a reference based on the corrected shape information, and in the corrected comparison region specified with the comparison target pixel as a reference based on the corrected shape information. A parallax calculation method including calculating the parallax of the parallax calculation target pixel based on the information of the reference image.
   

Images