JP2013045276A - Image processing method and stereo camera system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing method capable of obtaining distance information in a shorter processing time.SOLUTION: The image processing method includes: a step for photographing a first image and a second image; a total value calculation step for allowing addition means 241 to calculate a first total value of the first image and a second total value of the second image; an approximate average value calculation step for allowing shift means 242 to calculate a first approximate average value by shifting the first total value to the right by a predetermined number of bits and a second approximate average value similarly; an integral multiple step for allowing integral multiple means 243 and 244 to multiply the value of each pixel data in small regions of the first image and the second image by an integer; and a correlation value calculation step for allowing correlation value calculation means 249 to obtain each of first data obtained by subtracting the first approximate average value from the value obtained by multiplying the value of the pixel data by an integer and second data obtained by subtracting the second approximate average value from the value obtained by multiplying the pixel data by an integer, and calculating the square-sum of the difference between the first data and the second data for each pixel data.

Description

  The present invention relates to an image processing method for detecting distance information with a stereo camera.

  A vehicle driver assistance system using an image captured by a stereo camera is known. The driving support system, for example, monitors the front of the vehicle with a stereo camera installed in front of the car, measures the distance from pedestrians, other vehicles, obstacles, etc. The user can be notified, or the vehicle brake can be operated to decelerate or stop.

  A stereo camera measures the distance and position of an object by taking advantage of the fact that the imaging position on the captured image changes according to the distance of the object when the same target object is captured from different viewpoint positions.

FIG. 1 is an example of a diagram illustrating the distance measuring principle of a stereo camera. From the base line length B (distance between cameras), the focal length f, and the parallax d (difference in the position of the imaging point of the object from the viewpoint), the distance Z to the object can be expressed by the following equation.
Z = B · f / d (1)
To calculate the parallax d, the correlation value of a small area of a pair of images obtained from the left and right cameras is calculated, and the shift amount between the images when the pixel position with the highest correlation is obtained is calculated as the parallax. Block matching is often used.

  As a correlation evaluation value, an absolute value sum SAD (sum of absolute difference) or a sum of squared differences (SSD) of differences between pixel values of two small regions is often used. Since SAD and SSD are easily affected by camera characteristics, a method of calculating a correlation value that can suppress the influence of the luminance difference between both images and noise has been proposed (for example, see Patent Document 1). Patent Document 1 discloses average value difference matching in which block matching is performed after the average value of each small region is subtracted from each pixel value of each small region of two images.

  However, the average value difference matching has a problem that the arithmetic unit must perform division processing. This is because, when the distance calculation of a stereo camera is used for in-vehicle applications, it is desirable that it can be processed with high accuracy and in real time, and dedicated hardware is often mounted. Although the processing by hardware is faster than the processing by software, the division processing of the division circuit takes longer processing time (a larger number of clocks are required), so it is ideal that no division circuit is used.

  However, in average value difference matching, a division circuit is required to divide the sum of pixel values in a small region by the number of pixels. For example, if the pixel average value of a 7 × 7 small area is to be obtained, the division circuit performs “sum of pixel values / 49” in the small area. Therefore, the average value difference matching requires a division circuit, and there is a problem that the processing time for obtaining the distance becomes long.

  In addition, the average value difference matching proposed in Patent Document 1 has a problem that a mismatch occurs due to the influence of high-frequency noise, and an accurate parallax value cannot be obtained. Therefore, a technique has also been proposed in which a determination value calculated from the average value of a small area is compared with a pixel value in the small area to determine whether it is valid, and correlation calculation is performed using only valid pixels (for example, patents). Reference 2). However, the mismatch reduction method of Patent Document 2 has a problem in that the calculation processing cost is high because matching processing is performed after valid determination is performed on all pixels in a target small region.

  In view of the above problems, an object of the present invention is to provide an image processing method capable of obtaining distance information in a shorter processing time.

  In the present invention, a pair of stereo cameras captures the first image and the second image, and the adding means includes a first total value of pixel data in a small area of the first image, and A total value calculating step for calculating a second total value of the pixel data in the small area of the second image; and a shift means shifts the first total value to the right by a predetermined number of bits to make a first approximation An approximate average value calculating step of calculating an average value, shifting the second total value to the right by the predetermined number of bits to calculate the second approximate average value, and an integer multiplying unit include: An integer multiple step of multiplying the value of each pixel data in the small region by an integer and multiplying the value of each pixel data in the small region of the second image by an integer, and a correlation value calculating means includes: For each pixel data in the small area, the first approximate plane is calculated from a value obtained by multiplying the value of the pixel data by an integer. First data obtained by subtracting a value and second data obtained by subtracting the second approximate average value from a value obtained by multiplying the pixel data by an integer for each pixel data in the small area of the second image. And a correlation value calculating step of calculating a sum of squares of differences or a sum of absolute values of differences between the first data and the second data for each pixel data.

  An image processing method capable of obtaining distance information in a shorter processing time can be provided.

It is an example of the figure explaining the ranging principle of a stereo camera. It is an example of the figure explaining the calculation method of a correlation value. It is an example of the figure which shows typically the mounting position of four stereo cameras. It is an example of the figure which shows the example of mounting to the vehicle of a stereo camera system. It is a figure which shows an example of the basic composition of an image processing IC. It is a flowchart which shows the flow of the parallax value calculation in a stereo calculation process part. It is an example of the figure explaining block matching. It is an example of the figure explaining equiangular straight line fitting. It is an example of the flowchart figure which shows the detailed procedure of the process of step S12. It is an example of the figure which shows the circuit structure of a stereo arithmetic processing part typically. It is a figure which shows the comparative example of the parallax value of the sub pixel calculated | required by the division process, and the parallax value of the sub pixel calculated | required by the method of the present Example. It is an example of the figure which shows the graph of the relationship between a translation amount and a sub pixel. FIG. 10 is an example of a flowchart illustrating a detailed procedure of processing in step S12 (second embodiment). It is an example of the figure which shows the circuit structure of a stereo arithmetic processing part typically (Example 2). It is an example of the flowchart figure which shows the detailed procedure of the process of step S12 (modified example of Example 2). It is an example of the figure which shows the circuit structure of a stereo arithmetic processing part typically (modified example of Example 2).

  Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings.

FIG. 2 is an example of a diagram for explaining a correlation value calculation method according to the present embodiment. One image of the stereo camera is called a reference image, and the other is called a comparative image. In block matching, a correlation value is calculated while shifting a small region (7 × 7 in the figure) including the target pixel of the comparison image pixel by pixel with respect to the reference image.
S1: The stereo camera system first calculates the sum of the pixel values of 48 pixels of each of the comparison image and the reference image. The reason why the number of pixels is 48 is that the pixel of interest is excluded.
S2: The stereo camera system shifts the sum of the pixel values of 48 pixels to the right by 4 bits. Since the right shift is equivalent to division, a division result equivalent to division by 16 is obtained. Dividing by 16 with respect to 48 pixels means that a value obtained by multiplying the average of the pixel values of 48 pixels by 3 is obtained. Hereinafter, the 4-bit right shift result is referred to as an approximate average value.
S3: Next, the stereo camera system triples the pixel value of each pixel of the comparison image and the reference image.
S4: Next, the stereo camera system calculates a ZSSD (zero-mean sum of squared difference) value as a correlation value. That is, in the comparative image, the approximate average value is subtracted from the tripled pixel value for each pixel, and the approximate average value is subtracted from the tripled pixel value in the reference image. Then, the ZSSD value is obtained by calculating the sum of squares of the difference between these (“3 × S _ij -4 bit shift result” and “3 × M _ij -4 bit shift result”) for 48 pixels.

  Therefore, the stereo camera system according to the present embodiment uses a shift operation for calculating the approximate average value and does not perform a division process, so that the operation can be performed at high speed. In addition, since the approximate average value is subtracted from the pixel value, the camera characteristics of the two cameras can be adjusted with high accuracy, and even if a slight aging occurs, the distance can be measured with high accuracy without requiring readjustment. It is possible.

[Example of stereo camera]
FIG. 3 is an example of a diagram schematically illustrating the mounting positions of the four stereo cameras 1 to 4. The stereo camera 1 is disposed forward, the stereo camera 2 is disposed on the right side, the stereo camera 3 is disposed on the left side, and the stereo camera 4 is disposed on the rear. The stereo camera 1 is disposed on the indoor mirror or the front bumper of the vehicle with the optical axis directed slightly downward horizontally in front of the vehicle. The stereo camera 2 is, for example, a right door mirror, a depression in a door knob installation portion on the right side of the vehicle, a door window frame, an A pillar, a B pillar, or a C pillar. It is arranged with the optical axis facing slightly ahead of the direction. The stereo camera 3 is, for example, a left door mirror, a depression in a door knob installation portion on the left side of the vehicle, a door window frame, an A pillar, a B pillar, or a C pillar. It is arranged with the optical axis facing slightly ahead of the direction. The stereo camera 4 is arranged around the rear license plate, the rear bumper, and the like.

  The stereo camera 1 is mainly used to measure the distance from a pedestrian in front of the vehicle, the distance from a preceding vehicle, other features (signs, traffic lights, utility poles, guardrails, etc.) and distances from obstacles. . Depending on the distance to the obstacle, the stereo camera system can alert the driver or apply braking.

  The stereo cameras 2 and 3 are used to detect the distance from a person or an object approaching from the surroundings when the passenger opens or closes the door. At a parking lot or the like, measure the distance to other adjacent vehicles, and measure the distance from a bicycle, motorcycle, or pedestrian approaching from the rear side during parking. The stereo camera system can warn the driver or prohibit the opening of the door according to the distance.

  The stereo camera 4 detects the distance from the obstacle behind and measures the distance from the obstacle when the occupant opens the rear door. When the obstacle is within a predetermined distance, the opening of the door can be prohibited. In addition, during back travel, the stereo camera can warn the driver or apply braking according to the distance from the obstacle.

  In addition, a stereo camera that detects an occupant already in the vehicle may be mounted. The installation position of the occupant detection camera is a position where the interior of the vehicle can be imaged from the front to the rear (for example, a room mirror, a sun visor, an A pillar, a ceiling portion, etc.) For example, C pillar, rear door frame, ceiling part, etc.). It is also effective to install a passenger at the position where the inside of the vehicle can be imaged from the side of the vehicle (for example, a door, an A-pillar, a B-pillar, and a C-pillar) and image the passenger from the side.

[Configuration example]
FIG. 4 shows an example of a diagram illustrating an example of mounting the stereo camera system 100 on a vehicle. Many microcomputers are mounted on a vehicle, and one device on which one or more microcomputers are mounted is often referred to as an ECU (Electronic Control Unit). As typical ECUs, an engine ECU that controls an engine, a brake ECU that controls a brake, a body ECU that controls a door and a seat, an information system ECU that controls a navigation system and an AV device, and the like are known. In this embodiment, the ECU that acquires distance information is referred to as an image processing ECU 12.

  The image processing ECU 12 is communicably connected to another ECU (not shown) via a standard in-vehicle LAN such as CAN (Controller Area Network) or FlexRay. All ECUs connected to the in-vehicle LAN can refer to data on the in-vehicle LAN, and each ECU including the image processing ECU 12 receives predetermined data and uses it for processing.

  The image processing ECU 12 is connected to the display device 11 and the stereo camera 13 (only one in the figure, but a plurality may be mounted). The image processing ECU 12 is equipped with one or more microcomputers, and has a CPU, RAM, ROM, CANC (CAN Controller), I / O, etc., as well as general microcomputers, and image processing for image processing. It has IC20. These are connected via a system bus, an external bus, and a bus controller.

  The stereo camera 13 is connected to the I / O, and the captured image data is appropriately processed by the image processing IC 20 using the RAM. The CANC communicates with other ECUs based on the CAN protocol. The CPU performs various controls such as executing a program stored in the ROM in the working memory and transmitting the image processing result to another ECU via the CANC. The display device 11 is a display such as a liquid crystal display or an organic EL display that displays an image captured by the stereo camera 13 or a HUD (Head Up Display). The display device 11 is used to emphasize and display, for example, a pedestrian or an obstacle detected by image processing, but is not necessarily required. The display device 11 is often used also as a navigation device that displays a road map.

  FIG. 5 is a diagram illustrating an example of a basic configuration of the image processing IC 20. The image processing IC 20 includes a stereo image input unit 21, a stereo image calibration unit 22, a stereo image storage unit 23, a stereo calculation processing unit 24, and a parallax data storage unit 25.

  The stereo camera 13 inputs image data captured by the two monocular cameras almost simultaneously to the stereo image calibration unit 22.

  The stereo image calibrating unit 22 corrects the optical shift in the monocular camera and the geometric shift between the monocular cameras with respect to the two image data input from the stereo camera 13. The stereo image storage unit 23 stores the two image data calibrated by the stereo image calibration unit 22 as an original image for stereo calculation. The stereo calculation processing unit 24 processes two pieces of image data to calculate parallax data. The parallax data storage unit 25 stores parallax data for calculating distance information.

  The stereo camera 13 includes a monocular camera 1L and a monocular camera 1R each including a lens optical system having a certain focal length f and an image sensor. These two monocular cameras 1L and 1R are CCD cameras with variable exposure times and gains, and they are synchronized with each other and take images at the same timing. The image sensor is not limited to a CCD, and may be an image sensor such as a CMOS. The stereo camera 13 is arranged such that the optical axes are parallel with a distance of a predetermined baseline length B between the two monocular cameras 1L and 1R.

  The stereo image calibration unit 22 corrects mechanical displacement between the lens optical system and the image sensor, distortion aberration caused by the lens optical system, and displacement of the mounting position of the monocular camera. Specifically, for example, the pixel position is replaced by a LUT (Look Up Table). The stereo image calibrating unit 22 is an electric circuit including, for example, an FPGA that realizes a predetermined image processing function. The correction amount in the above correction item is obtained in advance by a calibration method such as Non-Patent Document 1, but the correction item and the calibration method are not limited to the contents of Non-Patent Document 1.

  The stereo calculation processing unit 24 applies the left camera image data (referred to as a comparison image) and the right camera image data (referred to as a reference image) stored in the stereo image storage unit 23 for each small region of both images. Block matching for detecting corresponding points is performed by obtaining a correlation value, and pixel shift (parallax) of corresponding points between both images is calculated. This parallax data corresponds to the parallax d of the equation (1), and makes it possible to acquire three-dimensional distance information to the measurement object using the focal length f and the base line length B. Similarly to the stereo image calibration unit 22, the stereo calculation processing unit 24 is an electric circuit including an FPGA or the like. Details of the stereo operation processing unit 24 will be described later.

  The parallax data obtained by the stereo arithmetic processing unit 24 is stored in the parallax data storage unit 25 and used for vehicle control, recognition processing, and the like. Note that when storing in the parallax data storage unit 25, the parallax data may be converted into distance information using the equation (1).

[Details of the processing of the stereo processing unit]
FIG. 6 is a flowchart showing a flow of parallax value calculation in the stereo calculation processing unit 24.

  S 11: The stereo camera 13 captures two image data, and inputs the input image to the stereo image calibrating unit 22. The stereo image calibration unit 22 calibrates two pieces of image data using an LUT or the like, and stores them in the stereo image storage unit 23.

S12: The stereo calculation processing unit 24 performs block matching on the two left and right image data (comparison image and reference image), and calculates a parallax value d _int in pixel units. The processing procedure of S12 will be described in detail later.

FIG. 7 is an example of a diagram illustrating block matching. In block matching, an input image is divided into search blocks of small regions, and the position of the search block of the comparative image is shifted by one pixel in the horizontal direction of the image with respect to the search block of the reference image (while shifting). Perform the calculation. In this example,
Search block size: 7 × 7
Luminance value of each pixel of the reference image: Mi, j (i = 1 to 7, j = 1 to 7)
Luminance value of each pixel of the comparison image: Si, j (i = 1 to 7, j = 1 to 7)
And However, the size of the search block is only an example.

To limit the advance search range and search (shifted) (if the correlation value is small) showing the highest correlation is the correlation value at the time when the number of pixels satisfying the threshold value TH ₁ to determine the shift amount. The shift amount at the pixel position with the smallest correlation value represents the parallax value d _int in pixel units.

  As described above, in this embodiment, the ZSSD value is used as a correlation value, but this ZSSD value is a value obtained by multiplying a general ZSSD value by three.

  S13: The stereo calculation processing unit 24 calculates a parallax value in units of sub-pixels using the correlation value calculated in step S12.

As a method for calculating the parallax value of the sub-pixel, equiangular straight line fitting, parabolic fitting or the like is known. In this embodiment, the disparity value d _sub of the sub-pixel is calculated using equiangular straight line fitting.

FIG. 8 is an example of a diagram illustrating equiangular straight line fitting. In the equiangular straight line fitting, the smallest correlation value R (0), the second smallest correlation value R (1), and the third smallest correlation value R (-1) are used.
(i) Draw a straight line that passes through the smallest correlation value (black circle in the figure) and the third smallest correlation value (R (-1) in the figure)
(ii) A straight line with the sign of the slope of the straight line reversed, and a straight line that passes through the second smallest correlation value ((R (1)) in the figure)
(iii) Finding the position of the intersection of two straight lines, and using the distance from the smallest correlation value to the intersection as an estimated value of the parallax value d _sub in sub-pixel units. Is done. The correlation value of the pixel with the highest correlation is R (0), and the correlation values of adjacent pixels are R (-1) and R (1). In this embodiment, three times the correlation value is calculated, but the parallax value d _sub at the subpixel does not change even if the correlation value is multiplied by a constant.

Ｓ１４：ステレオ演算処理部２４は、ピクセル単位での視差値d_intとサブピクセル単位での視差値d_subを加算することで、視差データdを算出する。

S14: The stereo calculation processing unit 24 calculates the parallax data d by adding the parallax value d _int in pixel units and the parallax value d _sub in sub-pixel units.

<Processing of S12>
FIG. 9 shows an example of a flowchart showing the detailed procedure of the processing in step S12, and FIG. 10 shows an example of a diagram schematically showing the circuit configuration of the stereo arithmetic processing unit 24. The adder 241 adds the pixel values of 48 pixels of the comparison image search block, and the shift unit 242 shifts the addition result to the right by 4 bits. The shift unit 243 shifts the pixel values of 48 pixels of the search block of the comparison image by 1 bit to the left, and the adder 244 adds the pixel value shifted by 1 bit to the left and the original pixel value. The same applies to the block on the reference image side. The ZSSD calculator 249 calculates a ZSSD value.
S12-1: The adder 241 calculates the sum of 48 pixels other than the target pixel of the search block of the comparison image, and the adder 245 calculates the sum of 48 pixels other than the target pixel of the search block of the reference image. Note that the pixel of interest means a central pixel in the search block, which is M4,4, S4,4 in this embodiment.

  The adders 241 and 245 may be a circuit that repeats the addition 48 times for each pixel, but a circuit that performs the addition of 48 pixels at a time, or an addition of m pixels at a time and performs 48 / m There are various mounting forms such as a circuit that repeats times. In general, the larger the number of pixels that can be added at one time, the faster the operation is, but the circuit scale tends to increase. Therefore, the adders 241 and 245 are designed from the limitation of the circuit scale and the required calculation speed.

S12-2: Next, the shift unit 242 shifts the addition result of 48 pixels added by the adder 241 to the right by 4 bits, and the shift unit 246 shifts the addition result of 48 pixels added by the adder 245 to the right by 4 bits. shift. This is an arithmetic shift where the upper 4 digits are filled with zeros (meaning positive values) and the lower 4 digits are discarded. A 4-bit right shift is equivalent to dividing by 16. Originally, if an average value of pixel values of 48 pixels is to be obtained, it is sufficient to divide by 48. However, in a shift operation, division can be performed only by the power of 2. Therefore, in this embodiment, by shifting right by 4 bits, division is performed by 16, and an average value × 3 of 48 pixel values other than the target pixel is set as an approximate average value.
Approximate average value = value obtained by shifting the total of the pixel values of 48 pixels to the right by 4 bits = average value of pixel values of 48 pixels × 3
S12-3: The shift unit 243 and the adder 244 calculate a value obtained by multiplying the pixel values of 48 pixels in the comparison image search block by three, and the shift unit 247 and the adder 248 perform the reference image search block. A value obtained by multiplying the pixel values of 48 pixels among them by three is calculated. That is, the shift unit 243 shifts the pixel value of each pixel of the comparison image to the left by 1 bit, and the shift unit 247 shifts the pixel value of each pixel of the reference image to the left by 1 bit. While the most significant bit holds a zero representing a positive value, the lower bits other than the most significant bit are shifted to the left by 1 bit, respectively, and the least significant bit is supplemented with zero. Thereby, a doubled value is obtained.

  The shift unit 243 inputs a value obtained by doubling the pixel value to the adder 244, and the adder 244 adds the pixel value of each pixel of the comparison image and the doubled pixel value. Similarly, the shift unit 247 inputs a value obtained by doubling the pixel value to the adder 248, and the adder 248 adds the pixel value of each pixel of the comparison image and the doubled pixel value. Thereby, a value obtained by multiplying the pixel value of each pixel of the comparison image and the reference image by three is obtained.

S12-4: The ZSSD computing unit 249 calculates a value equivalent to a value obtained by multiplying the ZSSD value by three, using Expression (3). Hereinafter, this value is simply referred to as a ZSSD value.
3 × ZSSD = Σ {(3 × S _ij −approximate average value) − (3 × M _ij −approximate average value)} ² (3)
That is, the ZSSD computing unit 249 subtracts the approximate average value from the value obtained by multiplying the pixel value by 3 for each prime of the comparison image, and approximates the reference image from the value obtained by multiplying the pixel value by 3 for each prime. Decrease the value. Then, the value obtained by subtracting the subtraction value (3 × M _{ij −approximation} average value) of the reference image from the subtraction value (3 × S _ij −approximation average value) of the comparison image is squared. Then, all the 48 square values obtained are added.

  By such an operation, a ZSSD value can be obtained as a correlation value without performing division processing.

S12-5: stereo processing unit 24 determines whether or not the search number of pixels is the threshold value TH ₁ or more.

S 12 - 6: If the search the number of pixels is not greater than the threshold value TH _1, the stereo processing unit 24, the number of search pixel incremented by one (by shifting by one pixel the pixel of interest in the horizontal direction), S12-1～12- 4 processing of calculating to search the number of pixels is larger than the threshold value TH _1. Therefore, to obtain the threshold value TH ₁ correlation values, the shift amount of the smallest ZSSD value is obtained among this is the disparity value d _int in pixels.

  In step S12-2, the 4-bit right shift and triple processing are values when the size of the search block is 7 × 7 pixels, and the shift amount and the triple value are not limited to the above. For example, when the size of the search block is 4 × 4, first, the sum of pixel values of 16 pixels including the target pixel is calculated. In this case, the shift amount may be 4 bits on the right, and it is not necessary to triple the pixel value of each pixel.

  For example, when the size of the search block is set to 6 × 6, first, the sum of pixel values of 36 pixels including the target pixel is calculated. The shift amount is 2 bits to the right. In this case, the pixel value of each pixel is multiplied by nine. As described above, the size of the search block, whether to add the pixel value of the pixel of interest, the shift amount, and the fixed multiple can be determined as appropriate.

[Comparison with conventional methods]
FIG. 11 is a diagram showing a comparative example of the sub-pixel parallax value obtained by the division process and the sub-pixel parallax value obtained by the method of the present embodiment, and FIG. 12 is a graph showing the relationship between the translation amount and the sub-pixel. It is an example of the figure shown. The reason why the parallax values d _sub are compared in units of sub-pixels is that the calculation results of both are completely the same in units of pixels, and the difference cannot be confirmed.

For example, the experimenter kept the distance between the distance measurement target and the stereo camera 13 constant, and translated the stereo camera 13 five times by 0.2 pixels in a direction parallel to the distance measurement target. For each translation amount, the parallax value d _sub of the sub-pixel was calculated by the stereo calculation processing unit 24 using the conventional method and the calculation method of the present embodiment. In FIG. 12, the horizontal axis represents the translation amount [pix] of the stereo camera 13, and the vertical axis represents the parallax value d _sub of the subpixel.

As is clear from FIGS. 11 and 12, the calculation result of the parallax value d _sub of the present embodiment and the calculation result of the parallax value d _sub by the division process are equivalent (substantially the same).

  Therefore, it was verified that the stereo camera system 100 according to the present embodiment can realize the same accuracy as that obtained when the average value difference matching is performed using the division circuit or the division process by the higher-speed processing.

  In this embodiment, a stereo camera system 100 that prevents a small area having low correlation due to the influence of noise from being mismatched will be described. In this embodiment, after calculating the average value of the small areas of the reference image and the comparison image, the absolute value of the difference between the two average values is calculated. If the absolute value of the difference is larger than the threshold value, the two areas are different areas. Judgment is not performed by matching. By doing so, it is possible to prevent mismatching of small areas with low correlation. In addition, since the calculation is not performed for each pixel in the small area as in the prior art, the calculation cost can be reduced.

  In the present embodiment, the processing procedure of FIG. 6 is the same as that of the first embodiment, but the processing content of step S12 is different.

<Processing of S12>
FIG. 13 shows an example of a flowchart showing a detailed procedure of the process of step S12 of FIG. FIG. 14 shows an example of a diagram schematically showing the circuit configuration of the stereo arithmetic processing unit 24. The stereo arithmetic processing unit 24 of this embodiment has a subtracter 250. The subtractor 250 calculates the absolute value of the difference between the two approximate average values output from the shift units 242 and 246, and outputs a determination result as to whether or not it is equal to or less than the threshold value to the ZSSD calculator 249.

  S12-11: The adder 241 calculates the sum of 48 pixels other than the target pixel of the comparison image, and the adder 245 calculates the sum of 48 pixels other than the target pixel of the reference image.

  S12-12: Next, the shift unit 242 shifts the addition result of the 48 pixels added by the adder 241 to the right by 4 bits. Thereby, the approximate average value 2 is obtained. Similarly, the shift unit 246 shifts the addition result of 48 pixels added by the adder 245 to the right by 4 bits. Thereby, the approximate average value 1 is obtained.

  Note that it is not always necessary to shift, and the absolute value of the difference between the total values of the 48 pixels or 49 pixels of the reference image and the comparison image may be obtained.

S12-13: The subtracter 250 calculates the absolute value of the difference between the two approximations average 1,2, determines the difference is whether the difference is less than the similarity determination threshold _{TH 21.}
When two search blocks are similar, that is, when both search block areas are corresponding areas in the two images, the approximate average values calculated by the respective search blocks are substantially equivalent. On the other hand, when the two search blocks are different areas, the approximate average values calculated in the respective search blocks are also different values.

This concept, when the absolute value of the difference between the approximate average 1,2 in both search block is smaller than the similarity determination threshold TH _21, it can be determined that both the search blocks are similar. When the absolute value of the difference between the approximate average values 1 and 2 in both search blocks is equal to or greater than the similarity determination threshold TH ₂₁ , it is determined that the two search blocks are different areas.

  S12-14: When it is determined that both search blocks are similar, the ZSSD calculation unit 249 calculates a ZSSD value. The calculation method is the same as in the first embodiment.

S12-15: When both the search block is determined not to be similar, the stereo processing unit 24 determines whether the number of pixels of searching a threshold TH ₁ or more preset. If the searched number of pixels is the threshold value TH ₁ or more set in advance, the amount of shift smallest correlation value is obtained among several ZSSD value calculated is the disparity value d _int in pixels.

S12-16: when the number of pixels of searching is not the threshold value TH ₁ or more set in advance, the stereo processing unit 24 shifts to the pixel position of one pixel adjacent to continue the process of S12-11～S12-15.

  In this embodiment, since the average of the search blocks of the reference image and the comparison image is simply obtained, a process for subtracting the average value from the pixel values of the pixels of the search block is unnecessary, and the calculation cost can be reduced.

<Modification>
The mismatch prevention process of the present embodiment can also be realized by the stereo arithmetic processing unit 24 using a division circuit.

FIG. 15 shows an example of a flowchart showing a detailed procedure of the process of step S12 of FIG. FIG. 16 shows an example of a diagram schematically showing the circuit configuration of the stereo arithmetic processing unit 24. The stereo calculation processing unit 24 in FIG. 16 includes division circuits 251 and 252. The division circuits 251 and 252 divide the sum of the pixel values of 49 pixels added by the adders 241 and 245 by a constant 49. Therefore, the average values M _AVE and S _AVE of the pixel values in the reference image and comparison image search blocks are calculated.

減算器２５０は、平均値Ｍ_AVE、Ｓ_AVEの差の絶対値を算出し、閾値以下か否かの判定結果をＺＳＳＤ演算器２４９に出力する。したがって、図１３と同様に、両探索ブロックが２つの画像中で対応する領域の場合にだけ、ブロックマッチング処理を行うことができる。

The subtractor 250 calculates the absolute value of the difference between the average values M _AVE and S _AVE , and outputs a determination result as to whether the difference is equal to or less than the threshold value to the ZSSD calculator 249. Therefore, as in FIG. 13, the block matching process can be performed only when both search blocks are corresponding regions in the two images.

  S12-11: The adder 241 calculates the sum of 49 pixels of the comparison image, and the adder 245 calculates the sum of 49 pixels of the reference image.

S12-12 ′: Next, the division circuit 251 divides the addition result of 49 pixels added by the adder 241 by 49, and the division circuit 252 adds the addition result of 49 pixels added by the adder 245 to 49. Divide by. Thereby, average values M _AVE and S _AVE are obtained.

S12-13': subtractor 250 determines two average values M _AVE, calculates the absolute value of the difference between S _AVE, the difference is whether the difference is less than the similarity determination threshold _{TH 22.} If the absolute value of the difference between the approximate average values in both search blocks is smaller than the similarity determination threshold TH ₂₂ , it can be determined that both search blocks are similar. If the absolute value of the difference between the approximate average value in both the search block is greater than similarity determination threshold TH _22, it is determined that both the search block is different regions.

S12-14: When it is determined that both search blocks are similar, the ZSSD calculation unit 249 calculates a ZSSD value. The calculation method may be the same as that of the first embodiment, but when the division circuits 251 and 252 are included, the calculation can be performed as follows.
ZSSD = Σ {(S _ij −S _AVE ) − (M _ij −M _AVE } ² (4)
That is, the ZSSD calculator 249 subtracts the average value S _AVE from the pixel value S _{ij for} each pixel of the comparison image of the search block, and subtracts the average value M _AVE from the pixel value M _{ij for} each pixel of the reference image. The square sum of these differences is the ZSSD value.

S12-15: When both the search block is determined not to be similar, the stereo processing unit 24 determines whether the number of pixels of searching a threshold TH ₁ or more preset. If the searched number of pixels is the threshold value TH ₁ or more set in advance, the amount of shift smallest ZSSD value is obtained among several ZSSD value calculated is the disparity value d _int in pixels.

S12-16: when the number of pixels of searching is not the threshold value TH ₁ or more set in advance, the stereo processing unit 24 shifts to the pixel position of one pixel adjacent to continue the process of S12-11～S12-15.

  Even with such a calculation method, a process of subtracting the average value from the pixel value of each pixel of the search block is unnecessary, and the calculation cost can be reduced.

11 Display device 12 Image processing ECU
13 Stereo camera 20 Image processing IC
21 Stereo image input unit 22 Stereo image calibration unit 23 Stereo image storage unit 24 Stereo calculation processing unit 25 Parallax data storage unit 100 Stereo camera system

Japanese Patent Laid-Open No. 11-234701 JP 2003-288484 A

Z. Zhang, "A Flexible New Technique for Camera Calibration", Technical Report MSR-TR-98-71, Microsoft Research, 1998

Claims (8)

A pair of stereo cameras capturing a first image and a second image;
A total value calculating step in which the adding means calculates a first total value of the pixel data in the small area of the first image and a second total value of the pixel data in the small area of the second image. When,
A shift means shifts the first total value to the right by a predetermined number of bits to calculate a first approximate average value, and shifts the second total value to the right by the predetermined number of bits to obtain a second An approximate average value calculating step for calculating an approximate average value;
An integer multiplying step, the integer multiplying unit multiplying the value of each pixel data in the small area of the first image by an integer, and multiplying the value of each pixel data in the small area of the second image by an integer;
Correlation value calculation means, for each pixel data in the small area of the first image, first data obtained by subtracting the first approximate average value from a value obtained by multiplying the value of the pixel data by an integer, and the first data Second data obtained by subtracting the second approximate average value from a value obtained by multiplying the pixel data by an integer is obtained for each pixel data in the small area of the second image, and the first data and the second data are obtained. A correlation value calculating step for calculating a sum of squares of differences with respect to data or a sum of absolute values of differences;
An image processing method. The integer multiplication means adds the value of the pixel data to a value obtained by shifting the value of the pixel data to the left by a predetermined number of bits for each pixel data in the small area of the first image. Multiply the value of each pixel data in a small area of one image by an integer,
For each pixel data in the small area of the second image, the value of the pixel data is added to a value obtained by shifting the value of the pixel data to the left by a predetermined number of bits. Multiply the value of each pixel data in the area by an integer,
The image processing method according to claim 1. The number of pixel data in the small area of the first image or the number of pixel data in the small area of the second image, which is added by the adding means, is a power of 2 of the predetermined number of bits shifted by the shifting means. And the number divided by
Multiples of integer multiples by integer multiple means are equal,
The image processing method according to claim 1, wherein the image processing method is an image processing method. The subtracting means has a step of determining whether or not an absolute value of a difference between the first approximate average value and the second approximate average value is smaller than a threshold;
Only when the absolute value is smaller than a threshold value, the integer multiplication means performs an integer multiplication step, and the correlation value calculation means performs a correlation value calculation step.
The image processing method according to any one of claims 1 to 3. A first division result obtained by dividing the first total value obtained by addition by the adding means by the number of pixel data in the small area of the first image;
A division step for respectively obtaining a second division result obtained by dividing the second total value by the number of pixel data in a small area of the second image;
Determining whether an absolute value of a difference between the first division result and the second division result is smaller than a threshold;
Only when the absolute value of the difference between the first division result and the second division result is smaller than a threshold value, the integer multiplication unit performs an integer multiplication step, and the correlation value calculation unit performs a correlation value calculation step.
The image processing method according to any one of claims 1 to 3. The computing means selects the smallest correlation value from a plurality of correlation values obtained by shifting the small area of the first image pixel by pixel, and the first correlation value is obtained based on the shift amount when the correlation value is obtained. Calculating a distance between the image of the object and an object photographed in common with the second image;
The image processing method according to claim 1, further comprising: The calculation means extracts three correlation values in ascending order from the plurality of correlation values, performs equiangular straight line fitting on the three correlation values, and calculates disparity information in sub-pixel units.
The image processing method according to claim 6.   The stereo camera system carrying the image processing method of any one of Claims 1-7.

Images