JP2008151659A - Object detector - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To allow successive calculation of distance information even when an object is lost during stereo range finding. <P>SOLUTION: A first range finding part 6 performs stereo matching between images of a right and left pair photographed by a stereo camera 2 to perform range finding by detecting parallax with respect to the same object. A second range finding part 7 performs stereo matching based on a luminance difference to detect the parallax to cope with such a case that reliability of the range finding images generated by the first range finding part 6 is deteriorated due to deviation in sensitivity balance between the stereo photographed images of the right and left pair caused by backlight or the like. The range finding by the first range finding part 6 and the range finding by the second range finding part 7 are selectively switched so as to be input to a recognition processing part 10 so that the successive range finding is allowed against the degradation of environmental conditions such as those in the backlight. The deterioration of recognition reliability due to change in environmental conditions is prevented thereby. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an object detection device that detects an object by processing an image captured by a stereo camera.

  Conventionally, as a technique for object detection based on an image, a correlation between a plurality of images obtained by capturing an object from different positions with a stereo camera is obtained, and a camera parameter such as a stereo camera mounting position and a focal length is used from a parallax for the same object. An object detection technique that uses the so-called stereo method to determine the distance based on the principle of triangulation is known.It can be applied to vehicles such as automobiles to recognize the environment outside the vehicle, and to control the distance between vehicles and obstacles. Is used for collision avoidance control.

For example, Patent Document 1 stores a region based on a vertical edge pair in a stereo image as a reference pattern and detects a vertical edge regarding a technology for recognizing an environment outside a vehicle using a combination of a monocular camera and a radar or a stereo camera. If not, a technique is disclosed in which an object is detected (tracked) by pattern matching using a reference pattern, assuming that the object has been lost.
JP 2005-38407 A

  In the technique of Patent Document 1, when a vertical edge cannot be detected in an image by a stereo camera, that is, when an object is lost, the technique is switched to a method of detecting an object by template matching. However, template matching can detect the presence or absence of an object, but cannot obtain object distance information, so it is difficult to continue obstacle avoidance control and tracking control by tracking the preceding vehicle. It is not enough to ensure safety.

  The present invention has been made in view of the above circumstances, and an object of the present invention is to provide an object detection apparatus capable of continuously calculating distance information even if an object is lost during stereo ranging.

  In order to achieve the above object, an object detection apparatus according to the present invention searches a pair of images for a stereo camera that acquires a pair of images and searches for a region in which the brightness values correspond to each other between the pair of images. Calculating a luminance difference between the first distance information calculating means for calculating the distance information of the object and the adjacent pixels for the predetermined pixel in each of the pair of images, and the luminance difference between the pair of images. A second distance information calculating means for searching the corresponding area and calculating the distance information of the object; and the second distance information calculating means when the object is lost by the first distance information calculating means. Selection means for selecting the distance information to be displayed.

  The object detection device according to the present invention can continuously calculate distance information even if an object is lost during stereo ranging, and can improve the controllability and reliability of various controls based on object detection. It becomes possible.

  Embodiments of the present invention will be described below with reference to the drawings. 1 to 12 relate to an embodiment of the present invention, FIG. 1 is a block diagram of a preceding vehicle tracking control system, FIG. 2 is a block diagram of an object detection device, and FIG. 3 is a luminance distribution of left and right images during backlighting. FIG. 4 schematically illustrates the luminance distribution of the left and right images after luminance difference, FIG. 5 illustrates the tracking failure determination, and FIG. 6 illustrates the concept of left / right inversion in the preceding vehicle position correction. FIG. 7 is an explanatory diagram showing correction of the preceding vehicle position, FIG. 8 is an explanatory diagram showing the correction result of the preceding vehicle position, FIG. 9 is a flowchart of the preceding vehicle recognition process, and FIG. 10 is a flowchart of the preceding vehicle tracking process. FIG. 11 is a flowchart of the lost determination process, and FIG. 12 is a flowchart of the preceding vehicle follow-up control process.

  The object detection device of the present invention enables continuous distance information acquisition even when the sensitivity balance between images captured by a stereo camera deteriorates. For example, the obstacle detection control of a vehicle and the preceding vehicle It can be applied to the inter-vehicle distance control by tracking the vehicle. In this embodiment, as shown in FIG. 1, stereo image recognition is performed on a vehicle 1 by performing stereo processing on a stereo camera 2 and an image captured by the stereo camera 2 and recognizing the external environment mainly using object distance information. The example which mounts the apparatus 5 as an object detection apparatus, connects various vehicle control apparatuses to the stereo image recognition apparatus 5, and comprises a preceding vehicle follow-up control system is demonstrated.

  The stereo camera 2 includes a plurality of cameras having image sensors such as a CCD and a CMOS, and images a target object from different viewpoints. In this embodiment, as the stereo camera 2, two left and right cameras 2a and 2b synchronized with each other with variable shutter speed are mechanically mechanically with a predetermined base line length (optical axis interval) so that their optical axes are substantially parallel to each other. It is fixed and installed at the front of the vehicle 1.

  Further, as the vehicle control device, an engine control device 21 that receives the external environment recognition information from the stereo image recognition device 5 and controls the engine output to the central control device 20 that controls the vehicle control based on the external environment recognition result, A brake control device 22 that controls the brake and an alarm device 23 that alerts the driver by issuing a warning by voice output, display display, or the like when a rear-end collision risk or abnormality occurs in the preceding vehicle are connected.

  The central controller 20 receives the distance and speed between the preceding vehicle and the host vehicle, the lost information when the preceding vehicle is lost, and the like as a result of the external environment recognition from the stereo image recognition device 5 and controls the engine output. A throttle opening command, a target deceleration for controlling the braking force, an alarm command, and the like are generated and output to the engine control device 21, the brake control device 22, and the alarm device 23, respectively.

  The recognition of the external environment in the stereo image recognition device 5 is based on acquisition of distance information for processing a pair of images captured by the stereo camera 2 and calculating a distance to an object such as a preceding vehicle or an obstacle existing around the own vehicle. And has two distance measuring units for acquiring distance information. That is, as shown in FIG. 2, the stereo image recognition device 5 includes a first distance measuring unit 6 and a second distance measuring unit 7 that perform three-dimensional distance measurement of an object in parallel by different processes, and further A tracking processing unit 8, a distance measurement selection unit 9, and a recognition processing unit 10.

  The first distance measuring unit 6 corresponds to the first distance information calculating means in the present invention, and performs normal distance measurement under conditions that do not inconvenience stereo processing. The distance is measured by performing stereo matching between the images and detecting the parallax with respect to the same object.

  The second distance measuring unit 7 corresponds to the second first distance information calculating means in the present invention, and the sensitivity balance between the pair of left and right images captured in stereo by backlight or the like is distorted, and the first distance measuring unit 7 This is to deal with a case where the reliability of the distance image generated by the unit 6 is lowered. As will be described below, the second distance measuring unit 7 detects the parallax by performing stereo matching that avoids the influence of the luminance balance shift.

  The distance measurement by the first distance measurement unit 6 and the distance measurement by the second distance measurement unit 7 are selectively switched by the tracking processing unit 8 and the distance measurement selection unit 9 corresponding to the selection means in the present invention, and the recognition processing unit 10. Thus, continuous distance measurement can be performed even when the environmental conditions are deteriorated such as in backlight, and the recognition reliability can be prevented from deteriorating due to changes in the environmental conditions.

  The stereo matching in the first distance measuring unit 6 evaluates the correlation between the left and right images using, for example, a well-known region search method. That is, with one image of the stereo camera 2 as a reference image and the other image as a comparison image, a small region (block) is set around a certain point in the reference image, and a certain point in the comparison image is Corresponding points are searched for by providing a small region of the same size around and performing a correlation calculation of the blocks while shifting the blocks on the comparison image.

  As an evaluation function in this correlation calculation, a sum (SAD: Sum of Absolute Difference) of differences (absolute values) of pixel values (generally luminance values of the respective pixels) between the block of the reference image and the block of the comparison image is used. ) Is used. The value of the evaluation function by SAD is called city block distance. The higher the correlation between blocks (the more similar), the smaller the value of the city block distance, and the minimum city block distance. Parallax is given by the amount of horizontal shift between the blocks to be taken.

The city block distance CB defines a position on the image plane as an orthogonal coordinate in which the horizontal direction is i-coordinate and the vertical direction is j-coordinate. n, j = 0 to n), the reference image search block main (i, j) and the comparison image search block sub (i, j), as shown in the following equation (1): The SAD value is calculated by shifting the SAD value on the i-axis by a predetermined shift value.
CB = Σ | main (i, j) -sub (i, j) | ... (1)
Details of stereo matching between search blocks (small regions) are described in detail in Japanese Patent No. 3167775 of the present applicant.

  In the above SAD calculation, for example, for a search block of 4 × 4 pixels, a reference image search block (hereinafter referred to as “main block”) is compared with a comparison image search block (hereinafter referred to as “sub block”). When the SAD calculation is performed while shifting the position of (described) one pixel at a time on the horizontal line, the point where the city block distance is the minimum value C0 is the corresponding position (matching point) having the highest degree of correlation between the main block and the sub-block. Desired.

  The amount of shift in units of one pixel between the main block and the sub block at this coincidence point (the difference between the position im of the main block in the horizontal direction and the position is of the sub block) is given as parallax. This parallax, that is, the amount of deviation is replaced with the luminance value at the corresponding position in the image coordinate system and is generated as a distance image in the form of an image, and the distance in real space can be obtained using the distance information of the distance image.

  In this case, the small area of the image includes parts other than the target preceding vehicle (background telephone poles, trees, white line in front, etc.). There is sex. Therefore, the parallax data for each small image region is arranged in descending order, and data on the side larger than a predetermined ratio (for example, 25%) and data on the smaller side are excluded as singular points. Then, an average value is calculated for the remaining data (50% in the center), and this is used as the parallax of the preceding vehicle.

  On the other hand, in stereo matching in the second distance measuring unit 7, the original image when the stereo image processing is normally performed in the first distance measuring unit 6 is stored as a template image, and the template image is converted into the captured image. On the other hand, a matching area is set, and stereo matching is performed using the luminance difference between adjacent pixels in the matching area.

  For example, when sunlight enters the imaging surface of the stereo camera 2 and becomes backlit, the luminance values of the left image and the right image are schematically represented as shown in FIG. The horizontal axis of the graph in FIG. 3 is the horizontal coordinate, the vertical axis is the luminance value at the position where the horizontal line on the image is drawn, and the left image shown in FIG. 3A and the right image shown in FIG. In this case, the left and right sensitivity balance is greatly broken due to the incidence of sunlight and the like, and a left and right luminance offset is generated. For this reason, even if stereo processing is performed by SAD calculation of luminance, it is difficult to perform normal matching.

  However, although there is an offset in the left and right luminances, there is a commonality in how the luminance fluctuates, and when the difference in luminance between adjacent pixels (luminance value of a certain pixel−luminance value of the right pixel) is graphed, As shown in FIG. 4, it can be seen that the luminance difference of the left image shown in FIG. 4A and the luminance difference of the right image shown in FIG. Therefore, if the luminance difference is calculated for each of the left and right images, and the SAD value described above is calculated for the luminance difference, the parallax can be obtained while ensuring the necessary accuracy even if a luminance offset occurs between the stereo images. Can be detected.

  Switching between the distance measurement by the first distance measurement unit 6 and the distance measurement by the second distance measurement unit 7 is determined by the tracking processing unit 8 based on the presence or absence of lost in tracking the object of interest (tracking the preceding vehicle). When there is no lost, distance information calculated by the first distance measuring unit 6 with higher accuracy is selected by the distance measuring selecting unit 9 and input to the recognition processing unit 10. On the other hand, if it is determined that the lost has occurred, the distance information calculated by the second distance measuring unit 7 is selected by the distance measuring selection unit 9 and input to the recognition processing unit 10.

  The tracking process of the preceding vehicle in the tracking processing unit 8 cuts out the right or left camera image corresponding to the preceding vehicle recognized by the distance image when the stereo image processing by the first distance measuring unit 6 is normally executed, It is stored and held for each frame as a template image of the preceding vehicle. Then, when the luminance balance between the left and right images is lost due to backlight or the like, the position on the image corresponding to the preceding vehicle is specified by searching the left and right images using the template image, and the second distance measuring unit 7 Perform stereo matching of luminance differences.

  Note that the template image of the preceding vehicle is stored and held as a reduced image, so that the memory capacity and calculation load can be reduced and the processing speed can be increased.

  As a search method, various conventionally known methods such as a residual successive test method and a method using a cross-correlation coefficient (“Image Analysis Handbook”: The University of Tokyo Press, Mikio Takagi, Yoshihisa Shimoda, 1991 Year reference). The outline of the search by these methods will be described below.

A. Residual sequential test method The preceding vehicle image is used as a template image, and the image to be searched is scanned up and down. The SAD values of the template image and the search image are calculated by the following equation (2), and it is determined that the current preceding vehicle exists at the position in the search image where the minimum value of SAD is obtained. In the following equations, Σj represents that a sum is obtained for j = 0 to Nj−1, and Σi represents a sum for i = 0 to Ni−1.
R (a, b) = ΣjΣi | I (a, b) (i, j) −T (i, j) | ... (2)
Where a: horizontal coordinate in the search image
b: Vertical coordinate in search image
R (a, b): SAD value at coordinates (a, b)
Nj: Wide search range (number of vertical pixels)
Ni: Wide search range (number of horizontal pixels)
j: Vertical coordinate in the template image (vertical pixel count)
i: Horizontal coordinate in template image (number of horizontal pixels)
I (a, b) (i, j): luminance value of the search image at coordinates (a + i, b + j)
T (i, j): Brightness value of template image at coordinates (i, j) In this residual sequential test method, if the added value exceeds a threshold value in the middle of SAD addition, the overlay of images is shifted. Judgment is made, and addition is interrupted, and it moves to the next coordinate (a, b). Thereby, processing time can be shortened.

B. Cross-correlation coefficient method The search range is scanned in the same way as the residual sequential test method, but the cross-correlation coefficient is used instead of the SAD value, and it is determined that the preceding vehicle is present at the position where the maximum value is obtained. . The cross correlation coefficient is calculated by the following equation (3).
C (a, b) = ΣjΣi {I (a, b) (i, j) −Iave} {T (i, j) −Tave} / √ (Iσ · Tσ) (3)
Where C (a, b): cross-correlation coefficient at coordinates (a, b)
Iave: Average brightness value within the corresponding range of the search image
Tave: Average brightness of template image
Iσ: luminance dispersion value within the corresponding range of the search image
Tσ: brightness dispersion value of template image Note that when tracking the image of the preceding vehicle on the original image, there is a problem that the brightness offset changes with time. In addition, tracking can be performed effectively by performing a residual sequential test method or a method using a correlation coefficient.

  Even with the above-described residual sequential test method or cross-correlation coefficient method, there is a possibility that an image that is not a preceding vehicle is erroneously determined as a preceding vehicle because processing is not successful. Accordingly, in order to reliably detect the tracking failure even in such a case, the tracking failure is determined by the method described in (a) or (b) below.

(A) Judgment of tracking failure in residual sequential test method The SAD value R (a, b) at the preceding vehicle position and its surroundings is calculated by the above-mentioned equation (2), and the depth of the SAD valley is calculated from the threshold value. If it is shallow, it is determined that tracking has failed. The depth of the SAD valley is a reliability parameter SAD_reliability calculated as a ratio of the difference between the SAD average value Rave and the SAD value R (a, b) to the SAD average value Rave as shown in the following equation (4). Evaluate with.
SAD_reliability = {Rave−R (a, b)} / Rave (4)
However, Rave: average value of SAD values of peripheral pixels surrounding the pixel of coordinates (a, b) The SAD average value Rave is, for example, the following equation (5) or (6) ) Equation.
Rave = R (a-1, b) + R (a, b-1) + R (a + 1, b) + R (a, b + 1) + R (a, b) (5)
Rave = R (a-1, b) + R (a, b-1) + R (a + 1, b) + R (a, b + 1) (6)
The reliability parameter SAD_reliability takes a value in the range of 0 to 1, and it is determined that the reliability is higher as it is larger. That is, when the template image and the search image are normally matched, a minimum point appears in the SAD value R (a, b) as shown in FIG. The ratio of the SAD value R (a, b) is small as shown in FIG. 5 (b) when another vehicle interrupts between the preceding vehicle and the own vehicle. Thus, the ratio of the difference with respect to the average value Rave becomes small. Therefore, for example, when the threshold value is 0.4, it is determined that tracking failure occurs when SAD_reliability ≦ 0.4.

(B) Judgment of tracking failure in the method using the cross-correlation coefficient The cross-correlation coefficient at the preceding vehicle position and its surroundings is calculated by the above-described equation (3), and if the peak of the cross-correlation coefficient is low, Judge that tracking has failed.

  In addition, the template image (preceding vehicle image) in the above processing selects one of the left and right original images. The right or left image is suitable indicates the effectiveness of SAD (4). The determination is made using the reliability parameter SAD_reliability of the equation, and the image with the larger value (either left or right) is selected as the preceding vehicle image. For example, when the reliability parameter SAD_reliability of the preceding vehicle range searched from the left image is 0.3 and the reliability parameter SAD_reliability of the preceding vehicle range searched from the right image is 0.7, the calculated value of the reliability parameter SAD_reliability is right Since the image is larger, the right image is used as the main image in the subsequent processing.

  Furthermore, the range of the selected preceding vehicle image may be shifted vertically and horizontally without accurately cutting out the preceding vehicle. Even if it deviates up and down, no major problem will occur, but if it deviates left and right and does not include the contour of the preceding vehicle, the strength of the edge used in the subsequent parallax calculation will be weakened and the distance accuracy will be reduced. Therefore, the position of the preceding vehicle is corrected using the left-right symmetry of the car.

  That is, as shown in FIG. 6, when the left end of the contour of the preceding vehicle in the main image is missing, the right half is extracted from the main image before position correction, and this is reversed horizontally. Then, the left-right inverted image is searched on the main image by the above-described residual sequential test method, and the matching position is specified. Next, the reliability of the matching position is evaluated by the above-described reliability parameter SAD_reliability. If the value of the reliability parameter SAD_reliability exceeds the threshold value, it is determined that the matching by the left-right inverted image is successful, and the preceding vehicle Correct the image.

  When correcting the preceding vehicle image, as shown in FIG. 7, the center of the left half and the right half are adjusted while moving the center in accordance with the center position of the right half and the left half so that the width of the image is not changed. The preceding vehicle image is corrected so that the center of the corrected preceding vehicle image coincides with. In FIG. 7, a white frame indicates a region before correction, and a black frame indicates a region after correction.

  As a result, as shown in FIG. 8, the preceding vehicle image before correction, which did not include the left end of the preceding vehicle, is corrected to the preceding vehicle image representing the outline of the entire preceding vehicle. The image is used as an image, and the left and right images taken in stereo are searched to identify the position on the image corresponding to the preceding vehicle, and stereo matching is performed by the luminance difference.

  Here, when the matching of the preceding vehicle region using the template image in the second distance measuring unit 7 and the stereo matching by the normal small region in the first distance measuring unit 6 are compared, the stereo matching using the template image is: Since the number of data used for matching target objects increases, the possibility of mismatching is low, and it is possible to detect parallax effectively even for stereo images with an unbalanced luminance balance, but the obtained parallax data is limited. Therefore, statistical processing is difficult, and high accuracy cannot be expected as compared with normal stereo matching using a small area.

  On the other hand, since stereo matching with normal small regions can obtain the number of data as many as the number of small regions, accuracy can be improved by statistical processing, and mismatching can be excluded by removing singular points by statistical processing. In order to increase the accuracy, it is necessary to increase the size / number of small areas.

  The distance image data from the first distance measuring section 6 or the second distance measuring section 7 is selectively sent to the recognition processing section 10 via the distance measuring selection section 9. The recognition processing unit 10 recognizes the preceding vehicle by extracting a crowded portion of the same distance data from the distance image. In the preceding vehicle recognition, the distance to the preceding vehicle is calculated using the distance image data, the speed of the preceding vehicle is calculated by taking the inter-frame difference in the distance, and the lost determination result in the preceding vehicle tracking process is calculated. The lost determination flag is output to the central controller 20.

  Based on the data from the recognition processing unit 10, the central control device 20 operates the throttle / brake via the engine control device 21 and the brake control device 22 to maintain the inter-vehicle distance at the target value. Further, when there is a danger of a rear-end collision with the preceding vehicle from the distance / speed data, an inter-vehicle distance alarm is issued via the alarm device 23.

  Next, the preceding vehicle following control in the preceding vehicle following control system to which the above processing is applied will be described with reference to the flowcharts of FIGS. The preceding vehicle follow-up control is realized by the preceding vehicle recognition and tracking process of FIGS. 9 to 11 in the stereo image recognition device 5 and the preceding vehicle follow-up control process of FIG.

[Previous vehicle recognition processing]
First, the image processing sequence shown in FIG. 9 will be described as processing on the stereo image recognition device 5 side. In this image processing sequence, in the first step S1, the front of the host vehicle is imaged by the stereo camera 2 to acquire left and right image data. Next, the process proceeds to step S2, in which normal stereo processing by the first distance measuring unit 6 is executed as first stereo processing, and distance image data is generated. In subsequent step S3, recognition processing of the distance image generated in the first stereo processing is performed to detect object data in real space coordinates. In step S4, it is determined whether or not the tracking data flag indicates “inserting”. Investigate.

  The tracking data flag is “inserting” indicating that the tracking data of the object generated by the preceding vehicle tracking process performed in parallel is being added to the object data obtained by the first stereo process, This is set to one of “cancel insertion” indicating that the preceding vehicle is normally captured in one stereo process and the tracking data is not required.

  If the image captured by the stereo camera 2 is not out of brightness balance due to backlight, etc., or if there is no other interrupting vehicle between the preceding vehicle and the own vehicle, the preceding vehicle can be captured smoothly. The tracking data flag is set to “cancel insertion”, and the process proceeds from step S4 to step S5 to refer to the flag indicating the occurrence of lost. The flag indicating the occurrence of lost is set by the lost determination process shown in FIG. 11, and the lost determination process will be described later.

  As a result of referring to the flag in step S5, if no loss has occurred, the process proceeds from step S5 to step S8, the tracking data flag is set to “cancel insertion”, and the preceding vehicle is selected from the object data in step S10. Exit processing. In a state where no loss has occurred, an object existing at the closest distance on the own lane is selected as the preceding vehicle from the object data acquired by the first stereo process.

  On the other hand, if lost as a result of referring to the flag in step S5, the process proceeds from step S5 to step S6, the tracking data flag is set to “inserting”, and data addition to the object is stopped in step S7. Judge whether to do. The condition for determining that object data addition is to be stopped is usually when the preceding vehicle shown in (1) below is re-acquired, but in addition, the object is also subject to the conditions (2) to (4) described below. It is determined that data addition has been canceled.

(1) Canceling object data addition due to re-acquisition of a preceding vehicle Once lost occurs, object data generated by the tracking process is continuously added for each frame. However, if there is an object that is considered to be the same object as the object generated by the tracking process among the recognized objects by the distance image of the first stereo process in the next frame or later, the distance image recognition by the first stereo process is normal. It is determined that the state has been restored, and the addition of object data is stopped.

  For example, the same object is determined by calculating the probability that the recognized object based on the distance image of the first stereo process and the object generated by the tracking process are the same object from each distance and the horizontal position, and this probability is set to a threshold value. If there is something that exceeds, it is determined that the same object exists. Details of the same object determination are described in detail in Japanese Patent Application Laid-Open No. 2004-037239 by the present applicant.

(2) Canceling addition of object data due to lowering of tracking reliability An evaluation value representing tracking reliability is introduced, and a predetermined value (for example, 100 points) is set as an initial value when a loss occurs. Every time a phenomenon that is considered to have low tracking reliability occurs, the evaluation value is subtracted, and when the score reaches 0, the tracking is considered to have failed and the subsequent addition of object data is stopped. The subtraction conditions for the reliability evaluation value are as shown in the following (2-1) to (2-6).

(2-1) When the value of the reliability parameter SAD_reliability calculated by the above equation (4) is smaller than the threshold value, the reliability evaluation value is subtracted.

(2-2) Calculate the cross-correlation coefficient between the preceding preceding vehicle image and the current preceding vehicle image according to the method using the cross-correlation coefficient in the above equation (3), and the calculated cross-correlation coefficient is below the threshold value. If it becomes, the reliability evaluation value is subtracted.

(2-3) If the position identified as the preceding vehicle in the tracking process is on the boundary of the search range (the top, bottom, left, or right end of the search range) Subtract gender.

(2-4) When the position in the screen of the object that has been set as the preceding vehicle in the tracking process moves to the left and right for each processing frame (the position is violent), the variance values of the left and right positions and the vertical position are observed over several frames. When the variance exceeds the threshold, the reliability evaluation value is subtracted.

(2-5) When the calculation result of the distance to the tracking vehicle derived in the tracking process fluctuates greatly every processing frame (the distance is violent), the dispersion value of the distance is observed over several frames, and the dispersion is a threshold value. The reliability evaluation value is subtracted when exceeding.

(2-6) Distance data by distance image recognition has occurred at the position identified as the preceding vehicle in the tracking process, and the distance is significantly different from the recalculation result of the distance, resulting in a different distance at the same position. When the number of distance data is larger than the threshold, it is determined that the tracking process and the distance image are contradictory, and the reliability evaluation value is subtracted.

(3) Canceling addition of object data due to night driving In the case of lost due to backlighting, tracking processing of a monocular image is often effective even if the distance image is disturbed, but at night, tracking with a monocular image is performed by the light of various surrounding lights. Processing becomes difficult. In order to prevent problems such as false braking due to the failure of tracking processing at night, the tracking function is stopped at night and the addition of object data is stopped. For example, the detection of nighttime is performed by utilizing the relationship between the average luminance in the screen and the exposure time. That is, the exposure time is lengthened and the average luminance within the screen is measured. If the average luminance is equal to or less than the threshold value, it is determined that the night.

(4) Object data addition stop by non-selection of additional object As will be described later, the tracking data of the object generated by the tracking process is added to the object data detected by the first stereo process, and these data are combined. The preceding car is extracted and selected from. If the tracking object by this tracking process is not selected as the preceding vehicle, it is determined that the tracking object has gone out of the lane or that there is an object by distance image recognition in the vicinity. The frame after the frame is canceled.

  As described above, when it is determined in step S7 that the addition of object data is to be stopped, the tracking data flag is set to “insertion stop” in step S8 described above, and the preceding vehicle is selected from the object data in step S10, and the current process is exited. . On the other hand, if it is determined in step S7 that the addition of object data is not to be stopped, the process proceeds from step S7 to step S9 to add tracking data to the object data detected in the first stereo process, and in step S10, the object data is preceded. Select a car and exit the process. In this case, the object existing at the closest distance on the own lane is selected as the preceding vehicle from the combination of the detected object by the distance image recognition generated by the first stereo process and the tracked object by the tracking process. become.

  Thereafter, when the preceding vehicle is re-captured (or the tracking reliability is lowered) from the state where the lost state is continued, “insertion” is made with reference to the tracking data flag in step S4, and the process proceeds to step S7. It is determined that the addition of data is stopped. In step S8, the tracking data flag is set to “cancel insertion”, and in step S10, the process returns to the selection of the preceding vehicle based on the object data of the first stereo process.

  In parallel with the image processing described above, the preceding vehicle tracking process shown in FIG. 10 and the lost determination process shown in FIG. 11 are executed. As described above, in this embodiment, the preceding vehicle tracking process is stopped at night.

[Previous vehicle tracking process]
In the preceding vehicle tracking process of FIG. 10, in the first step S11, the left and right original image data captured by the stereo camera 2 is acquired, and the image suitable for the template image (the preceding vehicle image) is selected and stored. save. As described above, the selection of an image is determined using the reliability parameter SAD_reliability of the equation (4).

  Next, it progresses to step S12 and it is investigated whether there was any preceding vehicle last time (whether it was recognized). As a result, if there is a preceding vehicle, a preceding vehicle tracking process is performed in step S13. In this preceding vehicle tracking process, the position corresponding to the preceding vehicle is cut out from the stored original image, and the preceding vehicle position is corrected while evaluating the matching reliability as necessary, but the calculation processing time is reduced. Therefore, thinning processing of image data to be tracked is performed, and tracking processing is performed on an image whose data number is reduced by thinning. In this case, the size of the preceding vehicle on the image changes as the preceding vehicle approaches or moves away. However, in order to enable tracking processing even if the size changes, the thinning rate is adjusted when thinning is performed. Then, the number of pixels on the screen of the template image and the preceding vehicle is made to substantially match.

  Thereafter, the process proceeds to step S14, and the stereo process by the second distance measuring unit 7 is performed as the second stereo process on the left and right images acquired in step S11 using the template image of the preceding vehicle, and the distance image is generated and tracked. The object is detected and the current process is exited. As described above, the second stereo process is a process of searching for a position on the image corresponding to the preceding vehicle image by the residual sequential test method, and performing stereo matching by the luminance difference with respect to the preceding vehicle region. It is possible to track the preceding vehicle effectively even when the lost due to.

  On the other hand, if there is no preceding vehicle in step S12, the process proceeds from step S12 to step S15, the tracking data flag is set to “cancel insertion”, and the current process is exited. With this “insertion stop” flag setting, normal preceding vehicle detection by the first stereo process is performed in the preceding vehicle recognition process of FIG. 9 described above.

  The distance image generated by the second stereo process is evaluated for its recognition reliability. If the recognition reliability is high, the template image used for tracking is updated. If the recognition reliability is low, the conventional template image is added to the template image. Keep the data retained. The reduction in the reliability of the distance image recognition is evaluated by, for example, the width of the recognized object.

[Lost judgment processing]
Next, the lost determination process of FIG. 11 will be described. In this lost determination process, first, in step S21, a flag for checking the occurrence of lost is initialized to 0. Setting this flag = 0 means the occurrence of lost. Next, in step S22, a counter k for counting the number of objects is initialized to 0, and the process proceeds to step S23.

  In step S23, the position Xobj [k] of the object detected this time is within the set value ΔX in the horizontal direction with respect to the previous preceding vehicle position Xtarget (Xtarget−ΔX <Xobj [k] <Xtarget + ΔX), and the distance It is checked whether or not a condition (Xtarget−ΔX <Xobj [k] <Xtarget + ΔX) that is within the set value ΔZ in the direction is satisfied.

  In step S23, if the above condition is not satisfied and no object is detected in the vicinity of the previous preceding vehicle, the process jumps to step S25 as lost and increments the counter k in step S25. On the other hand, if the above condition is satisfied, that is, if an object is detected again at a position close to the previous preceding vehicle position, the process proceeds from step S23 to step S24 and the flag is set to 1. In S25, the counter k is incremented.

  In step S26 following step S25, it is checked whether or not the counter k has reached the number of objects to be determined. If k <the number of objects, the process returns from step S26 to step S23 to check the positional relationship with respect to the next target object. If k ≧ the number of objects, the process proceeds from step S26 to step S27 to refer to the flag. Check whether the flag is 1 or not.

  As a result, if the flag is 1, it is determined in step S28 that the previous preceding vehicle is present and the current process is exited. If the flag is 0, the previous preceding vehicle disappears (lost) in step S29. It is determined that the process has been completed and the current process is exited.

[Leading vehicle tracking control process]
When the above loss occurs, the central controller 20 outputs a warning of recognition performance deterioration via the alarm device 23 and alerts the driver, but the preceding vehicle follow-up control continues and follows without any precautions. It is possible to prevent the driver from coping with the delay in driving. Even when it is determined that the preceding vehicle has been recognized again by normal distance image recognition (recognition by the first stereo process), the control is continued while outputting the recognition performance restoration notification to the driver. Next, the preceding vehicle follow-up control process in FIG. 12 will be described.

  In this preceding vehicle follow-up control process, it is checked in the first step S31 whether there is a preceding vehicle detected by image recognition. If there is no preceding vehicle, the present process is terminated. If there is a preceding vehicle, the process proceeds to step S32, and whether or not the normal distance image recognition by the first stereo processing is performed in the stereo image recognition device 5 is lost. Investigate whether the occurrence tracking process is performed.

  As a result, when normal distance image recognition is performed, the control process by steps S33 to S35 is executed, and when the tracking process is performed, the control process of steps S36 to S39 is executed.

  Steps S33 to S35 when normal distance image recognition is performed will be described. In step S33, when the distance / speed data of the preceding vehicle by the distance image recognition is read, the distance data is filtered, and the variation of the distance data used in the follow-up running is smoothed.

  In this case, filtering at the time of normal distance image recognition and filtering of distance data at the time of tracking processing change the respective filtering characteristics, and dispersion of distance data at the time of normal distance image recognition and distance data at the time of tracking processing Absorbs the difference from variation. For example, distance data is filtered using a moving average of 5 times during normal distance image recognition, and distance data is filtered using a moving average of 10 times during tracking processing.

  Next, in step S34, the target deceleration and throttle opening of the host vehicle are calculated from the distance / speed data of the preceding vehicle, and in step S35, the target deceleration and the throttle opening are respectively limited, and the brake control device. 22, output a control command to the engine control device 21. These limiters are for coping with the recognition performance degradation. Limiting the target deceleration to limit the maximum value of the target deceleration and setting the slow brake, the effect of the emergency operation of the brake due to misrecognition. By limiting the throttle opening, the maximum acceleration value is limited to reduce the effect of acceleration due to misrecognition.

  Even in this case, during normal distance image recognition and tracking processing, each limiter characteristic is changed according to the difference in recognition performance. For example, during normal distance image recognition, the target deceleration is set to 0.2 GMAX. During the tracking process, the target deceleration is set to 0.1 GMAX.

  Next, the tracking process in steps S36 to S39 will be described. In step S36, the distance / speed data of the preceding vehicle obtained by the tracking process is read, the distance data is filtered, and the variation of the distance data used for the follow-up driving is smoothed (for example, filtering by 10 moving averages). .

  In the subsequent step S37, the target deceleration and throttle opening of the host vehicle are calculated from the distance / speed data of the preceding vehicle, and in step S38, the target deceleration and the throttle opening are respectively applied with limiters for tracking processing. As described above, since the recognition performance is relatively low during the tracking process, the effect of the limiter is made stronger than that during normal distance image recognition. For example, the target deceleration limiter is set to 0.1 GMAX.

  After limiting the target deceleration and throttle opening in step S38 and outputting control commands to the brake control device 22 and the engine control device 21, it is determined in step S39 that the tracking control is stopped. For example, the determination of the stop of the control is performed in consideration of the possibility of forgetting the driver for a certain period of time even if a warning is issued. abort. However, during deceleration, control is discontinued only when the deceleration is smaller than the threshold value.

  As described above, in this embodiment, the sensitivity balance between the pair of left and right images captured by the stereo camera 2 is lost, and even if the object is lost due to a situation in which normal stereo processing is difficult, one of the left and right images is lost. Since the object is tracked from the template image and switched to the stereo processing based on the tracking result, distance information can be continuously acquired, and even when applied to vehicle control, controllability is improved and reliability is improved. Can be improved.

  In addition, when an object is detected by normal stereo processing, an object image is stored and tracked on the original image, so the calculation load is small and a highly reliable system is avoided by avoiding mistracking. Can be built.

Configuration diagram of the preceding vehicle tracking control system Block diagram of object detection device Explanatory diagram schematically showing the luminance distribution of the left and right images during backlighting Explanatory diagram schematically showing the luminance distribution of the left and right images after luminance difference Illustration of tracking failure judgment Explanatory drawing showing the concept of left-right reversal in leading vehicle position correction Explanatory drawing showing the correction of the preceding vehicle position Explanatory drawing showing the correction result of the preceding vehicle position Flow chart of preceding vehicle recognition process Flow chart of preceding vehicle tracking process Lost determination process flowchart Flow chart of preceding vehicle following control process

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Vehicle 2 Stereo camera 5 Stereo image recognition apparatus 6 1st ranging part 7 2nd ranging part 8 Tracking process part 9 Distance selection part 10 Recognition process part

Claims (18)

A stereo camera that acquires a pair of images;
A first distance information calculating means for calculating a distance information of an object shown in the pair of images by searching a region in which the luminance values correspond to each other in the pair of images;
Calculating a luminance difference between adjacent pixels of a predetermined pixel in each of the pair of images and calculating a distance information of the object by searching for a region corresponding to the luminance difference between the pair of images; Distance information calculation means,
An object detection apparatus comprising: selection means for selecting distance information calculated by the second distance information calculation means when the object is lost by the first distance information calculation means. The second distance information calculation means is:
Finding a correlation between a pair of images and a template image in a predetermined region, and searching for a corresponding region between the pair of images in a region obtained by tracking processing that tracks the same object based on the correlation. The object detection apparatus according to claim 1. As the above template image,
The object detection apparatus according to claim 2, wherein an image having a larger correlation with the pair of images captured last time is selected.   4. The object detection apparatus according to claim 2, wherein the correlation is obtained by a residual successive test method.   4. The object detection apparatus according to claim 2, wherein the correlation is obtained from a cross-correlation coefficient.   2. The object detection apparatus according to claim 1, wherein when the object is not detected by the current frame processing within the set range near the object calculated by the previous frame processing, it is determined that the lost has occurred.   The object detection apparatus according to claim 2, wherein the reliability of the tracking process is determined using an evaluation parameter based on the correlation.   The object detection apparatus according to claim 2, wherein the tracking processing area is corrected using symmetry of the shape of the object.   9. The object detection apparatus according to claim 8, wherein a part of the candidate region of the object is extracted and inverted, and the inverted image is matched with the original image to identify the symmetrical center position.   The object detection apparatus according to claim 2, wherein the reliability of the tracking process is determined by a variance value of the position of the object between frames.   The object detection apparatus according to claim 2, wherein the reliability of the tracking process is determined by a variance value of the distance information of the object between frames.   3. The object detection apparatus according to claim 2, wherein the reliability of the tracking process is determined by the number of different data in the distance information at the same position.   The object detection apparatus according to claim 2, wherein the tracking process is performed on an image obtained by thinning image data.   The object detection apparatus according to claim 13, wherein the thinning rate of the image data is adjusted so that the number of pixels on the screen of the template image and the object is approximately the same.   The object detection apparatus according to claim 2, wherein the template image is updated according to the reliability of the distance information.   The object detection apparatus according to claim 2, wherein the reliability of the distance information is evaluated based on a width of the recognized object.   The distance information calculated by the first distance information calculating means and the distance information calculated by the second distance information calculating means are subjected to filtering processes having different characteristics from each other, and the difference in data variation is determined. The object detection apparatus according to claim 1, wherein the object detection apparatus absorbs the object.   The preceding vehicle follow-up control of the vehicle is executed based on the distance information, the control amount based on the distance information calculated by the first distance information calculating means, and the distance information calculated by the second distance information calculating means The object detection apparatus according to claim 1, wherein a limiter having different characteristics is applied to the control amount based on the difference according to the difference in recognition performance.

Images