WO2007124502A2 - Apparatus and method for object detection and tracking and roadway awareness using stereo cameras - Google Patents

Abstract

The present invention provides a collision avoidance apparatus and method employing stereo vision applications for adaptive vehicular control The stereo vision applications are comprised of a road detection function and a vehicle detection and tracking function. The road detection function makes use of three-dimensional point data, computed from stereo image data, to locate the road surface ahead of a host vehicle information gathered by the road detection function is used to guide the vehicle detection and tracking function, which provides lead motion data to a vehicular control system of the collision avoidance apparatus. Similar to the road detection function, stereo image data is used by the vehicle detection and tracking function to determine the depth of image scene features, thereby providing a robust means for identifying potential lead vehicles in a headway direction of the host vehicle.

Description

APPARATUS AND IVIE I MOD FOR

OBJEC T DETECTION AND TRACKING AND

ROADWAY AWARENESS USING STEREO CAMERAS

CROSS-REFERFJVCE TO RELATED APPLICATIONS

[00011 This application claims the benefit of United States Provisional Patent Application No 60/794.064. filed April 21, 2006, which is hereby incorporated by reference in its entirety

BACkGEOt ND OF THE INVENTION

Field of the Invention

J0002J The present invention relates generally to a stereopsis-based application More particular^, the present invention is directed to an apparatus and method for detecting and tracking objects within corresponding detected roadway features utilizing a stereo \ ision configuration, thereby enabling and improving applications in autonomous systems for vehicular control

Description of Related Art

|0003[ Developments in the field of computer vision and the availability of processors with increased processing speeds and multitasking capabilities have led to new and improved video-based technologies, but it has not been until only in the past few years that these developments have begun to appear in automotive applications, such as collision systems having adaptive control capabilities

|0004] One such system that is increasingly becoming av ailable in vehicular collision avoidance systems is adaptive cruise control (ACC), which has also been commonly referred to as active, automatic, autonomous and intelligent cruise control A(T is similar to conventional cruise control systems in that they assist in maintaining a host vehicle's pre-set cruise speed Howev er, unlike conventional ciuise control systems. ACC systems are configured to automatical h adjust (e g , by braking or increasing throttle) the ρre~set speed of a vehicle in response to the detection of target vehicles that may enter into and impede the path of travel of the host vehicle, thereby automatically iegulating the speed of the host vehicle absent voluntary actions by a

er

{OOOSj Modem ACC systems, as w-eli as other collision avoidance s) stems, have been known to employ the use of a headway sensor, a digital signal processor, longitudinal controller and othei similar sensor and controller components coupled togefhei for regulating separation distance between vehicles, vehicular speed and detection of drift or departing from a tracked lane of travel b> a host vehicle In ACC systems, for example, these sensors are typically housed in a front body portion of a host vehicle and are positioned to detect vehicles ahead of the host vehicle^'s line of travel Information collected by these sensors are processed and supplied to the controller, which then regulates the corresponding host vehicle^'s throttle and braking units accordingly If a lead vehicle in the same lane (i e . the target vehicle) slows dtnvn or enters the lane of the host vehicle, the system sends a signal to the engine or braking units of the host vehicle to decelerate proportionally to sensor readings Similarly, when the lane becomes clcai, or the lead vehicle entering the lane accelerates, the svstem will re-accelerate the host vehicle back to the desired pre-set cruise speed

10006] Monocular v ision, radar, monocular vision with radar, steieo vision with iadar and lasers have all been used in enabling the sensory requirements of collision avoidance systems Those commonly used in todays ACC systems, for example, employ radar- based or laser-based sensors Radar-based sensors bounce microwaves off the target vehicle w-hile laser-based sensors read light reflected off the body of the vehicle ahead Rader-based sensors provide a slight advantage over laser-based s> stems in that the\ are unaffected by unfavorable weather conditions, such as dense fog, snow and rain Radar- based sensors also perform better when used on target vehicles that are extremely dirt\ (which severely inhibits the ability to reflect light back to the laser-based sensor) However, a considerable disadvantage of radar-based sensors over laser -based sensors is thcii cost

|0007| Accordingly, there exists a need to provide a more cost-effective and reliable means for detecting and tracking objects for use in vehicular collision avoidance systems SUMMARY OF TRE l^iVEISTlOλ

|OOOSJ In viev, of the foregoing, it is an object of the present i m ention to provide a collision avoidance system enabled by input of a scene image received solely from a stereo configuration, therein eliminating costly sensors based components and providing a performance-enhanced approach to vehicle detection and tracking

[00091 It h another object of the present invention to prov ide a collision avoidance system enabled to receive a focused estimation of three dimensional structures and regions, as well as detailed scene context relating to the immediate headway of a vehicle

|0010| It is yet another object of the piesem in\ cntion to provide a collision avoidance system enabled with a means for rejecting spurious scene image data and increasing system reliability

|001 ϊ | In accordance with the aforementioned objectives of the present invention, prov ided and described herein is a collision av oidance apparatus employing novel stereo vision applications for adaptive vehicular control To satisfy the detection requirements foτ employing the collision avoidance apparatus of the present invention, two major stereo vision processing functions are prov ided, a road detection and awareness (^"RDA) algorithm and a vehicle detection and tracking (VDT) algorithm

J0012J The RD_'\ function makes me of three-dimensional point data, computed from received stereo image data, to locate the road surface ahead of the host vehicle Once a set of image feaiuies that lie on the road surface are found, the RDA is configured to utilize the detected features to identify lane boundary markers that demarcate the edges of the τoad lane in which the host vehicle is traveling Data computed h\ the RDA function is used to guide the VD f function, which provides lead vehicle motion data to the collision avoidance comto! system by using stereo image data to determine the depth of scene features ahead of the host \ehicle₅ as well as making associations between the three-dimensional features so that they form groups that represent potential lead vehicle detections in the road ahead of the host vehicle The VDT function is further configured to use information from the RDA function to compensate for pitching motion of the host vehicle, and to direct the search for potential detections to concentrate on the lane ahead of the host vehicle

|OOΪ3j A number of target confirmation functions, based upon iD features, image content and temporal consistency, are used with the RJ )Λ and VI)T functions to \ erif\ potential detections so that only those potential detections that correspond to real vehicles are used to provide information to the collision avoidance eomrol system For confunied detections, a spatio-temporal tracking function is used to estimate the forward and lateral motion of the lead vehicle with respect to the host vehicle, information of which is transmitted to the collision avoidance control s\ stern in order to ultimately regulate the host vehicle's speed

jøθi4] The most visible advantages of the present invention is that of performance and cost The integration of steieo vision sensing in the present invention provides a more reliable three-dimensional estimation of a target vehicle structure than a monocular s> stem alone, and can provide more scene context than a radar s\ stem alone Moreover, it is less costly than the combination of an ACC radar and a monocular vision system, and offers a more straightforward integration solution than a multi-sensor s\ stem

[0015] These and other objects are accomplished in accordance with the principles of the present invention, wherein the novelty of the present i m ention will become apparent from the following detailed description and appended claims

BRIEF DESCRIPTION OF THE DRAWINGS

{0016] The above and other objects and

of the present imention will become apparent upon consideration of the following detailed description, taken in conjunction with the accompanying illustrative drawings, which are provided for exemplary purposes to assist in an understanding of the present invention described herein, and in which like reference characters refer to like parts throughout, wherein

[0017] KIG. 1 illustrates an autonomous collision avoidance apparatus employ ing a stereo vision-based configuration in accordance with an embodiment of the present invention £001.8] FIG. 2 illustrates a generalized flow diagram of the essential processes performed for roadway and object detection and tracking in accordance with an embodiment of the present invention.

£001.9] FIGS. 3A-3I) illustrate functional block diagrams of the road awareness and ground detection function performed by an image processing portion of the autonomous collision avoidance apparatus in accordance with an embodiment of the present invention.

[0020J FIGS. 4A and 4B illustrate a left and right stereo image scene of lead vehicles with detected edges in accordance with an embodiment of the present invention

£0021] FIG. 5 illustrates an x-disparity histogram of the stereo image scene illustrated in FIGS. 4A and 4B in accordance with an embodiment of the present invention.

{0022] FlG. 6 illustrates detection gates in a lane ahead of a host vehicle in accordance with an embodiment of the present invention.

[0023] FIG. 7 illustrates a generalized flow diagram of the image pyramid construction and corresponding processing of stereo images in accordance with an embodiment of the present invention.

[0024] FKL 8 illustrates a generalized flow diagram of the X-disparity construction and peak extraction processing in accordance with an embodiment of the present invention

|0025] FlG. 9 illustrates a generalized flow diagram of the computational processing of regions of interest for initialization in accordance with an embodiment of the present invention

|0026] FlG. 10 illustrates a generalized flow diagram of the computational processing of coarse disparity in accordance with an embodiment of the present invention.

[0027] FIG. J l illustrates a generalized flow diagram for the selection of peak pairs likely to represent the outer edges of a vehicle in accordance with an embodiment of the present invention. {0028] FlG. J 2 illustrates a generalized flow diagram for false-positive removal processing in accordance with an embodiment of the present invention

{0029] FlG. 13 illustrates a generalized flow diagram for remov al processing of multiple detections of the same target vehicle in a single image frame in accordance with an embodiment of the present invention

{0030] FlG. 14 illustrates a generalized flow diagram of the temporal consistency checking process in accordance w ith an embodiment of the present invention

|003I ] FIG. 15 illmtiates a generalized flou diagram of the nighttime featuie extraction process in accordance with an embodiment of the present invention

|0032| FIG, 16 illustrates a generalized flow diagram of the nighttime blob pairing process in accordance with an embodiment of the present im eπtϊoπ

{0033] FlG. J 7 illustrates a generalized flow diagram of the nighttime depth estimation process in accordance with an embodiment of the present invention

{0034] FlG. 18 illustrates a generalized flow diagram of the nighttime detection ROI size testing process in accordance with an embodiment of the present invention

{0035] FIG. 1*> illustrates a generalized tlo diagram of the nighttime final verification piocess in accordance with an embodiment of the present invention

DETAILED DESCRIPTION Ob THE PREFERRED EMBODIMENTS {0036] The present invention is directed towards an apparatus and method for enabling adaptiv e v ehicular systems employing solely the use of stereo cameras for detection and tracking applications For purposes of clarity, and not bv wav of limitation, illustrative views of the present invention are described herein with references being made to the aforementioned drawing figures The following detailed description in conjunction vuih the referenced figures are provided to enable and assist in the understanding of the present invention and. as such, the drawing figures should not he deemed to be exhaustive of all the various modifications obvious to one skilled in the art Such obvious modifications are deemed to be well w ithin the spim and scope of the present invention

|0037] An autonomous collision avoidance apparatus 10 is illustrated in FIG. 1 In accordance with a pi ef erred embodiment of the present invention, an adaptive cruise controi (ACC) s\stem is depicted in connection with the illustrated autonomous collision avoidance apparatus IO to better exemplify the detection and tracking applications of the present invention The detection and tracking applications of the stereo vision configurations employed in the present invention can be modified accordingly bv one skilled in the art to enable additional active and passive collision avoidance or collision mitigation systems

J003S] In FIG. 1, autonomous collision avoidance apparatus Ϊ0 is composed of an AVV s>stem controller 12, which is coupled to a use? interface 14 having catise control snitches 16 for activating and deactivating ciiiiso control features controlled by 4.CC s> stem controller 12 Switches 16 may be integrated, for example, along an unobtrusive portion of a vehicle^'s steering wheel (not shown), thereby providing a driver with ease of access for empio) ing or disabling A(¹C features of the s elύcle

{0039] ACC s> stem controller 12 is further coupled to an image processing unit 18 Image processing unit 18 is comprised of a memory and processor components section 20 Memory and processor components section 20 may include an image preprocessor, a central processing unit {CPU}, a analog-to-digital converter (ADC), read-on!) memory (ROM), random access memory (RAM) video random access memory (VRAM), other necessary image processing and support circuitry or any combination thereof For example, an image preprocessor may he integrated into image piocessing unit 18 as a single chip video processor, such as Acadia^{ι xt} I developed by Pyramid Vision Technologies, a subsidiary of Sarnoff Corporation, in Princeton,

Jerse)

J0040] The CPt' included in memory and processor components section 20 of image processing unit IS may be comprised of any number of presently available high speed microcontrollers or microprocessor and supported, according!) , by support circuits that are generally well known in the art Such support circuits ma) include, for example. cache or any other suitable type of tentpoiary stoiage area where frequently accessed data can be stored for quick access, power supplies, clock circuits, input-output interfaces and the like

J0041 J The aforementioned various types of memory, or combinations thereof, that may be incorporated in memory and processor components section 20 of image processing unit IS may be used foi sioiing certain software routines, collections of image data, databases of information and image processing software for processing collected image data and facilitating the present invention Although the present invention is described herein primarily in the context of a series of method steps, these methods may similarly be performed by hardware, software or some suitable combination of both hardware and software

J0042| As illustrated in FIG. I, memory and processor components section 20 of image processing unit 18 may be coupled to stereo vision specific modules The stereo \ ision modules are a road detection and awareness module 22 and a vehicular detection and tracking module 24. each being employed at various' stages of processing of the collected stereo image data in order to facilitate the present invention These modules, although illustrated as separate component modules, ma> be similarly be made available by accessing their related algorithms from memory components in memory and piocessor section 20 The specific computational processes employed in connection with these two modules aie described in further detail herein

10043] When the cruise control features of a host vehicle (i e , the trailing vehicle equipped with ACC svstein controller 12) are activated, autonomous collision avoidance apparatus 10 begins receiving input image data from stereo camera configuration 26 Camera configuration 26 is comprised of a pair of stereo cameras, a first camera unit 26a and a second camera unit 26b Camera units 26a and 26b are prefeiably in a side-b\ -side configuration, having a known relation to one another and securely affixed in a suitable headway position of the host vehicle such that they can produce a stereo image of a scene J0044J Camera units 26a and 26b, configured in a stereo pair configuration as shown in FlG. 1, are capable of providing vast three-dimensional information about a target object in the headway direction of the host vehicle, such as the presence of a target object, its corresponding distance from the host vehicle and its speed of travel . By combining image data received from both stereo camera units 26a and 26t>, depth information for a particular scene that is typically lacking in a monocular image may be achieved by revealing the three-dimensional structure of the scene. This configuration provides a robust solution for detecting and tracking physical properties associated with moving objects.

|0045| ACC system controller 12 may also be further coupled to a speed sensor device 28 for tracking the speed of travel associated with the host vehicle. Speed sensor device 28 may be any speed sensor device or combination of components known to those skilled in the art for reliably tracking a range of speeds of travel typically associated with a moving vehicle. For example, sensors may be employed at the wheels of a moving vehicle and corresponding captured data reported back to ACX^' system controller 12 for executing applicable adjustments in speed as deemed necessary by ACC system controller 12.

10046] Similar to the memory and processing components of image processing unit 18, ACC system controller 12 may generally be comprised of an input/output (I/O) interface, a microprocessor or a centra! processing unit (CPU) for processing data received from the I/O interface and executing commands in response to said processed data, and a plurality of memory type components for regulating and accessing stored data necessary for achieving the present invention, ACC system controller 12 is configured to receive input from image processing unit 18 in order to translate the detected image data and communicate, accordingly, with various vehicle control units, such as, engine 30 having an engine control unit (ECU) 32 that is likely configured to communicate correspondingly with a braking control unit 34, a transmission control unit 36 and a throttle control unit 38 of the host vehicle

|0047| Image data related to the leading vehicle and its corresponding lane of travel is received from camera units 26a and 26b and processed using the detection and tracking modules 22 and 24 of image processing unit 18 The processed image data is then shared with ACC s> stera controller 12 to determine the need for adjusting a driver^'s preset cruise speed of the host vehicle If ACC system controller 12 determines thai an adjustment in the host vehicle^'s speed is necessary, the appropriate control signals are transmitted to. for example. ECU 32 ECU 32 may be configured to communicate with all the essential control units of the host \ehicle In an alternative embodiment, which is typical in modern vehicles, a plurality of F.€Us may be employed for controlling one or more of the Subsystems and/or control units of a vehicle l*or example, KCU 32 may be configured to communicate with and control the operation of throttle control unit 38 for regulating the fuel and air intake flow into engine 30 Similar!) IICV 32 may be configured to communicate with and control the operation of braking control unit 34 so that a predetermined safe distance between vehicles may be achieved automatically by ACC system controller 12

fOΘ4$| Λ general overview of the processing stages implemented in connection with autonomous collision avoidance appaiatυs 10 is illmtiated in the flow diagram of FIG. 2. with more elaborate descriptions of key processes further being provided herein When the cruise eontiol fealuie of a host vehicle is activated b> a driver, for example, by setting a desired cruise speed using caiise control sw itches 16 provided on user interface 14, ACC s\ stem controller 12 and image processing unit IS are enabled, thereby initialing the detection and tracking process 50 illustrated in FtG. 2 Process 50 begins by providing stereo image data collected bv camera units 26a and 26b as input to image processing unit 18 Stereo image data received by image processing unit 18 is subjected to two major image computational functions to satisfy the sensing requirements necessary for robust functionalit) of autonomous collision avoidance apparatus 10 These image computational functions are accomplished primarily in connection with those processes' describes in connection with modules 22 and 24 of image processing unit 18

{0049] Stereo image data received by image processing unit IS. at step 52, is subject initially to a road awareness and ground detection algorithm, which is implemented at step 54 Implementation of this algorithm is used to identify, at step 56, lane boundary markers for demarcating edges of a road on which the host vehicle is traveling While processing depictions of the road surface being tiaversed

the host v ehicle, a vehicle detection and tracking algorithm is implemented at step 58 to identify, at step 60, potential leading vehicle detections on the depicted road surface ahead of the host vehicle Potential detections arc verified, at step 62, using target confirmation algorithms Thereafter, at step 64, forward and lateral motion of the leading vehicle with respect to the host vehicle is estimated and communicated, at step 66, to AVC s>siera controller 12 of FIG, I in order to regulate, at step 68, the host vehicle's speed dynamics

J0O5Θ] Road Detection and Awareness

[0051 ] Road detection and awareness module 22 of image processing unit 18 opeia.es essentially as a ground point detector 110. as illustrated m FϊG> 3Λ, implementing the road detection and awareness (RDA) algorithm represented generally by module 22 illustrated in FtG, 1 The RDA algorithm makes use of three-dimensional point data computed by image processing unit 18 using stereo imago data received from camera units 26a and 26b. to locate the road surface in the headway direction of the host v ehicle Once a set image features' that lie on the road surface are found, the RD \ algorithm goes on to identify lane boundary markers that demarcate the edges of the road lane in which the host vehicle is traveling within Determinations made by the RL) Λ algorithm are used to guide the vehicle detection and tracking algorithm represented geneialiy by module 24 illustrated m FϊG. 1 (to be described in further detail herein) Road awareness data additionally may be used to compensate for pitching motion of the host vehicle and to judge when either the host or tracked lead vehicle changes into an adjacent lane or is going around a curs e along the detected road surface

J0052] Giouπd point detector HO depicted in FIG. 3A receives input image data 100 from stereo camera units 26a and 26b, as illustrated in FlG, 1, and provides for matched features 102, a ground model 104 and a lane model J 06 Matched features J 02 are a set of features that have been matched in the left and right images provided, respective! j . by- each of camera units 26a and 26b, to obtain three-dtmensional V, Y and 7 coordinates This three-dimensional point cloud is further anah zed to determine which points He on the ground, to prov ide ground model J 04, and which features are likelv to be lane markers, to provide lane model J 06 The lane center can be determined upon identification of those features that represent Sane markers In. an alternative embodiment, the three-dimensional points could be organized into vertical and horizontal clusters before locating the road surface, and then a road model could be fit through the appropriate horizontal clusters.

j0053| Obtaining a set of features is a separate task from analyzing the features to find the required road and lane markings. FIG. 3B depicts the separate tasks executed by ground point detector 110 of FlG. 3A. which provides for a computational block 120 for computing feature points and a computational block 130 for selecting corresponding ground points ^'These two separate tasks are illustrated by the functions of computational blocks 120 and 130, where computation of matched feature points 102 are processed from input 100 of the left and right stereo image data received by computational block J 20 and where selection of ground points is further processed in computational block 130 using matched feature points J 02 output by computational block J 20 The selection of ground points by computational block 130 contributes to yielding ground model 104, lane model 106 and annotated features 102'. which supplement the initial output matched features Ϊ02

j 00541 To compute feature points, computational block 120 processes the left and right images received from stereo camera units 26a and 26b one line at a time. On each line processed, intervals (e.g., on the order of 8 pixels wide) with significant intensity variation are found and then matched in the left and right lines to obtain the disparity for detected three-dimensional features.

|0055] FlG. 3C illustrates the necessary functions employed by computational block 120 for obtaining the three-dimensional point features In feature detection block 122 a derivative image is obtained, and regions with high gradients are identified as candidates for feature matching However, a problem experienced with the derivative image test is its locality For example, an intensity change of 70 over an 8 pixel width (i e., 100 1 10 120 130 140 150 160 170} could result in the same derivative score as a random variation that includes a 10-pixel intensity' change (e.g. 1 10 114 104 1 1 1 107 100 107 1 14).. even though the first pattern describes a more obvious intensity variation. Therefore, when searching for features in the present invention that are likely to result in a reliable match. particular criteria is employ ed to give this first pattern a much higher score than that of the second pattern B\ computing the difference in maximum and minimum intensities for a set of pixels strong features can be isolated, howev er, this may also iesuh in sensitivity to spurious intensity variations The RDA algorithm employed in the present invention is configured to compute the difference of second and ne\t-tυ-last values in a sorted list of intensities, therein providing a measure that vields more robust results while minimising spurious intensity variations

J0O56] Line features 123 are passed to a feature matching block 124. wherein the features being matched are actually intervals

ing the occurrence of a significant intensity variation An interval begins when the next 8 pixels have a significant intensity variation and it ends when the next 8 pixels are relatively constant in their intensity variation Ideally, corresponding features should have intensity profiles that match when starting points are aligned In practice, sum of absolute difference (SΛD) scores are computed for the nominal alignment ±3 pixels and the true alignment to sub-pixel precision is estimated by fitting a parabola through the lowest SAD seoies

J0057J \ote that matching features may not have the same width For example, consider a situation where rising and falling edges are grouped as one feature in one image, while in the other image the top of the profile is wide enough and flat enough that rising and failing edges are separate features To address such a situation, the RDA algorithm selects the larger of the two widths, at feature matching block 124, for the SAD interval used to align the intervals in the current example, the single wide profile will match the feature for the rising edge and include the falling edge as well because the width of the single wide feature will he used in the match The separate falling edge will not match anything and, therefore, will be discarded

[0058] Since response to light of stereo camera units 26a and 26b may differ (e g due to different aperture settings), it mav be necessary to make the matching scheme of the RDA algorithm employed at feature matchtng block 124 insensitive to differences in the overall intensity gain and offset In particular, an intensity shift is permitted between matching patterns by computing the algebraic average difference between the two intensity patterns, and then computing the sura of the absolute value of the difference in intensity minus the average difference

{0059] When looking for candidate matching features, a search range is computed to rule out some candidates because with reasonable camera geometry and orientation the proposed disparity could never occur. Detected line features 123 determined by feature detection block 122 and matched features determined by feature matching block 124 are passed to a disparity computation block 126 to be used for computing the disparity for corresponding points The disparity for corresponding points is defined as δ - x_R - x_; , where in the absence of image shifts or affme transformations the range is given by

Z ^~ - ^...-^...>^.. . Since the RDA algorithm has to deal with "warped" images, it also needs to δ take into account the possibility of an affme shift and/or an Acadia preshift These shifts affect the right image and aforementioned search range and, as such, may be defined in an ASP file that may be stored, for example, in memory and processor components section 20 of image processing unit 18.

[00601 Feature points output by computational block 120 may then be provided as input to computational block 130 of ground point detector 110 for selecting the corresponding ground points. This procedure includes both a search for the ground plane and a search for the lane. The ground plane is needed to supply outputs that provide the row of the image on which the ground plane will project for a particular depth. The search for the lane provides information that determines for a given row of the image, the column where the left lane boundary, right lane boundary and lane center will be located, FIG- 3D illustrates the necessary functions employed by computational block 150 for findinii the ground and lane markers

1006 J ] The ground plane search assumes that we have a good estimate for the height at which stereo camera units 26a and 26b are fixed. What is not known is the pitch angle between fixed stereo camera units 26a and 26b and the road, which can vary by several degrees. In one embodiment, the ground plane scheme of the RDA algorithm employed by computational block 130 may provide a histogram of the slopes that exist between the point Y (the camera height), Z=O and feature points K, Z. In an alternative embodiment. the RDA algorithm employed by computational block 130 may provide a ground plane scheme that forms a two-dimensional histogram of y, d values (image y and disparity') to use in searching for the ground. In either approach, the challenge is finding the "onset" of ground points, rejecting spurious below-ground points and also rejecting above-ground points. Yet, there may be more above-ground points than ground points and, therefore, one cannot simply look for a peak in the histograms. However, if vertical structures could be eliminated, which are typically identifiable by the clustering of points, then the remaining above-ground horizontal features (e.g., the top of a highway road barrier) may be rare enough that the first peak in a suitable histogram can be selected with confidence of detecting the ground.

(0062) A three-dimensional point cloud will likely have spurious below-ground points as well as legitimate above-ground points from cars and road-side objects. For points on the ground, we expect y to be a linear function of disparity In a typical plot of y versus disparity, the slope of the line is determined by the camera height. The intercept at zero disparity gives the location of the horizon in the image, and is related to the slope of the

* a_>r' ound relative to the camera axis.

10063 [ For a particular disparity value (i.e., a particular range), a y value that is likely to be near the ground must be found. In one embodiment, the RDA algorithm starts with y equal to the image height (i.e., start from the bottom of the image) and searches through the histogram until a bin with a count exceeding some threshold is found. The set of y, d values obtained in this way may then be analyzed by making a histogram of horizon values That is, for each y, d value, the corresponding y_^horizon is computed, assuming that the ground plane has no roll, the height of stereo camera units 26a and 26t> are known and the plane passes through the y, d point The y horizon histogram for a good set of y, d points will exhibit a peak at the location of the horizon, and will have relatively few horizon predictions from below-ground points. The difficulty, however, is that the threshold required to obtain a good set of y, d points is not known. Therefore, the RDA algorithm starts with a zero threshold and increases it until a good set of points are obtained, presumably because spurious ground points have been eliminated. An alternative to searching for yjhorizon may be to employ a search for the slope of the ground plane relative to the camera axis.

{0064] For the present analysis it is assumed that in camera coordinates the ground plane is described by

The usual coordinate system is applied where positive Y is down, if the ground pitches up, then the slope (s) is positive and Y becomes increasingly more negative as one moves away from stereo camera units 26a and 26b.

[0065] Alternatively, the ground plane can be described by expressing disparity as a function of the image coordinate y, thereby givirm the camera projection

and the stereo equation gives

M ^κ - Z For points in the ground plane, we have

s^ ^rbi^ iN cxusa*^ ^ u"\>

v ^nκ*m_*£ sh wnh ?k* si torro <>; ^- ?\.-mr

f^ ksn^ ϊ>f sht^c oooiH^K^u^v t? -; cϋϊ^son

s^eccn^er

jt>066| Thus, once reliable ground points have been identified, the ground plane pauimetefs are refined, at a ground plane fitting block 132, by fitting a Sine oi plane to the ground points The processes executed in ground plane fitting block 132 produces coefficients dθ, d !, d2 in the expression d(x,y) - dO (x - ex) f di y ^! ύ2 for disparity as a function of image x,y for points in the ground In an effort to get a more accurate horizon estimate, distant ground points are weighted more heavily than ground points that are near Thereafter vertical features associated uith the ground plane fitted points are removed at a sertieal feature removal block IM This procedure ielies on knowing which points are above ground and seeks to remove the foot-print, on the ground, of an above-ground feature In an alternative embodiment, if the RDA algorithm weie modified so that it only relied on finding sets of points with a large Y extent, then it could precede the ground detection

£0067) After vertical feaimes are remov ed, the RD \ algorithm proceeds to a lane searching block 136 Lane detection relies on searching for ground features that appear to line up when viewed from the proper Y position and direction in the .V-Z plane Detecting lane markers on a curve is also a critical, additional requirement and, therefore, autonomous collision avoidance apparatus 10 may also employ the use of an external sensor (not shown) The external sensor maj be used to estimate the vehicle yaw rate and, hence, the curvature of the lane

|0068| The search for lane markers tries a discrete set of slopes For each slope, lane searching block Ϊ36 computes the .V intercept at Z 0 for a line through the point with the specified slope A histogram of A^' intercepts is constructed for each trial slope, the result being a two-dimensional position-slope histogram The histogram is smoothed in A^', to mitigate effects due to the alignment of bin boundaries with the actual lane locations The overall peak location gives the position and slope of the lane hypothesis that best represents the data This peak represents one lane Histogram cells near this first peak are then ignored, and a second peak is found coiresponding to the second lane To improve

b\ distance

J0069] As described above, the RDA algorithm analyzes the lane points in \, y image coordinates In an alternative embodiment the RDA algorithm, which includes a curvature correction, may build a lane model in Λ, Z coordinates This ma) be done by clustering points belonging to each lane into nodes T he points are binned in Z, and regions of high density are identified as clusters Clustered points arc removed from the set and the process is repeated until all the points

been assigned to a cluster The nodes are used to construct polv-Hncs In addition, some curve-fitting is used to allow extrapolation off the ends of the lines, and to reduce the effect of kinks that occur when non-lane points {such as shadows or tires) happen to be included in the set of lane points.

|0070] When based on image coordinates x, y, the slope is computed between successive nodes and fitted as a function of y\ giving node coordinates (x_k , y_!: ) that are fit to a function of the form s(y) ^~ s_Λ + s-, v where s - dxjdy . At least two nodes are required, but if there are exactly two nodes, then slope is set as Λ\ - 0 and computed as follows;

If there are three nodes, then the curvature estimate is assumed to be unreliable and only the slope is estimated. In this case, the slope is estimated by fitting a line:

The least squares equations are

If the calculation is done using Ay ~ y ^••• J , then the matrix is diagonal and

Note that disregarding the curvature in these particular cases may be a mistake. Whether or not to estimate curvature with only three points will depend on the means employed for generating the nodes, as well as the noise associated with each of the nodes. If there x — x. are more than three points, then slopes *,. - -^:--*-"^!- -^' ------ are associated with y values

S'_k ^~ -^"--^:--"^!- -^: -----^:- - . These (s, y) values are weighted to favor observations near the median

slope: s* . ^■■

where r is the "kink slope threshold." The usual least squares equations are solved to obtain the slopes .¥,., and Λ_>

J0071 J In the case of a curved road, two additional steps may be taken in the search for Sane boundaries. The fust of these extra steps depends on whether or not the vehicle speed and yaw-rate are available If so, then the road curvature can be estimated by dividing the velocity by the yaw-rate. The estimated curvature may then be used during the construction of the Xs histogram, where the location of individual X, Z points are corrected by subtracting away the X offset due to curvature prior to entering the point into the histogram. This allows the slope and offset of lane markers to be estimated in the presence of the road curvature and, hence, the clustering of image features into nodes for each Sane marker. The other step may utilize, in place of a least-squares fit of a straight line to the lane marker node points, a quadratic equation of the form X(Z) === XO -^;- sX -^;- C0/2*Z*Z, where CX) is the road curvature Both individual quadratic fits to the left and right markers, and a fit where slope s and curvature CO are shared by both left and right markers, are valid alternatives for this procedure.

J0072J As depicted in FlG. 3B. the present invention decomposes the problem of road detection and awareness into two major tasks, namely computation of feature points and selection of ground points. However, there is no explicit incorporation of a task for the aggregation of individual three-dimensional points into groups of nearby points representing objects, such as lane markers or cars, in the world Thus, in an alternative embodiment, the RDA algorithm may be further configured to employ a scheme that bins three-dimensional points on a 0 5m by 0 5m grid or, alternatively, a Im by Im grid, covering a region Z ^~ 0 to 100 and the corresponding X width range (something on the order of 10,000 cells), thereby enabling the algorithm to find "islands" where adjacent cells in a particular direction (e.g., N-E-S-W or NE-SE-SW-NW) are occupied.

{0073] islands determined to exceed 5m in X or Z (e g., a continuous lane marker that generates a large cluster) could be subdivided. Then, for each duster, a histogram of y values could be generated with a resolution of about 0 2m, wherein the histogram is subsequently tittered to mitigate effects due to how bins may be hned up with data and to locate a well-defined peak Points can be added in the peak to a list of "horizontal" points to use in finding the ground Once the ground is found, the set of horizontal points could be further refined to get ground points for use in finding lanes Cluster labels raa> then be used to keep points grouped together and aggregate properties of the clusters, instead of individual points, may be used In this way, the clusters nia\ be used to help address potential near-far weighting problems A near Sane marker and a far lane marker will both be represented bv one cluster, even though the near cluster is formed from many more points

J0074J Vehicle Detection and Tracking

$00751 Vehicle detection and tracking module 24 of image processing unit 18 implements the vehicle detection and tracking (VDT) algorithm, which pro\ idcs lead \ehicle motion data to AVC system controller 12 The VDF algorithm uses stereo image data received from camera units 26a and 26b to determine the depth of scene features ahead of the host vehicle, and then makes associations between these three-dimensional features so that groups are formed representing potential lead vehicle detections on the toad ahead of the host vehicle As previously described, the VDT algorithm uses data processed by the RDA algorithm of module 22 to compensate for the pitching motion of the host \ chicle, and to direct the search for potential detections to concentrate on the lane ahead of the host vehicle

{0076] A number of target confirmation algorithms are used in connection with the VDT algorithm The implementation of these algorithms are based upon three- dimensional features, image content and temporal

potential detections, so that onh those potential detections that correspond to real

are used to information to ACC system controller 12 For a confirmed detection in the same lane as the host vehicle, which is determined to be the closest to the host vchtcle, a spatio-temporal tracking algorithm is used to estimate the forward and lateral motion of the lead vehicle with respect to the host, this information is then also transmitted to ACC controller 12 in order to regulate the host vehicle's speed T hese aforementioned functional schemes employed by the VDT algorithm are described in connection with FIGS. 4-19, wherein FIGS 4-14 are referenced for describing daytime execution of the VDT algorithm and FIGS. 15-19 are referenced for describing nighttime execution of the VDT algorithm

j0077| For vehicle detection and tracking enabled during daylight hours, entire lead vehicle outlines are easily visible in the image received from stereo camera units 26a and 26b The basic principle behind the daytime scheme of the VDT algorithm is that for a given region of space ahead of the host vehicle, vertical image features that could represent the outside edges of a vehicle are searched for. Such identifiable edge features are illustrated, for example, in FlGvS. 4A and 4B, which correspond, respectively, to the left and right image views of stereo camera units 26a and 26b As illustrated, edge features 402a - 408b are identified, respectively, along the outside edges of lead vehicles 402, 406 and 4OS

{0078] Edge features associated with lead vehicles are searched for within a region of interest (ROl) 401 predefined, for example, by the ground and lane models calculated in the previously described RDA algorithm. Edge features may be associated with the edges of taitlights, wheels and license plates of a lead vehicle within ROl 401 For example, the nearest lead vehicle 402 to the host vehicle is detected within ROl 401 and has edge features identified along a left taillight 402a, a right tailHght 402b, a left rear tire 402c, a right rear tire 402d, a left edue of the license plate 402e and a risjht edae of the license plate 402f.

|0079] Once these vertical image features are identified, a stereo matching procedure may be performed to estimate the depth to the vertical image features. With the three- dimensional location of each of the vertical image features, a two-dimensional histogram 500, as illustrated in FIG. 5, of x (i e._; the corresponding column of the image where the feature is located) vs. D (the disparity of the feature) is constructed in order to cluster vertical features together.

[0080] The two peaks from histogram 500 that are most likely to be the outer edges of lead vehicle 402 in the imaiie scene are then selected and a bounding box is drawn in the image scene around those features that are associated with the two peaks. The image contents of the bounded box are then subjected to a number of tests to confirm that the detected features actually belong to a vehicle. If confirmed as a vehicle, the output of the VDT algorithm will be the location (in X and Z) and the size (in width and height) of the detected lead vehicle.

[0081 ] Referring to FlG. 6, during the initialization phase the VDT algorithm being employed in a host vehicle 600 searches for the previously described vehicle edge features in several regions of space, each region of space being referred to as a detection gate 602. Each of detection gates 602 specifies an ROl 603 and a depth range 604 in the image over which the search for lead vehicle edge features is performed. These detection gates are spread out along the length of the lane ahead of host vehicle 600, which is guided by the RDA algorithm, and each gate 602 slightly overlaps, as depicted by overlap regions 606, its adjacent neighbor gate

[0082] Once the potential detection is confirmed to be a lead vehicle, then the tracking scheme of the VDT algorithm is deployed for tracking the lead vehicle The tracking scheme of the VDT algorithm is similar to the detection scheme of the VDT algorithm, however, instead of the ROI being set by a fixed detection gate (in the image and in space), the location of the ROI at one instant (i.e., a current stereo pair image) is determined by the location of the ROI that was seen in the previous instant (i.e.. the previous stereo pair image). During initialization the tracker will give the location and size of the lead vehicle, wherein the data is fed into a Kalman filter for estimating the range, range rate and acceleration of the lead vehicle with respect to the host vehicle, as well as the lateral position and lateral velocity of the lead vehicle with respect to the host vehicle.

[0083] To assist in a better understanding of the algorithms employed by the present invention, a more detailed analysis of the computation engaged in by the VDI algorithm during a daytime search of an image scene having a lead vehicle, said scheme having already been overvlewed in the preceding paragraphs, is described with reference to the process flows illustrated in FIGS. 7- J 4. J00S4J Referring to KIG. 7, image pyramid construction and processing is engaged in by the VDT algorithm for stereo image data received from stereo camera configuration 26. in uhich camera unit 264 provides, at step 702a. a left image vieu and camera unit 26b provides, at step 702b. a right image view The two inputs are regions of interest (ROIs) from rectified left and right images recekcd. respectively, from camera units_* 26a and 26b of stereo pair configuration 26 Thereafter, image pyramid construction of the received left and right image data is engaged, wherein left image and right image py ramids are constructed, respectively, at steps 704a and 704b, by subsampUng only along the x -direction of their respective images A 5X5 Gaussian filter is then applied, respective!) , ai steps 706a and 706b, at ever) !e\ el of the constructed left and right image pyramids, thereby filtering and forming a Gaussian ps ramid Thereafter, each level of the left and right Gaussian pyramids arc subject to a 3X3 vertical Sobel application, respectively, at steps ⁷OSa and ⁷OSb. to compute Sobel edge images The Sobel edge images are the candidate image features used for executing the \ ehicie detection scheme of the VDT algorithm A 3X3 horizontal Sobel is also applied, at step 707. at level 0 of the left Gaussian pyiainid geneiatcd ai previous' step 706a to prov ide a left horizontal Sobe! image, at step 709 The horizontal edge image produced for the left image helps provide for a higher resolution image

[0085] The left and right vertical Sobe! pyramids are output.

at steps 710a and 7iø}>, and then subject to an adaptive thresholding scheme, wherein the most significant edges from the left and right image ROIs are selected This computation is repeated for every level of the left and right Sobel pyramids The adaptive thresholding scheme is initiated at steps 712a and 712b. computing the intensity histogram for the edge pixels associated, respective!} , with the left and right vertical Sobel pyramids provided at steps 710a and 710b The intensity histogram computed, at step 712a, for the left vertical Sobe! pyramid is further subjected to a computation, at step 713 to determine the 90^th percentile intensity to be used to threshold the vertical Sobel image The computed 90^th percentile intensity is then used to compete, at step 714a, the total number of edge pixels m the thresholded left vertical Sobel image (represented by a variable "m") and, similar!) , at step 714b, the tola! number of edge pixels in the right vertical Sobel image represented b> a \ariable "n^" ϊ is computed from the intensit\ histogram for the edge pixels computed, at step 712b, from the right vertical Søbei pyramid The number of edge pixels in the left vertical Sobel image (m) and the right vertical Sobel image (n) may then be provided to computation step 716, wherein a percentile for the right Sobel image is computed as: 100*f i -Cnv^'n)] and used to threshold the right vertical Sobel image. Computation of the percentile intensities and their application in thresholding the vertical Sobel images at steps 713 and 716 \ ield, respectively, at steps ?18a and 718b, the left edge pyramid and right edge pyramid, which are the inosi significant edges Horn the left and right image ROIs previously discussed The edge pyramids are binary images, where white corresponds to the significant edge pixels and black corresponds to all other pixels

|0086| Once the left and right edge pyramids have been constructed, respectively, at steps 718a and 71Bh. the X-disparity scheme of FlG, B is initiated to construct an X- disparity histogram and extract relevant peaks For the ROIs, a SAD-based stereo matching scheme is executed, at step 720, between the left and right Sobel pyramids for each pixel that is colored white in the left edge pyramid, yielding an estimated refined disparity for every edge pixel in the left edge image Thereafter, at step 722, an X- disparity histogram is constructed using the image X coordinates and the refined disparity values computed at step 720 for the edge pixels. Peaks are then located, at step 724, in the X-disparity histogram using a mean shift algorithm, thereby yielding, at step 726, a list of peaks (location, and the pixels that contribute to those peaks) in the X-disparity histogram for every pyramid level.

jO087| In FIG. 9. the ROl for initialization is computed. The detection gate locations are computed based upon the required spacing in depth ahead of the host vehicle, and the road plane and lane boundary information provided by the RDA algorithm This computation begins by retrieving, at steps 902a-902e, the ground plane, the lane marker positions and disparity interv al between gates. At step 904, the location of the initializer gates in the disparity domain are determined using the starting and the ending gate positions and the disparity interval between the gates. For some gates, the disparity interval will need to be reduced to ensure that there is at least one gate at every pyramid leseS The disparity interval between the nvo gates serves as the gate disparity search range for the closer of the two gates Then, at step 906, the depth for every gate is computed using the gate disparity and the gate inspection ROf is computed using the ground plane and the lane marker positions The gate inspection ROI is the projection (in the image) of the region of height and depth equal to the gale depth King between the two Sail c markers and the ground These computations, at step 906, yield gate disparity search ranges, at step 908, and gate inspection ROIs, at step 910

[0088] In FICJ. 10, coarse disparity is computed, wherein foi each gate inspection ROI, the integral disparity value that best aligns the majority of the extracted image edge points are computed from the left and right image pyramids Previously calculated and constructed information with respect to the pyramid

the inspection ROi the left edge image at the pytamid lc\ el and the light edge image at the pyramid level are all retrieved, respective!} , at steps IΘO2a-IOΘ2d, to execute the coarse disparity computation, the process being repeated for even.' gate inspection ROl The computation is initiated at step 1004, v\ herein the integral image based SΛD sums foτ the whole inspection ROI (represented by the illustrated hatched box 1004a) are computed for all shifts using the binary edge images Then, at step 1006, a horizontal sub-ROl (represented by the illustrated dotted box 1006a) of the inspection ROI is selected having equal borders ob opposing sides The disparity foi the sub-ROΪ ma\ then be computed, at step 1008, using the integral image based SAD sums computed for the whole ROI The border is then increased, at step 1009, to consider a sraaHeϊ sub-ROΪ and loops back to step J 006 The median disparity is then computed, at step JOΪO, from all disparities computed for sub-ROIs. yielding the target coarse disparity at step 1012

[0089] Having generated a list of peaks from the X -disparity histogram construction, together with the integer (coarse) disparity estimate for a ghen detection gate, the pair of peaks that are most likely to represent the outer edges of a vehicle may now be selected in accordance with the process illustrated in FtG. 11 Once the appropriate peak pair is determined, the consensus set of pixels that belong to those peaks and the region between the two peaLs. the detection ROl, which is the bounding box in the image that surrounds the consensus points, can be calculated In addition, a sub-pixel disparity estimate foi the consensus points may also be calculated once the appiopriate peak paii has been determined J009ΘJ The aforementioned peak selection process is repeated few every gate inspection ROI and is initiated b\ retrieving, at steps 1102a-U02e, respectively, the calculated and constructed information pertaining to the coarse disparity, the inspection ROI, the pyramid level the iist of peaks in the X-disparity histogram at the pyramid level and the X-disρarity histogram at the pyramid level With this, the top four (4) peaks, from the list of input peaks, that He within the inspection RUI are found, at step 1104 Then, at step J l 06. peak pairs that satisfy a vehicle width constraint, in pixels estimated at coarse disparity, are found, wherein the peak pair for which the sura of the strengths of the two peaks is maximum, is found at step 1108 The list of points, pushed into the X- disparity histogram which lie tn the rectangular region defined by the peak pail locations are then found, at step 1110 These points form the consensus set at step 1111 The bounding box (in the image) of the points in the consensus set is then found, at step 1 112, which are used to form the detection RUl, at step 1113 I hen, at step 1114, a histogram of the disparity values of the points tn the consensus set me created, the highest peak in the histogram is found and an average of the disparity values which belong to the peak aie taken, yielding the iefined taiget disparity, at step 1 115

J0091J Referring now to FIG. J 2, a single gate false-positive (KP) removal scheme is described For am detection from a gate, as determined by the previously described process in connection with FIG. J l, three verification tests ate applied in ordeτ to eliminate potential false detections The first is a SΛD based FP removal scheme that tests, at step Ϊ204, how flat the region is between the two selected edge peaks For a vehicle, the region is expected to be relatively flat If the region between the two selected edge peaks is relatively Oat, then a horizontal structure based FP removal scheme is executed, at step 1206, to determine whether a sufficient amount of horizontal edge energ\ is present in the image For a vehicle, horizontal edges from the top and bottom of the vehicle as well as the shadow edge on the road below the vehicle, is expected Finaii\, if it is determined that a sufficient amount of horizontal edge energy is present, then a ground plane based FP removal scheme is engaged, at step 1208, to check how many of the consensus points overlap with image pixels that have been identified as lying on the road plane by the RD \ algorithm If too many consensus points are identified as lying on the ground plane, then the detection is most likely a false positiv e from the road surface, and will be rejected. If the detection ROl being tested meets the requirements of all three testing stages, then the detection is accepted at step 1210, otherwise the testing scheme is terminated, at step 1212, at the first instance of non-compliance in any one of the testing stages

j0092| At this stage, since the detection scheme has been executed multiple times for analyzing each detection gate, it ia possible that a given vehicle has been detected by more than one gate Therefore, a scheme for the removal of multiple detections of the same lead vehicle in a single frame is engaged, as illustrated by the process flow provided in HG. 13, which shows how these detections are consolidated into a set of unique detections (i e., one per vehicle in the scene) Detections from all the initializer gates, after false positive removal, are retrieved, at step 1302, to construct, at step 1304, a graph for all the detection ROIs. Nodes corresponding to two detections are connected by an edge if the overlap between the detection ROIs is greater than a threshold and the difference between the disparities for the detections is less than a threshold. Thereafter, the constructed graph is partitioned, at step 1306, into cliques (i.e , completely connected components). From each clique, the detection with the lowest value of the normalized SAD score, which was computed in the SAD based FP removal scheme described in connection with FIG. 12, is chosen at step 1308. This yields, at step 1310, the unique detections corresponding to the same lead vehicle

J0093] A final test to remove false detections is based upon temporal consistency of the detection. For every detection that survives the tests implemented in the single gate FP removal scheme (FIG. 12) and the multiple detection removal scheme (FlG. 13), a temporal history is gathered. Only if a detection persists for a given number of frames is it finally declared to be a reliable detection that can be passed on to the tracking scheme in the VDT algorithm of the present invention. A temporal consistency based FP removal scheme is implemented according to the process flow illustrated in FIG- 14, wherein the process is initiated by retrieving information related to the detections after removal of multiple detections corresponding to the same vehicle and vehicle detection histories from previous frames at steps J 402a and 1402b, respectively Thereafter, at step 1404, detections that fall within the inspection regions of the previous detection histories and have disparities consistent vvtth those for the detection histories are found The detection histories for which consistent detections are found are then updated, at step 1406, and the detection lost count for the rest of the previous detection histories is increased, at step 1408 Detection histories for which the detection lost count is greater than one are thrown awaj , at step I4Ϊ0, and new detection histories foτ those detections which not been associated with any prev ious detection historx are started, at step 1412 Detections for which the corresponding history length is greater than five aie yielded at step 1414

j0094| The nighttime detection embodiment of the VDF algorithm is designed to operate on stereo images, where the only v isible parts may be illuminated portions of the vehicle (e g . taillights and headlights) [lie nighttime detection embodiment is structured similarly to the daytime detection embodiment, m that there is an initialization scheme that uses a set of detection gates, and a tracking scheme that will tiack vehicles that have been detected by the initialization scheme Λ more detailed analysis of the computation engaged in by the VDT algorithm during a nighttime search of an image scene having a lead vehicle is described with reference to the process flows illustrated in FIGS. 15-19

10095] In the daytime detection application, the VDT algorithm extracts the edges of vehicles and estimates the three-dimensional location of those edges In the nighttime detection application, the VDT algorithm looks for pahs of bright "blobs" in the image that correspond to the two lights of a lead vehicle ahead of the host vehicle, and then estimates the depth to the blob pairs The fust stage is to extiact all of the blobs from the intensity image, as described in connection with FlG. 15, which illustrates the process flow for night-time feature extraction The process is initiated by receiving, at step 1502, the left image from stereo camera configuration 26 The left image of the stereo pair is thresholded, at step 1506, to extract the brightest pans of the image data, found in step 1504, since it is those parts that are HLeIy to correspond to a vehicles headlamps or taillights After thresholding, a binary image is stored and a connected components scheme is then applied, at step 1508, to this stored data to extract coherent "blobs^" of

20 image data that are likely coriespond to single objects in the scene, such as a Sight or a reflective number plate

føθ%] rhc second stage works on a region of interest (such as a detection gate, or the ROl set by the trackei) to make pahs of blobs thai could ieasonably belong to a single car Simitar to the daytime detection application of the VDF algorithm, the nighttime detection application analyzes each gate traversed by the host vehicle The blob pairing scheme that takes coherent blobs of pixels from the feature detection scheme of FlG. 15 and a detection gate inspection ROl is illustrated in the process flow diagram of FiG. 16 This process is repeated for every gate inspection ROl recehed at step 1602, wherein blobs lying inside the inspection region of the gate arc selected, at stop 1604 The blobs where the pixel area of the blob is determined to be greater than the threshold dependent on the gate depth is selected, at step 1606 Then, at step 1608, the maximum and minimum detection ROl width from the maximum and minimum depth for the gate is calculated This calculation guides the pairing of blobs within the image ROI, wherein the blobs within the image ROl of the gate aie paired based upon proximity measures, and a set of putative detection ROb are formed as the bounding boxes that surround the various blob pahs The blob pairing scheme proceeds, at step 1610, to find all possible pairs from the selected blobs where the \ertical position of the two blobs are determined to be the same within some defined threshold and wbeie the width of the bounding box of the blob lies between the minimum and maximum widths (as determined at step 1608) Thereafter, the bounding box for all possible blob pairs satisfying the prev ious constraint are computed, at step 1612, to \ ield, at step 1614, the possible detection ROl from all the bounding boxes

J0O97] The possible detections determined by the blob pairing scheme in FIG, 16 are then passed on to a depth estimation stage, described in connection with the process flow of FIG. 17, which estimates the range to the blob pairs The depth estimation process receives image data associated with the left sub-sampled image, at step 1702a₅ the right sub-sampled image, at step ϊ702b and the possible detection ROIs (determined in the process flow of FlG. 16), at step 1702c For each of the blob pair detection ROK, a disparity estimate using SAD scores between the left and right images at the appropriate pyramid levei is computed The SΛD between the left and right sub-sampled image for shifts between a specified range {e g , [gate presearch-5] to [gate presearch+7] ) is calculated, at step 1704, for passible detection ROIs Then, at step 1706, the scheme determines for which shift the SAD score is minimum and calculates the disparity using a quadratic interpolation of the SAD seoie foτ minimum shift This is the tefined disparity of the detection ROl To mitigate the effects of pixel blooming when using, for example, a CCD camera, in addition to estimating the disparity using SA.D for the whole ROl, the disparity for the left and right halves of the RO! are also estimated separately, and tiie ratio of the SAD scores for each half in addition to the disparity value calculated for the whole potential detection ROl are stored fhus to circumv ent blooming, with the assumption that both the left and right taiilights have similar blooming, the SAD score for the left and right half of the detection ROI is calculated, at step 1708, using the computed disparity, and the ratio of the SAD scores is found, yielding, at step 1710. disparity . SAD scores and SAD ratio for each detection ROI

|0098] For all of the detection ROIs, the si/e of the detected ROl and the disparity of the detected ROI are tested, in accordance with the process flow illustrated in FIG, 18, to determine whether the> are appropriate for the gate in which that 1101 was originally found Here, as prev iously described in connection with the dav tirne application, the flatness of a scene between a blob pair, as well as the vv idth and height (e g , in meters) of the region defined by the hounding box of the blob pair, is checked If the flatness and size of the detected pair are deemed satisfactory, then there may be a possible detection All of the detection ROIs received, at step 1802, are checked, at step 1804. to determine whether the detection ROl satisfies a maximum and minimum car width v alue, as well as a height constraint, at the refined disparity If so. then the associated refined disparity is checked, at step 1806. to determine whether it lies between the minimum and maximum gate disparities if either of these tests me failed the detection ROl is rejected at step 1807 If both tests are passed, then the detection ROl is identified, at step 1808, as an acceptable detection RX)] and the verification scheme yields, at step 1810, a Hst of accepted detection ROIs The tests employed by this verification scheme are then reiterated for the remaining possible detection ROIs J0099J Potential detections that pass both of the aforementioned tests me retained and passed to the next verification stage, illustrated b> the process flow of FIG. 19 A detection for a gate will only be accepted if its SΛD score foi the estimated disparity is below a pre-determined threshold, and also if its SAD score for the left and right halves of the ROI are similar This final verification stage

at step 1902, the list of accepted detection ROIs At step 1904, the detection Rϋϊ with the minimum SAD score in the list is then found and subjected to a condition, at step 1906. to determine whether or not this minimum SΛD score is less than the pre-determined threshold value If the value is determined to be acceptable, then the SAD ratio between the right and left halves is subsequently tested, at step 190S to determine if it is less than a tlueshoid value Failure to satisfy the threshold conditions in both of the tests at steps 1906 and 1908 will yield, at step 1909, in a determination that no vehicle has been detected for the particular gate Otherwise, satisfaction of both test conditions will result in the detection ROI being accepted, at step 1910, as a possible vehicle When, the detection passes and is accepted, it is \erifJed for that particular frame and is then passed to the temporal consistent test previously descπbed in connection with the daytime application of the VDT algorithm, as illustrated in the process flow diagram of FtG, 14 Only after the detection has persisted for a sufficient number of frames will that detected vehicle be introduced to the \ chicle tracker

JOlOO] One skilled in the art will appreciate that the present invention can be practiced by other than the descπbed embodiments, which are presented for purposes of illustration and not by way of limitation, and the present invention is limited only b\ the claims that follow

Claims

What is claimed is

1 A method for adaptiv e regulation of distance between a host vehicle and a lead vehicle, comprising receiving a preset cruise speed foi said host vehicle. receiving a stereo image of a roadway scene in a headway direction of said host vehicle, identifying lane boundary markers in said stereo image of said roadway scene, identifying potential lead vehicle detections on said foadwas scene in said headway direction of said host vehicle using said identified lane boundary marker data. verifying said potential lead vehicle detections. racking said v erified lead vehicle detection and estimating forwaϊd and lateral motion of said lead vehicle with respect to said host vehicle, and adjusting said preset cnnse speed of said host \ ehieic based on said estimated forward and lateral motion data associated with said lead \ehicle

2 An apparatus for adaptive regulation of distance between a host vehicle and a lead

\ehicle, comprising a stereo imaging device mountably affixed to said host vehicle for providing an image scene of a road vv as in a headwas direction of said host vehicle and at least one said lead vehicle on said roadway, an image processing part foi identifying road boundary markers and at least one said lead vehicle in said image scene, and a v ehicular system controller for adjusting a preset cruise speed of said host \ ehicie in response to image processing data associated with identification of said lead vehicle

J ^

3. A computer readable medium having stored thereon a plurality of instructions, the plurality of instructions including instructions which, when executed by a processor, cause the processor to perform the steps of a method for adaptive regulation of distance between a host vehicle and a lead vehicle, comprising the steps of receiving a preset cruise speed for said host vehicle; receiving a stereo image of a roadway scene in a headway direction of said host vehicle; identifying lane boundary markers in said stereo image of said roadway scene: identifying potential lead vehicle detections on said roadway scene in said headway direction of said host vehicle using said identified lane boundary marker data; verifying said potential lead vehicle detections; tracking said verified lead vehicle detection and estimating forward and lateral motion of said lead vehicle with respect Io said host vehicle; and adjusting said preset cruise speed of said host vehicle based on said estimated forward and lateral motion data associated with said lead vehicle.