CN107423675A - The advanced warning system of front shock warning is carried out to trap and pedestrian - Google Patents
The advanced warning system of front shock warning is carried out to trap and pedestrian Download PDFInfo
- Publication number
- CN107423675A CN107423675A CN201710344179.6A CN201710344179A CN107423675A CN 107423675 A CN107423675 A CN 107423675A CN 201710344179 A CN201710344179 A CN 201710344179A CN 107423675 A CN107423675 A CN 107423675A
- Authority
- CN
- China
- Prior art keywords
- model
- patch
- described image
- point
- collision
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/588—Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
-
- G—PHYSICS
- G08—SIGNALLING
- G08G—TRAFFIC CONTROL SYSTEMS
- G08G1/00—Traffic control systems for road vehicles
- G08G1/16—Anti-collision systems
- G08G1/165—Anti-collision systems for passive traffic, e.g. including static obstacles, trees
-
- G—PHYSICS
- G08—SIGNALLING
- G08G—TRAFFIC CONTROL SYSTEMS
- G08G1/00—Traffic control systems for road vehicles
- G08G1/16—Anti-collision systems
- G08G1/166—Anti-collision systems for active traffic, e.g. moving vehicles, pedestrians, bikes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30248—Vehicle exterior or interior
- G06T2207/30252—Vehicle exterior; Vicinity of vehicle
- G06T2207/30256—Lane; Road marking
Abstract
The video camera that can be filled in a motor vehicle the present invention relates to use carries out the advanced warning system and method for front shock warning to trap and pedestrian.This method obtains picture frame by known spacings.Patch can be selected at least one picture frame.Light stream between the picture frame of the multiple images point of traceable patch.Picture point can be fitted to at least one model.Fitting based on picture point Yu at least one model, if contemplating that collision, it may be determined that collision time (TTC).Picture point can be fitted to road surface model and its part is modeled as imaging from road surface.Fitting based on picture point and road surface model determines that expection does not have collision.At least one model may also include mixed model, and picture point Part I can be modeled as imaging from road surface, and its Part II can be modeled as imaging from substantial orthogonality object.Picture point can be fitted to vertical surface model, and picture point part can be modeled as imaging from perpendicular objects.TTC can be determined based on the fitting of picture point and vertical surface model.
Description
It is on December 7th, 2011 applying date that the application, which is, and Application No. 201110404574.1 is entitled " to falling into
The divisional application of the application of trap and the advanced warning system of pedestrian's progress front shock warning ".
Background
1. technical background
The present invention relates to the driver assistance system for providing front shock warning.
2. description of related art
Driver assistance system based on video camera (driver assistance system, DAS) in recent years
Come into market；The driver assistance system include lane departur warning (lane departure warning, LDW), from
Dynamic distance light control (Automatic High-beam Control, AHC), pedestrian's identification and front shock warning (forward
Collision warning, FCW).
Lane departur warning (LDW) system is designed to give a warning in the case of unintentional deviation.When
Vehicle by or give a warning when will pass through lane markings.Use based on turn signal, the change of the angle of steering wheel, car
Speed and brake activation determine driver intention.
In image procossing, Moravec Corner Detection Algorithms are probably that earliest Corner Detection Algorithm defines angle in the lump
Point is the point with relatively low self-similarity.Moravec algorithms by consider the patch (patch) that concentrates in pixel with it is big nearby
How similar partly overlapping patch has, and angle point whether there is from the point of view of each pixel in test image.By using two spots
The difference of two squares and (sum of squareddifference, SSD) measurement similitude between block.The smaller explanation similitude of numeral is got over
Greatly.The method of angle point in optional detection image based on the method by Harris and Stephens propositions, this method be to by
The improvement for the method that Moravec is proposed.Harris and Stephens is directly micro- with the angle point fraction of directional correlation by considering
Divide rather than using Moravec adjacent to patch, Moravec Corner Detection Algorithm is improved.
In computer vision, the widely used differential method for light stream estimation be by Bruce D.Lucas and
Takeo Kanade exploitations.Lucas-Kanade methods are assumed
Constant, and basic optical flow equation is solved to all pixels in the neighborhood by criterion of least squares.By comprehensive from several
The information of individual neighborhood pixels, Lucas-Kanade methods are generally possible to the inherent ambiguity for solving optical flow equation.With point-by-point method
Compare, this method is also insensitive to picture noise.On the other hand, because this method is pure partial approach, it can not
Stream information in the internal unity region of image is enough provided.
General introduction
According to the feature of the present invention, there is provided for sending the distinct methods of front shock caution signal, methods described makes
With the video camera that can be installed in a motor vehicle.Multiple images frame is obtained by known time interval.Can be at least one picture frame
Middle selection image patch.The light stream of the multiple images point of patch can be tracked between picture frame.Picture point can be fitted at least one
Individual model.Fitting based on picture point, it may be determined that collision whether is contemplated that, and if it is expected that having, it may be determined that touch
Hit the time (TTC).Picture point can be fitted to road surface model, and a part for picture point can be modeled as being imaged from road surface.Can base
Determine that expection does not have collision in the fitting of picture point and road surface model.Picture point can be fitted to vertical surface model, wherein scheming
A part for picture point can be modeled as imaging from perpendicular objects.When fitting determination that can be based on picture point and vertical surface model is collided
Between TTC.Picture point can be fitted to mixed model, and the wherein Part I of picture point can be modeled as being imaged from road surface, and picture point
Part II can be modeled as imaging from the object in substantial orthogonality or upright object rather than traverse road surface.
In picture frame, the candidate image of pedestrian is can detect, wherein, the patch is selected to include candidate's figure of pedestrian
Picture.When best fit model is vertical surface model, it may be verified that candidate image be upright pedestrian image rather than road surface in
Object.In picture frame, vertical line is can detect, wherein, the patch is selected to include the vertical line.When best fit mould
When type is vertical surface model, it may be verified that vertical line is the image of the image of perpendicular objects rather than the object in road surface.
In distinct methods, collision time can be based on and be less than threshold value and give a warning.In distinct methods, image can be based on
Light stream between frame determines the relative scale of patch, and may be in response to the relative scale and time interval determines collision time
(TTC).This method can be avoided it is determined that carrying out Object identifying before relative scale in patch.
According to the feature of the present invention, there is provided the system including video camera and processor.The system may be used in can
The video camera of installation in a motor vehicle provides front shock warning.The system can also be used for more by the acquisition of known time interval
Individual picture frame, at least one middle selection patch in picture frame；For track patch multiple images point picture frame it
Between light stream；For picture point to be fitted at least one model and determined based on picture point and the fitting of at least one model
Whether be expected to have collision, if being expected if determine collision time (TTC).The system can be additionally used in picture point
It is fitted to road surface model.Fitting determination expection that can be based on picture point and road surface model does not have collision.
According to the other embodiment of the present invention, the patch in picture frame may be selected, the patch can correspond to motor vehicle will be
The location of after predetermined time interval.The patch can be monitored；Front shock is sent if object is imaged in the patch
Warning.Can be substantially to determine object by the light stream between the picture frame for the multiple images point for tracking the object in patch
No is vertical, upright or not in road surface.Picture point can be fitted at least one model.A part for picture point can model
To be imaged from object.Fitting based on picture point Yu at least one model, it is determined whether it is expected that collision is had, if it is expected that have
Words then determine collision time (TTC).Front shock warning can be sent when best fit model includes vertical surface model.Image
Point can be fitted to road surface model.Fitting determination expection that can be based on picture point and road surface model does not have collision.
According to the feature of the present invention, there is provided a kind of system for being used to provide front shock warning in a motor vehicle.It is described
System includes that video camera and processor in a motor vehicle can be installed.Video camera can be used for multiple by known time interval acquisition
Picture frame.The patch that processor can be used in selection picture frame, the patch corresponds to motor vehicle will institute after a predetermined interval of time
The position at place.If object is imaged in patch, if it find that object is upright and/or front portion can be then sent not in road surface
Conflict alert.Processor can be additionally used in the multiple images point that the object in patch is tracked between picture frame, and picture point is intended
Close one or more models.The model may include perpendicular objects model, road surface model and/or mixed model, mixed model
Comprise provide that one or more picture points from road surface and one or more images from the upright object not in road surface
Point.Fitting based on picture point and model, it is determined whether collision is contemplated that, if it is expected that there is collision then to determine collision time
(TTC).Processor can be used for sending front shock warning less than threshold value based on TTC.
The application further relates to herein below：
1) a kind of method for being used to provide front shock warning, the method use the shooting that can be installed in a motor vehicle
Machine, methods described include：
Multiple images frame is obtained by known time interval；
In at least one middle selection patch of described image frame；
Track the light stream between the picture frame of the multiple images point of the patch；
Described image point is fitted at least one model；And
Fitting based on described image point Yu at least one model, if it is expected that there is collision, it is determined that collision time
(TTC)。
2) method as described in 1), in addition to：
Described image point is fitted to road surface model, at least a portion of wherein described image point is modeled as imaging from road
Face；
Fitting based on described image point Yu the model, it is determined that expection does not have collision.
3) method as described in 1), in addition to：
Described image point is fitted to vertical surface model, at least a portion of wherein described image point is modeled as being imaged
From vertical object；And
Fitting based on described image point Yu the vertical surface model, determines the TTC.
4) method as described in 3), in addition to：
The candidate image of pedestrian is detected in described image frame, wherein selecting the patch with including described in the pedestrian
Candidate image；And
When best fit model is the vertical surface model, it is the image of upright pedestrian to verify the candidate image
Rather than the image of the object in road surface.
5) method as described in 3), in addition to：
Vertical line is detected in described image frame, wherein selecting the patch with including the vertical line；
When best fit model is the vertical surface model, verify the vertical line be the image of vertical object and
It is not the image of the object in road surface.
6) method as described in 1), wherein at least one model also includes mixed model, wherein described image point
Part I is modeled as imaging from road surface, and the Part II of described image point is modeled as pair of the imaging from substantial orthogonality
As.
7) method as described in 1), in addition to：
Threshold value is less than based on the collision time and given a warning.
8) a kind of system including that can install video camera and processor in a motor vehicle, the system, which can operate, to be come
Front shock warning is provided, the system, which can operate, to be come：
Multiple images frame is obtained by known time interval；
In at least one middle selection patch of described image frame；
Track the light stream between the picture frame of the multiple images point of the patch；
Described image point is fitted at least one model；And
Fitting based on described image point Yu at least one model, if it is expected that there is collision, it is determined that collision time
(TTC)。
9) system as described in 8), additionally it is possible to which operation comes：
Described image point is fitted to road surface model；
Fitting based on described image point Yu the road surface model, it is determined that expection does not have collision.
10) a kind of method for providing front shock warning, the method use the video camera that can be installed in a motor vehicle
And processor, methods described include：
Multiple images frame is obtained by known time interval；
The patch in picture frame is selected, the patch corresponds to the motor vehicle by position residing after a predetermined interval of time
Put；And
The patch is monitored, front shock warning is sent if object is imaged in the patch.
11) method as described in 10), in addition to：
Determine whether the object includes the part of substantial orthogonality.
12) method as described in 11), wherein described determine to perform by following operation：
Track the light stream between the picture frame of the multiple images point in the patch；And
Described image point is fitted at least one model.
13) at least a portion of the method as described in 11), wherein described image point is modeled as imaging from vertical pair
As；And
Fitting based on described image point Yu at least one model, if it is expected that there is collision, it is determined that collision time
(TTC)。
14) method as described in 11), wherein at least one model includes road surface model, methods described also includes：
Described image point is fitted to road surface model；
Fitting based on described image point Yu the road surface model, it is determined that expection does not have collision.
15) method as described in 11), in addition to：
When best fit model is vertical surface model, the warning is sent.
16) a kind of system for being used to provide front shock warning in a motor vehicle, the system include：
Video camera, it can be arranged in the motor vehicle, and the video camera can be operated by known time interval
Obtain multiple images frame；
Processor, it, which can be operated, comes：
The patch in picture frame is selected, the patch corresponds to the motor vehicle by position residing after a predetermined interval of time
Put；
Monitor the patch；And
If object is imaged in the patch, front shock warning is sent.
17) system as described in 16), wherein the processor can also operate to determine whether the object includes essence
Upper vertical part, it is described to determine to perform by following operation：
The multiple images point of the object in the patch is tracked between described image frame；
Described image point is fitted at least one model；And
Fitting based on described image point Yu at least one model, if it is expected that there is collision, it is determined that collision time
(TTC)。
18) system as described in 16), wherein the processor can operate sends front portion to be based on TTC less than threshold value
Conflict alert.
The brief description of accompanying drawing
The present invention is only described with reference to the drawings by way of example herein, wherein：
Fig. 1 a and 1b schematically show according to the present invention it is feature, when vehicle is close to guardrail wires from installed in car
Two images of the forward looking camera capture in.
Fig. 2 a show feature, for using in main car (host vehicle) the video camera according to the present invention
The method that front shock warning is provided.
Fig. 2 b show the further of the step of determination collision time feature, showing in fig. 2 a according to the present invention
Details.
Fig. 3 a show the picture frame (back side of van) on feature, the upright surface according to the present invention.
Fig. 3 c show feature, mainly road surface the rectangular area according to the present invention.
Fig. 3 b show the point of function feature, on Fig. 3 a as vertical image position (y) according to the present invention
Move vertically δ y.
Fig. 3 d show the point of function feature, on Fig. 3 c as vertical image position (y) according to the present invention
Move vertically δ y.
Fig. 4 a show the image of feature including with horizontal line and rectangular patches the guardrail wires according to the present invention
Picture frame.
Fig. 4 b and 4c show the more details of rectangular patches feature, showing in fig.4 according to the present invention.
Fig. 4 d show the song relative to vertical point position (y) according to feature, point vertical movement (δ y) of the invention
Line chart.
Fig. 5 shows another example of feature, in picture frame the mirage according to the present invention.
Fig. 6 shows feature, for providing front shock warning trap the method according to the present invention.
Fig. 7 a and 7b are shown according to front shock trap warning example feature, being triggered for wall of the invention
Example.
Fig. 7 c show example example feature, being alerted for the front shock trap that box is triggered according to the present invention
Son.
Fig. 7 d show according to the present invention it is example feature, warned for the front shock trap that is triggered of side of automobile
The example of announcement.
Fig. 8 a show according to an aspect of the present invention, have on box the object of obvious vertical line example.
Fig. 8 b show according to an aspect of the present invention, have on lamppost the object of obvious vertical line example.
Fig. 9 and 10 shows according to an aspect of the present invention including in vehicle video camera or imaging sensor
System.
It is described in detail
With detailed reference to the feature of the present invention, its example is shown in the drawings, and wherein identical reference numeral is from beginning
To referring to identical element eventually.Carry out Expressive Features below with reference to figure to explain the present invention.
Before the feature of the present invention is explained in detail, it should be appreciated that the present invention is not only restricted to its institute in the following description
The state or application in the shown in the accompanying drawings design of part and the details of arrangement.The present invention has other features or energy
Enough different modes are practiced or carried out.In addition, it should be understood that the phraseology and terminology used herein be for descriptive purposes and
It is restricted that should not be construed.
By way of introduction, embodiments of the present invention are related to front shock warning (FCW) system.According to United States Patent (USP)
7113867, the image of front truck is identified.The width of vehicle can be used for detecting the ratio or relative scale S between picture frame
In change, and relative scale be used for determine collision time.Specifically, such as the width of front truck has in the first image and the
The length (length as measured by for example with pixel or millimeter) represented respectively with w (t1) and w (t2) in two images.So may be used
Selection of land, relative scale are S (t)=w (t2)/w (t1).
According to the teaching of United States Patent (USP) 7113867, front shock warning (FCW) system is relied on to barrier or the figure of object
The identification of picture, for example, such as the front truck identified in picture frame.In front shock warning system, such as United States Patent (USP) 7113867
Disclosed, the ratio of the size (such as width) of detected object (such as vehicle) changes for calculating collision time
(TTC).However, object first be detected and with the scene cut of surrounding.This disclosure has described changed using relative scale
System, its based on light stream determine collision time TTC and collision possibility, if it is desired, send FCW warning.Light stream causes
Mirage phenomenon (looming phenomenon)：As the object being imaged becomes nearer, the image of perception seems bigger.According to
The different characteristic of the present invention, object detection and/or identification are can perform, or object detection and/or identification can be avoided.
Mirage phenomenon has been extensively study in biology system.Mirage is seemingly a kind of to people very low-level to be regarded
Feel attention mechanism and natural reaction can be triggered.There are a variety of trials to detect mirage in computer vision, or even have silicon sensing
Device is designed for the mirage in the case of detecting pure flat shifting.
Can with changing lighting condition, include the complex scene of multiple objects and the actual environment of main car
Middle execution mirage detection, mirage detection include translation and move and rotate.
Term as used herein " relative scale " refers to image patch in a picture frame and in subsequent picture frame
Correspondence image patch relative size increase (or reduce).
Referring now to Fig. 9 and 10, according to an aspect of the present invention, Fig. 9 and 10 shows to include the shooting being arranged in vehicle 18
The system 16 of machine or imaging sensor 12.To the transmission figure picture in real time of imaging sensor 12 of field of front vision imaging, these images
It is captured with the time series of picture frame 15.Image processor 14 can be used for simultaneously and/or concurrently handling picture frame 15
For many driver assistance system services.It can be used soft in specific hardware circuit and/or memory 13 with onboard software
Part control algolithm realizes driver assistance system.Imaging sensor 12 can be monochromatic or black and white, i.e., no color divides
From, or imaging sensor 12 can be sense color.By the way that for the example in Figure 10, picture frame 15 is used for serving pedestrian's warning
(PW) 20, lane departur warning (LDW) 21, according to front portion of the teaching of United States Patent (USP) 7113867 based on object detection and tracking
Conflict alert (FCW) 22, the front shock based on image mirage alert (FCWL) 209 and/or based on FCW traps (FCWT) 601
Front shock warning 601.Image processor 14 is used to handle picture frame 15 to detect for being based on image mirage and FCWT
The mirage of image in the field of front vision of the video camera 12 of 601 front shock warning 209.Front shock based on image mirage
Alerting 209 and front shock warning (FCWT) 601 based on trap can perform parallel with traditional FCW22, and and other
Driver assistance function, pedestrian detection (PW) 20, lane departur warning (LDW) 21, road traffic sign detection and self motion detection
It is parallel to perform.FCWT 601 can be used for normal signal of the checking from FCW22.As used herein term " FCW signals " refers to
Front shock caution signal.Term " FCW signals ", " front shock warning " and " warning " are interchangeably used herein.
The feature of the present invention is shown in Fig. 1 a and 1b for showing the example of light stream or mirage.When vehicle 18 is close to metal
During guardrail 30, captured two image from the forward looking camera 12 in vehicle 18 is shown.Image in Fig. 1 a
The visual field and guardrail 30 are shown.Image in Fig. 1 b shows identical feature, and wherein vehicle 18 is closer to guardrail wires 30, if seen
The small rectangle p 32 (being indicated with dotted line) in guardrail is examined, may see that horizontal line 34 seems with the close shield of vehicle 18 in Figure 1b
Stretch on column 30.
Referring now to Fig. 2 a, it shows feature, for using in main car 18 the video camera 12 according to the present invention
The method 201 that front shock alerts 209 (FCWL209) is provided.Method 201 is independent of the object in the field of front vision of vehicle 18
Object identifying.In step 203, multiple images frame 15 is obtained by video camera 12.Time interval between the capture of picture frame
It is △ t.The patch 32 in picture frame 15 is selected in step 205, and determines the relative scale (S) of patch 32 in step 207.
In step 209, collision time (TTC) is determined based on the relative scale (S) between frame 15 and time interval (△ t).
Referring now to Fig. 2 b, it is shown according to the step of determination collision time feature, showing in fig. 2 a of the invention
209 further details.In step 211, the multiple images point in patch 32 can be tracked between picture frame 15.In step
In 213, picture point can be fitted to one or more models.First model can be vertical surface model, its may include such as pedestrian,
Vehicle, wall, shrub, the object of tree or lamppost.Second model can be road surface model, and it considers the spy of the picture point on road surface
Sign.Mixed model may include one or more picture points from road, and one or more figures from upright object
Picture point.For the model for a part of picture point at least assuming to include upright object, multiple collision times can be calculated
(TTC).In step 215, the best fit of picture point and road surface model, vertical surface model or mixed model makes it possible to select
Select collision time (TTC) value.Collision time (TTC) based on less than threshold value and when best fit model is vertical surface model
Or during mixed model, it can give a warning.
Alternatively, step 213 may additionally include the detection of the candidate image in picture frame 15.Candidate image can be pedestrian
Or the vertical line of perpendicular objects such as lamppost.In the case where being pedestrian or vertical line, patch 32 may be selected to scheme including candidate
Picture.Once selected patch 32, then it is the image of upright pedestrian and/or the image of vertical line to be possible to perform candidate image
Checking.The checking can confirm that candidate image is not the object in road surface when best fit model is vertical surface model.
Look back Fig. 1 a and 1b, the son of the patch 32 for the second image that the first image shown from Fig. 1 a is shown into Fig. 1 b
Pixel arrangement, which can cause size to increase by 8% or relative scale S, increases by 8% (S=1.08) (step 207).Assuming that between image
T=0.5 seconds time difference △, collision time (TTC) can use following equation 1 to calculate (step 209)：
If it is known that the speed of vehicle 18 is v (v=4.8m/s), then range-to-go Z also can use following equation 2 to count
Calculate：
According to the feature of the present invention, Fig. 3 b and 3d are shown as the vertical fortune of the point of the function of vertical picture position (y)
Dynamic δ y.The δ y that move vertically are zero at horizontal line, are negative value under horizontal line.The vertical movement δ y of point are with following equation 3
Show.
Equation (3) is the linear model on y and δ y and is of virtually two variables.Two points can be used to solve this
Two variables.
For vertical surface because all points are equidistant, as in shown image in fig 3b away from
From moving in horizontal line (y0) place is zero and linearly changes with picture position.For road surface, point in the picture more it is low then more
Closely (Z is smaller), as shown by following equation 4：
Therefore, image motion δ y are not only with the increase of linear rate, in following equation 5 and as shown in Fig. 3 d figure
's.
Equation (5) is the secondary equation of constraint for being of virtually two variables.
Equally, two points can be used to solve the two variables.
Referring now to Fig. 3 a and 3c for representing different picture frames 15.In Fig. 3 a and 3c, two rectangular areas are shown with dotted line
Go out.Fig. 3 a show upright surface (behind van).Square points are the points of tracked (step 211), are moved with scheming
In 3b compared to point height y image motion (δ y) image shown in upright surface motion model match (step
It is rapid 213).The motion of triangle point in fig. 3 a mismatches the motion model on upright surface.Referring now to Fig. 3 c, it shows mainly
It is the rectangular area on road surface.Square points be with Fig. 3 d compared to point height y image motion (δ y) image in institute
The point that the road surface model shown matches.The motion of triangle point mismatches the motion model on road surface and is exceptional value
(outlier).Therefore in general, task here be to determine which point belong to model (and which model belonged to) and which
Point is exceptional value, and this can be performed by robust approximating method as be explained below.
Referring now to Fig. 4 a, 4b, 4c and 4d, they show feature, in image two motions according to the present invention
The typical situation of the mixing of model.Fig. 4 a show the picture frame 15 for the image and rectangular patches 32a for including guardrail wires 30, wherein
The image of guardrail wires 30 has horizontal line 34.Patch 32a further details are shown in figs. 4 b and 4 c.Fig. 4 b show one
The details of the patch 32a in picture frame 15 before individual, Fig. 4 c show subsequent at one when vehicle 18 is closer to guardrail 30
Picture frame 15 in patch 32a details.In Fig. 4 c and 4d, some picture points are shown on vertical obstacle 30
Square, triangle and circle, and some picture points are illustrated on the road surface in the front of barrier 30.In the 32a of rectangular area
Trace point show, some point in the lower part corresponding to the region 32a of road model, and some point corresponding to upright
Surface model region 32a upper part in.Fig. 4 d show vertical movement (δ y) a little compared to vertical point position (y)
Curve map.In figure 4d, there are two parts with the model being resumed illustrated：Bend (parabolical) part 38a and line
Property part 38b.Transition point between part 38a and 38b corresponds to the bottom on upright surface 30.The transition point is also by Fig. 4 c
Horizontal dotted line 36 mark.There are some in figs. 4 b and 4 c by the point shown in triangle, their tracked but mismatching models,
The point that the tracked point of some Matching Models shows not tracked well and some by square is illustrated as circle.
Referring now to Fig. 5, it shows another example of the mirage in picture frame 15.In Fig. 5 picture frame 15, in patch
There is no upright surface in 32b, only the accessible road in front, and the transition point between two models at horizontal line with
Dotted line 50 marks.
The estimation of motion model and collision time (TTC)
Estimation (the step 215) of motion model and collision time (TTC) assumes one region 32 of offer, such as in picture frame
Rectangular area in 15.The example of rectangular area is the rectangle 32a and 32b for example shown in Fig. 3 and 5.It can be based on being examined
The object of such as pedestrian surveyed selects these rectangles based on the motion of main car 18.
1. trace point (step 211)：
(a) rectangular area 32 can be subdivided into 5x20 sub- rectangular grids.
(b) it can be that each sub- rectangle performs algorithm to find the angle point of image, such as use Harris and Stephens
Method, and the point can be traced.5x5Harris points are preferred, it is contemplated that the characteristic value of matrix below,
And search out two strong characteristic values.
(c) some optimal differences of two squares of the exhaustive search in the rectangular search region with width W and height H can be passed through
(SSD) match to perform tracking.When starting, the exhaustive search is critically important, as it means that motion before is not adopted
With, and the measurement from all sub- rectangles is statistically more independent.Light stream has been used to estimate after searching
Fine setting, wherein light stream estimated service life such as Lukas Kanade methods.Lukas Kanade methods allow sub-pixel motion.
2. models fitting (the step 213) of robust：
(a) two or three points are randomly selected from 100 tracked points.
(b) be selected to quantity (NIt is right) car speed (v) is depended on, such as be given by the following formula：
NIt is right=min (40, max (5,50-v)) (7)
Wherein v units are meter per second.Quantity (the N of triple (triplet)Triple) be given by the following formula：
NTriple=50-NIt is right (8)
(c) for two points, they can be fitted two model (steps 213).The two points of one model hypothesis are upright
Object on.The two points of second model hypothesis are all on road.
(d) for three points, they can also be fitted two models.Two points above one model hypothesis are in upright pair
As upper, the 3rd (nethermost) point is on road.Under the uppermost point of second model hypothesis is on upright object
Two points in face are on road.
Two models can solve on three points, and this solves the first model (equation 3) by using two points, then with knot
Fruit y0The second model (equation 5) is solved with the 3rd point.
(e) each model in (d) provides collision time TTC value (steps 215).Each model is also based on 98
Other individual points and models fitting, which obtain, how well obtains a fraction.By between the model sport that the y of point is moved and is predicted
The truncation quadratic sum (Sum ofthe Clipped Square ofthe Distance, SCSD) of distance provide the fraction.
SCSD values are converted to the function similar to probability：
Wherein N is the quantity (N=98) of point.
(f) based on TTC values, the speed of vehicle 18 and these are assumed on static object, can calculate these points
Distance Z=v x TTC.According to the x image coordinates of each picture point distance, the lateral attitude in world coordinates can be calculated：
(g) therefore calculate in time TTC lateral attitude.Binary system transverse direction fraction requirement from pair or triple point
In it is at least one must be in the path of vehicle 18.
3. the fraction of multiframe：Can produce new model in each frame 15, each new model have its related TTC and
Fraction.The model of 200 optimal (fraction highest) can be retained from 4 frames 15 before, its mid-score is weighted as follows：
Fraction (n)=αnFraction (12)
Wherein n=0.3 is the age (age) of fraction, and α=0:95.
4.FCW judges：If any one generation in three following conditions, send real FCW warnings：
(a) TTC of the model with highest score is under TTC threshold values and fraction is more than 0.75, and
(b) with highest score model TTC under TTC threshold values and
(c)
Fig. 3 and 4 has been shown how robustly to provide FCW warnings for the point in given rectangle 32.How rectangle is limited
Depending on the application as shown in other example features by Fig. 7 a-7d and 8a, 8b.
The FCW traps of in general stationary objects
Referring now to Fig. 6, its show according to the present invention it is feature, for providing front shock warning trap (FCWT) 601
Method 601.In step 203, multiple images frame 15 is obtained by video camera 12.In step 605, select in picture frame 15
Patch 32, the patch corresponds to motor vehicle 18 will the location of after a predetermined interval of time.Then monitor in step 607
Patch 32.In judgment step 609, if general object is imaged in patch 32 and is detected wherein, in step 611
In send front shock warning.Otherwise the capture of picture frame is continued by step 203.
Fig. 7 a and 7b show the example feature according to the present invention, the example alerted for the FCWT 601 that wall 70 is triggered
Son；The example of warning that is triggered of side example feature, for automobile 72 according to the present invention is shown in figure 7d；And
And example example feature, for box 74a and the 74b warning triggered according to the present invention is shown in figure 7 c.Figure
7a-7d is class-based detection, in general stationary objects the examples before not requiring.Dashed rectangle region is defined as
At a certain distance from, target that W=1m is wide, the distance is main car by residing distance after t=4s.
Z=vt (16)
Wherein v is the speed of vehicle 18, and H is the height of video camera 12, and w and y are the width of rectangle respectively and schemed
Upright position as in.The rectangular area is the example of FCW traps.If object is " dropped " into the rectangular area, if TTC
Less than threshold value, FCW traps can produce warning.
Performance is improved using multiple traps：
In order to improve verification and measurement ratio, FCW traps can be copied in 5 regions with 50% lap, to produce 3m
Wide total trap area.
The dynamic position of FCW traps can be selected according to yaw rate (yaw rate)：Can be based on according to Yaw rate sensor,
The path for the vehicle 18 that the dynamic model of the speed of vehicle 18 and main car 18 determines laterally translates trap area 32.
For verifying the FCW traps of front shock caution signal
Pattern recognition techniques can be used to be detected in image 15 for the special class object of such as vehicle and pedestrian.According to the U.S.
The teaching of patent 7113867, these objects are afterwards with time tracking, and the change in use ratio can produce FCW 22 and believe
Number.However, it is important that using the independent signals of technical identification FCW 22 before giving a warning.If system 16 will activate
If brake, then use independent technology, such as application method 209 (Fig. 2 b) to verify that the signals of FCW 22 may be weighed particularly
Will.In the system of radar/vision fusion, independent checking may be from radar.It is independent in the system 16 of vision is based only on
Checking is from independent vision algorithm.
The detection of object (such as pedestrian, front truck) is not problem.Very high detection rates can be realized and only had very
Low error rate.Feature of this invention is that it is produced without the reliable FCW signals of too many false alarm, too many false alarm
Driver will be made irritated, or further worsened ground can cause driver unnecessarily to brake.One on traditional pedestrian's FCW systems
Possible problem is to avoid the front shock warning of mistake because the substantial amounts of pedestrian in the scene and really front portion is touched
The quantity for hitting situation is then very small.It also will imply that driver will likely receive frequently false alarm even if 5% error rate,
And it may never undergo real warning.
Pedestrian target is especially challenging for FCW systems, because target is nonrigid, this make it that tracking is tired
Difficult (according to the teaching of United States Patent (USP) 7113867), and ratio change especially can be disturbed a lot.Therefore, the model (side of robust
Method 209) it can be used for checking to be alerted for the front shock of pedestrian.Rectangular area 32 can be determined by pedestrian detecting system 20.Root
According to United States Patent (USP) 7113867, FCW signals, and the FCW (methods of robust can just only be produced by the tracking of the performance objectives of FCW 22
209) give than can with or cannot the small TTC of predetermined one or more threshold values.Front shock warning FCW 22 can
With the different threshold value of the threshold value from being used in the model of robust (method 209).
It is that pedestrian is typically occurred in the road of less structuring that one of factor of quantity of false alarm, which may be increased,
The possible rather unstable of the driving model of driver in such road, including zig zag and lane change.Therefore for police
Sending for accusing may need to include some further constraints：
When detecting curb or lane markings, if pedestrian is in the distal side in curb or/and track and does not occur following
During any one in condition, then FCW signals are prevented from：
1. pedestrian passes through lane markings or curb (or approaching very fast).On the other hand, the pin of detection pedestrian may be very
It is important.
2. main car 18 does not pass through lane markings or curb (for example, as detected by the systems of LDW 21).
The more difficult prediction of intention of driver.If driver is honest to driving, do not activate turn signal and not in advance in respect of
Other lane markings, then it is reasonable to assume that driver will continue directly to march forward.Therefore, if pedestrian in the paths and
TTC can then send FCW signals under threshold value.However, if driver turns, then he/her will continue turn or
Stop turn and continue move ahead be it is also possible that.Therefore, when detecting yaw rate, only when assuming that vehicle 18 will be inclined with identical
Boat angle continue turn and pedestrian in the paths, and if vehicle straight trip and pedestrian in the paths when just send FCW signals.
The concept of FCW traps 601 may extend into the main object for including vertical line (or horizontal line).Such object is made
It is that good Harris (angle point) points as a rule pass through hanging down on the edge by object with the possibility problem of the technology based on point
Straight line and the horizontal line of distant place background intersect to produce.The vertical movement of these points will be similar to that the road surface of distant place.
Fig. 8 a and 8b show the example with the object of obvious vertical line 82, the lamppost of the vertical line 82 in figure 8b
On box 84 on 80 and in Fig. 8 a.Vertical line 82 is detected in trap area 32.It can track between images detected
The straight line 82 arrived.Straight line 82 can be matched by frame by frame and calculate the TTC models of each straight line pair, it is assumed that perpendicular objects, so
SCSD afterwards based on other straight lines 82 provides fraction, to perform the estimation of robust.Because the quantity of straight line may be smaller, generally
It is the possible line pair for testing all combinations.Only using the straight line pair for having important lap.For horizontal line, and use
The same when point, triple line also gives two models.
Indefinite article " one (a) " used herein, " one (an) ", such as " image " (" an image "), " a rectangle region
Domain " (" arectangular region "), has a meaning of " one or more ", i.e., " one or more images " or " one or
Multiple rectangular areas ".
While there has been shown and described that selected feature of the invention, but it is understood that the present invention is not only restricted to be retouched
The feature stated.Conversely, it should be appreciated that these features can be changed without departing from the principle and spirit of the present invention, the present invention
Scope limited by appended claims and their equivalents.
Claims (18)
1. a kind of method for being used to provide front shock warning, the method use the video camera that can be installed in a motor vehicle,
Methods described includes：
Multiple images frame is obtained by known time interval；
In at least one middle selection patch of described image frame；
Track the light stream between the picture frame of the multiple images point of the patch；
Described image point is fitted at least one model；And
Fitting based on described image point Yu at least one model, if it is expected that there is collision, it is determined that collision time
(TTC)。
2. the method as described in claim 1, in addition to：
Described image point is fitted to road surface model, at least a portion of wherein described image point is modeled as imaging from road surface；
Fitting based on described image point Yu the model, it is determined that expection does not have collision.
3. the method as described in claim 1, in addition to：
Described image point is fitted to vertical surface model, at least a portion of wherein described image point is modeled as imaging and hung down certainly
Straight object；And
Fitting based on described image point Yu the vertical surface model, determines the TTC.
4. method as claimed in claim 3, in addition to：
The candidate image of pedestrian is detected in described image frame, wherein selecting the patch with the candidate including the pedestrian
Image；And
When best fit model is the vertical surface model, verify the candidate image be upright pedestrian image without
It is the image of the object in road surface.
5. method as claimed in claim 3, in addition to：
Vertical line is detected in described image frame, wherein selecting the patch with including the vertical line；
When best fit model is the vertical surface model, verify the vertical line be vertical object image rather than
The image of object in road surface.
6. the method as described in claim 1, wherein at least one model also includes mixed model, wherein described image point
Part I be modeled as imaging from road surface, and the Part II of described image point is modeled as imaging from substantial orthogonality
Object.
7. the method as described in claim 1, in addition to：
Threshold value is less than based on the collision time and given a warning.
8. a kind of system including that can install video camera and processor in a motor vehicle, the system can be operated to provide
Front shock is alerted, and the system, which can operate, to be come：
Multiple images frame is obtained by known time interval；
In at least one middle selection patch of described image frame；
Track the light stream between the picture frame of the multiple images point of the patch；
Described image point is fitted at least one model；And
Fitting based on described image point Yu at least one model, if it is expected that there is collision, it is determined that collision time
(TTC)。
9. system as claimed in claim 8, additionally it is possible to which operation comes：
Described image point is fitted to road surface model；
Fitting based on described image point Yu the road surface model, it is determined that expection does not have collision.
10. a kind of method for providing front shock warning, the method use video camera and the place that can be installed in a motor vehicle
Device is managed, methods described includes：
Multiple images frame is obtained by known time interval；
Select picture frame in patch, the patch correspond to the motor vehicle will after a predetermined interval of time location；
And
The patch is monitored, front shock warning is sent if object is imaged in the patch.
11. method as claimed in claim 10, in addition to：
Determine whether the object includes the part of substantial orthogonality.
12. method as claimed in claim 11, wherein described determine to perform by following operation：
Track the light stream between the picture frame of the multiple images point in the patch；And
Described image point is fitted at least one model.
13. at least a portion of method as claimed in claim 11, wherein described image point is modeled as being imaged from vertical
Object；And
Fitting based on described image point Yu at least one model, if it is expected that there is collision, it is determined that collision time
(TTC)。
14. method as claimed in claim 11, wherein at least one model includes road surface model, methods described is also wrapped
Include：
Described image point is fitted to road surface model；
Fitting based on described image point Yu the road surface model, it is determined that expection does not have collision.
15. method as claimed in claim 11, in addition to：
When best fit model is vertical surface model, the warning is sent.
16. a kind of system for being used to provide front shock warning in a motor vehicle, the system include：
Video camera, it can be arranged in the motor vehicle, and the video camera can operate to be obtained by known time interval
Multiple images frame；
Processor, it, which can be operated, comes：
Select picture frame in patch, the patch correspond to the motor vehicle will after a predetermined interval of time location；
Monitor the patch；And
If object is imaged in the patch, front shock warning is sent.
17. system as claimed in claim 16, wherein the processor can also operate to determine whether the object includes
The part of substantial orthogonality, it is described to determine to perform by following operation：
The multiple images point of the object in the patch is tracked between described image frame；
Described image point is fitted at least one model；And
Fitting based on described image point Yu at least one model, if it is expected that there is collision, it is determined that collision time
(TTC)。
18. system as claimed in claim 16, wherein the processor can be operated to be based on before TTC sends less than threshold value
Portion's conflict alert.
Applications Claiming Priority (3)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US42040510P | 2010-12-07 | 2010-12-07 | |
| US61/420,405 | 2010-12-07 | ||
| CN201110404574.1A CN102542256B (en) | 2010-12-07 | 2011-12-07 | The advanced warning system of front shock warning is carried out to trap and pedestrian |
Related Parent Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201110404574.1A Division CN102542256B (en) | 2010-12-07 | 2011-12-07 | The advanced warning system of front shock warning is carried out to trap and pedestrian |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107423675A true CN107423675A (en) | 2017-12-01 |
| CN107423675B CN107423675B (en) | 2021-07-16 |
Family
ID=46349111
Family Applications (2)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710344179.6A Active CN107423675B (en) | 2010-12-07 | 2011-12-07 | Advanced warning system for forward collision warning of traps and pedestrians |
| CN201110404574.1A Active CN102542256B (en) | 2010-12-07 | 2011-12-07 | The advanced warning system of front shock warning is carried out to trap and pedestrian |
Family Applications After (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201110404574.1A Active CN102542256B (en) | 2010-12-07 | 2011-12-07 | The advanced warning system of front shock warning is carried out to trap and pedestrian |
Country Status (1)
| Country | Link |
|---|---|
| CN (2) | CN107423675B (en) |
Families Citing this family (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6822563B2 (en) | 1997-09-22 | 2004-11-23 | Donnelly Corporation | Vehicle imaging system with accessory control |
| US5877897A (en) | 1993-02-26 | 1999-03-02 | Donnelly Corporation | Automatic rearview mirror, vehicle lighting control and vehicle interior monitoring system using a photosensor array |
| US7655894B2 (en) | 1996-03-25 | 2010-02-02 | Donnelly Corporation | Vehicular image sensing system |
| EP1504276B1 (en) | 2002-05-03 | 2012-08-08 | Donnelly Corporation | Object detection system for vehicle |
| US7526103B2 (en) | 2004-04-15 | 2009-04-28 | Donnelly Corporation | Imaging system for vehicle |
| US7972045B2 (en) | 2006-08-11 | 2011-07-05 | Donnelly Corporation | Automatic headlamp control system |
| DE102013213812A1 (en) * | 2013-07-15 | 2015-01-15 | Volkswagen Aktiengesellschaft | Device and method for displaying a traffic situation in a vehicle |
| EP3095073A1 (en) * | 2014-01-17 | 2016-11-23 | KPIT Technologies Ltd. | Vehicle detection system and method thereof |
| WO2018049643A1 (en) * | 2016-09-18 | 2018-03-22 | SZ DJI Technology Co., Ltd. | Method and system for operating a movable object to avoid obstacles |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6327536B1 (en) * | 1999-06-23 | 2001-12-04 | Honda Giken Kogyo Kabushiki Kaisha | Vehicle environment monitoring system |
| CN1576123A (en) * | 2003-07-03 | 2005-02-09 | 黄保家 | Anticollision system for motor vehicle |
| US7113867B1 (en) * | 2000-11-26 | 2006-09-26 | Mobileye Technologies Limited | System and method for detecting obstacles to vehicle motion and determining time to contact therewith using sequences of images |
| EP1837803A2 (en) * | 2006-03-24 | 2007-09-26 | MobilEye Technologies, Ltd. | Headlight, taillight and streetlight detection |
| CN101305295A (en) * | 2005-11-09 | 2008-11-12 | 丰田自动车株式会社 | Object detection device |
| CN101633356A (en) * | 2008-07-25 | 2010-01-27 | 通用汽车环球科技运作公司 | System and method for detecting pedestrians |
| CN101837782A (en) * | 2009-01-26 | 2010-09-22 | 通用汽车环球科技运作公司 | Be used to collide the multiple goal Fusion Module of preparation system |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7136506B2 (en) * | 2003-03-03 | 2006-11-14 | Lockheed Martin Corporation | Correlation based in frame video tracker |
| JP2005226670A (en) * | 2004-02-10 | 2005-08-25 | Toyota Motor Corp | Deceleration control device for vehicle |
| WO2005098782A1 (en) * | 2004-04-08 | 2005-10-20 | Mobileye Technologies Limited | Collision warning system |
| CN101261681B (en) * | 2008-03-31 | 2011-07-20 | 北京中星微电子有限公司 | Road image extraction method and device in intelligent video monitoring |
-
2011
- 2011-12-07 CN CN201710344179.6A patent/CN107423675B/en active Active
- 2011-12-07 CN CN201110404574.1A patent/CN102542256B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6327536B1 (en) * | 1999-06-23 | 2001-12-04 | Honda Giken Kogyo Kabushiki Kaisha | Vehicle environment monitoring system |
| US7113867B1 (en) * | 2000-11-26 | 2006-09-26 | Mobileye Technologies Limited | System and method for detecting obstacles to vehicle motion and determining time to contact therewith using sequences of images |
| CN1576123A (en) * | 2003-07-03 | 2005-02-09 | 黄保家 | Anticollision system for motor vehicle |
| CN101305295A (en) * | 2005-11-09 | 2008-11-12 | 丰田自动车株式会社 | Object detection device |
| EP1837803A2 (en) * | 2006-03-24 | 2007-09-26 | MobilEye Technologies, Ltd. | Headlight, taillight and streetlight detection |
| CN101633356A (en) * | 2008-07-25 | 2010-01-27 | 通用汽车环球科技运作公司 | System and method for detecting pedestrians |
| CN101837782A (en) * | 2009-01-26 | 2010-09-22 | 通用汽车环球科技运作公司 | Be used to collide the multiple goal Fusion Module of preparation system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN102542256B (en) | 2017-05-31 |
| CN102542256A (en) | 2012-07-04 |
| CN107423675B (en) | 2021-07-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10940818B2 (en) | Pedestrian collision warning system | |
| CN102542256B (en) | The advanced warning system of front shock warning is carried out to trap and pedestrian | |
| EP2463843B1 (en) | Method and system for forward collision warning | |
| US11741696B2 (en) | Advanced path prediction | |
| US10690770B2 (en) | Navigation based on radar-cued visual imaging | |
| US10115027B2 (en) | Barrier and guardrail detection using a single camera | |
| CN106663193B (en) | System and method for curb detection and pedestrian hazard assessment | |
| RU2629433C2 (en) | Device for detecting three-dimensional objects | |
| US6191704B1 (en) | Run environment recognizing apparatus | |
| US8810653B2 (en) | Vehicle surroundings monitoring apparatus | |
| US20190294893A9 (en) | Monocular cued detection of three-dimensional structures from depth images | |
| CN106462727A (en) | Systems and methods for lane end recognition | |
| Lin et al. | Lane departure and front collision warning using a single camera | |
| RU2635280C2 (en) | Device for detecting three-dimensional objects | |
| KR102031635B1 (en) | Collision warning device and method using heterogeneous cameras having overlapped capture area | |
| CN113792634B (en) | Target similarity score calculation method and system based on vehicle-mounted camera | |
| JPH08320999A (en) | Vehicle recognizing device | |
| JP2023089311A (en) | Information processing device, image capturing system, information processing method, and computer program | |
| JP2006286010A (en) | Obstacle detecting device and its method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |