JP5187292B2 - Vehicle periphery monitoring device - Google Patents

Description

  The present invention relates to a vehicle periphery monitoring device that displays an obstacle on a bird's-eye view image of a host vehicle and performs driving support.

Conventionally, a vehicle is provided with a plurality of imaging devices, and images taken by the imaging devices are combined, an overhead view image of the entire circumference of the vehicle is generated and displayed on a driver's seat monitor, thereby enabling driving operations in a parking lot or the like. There is a vehicle periphery monitoring device that can be easily and safely performed.
In such a conventional vehicle periphery monitoring device, a front camera, a rear camera, a right camera, and a left camera are attached to the vehicle.
Then, the viewpoint images of the images captured by the front camera, the rear camera, the right camera, and the left camera are subjected to viewpoint conversion, and further an image synthesis is performed to generate an overhead view image.
The bird's-eye view image generated in this way is displayed on the bird's-eye view image display screen and is used to support driving operations.

As such a vehicle periphery monitoring device, it has an imaging unit, a display unit that provides information to the driver, a start scene selection unit, a vehicle speed sensor, a shift position detection unit, a winker detection unit, a steering angle, A device including a sensor and a screen mode switching unit is provided.
The vehicle periphery monitoring device can change the rear side area based on the blinker information and the steering wheel steering angle information. In particular, when the blinker input is detected in the high speed mode, the turn signal direction synthesis is performed. Zoom in on the image.
In addition, the said imaging part images the vehicle periphery with each camera attached to the several camera installation part installed in the vehicle, and the said display part provides a driver | operator with information.
The start scene selection unit selects a normal travel scene from a plurality of driving scenes when starting the engine, the vehicle speed sensor detects a vehicle speed, and the shift position detection unit detects a shift position.
The winker detection unit detects the direction of the winker, the steering angle sensor detects a steering angle of the steering wheel, and the screen mode switching unit switches a display screen configuration based on information detected by each sensor and each detection unit. (See Patent Document 1).

JP 2002-109697 A

  Therefore, in the conventional vehicle periphery monitoring device, a target to be focused on in the vicinity of the boundary on the bird's-eye view image that has been subjected to viewpoint conversion and image synthesis, in particular, a three-dimensional target, is detected at the boundary on the viewpoint video at the image synthesis processing stage. A state occurs in which almost no image is displayed. For this reason, there is a problem that a target near the boundary on the viewpoint image is overlooked on the overhead image display screen.

The present invention has been made in view of such circumstances, and for a target that is not entirely displayed on the overhead image, the boundary around the overhead image can be changed to display the entire vehicle periphery. An object is to provide a monitoring device.
It is another object of the present invention to provide a vehicle periphery monitoring device that can determine a target located on an expected route of the host vehicle and perform appropriate driving support.

According to the first aspect of the present invention, the vehicle is mounted from a plurality of imaging devices that are mounted at a plurality of locations in the vehicle and that images the surroundings of the vehicle in different directions from the locations, and the vehicle surroundings captured by the imaging devices. A bird's-eye view image composition processing unit that synthesizes a bird's-eye view image, a bird's-eye view image display unit that displays the bird's-eye view image synthesized by the bird's-eye view image composition processing unit, and the bird's-eye view image composition processing unit of the edge or the like of the corner or the image of the image, the feature point extraction unit tracking of changes in the position on the image processing technique for extracting as a feature point a portion having a comparison Tekiyo easy characteristics, chronological image of the overhead view image The feature point tracking processing unit that tracks the change in the position of the feature point on the overhead image extracted by the feature point extraction unit from the feature point, and the feature point tracked by the feature point tracking processing unit 3D measurement processing unit for calculating feature point information including 3D coordinate information of the feature point and relative motion information with respect to the vehicle based on a change in position on the image, and the 3D measurement processing unit An obstacle detection processing unit that determines whether the image from which the feature points have been extracted is an obstacle based on the feature point information calculated in step 5; a vehicle speed, a steering angle, a transmission shift position, an accelerator opening the expected route determination processing unit for determining the predicted path of the vehicle based on the status information of the vehicle including all degrees, on the expected path of the vehicle which is determined by the expected route determination processing unit, the obstacle An obstacle determination processing unit on the predicted route that determines whether or not an obstacle detected by the detection processing unit is located, and an obstacle is positioned on the predicted route of the vehicle in the obstacle determination processing unit on the predicted route Is determined to be Characterized by comprising an alarm output means for outputting an alarm.

According to the present invention, for a target that is located near the boundary on the viewpoint image and is not entirely displayed on the overhead image, the boundary line on the overhead image can be changed and displayed as a whole, and appropriate driving support is performed. There is an effect that it is possible to provide a vehicle periphery monitoring device that can be used.
Further, there is an effect that it is possible to provide a vehicle periphery monitoring device that can determine a target located on the predicted route of the host vehicle and perform appropriate driving support.

It is a block diagram which shows the structure of the vehicle periphery monitoring apparatus which is embodiment of this invention. It is explanatory drawing which shows the imaging range of each camera of the vehicle periphery monitoring apparatus of embodiment of this invention, and the bird's-eye view image produced | generated by carrying out viewpoint conversion of the image of these imaging ranges, and image synthesis. It is explanatory drawing which shows an example of the bird's-eye view image when the boundary line of the bird's-eye view image in the vehicle periphery monitoring apparatus of embodiment of this invention is changed. It is explanatory drawing which shows a bird's-eye view image when the obstacle located in the common area | region in the vehicle periphery monitoring apparatus of embodiment of this invention is contained in an estimated path | route. It is explanatory drawing which shows a bird's-eye view image when the obstruction in the vehicle periphery monitoring apparatus of embodiment of this invention is located out of a common area | region. It is explanatory drawing which shows the relationship between the three-dimensional space coordinates of the target object for demonstrating the obstruction detection in the vehicle periphery monitoring apparatus of embodiment of this invention, and the position on a bird's-eye view image. It is explanatory drawing which shows an example of the bird's-eye view image in the time t and the time t + F (DELTA) t in the vehicle periphery monitoring apparatus of embodiment of this invention. It is explanatory drawing which showed the motion of the feature point extracted from the bird's-eye view image in the vehicle periphery monitoring apparatus of embodiment of this invention with the vector. It is a flowchart which shows operation | movement of the vehicle periphery monitoring apparatus of embodiment of this invention. It is a flowchart which shows operation | movement of the vehicle periphery monitoring apparatus of embodiment of this invention. It is explanatory drawing which shows the approaching moving object which passes the front of the own vehicle in the vehicle periphery monitoring apparatus of embodiment of this invention. It is explanatory drawing which shows the obstacle area A by the distance in the vehicle periphery monitoring apparatus of embodiment of this invention, and the obstacle area B by the collision estimated time. It is a flowchart which shows operation | movement of the three-dimensional measurement process part and the obstruction detection process part in the vehicle periphery monitoring apparatus of embodiment of this invention. It is a figure which shows the numerical formula and matrix regarding a constraint matrix. It is operation | movement explanatory drawing of the warning output control part according to the predicted path | route of the own vehicle determined in the predicted path | route determination process part in the vehicle periphery monitoring apparatus of embodiment of this invention, and the expected locus | trajectory of a feature point. It is operation | movement explanatory drawing of the warning output control part according to the prediction path | route of the own vehicle determined in the prediction path | route determination process part in the vehicle periphery monitoring apparatus of embodiment of this invention, the prediction locus | trajectory of a feature point, and a bird's-eye view image display part. . It is operation | movement explanatory drawing of the warning output control part according to the prediction path | route of the own vehicle determined in the prediction path | route determination process part in the vehicle periphery monitoring apparatus of embodiment of this invention, the prediction locus | trajectory of a feature point, and a bird's-eye view image display part. . It is explanatory drawing of partial differentiation A and B. FIG.

Embodiments of the present invention will be described below. FIG. 1 is a block diagram showing a configuration of a vehicle periphery monitoring apparatus according to an embodiment of the present invention. This vehicle periphery monitoring apparatus includes a front camera 1, a rear camera 2, a right camera 3, and a left camera 4, frame memories 11, 12, 13, and 14, viewpoint conversion units 21, 22, 23, and 24, and a bird's-eye view image composition. A processing unit 31 is provided.
Further, a feature point extraction unit 41, a feature point tracking processing unit 42, a three-dimensional measurement processing unit 43, an obstacle detection processing unit 44, an obstacle boundary region determination unit 45, and a camera image priority processing unit 46 are provided.
The predicted route determination processing unit 51, the predicted route obstacle determination processing unit 52, the warning output control unit 53, the warning sound output unit 54, the buzzer 55, the overhead image display unit 56, the display 57, and the sudden acceleration suppression control unit 59 are provided. I have.
The front camera 1 is a camera that is attached to, for example, the center of the front grill at the front of the vehicle and images the front of the host vehicle.
The rear camera 2 is a camera that is attached to, for example, the rear portion of the vehicle provided with a rear window and images the rear of the host vehicle.
The right camera 3 is a camera that is attached to a protruding end of a door mirror support part that supports a right door mirror, for example, and images the right side of the vehicle.
The left camera 4 is a camera that is attached to a protruding end of a door mirror support portion that supports a left door mirror, for example, and images the left side of the vehicle.
The frame memory 11 temporarily stores an image signal captured by the front camera 1 and converted into digital data in units of frames.
The frame memory 12 temporarily stores an image signal captured by the rear camera 2 and converted into digital data in units of frames.
The frame memory 13 temporarily stores an image signal captured by the right camera 3 and converted into digital data in units of frames.
The frame memory 14 temporarily stores an image signal captured by the left camera 4 and converted into digital data in units of frames.
The viewpoint conversion unit 21 converts the viewpoint position of the image captured by the front camera 1 from the attachment position of the front camera 1 to a virtual viewpoint position above the host vehicle.
The viewpoint conversion unit 22 converts the viewpoint position of the image captured by the rear camera 2 from the attachment position of the rear camera 2 to the virtual viewpoint position above the host vehicle.
The viewpoint conversion unit 23 converts the viewpoint position of the image captured by the right camera 3 from the attachment position of the right camera 3 to a virtual viewpoint position above the host vehicle.
The viewpoint conversion unit 24 converts the viewpoint position of the image captured by the left camera 4 from the attachment position of the left camera 4 to a virtual viewpoint position above the host vehicle.
The overhead image synthesis processing unit 31 is captured by the front camera 1 and the viewpoint changed by the viewpoint conversion unit 21, the image captured by the rear camera 2 and the viewpoint changed by the viewpoint conversion unit 22, and the right camera 3. The image whose viewpoint is changed by the viewpoint conversion unit 23 and the image captured by the left camera 4 and whose viewpoint is changed by the viewpoint conversion unit 24 are combined to generate an overhead image when the host vehicle is viewed from the virtual viewpoint position. .

FIG. 2 shows the imaging ranges captured by the front camera 1, the rear camera 2, the right camera 3, and the left camera 4 attached to the vehicle 101, and the images in these imaging ranges are generated by performing viewpoint conversion and image synthesis. It is explanatory drawing which shows the bird's-eye view image displayed on the display 57 of a monitor apparatus centering | focusing on the own vehicle.
In the figure, reference numeral 111 denotes an imaging range by the front camera 1, reference numeral 112 denotes an imaging range by the rear camera 2, reference numeral 113 denotes an imaging range by the right camera 3, and reference numeral 114 denotes an imaging range by the left camera 4.
Reference numeral 116 indicates a shared visual field region in which the imaging range of the front camera 1 and the imaging range of the right camera 3 overlap. Reference numeral 117 indicates a shared visual field region where the imaging range of the right camera 3 and the imaging range of the rear camera 2 overlap. Reference numeral 118 denotes a shared visual field region in which the imaging range of the rear camera 2 and the imaging range of the left camera 4 overlap. Reference numeral 119 denotes a shared visual field region where the imaging range of the left camera 4 and the imaging range of the front camera 1 overlap.
Reference numeral 121 denotes a front partial overhead image that is converted from the viewpoint of the imaging range 111 of the front camera 1 by the viewpoint conversion unit 21 and displayed on the display 57 of the monitor device.
Reference numeral 122 denotes a right side partial overhead view image that is similarly converted from the viewpoint of the image in the imaging range 112 of the rear camera 2 and displayed on the display 57 of the monitor device.
Reference numeral 123 denotes a rear partial overhead image which is similarly converted from the viewpoint of the image in the imaging range 113 of the right camera 3 and displayed on the display 57 of the monitor device.
Reference numeral 124 indicates a left side partial overhead view image that is similarly converted from the viewpoint of the image in the imaging range 114 of the left camera 4 and displayed on the display 57 of the monitor device.
The front partial overhead image 121, the right partial overhead image 123, the rear partial overhead image 122, and the left partial overhead image 124 are overhead images when the surroundings of the host vehicle are viewed from the virtual viewpoint position around the host vehicle 101. Images are synthesized by the overhead image synthesis unit 31.
In FIG. 2, reference numeral 126 denotes a common area in which the front partial overhead image 121 and the right side partial overhead image 123 are superimposed, and reference numeral 127 denotes a common area in which the right side partial overhead image 123 and the rear partial overhead image 122 are superimposed. Reference numeral 128 denotes a common area where the rear partial overhead image 122 and the left side partial overhead image 124 overlap, and reference numeral 129 denotes a common area where the front partial overhead image 121 and the left partial overhead image 124 overlap.

Returning to FIG. 1, the feature point extraction unit 41 compares the tracking of position changes in the image processing technique, such as an image corner or an image edge, with respect to the overhead image synthesized by the overhead image synthesis processing unit 31. extracting a portion having a Tekiyo easy features as feature points.
The feature point tracking processing unit 42 tracks a change in the movement position of the feature point accompanying a time change of the overhead image.
The three-dimensional measurement processing unit 43 calculates relative motion information and three-dimensional coordinate information of the feature point from a change in the movement position of the feature point on the overhead view video.
The relative motion information and the three-dimensional coordinate information are calculated using equations (1) and (2) that model the movement of the target object on the overhead image shown below.

  The obstacle detection processing unit 44 determines an obstacle based on the calculated relative motion information and three-dimensional coordinate information. In this obstacle determination, an obstacle is detected from feature points in the obstacle area based on the distance to the host vehicle and the obstacle area based on the estimated collision time.

The obstacle display area determination unit 45 includes the position of the obstacle detected by the obstacle detection processing unit 44 in the common areas 126, 127, 128, and 129 of the overhead image described with reference to FIG. Determine which common area we are in.
The camera image priority processing unit 46, based on the relative motion information and the three-dimensional coordinate information calculated by the three-dimensional measurement processing unit 43 for the feature points of the common area, the feature points when a certain elapsed time is assumed. Find the expected trajectory.
When the obstacle moves along the predicted locus, a camera that uses the common area determined by the obstacle display area determination unit 45 as a visual field area, for example, the front camera 1 and the left camera in the case of the common area 129 As shown in FIG. 4, a prediction result of whether or not the image is captured by each camera on which the imaging range is superimposed is obtained.
Further, with respect to the common region, priority is given to a partial overhead view image of a camera that captures the obstacle in a wider range, and the overhead view image synthesis processing unit 31 causes the overhead view image to be synthesized.
The predicted route determination processing unit 51 is based on the vehicle information based on the vehicle speed, the steering wheel operation angle, the transmission shift position, and the accelerator opening, or the state information of the vehicle including any one of the parameters. The expected route is determined.
The obstacle determination processing unit 52 on the predicted route determines whether or not the obstacle detected by the obstacle detection processing unit 44 is located on the predicted route of the host vehicle determined by the prediction route determination processing unit 51. .
The warning output control unit 53 instructs the warning sound output unit 54 to output a warning sound when the obstacle determination processing unit 52 on the predicted route determines that an obstacle is located on the predicted route of the host vehicle. Outputs alarm sound output control signal.
In addition, a warning display control signal for displaying a warning indicating that an obstacle is located on the predicted route of the host vehicle is output to the overhead image display unit 56 on the display 57 of the monitor device displaying the overhead image.
The warning sound output unit 54 operates the buzzer 55 based on the warning sound output control signal to output a warning sound.
The overhead image display unit 56 displays the overhead image synthesized by the overhead image synthesis processing unit 31 on the display 57 and displays a warning on the display 57 based on the warning display control signal output from the warning output control unit 53. I do.
The sudden acceleration suppression control unit 59 suppresses sudden acceleration if there is a sudden accelerator operation when the obstacle determination processing unit 52 determines that an obstacle is located on the predicted route of the host vehicle. A sudden acceleration suppression request is output to the engine ECU 60 to control the engine and suppress sudden acceleration.
Detection of this rapid accelerator operation can be realized as follows, for example.
That is, it is determined whether or not the change rate of the accelerator opening exceeds a certain reference value, and when the change rate of the accelerator opening exceeds a certain reference value, a sensor circuit that outputs a sudden acceleration accelerator operation determination signal is detected for the accelerator operation. Provide as a means.
Alternatively, it is possible to detect whether or not there has been a sudden accelerator operation based on the output of the kick-down switch as an accelerator operation detection means.
Viewpoint conversion units 21, 22, 23, 24, overhead image synthesis processing unit 31, feature point extraction unit 41, feature point tracking processing unit 42, three-dimensional measurement processing unit 43, obstacle detection processing unit 44, obstacle display region determination The unit 45 is realized by an ECU having a microcomputer as a main component.
The camera image priority processing unit 46, the predicted route determination processing unit 51, the obstacle determination processing unit 52 on the predicted route, the warning output control unit 53, the overhead image display unit 56, and the rapid acceleration suppression control unit 59 are mainly microcomputers. This is realized by an ECU having a different configuration.
The ECU performs various processes for realizing functions unique to the vehicle periphery monitoring device, including viewpoint conversion processing and image synthesis processing of images captured by the front camera 1, the rear camera 2, the right camera 3, and the left camera 4. Do.

  Here, the derivation of the equation (2) indicating the motion model of the three-dimensional object on the overhead image when the obstacle is detected by the three-dimensional measurement processing unit 43 and the obstacle detection processing unit 44, and the equation from the equation (2) Derivation of (3) will be described.

As shown in FIG. _{6, [X p, Y p} , Z p] the position vector Xp of a three-dimensional real space of the feature point p is ^T, the camera at time t [0,0, H] to the position of Suppose there is. Here, H is the height of the camera with respect to the road surface.
_{[X p, y p, 0} ] position vector _{x p} on the overhead image of the feature point p when ^{T, the} vector _{Xp = [X p, Y p} , Z p] T and the vector _x p = _[x p, Equation (1) holds between y _p , 0] and ^T.
Here, if k _p = H / (H−Z _p ), Expression (1) can be rewritten as Expression (4) shown below.

On the other hand, a true bird's-eye view image is when the camera is at an infinite height (H = ∞), and k _p = 1, and from Equation (4), Equation (5) is obtained.

That is, the feature point position vector x _p = [x _p , y _p , 0] ^T reflected in the overhead view video is the camera height and the feature point with respect to the true overhead view position [X _p , Y _p , 0] ^T. Is multiplied by a coefficient k _p corresponding to the height of.
Now, the object p is at the position of the vector Xp = [X _p , Y _p , Z _p ] ^T in the three-dimensional real space at time t with respect to the camera, and is relative to the yaw rate ω and the relative velocities V _X and V _Y. Suppose you were moving by exercise. It is assumed that there was no bouncing, roll, or pitching movement.
At this time, the position vector Xp (t + Δt) of the object p at the time t + Δt in the three-dimensional real space = [X _p (t + Δt), Y _p (t + Δt), Z _p (t + Δt)] ^T is given by equation (6) as follows.

From Equation (6), if there is no bouncing, roll, or pitching motion, Z _p (t) = Z _p (t + Δt), and k _p = H / (H−Z _p ), the coefficient k _p Is constant regardless of time. Equation (6) can be rewritten as Equation (7) as follows.

By the way, the position vector x _p = [x _p (t), y _p (t), 0] ^T on the bird's-eye view image of the object p at time t and t + Δt, the vector x _p = [x _p (t + Δt). , Y _p (t + Δt), 0] ^T is given by the following equation (8) from equation (5) because the coefficient k _p is constant regardless of time.

A position vector x _p (t + Δt) = [x _p (t + Δt), y _p (t + Δt), 0] ^T on the bird's-eye view image of the object p at time t + Δt is expressed by Equation (7). , (8), the result is as shown in equation (9).

As a result, the position vector x _p (t) = [x _p (t), y _p (t), 0] ^T on the bird's-eye view image of the object p at the times t and t + Δt, the vector x _p (t + Δt) = [x _p (t + Δt), y _p (t + Δt), 0] The following relational expression (10) holds for ^T.

Now, it is assumed that the relative motion of the object p with respect to the camera is caused by the fact that the object p is a stationary object and the camera moves. In other words, it is caused by the movement of the host vehicle. Further, it is assumed that there are P feature points in the stationary object shown in the overhead view video in addition to the object p.
At this time, since the relative motion to the stationary object is common, the position vector x _p (t) = [x _p (t), y _p (t), 0] ^T , vector x _p (t + Δt) = [x _p (t + Δt), y _p (t + Δt), 0] ^T , 1 <p <P is obtained from equation (10) as follows: It is given by (11).

  Equation (11) can be rewritten as Equation (12) as follows.

Here, the following expressions (13) and (14) are _satisfied , and the vectors m (t + Δt), n (t + Δt), c _x (t + Δt), and _cy (t + Δt) are times. It shows the relative movement rotational motion and the translational motion of the camera and the object at time t + Delta] t from t, the vector s _{p (t)} is the three-dimensional feature point p at substantially time t from equation (4) It can be regarded as a position in real space.

By the way, relative motion information m (t + 2Δt), n (t + 2Δt), c _x (t + 2Δt), and c _y (t + 2Δt) between the camera and the object from time t to time t + 2Δt are obtained. If obtained, the position vector x _p (t + 2Δt) = [x _p (t + 2Δt), y _p (t + 2Δt), 0] ^{T at} the time t + 2Δt, 1 <p <P is given by equation (15) as follows from equation (12).

  When formulas (12) and (15) are put together, formula (16) shown below is obtained.

  Now, it is assumed that the positions of P feature points in total are observed on the bird's-eye view video from time t to t + FΔt at time interval Δt. This observation is performed by extracting and tracking feature points. Then, equation (16) is expanded to obtain the following relational equation (17).

Expression (17) is a mathematical model of the movement of the three-dimensional object on the overhead view video.
Equation (17) can be rewritten as a matrix product as shown in Equation (18) below.

  Here, the matrices W, M, and S are given as shown in the following equation (19), and the matrix W represents the feature points on the overhead video from time t to t + FΔt obtained by feature point extraction / tracking. The matrix M represents the relative motion from time t to t + FΔt, and the matrix S represents the position of each feature point in the three-dimensional real space at time t.

  Using the relational expression of Expression (18), the relative motion and the position of the three-dimensional real space are obtained from the change of the position of the feature point on the overhead image.

Next, an outline of the model estimation method will be described.
In order to divide the matrix W indicating the change of the position of the feature point on the overhead view video into the matrix M indicating the relative motion and the matrix S indicating the three-dimensional shape from W = MS shown in Expression (18), the following procedure is used. Do.
From Equation (19), the matrix W is a matrix having a size of 2F × P. The matrix M is a 2F × 3 matrix because the vectors m and n are two-dimensional, and the matrix S is a 3 × P matrix because the vector s is three-dimensional. That is, the matrix W is a rank 3 matrix. Therefore, singular value decomposition is once performed on the matrix W as follows.

Here, U is a 2F × H orthogonal matrix, Σ is a H × H diagonal matrix, ^VT is a H × P orthogonal matrix, and H is given as H = min {2F, P}. Further, it is assumed that diagonal elements of Σ are arranged in descending order as σ ₁ > σ ₂ >... Σ _p .
Since the matrix W is a matrix of rank 3, since σ ₄ can be regarded as a noise component, three columns of the matrix U are extracted as shown in Expression (21).

  From this, a temporary decomposition result can be obtained as shown in equation (22).

  Next, the constraint matrix is calculated. The decomposition result of Expression (22) satisfies (Expression 1) of FIG. 14, but the obtained matrix (1) of FIG. 14 generally does not satisfy (Expression 2) of FIG. For example, even if (matrix 1) in FIG. 14 is converted as shown in the following equation (23) using a matrix 3 having an appropriate size of 3 × 3, W = MS is satisfied. That is, it is necessary to select the matrix A appropriately.

In order to select the matrix A appropriately, the following constraint conditions are introduced.
(A) From the constraint equation (19) of the rotation component, the matrix M is configured as shown by the following equation (24).

  Here, since the vector mn indicates a rotational component as shown in the equation (13), the following constraint condition exists.

(B) Road surface constraint condition From equation (19), the matrix S is constructed as follows.

Further, the vector S _P (t) is S _P (t) = [x _p (t) y _p (t) k _p ] ^T from the equation (14), and k _p is k _p = H / (H−Z _p ). Here, k _p is a condition of Z _p <H, and if Z _p increases, k _p also increases. That is, as long as the object is below the camera, the higher the object, the greater the value of k _p .
k _p is minimized when the object is on the road surface, that is, Z _p = 0, and then k _p = 1. Since the bird's-eye view video always shoots the road surface, there is always a feature point that satisfies k _p = 1 on the bird's-eye view video. That is, the following constraints exist for k _p.

  As described above, the matrix A may be determined so as to satisfy the expressions (25) and (27).

Next, the operation will be described.
FIG. 6 is an explanatory diagram showing the relationship between the three-dimensional spatial coordinates of the object and the position on the overhead image for explaining the obstacle detection in the vehicle periphery monitoring apparatus of this embodiment.
As shown in FIG. 6, the camera 201 is disposed at a height H on the z axis. In FIG. 6, reference numeral 300 denotes an object, and its three-dimensional spatial coordinates are [X _p , Y _p , Z _p ] ^T. Reference numeral 301 denotes an object image on the bird's-eye view image, and its three-dimensional spatial coordinates are [x _p , y _p , 0] ^T. Reference numeral 302 denotes a planar bird's-eye view image including the captured object.
When the object and the vehicle are moving relative to each other at the relative speed [V _X , V _Y ] ^T and the yaw rate ω, the relationship expressed by the above equation (2) is established.
The vectors m (t + Δt) and n (t + Δt) in equation (2) are relative motions that store the relative velocity and yaw rate, and the vector _SP is obtained from the above equation (1) to the three-dimensional coordinates of the feature point P. Conversion is possible uniquely.
In this embodiment, using the relationship of Expression (1) and Expression (2), the feature point P on the bird's-eye view image is converted into three-dimensional coordinates, and a solid object with respect to the host vehicle is determined to detect an obstacle. Do. When the detected obstacle is in the common area on the overhead image and the image is not entirely displayed on the overhead image, the obstacle is copied more widely from the expected locus of the obstacle. Priority is given to the image of the camera which can perform, the boundary line of the bird's-eye view image is changed so that the whole obstacle is displayed, and the bird's-eye view image is displayed again.
Also, it is determined whether or not the detected obstacle is on the predicted route of the host vehicle. If the detected obstacle is on the predicted route, a warning is given by a display or sound. The engine control which suppresses is realized.

Hereinafter, with reference to the flowchart shown in FIG. 9, the obstacle detection in the vehicle periphery monitoring device of this embodiment, and the boundary line of the overhead image is changed so that the entire obstacle is displayed, The operation when redisplaying will be described.
As shown in FIG. 2, the images captured by the front camera 1, the rear camera 2, the right camera 3, and the left camera 4 are subjected to viewpoint conversion by the viewpoint conversion units 21, 22, 23, and 24, and further, an overhead image synthesis processing unit. An image is synthesized by 31 and an overhead image is generated. This overhead image is generated at regular time intervals Δt and output from the overhead image synthesis processing unit 31 (step S1). The overhead image output from the overhead image synthesis processing unit 31 is displayed and output to the display 57 by the overhead image display unit 56. On the other hand, the overhead image output from the overhead image synthesis processing unit 31 is supplied to the feature point extraction unit 41, and the feature point P is extracted (step S2). Extraction of the feature point P in the feature point extraction unit 41 extracts, for example, an image such as edges of the corners or image, the location tracking of changes in the position on the image processing technique has a comparison Tekiyo easy feature as the feature point P To do.

FIG. 7 is an explanatory diagram showing an example of an overhead view image at time t and time t + FΔt in the vehicle periphery monitoring device of this embodiment. In FIG. 7, a symbol P indicates a feature point extracted by the feature point extraction unit 41.
As shown in FIGS. 7A and 7B, the feature point tracking processing unit 42 performs tracking processing to determine how the feature point P extracted by the feature point extracting unit 41 moves.
The tracking process of the feature point tracking processing unit 42 is performed as follows.
That is, what is the feature point P from the positional relationship between the feature point P on the overhead image at time t shown in FIG. 7A and the feature point P on the overhead image at time t + FΔt shown in FIG. A tracking process is performed to determine the position of the movement (step S3).
In the tracking process of the feature point P, the motion vector is obtained by calculating the optical flow of the feature point P.
FIG. 8 is an explanatory diagram showing motion vectors of feature points extracted from the overhead image shown in FIG.
The calculation process of the optical flow of the feature point P is performed as follows.
When a coordinate change of a feature point P that is common to each other is detected on the overhead image captured at a predetermined period and time interval, whether the feature point P is moving or not, Then, the movement of the feature point P, the direction of the coordinate change and the extent of the magnitude are calculated.

Next, in the three-dimensional measurement processing unit 43, the relative motion information between the host vehicle and the feature point P from the change in the position of the feature point P on the overhead image using the formula (2), and the three-dimensional of the feature point P are used. Coordinate information is calculated (step S4).
Further, the obstacle detection processing unit 44 detects an obstacle from the relative motion information calculated by the three-dimensional measurement processing unit 43 and the three-dimensional coordinate information (step S5).

Here, the three-dimensional measurement processing of the three-dimensional measurement processing unit 43 and the obstacle detection processing of the obstacle detection processing unit 44 will be further described.
FIG. 12 is an explanatory diagram showing an obstacle area A based on the distance to the host vehicle 101 detected from the three-dimensional information and relative motion information of the three-dimensional object obtained from the overhead view image, and an obstacle area B based on the estimated collision time. is there.
FIG. 13 is a flowchart showing the operations of the three-dimensional measurement processing unit 43, the obstacle detection processing unit 44, and the predicted route determination processing unit 51 in the vehicle periphery monitoring device of this embodiment.
FIG. 15 is an operation explanatory diagram of the warning output control unit 53 according to the predicted route of the host vehicle and the predicted trajectory of the feature point determined by the predicted route determination processing unit 51.
16 and 17 are explanatory diagrams of operations of the warning output control unit 53 and the bird's-eye view image display unit 56 according to the predicted route of the host vehicle determined by the predicted route determination processing unit 51 and the predicted trajectory of the feature point.
In the three-dimensional measurement process of the three-dimensional measurement processing unit 43, as described above, the three-dimensional coordinate information of the feature points on the overhead image and the relative motion information between the feature points and the host vehicle 11 are expressed by the equations (1) and (1). Calculate from (2) (step S41).
The three-dimensional coordinate information of the feature points is obtained as a three-dimensional position matrix S (step S42), and the relative motion information between the feature points and the host vehicle 101 is obtained as a relative motion matrix M (step S43). .
Then, threshold processing based on the height in the z-axis direction is performed on the three-dimensional coordinate information of the feature points obtained as the three-dimensional position matrix S (step S44).
In this threshold value processing, feature points having a value whose height information in the z-axis direction is not zero are determined as obstacle candidate feature points based on a predetermined height reference value (step S45). A feature point whose direction height information is zero is set as a non-obstacle feature point (step S46).
And about the said obstacle candidate feature point, since the distance between the own vehicle 101 becomes clear based on the three-dimensional coordinate information of the feature point, between the own vehicle 101 and an obstacle candidate feature point. Threshold processing is performed based on a predetermined distance reference value for the distance (step S47). Then, the long-distance obstacle feature point and the short-distance obstacle feature point are identified (step S48, step S49).
The feature points identified as short-distance obstacle feature points are, for example, feature points included in the obstacle area A shown in FIG.

For the feature points identified as long-distance obstacle feature points, threshold processing is performed based on a predetermined expected collision time reference value for the expected collision time based on the relative motion matrix M obtained in step S43. (Step S51). Then, the feature points are identified as low-risk obstacle feature points and early-arrival obstacle feature points (steps S52 and S53).
The early-arrival obstacle feature points are, for example, feature points included in the obstacle area B shown in FIG.
Subsequent Step S61, Step S62, Step S71, Step S72, and Step S81 show processing for determining the predicted route of the host vehicle and the predicted trajectory of the feature point in the predicted route determination processing unit 51.
The predicted route determination processing unit 51 determines whether to output a warning based on the determination result of the predicted route of the host vehicle and the predicted trajectory of the feature point, which camera image is given priority, and the boundary line of the overhead image is changed. Determine.
The warning output control unit 53 performs warning output control based on the determination result of the predicted route determination processing unit 51 as shown in FIG. 15, and the overhead image display unit 56 performs control as shown in FIGS. The boundary line of the overhead image is changed based on the determination result of the predicted route determination processing unit 51.
That is, the predicted route determination processing unit 51 includes state information including all of the vehicle speed, steering wheel operation angle, and transmission shift position of the host vehicle, and the relative motion information of the feature points calculated by the three-dimensional measurement processing unit 43. Based on the above, the predicted route of the host vehicle and the predicted trajectory of the feature point are determined, and the obstacle determination processing unit 52 on the predicted route is on the predicted route of the host vehicle determined by the predicted route determination processing unit 51. Similarly, the predicted path determination processing unit 51 determines the predicted trajectory, and the obstacle detection processing unit 44 determines whether or not the feature point detected as an obstacle is located.

In addition, as shown in FIG. 17, even if a feature point and an obstacle have the same relative motion, whether or not a warning is output depending on the positional relationship between the feature point and the obstacle and the host vehicle, Judgment results for changes vary.
Therefore, the predicted route determination processing unit 51 includes state information including all of the vehicle speed, the steering wheel operation angle, and the transmission shift position of the host vehicle, the relative motion information of the feature points calculated by the three-dimensional measurement processing unit 43, and Based on the positional relationship between the host vehicle and the feature point, the predicted route of the host vehicle and the predicted trajectory of the feature point are determined, and the obstacle determination processing unit 52 on the predicted route is a predicted route determination processing unit 51. In the same way, the predicted path is determined by the predicted path determination processing unit 51 on the predicted route of the host vehicle, and it is determined whether or not the feature point detected as an obstacle by the obstacle detection processing unit 44 is located. .

Returning to the flowchart of FIG. 9, the obstacle display area determination unit 45 causes the obstacle on the detected obstacle to be located in any common area 126, 127, 128, 129 shown in FIG. Whether it is included is determined from the feature points of the obstacle (step S6).
If the obstacle display area determination unit 45 determines that the feature point of the obstacle is included in any one of the common areas 126, 127, 128, and 129, for example, the common area 129, camera image priority The processing unit 46 operates as follows.
The camera image priority processing unit 46 calculates the time from the relative motion information calculated by the three-dimensional measurement processing unit 43 and the feature point P of the obstacle and the three-dimensional position information of the feature point P of the obstacle. A three-dimensional position and an expected trajectory of the feature point P of the obstacle when Δt has elapsed are calculated.
Then, for the front camera 1 and the left camera 4 having the common area 129 determined to include the feature point of the obstacle as a common viewing area, the obstacle is reflected at the calculated three-dimensional position. Whether or not there is a certain elapsed time is determined for the feature point of the obstacle.
Then, from the magnitude relationship between the weighted average of the feature points that appear only in the front camera 1 and the weighted average of the feature points that appear only in the left camera 4 at the assumed constant elapsed time, The image of the camera that can capture a wider range is determined.
Further, priority is given to the determined camera image for the common area of the overhead image so that the entire obstacle is displayed, and the boundary line of the overhead image is changed (step S7).
Then, the bird's-eye view image synthesis processing unit 31 redisplays the bird's-eye view image including the entire obstacle (step S8).

FIG. 3 is an explanatory diagram illustrating an example of an overhead image when a boundary line of the overhead image is changed so that the entire obstacle is displayed when the entire obstacle is not displayed on the overhead image.
FIG. 3A shows an overhead image in which the entire obstacle is not displayed, and reference numeral 131 indicates an obstacle in which the entire obstacle is not displayed.
When the front camera 1 hardly captures the whole image, but the left camera 4 has an obstacle in the common area 129 where the whole image is reflected, for example, the obstacle is located at a position indicated by reference numeral 132 in FIG. The case where it is doing is demonstrated.
At this time, if the arrangement form of each region defined by the boundary line of the overhead image has the configuration shown in FIG. 3A, the obstacle 131 is only partially displayed on the overhead image.
Note that the areas defined by the boundary lines of the overhead image are areas of the front partial overhead image 121, the rear partial overhead image 122, the right side partial overhead image 123, and the left side partial overhead image 124 of FIG.
Such a phenomenon occurs according to the position of the obstacle 131 around the common visual field area indicated by reference numerals 116, 117, 118, and 119 in FIG. 2 and the arrangement form of each area of the partial overhead image at this time. However, as shown in FIG. 3, the obstacle 131 is only partially displayed on the overhead image.
For this reason, if the obstacle 131 is located in the common areas 126, 127, 128, and 129 that cover the common visual field area shown in FIG. 2, the relative motion information between the host vehicle and the feature point P of the obstacle. The predicted position of the obstacle is calculated from the predicted trajectory obtained based on the three-dimensional position information of the obstacle and assuming a certain elapsed time.
Then, from the prediction result of whether or not the obstacle at the calculated predicted position is reflected in each camera having the common area as a common visual field area, the obstacle is copied in a wider range for the common area. To determine a partial overhead view of the camera.
Further, priority is given to the determined partial overhead view image of the camera for the common area, and the overhead view image is redisplayed.
As a result, the arrangement of the front partial overhead image 121, the rear partial overhead image 122, the right side partial overhead image 123, and the left side partial overhead image 124 is changed, so that the obstacle can be more widely captured in the common area 129. The partial overhead view image of the left camera 4 that can be given priority, and as shown in FIG. 3B, the overhead view image including the entire obstacle is redisplayed.
That is, in the arrangement form of each area of the partial overhead image in FIG. 3A, only a part of the obstacle 131 is displayed in the overhead image, whereas in FIG. Is displayed.
That is, the common area is determined from the predicted trajectory of the obstacle assumed for a certain elapsed time based on the relative motion information between the own vehicle and the feature point P of the obstacle, and the three-dimensional position information of the obstacle. It is predicted that “the obstacle 131 can be photographed” or “cannot be photographed” for each camera having a common visual field region.
Then, for the common region, priority is given to the partial overhead view image of the camera that is predicted from the prediction result that “the obstacle 131 can be captured in a wider range”, and the boundary of the overhead view image is changed.

Here, the camera image priority process for changing the boundary line of the overhead image will be further described.
This camera image priority processing is a visual field region in which the common region is shared by a plurality of feature points extracted from an image determined as an obstacle located in the common region based on the expected trajectory and time change. Is predicted for each camera, and the prediction result is obtained.
The expected trajectory and time change are the expected trajectory and time change when a certain elapsed time is assumed based on the relative motion information obtained in the three-dimensional measurement processing unit 43.
Then, from this prediction result, priority is given to the whole image of the obstacle 131 from which the feature points have been extracted or the left side partial overhead view image 124 by the left camera 4 showing the obstacle in a wider range, and is fed forward. Change the seam and boundary of the overhead image.
The processing procedure for this is as follows.
First, assuming that the feature point i is in the shared visual field range of the camera A and the camera B, the three-dimensional position of the feature point i is set as a vector X _i = [X _i , Y _i , Z _i ] from the three-dimensional measurement result, and the vector V = [V _x , V _y ], ω is a relative speed and a yaw rate (step S21). Next, the three-dimensional position vector X ′ _i (t) = [X ′ _i (t), Y of the feature point i when the time t has elapsed since the measurement from the relative motion information V, ω and the three-dimensional position. ' _I (t), Z' _i (t)] is obtained (step S22).
Further, from the three-dimensional coordinate vector X ′ _i (t) of the feature point i at time t, the position vector x ^(a) _i (t) on the overhead image of the feature point i by the camera A and camera B at time t. , Vector x ^(b) _i (t) is obtained (step S23).
Then, it is determined from the vectors x ^(a) _i (t) and the vectors x ^(b) _i (t) whether or not the feature point i is reflected in the overhead images of the camera A and the camera B, and Expression (28), Functions R _a (vector x ^(a) _i (t)) and R _b (vector x ^(b) _i (t)) shown in (29) are obtained (step S24). Here, the functions R _a (vector x ^(a) _i (t)) and R _b (vector x ^(b) _i (t)) give a real number of “0” or more and “1” or less depending on the visibility. Also good.
Next, from the functions R _a (vector x ^(a) _i (t)), R _b (vector x ^(b) _i (t)), an expression (30) indicating that the feature point i is reflected only in the camera A ) To obtain a function S _a (vector x ^(a) _i (t)) represented by () and S _b (vector x ^(b) _i (t)) represented by equation (31) indicating that only the camera B is reflected. (Step S25).
Subsequently, from the function S _a (vector x ^(a) _i (t)) and S _b (vector x ^(b) _i (t)) until the time T elapses after the measurement shown in the equation (32). A weighted average N _{a of the} number of feature points reflected only in the camera A and a weighted average N _{b of the} number of feature points reflected only in the camera B shown in Expression (33) are obtained. Here, w (t) is a weighting function that satisfies w (0) = 1 and w (T) = 0 and decreases with time (step S26).
From the magnitude relationship between N _a and N _b , the overhead image of camera A is used for the common area when N _a > N _b . When N _a = N _b , a camera overhead image determined as a default value of the cameras A and B is used for the common area. When N _a <N _b , the overhead image of the camera B is used for the common area.
The common area may be weighted and combined with the luminance of the overhead images of the cameras A and B. That is, when N _a / N _b > k, the overhead image of camera A is applied to the common area. When 1 / k ≦ N _a / N _b ≦ k, the weighted average of the overhead images of the cameras A and B is applied to the common area. The weighted average of the bird's-eye view image is as shown in Expression (34) and Expression (35). At this time, _Ia is an overhead image of camera A, _Ib is an overhead image of camera B, k is an appropriate constant that satisfies k> 1, and I is an overhead image after synthesis. When N _a / N _b <1 / k, the overhead image of camera B is applied to the common area.

Next, with reference to a flowchart shown in FIG. 10, a warning display or alerting by a warning sound when there is an obstacle on the predicted route of the host vehicle, and a rapid acceleration suppression operation for a sudden accelerator operation will be described.
In FIG. 10, the same or equivalent processes as those in FIG. 9 are denoted by the same reference numerals and description thereof is omitted.
Each process of step S1, step S2, step S3, step S4 and step S5 is equivalent to the description of FIG.
In the subsequent predicted route determination process, the predicted route determination processing unit 51 obtains the state information including the position, the vehicle speed, the traveling direction, and the direction of the host vehicle 101 after a predetermined time Δt from the vehicle information of the host vehicle, and the predicted track of the vehicle 101. The predicted route 151 is determined by calculation (step S16).
The vehicle information includes all or one of the current vehicle speed, steering wheel operation angle, transmission shift position, accelerator opening, and vehicle dimensions.
Subsequently, the obstacle detection processing unit 52 on the predicted route determines whether or not the obstacle detected by the obstacle detection processing unit 44 is on the predicted route 151 (step S17).
The determination as to whether or not the route is on the predicted path is based on the fact that the three-dimensional position information of the feature point of the obstacle is obtained by the three-dimensional measurement processing unit 43. This is possible by determining whether or not the route 151 is included.

FIG. 4 is an explanatory diagram showing an overhead image when an obstacle located in the common area is included in the predicted route 151.
As shown in FIG. 4, when the obstacle 131 is located in the common area 129, the camera image priority processing unit 46 obtains the feature point in the common area 129 in the three-dimensional measurement processing unit 43. Based on the relative motion information, an expected trajectory when a certain elapsed time is assumed is obtained.
Based on the time change, a prediction result that “the obstacle 131 can be copied” or “cannot be copied” is obtained for each camera having the common area 129 as a common visual field area.
Further, a camera capable of capturing the obstacle 131 in a wider range is determined for each camera based on the prediction result so that the entire image of the obstacle 131 is displayed.
In the example shown in FIG. 4, the cameras that use the common area 129 as the common visual field area are the front camera 1 and the left camera 4, and the camera that can capture the obstacle 131 in a wider range is the left camera 4.
Then, the left partial overhead view image 124 by the left camera 4 is prioritized as the overhead view image of the common area.
In the bird's-eye view image shown in FIG. 4, when the obstacle 131 is further located in the predicted route 151, the obstacle determination processing unit 52 on the predicted route determines that the obstacle is on the predicted route 151, and warning output control is performed. The unit 53 drives the buzzer 55 by the warning sound output unit 54 and outputs a warning sound.
Further, the warning output control unit 53 causes the overhead image display unit 56 to display a warning on the display 57 (step S18).

FIG. 5 is an explanatory diagram showing an overhead image when the obstacle is located outside the common area.
As shown in FIG. 5, when the obstacle is not located in the common area, the boundary line of the overhead image is not changed, and a warning display and a warning sound are not output.
Subsequently, it is determined whether or not a rapid accelerator operation has been performed (step S19).
This rapid accelerator operation determination is performed, for example, by detecting whether the rate of change of the accelerator opening exceeds a certain reference value, and when the rate of change of the accelerator opening exceeds a certain reference value, a rapid acceleration accelerator operation determination signal Can be realized by a sensor circuit (accelerator operation detecting means) that outputs.
Alternatively, it can be detected based on the output of the kick-down switch.
When it is determined that the obstacle detected by the obstacle detection processing unit 44 is on the expected route 151, the rapid acceleration suppression control unit 59 is based on the rapid acceleration accelerator operation determination signal output from the sensor circuit. A sudden acceleration suppression request is output to the engine ECU 60 to suppress sudden acceleration of the vehicle (step S20).
As a result, when the obstacle is on the expected route 151, the rapid acceleration of the vehicle is suppressed even if a rapid accelerator operation is performed.

Here, the extraction of the feature point P in the feature point extraction processing unit 41 will be further described.
FIG. 11 is an explanatory diagram illustrating an example of an image on the overhead image.
For example, it is assumed that the image on the bird's-eye view image captured by the camera is on coordinates where the horizontal axis is the x-axis and the vertical axis is the y-axis with the upper left of the image data as the origin as shown in FIG.
First, the feature point extraction processing unit 3 obtains a partial differential A and a partial differential B with respect to the x-axis and y-axis directions of the image I as shown in FIG. Next, a spatial arrangement matrix for all pixels in the image I is obtained.
Then, the feature point extraction processing unit 4 calculates eigenvalues λmin (p) and λmax (p) for the spatial arrangement matrix, extracts predetermined values recognized as having features, and determines them as feature points.

  Next, the feature point tracking processing unit 42 tracks the temporal change of the feature points in the image on the overhead image obtained as described above.

  Therefore, in the vehicle periphery monitoring device of this embodiment, a camera that captures the entire obstacle 131 determined based on the relative motion information between the own vehicle and the feature point P of the obstacle and the three-dimensional position information of the obstacle. The overall image of the obstacle located in the common area that was hardly displayed can be displayed on the overhead image that prioritizes the partial overhead image.

  According to this embodiment, when an obstacle is located on the expected route, the obstacle is detected on the overhead image, and the driver is alerted visually or audibly with a display or sound, Furthermore, if there is a sudden accelerator operation, engine control that suppresses sudden acceleration can be realized.

  DESCRIPTION OF SYMBOLS 1 ... Front camera (imaging device), 2 ... Rear camera (imaging device), 3 ... Right camera (imaging device), 4 ... Left camera (imaging device), 21, 22, 23, 24 ... Viewpoint Conversion unit 31 .. bird's-eye view image synthesis processing unit 41... Feature point extraction processing unit 42... Feature point tracking processing unit 43 .. 3D measurement processing unit 44. ... Obstacle display area determination unit, 46 ... Camera image priority processing unit, 51 ... Expected route determination processing unit, 52 ... Obstacle on-route obstacle determination processing unit, 53 ... Warning output control unit (alarm output means) , 54... Warning sound output unit (alarm output unit), 55... Buzzer (alarm output unit), 57... Display (alarm output unit), 56 .. overhead image display unit (overhead image display unit), 59. ... Sudden acceleration suppression control unit, 60 ... Engine ECU, 101 ... Self vehicle (Vehicle), 131 ...... obstacle.

Claims (4)

A plurality of imaging devices that are mounted in a plurality of locations of the vehicle and that image the vehicle periphery in different directions from those locations;
An overhead image synthesis processing unit that synthesizes an overhead image centered on the vehicle from an image around the vehicle imaged by each imaging device;
An overhead image display means for displaying the overhead image synthesized by the overhead image synthesis processing unit;
Extracts such as edges of the corners or the image of the image on the synthesized bird's-eye image by the bird's-eye image synthesis processing unit, track change in position on the image processing technique of the parts having a comparison Tekiyo easy feature as feature points A feature point extraction unit;
A feature point tracking processing unit that tracks a change in the position of the feature point on the overhead image extracted from the time series video of the overhead image by the feature point extraction unit;
Feature point information including three-dimensional coordinate information of the feature point and relative motion information with respect to the vehicle based on a change in the position of the feature point tracked by the feature point tracking processing unit on the overhead image. A three-dimensional measurement processing unit to calculate,
Based on the feature point information calculated by the three-dimensional measurement processing unit, an obstacle detection processing unit that determines whether the image from which the feature point has been extracted is an obstacle,
A predicted route determination processing unit for determining a predicted route of the vehicle based on the vehicle state information including all of the vehicle speed, the steering wheel operation angle, and the transmission shift position;
On the expected path of the vehicle which is determined by the expected route determination processing unit, and the obstacle detection process determines expected route over obstacles whether the detected obstacle is located in the section determination processing unit,
An alarm output means for outputting an alarm when it is determined that an obstacle is located on the predicted route of the vehicle in the obstacle determination processing unit on the expected route;
A vehicle periphery monitoring device comprising: Accelerator operation detecting means for detecting an abrupt accelerator operation, and the accelerator operation detecting means when the obstacle determination processing unit on the predicted route determines that an obstacle is located on the predicted route of the vehicle. When sudden accelerator operation is detected rapidly accelerated the rapid acceleration suppressing required to control the output by the engine to the engine ECU inhibits the, characterized by comprising a rapid acceleration suppressing control portion to suppress the rapid acceleration claim 1 The vehicle periphery monitoring device described. The predicted route determination processing unit, based on the state information of the vehicle and the relative motion information of the feature points calculated by the three-dimensional measurement processing unit, the predicted route of the vehicle and the predicted trajectory of the feature points The obstacle determination processing unit on the predicted route determines an expected locus on the predicted route of the vehicle determined by the predicted route determination processing unit, and the predicted route determination processing unit also determines the obstacle. detection processing vehicle periphery monitoring device according to claim 1, wherein the detected characteristic points as an obstacle to determine whether or not the position unit. The predicted route determination processing unit is based on the state information of the vehicle, the relative motion information of the feature points calculated by the three-dimensional measurement processing unit, and the positional relationship between the vehicle and the feature points. The predicted route of the vehicle and the predicted trajectory of the feature point are determined, and the obstacle determination processing unit on the predicted route also determines the predicted route on the predicted route of the vehicle determined by the predicted route determination processing unit. is determined predicted locus in the processing unit, the obstacle detection processing vehicle periphery monitoring device according to claim 1, wherein the detected characteristic points as an obstacle to determine whether or not the position unit.

Images