JP2011070593A - Vehicle periphery monitoring device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a vehicle periphery monitoring device for easily grasping an obstacle displayed with distortion on a bird's eye view video by emphasizing and displaying an obstacle on the bird's eye view video. <P>SOLUTION: This vehicle periphery monitoring device is configured to capture a bird's eye view video by constant time interval, and to extract a feature point P by a feature point extraction processing part 3, and to perform the tracking processing of the movement of the extracted feature point P by a feature point tracking processing part 4, and to calculate the optical flow of the feature point P to determine its moving vector by the tracking processing of the feature point P, and to calculate the relative motion information of the self-vehicle and the feature point P and the three-dimensional coordinate information of the feature point P from the change of the position of the feature point P on the bird's eye view video by a three-dimensional measurement processing part 5, and to detect an obstacle by an obstacle detection processing part 6 from the calculated relative motion information and three-dimensional coordinate information, and to determine and emphasize a region on the bird's eye view video including the detected obstacle by an emphasized region plotting processing part 8, and to display and output it by a display part 10. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a vehicle periphery monitoring device that displays an area on an overhead view image of an obstacle with respect to a host vehicle.

Conventionally, a vehicle is provided with a plurality of imaging devices, and images taken by the imaging devices are combined, an overhead view image of the entire circumference of the vehicle is generated and displayed on a driver's seat monitor, thereby enabling driving operations in a parking lot or the like. There is a vehicle periphery monitoring device that can be easily and safely performed.
In such a conventional vehicle periphery monitoring device, a front camera, a rear camera, a right camera, and a left camera are attached to the vehicle.
Then, the viewpoint images of the images captured by the front camera, the rear camera, the right camera, and the left camera are converted, and further, the images are synthesized to generate an overhead image.
The overhead view video generated in this way is displayed on the overhead view video display screen, and is used to support driving operations.

As such a vehicle periphery monitoring device, it has an imaging unit, a display unit that provides information to the driver, a start scene selection unit, a vehicle speed sensor, a shift position detection unit, a winker detection unit, a steering angle, A device including a sensor and a screen mode switching unit is provided.
The vehicle periphery monitoring device can change the rear side area based on the blinker information and the steering wheel steering angle information. In particular, when the blinker input is detected in the high speed mode, the turn signal direction synthesis is performed. Zoom in on the image.
In addition, the said imaging part images the vehicle periphery with each camera attached to the several camera installation part installed in the vehicle, and the said display part provides a driver | operator with information.
The start scene selection unit selects a normal travel scene from a plurality of driving scenes when starting the engine, the vehicle speed sensor detects a vehicle speed, and the shift position detection unit detects a shift position.
The winker detection unit detects the direction of the winker, the steering angle sensor detects a steering angle of the steering wheel, and the screen mode switching unit switches a display screen configuration based on information detected by each sensor and each detection unit. (See Patent Document 1).

JP 2002-109697 A

  Therefore, in the conventional vehicle periphery monitoring device, for the target to be focused on, particularly the three-dimensional target, the image is distorted at the processing stage where the viewpoint is converted and the image is synthesized, and the image is displayed on the overhead view video display screen. There was a problem that it was difficult to grasp the sense of distance of the target.

  The present invention has been made in view of such circumstances, and highlights obstacles on a bird's-eye view image, and facilitates the understanding of obstacles displayed distorted on the bird's-eye view image. An object is to provide a monitoring device.

  The invention according to claim 1 is a vehicle periphery monitoring device that displays a bird's-eye view image around the vehicle imaged by a plurality of imaging devices mounted on the vehicle. The vehicle periphery monitoring device is mounted at a plurality of locations of the vehicle and is different from the locations. A plurality of imaging devices that capture the surroundings of the vehicle in the direction, a video synthesis unit that synthesizes a bird's-eye view image centered on the vehicle from videos around the vehicle captured by the imaging devices, and a bird's-eye view synthesized by the video synthesis unit A feature point extraction processing unit that extracts, as a feature point, a corner of an image, an edge of the image, a portion where the image signal level of the image changes, or a portion where the hue of the image changes, and the time series video of the overhead video A feature point tracking processing unit that tracks a change in the position of the feature point extracted by the feature point extraction processing unit on the overhead video, and a feature point tracking processing unit that tracks A three-dimensional measurement processing unit that calculates feature point information including the height of the feature point in three-dimensional coordinates based on a change in the position of the feature point on the overhead view image, and the three-dimensional measurement processing unit An obstacle detection processing unit for determining whether the image from which the feature point has been extracted is an obstacle based on the feature point information, and a display unit for displaying an overhead video synthesized by the video synthesis unit And an area including the image determined as an obstacle by the obstacle detection processing unit is determined according to an imaging position of the image determined as the obstacle, and the determined area is emphasized and drawn as an emphasized area. An enhancement region drawing processing unit, and an enhancement region addition unit that superimposes the enhancement region on the overhead video and displays the enhancement region on the display unit are provided.

  According to the present invention, since the obstacle on the overhead image can be displayed in an emphasized manner according to the characteristic that the obstacle on the overhead image is distorted and displayed on the overhead image, the obstacle on the overhead image is distorted. There is an effect that it is possible to provide a vehicle periphery monitoring device that makes it possible to easily grasp an obstacle that is distorted and displayed on the bird's-eye view image by clarifying the state in which the vehicle is standing.

It is a block diagram which shows the structure of the vehicle periphery monitoring apparatus which is embodiment of this invention. It is explanatory drawing which shows the imaging range of each camera of the vehicle periphery monitoring apparatus of embodiment of this invention, and the bird's-eye view video produced | generated by carrying out viewpoint conversion of the image | video of these imaging ranges, and image-synthesising. It is explanatory drawing which shows the camera position of the vehicle periphery monitoring apparatus of embodiment of this invention, and distortion of a bird's-eye view image | video. It is explanatory drawing which shows the determination method of the emphasis area | region in the bird's-eye view image of the vehicle periphery monitoring apparatus of embodiment of this invention. It is explanatory drawing which shows an example of the frame display which shows the emphasis area | region in the bird's-eye view image of the vehicle periphery monitoring apparatus of embodiment of this invention. It is explanatory drawing which shows an example of the other display form of the frame which shows the emphasis area | region in the bird's-eye view image of the vehicle periphery monitoring apparatus of embodiment of this invention. It is explanatory drawing which shows the relationship between the three-dimensional space coordinate of the target object for demonstrating the obstruction detection in the vehicle periphery monitoring apparatus of embodiment of this invention, and the position on a bird's-eye view image | video. It is explanatory drawing which shows an example of the bird's-eye view image in the time t for demonstrating the obstruction detection in the vehicle periphery monitoring apparatus of embodiment of this invention, and the bird's-eye view image in the time t + F (DELTA) t. It is explanatory drawing which showed the motion of the feature point extracted from the bird's-eye view image in the vehicle periphery monitoring apparatus of embodiment of this invention with the vector. It is a flowchart which shows operation | movement of the vehicle periphery monitoring apparatus of embodiment of this invention. It is explanatory drawing which shows an example of the image on the bird's-eye view image in the vehicle periphery monitoring apparatus of embodiment of this invention. It is explanatory drawing which shows the obstacle area A by the distance in the vehicle periphery monitoring apparatus of embodiment of this invention, and the obstacle area B by the collision estimated time. It is a flowchart which shows operation | movement of the three-dimensional measurement process part and the obstruction detection process part in the vehicle periphery monitoring apparatus of embodiment of this invention. It is a figure which shows the numerical formula and matrix regarding a constraint matrix. It is explanatory drawing of partial differentiation A and B. FIG.

Embodiments of the present invention will be described below. FIG. 1 is a block diagram showing a configuration of a vehicle periphery monitoring apparatus according to an embodiment of the present invention. This vehicle periphery monitoring apparatus includes a time-series video capturing unit 2 for the overhead view video 1, a frame memory 3 for temporarily storing the captured overhead video in units of frames, a feature point extraction processing unit 4, a feature point tracking processing unit 5, a tertiary An original measurement processing unit 6, an obstacle detection processing unit 7, an emphasis region drawing processing unit 8, an emphasis region addition unit 9 and a display unit 10 are provided.
The bird's-eye view image 1 is generated by performing viewpoint conversion on images captured by a front camera, a rear camera, a right camera, and a left camera attached to a vehicle and further synthesizing images.

FIG. 2 shows an imaging range captured by the front camera 114, the rear camera 112, the right camera 111, and the left camera 113 attached to the vehicle 11, and the images of these imaging ranges are generated by performing viewpoint conversion and image synthesis. It is explanatory drawing which shows the bird's-eye view image displayed on a monitor apparatus centering | focusing on the own vehicle.
The front camera 114 is a camera that is attached to the center of the front grill at the front of the vehicle 11 and images the front of the host vehicle, for example.
The rear camera 112 is a camera that is attached to the rear portion of the vehicle 11 provided with a rear window, for example, and images the rear of the host vehicle.
The right camera 111 is a camera that is attached to a protruding end of a door mirror support portion that supports a right door mirror, for example, and images the right side of the vehicle 11.
The left camera 113 is a camera that is attached to a protruding end of a door mirror support portion that supports a left door mirror, for example, and images the left side of the vehicle 11.
Reference numeral 12 denotes an imaging range by the front camera 114, reference numeral 13 denotes an imaging range by the right camera 111, reference numeral 14 denotes an imaging range by the rear camera 112, and reference numeral 15 denotes an imaging range by the left camera 113.
Reference numeral 22 denotes a front partial overhead view image that has been subjected to viewpoint conversion by a viewpoint conversion unit (not shown) from the image of the imaging range 12 of the front camera 114 and displayed on the monitor device. Reference numeral 23 denotes a right side partial overhead view image that is similarly converted from the viewpoint of the image in the imaging range 13 of the right camera 111 and displayed on the monitor device. Reference numeral 24 denotes a rear partial overhead view image that is similarly converted from the viewpoint of the image in the imaging range 14 of the rear camera 112 and displayed on the monitor device. Reference numeral 25 denotes a left side partial overhead view image which is similarly converted from the viewpoint of the image of the imaging range 15 of the left camera 113 and displayed on the monitor device.
The front partial overhead view image 22, the right side partial overhead view video 23, the rear partial overhead view video 24, and the left side partial overhead view video 25 are illustrated as overhead views when the surroundings of the host vehicle are viewed from above with the host vehicle 11 as a center. The image is synthesized by the image synthesizing means.

Returning to FIG. 1, the time-series video capturing unit 2 captures the overhead video synthesized by the image synthesizing unit described in FIG. 2 as the time-series video 1 at a predetermined time interval.
The frame memory 3 is a memory that temporarily stores the captured overhead image in units of frames.
The feature point extraction processing unit 4 is characterized in that the time-series video 1 of the bird's-eye view video has a feature that makes it relatively easy to track a change in position on an image processing technique such as an image corner or an image edge. Extract as a point.
The feature point tracking processing unit 5 tracks the change in the movement position of the feature point with time change on the overhead view video.
The three-dimensional measurement processing unit 6 calculates relative motion information and three-dimensional coordinate information of the feature point from a change in the movement position of the feature point on the overhead view video.
The relative motion information and the three-dimensional coordinate information are calculated using equations (1) and (2) that model the movement of the target object on the overhead image shown below.

  In this driving support device, since the movement of the target object varies depending on the viewpoint, that is, the height of the camera in the overhead image of the monocular camera image, the movement of the target object on the overhead image corresponding to the camera height is modeled. The model is estimated with respect to the actual video, and the target object is three-dimensionally measured to detect the obstacle.

The obstacle detection processing unit 7 determines an obstacle based on the calculated relative motion information and three-dimensional coordinate information. In this obstacle determination, an obstacle is detected from feature points in the obstacle area based on the distance to the host vehicle and the obstacle area based on the estimated collision time.
The enhancement region drawing processing unit 8 determines an area including the image determined as the obstacle by the obstacle detection processing unit 7 according to the imaging position of the image determined as the obstacle, and emphasizes the determined region. A process for rendering as an emphasized region is performed.
The enhancement region adding unit 9 causes the display unit 10 to display the enhancement region so as to be superimposed on the overhead view video.
The viewpoint converting means and the image synthesizing means, a time-series video capturing unit 2, a feature point extraction processing unit 4, a feature point tracking processing unit 5, a three-dimensional measurement processing unit 6, an obstacle detection processing unit 7, and an emphasized area drawing process The unit 8 and the enhancement region adding unit 9 are realized by an ECU configured by a microcomputer that controls each unit.
This ECU performs various processes for realizing functions unique to the vehicle periphery monitoring device, including viewpoint conversion processing and image synthesis processing of images captured by the front camera 114, the rear camera 112, the right camera 111, and the left camera 113. Do.
The display unit 10 displays the overhead video synthesized by the video synthesizing unit.

Next, the operation will be described.
In FIG. 3, reference numeral 201 denotes a camera attached at the height of the attachment position H. FIG. 3A shows an example of an image of the three-dimensional object 202 captured from the camera 201. FIG. 7B shows an image when the viewpoint shown in FIG. 6A is converted into a bird's-eye view image, and the three-dimensional object of the image shown in FIG. Appears.
As shown in FIG. 7, the camera 201 is disposed at a height H on the z-axis. In FIG. 7, reference numeral 300 denotes an object, and its three-dimensional spatial coordinates are [X _p , Y _p , Z _p ] ^T. Reference numeral 301 denotes an object image on the bird's-eye view image, and its three-dimensional spatial coordinates are [x _p , y _p , 0] ^T. Reference numeral 302 indicates a planar bird's-eye view image including the captured object.
The operation of the vehicle periphery monitoring apparatus according to this embodiment will be described with reference to the flowchart shown in FIG. 10. The object and the host vehicle are in relative motion at a relative speed [V _X , V _Y ] ^T and yaw rate ω. Then, the relationship shown by the above-described equation (2) is established.
The vectors m (t + Δt) and n (t + Δt) in equation (2) are relative motions that store the relative velocity and yaw rate, and the vector _SP is obtained from the above equation (1) to the three-dimensional coordinates of the feature point P. Conversion is possible uniquely.
In this embodiment, the feature point P on the bird's-eye view image is converted into three-dimensional coordinates using the relationship of the expressions (1) and (2), and the obstacle is detected by determining the three-dimensional object with respect to the host vehicle. Do. Then, an emphasis area is determined on the overhead view video based on the camera position where the detected obstacle image is captured, and the determined emphasis area is superimposed on the overhead image and drawn.

Hereinafter, the operation for detecting an obstacle and determining and drawing an emphasis area on the overhead view video for the detected obstacle will be described with reference to a flowchart shown in FIG.
As shown in FIG. 2, the bird's-eye view video that has undergone viewpoint conversion and image synthesis is captured from the time-series video capturing unit 2 at regular time intervals Δt (step S1). The overhead video captured from the time-series video capturing unit 2 is temporarily stored in the frame memory 3.
And the feature point extraction process part 4 extracts the feature point P from the said bird's-eye view image | video (step S2). The feature point P is extracted by the feature point extraction processing unit 4 by extracting, as the feature point P, a portion having a feature that makes it relatively easy to track a change in position on an image processing technique such as an image corner or an image edge. .

FIG. 11 is an explanatory diagram illustrating an example of an image on the overhead view video.
For example, as shown in FIG. 11, it is assumed that an image on a bird's-eye view image captured by a camera is on coordinates where the upper left of the image data is the origin and the horizontal axis is the x axis and the vertical axis is the y axis.
The feature point extraction processing unit 4 obtains a partial differential A and a partial differential B with respect to the x-axis and y-axis directions of the image I as shown in FIG. Next, a spatial arrangement matrix for all pixels in the image I is obtained.
Then, the feature point extraction processing unit 4 calculates eigenvalues for the spatial arrangement matrix, extracts predetermined values recognized as having features, and determines them as feature points. Next, the feature point tracking processing unit 5 tracks the temporal change of the feature points in the image on the overhead video obtained as described above.

FIG. 8 is an explanatory diagram showing an example of the bird's-eye view video at time t and the bird's-eye view video at time t + FΔt, which is the time-series video 1 captured from the time-series video capture unit 2 in the vehicle periphery monitoring device of this embodiment.
As shown in FIGS. 8A and 8B, the feature point tracking processing unit 5 performs a tracking process to determine how the feature point P extracted by the feature point extraction processing unit 4 moves.
The tracking process of the feature point tracking processing unit 5 is performed as follows.
That is, what is the feature point P from the positional relationship between the feature point P on the overhead image at time t shown in FIG. 8A and the feature point P on the overhead image at time t + FΔt shown in FIG. A tracking process is performed to determine the position of the movement (step S3).
In the tracking process of the feature point P, the motion vector is obtained by calculating the optical flow of the feature point P.
FIG. 9 is an explanatory diagram showing motion vectors of feature points extracted from a bird's-eye view video in the vehicle periphery monitoring device of this embodiment shown in FIG.
The calculation process of the optical flow of the feature point P is performed as follows.
When a coordinate change of a feature point P that is common to each other is detected on the bird's-eye view image captured at a predetermined cycle and time interval, whether the feature point P is moving or not, Then, the movement of the feature point P, the direction of the coordinate change and the extent of the magnitude are calculated.

  Next, in the three-dimensional measurement processing unit 6, relative motion information between the host vehicle and the feature point P is calculated from the change in the position of the feature point P on the overhead view video using the equations (1) and (2), and the feature point. P three-dimensional coordinate information is calculated (step S4).

Further, the obstacle detection processing unit 7 detects an obstacle from the relative motion information calculated by the three-dimensional measurement processing unit 6 and the three-dimensional coordinate information (step S5). Then, an obstacle region is determined in the obstacle detection processing unit 7 on the bird's-eye view video for the detected obstacle. Further, with respect to the determined obstacle area, the enhancement area drawing processing unit 8 determines and emphasizes the enhancement area based on the camera position where the image of the obstacle area is captured. Then, the emphasis area adding unit 9 draws the emphasis area so as to overlap the overhead view video, and displays it on the display unit 10 (step S6).
FIG. 4 is an explanatory diagram illustrating a method for determining an emphasis region in the overhead view image of the vehicle periphery monitoring device according to this embodiment.
In FIG. 4, reference numeral 311 indicates the camera position, and reference numeral 312 indicates the distance and depth between the camera and the obstacle. Reference numeral 313 indicates an obstacle area, and reference numeral 314 indicates an emphasis area including the obstacle. The highlighted area 314 is displayed surrounded by a frame 315. The determination method of the enhancement region 314 is performed as follows.
First, the obstacle detection processing unit 7 determines the obstacle region 313 on the overhead view video.
Next, two circumscribed lines 501 and 502 for the obstacle region 313 are determined with the camera position 311 as an end point on the overhead view video.
Furthermore, the minimum depth of the obstacle 312 with respect to the camera position 311 on the overhead view image, that is, the minimum distance between the camera position 311 and the obstacle in the obstacle region 313 is determined.
Next, a frame 315 surrounding the emphasis region 314 is determined from the determined two circumscribed lines 501 and 502 and the minimum distance between the camera position 311 and the obstacle in the obstacle region 313. At this time, as shown in FIG. 5, the display of the frame 315 is displayed and output in different display colors according to the distance between the camera position 311 and the obstacle in the obstacle region 313. For example, if the depth is large (separated from the vehicle), it is displayed in orange, and if the depth is small (close to the vehicle), it is displayed in red.
FIG. 5 is an explanatory diagram showing an example of a frame display showing the emphasis region 314 in the overhead view video of the vehicle periphery monitoring device of this embodiment.
As shown in FIG. 5, the frame 315 is not determined and displayed and output so as to surround the entire emphasis area 314, but only the frame portion on the camera side of the emphasis area 314 is displayed as shown in FIG. May be. FIG. 6 is an explanatory diagram showing an example of another display form of the frame 315 showing the emphasis region 314.

  Next, the movement of the target object on the overhead image according to the height of the camera is modeled, and the model is estimated from the actual video, so that the target object is three-dimensionally measured and the obstacle image is detected Derivation of Expression (2) indicating the motion model of the upper three-dimensional object and derivation of Expression (3) from Expression (2) will be described.

As shown in FIG. _{3, [X p, Y p} , Z p] the position vector Xp of a three-dimensional real space of the feature point p is ^T, the camera at time t [0,0, H] to the position of Suppose there is. Here, H is the height of the camera with respect to the road surface.
_{[X p, y p, 0} ] position vector _{x p} on the overhead image of the feature point p when ^{T, the} vector _{Xp = [X p, Y p} , Z p] T and the vector _x p = _[x p, Equation (1) holds between y _p , 0] and ^T.
Here, if k _p = H / (H−Z _p ), the equation (1) can be rewritten as the following equation (4).

On the other hand, a true bird's-eye view image is when the camera is at an infinite height (H = ∞), and k _p = 1, and from Equation (4), Equation (5) is obtained.

That is, the feature point position vector x _p = [x _p , y _p , 0] ^T reflected in the overhead view video is the camera height and the feature point with respect to the true overhead view position [X _p , Y _p , 0] ^T. Is multiplied by a coefficient k _p corresponding to the height of.
Now, the object p is at the position of the vector Xp = [X _p , Y _p , Z _p ] ^T in the three-dimensional real space at time t with respect to the camera, and is relative to the yaw rate ω and the relative velocities V _X and V _Y. Suppose you were moving by exercise. It is assumed that there was no bouncing, roll, or pitching movement.
At this time, the position vector Xp (t + Δt) of the object p at the time t + Δt in the three-dimensional real space = [X _p (t + Δt), Y _p (t + Δt), Z _p (t + Δt)] ^T is given by equation (6) as follows.

From Equation (6), if there is no bouncing, roll, or pitching motion, Z _p (t) = Z _p (t + Δt), and k _p = H / (H−Z _p ), the coefficient k _p Is constant regardless of time. Equation (6) can be rewritten as Equation (7) as follows.

By the way, the position vector x _p = [x _p (t), y _p (t), 0] ^T on the bird's-eye view image of the object p at time t and t + Δt, the vector x _p = [x _p (t + Δt). , Y _p (t + Δt), 0] ^T is given by the following equation (8) from equation (5) because the coefficient k _p is constant regardless of time.

A position vector x _p (t + Δt) = [x _p (t + Δt), y _p (t + Δt), 0] ^T on the bird's-eye view image of the object p at time t + Δt is expressed by Equation (7). , (8), the result is as shown in equation (9).

As a result, the position vector x _p (t) = [x _p (t), y _p (t), 0] ^T on the bird's-eye view image of the object p at the times t and t + Δt, the vector x _p (t + Δt) = [x _p (t + Δt), y _p (t + Δt), 0] The following relational expression (10) holds for ^T.

Now, it is assumed that the relative motion of the object p with respect to the camera is caused by the fact that the object p is a stationary object and the camera moves. In other words, it is caused by the movement of the host vehicle. Further, it is assumed that there are P feature points in the stationary object shown in the overhead view video in addition to the object p.
At this time, since the relative motion to the stationary object is common, the position vector x _p (t) = [x _p (t), y _p (t), 0] ^T , vector x _p (t + Δt) = [x _p (t + Δt), y _p (t + Δt), 0] ^T , 1 <p <P is obtained from equation (10) as follows: It is given by (11).

  Equation (11) can be rewritten as Equation (12) as follows.

Here, the following expressions (13) and (14) are _satisfied , and the vectors m (t + Δt), n (t + Δt), c _x (t + Δt), and _cy (t + Δt) are times. It shows the relative movement rotational motion and the translational motion of the camera and the object at time t + Delta] t from t, the vector s _{p (t)} is the three-dimensional feature point p at substantially time t from equation (4) It can be regarded as a position in real space.

By the way, relative motion information m (t + 2Δt), n (t + 2Δt), c _x (t + 2Δt), and c _y (t + 2Δt) between the camera and the object from time t to time t + 2Δt are obtained. If obtained, the position vector x _p (t + 2Δt) = [x _p (t + 2Δt), y _p (t + 2Δt), 0] ^{T at} the time t + 2Δt, 1 <p <P is given by equation (15) as follows from equation (12).

  When formulas (12) and (15) are put together, formula (16) shown below is obtained.

  Now, it is assumed that the positions of P feature points in total are observed on the bird's-eye view video from time t to t + FΔt at time interval Δt. This observation is performed by extracting and tracking feature points. Then, equation (16) is expanded to obtain the following relational equation (17).

Expression (17) is a mathematical model of the movement of the three-dimensional object on the overhead view video.
Equation (17) can be rewritten as a matrix product as shown in Equation (18) below.

  Here, the matrices W, M, and S are given as shown in the following equation (19), and the matrix W represents the feature points on the overhead video from time t to t + FΔt obtained by feature point extraction / tracking. The matrix M represents the relative motion from time t to t + FΔt, and the matrix S represents the position of each feature point in the three-dimensional real space at time t.

  Using the relational expression of Expression (18), the relative motion and the position of the three-dimensional real space are obtained from the change of the position of the feature point on the overhead image.

Next, an outline of the model estimation method will be described.
In order to divide the matrix W indicating the change of the position of the feature point on the overhead view video into the matrix M indicating the relative motion and the matrix S indicating the three-dimensional shape from W = MS shown in Expression (18), the following procedure is used. Do.
From Equation (19), the matrix W is a matrix having a size of 2F × P. The matrix M is a vector m, n
Is a two-dimensional matrix, and the matrix S is a 3 × P matrix because the vector s is three-dimensional. That is, the matrix W is a rank 3 matrix. Therefore, singular value decomposition is once performed on the matrix W as follows.

Here, U is a 2F × H orthogonal matrix, Σ is a H × H diagonal matrix, ^VT is a H × P orthogonal matrix, and H is given as H = min {2F, P}. Further, it is assumed that diagonal elements of Σ are arranged in descending order as σ ₁ > σ ₂ >... Σ _p .
Since the matrix W is a matrix of rank 3, since σ ₄ can be regarded as a noise component, three columns of the matrix U are extracted as shown in Expression (21).

  From this, a temporary decomposition result can be obtained as shown in equation (22).

  Next, the constraint matrix is calculated. The decomposition result of Expression (22) satisfies (Expression 1) of FIG. 14, but the obtained matrix (1) of FIG. 14 generally does not satisfy (Expression 2) of FIG. For example, even if (matrix 1) in FIG. 14 is converted as shown in the following equation (23) using a matrix 3 having an appropriate size of 3 × 3, W = MS is satisfied. That is, it is necessary to select the matrix A appropriately.

In order to select the matrix A appropriately, the following constraint conditions are introduced.
(A) From the constraint equation (19) of the rotation component, the matrix M is configured as shown by the following equation (24).

  Here, since the vector mn indicates a rotational component as shown in the equation (13), the following constraint condition exists.

(B) Road surface constraint condition From equation (19), the matrix S is constructed as follows.

Further, the vector S _P (t) is expressed as S _P (t) = [x _p (t) y _p (t) from the equation (14).
k _p ] ^T , where k _p is given by k _p = H / (H−Z _p ). Here, k _p is a condition of Z _p <H, and if Z _p increases, k _p also increases. That is, as long as the object is below the camera, the higher the object, the greater the value of k _p .
k _p is minimized when the object is on the road surface, that is, Z _p = 0, and then k _p = 1. Since the bird's-eye view video always shoots the road surface, k _p = 1 on the bird's-eye view image
There is always a feature point that satisfies. That is, the following constraints exist for k _p.

  As described above, the matrix A may be determined so as to satisfy the expressions (25) and (27).

Next, the three-dimensional measurement processing of the three-dimensional measurement processing unit 6 and the obstacle detection processing of the obstacle detection processing unit 7 will be further described.
FIG. 12 is an explanatory diagram showing an obstacle area A based on the distance to the host vehicle 11 detected from the three-dimensional information and relative motion information of the three-dimensional object obtained from the overhead view image, and an obstacle area B based on the estimated collision time. is there.
FIG. 13 is a flowchart showing the operations of the three-dimensional measurement processing unit 6 and the obstacle detection processing unit 7 in the vehicle periphery monitoring device of this embodiment.
In the three-dimensional measurement process of the three-dimensional measurement processing unit 6, as described above, the three-dimensional coordinate information of the feature points on the overhead image and the relative motion information between the feature points and the host vehicle 11 are expressed by the equations (1) and (1). Calculate from (2) (step S41).
The three-dimensional coordinate information of this feature point is obtained as a three-dimensional position matrix S (step S42).
Moreover, the relative motion information between the feature point and the host vehicle 11 is obtained as a relative motion matrix M (step S43).
Then, threshold processing based on the height in the z-axis direction is performed on the three-dimensional coordinate information of the feature points obtained as the three-dimensional position matrix S (step S44).
In this threshold value processing, feature points having a value whose height information in the z-axis direction is not zero are determined as obstacle candidate feature points based on a predetermined height reference value (step S45). A feature point whose direction height information is zero is set as a non-obstacle feature point (step S46).
And about the said obstacle candidate feature point, since the distance between the own vehicle 11 becomes clear based on the three-dimensional coordinate information of the feature point, between the own vehicle 11 and an obstacle candidate feature point Threshold processing is performed based on a predetermined distance reference value for the distance (step S47). Then, the long distance obstacle feature point and the short distance obstacle feature point are identified (step S48, step S49).
.
The feature points identified as short-distance obstacle feature points are, for example, feature points included in the obstacle area A shown in FIG.
For the feature points identified as long-distance obstacle feature points, threshold processing is performed based on a predetermined expected collision time reference value for the expected collision time based on the relative motion matrix M obtained in step S43. (Step S51). Then, the feature points are identified as low-risk obstacle feature points and early-arrival obstacle feature points (steps S52 and S53).
The early-arrival obstacle feature points are, for example, feature points included in the obstacle area B shown in FIG.

  As described above, the vehicle periphery monitoring device according to the present embodiment detects an obstacle based on the identified low risk obstacle feature point and early arrival obstacle feature point. Then, the emphasis area drawing processing unit 8 changes the display color of the frame surrounding the emphasis area including the obstacle depending on whether the obstacle is a low risk obstacle or an early arrival obstacle, and the enhancement It is possible to display and output the area on the overhead view video.

Further, according to this embodiment, since the obstacle area on the overhead image can be emphasized and displayed according to the characteristic that the obstacle on the overhead image is distorted and displayed, the obstacle on the overhead image is distorted. There is an effect that it is possible to provide a vehicle periphery monitoring device that can easily grasp an obstacle that is distorted and displayed on the bird's-eye view image.
In addition, because the obstacle displayed on the bird's-eye-view video is clarified and emphasized by surrounding it with a frame, it is easy to grasp the location and state of the obstacle displayed on the bird's-eye-view video as distorted There is an effect that a monitoring device can be provided.
The display color of the frame 305 surrounding the emphasis region 304 is changed according to the distance between the camera position 301 and the obstacle, and whether the obstacle is a low-risk obstacle or an early-arrival obstacle. Different display output. For this reason, not only the obstacles that are distorted and displayed on the bird's-eye view image are clarified and emphasized, but it is also intuitive on the bird's-eye view image whether the obstacle is close to or away from the vehicle. It is possible to provide a vehicle periphery monitoring device that can be easily grasped. Further, there is an effect that it is possible to provide a vehicle periphery monitoring device that can intuitively grasp the positional relationship between the obstacle and the own vehicle in consideration of the relative motion between the obstacle and the own vehicle on the overhead view video.

  4 ... Feature point extraction processing unit, 5 ... Feature point tracking processing unit, 6 ... 3D measurement processing unit, 7 ... Obstacle detection processing unit, 8 ... Emphasis area drawing processing unit, 9 ... Enhancement area Adder unit, 10... Display unit, 11 .. own vehicle, 114... Front camera (imaging device), 112 .. rear camera (imaging device), 111... Right camera (imaging device), 113. (Imaging device).

Claims (8)

A plurality of imaging devices that are mounted in a plurality of locations of the vehicle and that image the vehicle periphery in different directions from those locations;
Video synthesizing means for synthesizing a bird's-eye view image centered on the vehicle from a video around the vehicle imaged by each imaging device;
A feature point extraction processing unit for extracting, as a feature point, a portion having a feature that makes it relatively easy to track a change in position on an image processing technique such as an image corner or an image edge on an overhead video synthesized by the video synthesis means. When,
A feature point tracking processing unit that tracks a change in the position of the feature point on the overhead view video extracted by the feature point extraction processing unit from the time-series video of the overhead view video;
A three-dimensional measurement processing unit that calculates feature point information including a height of the feature point in three-dimensional coordinates based on a change in a position of the feature point tracked by the feature point tracking processing unit on the overhead view image; ,
Based on the feature point information calculated by the three-dimensional measurement processing unit, an obstacle detection processing unit that determines whether the image from which the feature point has been extracted is an obstacle,
A display unit for displaying an overhead video synthesized by the video synthesizing unit;
An area including the image determined to be an obstacle by the obstacle detection processing unit is determined according to an imaging position of the image determined to be the obstacle, and the determined area is emphasized and drawn as an emphasized area A drawing processing unit;
An emphasis region adding unit that causes the display unit to display the emphasis region superimposed on the overhead view video;
A vehicle periphery monitoring device comprising:   The enhancement region includes a circumscribed line for an image region determined as the obstacle from a camera position that captures an image determined as the obstacle by the obstacle detection processing unit, and the image region and the camera position. The vehicle periphery monitoring device according to claim 1, wherein the vehicle periphery monitoring device is determined based on a depth which is a distance on the overhead view image.   The enhancement region rendering processing unit determined based on the circumscribing line and the depth is based on the circumscribing line and the depth, and at least a part or all of the periphery of the enhancement region is a frame. The vehicle periphery monitoring device according to claim 2, characterized in that the vehicle periphery monitoring device is defined and emphasized.   The vehicle periphery monitoring device according to claim 3, wherein the frame defining the emphasis region has a different display color depending on the depth.   The three-dimensional measurement processing unit models the movement of the target object on the overhead image according to the height of the imaging device based on the feature points, and performs model estimation on the actual video, thereby The driving support device according to claim 1, wherein feature point information including a height in three-dimensional coordinates is calculated.   The determination of whether or not the obstacle is detected by the obstacle detection processing unit is an obstacle that becomes an obstacle candidate based on a predetermined height reference value based on information about the height of the feature point in three-dimensional coordinates. The vehicle periphery monitoring according to claim 1, wherein candidate feature points are identified, and it is determined whether or not an image obtained by extracting the feature points is an obstacle based on the identified obstacle candidate feature points. apparatus.   The determination of whether or not the obstacle is detected by the obstacle detection processing unit is an obstacle that becomes an obstacle candidate based on a predetermined height reference value based on information about the height of the feature point in three-dimensional coordinates. Candidate feature points are identified, and the distance between the identified obstacle candidate feature points and the vehicle is a distance included in an obstacle range close to the vehicle, or included in an obstacle range at a long distance It is determined whether it is a distance based on a predetermined distance reference value, and the obstacle candidate feature point is identified as a long-distance obstacle feature point and a short-distance obstacle feature point from the determination result, and the short-distance obstacle 2. The vehicle periphery monitoring device according to claim 1, wherein it is determined based on the identification result whether or not an image from which object feature points are extracted is an obstacle at a short distance to the vehicle.   Whether the feature point information calculated by the three-dimensional measurement processing unit includes relative motion information on the overhead view image of the vehicle and the feature point is an obstacle performed by the obstacle detection processing unit. The determination identifies obstacle candidate feature points that are obstacle candidates based on a predetermined height reference value from information about the height of the feature points in three-dimensional coordinates, and the identified obstacle candidate feature points It is determined based on a predetermined distance reference value whether the distance to the vehicle is a distance included in an obstacle range close to the vehicle or a distance included in an obstacle range far away. The obstacle candidate feature point is identified as a long-distance obstacle feature point and a short-distance obstacle feature point from the determination result, and the identified long-distance obstacle feature point is calculated by the three-dimensional measurement processing unit. The relative motion information and a predetermined expected collision time reference value. The estimated collision time is determined, and a low risk obstacle feature point and an early arrival obstacle feature point are identified from the determination result, and an image obtained by extracting the early arrival obstacle feature point reaches the vehicle early. 2. The vehicle periphery monitoring apparatus according to claim 1, wherein it is determined whether or not the vehicle is an obstacle based on the identification result.

Images