JP2006268076A - Driving assistance system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a driving assistance system for facilitating obstacle detection. <P>SOLUTION: The driving assistance system includes: a display mounted in a vehicle; an image pickup device mounted in the vehicle for picking up images of areas around the vehicle; a bird's eye view image creation means for creating bird's eye view images based on two images picked up by the image pickup device with a time difference; a corresponding point extracting means for extracting two or more sets of corresponding points on a road surface between two bird's eye view images created by the bird's eye view image creation means; a coordinate transformation means for transforming the coordinates of one of the bird's eye view images based on the two or more sets of corresponding points on the road surface extracted by the corresponding point extraction means, so that the corresponding points between the bird's eye view images match; and an obstacle area extracting means for extracting any high obstacle area by determining a difference between the one of the bird's eye view images whose coordinates have been transformed and the other of the bird's eye view images. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a driving support system.

  For a driver of an automobile or the like, when checking the back, a blind spot is generated, so that it is difficult to confirm the rear. Therefore, a system has been developed that is equipped with an in-vehicle camera that monitors the rear of a vehicle that is likely to become a driver's blind spot and displays the captured image on a screen such as a car navigation system.

  However, if a wide-angle lens is used to project a wide range, lens distortion occurs in the image. Furthermore, the farther away from the camera, the smaller the image appears than the normal lens, and it becomes difficult to grasp the distance and space behind the vehicle from the captured image.

  Therefore, research is being conducted to display a more human-friendly video by using image processing technology rather than simply displaying the video from the camera. One of them is to convert the captured image into coordinates and generate and display a bird's eye view image as seen from above the ground (Japanese Patent Laid-Open Nos. 10-211849 and 2002-87160). reference). By displaying a bird's eye view from above, the driver can easily grasp the distance and space behind the vehicle.

  In addition, various image processing technologies and sensors are used to detect obstacles in the captured image, determine the degree of danger that the vehicle will collide with the obstacles, issue an alarm, and between the obstacle and the vehicle A technique for ensuring safety by displaying a distance or reducing the speed of a vehicle according to the distance has been devised.

  As such a technique, there is one that detects an obstacle based on a stereo image captured by two cameras mounted on the vehicle, or detects a distance between the obstacle and the vehicle. Japanese Patent Laid-Open No. 2001-187553 identifies an obstacle from vehicle movement information acquired by a rudder angle sensor and a wheel speed sensor and images taken by a back camera before and after the movement of the vehicle, from the vehicle to the obstacle. Is calculated, and a bird's eye view image in which the distance between the obstacle region and the obstacle is clearly displayed is displayed on the monitor.

  By the way, when the bird's-eye view image is displayed, an easy-to-see image viewed from above the ground can be presented to the driver, but since the obstacle with the original height is projected on the plane, the height information of the obstacle is displayed. It has the disadvantage of losing and making it difficult to perceive obstacles.

In the technique disclosed in Japanese Patent Laid-Open No. 2001-187553, an obstacle can be detected and a distance from the vehicle to the obstacle can be calculated. However, it is necessary to obtain accurate vehicle movement information from the rudder angle sensor and the wheel speed sensor. Yes, strict accuracy is required. Further, since it is necessary to take a difference between original images taken at different points in time, the scale difference between the two images must be eliminated, and a device is required for the calculation.
Japanese Patent Laid-Open No. 10-211849 JP 2002-87160 A JP 2001-187553 A

  An object of this invention is to provide the driving assistance system which can perform an obstacle detection easily.

  It is another object of the present invention to provide a driving support system that can easily detect an obstacle and calculate a distance from the vehicle to the obstacle.

  According to the first aspect of the present invention, a bird's-eye view image is obtained from each of a display mounted on a vehicle, an imaging device that captures an image around the vehicle mounted on the vehicle, and two images with a time difference captured by the imaging device. A bird's-eye view image generating means to generate, a corresponding point extracting means for extracting two or more sets of corresponding points on the road surface between the two bird's-eye view image generated by the bird's-eye view image generating means, and 2 on the road surface extracted by the corresponding point extracting means Coordinate conversion means for converting the coordinates of one bird's-eye view image so that corresponding points between the two bird's-eye view images match each other based on the pair of corresponding points or more, and the coordinate-converted one bird's-eye view image and the other bird's-eye view image By taking the difference, the obstacle region extracting means for extracting the obstacle region having a height, and displaying one bird's-eye view image of the two bird's-eye view images on the display, Characterized in that it comprises a display means for displaying on the display unit as an obstacle area extracted by the obstacle region extracting means can be distinguished from other portions of the's eye view image.

  The invention according to claim 2 comprises distance calculation means for calculating the distance from the vehicle for each obstacle area extracted by the obstacle area extraction means in the invention according to claim 1, and displays The means includes means for displaying a distance corresponding to each obstacle area calculated by the distance calculating means.

  According to the present invention, obstacle detection can be easily performed. Moreover, according to this invention, high visibility comes to be obtained also in the area | region away from the vehicle.

  Embodiments of the present invention will be described below with reference to the drawings.

[1] Description of Configuration of Driving Support System FIG. 1 shows a configuration of a driving support system provided in an automobile.

  The driving support system includes a camera (imaging device) 1 disposed rearward and obliquely downward at the rear of the vehicle, and an image processing unit 2 for generating a bird's eye view image or the like from an image provided in the vehicle and captured by the camera 1. And a monitor (display device) 3 that is arranged on a dashboard in the vehicle and displays a bird's eye view image or the like generated by the image processing unit 2.

  As the camera 1, for example, a CCD camera is used. As the image processing unit 2, for example, a microcomputer is used. As the monitor 3, for example, a monitor of a navigation system is used.

  There are two types of angles between the horizontal plane and the optical axis of the camera 1, the angle represented by α and the angle represented by β in FIG. 1. α is generally called a look-down angle or depression angle α. In this specification, the angle β is referred to as the tilt angle θ of the camera 1 with respect to the horizontal plane.

[2] Description of Processing Procedure by Image Processing Unit 2 FIG. 2 shows a processing procedure by the image processing unit 2.
It is assumed that parameters such as the height h of the camera 1 from the ground, the tilt angle θ of the camera 1 with respect to the horizontal plane, and the lens focal length f of the camera 1 are set in advance.

  First, a captured image of the camera 1 (first image: image at time t) is read (step S1). Thereafter, when the vehicle moves (YES in step S2), an image captured by the camera 1 (second image: image at time t + 1) is read (step S3).

  While converting a 1st image into a 1st bird's-eye view image, a 2nd image is converted into a 2nd bird's-eye view image (step S4).

  Next, matching is performed between the first bird's-eye view image and the second bird's-eye view image, and two sets of corresponding points on the road surface between the first bird's-eye view image and the second bird's-eye view image are extracted (step S5).

  Based on the two sets of corresponding points on the road surface between the first bird's-eye view image and the second bird's-eye view image, the two sets of corresponding points between the two images match each other. The coordinates are converted (step S6). In this example, the coordinates of the first bird's eye view image are converted so that two sets of corresponding points on the road surface between the first bird's eye view image and the second bird's eye view image match.

  By taking the difference between the one bird's-eye view image and the other bird's-eye view image that have undergone coordinate conversion, an obstacle region having a height is extracted (step S7). In this example, an obstacle region having a height is extracted by taking the difference between the first bird's-eye view image and the second bird's-eye view image that have undergone coordinate conversion.

  Next, the distance from the vehicle to each obstacle area extracted in step S7 is calculated (step S8). Then, an image in which each obstacle region in the image can be identified from other parts and distance data from the vehicle of each obstacle region on the second bird's eye view image is displayed on the display. (Step S9).

  Hereinafter, the processes of steps S4, S5, S6, S7, S8 and S9 will be described in detail.

[3] Explanation of processing in step S4 In step S4, a bird's-eye view image is generated by perspective-projecting the original image on the road surface. Hereinafter, a method for generating the bird's eye view image will be described.

FIG. 3 shows the relationship between the camera coordinate system XYZ, the coordinate system X _bu Y _bu of the imaging surface S of the camera 1, and the world coordinate system X _w Y _w Z _w including the two-dimensional ground coordinate system X _w Z _w. ing.

In the camera coordinate system XYZ, with the optical center of the camera as the origin O, the Z axis in the optical axis direction, the X axis in the direction perpendicular to the Z axis and parallel to the ground, and the Y axis in the direction perpendicular to the Z axis and the X axis The axis is taken. In the coordinate system X _bu Y _bu of the imaging surface S, the origin is set at the center of the imaging surface S, the X _bu axis is taken in the horizontal direction of the imaging surface S, and Y _bu is taken in the vertical direction of the imaging surface S.

In the world coordinate system X _w Y _w Z _w, the intersection of the perpendicular with the ground passing through the origin O of the camera coordinate system XYZ with the origin O _w, Y _w axis to the ground and perpendicular directions, X axis of the camera coordinate system XYZ X _w axis in a direction parallel to the can, Z _w axis is taken in a direction orthogonal to the X _w axis and Y _w axis.

Translation amount between the world coordinate system X _w Y _w Z _w and the camera coordinate system XYZ is [0, h, 0], the rotation amount about the X-axis is θ (= θo).

Therefore, the conversion formula between the coordinates (x, y, z) of the camera coordinate system XYZ and the coordinates (x _w , y _w , z _w ) of the world coordinate system X _w Y _w Z _w is the following formula (1): It is represented by

In addition, the conversion formula between the coordinates (x _bu , y _bu ) of the coordinate system X _bu Y _{bu on} the imaging surface S and the coordinates (x, y, z) of the camera coordinate system XYZ represents the focal length of the camera 1. If it is set to f, it represents with following Formula (2).

From the above equations (1) and (2), the coordinates (x _bu , y _bu ) of the coordinate system X _bu Y _bu of the imaging surface S and the coordinates (x _w , z _w ) of the two-dimensional ground coordinate system X _w Z _w (3) is obtained.

Further, the projection from the two-dimensional ground coordinate system X _w Z _w to the bird's eye view coordinate system X _au Y _{au of the} virtual camera is performed by parallel projection. Assuming that the focal length of the camera 1 is f and the height position of the virtual camera is H, the coordinates (x _w , z _w ) of the two-dimensional ground coordinate system X _w Z _{w and} the coordinates of the bird's eye view coordinate system X _au Y _au (x A conversion formula between _au and y _au ) is expressed by the following formula (4). The height position H of the virtual camera is set in advance.

  From the above equation (4), the following equation (5) is obtained.

  Substituting the obtained equation (5) into the above equation (3), the following equation (6) is obtained.

From the above equation (6), the equation (7) for converting the coordinates (x _bu , y _bu ) of the input image I into the coordinates (x _au , y _au ) of the bird's eye view coordinate system X _au Y _au is obtained.

  In step S4, the first image is converted into the first bird's-eye view image and the second image is converted into the second bird's-eye view image based on the equation (7).

  FIG. 4A shows an original image of a parking frame, and FIG. 4B shows a bird's eye view image corresponding to the original image of FIG. From FIG. 4A and FIG. 4B, it can be seen that the bird's-eye view image is converted to an image viewed from above.

[4] Description of Step S5 In step 5, two sets of corresponding points on the road surface between the first bird's-eye view image and the second bird's-eye view image are extracted. In this case, the two sets of corresponding points must be points on the road surface. This is because the first bird's-eye view image and the second bird's-eye view image are converted into the same coordinate system by utilizing the fact that the scale on the road surface is always the same regardless of the shooting point in the bird's-eye view.

  First, three or more feature points are extracted from one image of the first bird's eye view image and the second bird's eye view image. The feature point is, for example, a point at the edge portion. Then, by matching between the first bird's-eye view image and the second bird's-eye view image, corresponding points on the other image corresponding to the feature points on one image are obtained. As a result, a plurality of sets of three or more sets of corresponding points between both bird's-eye view images are obtained. Then, two sets of corresponding points are selected at random from the determined sets of corresponding points. Let two sets of corresponding points selected at random be (a1, a2) and (b1, b2).

  In the bird's eye view, the scale on the road surface is always the same regardless of the shooting point. Therefore, when these corresponding points (a1, a2), (b1, b2) are on the road surface, the length a1b1 between ab, The length a2b2 between a2b2 is equal. Therefore, it is determined whether or not the length a1b1 between a1b1 and the length a2b2 between a2b2 are equal for the two selected pairs of corresponding points (a1, a2) and (b1, b2). Points (a1, a2) and (b1, b2) are taken as two corresponding points on the road surface between the first bird's-eye view image and the second bird's-eye view image.

  If they are not equal, two pairs of corresponding points are selected at random, and for the two selected pairs of corresponding points (a1, a2) and (b1, b2), the length a1b1 between a1b1 and the length a2b2 between a2b2 are Determine whether they are equal. Such processing is repeated for the two pairs of corresponding points (a1, a2) and (b1, b2) until it is determined that the length a1b1 between a1b1 and the length a2b2 between a2b2 are equal. .

[5] Explanation of Step S6 In step S6, two sets of corresponding points between the two images are determined based on two sets of corresponding points on the road surface between the first bird's-eye view image and the second bird's-eye view image. The coordinates of one of the two images are converted so as to match. The basic concept of this coordinate transformation will be described.

  FIG. 5A shows an example of the first bird's-eye view image, and FIG. 5B shows an example of the second bird's-eye view image.

  It is assumed that (a1, a2) and (b1, b2) are two sets of corresponding points extracted in step S6 in both bird's-eye view images. As described above, the length a1b1 between a1b1 and the length a2b2 between a2b2 are equal. Using this property, the coordinates of the first bird's-eye view image can be converted into the coordinates on the second bird's-eye view image so that the corresponding points (a1, a2) and (b1, b2) match.

  First, as shown in FIG. 6, the first bird's-eye view image is translated by (Δx, Δy) so that the point a1 of the first bird's-eye view image matches the point a2 of the second bird's-eye view image. Next, as shown in FIG. 7, the first bird's-eye view image is rotated by Δθ so that the point b1 coincides with the point b2. Note that (Δx, Δy) and Δθ are movement amounts of the vehicle, and can be easily obtained from the image relative positions of the straight line a1b1 and the straight line a2b2.

[6] Description of Step S7 In step S7, the obstacle region having a height is detected by taking the difference between the one bird's-eye view image and the other bird's-eye view image that have undergone coordinate transformation.

  By the coordinate transformation in step S6, the planar image area on the road surface is the same between the first bird's-eye view image and the second bird's-eye view image after the coordinate transformation, but in the obstacle area having a height. Does not match.

  FIG. 8 shows an image Q1 on the first bird's eye view image after coordinate conversion for an obstacle Q and an image Q2 on the second bird's eye view image. Q1 and Q2 partially match, but a mismatched portion has occurred.

  Therefore, in step S7, by taking the difference between the first bird's-eye view image after the coordinate conversion and the second bird's-eye view image, as shown in FIG. 9, the inconsistent portion Q1 between the obstacle images Q1 and Q2 is shown. ', Q2' is extracted as an obstacle area.

[7] Description of Step S8 In step S8, the distance from the vehicle to each obstacle extracted in step S7 is calculated. That is, the distance from the vehicle to each obstacle is calculated on the second bird's eye view image. Specifically, based on the above equation (5), the coordinates (x _w , z _w ) of the two-dimensional ground coordinate system corresponding to the position (x _au , y _au ) of the obstacle on the second bird's eye view image are obtained. calculate. The obtained z _w is the distance from the vehicle.

[8] Explanation of processing in step S9 In step S9, the second bird's-eye view image displays on the second bird's-eye view image such that each obstacle region can be identified from other parts, and distance data from the vehicle in each obstacle region. The image with the added is displayed on the display.

  As a display that can identify the obstacle area from other parts, a special color is given to the obstacle area or a blinking display is performed. The distance data is displayed numerically near the obstacle area.

The structure of the driving assistance system provided in the motor vehicle is shown. 4 is a flowchart illustrating a processing procedure performed by the image processing unit 2. A camera coordinate system XYZ, a coordinate system X _bu Y _bu of the imaging surface S of the camera 1, is a schematic diagram showing the relationship between the world coordinate system X _w Y _w Z _w including a two-dimensional ground surface coordinate system X _w Z _w . It is a schematic diagram which shows the original image of a parking frame, and the bird's-eye view image corresponding to the original image. It is a schematic diagram which shows the example of a 1st bird's-eye view image and a 2nd bird's-eye view image. It is a schematic diagram which shows the state which translated the 1st bird's-eye view image so that the point a1 of a 1st bird's-eye view image may correspond to the point a2 of a 2nd bird's-eye view image. It is a schematic diagram which shows the state which rotated the 1st bird's-eye view image so that the point b1 may correspond to the point b2 from the state of FIG. It is a schematic diagram which shows image Q1 on the 1st bird's-eye view image after coordinate transformation with respect to a certain obstacle Q, and image Q2 on the 2nd bird's-eye view image. It is a schematic diagram which shows the obstruction area | region extracted by taking the difference of the 1st bird's-eye view image and 2nd bird's-eye view image after coordinate transformation.

Explanation of symbols

1 Camera 2 Image processing unit 3 Monitor

Claims (2)

Indicator mounted on the vehicle,
An image pickup apparatus for picking up an image around the vehicle mounted on the vehicle;
A bird's-eye view image generating means for generating a bird's-eye view image from each of two time-difference images captured by the imaging device;
Corresponding point extracting means for extracting two or more sets of corresponding points on the road surface between two bird's eye view images generated by the bird's eye view image generating means;
Coordinate conversion means for converting the coordinates of one bird's-eye view image so that the corresponding points between the two bird's-eye view images match based on two or more sets of corresponding points on the road surface extracted by the corresponding point extraction means; By taking the difference between the one bird's-eye view image and the other bird's-eye view image, the obstacle region extracting means for extracting the obstacle region having a height, and one bird's-eye view image of the two bird's-eye view images is displayed on the display And a display means for displaying the obstacle area extracted by the obstacle area extraction means in the bird's eye view image on the display so as to be distinguishable from other parts,
A driving support system characterized by comprising: For each obstacle area extracted by the obstacle area extraction means, a distance calculation means for calculating the distance from the vehicle is provided, and the display means indicates the distance corresponding to each obstacle area calculated by the distance calculation means. The driving support system according to claim 1, further comprising a display unit.

Images