JP2008085710A - Driving support system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a driving support system for eliminating conventional defects so as to achieve highly reliable obstacle detection. <P>SOLUTION: The driving support system comprises: an indicator mounted in a vehicle; two image pickup devices which are provided with a space in the vehicle so that imaging areas mutually include a common region, and photograph images in the vicinity of the vehicle; a bird's-eye view image generating means for converting each of two images photographed by each of the image pickup devices into a bird's eye-view image of the same bird's eye-view coordinate system; an obstacle region extracting means for extracting a high obstacle region by taking a difference between the two bird's eye view images generated by the bird's eye image generating means; and a display means for displaying one of the two bird's eye view images in the indicator, and for displaying the obstacle region extracted by the obstacle region extracting means among the bird's eye-view images in the indicator in a way that it can be identified from other parts. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a driving support system.

  As a driving support system for parking, it has a function to convert the image captured by the in-vehicle camera that monitors the back of the vehicle into a bird's-eye view image, and detects and detects obstacles from the bird's-eye view images obtained at two different points A system for displaying the obstacle on the monitor has already been developed (see Japanese Patent No. 3301421).

  However, in this conventional system, in order to detect an obstacle, it is necessary that the vehicle is not in a stationary state, and the movement amount and movement angle of the vehicle are accurately determined in real time using a vehicle speed sensor, a steering angle sensor, or the like. It is necessary to measure.

In addition, a technique for detecting an obstacle using a stereo camera has already been developed. However, in order to detect an obstacle using a stereo camera, matching between images picked up by both cameras is required, and the amount of calculation becomes enormous.
Japanese Patent No. 3301421

  An object of this invention is to provide the driving assistance system which can eliminate the conventional fault and can perform highly reliable obstacle detection.

  According to the first aspect of the present invention, there are provided a display device mounted on a vehicle, and two image pickup devices which are provided at intervals in the vehicle so as to include a common area in the image pickup range of each other and pick up images around the vehicle By taking the difference between the two bird's-eye view images generated by the bird's-eye view image generating means and the bird's-eye view image generating means for converting each of the two images taken by each imaging device into a bird's-eye view image in the same bird's-eye view coordinate system, Obstacle area extracting means for extracting a certain obstacle area, and one of the two bird's-eye view images is displayed on the display, and the obstacle extracted from the bird's-eye view image by the obstacle area extracting means A display means for displaying the area on the display so that the area can be distinguished from other parts is provided.

  The invention according to claim 2 includes distance calculation means for calculating a distance from the vehicle to the obstacle area extracted by the obstacle area extraction means in the invention according to claim 1, wherein the display means is a distance. A means for displaying the distance from the vehicle to the obstacle area calculated by the calculating means is provided.

  According to a third aspect of the present invention, in the first or second aspect of the present invention, each of the imaging devices may be configured with respect to the same virtual axis that is horizontal to the ground or two virtual axes that are horizontal to the ground and parallel to each other. The image pickup apparatus is attached in a state where the optical axes thereof are orthogonal to each other.

  According to the present invention, obstacle detection can be easily performed. Moreover, according to this invention, high visibility comes to be obtained also in the area | region away from the vehicle.

  Embodiments of the present invention will be described below with reference to the drawings.

[1] Explanation of basic concept of the present invention

  FIG. 1 shows a processing flow of the present invention.

In the present invention, for example, at the rear of the vehicle, two cameras are attached at an interval so that a common area is included in the mutual imaging range. Each camera is attached rearward and obliquely downward. However, each camera is attached in a state where the optical axis of the camera is orthogonal to two identical or different virtual axes that are horizontal to the ground and parallel to the lateral direction of the vehicle.
Information on the height position of each camera from the ground, information on the mounting angle of each camera, and the horizontal distance (relative position information) of each camera are set in advance.

  Video captured by the two cameras is input. Then, two bird's-eye view images in the same bird's-eye view coordinate system are generated from each input video. The difference between the two obtained bird's-eye view images is taken. The obstacle area is specified from the obtained difference image, and the distance from the vehicle to the obstacle area is calculated. Here, the obstacle means an object having a height. Then, on the one bird's-eye view image, an image in which the obstacle area in the image can be identified from other parts and an image in which the distance data from the vehicle to the obstacle area is added is displayed on the monitor.

  2A, 2B, and 2C show how the two cameras 1 and 2 are attached.

  In order to detect obstacles in a wide range, it is preferable that the common area of the imaging range of both cameras is wide, but if the installation positions of both cameras are too close, the difference between the captured images of both cameras will be reduced. Obstacle detection accuracy may be degraded. That is, widening the obstacle detection range and increasing the obstacle detection accuracy are in a trade-off relationship. Therefore, it is necessary to set the mounting positions of the two cameras in consideration of this trade-off relationship. In this embodiment, three attachment methods indicated by (a), (b), and (c) in FIG. 2 are proposed.

  FIG. 2A shows an example in which two cameras 1 and 2 are arranged at a predetermined distance in the vertical direction, and FIG. 2B shows two cameras 1 and 2 at a predetermined distance in the horizontal direction. FIG. 2C shows an example in which two cameras 1 and 2 are arranged at a predetermined distance in an oblique direction.

  As shown in FIG. 3, the camera 1 is attached in a state where the optical axis of the camera is orthogonal to a virtual axis that is horizontal to the ground and parallel to the lateral direction of the vehicle. Although not shown, the camera 2 is similarly attached in a state where the optical axis of the camera is orthogonal to a virtual axis that is horizontal to the ground and parallel to the lateral direction of the vehicle.

  As shown in FIG. 3, there are two types of angles formed by the horizontal plane and the optical axis of the camera 1 (camera 2): an angle represented by α and an angle represented by β. α is generally called a look-down angle or depression angle. In this specification, the angle β is referred to as a camera tilt angle (attachment angle) θ with respect to a horizontal plane. The mounting angle β of the camera 1 and the mounting angle β of the camera 2 may or may not match.

  FIG. 4 shows an imaging scene by two cameras 1 and 2 attached to the vehicle as shown in FIG. In FIG. 4, reference numeral 200 denotes a triangular pyramid-shaped obstacle existing on the ground, and 201 and 202 are white lines drawn on the ground.

  FIG. 5 shows captured images captured by the cameras 1 and 2 in the imaging scene shown in FIG. In FIG. 5, 101 is a captured image of the camera 1, and 102 is a captured image of the camera 2. Although the shooting is for the same scene, the captured images of the cameras 1 and 2 are slightly different because the positions of the cameras 1 and 2 are different.

  When the captured images 101 and 102 shown in FIG. 5 are converted into bird's-eye view images of the same bird's-eye view coordinate system, two bird's-eye view images as shown in FIG. 6 are obtained. In FIG. 6, 111 is a bird's eye view image generated from the captured image 101, and 112 is a bird's eye view image generated from the captured image 102. Between the two bird's-eye view images 111 and 112, the planar image regions on the road surface coincide with each other, but inconsistent portions occur in the regions Q1 and Q2 of the obstacle 200 having a height.

  When the difference between the two bird's-eye view images 111 and 112 shown in FIG. 6 is taken, a difference image as shown in FIG. 7 is obtained. In this difference image, unmatched portions Q1 'and Q2' of the images 200 and 112 of the obstacle 200 in the two bird's eye view images 111 and 112 are extracted. Therefore, the inconsistent portions Q1 'and Q2' can be specified as an obstacle region. When the obstacle area is specified, the distance D from the vehicle to the obstacle can be calculated.

  Of the camera arrangements shown in FIGS. 2A, 2B, and 2C, the camera arrangement shown in FIG. 2C is preferable. Hereinafter, this reason will be described.

  In the case where the arrangement of the cameras 1 and 2 is FIG. 2A, an imaging scene is assumed in which a vertical bar (obstacle) 310 as shown in FIG. 8 exists. This vertical bar 310 is present in a vertical plane 300 that includes both cameras 1 and 2 and extends rearward of the vehicle. In this case, since the vertical bar 310 appears in the left and right center of the captured images of the cameras 1 and 2, the bird's-eye view images of the same bird's-eye view coordinate system corresponding to the captured images of the cameras 1 and 2 are indicated by 311 and 312 in FIG. Such a difference image becomes such an image as shown in FIG. In the difference image, only the upper end portion of the vertical bar 310 can be extracted.

  In the case where the cameras 1 and 2 are arranged as shown in FIG. 2B, an imaging scene in which a card rail 410 as shown in FIG. 11 is present is assumed. The card rail 410 has a symmetrical shape with respect to a vertical plane that passes between the cameras 1 and 2 and extends rearward of the vehicle. In this case, since the card rail 410 is at the same distance from the cameras 1 and 2, the bird's-eye view images of the same bird's-eye view coordinate system corresponding to the captured images of the cameras 1 and 2 are as shown by 411 and 412 in FIG. These images become images, and the difference images are as shown in FIG. In the difference image, only both end portions of the card rail 410 can be extracted.

[2] Explanation of the configuration of the driving support system

  FIG. 14 shows a configuration of a driving support system provided in the automobile.

  The driving support system includes two cameras (imaging devices) 1 and 2 that are disposed rearward and obliquely downward at the rear of the vehicle, and images that are provided in the vehicle and captured by the cameras 1 and 2 in the same bird's-eye view coordinate system. An image processing unit 3 that performs processing such as conversion to a bird's-eye view image and detects an obstacle based on the obtained both bird's-eye view images, and one of the image processing units 3 that is disposed on the dashboard in the vehicle and obtained by the image processing unit 3 A monitor (display) 4 is provided that displays an image in which a bird's eye view image is added with a display or the like that can identify an obstacle region in the image from other parts.

  As the arrangement of the cameras 1 and 2, the camera arrangement shown in FIG. As the cameras 1 and 2, for example, CCD cameras are used. As the image processing unit 3, for example, a microcomputer is used. As the monitor 4, for example, a monitor of a navigation system is used.

[2] Explanation of processing procedure by the image processing unit 3

  FIG. 15 shows a processing procedure by the image processing unit 3.

  The height h of the cameras 1 and 2 from the ground, the tilt angle θ of the camera 1 with respect to the horizontal plane, the horizontal distance (relative position information) L between the cameras 1 and 2 (see FIG. 17C), It is assumed that parameters such as the lens focal length f are set in advance. Based on such information, a coordinate conversion table (look-up tables LUT1, LUT2) for converting the captured images of the cameras 1 and 2 into a bird's-eye view image in the same bird's-eye view coordinate system is generated and stored in advance. It shall be. A method of creating LUT1 and LUT2 will be described later.

  In this example, a reference bird's-eye view image is generated based on the LUT 1 and the captured image of the camera 1, and based on the LUT 2 and the captured image of the camera, the bird's-eye view image in the same bird's-eye view coordinate system as the reference bird's-eye view image. Is generated. The LUT 1 is a lookup table for converting a captured image of the camera 1 into a reference bird's-eye view image (first bird's-eye view image). The LUT 2 is a lookup table for converting a captured image of the camera 2 into a bird's eye view image (second bird's eye view image) in the same bird's eye view coordinate system as the first bird's eye view image.

First, the captured image (first input image I ₁ ) of the camera ₁ is read and the captured image (second input image I ₂ ) of the camera 2 is read (step S1).

Then, the first input image I ₁ is converted into a bird's-eye view image (first bird's-eye view image) using the lookup table LUT1, and the second input image I ₂ is converted into a bird's-eye view image (first view) using the lookup table LUT2. 2) (step S2).

  Next, the difference between the first bird's-eye view image and the second bird's-eye view image is taken (step S3). Then, the obstacle area is specified based on the obtained difference image, and the distance from the vehicle to the obstacle area is calculated (step S4). A method for calculating the distance from the vehicle to the obstacle area will be described later.

  Then, the first bird's-eye view image (reference bird's-eye view image) is displayed with an image in which the obstacle area in the image can be identified from other parts and distance data from the vehicle in the obstacle area. It is displayed on the monitor (step S5). Then, the process returns to step S1. As a display that can identify the obstacle area from other parts, for example, a special color is given to the obstacle area, a blinking display is performed, or the obstacle area is surrounded by a red line.

[3] Description of how to create lookup tables LUT1 and LUT2

[3-1] A method for creating the LUT 1 will be described.

FIG. 16 shows the relationship between the camera coordinate system XYZ, the coordinate system X _bu Y _bu of the imaging surface S of the camera 1, and the world coordinate system X _w Y _w Z _w including the two-dimensional ground coordinate system X _w Z _w. ing.

In the camera coordinate system XYZ, with the optical center of the camera as the origin O, the Z axis in the optical axis direction, the X axis in the direction perpendicular to the Z axis and parallel to the ground, and the Y axis in the direction perpendicular to the Z axis and the X axis The axis is taken. In the coordinate system X _bu Y _bu of the imaging surface S, the origin is set at the center of the imaging surface S, the X _bu axis is taken in the horizontal direction of the imaging surface S, and Y _bu is taken in the vertical direction of the imaging surface S.

In the world coordinate system X _w Y _w Z _w, the intersection of the perpendicular with the ground passing through the origin O of the camera coordinate system XYZ with the origin O _w, Y _w axis to the ground and perpendicular directions, X axis of the camera coordinate system XYZ X _w axis in a direction parallel to the can, Z _w axis is taken in a direction orthogonal to the X _w axis and Y _w axis.

Translation amount between the world coordinate system X _w Y _w Z _w and the camera coordinate system XYZ is [0, h, 0], the rotation amount about the X-axis is theta.

Therefore, the conversion formula between the coordinates (x, y, z) of the camera coordinate system XYZ and the coordinates (x _w , y _w , z _w ) of the world coordinate system X _w Y _w Z _w is the following formula (1): It is represented by

In addition, the conversion formula between the coordinates (x _bu , y _bu ) of the coordinate system X _bu Y _{bu on} the imaging surface S and the coordinates (x, y, z) of the camera coordinate system XYZ represents the focal length of the camera 1. If it is set to f, it represents with following Formula (2).

From the above equations (1) and (2), the coordinates (x _bu , y _bu ) of the coordinate system X _bu Y _bu of the imaging surface S and the coordinates (x _w , z _w ) of the two-dimensional ground coordinate system X _w Z _w (3) is obtained.

Further, the projection from the two-dimensional ground coordinate system X _w Z _w to the bird's eye view coordinate system X _au Y _{au of the} virtual camera is performed by parallel projection. Assuming that the focal length of the camera 1 is f and the height position of the virtual camera is H, the coordinates (x _w , z _w ) of the two-dimensional ground coordinate system X _w Z _{w and} the coordinates of the bird's eye view coordinate system X _au Y _au (x A conversion formula between _au and y _au ) is expressed by the following formula (4). The height position H of the virtual camera is set in advance.

  From the above equation (4), the following equation (5) is obtained.

  Substituting the obtained equation (5) into the above equation (3), the following equation (6) is obtained.

From the above equation (6), the equation (7) for converting the coordinates (x _bu , y _bu ) of the input image I ₁ into the coordinates (x _au , y _au ) of the bird's eye view coordinate system X _au Y _au is obtained. .

Therefore, based on the above equation (7), for each coordinate (x _au , y _au ) of the bird's eye view coordinate system X _au Y _au , the corresponding coordinate (x _bu , y _bu ) of the input image I ₁ of the camera 1 Can be created.

[3-2] A method for creating the LUT 2 will be described.

  Similarly to the camera 1, the camera (2) also holds the above formula (7). However, when the bird's-eye view image is generated from the input image of the camera 2 based on the equation (7), the bird's-eye view coordinate system is a bird's-eye view coordinate system (reference) of the bird's-eye view image generated from the input image of the camera 1. It is not the same as the bird's eye view coordinate system).

  In order to generate a bird's eye view image from the input images of the cameras 1 and 2 according to Expression (7), the mounting angle θ and the mounting height h of the cameras 1 and 2 are required. In the bird's-eye view image generated by Expression (7), the vertical projection point on the ground of each camera 1 and 2 is the coordinate origin. Therefore, in order to convert the input images of the cameras 1 and 2 into a bird's-eye view image of the same bird's-eye view coordinate system, a horizontal distance (relative position information) L between both cameras is required.

That is, as shown in FIG. 17, (shown by dashed lines in FIG. 17) Z _w axis of the bird's eye view coordinate system of the bird's-eye view image generated from the input image of the camera 2 based on the equation (7) is based on the equation (7) Te relative Z _w axis of the bird's eye view coordinate system of the generated bird's-eye view image from the input image of the camera 1 (bird's-eye view coordinate system as a reference), are separated in the -X _w direction of the bird's eye view coordinate system by a distance L.

Therefore, the equation for converting the coordinates (x _bu , y _bu ) of the input image I ₂ of the camera 2 into the coordinates (x _au , y _au ) of the reference bird's-eye view coordinate system X _au Y _au is as follows: 8).

Thus, based on the equation (8), bird's eye view coordinate system as the reference X _au Y coordinates of _au (x _au, y _au) for each input image I ₂ of the coordinates of the camera 2 corresponding thereto (x _bu, A LUT2 storing y _bu ) can be created.

In the above embodiment, since LUT1 and LUT2 do not consider lens distortion correction, input images I ₁ and I ₂ are images after performing lens distortion correction on the original image captured by the camera. Is preferably used.

In addition, look-up tables taking lens distortion correction into consideration can be used as LUT1 and LUT2. In this case, since the original images picked up by the camera can be used as the input images I ₁ and I ₂ , it is not necessary to perform lens distortion correction on the original images.

[4] Explanation of the method for calculating the distance from the vehicle to the obstacle area

After extracting the obstacle area based on the difference image obtained in step S3 of FIG. 15, the position (x _au , y _au ) of the obstacle area on the first bird's eye view image based on the above equation (5). The coordinates (x _w , z _w ) of the two-dimensional ground coordinate system corresponding to are calculated, and the obtained z _w is the distance D from the vehicle to the obstacle.

It is a block diagram which shows the flow of a process of this invention. It is a schematic diagram which shows the attachment aspect of the two cameras 1 and 2. FIG. It is a schematic diagram which shows the attachment angle (beta) of the camera. It is a schematic diagram which shows the imaging scene by the two cameras 1 and 2 attached to the vehicle as shown in FIG. FIG. 5 is a schematic diagram illustrating captured images captured by the cameras 1 and 2 in an imaging scene as illustrated in FIG. 4. It is a schematic diagram which shows two bird's-eye view images obtained by converting each captured image 101,102 shown in FIG. 5 into the bird's-eye view image of the same bird's-eye view coordinate system. It is a schematic diagram which shows the difference image of two bird's-eye view images shown in FIG. FIG. 3 is a schematic diagram showing an imaging scene in which a vertical bar is present when the cameras 1 and 2 are arranged as shown in FIG. FIG. 9 is a schematic diagram showing two bird's-eye view images obtained by converting captured images taken by the cameras 1 and 2 into bird's-eye view images in the same bird's-eye view coordinate system in the imaging scene shown in FIG. 8. It is a schematic diagram which shows the difference image of two bird's-eye view images shown in FIG. FIG. 3 is a schematic diagram illustrating an imaging scene in which a card rail exists when the arrangement of the cameras 1 and 2 is FIG. 2B. FIG. 12 is a schematic diagram showing two bird's-eye view images obtained by converting captured images taken by the cameras 1 and 2 into bird's-eye view images in the same bird's-eye view coordinate system in the image pickup scene as shown in FIG. 11. It is a schematic diagram which shows the difference image of two bird's-eye view images shown in FIG. It is a block diagram which shows the structure of the driving assistance system provided in the motor vehicle. 4 is a flowchart illustrating a processing procedure performed by the image processing unit 3. FIG. 16 shows the relationship between the camera coordinate system XYZ, the coordinate system X _bu Y _bu of the imaging surface S of the camera 1, and the world coordinate system X _w Y _w Z _w including the two-dimensional ground coordinate system X _w Z _w. It is a schematic diagram. The bird's eye view generated from the input image of the camera 1 based on the Y axis (indicated by a broken line in FIG. 17) of the bird's eye view coordinate system of the bird's eye view image generated from the input image of the camera 2 based on the equation (7). It is a schematic diagram which shows the positional relationship with the Y-axis of the bird's-eye view coordinate system (reference bird's-eye view coordinate system) of an image.

Explanation of symbols

1, 2 Camera 3 Image processing unit 4 Monitor

Claims (3)

Indicator mounted on the vehicle,
Two imaging devices that are provided at an interval in the vehicle so as to include a common area in each other's imaging range and that capture an image around the vehicle,
Bird's-eye view image generation means for converting each of the two images captured by each imaging device into a bird's-eye view image of the same bird's-eye view coordinate system;
By taking the difference between the two bird's-eye view images generated by the bird's-eye view image generating means, the obstacle region extracting means for extracting the obstacle region having a height, and one bird's-eye view image of the two bird's-eye view images is displayed on the display. Display means for displaying on the display so that the obstacle area extracted from the bird's eye view image by the obstacle area extraction means can be distinguished from other parts.
A driving support system characterized by comprising: Distance calculating means for calculating the distance from the vehicle to the obstacle area extracted by the obstacle area extracting means is provided, and the display means displays the distance from the vehicle to the obstacle area calculated by the distance calculating means. The driving support system according to claim 1, further comprising: Each imaging device is mounted in a state where the optical axis of the imaging device is orthogonal to the same virtual axis that is horizontal to the ground or two virtual axes that are horizontal to the ground and parallel to each other The driving support system according to claim 1.

Images