JP4052310B2 - Method, apparatus and system for calculating distance to intersection - Google Patents

Description

  The present invention relates to a technology for realizing a high-precision, high-function safe driving support system and navigation system capable of photographing a traffic signal at the next intersection with an in-vehicle camera mounted on a vehicle and calculating a distance to the intersection. It is.

For a traveling vehicle, the information on the distance to the nearest intersection is information necessary for traveling safety. In addition, information on the distance to the nearest intersection is indispensable even in a system that supports automatic driving.
Conventionally, the following methods have been considered as a method for calculating the distance to the nearest intersection.

(1) Knowing the position of the vehicle using a radio wave positioning system such as GPS (Global Positioning System). The distance to the nearest intersection is calculated based on the position of the nearest intersection in the map database.
(2) Use a radio wave positioning system such as GPS and know the position of the vehicle using a distance sensor. A map database is used to obtain the position of the vehicle on the road on the assumption that the vehicle travels on the road. Based on these pieces of information, the distance to the nearest intersection is calculated.

(3) By communicating with the roadside beacon, the distance information to the nearest intersection when the roadside beacon passes is obtained. The distance to the nearest intersection at an arbitrary time is calculated from this distance information and the travel distance data of the on-vehicle distance sensor.
However, if the radio wave condition is bad, the position error of the radio wave positioning system such as GPS becomes large, and the position may not be obtained.

In addition, roads in the map database are often expressed as vectors, and the road width is not taken into consideration, so an accurate distance to the stop line cannot be obtained at a large intersection.
Moreover, when using a distance sensor, aging may occur in the long term. In addition, a system aiming at safe driving assistance does not necessarily have vehicle position detection, and therefore may not be equipped with a distance sensor.

On the other hand, a system has been proposed in which a front image is taken with an in-vehicle camera and a signal lamp is automatically detected and recognized by image processing.
Yuichiro Kobashi, Naoto Ishikawa, Masato Nakajima "Automatic Recognition of Road Signs and Traffic Lights" 1st ITS Symposium 2002, Proceedings pp321-327, 2002 JP 2000-149195 A

If the distance to the nearest intersection can be accurately calculated using the signal lamp recognition system, it is effective for subtle guidance before the intersection.
Therefore, the present invention installs an in-vehicle camera in the vehicle and obtains information such as the height of the signal lamp by way of road-to-vehicle communication so that the driver can safely stop before the intersection or pass through the intersection. An object is to provide a method, an apparatus, and a system for calculating a distance to an intersection that can be supported.

Distance calculation method to the intersection of the present invention, the vehicle-mounted camera installed in the vehicle, recognizes the signal lamp device intersection traffic signal from the image captured in front of the vehicle, by communicating with the installed communications device to the roadside, the communication Obtaining distance information from the device to the traffic light, calculating and storing the height of the signal lamp based on the coordinates of the signal lamp in the image at the time when the distance information was acquired, and the distance information, the coordinates of the signal lamp device in the image, based on the high-ri of the signal lamp device, a method of calculating the distance from the vehicle to the intersection (claim 1).

According to this method, an image of the signal lamp is captured by the in-vehicle camera, the position of the signal lamp is grasped , distance information from the communication device to the signal is obtained, and the height of the signal lamp is calculated. Then, based on these pieces of information, the distance from the vehicle to the intersection can be calculated as the vehicle travels. By providing this distance information to the driver, it is possible to avoid as much as possible the situation in which the driver is at a loss in determining whether to stop or pass before the intersection. Therefore, safe driving support can be enhanced.

The distance calculation device to the intersection of the present invention includes a receiving unit that receives distance information from the communication device installed on the road side to the traffic signal at the intersection, an in-vehicle camera installed in the vehicle, and the in-vehicle camera. Based on the image recognition processing unit for recognizing the signal lamp of the traffic light from the image taken in front of the vehicle , the coordinates of the signal lamp in the image when the distance information is received, and the distance information, the height of the signal lamp A height calculation processor that calculates the height; a memory that records the height of the signal lamp calculated by the height calculation processor; the coordinates of the signal lamp in the image; and the signal lamp recorded in the memory. And a distance calculation processing unit that calculates a distance from the vehicle to the intersection from the height (claim 2 ).

This device records the distance information acquired from the communication device, estimates the height of the signal lamp based on the distance information, and then the coordinates of the signal lamp on the screen of the in-vehicle camera, The distance from the vehicle to the intersection is calculated from the height information of the signal lamp.
The distance from the vehicle to the intersection may be a distance from the vehicle to the stop line of the intersection (claim 3 ). This is because the distance to the stop line at the intersection is particularly important in systems that support safe driving and automatic driving.

Distance calculation system to signal machine of the present invention includes a communication device installed in the roadside, and a distance calculation device according to claim 2 or claim 3 mounted on the vehicle (claim 4). This system is a system according to the same invention as the apparatus according to claim 2 or claim 3 .

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.
-Distance calculation method 1-
FIG. 1 is a layout diagram of a traffic signal and a roadside communication device for implementing a method for calculating a distance to an intersection when the height information of a signal lamp is known .
In FIG. 1, a roadside communication device 6 is installed on the side of the road before the intersection. The installation point of the roadside communication device 6 needs to be close to the intersection so that the in-vehicle device can recognize the signal lamp device by image processing.

Further, the installation position of the roadside communication device 6 is not limited to the side of the road as long as the vehicle A can communicate with the vehicle A. For example, the roadside communication device 6 may be attached to the traffic light 4 or may be built in a signal controller that controls the color of the signal lamp of the traffic light 4.
In FIG. 1, a vehicle A equipped with an in-vehicle camera 3 capable of photographing the front is traveling toward an intersection. It is assumed that the in-vehicle camera 3 is attached to the vehicle body close to the front at a height h from the tire ground contact point of the vehicle A.

A traffic light 4 is installed at a position crossing the intersection as viewed from the vehicle A.
In addition, there is a pedestrian crossing at the intersection, and there is a stop line 5 in front of it.
FIG. 2 is a coordinate diagram showing the relative relationship between the road coordinate system and the in-vehicle camera coordinate system.
The coordinate system of the road is (X, Y, Z), the coordinate system of the in-vehicle camera 3 is (X ′, Y ′, Z ′), and the origin is the center of the captured image of the in-vehicle camera 3.

In the road coordinate system, the road direction toward the intersection is the Y axis (forward direction is positive), the direction perpendicular to this is the X axis (right direction is positive), and the direction perpendicular to the road is Z (upward is positive). ).
In the in-vehicle camera coordinate system, the optical axis of the in-vehicle camera 3 is the Y ′ axis, the right direction of the in-vehicle camera 3 on the plane perpendicular to the Y ′ axis is the X ′ axis, and the upward direction of the in-vehicle camera 3 is the Z ′ axis.
Furthermore, the rotation angle of the in-vehicle camera coordinate axis (the in-vehicle camera 3 parameter) with respect to the road coordinate axis is set to θ (pitch angle), φ (roll angle), and ψ (yaw angle), respectively, and the direction in which the right screw advances is positive (θ: The upward direction from the horizontal plane is positive, φ: clockwise is positive, ψ: counterclockwise is positive). θ, φ, and ψ are all known.

  At this time, the conversion formula between the road coordinate system and the in-vehicle camera coordinate system can be expressed by the following formula.

Here, each element of the matrix is as follows.
P ₁₁ = cos φcos ψ−sin θsin φsin ψ
P ₁₂ = cos φsin ψ + sin θsin φcos ψ
P ₁₃ = −cos θsin φ
P ₂₁ = −cos θsin ψ
P ₂₂ = cos θcos ψ
P ₂₃ = sin θ
P ₃₁ = sin φcos ψ + sin θcos φsin ψ
P ₃₂ = sin φsin ψ−sin θcos φcos ψ
P ₃₃ = cos θcos φ
FIG. 3 is a diagram illustrating a captured image of the in-vehicle camera 3, where the right direction is the x axis and the upward direction is the y axis. The x axis coincides with the X ′ axis, and the y axis coincides with the Z ′ axis.

At this time, the point (X, Y, Z) in the road coordinate system is converted to the point (x, y) on the captured image of the in-vehicle camera 3 as follows. f is the focal length of the lens.
x = f · X ′ / Y ′
_{= F (P 11 X + P} 12 Y + P 13 Z) / (P 21 X + P 22 Y + P 23 Z) (1)
y = f · Z ′ / Y ′
_{= F (P 31 X + P} 32 Y + P 33 Z) / (P 21 X + P 22 Y + P 23 Z) (2)
The in-vehicle camera 3 parameters such as θ (pitch angle), φ (roll angle), and ψ (yaw angle) can be set at the time of attachment to the vehicle body. It can be changed by (variation of yaw angle). Therefore, the in-vehicle camera 3 parameter may not be handled as a fixed value. For this reason, for example, the following method can be used. However, here, in order to obtain a simple notation, the roll angle φ is set to 0 at the time of attachment.

From the captured image (FIG. 3), the vanishing point (x _a , y _a ) that is the intersection of the white lines of the road is obtained. These are obtained when Y → ∞ in the equations (1) and (2).
xa = f (P ₁₂ / P ₂₂ ) = sin ψ / cos θcos ψ (3)
ya = f (P ₃₂ / P ₂₂ ) = − sin θ / cos θ (4)
From these equations (3) and (4), when θ and ψ are obtained, the in-vehicle camera 3 parameters can be determined.

In addition, when the road is relatively crowded, the vanishing point (x _a , y _a ) may be difficult to recognize. However, for the purpose of safe driving support, there is a case where the vehicle A is not in front of the own vehicle. Since it is considered important, the method for obtaining the in-vehicle camera 3 parameters using the vanishing point (x _a , y _a ) is sufficiently realistic.
On the other hand, the height of the road surface viewed from the camera is -h. If the height of the signal lamp from the road surface is H, the signal lamp is represented by a road coordinate system (X, Y, Hh). Let the point of the signal lamp on the captured image of the in-vehicle camera 3 be (x0, y0). These x0 and y0 are also observable values.

Then, the equations (1) and (2) become equations with X and Y as unknowns, and since two equations are two unknowns, X and Y can be solved. Therefore, the distance Y on the plane from the in-vehicle camera 3 to the traffic light 4 can be obtained.
The calculated coordinates X and Y of the signal lamp or the coordinates x and y of the signal lamp in the captured image become a function of time since blurring occurs as the vehicle A travels. These are written as coordinates X (t), Y (t), coordinates x (t), y (t). Temporal variations can be suppressed by taking a temporal moving average, exponential smoothing, or linear regression, respectively.

Note that the calculated coordinates X (t) of the signal lamp and the coordinates x (t) in the captured image are constant values unless there is a lane change. Y may be calculated by smoothing or the like.
-Distance calculation device and system 1-
Next, a distance calculation system that implements the distance calculation method 1 to the intersection will be described.

FIG. 4 is a conceptual diagram showing the overall configuration of the distance calculation system to the traffic light 4. The distance calculation system includes a roadside communication device 6 and an in-vehicle device 7.
As described above, the roadside communication device 6 is installed on the side of the road in front of the intersection. The roadside communication device 6 has the height information of the signal lamp 4a at the intersection. The height of the traffic light 4 is grasped by an investigation when the traffic light 4 is installed, and is stored in the roadside communication device 6 in advance.

The roadside communication device 6 periodically transmits the height information of the signal lamp 4a. Considering transmission to the in-vehicle device 7, radio or light is appropriate for the transmission medium. In particular, spot road-to-vehicle communication such as optical beacons and DSRC (Dedicated Short Range Communication) is desirable. The modulation method and the coding method are arbitrary.
The data sent from the roadside communication device 6 to the in-vehicle device 7 includes the distance between the traffic signal 4 and the stop line 5 in addition to the height, and other data on the distance from the roadside communication device 6 to the traffic signal 4, the shape of the intersection May also be included.

The in-vehicle device 7 calculates the distance from the vehicle A to the intersection, an image recognition processing function for recognizing the signal lamp 4a from the image captured by the communication device 8, the in-vehicle camera 3, and the in-vehicle camera 3 and specifying the coordinates thereof. It comprises a processing device 11 having a distance calculation function, a man-machine device 9, a memory 10, a display 14, a speaker 15, and the like.
The communication device 8 receives data sent from the roadside communication device 6, and the demodulation method and decoding method correspond to those of the roadside communication device 6.

The display 14 is a screen for displaying the distance to the intersection, and may be provided independently, or an in-vehicle television screen or an in-vehicle navigation device screen may be used.
The speaker 15 is used to announce the distance to the intersection to the driver.
The memory 10 stores data such as the height of the signal lamp 4a, the distance between the traffic light 4 and the stop line 5, the shape of the intersection, and the like sent from the roadside communication device 6.

The processing device 11 calculates the distance to the intersection based on the height data of the signal lamp 4a stored in the memory 10, and creates display and audio output data including this distance.
Hereinafter, the function of the processing apparatus 11 will be described in detail based on a flowchart (FIG. 5).
The processing device 11 confirms whether or not communication with the roadside communication device 6 has been established (step S1), and if established, from the roadside communication device 6, the height of the signal lamp 4a, the traffic light 4 and the stop line 5 is received and stored in the memory 10 (step S2).

If it becomes a fixed processing cycle (step S3), the signal lamp 4a is recognized from the captured image of the vehicle-mounted camera 3 (step S4).
FIG. 6 shows a front view of the traffic light 4. The traffic light 4 has a signal lamp 4a and a pole 4b that supports the signal lamp 4a.
The signal lamp 4a is recognized using, for example, the following features of the traffic light 4.

First, an elongated line on the captured image corresponding to the pole 4b of the traffic light 4 is detected. The detection of the elongated line is a method using Hough transformation in image processing (see Japanese Patent Laid-Open No. 7-78240), a method using edge detection and pixel connection (see Japanese Patent Laid-Open No. 2003-168110), This is possible with the min-max method.
Next, three signal lights of red, blue, and yellow and a signal light that is lit (high brightness) are detected. The signal lamp to be detected is circular as shown in FIG. Therefore, originally, an edge can be extracted and a circle having a signal use color can be easily determined as a signal candidate. However, since the in-vehicle camera 3 captures a wide range, the lighting portion of the imaged signal has a radius of about 2 to 12 pixels. Under this condition, it is difficult to discriminate between a circle and a rectangle even if an edge is extracted. Therefore, for example, a signal lamp is considered to be a connection of pixels having a color in the signal use color range. That is, color extraction is performed, and post-extraction labeling is performed. The connected pixels are captured by labeling, and the circumscribed rectangle of each label is taken. An appropriate label is recognized as a signal in consideration of the size of the circumscribed rectangle, the aspect ratio, the position on the captured image, the occupation rate of the number of extracted pixels in the circumscribed rectangle, and the like (see Non-Patent Document 1).

Further, the signal lamp 4a containing three signal lamps is detected. And the center G of a signal lamp is selected as a position coordinate of the signal lamp 4a.
The relative position (X, Y) of the traffic light 4 with reference to the vehicle A based on the position coordinates (x0, y0) of the signal lamp 4a in the captured image of the in-vehicle camera 3 described in the method for calculating the distance to the intersection. Is calculated. And the distance Y on the plane from the vehicle A to the traffic light 4 is calculated based on (X, Y) at this position.

Further, the distance between the traffic light 4 and the stop line 5 received from the roadside communication device 6 is subtracted from the distance Y.
In this way, the distance from the vehicle A to the stop line 5 at the intersection can be obtained.
The distance information from the vehicle A to the stop line 5 at the intersection is displayed on the display 14 and / or announced to the driver from the speaker 15 (step S5).

  Next, it is determined whether or not the distance from the vehicle A to the intersection stop line 5 is equal to or less than a certain value (step S6). If it is longer than a certain value, step S3 and subsequent steps are repeated, and if it is less than a certain value, the distance display is discontinued in order not to disperse the driver's concentration. The “constant value” is a distance necessary for the driver to perform direction indication, deceleration, brake operation, etc. in order to safely stop at the stop line 5 at the intersection.

-Method for calculating distance to intersection 2-
Next, the method for calculating the distance to the intersection according to the present invention when the vehicle A communicates with the roadside communication device 6 installed on the roadside and distance information from the roadside communication device 6 to the traffic signal 4 at the intersection can be obtained. To do.
Since the in-vehicle device obtains distance information from the roadside communication device 6 to the traffic signal 4 at the intersection when it passes through the roadside communication device 6, it is set as Y0. The signal lamp 4a at this time is represented by a road coordinate system (X, Y0, Zh).

The equations (1) and (2) become equations with X and Z as unknowns, and X and Z can be solved. Therefore, the height Z of the signal lamp 4a can be obtained. Let this be H.
Thereafter, when the vehicle A passes through the roadside communication device 6, the signal lamp 4a is represented by a road coordinate system (X, Y, Hh) using the calculated height H of the signal lamp 4a. Therefore, X and Y can be calculated from the equations (1) and (2).

  As described above, the roadside communication device 6 transmits the distance information from the roadside communication device 6 to the traffic signal 4 at the intersection to the vehicle A, so that the vehicle A calculates the height of the signal lamp 4a by image processing. Can do. The roadside communication device 6 does not need to transmit the height information of the signal lamp 4a. Therefore, it is not necessary to measure the height of the signal lamp 4a when installing the traffic light 4, and the work for installing the traffic light 4 is reduced.

If the roadside communication device 6 transmits both the height information of the signal lamp 4a and the distance information to the traffic signal 4 at the intersection, the redundancy of the expressions (1) and (2) increases. The distance calculation accuracy to the traffic light 4 can be improved.
-Distance calculation device and system 2-
The configuration of the distance calculation system when the vehicle A communicates with the roadside communication device 6 installed on the roadside to obtain distance information from the roadside communication device 6 to the traffic signal 4 at the intersection is basically the same as that shown in FIG. Absent.

However, the difference is that the roadside communication device 6 periodically transmits information on the distance Y0 to the traffic light 4, and the on-vehicle computer passes the roadside communication device 6 side and the distance Y0 to the traffic light 4 is shown. The height H of the signal lamp 4 a is calculated based on the coordinates of the signal lamp 4 a in the image captured by the in-vehicle camera 3, and the height H is stored in the memory 10.
After the vehicle-mounted computer passes the side of the roadside communication device 6, the distance Y to the traffic light 4 is calculated based on the height H of the signal lamp 4 a and the coordinates of the signal lamp 4 a in the image captured by the vehicle-mounted camera 3. Calculation and data for display and audio output including the distance Y are created.

This distance information is displayed on the display 14 and / or announced from the speaker 15 to the driver.
-Others-
Although the embodiments of the present invention have been described above, the embodiments of the present invention are not limited to the above-described embodiments. For example, in the description so far, the traffic light 4 has been installed on the other side of the intersection. However, as shown in FIG. 7, it may be installed in front of the intersection along with the other side of the intersection. In this case, the in-vehicle device 7 may recognize the signal lamp of the traffic light 4 ′ installed before the intersection, determine the position on the captured image, and obtain the distance to the traffic light 4 ′.

Information on whether the traffic light is installed on the other side of the intersection or on the front side of the intersection is held by the roadside communication device 6 and may be notified from the roadside communication device 6 to the in-vehicle device. Further, using the vanishing point (x _a , y _a ) of the road shown in FIG. 3, if the signal lamp is on the right side of the vanishing point, the traffic light is judged to be on the right side of the road, that is, before the intersection, and disappeared. If it is on the left side of the point, it may be determined that it is on the left side of the road, that is, beyond the intersection (FIG. 3 shows the latter case).

  As a result, the number of traffic lights to be recognized increases, so even if there is another large vehicle in front of the vehicle A and the traffic lights 4 beyond the intersection cannot be seen, the front side of the intersection The distance can be calculated using the traffic light 4 'in On the other hand, when there is another large vehicle on the right front side of the vehicle A and the traffic light 4 'on the front side of the intersection cannot be seen, the distance can be calculated using the traffic light 4 on the other side of the intersection. .

  In the above embodiment, the monocular in-vehicle camera 3 is used, but a stereo in-vehicle camera 3 may be used. In addition, various modifications can be made within the scope of the present invention.

Traffic 4 for carrying out the distance calculation method to the intersection difference point is a layout view of a roadside communication device 6. It is a coordinate diagram which shows the relative relationship of a road coordinate system and a vehicle-mounted camera coordinate system. It is a figure which shows the captured image of the vehicle-mounted camera. It is a conceptual diagram which shows the whole structure of the distance calculation system to a traffic light. 3 is a flowchart for explaining functions of the processing apparatus 11; It is a front view of the signal lamp device 4a. It is a road map showing the installation state of traffic lights 4, 4 '.

Explanation of symbols

3 In-vehicle camera 4, 4 ′ Traffic light 4 a Signal lamp 5 Stop line 6 Roadside communication device 7 In-vehicle device 8 Communication device 10 Memory 11 Processing device 14 Display 15 Speaker A Vehicle

Claims (4)

The in-vehicle camera installed in the vehicle recognizes the signal lamp of the traffic light at the intersection from the image taken in front of the vehicle,
By communicating with a communication device installed on the roadside, obtain distance information from the communication device to the traffic light,
Based on the coordinates of the signal lamp device in the image at the time of obtaining the distance information and the distance information, the height of the signal lamp device is calculated and stored,
The coordinates of the signal lamp device in the image, based on the high-ri of the signal lamp device, the distance calculation method from the vehicle to the intersection, characterized in that to calculate a distance to the intersection. A distance calculation device to an intersection that is mounted on a vehicle and calculates a distance from the vehicle to the intersection,
A receiving unit that receives distance information from the communication device installed on the roadside to the traffic signal at the intersection;
An in-vehicle camera installed in the vehicle;
An image recognition processing unit for recognizing the signal lamp of the traffic light from an image of the front of the vehicle captured by the in-vehicle camera;
A height calculation processing unit that calculates the height of the signal lamp based on the coordinates of the signal lamp in the image at the time of receiving the distance information, and the distance information;
A memory for recording the height of the signal lamp calculated by the height calculation processing unit;
A distance calculation processing unit that calculates a distance from the vehicle to the intersection from the coordinates of the signal lamp in the image and the height of the signal lamp recorded in the memory. apparatus. The distance calculation device for an intersection according to claim 2, wherein the distance from the vehicle to the intersection is a distance from the vehicle to a stop line of the intersection. The distance calculation system containing the communication apparatus installed in the roadside, and the distance calculation apparatus of Claim 2 or Claim 3 mounted in the vehicle.

Images