JPH06107096A - Forward monitoring method for vehicle - Google Patents

Abstract

PURPOSE:To judge a degree of danger due to a traveling vehicle and the like in front and other obstacles on the road automatically and at high speed by using a foreground image by a video camera without using a distance sensor for measuring an inter-vehicle distance to the preceding vehicle. CONSTITUTION:Movement of the same point on an object shown in two frames of images chronologically continuing in a series of foreground animations is detected as an optical flow vector, and danger is judged by its size. A narrow window set in the radical direction from FOE of the previous image is moved in the same direction on the successive image, and the optical flow vector of a point to be aimed at is set as an arrow connecting each of the central points of the position of the window where the sum of absolute values of difference in brightness of the area on the successive image that the window overlaps the window becomes the minimum, and the set position of the window in the previous image. However, the point to obtain the optical flow is the only point that the difference in brightness between the previous and successive images exceeds a certain threshold value so that influence of landscape outside the road or a traffic lane, letters or characters drawn on the road should be eliminated in obtaining the optical flow.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention detects a vehicle in front of a vehicle or an obstacle when the vehicle is traveling by using an image picked up by a video camera installed in a front portion of a vehicle such as an automobile and warns a driver. The present invention relates to a front monitoring method for a vehicle for giving

[0002]

2. Description of the Related Art Conventionally, as this type of method, for example, there is one disclosed in Japanese Patent Application No. 2-241855. According to the method disclosed in the publication, a foreground is photographed from a moving vehicle, the movement of the same point of the photographed foreground is recognized as an optical flow at every predetermined time, and the distance between the optical flow and a vehicle in front is detected. Information about the position and relative speed of the preceding vehicle with respect to the own vehicle is obtained based on the distance sensor that measures the distance, and the driver is informed of that fact based on this information.

Further, in order to obtain an optical flow, a method called a matching method for detecting corresponding points between images of two frames has been conventionally used. In this matching method, a window W1 is taken for a pixel P of interest on the image at time t shown in FIG. 9A, and a correlation value is calculated while moving the window to the entire area of the image or a peripheral area,
As shown in FIG. 9B, the window W2 having the maximum correlation value
Is obtained as the corresponding point, that is, the corresponding pixel Q, and this PQ (arrow) becomes the optical flow. To obtain the above correlation value, Σ (W1 _{(x, y)} × W2 _{(x, y)} ) / (ΣW1 ² _{(x, y)} × ΣW2 ² _{(x, y)} ) ^1/2 (1 ) Is calculated. Note that W1 _{(x, y)} and W2 _{(x, y)} are windows W
1, the output of (x, y) coordinates in W2.

[0004]

However, in such a conventional method, in addition to a video camera for photographing the foreground in order to judge a danger, in order to know the position of the preceding vehicle with respect to the own vehicle, At the same time, a distance sensor for measuring the inter-vehicle distance is needed.

Further, in order to obtain the optical flow, it is necessary to perform a calculation process of detecting a corresponding point between two images. As described above, this calculation process detects a certain corresponding point. In order to do so, one pixel in the image is searched for in all pixels in the entire area in the image or in the peripheral area. This process must be performed on all pixels in order to detect corresponding points throughout the image. Also, many calculations are required to obtain the correlation value used as an index for performing the search.

Therefore, there is a problem in that the amount of calculation becomes enormous, so that real-time processing is difficult, and in order to realize it, a high-speed arithmetic processing device is required and the device itself becomes expensive. Can be given. In addition, it is difficult to distinguish whether the optical flow is caused by the part of the preceding vehicle, the scenery outside the road, the characters or symbols on the road surface, or the white line. Can also be given.

Therefore, in view of the above-mentioned conventional problems, the present invention uses the foreground image from the video camera installed in the front part of the vehicle without measuring the distance to the preceding vehicle,
It is an object of the present invention to provide a vehicle front monitoring method capable of automatically determining the degree of danger due to a traveling vehicle in front and other obstacles.

It is another object of the present invention to provide a vehicle front monitoring method capable of quickly recognizing movement of the same point as an optical flow in a foreground imaged by a video camera every predetermined time. There is.

[0009]

In order to achieve the above object, a vehicle front monitoring method according to the present invention takes a foreground image from a moving vehicle, and a predetermined movement of the same point of the taken foreground is determined. A front monitoring method for a vehicle, which recognizes an optical flow based on images of two frames before and after a time phase, and monitors a preceding vehicle or an obstacle with respect to the own vehicle based on the optical flow. It is characterized in that the degree of danger is judged based on the magnitude of the optical flow of the points.

In the above vehicle front monitoring method, one point to be noted from the point at infinity corresponding to one point indicating the traveling direction of the own vehicle traveling in the previous image in the two frames before and after the predetermined time. On the other hand, a long and narrow window centered on that one point is set in the radial direction, and while the window is moved in the radial direction from the point at infinity on the subsequent screen, the brightness difference from the window in the previous image is The feature is that the total sum of absolute values is obtained, and the movement amount of the window when the total sum is minimized is obtained as an optical flow of one point of interest.

The above vehicle front monitoring method is characterized in that a brightness difference is obtained from images of two frames before and after a predetermined time period, and an optical flow is calculated at a point where the brightness difference exceeds a certain threshold value. .

In the above vehicle front monitoring method, the optical flow generated from the scenery outside the road, the lanes on the road surface, the characters, the symbols, etc. is removed when the optical flow is obtained.

In obtaining the optical flow, a region corresponding to the scenery outside the road is set in advance, and the set region is not processed.

A video camera for picking up a foreground image from a running vehicle is provided, and the height from the lane, characters, or symbols on the road surface where the optical flow is generated to the video camera is obtained based on the optical flow information. The optical flow whose height matches the height of the camera from the road surface is removed.

The predetermined area is divided into a plurality of areas, the total sum of the lengths of optical flow vectors existing in each area is weighted for each area, and the degree of danger is judged by the weighted value. I am trying.

A predetermined threshold value is set for each area, and an area in which the sum of the weighted optical flow vector lengths exceeds the threshold value is judged to be dangerous.

The threshold is composed of several levels, and the level of risk is judged by the level of the threshold exceeded by the sum of the weighted optical flow lengths.

It is characterized in that an alarm is issued in accordance with the determined degree of danger.

[0019]

According to the above method, the optical flow becomes larger as the distance to the preceding vehicle or the obstacle is smaller and the relative speed is larger. Since the danger is judged based on the magnitude of the optical flow, it is not necessary to provide a range finder for measuring the distance to the preceding vehicle.

Paying attention to the fact that the optical flow is formed in the radial direction from the point at infinity corresponding to one point indicating the traveling direction of the host vehicle, traveling in the previous image of the two frame images is performed. From a point at infinity corresponding to a point indicating the traveling direction of the own vehicle, a narrow window is set in a radial direction with respect to the point of interest, and the window is set in a radial direction from the point at infinity on a later screen. While moving, the sum of the absolute values of the brightness differences between the window in the previous image and the area of the subsequent image overlapping the window is obtained, and the position of the window when the total is minimum and the front of the window. Since the arrow connecting the respective center points with the set position in the image is defined as the optical flow of one point, the amount of calculation is reduced and the processing speed can be increased.

Paying attention to the place where the brightness does not change with time such as the sky or the road surface included in the image, the brightness difference is obtained from the images of two frames which are before and after a predetermined time, and there is the brightness difference. Since the optical flow is calculated for points that exceed a certain threshold, the number of points in the image for which the optical flow is calculated is greatly reduced, and unnecessary processing can be omitted, resulting in high speed processing.

In order to remove the optical flow generated from the scenery outside the planned driving lane in obtaining the optical flow, an area corresponding to the scenery outside the planned driving lane is set in advance, and the processing of the set area is not performed. Therefore, the processing time can be shortened.

A video camera for picking up a foreground image from a running vehicle is used, and the height from the lane, characters, or symbols on the road surface where the optical flow is generated to the video camera is obtained using the optical flow information. By removing the optical flow whose height matches the height of the camera from the road surface, it is possible to process only the optical flow generated only by other vehicles in front and obstacles, and it is dangerous to use the optical flow. It is possible to know the degree and speed up the processing time.

Since the predetermined area is divided into a plurality of areas, the sum of the lengths of the optical flows existing in each area is weighted for each area, and the existence of danger is judged by the weighted value. You can know which position is dangerous. Further, a predetermined threshold value can be set for each area, and the area where the sum of the optical flow vector lengths exceeds the threshold value is judged to be dangerous, and the degree of danger can be set for each area. Furthermore, the threshold consists of several levels,
The level of risk is judged by the level of the threshold value over which the sum of the optical flow vector lengths exceeds, and an alarm is issued according to the calculated level of risk.

[0025]

Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 shows the arrangement of an apparatus for carrying out the method according to the present invention, in which 1 is a video camera for capturing the foreground, 2 is an image obtained by the video camera 1, and image processing described later is performed. An arithmetic processing device for performing 3, a speedometer for measuring the speed of the own vehicle, and 4 for inputting an image processing result of the arithmetic processing device 2 and the own vehicle speed obtained from the speedometer 3 to perform risk determination processing. Devices 5 are alarm devices.

FIG. 2 is a diagram for explaining changes in the foreground image obtained by the video camera 1, and FIG. 2 (b) is a video camera 1 in the situation including the own vehicle shown in FIG. 2 (a).
Shows an image taken at time t, and (c) shows an image taken at time t + Δt.

Now, it is assumed that the vehicle is going straight on a flat road. For example, when attention is paid to the road signs and buildings shown in (a), the images shown in (b) and (c) are obtained at time t and time t + Δt as time passes.
By searching for corresponding points in these two images and connecting them, a velocity vector as shown in (d) is obtained. This is the optical flow.

Here, these optical flows appear radially from one point called FOE (Focus of Expansion) in the image. The FOE is called an infinity point or vanishing point, and corresponds to one point indicating the traveling direction of the host vehicle on the image when the vehicle is traveling straight. In this way, the optical flow required when the host vehicle is traveling is
Radial direction from FOE. Here, the optical flow emitted from the preceding vehicle includes information including the position and the relative speed of the preceding vehicle with respect to the own vehicle, and it is considered that the longer the optical flow, the higher the risk.

Next, the details will be described with reference to FIG. In the optical arrangement of the figure, 11 is the lens of the video camera, 12 is the image plane of the video camera, f is the distance from the lens 11 to the image plane 12, and P
(X, Y, Z) is any one point on the preceding vehicle or obstacle,
If p (x, y) is a point corresponding to the point P on the image plane 12, then x = f · X / Z (2) from the similarity ratio of the triangle.

When this equation is modified and time-differentiated, X '= (Δx / ΔtZ + xZ') / f (3) Further, the x-direction component u of the optical flow is u = Δx / Δt (4), so that it is used as Z = (f · X′−x · Z ′) / u (5) Become.

Here, Z ′ = relative speed between the preceding vehicle or the obstacle and the host vehicle = −α (6) Therefore, the above equation (5) is Z = (f · X ′ + xα) / u. (7) Therefore, the x-direction component u of the optical flow is u = (f · X ′ + xα) / Z (8). The same applies to Y.

Therefore, from the above equation (8), the smaller the Z is, that is, the smaller the distance to the preceding vehicle or the obstacle, or the larger is the α, that is, the larger the relative speed with respect to the preceding vehicle, the optical flow. X component of is large. This also applies to the Y direction. Therefore, the optical flow becomes longer as the distance to the preceding vehicle becomes smaller and the relative speed becomes larger, and when the optical flow is longer than this, the risk to the preceding vehicle or obstacle is relatively high. Is considered to be large.

In the present invention, the optical flow is FOE.
The optical flow is obtained at a high speed by utilizing the fact that the optical flow is obtained in a radial direction from, and the method will be described below with reference to FIG.

FIG. 4 is a diagram showing an embodiment of a method for obtaining an optical flow at high speed. First, a narrow window is set in the radial direction from the FOE for one point of interest in the image at the time t (FIG. 4A). Next, in the image at time t + Δt, the sum of the absolute values of the brightness differences from the window at time t is obtained while moving the window from the FOE in the radial direction one point at a time. Then, the amount of movement of the window when the total sum becomes the minimum is obtained as the velocity vector of one point of interest (FIG. 4 (b)). Note that the above-mentioned luminance difference is, for each pixel forming the window, between pixels at corresponding positions indicated by ◯ in (a) and (b), for example.
The optical flow of the entire image can be obtained by repeatedly performing the above processing at all points of the image at time t.

Further, in the conventional method, the cross-correlation value is used when comparing the corresponding windows. On the other hand, in the method of the present invention, the total sum of the absolute values of the brightness differences is used, so the amount of calculation is reduced and the processing speed can be increased.

Further, the present invention does not obtain the velocity vector for every point in the image, but rather at time t and time t.
The difference is calculated from the image of + Δt, and the processing is performed only for points that exceed a certain threshold value that is the difference.

In general, an image showing the foreground while the vehicle is traveling is often a place where there is no temporal change in luminance such as the sky or the road surface. In such a place, in principle, it is impossible to obtain the optical flow. Therefore, as in the method of the present invention, within the image at time t and the image at time t + Δt,
By focusing on the fact that there is a brightness difference,
Useless processing can be omitted, and the speed can be increased.

By the method as described above, the optical flow can be obtained at high speed. However, the optical flow obtained here includes the optical flow caused by non-obstacles such as landscapes other than roads and lanes on the road surface. Therefore, it is difficult to determine the presence of a front obstacle or the risk of the obstacle by using the optical flow obtained here.
Therefore, it is necessary to remove the optical flow generated from the scenery outside the road or the lane on the road surface. Next, this method will be described.

FIG. 5 shows an embodiment of a method for removing the optical flow due to the scenery outside the road. It is assumed that the hatched position in the figure indicates the road, and the process is not performed at other positions.
By doing so, the optical flow generated from the scenery outside the road cannot be found from the beginning, and the processing time can be shortened by limiting the area.

Next, a method of erasing the mark on the road surface will be described with reference to the drawings. FIG. 6 shows an example of this method, and the optical configuration will be described first. FIG. 6 is drawn on the assumption that the entire road surface is approaching instead of the vehicle equipped with the video camera going straight. Reference numeral 11 denotes a lens of a video camera which images the front, and 12 denotes an image plane of the video camera. Reference numerals 13 and 14 represent lanes before and after movement. Consider three-dimensional coordinates with the center of the lens as the origin and two-dimensional coordinates with the FOE on the image plane 2 as the origin.

The position of the lane before movement is P (X, Y, Z + Δ
Z), and the position after ΔZ movement in the Z direction is Q (X, Y, Z), from the similarity ratio of the two triangles to the video camera in the lane where the optical flow occurs as shown in equation (1). You can know the height. Y = y ² ΔZ / fΔy (9) By using the obtained height information, the optical flow whose height matches the height of the video camera, that is, the optical flow caused by marks on the road surface is removed. You can

Here, ΔZ is the time interval Δt between two images.
It is the distance traveled during the period and can be obtained by knowing the vehicle speed. By the processing as described above, it becomes possible to remove the optical flow generated from other than the obstacle ahead. Therefore, it is possible to determine the degree of danger indicating how dangerous the obstacle ahead is from the position and length of the remaining optical flow.

Next, a method of obtaining the degree of risk will be described. FIG. 7 shows an embodiment of this method. First, the configuration will be described. Each of the areas I to IV indicates the distance of the own lane, the distance of the own lane, the distance of the adjacent lane, and the distance of the adjacent lane. Here, the degree of risk is obtained by weighting the total sum of the lengths of the optical flows existing in each region for each area and using the weighted value. Further, a predetermined threshold value is set for each area, and if the total flow length exceeds this threshold value, the area exceeding the threshold value is judged to be dangerous. Set several threshold levels,
It is also possible to judge the level of risk.

Finally, it becomes possible to call the driver's attention by sounding an alarm according to the calculated degree of danger. It is also possible to display which area is dangerous and how dangerous it is on the display. Furthermore, it is possible to change the type of the alarm tone, etc., depending on the danger level.

The image processing procedure according to the method of the present invention described above is summarized and shown in FIG. First, in step S1, the image at time t is captured, and then in step S2, the image at time t + Δt is captured.
After that, the FOE is set in step S3, and the processing area is set in the following step S4. Then, the process proceeds to step S5, and the region where the brightness difference exceeds a certain threshold is extracted from the image at time t and time t + Δt. After that, the process proceeds to step S6, an optical flow is obtained in the extracted region, the optical flow on the road is removed in the next step S7, and the risk degree is calculated by the value weighted for each area in the next step S8.

[0046]

As described above, according to the present invention, it is possible to automatically determine the presence or the degree of danger of a front obstacle, so that the vehicle can be driven safely.

In particular, the danger is judged by the magnitude of the optical flow of the preceding vehicle or the point on the obstacle, and one video camera for picking up the foreground from the moving own vehicle is installed in front of the vehicle. Since it is no longer necessary to provide a range finder for measuring the distance to a particularly preceding vehicle, it can be realized at low cost.

Further, the calculation amount is reduced, unnecessary processing is omitted,
By limiting the processing area, the processing speed can be increased and real-time processing becomes easier.

Further, it is possible to obtain an effect that an appropriate alarm can be issued by knowing the dangerous area and its degree.

[Brief description of drawings]

1 is a block diagram showing an example of an apparatus for carrying out the method according to the invention.

2 is a diagram showing a foreground image captured by the video camera of FIG. 1, an image, and an obtained optical flow. FIG.

FIG. 3 is a diagram for explaining a method of detecting an obstacle or the like by the method of the present invention.

FIG. 4 is a diagram for explaining how to obtain an optical flow by the method of the present invention.

FIG. 5 is a diagram showing an example of a region set to obtain an optical flow by the method of the present invention.

FIG. 6 is a diagram for explaining a method of removing optical flow due to a lane on a road surface by the method of the present invention.

FIG. 7 is a diagram showing an example of divided areas for determining the degree of risk by the method of the present invention.

FIG. 8 is a diagram showing a series of processes of the method of the present invention.

FIG. 9 is a diagram for explaining a problem of the conventional method.

[Explanation of symbols]

 1 Video camera 2, 4 Arithmetic processing device 5 Alarm device

[Procedure amendment]

[Submission date] May 12, 1993

[Procedure Amendment 1]

[Document name to be amended] Statement

[Correction target item name] 0041

[Correction method] Change

[Correction content]

The position of the lane before movement is P (X, Y, Z) ,
Assuming that the position after moving ΔZ in the Z direction is Q (X, Y, Z−ΔZ) , from the similarity ratio of the two triangles, the height up to the video camera in the lane where the optical flow is generated is expressed by the equation (1). You can know that. Y = y ² ΔZ / fΔy (9) By using the obtained height information, the optical flow whose height matches the height of the video camera, that is, the optical flow caused by marks on the road surface can be removed. ─────────────────────────────────────────────────── ───

[Procedure amendment]

[Submission date] May 12, 1993

[Procedure Amendment 1]

[Document name to be corrected] Drawing

[Name of item to be corrected] Fig. 4

[Correction method] Change

[Correction content]

[Figure 4]

[Procedure Amendment 2]

[Document name to be corrected] Drawing

[Correction target item name] Figure 8

[Correction method] Change

[Correction content]

[Figure 8]

 ─────────────────────────────────────────────────── ─── Continuation of the front page (72) Inventor Kazuyuki Sasaki 1500 Onjuku, Susono, Shizuoka Prefecture Yazaki Corporation

Claims (10)

[Claims] 1. An image of a foreground is taken from a moving vehicle,
A forward monitoring method for a vehicle, which detects a movement of the same point in images of two frames before and after a predetermined time period as an optical flow, and monitors a relative relationship of a preceding vehicle or a road obstacle with respect to the own vehicle. A forward monitoring method for a vehicle, characterized in that the degree of danger is judged according to the magnitude and appearance position of a vector of an optical flow that appears with respect to a point on an object. 2. In the previous image of the two frame images which are before and after a predetermined time period, from the point at infinity corresponding to one point indicating the traveling direction of the traveling own vehicle, with respect to one point of interest Setting a narrow window in a radial direction, and moving the window in the subsequent image from the point at infinity in the radial direction, the brightness difference between the window in the previous image and the region in the subsequent image overlapping the window. The sum of the absolute values of is calculated, and the arrow connecting the center points of the position of the window when the total is minimized and the set position in the image in front of the window is defined as one optical flow. The vehicle front monitoring method according to claim 1, wherein the front monitoring method is a vehicle. 3. A brightness difference at each point between images of two frames before and after a predetermined time period, and an optical flow is calculated only at a point where the brightness difference exceeds a certain threshold value. The vehicle forward monitoring method according to claim 1. 4. The front monitoring method for a vehicle according to claim 1, wherein in obtaining the optical flow, the optical flow generated from the scenery outside the road or the lanes, characters, symbols, etc. drawn on the road surface is removed. 5. The front monitoring method for a vehicle according to claim 1, wherein an area corresponding to a landscape outside the planned lane of travel is preset when the optical flow is obtained, and the processing of the set area is not performed. 6. A lane or character on a road surface in which an optical flow is generated based on the optical flow information, which is provided with a video camera for capturing a foreground from a moving own vehicle.
The vehicle front monitoring method according to claim 4, wherein the height from the symbol or the like to the video camera is obtained, and the optical flow whose height matches the height of the camera from the road surface is removed. 7. A predetermined area is divided into a plurality of areas, the total length of optical flow vectors existing in each area is weighted for each area, and the degree of danger is determined by the magnitude of the weighted value. The vehicle front monitoring method according to claim 1, wherein the determination is made. 8. The method according to claim 7, wherein a predetermined threshold value is set for each area, and an area in which the sum of the lengths of the optical flows exceeds the threshold value is judged to be dangerous. Forward monitoring method for vehicles. 9. The vehicle according to claim 8, wherein the threshold value is composed of several levels, and the risk level is determined by the threshold value level over which the sum of the optical flow lengths exceeds. For forward monitoring method. 10. The vehicle front monitoring method according to claim 9, wherein an alarm is issued according to the determined degree of danger.