JP3292292B2 - Optical flow detection method - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION This invention is, whether the vehicle you are traveling
Of two frames before and after a predetermined time
Optical that detects a single point movement as an optical flow
The present invention relates to a physical flow detection method.

[0002]

2. Description of the Related Art Conventionally, as this kind of method, there is a method disclosed in Japanese Patent Application No. 2-241855. In the method disclosed in the publication, a foreground is photographed from a running own vehicle, the movement of the same point of the photographed foreground is recognized as an optical flow at predetermined time intervals, and the distance between the optical flow and a preceding vehicle is determined. Based on a distance sensor that measures the distance, information on the position and relative speed of the preceding vehicle with respect to the own vehicle is obtained, and when it is determined that there is danger based on this information, the driver is notified of the danger.

Further, in order to obtain an optical flow, a technique called a matching method for detecting a corresponding point between two frames of images has conventionally been adopted. In this matching method, a window W1 is taken for a pixel P of interest on the image at the time t shown in FIG. 9A, and a correlation value is calculated while moving the window for the entire region or a peripheral region in the image,
As shown in FIG. 9B, the window W2 having the maximum correlation value
Is obtained as a corresponding point, that is, a corresponding pixel Q, and this PQ (arrow) becomes an optical flow. To obtain the correlation value, 値 (W1 _{(x, y)} × W2 _{(x, y)} ) / (ΣW1 ² _{(x, y)} × ΣW2 ² _{(x, y)} ) ^1/2 (1) Perform the following calculation. Note that W1 _{(x, y)} and W2 _{(x, y)} are the windows W
1, output of (x, y) coordinates in W2.

[0004]

However, in such a conventional method, since the optical flow is obtained, two
It is necessary to perform an arithmetic process of detecting a corresponding point between two images, but as described above, this arithmetic process is performed on one pixel in the image in order to detect a certain corresponding point. The search is performed for all the pixels in the whole area or the surrounding area. This process must be performed on all pixels in order to detect corresponding points over the entire image. Also, many calculations are required to obtain a correlation value used as an index for performing a search.

[0005] Accordingly, there is a problem that the amount of calculation is enormous and it is difficult to realize real-time operation, or a high-speed arithmetic processing unit is required for realization, so that the apparatus itself becomes expensive. Is raised.

Accordingly, the present invention has been made in view of the above-mentioned conventional problems, and enables an image taken by a video camera to detect a movement of the same point at predetermined time intervals as an optical flow at a high speed. Optical flow
It aims to provide a detection method.

[0007]

To achieve the above object, an optical flow detecting method according to the present invention comprises:
An optical flow detection method for detecting, as an optical flow, a movement of the same point in two frames preceding and following a predetermined time obtained by capturing an image of a running vehicle,
In the previous image of the two frames before and after the predetermined time, from the point at infinity corresponding to one point indicating the imaging direction from the running own vehicle in the radial direction with respect to the point of interest An elongated window is set, and in the subsequent image, while moving the window in the radial direction from the point at infinity, the absolute value of the brightness difference between the window in the previous image and the area of the image after the window overlapping the window is set. A sum is obtained, and an arrow connecting between respective center points of the position of the window when the sum is minimized and the set position in the image in front of the window is defined as one optical flow.

[0008]

According to the above method, focusing on the fact that the optical flow is formed in a radial direction from the point at infinity corresponding to one point indicating the imaging direction from the own vehicle, the front of the two-frame image In the image of, an elongated window is set for one point in the radial direction from the point at infinity corresponding to one point indicating the imaging direction from the running own vehicle, and this window is radiated from the point at infinity. While moving on the subsequent screen in the direction, find the sum of the absolute values of the luminance differences between the window in the previous image and the area of the image that overlaps this window, and the position of the window when this sum is minimized The arrow connecting between the center points of the image and the set position in the image in front of the window is defined as one optical flow, so that the calculation may be performed within the window set in the processing. Of processing Speed up can be performed.

[0009]

Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 shows the configuration of an apparatus for carrying out a method according to the present invention. In FIG. 1, reference numeral 1 denotes a video camera for capturing a foreground, 2 denotes an input of an image obtained by a video camera 1, and image processing described below is performed. The arithmetic processing unit 3 performs a speedometer for measuring the speed of the own vehicle, and the arithmetic processing unit 4 inputs the image processing result of the arithmetic processing device 2 and the own vehicle speed obtained from the speedometer 3 to perform a risk determination process. The device 5 is an alarm device.

FIGS. 2A and 2B are diagrams for explaining changes in the foreground image obtained by the video camera 1. FIG. 2B shows the situation in which the video camera 1 includes the own vehicle shown in FIG.
Indicates an image captured at time t, and (c) indicates an image captured at time t + Δt.

Now, it is assumed that the vehicle is traveling straight on a flat road. For example, when attention is paid to the road sign and the building shown in (a), images as shown in (b) and (c) are obtained at time t and time t + Δt with the passage of time.
By searching for corresponding points in these two images and connecting them, a velocity vector as shown in (d) is obtained. This is the optical flow.

[0012] Here, these optical flows appear radially from one point called FOE (Focus of Expansion) in the image. The FOE is called an infinity point or a vanishing point, and corresponds to one point indicating an imaging direction which is a traveling direction of the own vehicle on an image when the vehicle is traveling straight. Thus, the optical flow required when the host vehicle is traveling is in a radial direction from the FOE. Here, the optical flow issued from the preceding vehicle includes information including the position and the relative speed of the preceding vehicle with respect to the own vehicle, and it is considered that the longer the optical flow, the higher the risk.

Next, the details will be described with reference to FIG. In the optical arrangement shown in the figure, 11 is a lens of a video camera, 12 is an image plane of the video camera, f is a distance from the lens 11 to the image plane 12, P
(X, Y, Z) is an arbitrary point on another preceding vehicle or obstacle, and p (x, y) is a point P on the image plane 12.
X = f × X / Z (2) from the similarity ratio of the triangle.

By transforming this equation and differentiating it over time, X '= (Δx / Δt · Z + x · Z ′) / f (3) Further, since the x-direction component u of the optical flow is u = Δx / Δt (4), using this, Z = (fX′−xZ ′) / u (5) Become.

Here, Z '= the relative speed between the preceding vehicle and the other vehicle or obstacle and the host vehicle = -α (6). u ... (7) Therefore, the x-direction component u of the optical flow is as follows: u = (f × X ′ + xα) / Z (8) The same applies to Y.

Therefore, from the above equation (8), as Z is smaller, that is, as the distance to another preceding vehicle or obstacle is smaller,
Alternatively, the larger the α is, that is, the larger the relative speed with the preceding vehicle is, the larger the x component of the optical flow becomes. This is the same in the Y direction. Therefore, the optical flow becomes longer as the distance to the preceding vehicle or the like becomes smaller and further as the relative speed becomes larger, and when the optical flow is longer than this, the other preceding vehicle or obstacle is relatively longer. It is considered that the danger to the object is high.

In the present invention, the optical flow is FOE.
The optical flow is obtained at a high speed by utilizing the fact that the optical flow is obtained in a radial direction from. The method will be described below with reference to FIG.

FIG. 4 is a diagram showing an embodiment of a method for obtaining an optical flow at high speed. First, an elongated window is set in a radial direction from the FOE for a point of interest in an image at time t (FIG. 4A). Next, in the image at the time t + Δt, the window is moved one point at a time in the radial direction from the FOE, and the sum of the absolute values of the luminance differences from the window at the time t is obtained. Then, the moving amount of the window when the sum is minimized is obtained as a speed vector at a point of interest (FIG. 4B). The luminance difference is, for example, between pixels at corresponding positions indicated by a circle in (a) and (b) for each pixel constituting the window.
By repeating the above processing at all points of the image at the time t, the optical flow of the entire image can be obtained.

Further, in the conventional method, when comparing corresponding windows, a cross-correlation value is used. On the other hand, in the method of the present invention, since the sum of the absolute values of the luminance differences is used, the calculation amount is reduced, and the processing can be speeded up.

In the present invention, instead of determining the velocity vector for every point in the image, time t and time t
The difference is obtained from the image with + Δt, and processing is performed only on the point that exceeds a certain threshold value, which is the difference.

Generally, in an image showing a foreground while a vehicle is running, there are many places where there is no temporal luminance change, such as the sky or a road surface. In such a place, it is impossible in principle to find an optical flow. Thus, as in the method of the present invention, within the image at time t and the image at time t + Δt,
By focusing on the fact that there is a luminance difference,
Unnecessary processing can be omitted, and the speed can be increased.

An optical flow can be obtained at high speed by the above-described method. However, the optical flows obtained here include optical flows generated by non-obstacles such as landscapes other than roads and lanes on the road surface. Therefore, it is difficult to determine the presence of an obstacle ahead or the degree of danger due to the obstacle using the optical flow determined here.
Therefore, it is necessary to remove the optical flow generated from the scenery outside the road or the lane on the road surface. Next, this method will be described.

FIG. 5 shows an embodiment of a method for removing an optical flow due to a scene outside the road. In the figure, it is assumed that the shaded position indicates a road, and that no processing is performed in other places.
By doing so, the optical flow generated from the scenery outside the road is not obtained from the beginning, and the processing time can be shortened by limiting the area.

Next, a method for erasing a mark on a road surface will be described with reference to the drawings. FIG. 6 shows an embodiment of this method. First, the optical configuration will be described. FIG. 6 is drawn on the assumption that the entire road surface is approaching, instead of the vehicle equipped with the video camera going straight. Reference numeral 11 denotes a lens of a video camera that captures an image of the front, and reference numeral 12 denotes an image plane of the video camera. 13 and 14 represent lanes before and after the movement. Consider three-dimensional coordinates with the center of the lens as the origin and two-dimensional coordinates with the FOE as the origin on the image plane 2.

Let the position of the lane before moving be P (X, Y, Z + Δ
Z), assuming that the position after the ΔZ movement in the Z direction is Q (X, Y, Z), the similarity between the two triangles indicates the distance from the video camera to the lane where the optical flow occurs as shown in equation (1). You can know the height. Y = y ² ΔZ / fΔy (9) Using the obtained height information, removing the optical flow whose height matches the height of the video camera, that is, which is caused by a mark on the road surface or the like. Can be.

Here, ΔZ is a time interval Δt between two images.
Is the distance that the vehicle has advanced, and can be obtained by knowing the vehicle speed. By the processing described above, it is possible to remove an optical flow generated from an object other than the obstacle in front. Therefore, it is possible to determine the degree of danger indicating how dangerous the obstacle ahead is from the position and length of the optical flow that is currently left.

Next, a method for obtaining the degree of risk will be described. FIG. 7 shows one embodiment of this method. First, the configuration will be described. Each of the regions I to IV indicates the distant position of the own lane, the near position of the own lane, the distant position of the adjacent lane, and the close position of the adjacent lane, respectively. Here, the risk level is obtained by weighting the sum of the lengths of the optical flows existing in each area for each area, and using the weighted value. Further, a predetermined threshold value is set for each area, and if the sum of the flow lengths exceeds this threshold value, the area exceeding the threshold value is determined to be dangerous. A plurality of threshold values can be set, and the level of risk can be determined.

Finally, it is possible to call the driver's attention by sounding an alarm according to the magnitude of the determined degree of danger. It is also possible to display which area is dangerous and how much is on the display. Further, it is also possible to change the type of alarm sound and the like depending on the danger level.

FIG. 8 summarizes the procedure of the image processing according to the method of the present invention described above. First, an image at time t is captured in step S1, and then an image at time t + Δt is captured in step S2.
Thereafter, the FOE is set in step S3, and the processing area is set in subsequent step S4. Subsequently, the process proceeds to step S5, where an area where the luminance difference exceeds a certain threshold value is extracted from the images at time t and time t + Δt. Thereafter, the process proceeds to step S6, where an optical flow is obtained in the extracted area. In the next step S7, the optical flow on the road is removed, and in the next step S8, the degree of risk is calculated by a value weighted for each area.

[0030]

As described above, according to the present invention, 2
Without performing an arithmetic process of detecting a corresponding point between two images, an elongated window set for one point of interest in a radial direction from an infinite point in a previous image of two frames is used. While moving in the radial direction from the point at infinity on the subsequent screen, the sum of the absolute values of the luminance differences between the window in the previous image and the area of the subsequent image overlapping this window is obtained, and this sum is minimized. The arrow connecting the center point between the window position at the time of the window and the set position in the image in front of the window is defined as one optical flow, and calculation is performed within the window set in the processing. All right,
Since the amount of calculation is reduced and the processing speed is increased, real-time processing is facilitated.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating an example of an apparatus for performing a method according to the present invention.

FIG. 2 is a diagram showing a foreground, an image, and an optical flow obtained by the video camera shown in FIG. 1;

FIG. 3 is a diagram for explaining how to detect an obstacle or the like by the method of the present invention.

FIG. 4 is a diagram for explaining how to determine an optical flow by the method of the present invention.

FIG. 5 is a diagram showing an example of an area set for obtaining an optical flow according to the method of the present invention.

FIG. 6 is a diagram for explaining a method of removing an optical flow by a lane on a road surface according to the method of the present invention.

FIG. 7 is a diagram showing an example of an area divided for judging the degree of risk by the method of the present invention.

FIG. 8 is a diagram showing a series of processes of the method of the present invention.

FIG. 9 is a diagram for explaining a problem of the conventional method.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Video camera 2, 4 Arithmetic processing unit 5 Alarm device

────────────────────────────────────────────────── ─── Continued on the front page (51) Int.Cl. ⁷ Identification code FI G08B 21/00 G08B 21/00 H H04N 7/18 H04N 7/18 J (72) Inventor Kazuyuki Sasaki 1500 Onjuku 1500, Susono City, Shizuoka Prefecture Yazaki In-house company (56) References JP-A-2-241855 (JP, A) JP-A-4-85148 (JP, A) JP-A-4-16647 (JP, A) JP-A-4-260979 (JP) , A) (58) Fields investigated (Int. Cl. ⁷ , DB name) B60R 21/00 G06T 7/20 H04N 7/18 G08B 21/00 G03B 15/00

Claims (1)

(57) [Claims] An optical flow detection method for detecting, as an optical flow, a movement of the same point in two frames before and after a predetermined period of time obtained by capturing an image of a running host vehicle, wherein the predetermined time period is determined. In the preceding image of the two successive images, a window elongated in the radial direction from the point at infinity corresponding to one point indicating the imaging direction from the running own vehicle to the point of interest. While moving the window in the subsequent image in the radial direction from the point at infinity, the sum of the absolute values of the luminance differences between the window in the previous image and the area of the image after the window overlapping the window is calculated. Optical flow detection , wherein an arrow connecting between respective center points of the position of the window when the sum is minimized and the set position in the image in front of the window is determined as one point of optical flow. Method.