JP3585071B2 - Vehicle periphery monitoring device - Google Patents

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention uses an image captured by a video camera installed at the front or rear of a vehicle such as an automobile, and, when the vehicle is running, an obstacle existing in the front, behind, or in an adjacent lane or around the vehicle. The present invention relates to a vehicle periphery monitoring device that detects a vehicle periphery and warns a driver.
[0002]
As this type of vehicle periphery monitoring device, for example, there is a device described in Japanese Patent Application Laid-Open No. 6-107096 (hereinafter, referred to as a first conventional device).
This first conventional apparatus captures an image of a foreground from a running own vehicle, obtains the movement of the same point of the captured foreground from two frames before and after a predetermined time, and determines the movement of the same point by an optical flow. The monitoring is performed as follows.
[0003]
More specifically, in the first conventional apparatus, one point of interest is set in a previous image in two consecutive frames, which is an image captured in a predetermined imaging cycle. From the point at infinity corresponding to one point indicating the traveling direction of the traveling own vehicle, an elongated window centered on the point of interest is set in the radial direction with respect to the set point of interest, The luminance value of the pixel group constituting the image in the set window is stored.
[0004]
Then, in the subsequent screen, while moving a window similar to the above-described window in the radial direction from the point at infinity in the subsequent image, the luminance value of the pixel group forming the image in this window and the pixel in the previous image are moved. The sum total of the absolute values of the differences between the luminance values of the pixels corresponding to each other is calculated.
Then, the amount of movement of the window when the sum is minimized is determined as the optical flow of the point of interest at the point approaching the vehicle, and the object approaching the vehicle based on the position and size of the optical flow, For example, it monitors a preceding vehicle traveling at a lower speed than the own vehicle or an obstacle on the road.
[0005]
In addition, there is an apparatus described in Japanese Patent Application Laid-Open No. 7-50769 (hereinafter, referred to as a second conventional apparatus).
This second conventional apparatus captures an image of a rear view from a running own vehicle, obtains the movement of the same point of the captured rear view from images of two frames before and after a predetermined time, and obtains the image of the same point. The movement is monitored as an optical flow.
[0006]
More specifically, one point of interest is set in a previous image of two consecutive frames that are imaged at a predetermined imaging cycle and indicate a direction directly opposite to the traveling direction of the own vehicle while traveling. From the point at infinity corresponding to the point, an elongated window centering on the set point of interest is set in the radial direction with respect to the set point of interest.
[0007]
Then, on the subsequent screen, one point of interest is detected while moving a window similar to the above-mentioned window from the point at infinity in the subsequent image in the diverging direction, and the point between the previous image and the subsequent image is detected. The connecting arrow is obtained as one optical flow, and an object approaching the own vehicle, for example, a rear vehicle running at a higher speed than the own vehicle is monitored based on the position and the size of the optical flow.
[0008]
[Problems to be solved by the invention]
However, in these first and second conventional apparatuses, an elongated window is set in a radial direction with respect to a point of interest from a point at infinity in a previous image, and the elongated window is set in a diverging direction in a subsequent image. Is detected while moving to the point of interest. If the point of interest in the previous image and the point of interest in the subsequent image are similar patterns, the point of interest is detected. May be erroneously detected as the same point of interest even if the points are different object points.
[0009]
In particular, when objects of the same shape, such as fences, guardrails, or utility poles, are arranged at intervals on a straight line, the objects will appear periodically, and between the two images described above, The one point of interest in the previous image and the one point of interest detected in the later image may be different object points.
For example, as shown in the images of two frames before and after a predetermined time in FIGS. 12A and 12B, the point P1 of the object S1 in the previous image (FIG. In some cases, the point S2 is erroneously detected as the point P2 of the object S2 in FIG.
[0010]
The present invention has been made in view of such circumstances, and a main object of the present invention is to provide a vehicle periphery monitoring device that can reduce erroneous detection of a single point of interest.
Another object of the present invention is to provide a vehicle periphery monitoring device that can determine the erroneous detection by a simple process. As described above, objects of the same shape such as a fence, a guardrail, and a utility pole periodically appear. Even in such a case, it is still another object to provide a vehicle periphery monitoring device capable of reducing erroneous detection of one point of interest.
[0011]
[Means for Solving the Problems]
In order to solve the above problem, a vehicle periphery monitoring device according to the present invention captures a peripheral image from a running own vehicle as shown in a basic configuration diagram of FIG. The movement of the same point in the image is defined as an infinity corresponding to one point indicating the traveling direction of the own vehicle traveling in the previous image of the two frames preceding and succeeding the predetermined time or the direction opposite to the traveling direction. From the far point, set a narrow window in the radial direction for one point of interest, detect the one point of interest while moving the window in the radial direction in a later image, and detect the one point of interest in the previous image. A vector connecting one point of interest and one point of interest in a subsequent image is set as an optical flow, and another vehicle traveling around the traveling own vehicle, or an obstacle present in the vicinity is identified as an optical flow. Run around Another vehicle in the middle, or in a vehicle periphery monitoring device that monitors according to the magnitude and the appearance position of the vector of the optical flow of an obstacle present in the vicinity, whether or not the point of interest is erroneously detected An erroneous detection judging means (31a) for judging that the optical flow is not set for one point of interest determined to be erroneously detected by the erroneous detection judging means (31a). It is characterized by doing. (Claim 1)
[0012]
In the configuration of the first aspect, the surrounding image is captured at predetermined time intervals from the running own vehicle.
One point of interest is selected for the previous image of the two images before and after the predetermined time, and attention is paid from the infinity point indicating the traveling direction of the own vehicle or the opposite direction of the traveling direction. An elongated window is set in a radial direction for one point.
For the subsequent image, an elongated window similar to the previous image is set at the same position as the elongated window in the previous image described above, and the set window is moved in the radial direction described above. , The same point as the point of interest in the previous image is detected.
If a point of interest is detected in a later image, the erroneous detection determination unit determines whether the detected point of interest is correct or not.
[0013]
Then, regarding the point of interest determined to be properly detected by the erroneous detection determination means, a point between the point of interest set in the previous image and the point of interest detected in the subsequent image is determined. The connected vector is defined as an optical flow. On the other hand, the optical flow is not set for one point of interest determined to be erroneously detected by the erroneous detection determination unit.
[0014]
Therefore, in the configuration of the first aspect, it is determined whether one point of interest is an erroneously detected point from one point of interest in the previous image and one point of interest detected in the subsequent image. It is configured such that no optical flow is set for a point determined to be erroneously detected, so that erroneous detection of one point of interest can be reduced.
[0015]
A first point-of-interest detection means (31b) for detecting the corresponding point of the one point of interest while moving the window from the set position in the previous image in the diverging direction from the point at infinity in the subsequent image ) And when the corresponding point is detected by the first point-of-interest detection means (31b), the third image picked up after the predetermined time from the subsequent image is infinitely shifted from the detection position in the subsequent image. A second point-of-interest detection means (31c) for detecting the corresponding point of the one point of interest while moving in the direction of divergence from the far point, wherein the erroneous detection determination means (31a) A vector connecting a point of interest in the image of interest and the point of interest in the subsequent image is set as a first provisional optical flow, and a point of interest in the subsequent image and the vector in the third image are set. A vector connecting one point of interest is set as a second temporary optical flow, and the one point of interest is an erroneously detected point by comparing the first temporary optical flow and the second temporary optical flow. Is determined. (Claim 2)
[0016]
In the configuration of the second aspect, the first point-of-interest detection means detects a corresponding point of one point of interest while moving the window in the diverging direction from the position set in the previous image in the subsequent image. When the corresponding point is detected by the first point-of-interest detection means, the second point-of-interest detection means opens a window in the diverging direction from the detection position in the subsequent image in the third image taken a predetermined time after the subsequent image. While moving, one corresponding point of interest is detected.
[0017]
The erroneous detection determination means sets a vector connecting a point of interest in the previous image and a corresponding point in the subsequent image as a first provisional optical flow, and sets a corresponding point in the subsequent image and a corresponding point in the third image. Is set as the second temporary optical flow.
Then, the imaging interval between the previous image and the subsequent image, and the imaging interval between the subsequent image and the third image are equal to each other, and the approaching state of the object approaching the own vehicle is determined by the Paying attention to the fact that there is no significant change between the imaging interval between the subsequent images and the imaging interval between the subsequent image and the third image, the erroneous detection determination unit determines the first temporary optical flow and the second temporary optical flow. By comparing the flow with the flow, it is determined whether or not one point of interest is an erroneously detected point.
[0018]
Therefore, according to the configuration of the second aspect, the first provisional optical flow set for the previous image and the subsequent image, and the third image captured after a predetermined time from the subsequent image and the subsequent image By comparing with the second provisional optical flow set between the above, the erroneous detection of one point of interest can be easily determined, and the erroneous detection can be reduced.
[0019]
The erroneous detection determination means (31a) determines whether the point of interest is an erroneously detected point by comparing the lengths of the first tentative optical flow and the second tentative optical flow. It is characterized by determining. (Claim 3)
[0020]
In the configuration according to the third aspect, the erroneous detection determination means may determine that the optical flow appears in a radial direction from the point at infinity, specifically, the optical flow at the same point is a straight line connecting this point and the point at infinity. By comparing the lengths of the above-mentioned first temporary optical flow and second temporary optical flow, it is determined whether or not one point of interest is an erroneously detected point by paying attention to the property appearing in the direction of. .
[0021]
Therefore, according to the configuration of the third aspect, the determination is made by comparing the lengths of the first temporary optical flow and the second temporary optical flow, so that the erroneous detection of one point of interest is further improved. The determination can be made easily, and erroneous detection can be reduced.
[0022]
A first point-of-interest detecting means (31b) for detecting the correspondence of the one point of interest while moving the window from the set position in the previous image in the diverging direction from the point at infinity in the subsequent image And a third point-of-interest detection means (31d) for detecting the corresponding point of the one point of interest while moving the window in the subsequent image in the direction of convergence from the set position in the previous image to the point at infinity. ), And the erroneous detection determination means (31a) receives the corresponding point of the one point of interest from both the first point of interest detection means (31b) and the third point of interest detection means (31d). Is detected, it is determined that the point of interest is erroneously detected. (Claim 4)
[0023]
In the configuration of the fourth aspect, the first point-of-interest detection means detects a corresponding point of one point of interest while moving the window in the subsequent image from the position set in the previous image in the diverging direction. The third point of interest detection means detects a corresponding point of one point of interest while moving the window from the position set in the previous image to the point at infinity in the subsequent image.
The erroneous detection determination means, based on the detection results by the first point of interest detection means and the third point of interest detection means, when a corresponding point is detected by both the first point of interest detection means and the second point of interest detection means, This point of interest is determined to be erroneously detected as having a high possibility of being erroneously detected. This determination is made by focusing on one point of interest in the previous image and one point corresponding to this one point of interest in the subsequent image.
The optical flow is not set for one point of interest determined to be erroneously detected by the erroneous detection determination unit.
[0024]
Therefore, according to the configuration of claim 4, when one point of interest in the previous image appears in two directions, that is, the divergence direction and the convergence direction with respect to the point at infinity, the point of interest is erroneously determined. Since it is determined that a point has been detected, erroneous detection of one point of interest can be eliminated.
[0025]
The third point-of-interest detection means (31d) performs detection on the detected one point when the first point-of-interest detection means (31b) detects one point of interest. I have. (Claim 5)
[0026]
In the configuration of the fifth aspect, when the first point of interest detection means detects one point of interest, the third point of interest detection means executes detection processing for the detected one point.
This makes it possible to easily and easily perform the processing without performing useless processing.
[0027]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, specific examples of the embodiments of the present invention will be described with reference to the drawings.
FIG. 2 is a block diagram showing a configuration of the first embodiment of the vehicle periphery monitoring device to which the present invention is applied.
In FIG. 1, reference numeral 10 denotes an imaging unit, 20 denotes a storage unit, 30 denotes a data processing unit, and 40 denotes a display alarm unit.
[0028]
The imaging unit 10 is configured as a mechanism that captures an image around the vehicle, and has a video camera 11. The video camera 11 is attached to a ceiling (not shown) of a vehicle, and captures an image of the front, side, or rear of the vehicle.
[0029]
The storage unit 20 is configured as a mechanism for temporarily storing an image captured by the imaging unit 10, and has a first frame memory 21, a second frame memory 22, a third frame memory 23, and a divergent optical flow memory 24. are doing.
Each of the first to third frame memories 21 to 23 is a memory for storing a captured image from the video camera 11, and the first frame memory 21 has a time t, and the second frame memory 22 has a time t + △. t, the third frame memory 23 stores the time t + {t + {t,. . . In this way, images captured at every predetermined time Δt are sequentially stored.
The divergent optical flow memory 24 stores an optical flow in a divergent direction obtained by image processing (described later).
[0030]
The data processing unit 30 is configured as a mechanism that inputs an image obtained by the imaging unit 10 and performs image processing and risk determination processing (described later), and operates according to an operation program describing the image processing and risk determination processing. And a RAM 33 for temporarily storing information necessary for the operation of the CPU 31, such as a CPU 32, an operation program, predetermined coefficient information, and the like.
[0031]
The display / alarm unit 40 is configured as a mechanism that displays an image obtained by the imaging unit 10 and that issues an alarm to notify a danger by an image or a sound, and provides a driver or the like with an image obtained by the imaging unit 10 or a danger. And a speaker 42 that emits a voice message indicating danger.
[0032]
Next, in the first embodiment having the above configuration, a detection principle when another vehicle or an obstacle is detected by an optical flow will be described.
FIG. 3 is a diagram for explaining a change in the foreground image obtained by the video camera 11, and FIG. 3B illustrates a state in which the video camera 11 is moved at time t in a situation including the host vehicle 100 illustrated in FIG. 3 (c) shows a foreground image captured at time t + Δt.
[0033]
Hereinafter, an example will be described in which the foreground image is captured, and other vehicles traveling in the front and adjacent lanes of the own vehicle 100 or obstacles existing in the front, adjacent lanes, and roadside zones are monitored from the foreground image. However, the process of capturing the foreground image of the own vehicle 100 with the video camera 11 and monitoring the other vehicles traveling behind and adjacent lanes or the obstacles existing in the adjacent lane and roadside zone from the foreground image includes: Since the processing is the same as the processing with the foreground image, the processing with the foreground image will be described here, and the description thereof will be omitted.
[0034]
Now, it is assumed that the vehicle is traveling straight on a flat road. For example, paying attention to the road sign and the building shown in FIG. 3A, images as shown in FIGS. 3B and 3C are obtained at time t and time t + Δt with the passage of time. By searching for corresponding points in these two images and connecting them, a velocity vector as shown in FIG. 3D is obtained. This is the optical flow.
[0035]
Here, these optical flows appear radially from one point called an FOE (Focus of Expansion) in the image. The FOE is called a point at infinity or a vanishing point, and corresponds to one point indicating the traveling direction of the own vehicle on the foreground image when the vehicle is traveling straight. When monitoring is performed using the background image, the FOE corresponds to one point on the background image that indicates the direction opposite to the traveling direction of the host vehicle.
Thus, the optical flow required when the host vehicle is traveling is in a radial direction from the FOE. Here, the optical flow emitted from the preceding vehicle (in the case of rearward monitoring, the following vehicle) includes information including the position and the relative speed of the preceding vehicle with respect to the own vehicle, and the longer the optical flow, the higher the risk. Conceivable.
[0036]
Next, the details will be described with reference to FIG. FIG. 1 is a diagram showing an optical arrangement for explaining how to detect an obstacle or the like.
In the figure, 11a is a lens of the video camera 11, 12 is an image plane of the video camera 11, f is a distance from the lens 11a to the image plane 12, and P (X, Y, Z) is on a preceding vehicle or an obstacle. Is an arbitrary point, p (x, y), corresponding to the point P on the image plane 12.
[0037]
In this optical arrangement, the x coordinate value of the point p on the image plane 12 can be expressed by the following equation (1) from the similarity ratio of the triangles.
x = f · X / Z (1)
[0038]
By transforming equation (1) and performing time differentiation, the following equation (2) is obtained.
X ′ = (Δx / Δt · Z + x · Z ′) / f (2)
[0039]
Further, since the x-direction component u of the optical flow is a change amount (Δx) of the x coordinate value per unit time (Δt), the following equation (3) is obtained.
u = Δx / Δt (3)
[0040]
The following equation (4) is obtained from the equations (2) and (3).
Z = (f.X'-x.Z ') / u (4)
[0041]
Here, since Z 'is the relative speed between the preceding vehicle or obstacle and the host vehicle, if this relative speed is -α, the following equation (5) is obtained.
Z ′ = − α (5)
[0042]
From the equations (4) and (5), the following equation (6) is obtained.
Z = (f · X ′ + xα) / u (6)
[0043]
Therefore, the x-direction component u of the optical flow can be expressed by the following equation (7).
u = (f × X ′ + xα) / Z (7)
Also, Y can be obtained in the same manner.
[0044]
Therefore, according to equation (7), Z is smaller, that is, the smaller the distance to the preceding vehicle (the following vehicle) or the obstacle, or the larger α, that is, the larger the relative speed with the preceding vehicle (the following vehicle). The x component of the optical flow becomes larger as the optical flow becomes larger. This is the same in the Y direction.
[0045]
Therefore, the optical flow becomes longer as the distance from the preceding vehicle or the like becomes shorter and the relative speed becomes larger, and the optical flow becomes longer when the optical flow is shorter than when the optical flow is shorter. Alternatively, it is considered that there is a high degree of danger to obstacles.
[0046]
In the present invention, the optical flow is obtained by utilizing the fact that the optical flow is obtained in a radial direction from the FOE. This procedure will be described with reference to the schematic diagram of FIG.
[0047]
First, as shown in FIG. 5A, in the image at the time t (that is, the previous image), a point of interest is radiated from the FOE in the radial direction (that is, FOE An elongated window is set in the direction connecting one point of interest). Next, as shown in FIG. 5B, in the image at the time t + Δt (that is, the subsequent image), while moving the window one point at a time in the radial direction from the FOE, the brightness difference from the window at the time t is obtained. Find the sum of absolute values. Then, the moving amount of the window when the sum is minimized is obtained as a speed vector of one point of interest.
[0048]
Note that the above-mentioned luminance difference is between the pixels at the corresponding positions for each pixel constituting the window, for example, as indicated by ● in FIGS. 5 (a) and 5 (b).
By repeating the above processing for all points of the image at time t, the optical flow of the entire image can be obtained.
[0049]
Next, the operation of the first embodiment having the above-described device configuration and detection operation (principle) will be described with reference to the flowchart of FIG.
In the first embodiment, in step S110, a foreground image (background image) at time t is captured by the video camera 11, and the obtained foreground image (background image) is used as a first frame image in the first frame. It is stored in the memory 21.
In step S111, a foreground image (background image) is captured at time t + Δt after a lapse of a predetermined time Δt from the time of step S110, and the obtained foreground image (background image) is used as a second frame image in the second frame. It is stored in the memory 22.
In step S112, a foreground image (background image) is captured at a time t + Δt + Δt after a lapse of a predetermined time Δt from the time of step S111, and the obtained foreground image (background image) is set as the third frame image. It is stored in the third frame memory 23.
[0050]
In step S113, an FOE is set for each of the images obtained in steps S110 to S112. Since the FOE is determined by the installation position and the angle of the video camera 11, a predetermined coordinate value is given as the FOE in the setting processing in step S113.
[0051]
In step S120, one point of interest is selected. The one point of interest selected in step S120 is selected, for example, as a point on the edge in the edge image of the object obtained by differentiating the image of the first frame (the image at time t). In addition, the point is selected as a point existing in a predetermined area (for example, an area in a straight traveling direction) having a risk of collision with the own vehicle.
[0052]
In step S130, the process described with reference to FIG. 5 is performed on the first frame image and the second frame image. More specifically, a point of interest and an elongated window are set in the image of the first frame, and the window is moved in the diverging direction along the radial direction from the same position as the image of the first frame in the image of the second frame. By calculating the sum of the absolute values of the luminance values with the window set in the first frame, the optical flow between the first frame image and the second frame image for the point of interest set in step S120 is determined. FL1) is detected.
[0053]
In a succeeding step S131, a determination based on the processing of the step S130 is performed. That is, in this step S131, when the optical flow is detected in the above step S130 (Y), the process proceeds to step S132, and when the optical flow is not detected (N), the process proceeds to step S150.
[0054]
In step S132, the process described with reference to FIG. 5 is performed on the second frame image and the third frame image. More specifically, in the image of the third frame, the position where the sum of the absolute values becomes the minimum in the second frame while moving the window in the diverging direction from the position where the sum of the absolute values becomes the minimum in step S130. The sum of the absolute value of the luminance difference with the window of the image is calculated, and the optical flow (FL2) between the image of the second frame and the image of the third frame is detected for the point of interest set in step S120. . Then, control goes to a step S140.
[0055]
In step S140, the length of the optical flow FL1 obtained in step S130 is compared with the length of the optical flow FL2 obtained in step S132, and whether the optical flows FL1 and FL2 are the same point optical flow is determined. Is determined.
[0056]
Here, the reason why it is possible to determine whether or not both optical flows FL1 and FL2 are optical flows at the same point by comparing the lengths of optical flow FL1 and optical flow FL2 will be described.
[0057]
In this description, when the other vehicle is traveling at a speed of 50 km / h and the own vehicle is traveling behind the other vehicle at a speed of 60 km / h, the distance that the vehicle approaches during the imaging interval Δt is equal to the distance between the other vehicle and the own vehicle. And a relative speed of 10 km. At this time, since the imaging interval Δt is short, it can be considered that there is no significant change in the relative speed and the steering angle of the vehicle during the imaging period. The same applies to the case where the change between time t and time t + Δt is compared with the change between time t + Δt and time t + 2Δt.
[0058]
As described with reference to FIG. 4, the optical flow has a property that it appears larger as it comes closer to the host vehicle. However, since the imaging interval Δt is as short as described above, the time t, which is the imaging period, is short. , T + Δt and t + 2Δt, there is no significant change in the distance that the two vehicles approach, and it can be considered that the distance falls within a certain range on the obtained captured image.
The optical flow at the same point can be regarded as appearing in a direction along a straight line connecting the point of interest and the FOE point.
[0059]
From the above, regarding the optical flow FL1 and the optical flow FL2 acquired during the period from the time t to the time t + 2 △ t, if the optical flow FL1 and the optical flow FL2 are at the same point, they have the same direction. It can be seen that the length is within a certain range based on one of the lengths.
In the first embodiment, the length of the optical flow FL2 is in a range from half (× 0.5) to twice (× 2) the length of the optical flow FL1 because (the base of the range is required). If they are within the range, it is determined that both optical points are the same.
[0060]
Therefore, in step S140, if the length of the optical flow FL2 is within the range represented by the following equation (8), it is determined that the two optical flows FL1 and FL2 are at the same point.
0.5FL1 <FL2 <2FL1 (8)
[0061]
In this step S140, if the length of the optical flow FL2 is within the predetermined range with respect to the optical flow FL1 (Y), the process proceeds to step S141, and if it is outside the predetermined range (N), the process proceeds to step S142. Move to
In step S141, the optical flow FL1 detected in step S131 is stored in the divergent optical flow memory 24. In step S142, the optical flow FL1 detected in step S131 is determined to be not the same point optical flow. Do not set optical flow for points.
Then, after the processing of step S141 or S142 is completed, the process proceeds to step S150.
[0062]
In step S150, it is determined whether or not processing has been performed for all target points with respect to the points to be detected for the optical flow. If the processing of all points has been completed (Y), the process proceeds to step S160, and if there are still unprocessed target points (N), the process proceeds to step S120 to newly focus one point of interest. Is set and processing is continued.
[0063]
In step S160, the degree of risk is calculated. In this step S160, the optical flows stored in the divergent optical flow memory 24 are compared with the regions of the own lane divided in advance, the near lane of the own lane, the farthest of the adjacent lane, and the vicinity of the adjacent lane. The sum of the lengths of the optical flows existing in each region is calculated.
[0064]
In step S161, it is determined whether it is dangerous. The determination process in step S161 is performed based on the threshold value of the risk level set by weighting for each region. When the total sum of the lengths of the optical flows for each region exceeds this threshold value, It is determined that an area exceeding the threshold is dangerous.
Note that the threshold value for each area in step S161 is stored in the ROM 32 in advance. Further, regarding the determination processing in step S161, the threshold may be set at several levels, and the determination may be made based on the level of risk.
If it is determined in step S161 that the image is dangerous (Y), the process proceeds to step S162. If it is determined that the image is not dangerous (N), the process proceeds to step S112 to perform the process for the next screen.
[0065]
In step S162, a warning process and a display process for notifying a danger are performed. In this step S162, the driver is alerted by sounding an alarm in accordance with the calculated degree of danger or displaying which area is danger on the display. At this time, the type of alarm tone or display color may be changed according to the level of danger.
When the process of step S162 is completed, the process shifts to step S112 to perform a process for the next screen.
[0066]
When the process of step S162 is completed or the process proceeds to step S112 due to the determination of N in step S161, the image of the third frame is captured in step S112. The image is stored in the first frame memory 21. Accordingly, the image stored in the second frame memory 22 is set as the first frame image, and the image stored in the third frame memory 23 is set as the second frame image, and the subsequent processing is performed.
[0067]
Similarly, when the process of step S112 is performed again, the image of the third frame captured in step S112 is stored in the second frame memory 22, and the image stored in the third frame memory 23 at the same time. The image of the first frame and the image stored in the first frame memory 21 are the images of the first frame.
[0068]
Next, an actual operation will be described with reference to FIG. FIG. 7 is a diagram showing a state in which another vehicle 200 traveling in front of the own vehicle is approaching the own vehicle with the passage of time, and FIG. 7) shows the foreground image at time t + Δt, and FIG. 7C shows the foreground image at time t + 2Δt (= Δt + Δt).
The case where the point 201 focused on the right rear portion of the other vehicle 200 is set will be described. In these figures, for convenience of explanation, a case where the imaging interval Δt is relatively long and the amount of movement is large is illustrated. However, since the actual imaging interval Δt is short as described above, The movement of the other vehicle 200 in each of the foreground images at time t, time t + Δt, and time t + 2Δt is minute.
[0069]
In this case, first, each foreground image at time t, time t + Δt, and time t + 2Δt is captured and stored in the first frame memory 21, the second frame memory 22, and the third frame memory 23, respectively (S110 to S112). ). Then, the FOE is set for each foreground image (S113).
[0070]
Next, a point 201 (t) of interest is set with reference to the image at time t, and at the same time, an elongated window W1 is set in the radial direction from the FOE centering on this point 201 (S120).
Then, in the image at time t + Δt, a point 201 on the image at time t + Δt is searched while moving the window W1 in the direction W from the position W1 (t) set in the image at time t. Note that the direction W at the time of this search is a radial direction from the FOE and a direction diverging from the FOE.
When the point 201 (t + Δt) is searched at the position W1 (t + Δt) in the image at the time t + Δt, as shown in FIG. 8, the point 201 (t) and the point 201 (t + Δt) are obtained. The vector connecting to (t) is set as a temporary optical flow FL1 (S130).
[0071]
Next, in the image at the time t + 2 △ t, the point 201 on the image at the time t + 2 △ t is moved while moving the window W1 in the direction from the position W1 (t + △ t) of the window W1 in the image at the time t + △ t. Search for.
When the point 201 (t + 2 @ t) is detected at the position W1 (t + 2 @ t) in the image at the time t + 2 @ t, as shown in FIG. (t + 2 △ t) is set as a temporary optical flow FL2 (S132).
[0072]
Then, the length L1 of the optical flow FL1 is compared with the length L2 of the optical flow FL2, and it is determined whether or not L2 is in a range from half the length of L1 to twice the length of L1 (S140). ).
In this case, since L1 and L2 are substantially equal, it is determined that L2 is within the range of half to twice the length of L1 (Y in S140), and the optical flow FL1 is stored in the divergent optical flow memory 24 (S141). ).
[0073]
Then, the same processing is performed on the other points of interest 202 (t) and 203 (t) (S120). When the processing on all the points of interest is completed (Y in S150), a dangerous The calculation of the degree (S160) and the alarm / display processing (S162) are performed, the image of the next imaging cycle (t + 3 △ t) is captured (S112), and the processing is continued.
[0074]
In the above-described processing, regarding the length L1 of the optical flow FL1 and the length L2 of the optical flow FL2, if L2 is out of the range of half the length of L1 to twice the length of L1. In (N in S140), this optical flow FL1 is not stored in the divergent optical flow memory 24 (S142), and the other points of interest 202 (t) and 203 (t) are processed. (S120).
[0075]
In short, in the first embodiment, a peripheral image (foreground image in the above-described example) is captured at predetermined time intervals from the own vehicle during traveling, and the first frame image of the two frames that are in front of and behind the predetermined time period is taken. A point of interest (point 201) is set for the image (image at time t), a narrow window W1 is set in the radial direction from the FOE to the point of interest (point 201), and the second frame (Image at time t + Δt), the window W1 is set at the same position as the window W1 in the image of the first frame, and the image of the first frame is moved while moving in the direction in which the window W1 diverges. , The same point (point 201) as one point of interest is detected.
[0076]
When one point of interest is detected in the image of the image of the second frame, the same point of interest is detected by performing similar processing on the image of the third frame (image at time t + 2 △ t). A vector connecting one point of interest between the first frame image and the second frame image is set as a temporary optical flow FL1, and the one point of interest between the second frame image and the third frame image is defined as a temporary optical flow FL1. The connected vector is assumed to be a temporary optical flow FL2.
Next, the length L1 of the optical flow FL1 is compared with the length L2 of the optical flow FL2, and if both have a length within a predetermined range (0.5L1 <L2 <2L1), the image of the first frame The optical flow FL1 is adopted assuming that the point of interest in the above and the point of interest in the image of the second frame are the same point.
[0077]
Thus, the optical flow FL1 set between the image of the first frame and the image of the second frame and the optical flow FL2 set between the image of the second frame and the image of the third frame By the comparison, erroneous detection of one point of interest can be easily determined, and erroneous detection can be reduced.
[0078]
Also, paying attention to the property that the optical flow at the same point appears in the direction of a straight line connecting this point and the point at infinity, and comparing the lengths of the first temporary optical flow and the second temporary optical flow described above, Since it is determined whether or not one point to be detected is a point that is erroneously detected, erroneous detection of one point of interest can be more easily determined, and erroneous detection can be reduced.
[0079]
By the way, the first embodiment has an advantage that erroneous detection of one point of interest can be easily determined. However, in the first embodiment, when an object of the same shape such as a fence, a guardrail, or a utility pole periodically appears, an object newly appearing in the imaging range focuses on the third frame image. It is rarely located at the point movement position. In this case, there is a high possibility that the point on the new object is determined to be the same point.
Next, a second embodiment will be described in which erroneous detection is not performed even when such an object having the same shape appears periodically.
[0080]
FIG. 9 is a block diagram showing the configuration of the second embodiment. In FIG. 9, reference numeral 10 denotes an imaging unit, 20 denotes a storage unit, 30 denotes a data processing unit, and 40 denotes a display alarm unit.
Regarding the imaging unit 10, the storage unit 20, the data processing unit 30, and the display alarm unit 40, the storage unit 20 includes a first frame memory 21, a second frame memory 22, and a divergent optical flow memory 24, and a third frame memory 23 is different from the first embodiment in that the data processing unit 30 is not provided and an operation program executed by the CPU 31 of the data processing unit 30 is different. Then, the first frame memory 21 and the second frame memory 22 store the captured images at predetermined intervals Δt alternately.
Other configurations and functions are the same as those described in the first embodiment, and thus description thereof will be omitted.
[0081]
Next, the operation of the second embodiment will be described with reference to the flowchart of FIG.
In the second embodiment, in step S210, a foreground image (background image) at time t is captured and stored in the first frame memory 21 as a first frame image, and in step S211 the foreground image at time t + Δt is obtained. (Background image) is captured and stored in the second frame memory 22 as a second frame image.
Then, in step S212, an FOE is set for each of the images obtained in steps S210 and S211.
[0082]
In the following step S220, one point of interest is selected. The selection of one point of interest is performed in the same manner as in step S120 in the first embodiment, and is set for the image of the first frame.
In step S221, an elongated window is set in the radial direction around the point of interest set in step S220.
In step S222, the sum of the absolute values of the brightness values of the window set in the first frame while moving the window in the diverging direction along the radial direction from the same position as the image of the first frame in the image of the second frame , A search for one point of interest set in step S220 is performed.
[0083]
In step S230, based on the processing in step S222, it is determined whether one point of interest has been detected in the image of the second frame. If one point of interest is detected (present), the process proceeds to step S231, and if one point of interest is not detected (no), the process proceeds to step S250.
In step S231, a vector connecting the point of interest set in step S220 and the point of interest detected in step S222 is set, and this vector is set as an optical flow.
[0084]
In the subsequent step S232, a search is performed for a point of interest while moving the window in the convergence direction along the radial direction from the same position as the image of the first frame, that is, toward the FOE, in the image of the second frame. This search operation is performed in the same manner as in step S222.
[0085]
In step S240, based on the processing in step S232, it is determined whether one point of interest has been detected in the image of the second frame. Then, when one point of interest is detected (present), the process proceeds to step S241, and when one point of interest is not detected (absent), the process proceeds to step S242.
[0086]
In step S241, regarding the point of interest set in the image of the first frame, the point corresponding to the point of interest is detected based on the detection of two points in the image of the second frame, the divergence direction and the convergence direction. The optical point set at the point of interest is ignored since it is highly likely that the point of interest is erroneously detected.
In step S242, regarding the point of interest set in the image of the first frame, based on the fact that the point corresponding to the point of interest is only one point in the divergent direction in the image of the second frame, Assuming that the one point of interest is properly detected, the optical flow set for this one point of interest is stored in the divergent optical flow memory 24.
Then, after the processing of step S241 or S242 is completed, the process proceeds to step S250.
[0087]
In step S250, it is determined whether or not the processing has been performed for all the target points regarding the points to be detected for the optical flow. When the processing of all the points is completed (Y), the processing shifts to step S260. When the unprocessed target points still remain (N), the processing shifts to step S220 to newly focus one point to be noted. Is set and processing is continued.
[0088]
In step S260, the degree of risk is calculated. In step S260, the sum of the lengths of the optical flows existing in each of the divided areas is calculated by the same processing as in step S160.
[0089]
In step S261, it is determined whether it is dangerous. The determination processing in step S261 is performed in the same manner as the processing in step S161, and the sum of the optical flow lengths for each area is calculated based on the risk threshold set by weighting for each area. If the threshold value is exceeded, it is determined that an area exceeding the threshold value is dangerous.
Then, if it is determined in this step S261 that it is dangerous (Y), the process proceeds to step S262, and if it is determined that it is not dangerous (N), the process proceeds to step S212 to perform processing for the next screen.
[0090]
In step S262, a warning process and a display process for notifying a danger are performed. In step S262, as in step S162, a warning is sounded in accordance with the magnitude of the obtained degree of danger, or the driver is warned by displaying which area is dangerous and how much.
Then, when the process of step S262 is completed, the process shifts to step S212 to perform a process for the next screen.
[0091]
Next, an actual operation of the second embodiment will be described with reference to FIGS. 7 (a) and 7 (b).
In this case, first, each foreground image at time t and time t + Δt is captured and stored in the first frame memory 21 and the second frame memory 22, respectively (S210, S211). Then, the FOE is set for each foreground image (S212).
[0092]
Next, a point 201 (t) of interest is set with reference to the image at time t (S220), and an elongated window W1 is set in the radial direction from the FOE centering on this point 201 (S221).
Then, in the image at the time t + Δt, the point 201 on the image at the time t + Δt is searched while moving the window W1 in the direction W from the position W1 (t) set in the image at the time t (S222). ).
[0093]
Based on the search result, it is determined whether or not the point 201 (t) set on the image at time t is detected on the image at time t + Δt (S230). In the case of FIG. 7B, since the point 201 (t + Δt) is detected at the position W1 (t + Δt), it is determined that the point 201 has been detected (Yes in S230).
[0094]
Then, as two points 201 (t) and 201 (t + Δt) are detected, an optical flow FL1 connecting these points is set (S231, see FIG. 11).
Next, in the image at the time t + Δt, the image at the time t + Δt is moved while moving the window W1 from the position W1 (t) set in the image at the time t to the direction −W which is the direction of convergence to the FOE. The upper point 201 is searched (S232).
[0095]
Based on the search result, it is determined whether or not another point 201 (t) set on the image at time t is detected on the image at time t + Δt (S240). In this case, since the corresponding point on the direction -W side is not detected, it is determined that there is no corresponding point (N in S240).
[0096]
As a result, the point 201 (t + Δt) detected in the process of step S222 is determined to be a normally detected point, and the optical flow FL1 set in step S231 is stored in the divergent optical flow memory 24. (S242).
[0097]
Then, the same processing is performed for the other points of interest 202 (t) and 203 (t) (S220). When the processing for all the points of interest is completed (Y in S150), a dangerous The calculation of the degree (S260) and the alarm / display processing (S262) are performed, the image of the next imaging cycle (t + 2 △ t) is captured (S212), and the processing is continued.
[0098]
Further, in the above-described corresponding point detection process on the −W side (S232), if the corresponding point on the −W side is detected (YES in S240), the point set on the image at time t is set. Regarding 201 (t), there are two points corresponding to this point at time t + Δt, so the optical flow FL1 relating to this point 201 is not stored in the divergent optical flow memory 24 (S241). Then, the processing of 202 (t) or 203 (t), which is another point of interest, is performed (S220).
[0099]
In short, in the second embodiment, the peripheral image is taken at predetermined time intervals from the own vehicle during traveling, and the image of the first frame (the image at time t) of the two frames preceding and following the predetermined time interval is taken. One point (point 201) of interest is set from the FOE, an elongated window W1 is set in the radial direction with respect to this point of interest (point 201), and the image of the second frame (the image at time t + Δt) is set. ), The window W1 is set at the same position as the window W1 in the image of the first frame, and the same point as the point of interest in the image of the first frame while moving in the direction in which the window W1 diverges. (Point 201) is detected.
[0100]
If one point of interest is detected, the same point (point 201) as the point of interest in the first frame image is detected while moving the window W1 in the convergence direction. When the same point as the point of interest is detected by the search in the convergence direction, two points corresponding to the point of interest in the image of the first frame exist on the image of the second frame. Therefore, it is determined that there is a high possibility that any of these points is erroneously detected, and the optical flow is not set for this noted point.
[0101]
As described above, regarding one point of interest in the image of the first frame, if this point of interest appears on the image of the second frame, in two directions, that is, the divergence direction and the convergence direction with respect to the point at infinity, the point of interest becomes Since it is determined that they have been erroneously detected, even if objects of the same shape such as fences, guardrails, or telephone poles are arranged at intervals on a straight line, it is possible to eliminate erroneous detection of one point of interest. it can.
[0102]
Further, since the search in the convergence direction described above is performed only when a corresponding point is detected by the search in the divergence direction, unnecessary processing is not performed and the processing can be easily and easily performed.
[0103]
As is clear from the above description, the basic configuration of the present invention and the flowchart in each embodiment have the following correspondence.
That is, the erroneous detection determination unit in the basic configuration of the present invention corresponds to steps S140 to S142 in the flowchart of FIG. 6 and steps S240 to 242 in the flowchart of FIG. This corresponds to step S130 and steps S221 to S222 in the flowchart of FIG.
Further, the second point of interest detecting means corresponds to step S132 in the flowchart of FIG. 10, and the third point of interest detecting means corresponds to step S132 in the flowchart of FIG.
[0104]
【The invention's effect】
According to the vehicle periphery monitoring device of the present invention, the following effects are obtained.
That is, it is determined from the one point of interest in the previous image and the one point of interest detected in the subsequent image whether or not the one point of interest is an erroneously detected point. Since the optical flow is not set for the point determined to be wrong, it is possible to reduce the erroneous detection of one point of interest.
[0105]
Further, a first temporary optical flow set for the previous image and the subsequent image, and a second temporary optical flow set between the subsequent image and the third image captured a predetermined time after the subsequent image. By comparison with the provisional optical flow, erroneous detection of one point of interest can be easily determined, and erroneous detection can be reduced.
[0106]
In addition, since the determination is made by comparing the length of the first temporary optical flow and the length of the second temporary optical flow, the erroneous detection of one point of interest can be more easily determined and the erroneous detection can be performed. Detection can be reduced.
[0107]
Also, regarding one point of interest in the previous image, when this point of interest appears in two directions, that is, the divergence direction and the convergence direction with respect to the point at infinity, it is determined that this point of interest is erroneously detected. It is possible to eliminate erroneous detection of one point of interest.
[0108]
Further, with respect to one point of interest in the previous image, processing is performed in the convergence direction only when this point of interest is detected in the diverging direction with respect to the point at infinity, so that unnecessary processing is not performed. Can be performed easily and easily.
[Brief description of the drawings]
FIG. 1 is a diagram illustrating a basic configuration of the present invention.
FIG. 2 is a block diagram showing the configuration of a first embodiment of a vehicle periphery monitoring device to which the present invention is applied.
FIG. 3 is a diagram illustrating a change in a foreground image obtained by a video camera 11;
FIG. 4 is a diagram showing an optical arrangement for explaining how to detect an obstacle or the like;
FIG. 5 is a schematic diagram illustrating a procedure for obtaining an optical flow.
FIG. 6 is a flowchart illustrating the operation of the first embodiment.
FIG. 7 is a diagram showing a state in which another vehicle 200 traveling ahead of the host vehicle approaches the host vehicle with the passage of time.
FIG. 8 is a diagram showing a detected point of interest and an optical flow.
FIG. 9 is a block diagram showing a configuration of a second embodiment of the vehicle periphery monitoring device to which the present invention is applied.
FIG. 10 is a flowchart illustrating the operation of the second embodiment.
FIG. 11 is a diagram showing a detected point of interest and an optical flow.
FIG. 12 is a diagram illustrating a problem in a conventional device.
[Explanation of symbols]
10 Imaging unit
11 Video camera
20 storage unit
21 First frame memory
22 Second frame memory
23 Third frame memory
24 Divergent Optical Flow Memory
30 Data processing unit
31 CPU
32 ROM
33 RAM
40 Display / alarm unit
41 Display
42 speakers
100 own vehicle
200 other vehicles
FL1, FL2 optical flow
W1 window
FOE infinity point

Claims (5)

A surrounding image is captured from the own vehicle while traveling, and the movement of the same point in the two frames preceding and following the predetermined time is moved in the previous image of the two frames preceding and following the predetermined time. From the point at infinity corresponding to one point indicating the traveling direction of the own vehicle or the direction opposite to the traveling direction, an elongated window is set in a radial direction with respect to one point of interest, and the window is set in the later image. The one point of interest is detected while moving in the radial direction, a vector connecting the one point of interest in the previous image and the one point of interest in the subsequent image is set as an optical flow, and Other vehicles traveling around the vehicle or obstacles existing in the vicinity are compared with other vehicles traveling around the vehicle or the magnitude and the appearance position of the optical flow vector of the obstacle existing around the vehicle. In the vehicle periphery monitoring apparatus for monitoring by anything,
Erroneous detection determining means for determining whether the one point of interest is erroneously detected,
An apparatus for monitoring the periphery of a vehicle, wherein the optical flow is not set for one point of interest determined to be erroneously detected by the erroneous detection determination means. A first point-of-interest detection means for detecting the corresponding point of the one point of interest while moving the window in a diverging direction from the infinity point from the set position in the previous image in the subsequent image,
When the corresponding point is detected by the first point-of-interest detection means, a divergence direction from the infinity point from the detection position in the subsequent image in the third image captured after the predetermined time from the subsequent image And a second point of interest detecting means for detecting the corresponding point of the one point of interest while moving to
The erroneous detection determination means sets a vector connecting a point of interest in the previous image and a point of interest in the subsequent image as a first provisional optical flow, and sets a vector of interest in the subsequent image. A vector connecting one point of interest in the third image is set as a second temporary optical flow, and the one point of interest is erroneously determined by comparing the first temporary optical flow and the second temporary optical flow. 2. The vehicle periphery monitoring device according to claim 1, wherein it is determined whether the detected point is a detected point. The erroneous detection determination means determines whether the one point of interest is a point erroneously detected by comparing the lengths of the first temporary optical flow and the second temporary optical flow. The vehicle periphery monitoring device according to claim 2, wherein A first point-of-interest detection means for detecting the correspondence of the one point of interest while moving the window in a diverging direction from the infinity point from the set position in the previous image in the subsequent image,
A third point of interest detecting means for detecting the corresponding point of the one point of interest while moving the window in the direction of convergence from the set position in the previous image to the point at infinity in the after image. Become
The erroneous detection determination means is configured to erroneously detect the one point of interest when the corresponding point of the one point of interest is detected from both the first point of interest detection means and the third point of interest detection means. The vehicle surroundings monitoring device according to claim 1, wherein it is determined that the vehicle has been hit. 5. The vehicle periphery according to claim 4, wherein the third point-of-interest detection means detects the detected one point when the first point-of-interest detection means detects one point of interest. Monitoring device.

Images