JPH06325287A - Device for detecting traffic flow abnormality - Google Patents

Abstract

PURPOSE:To make possible detecting abnormality before an accident happens by detecting the abnormal movement of a running vehicle. CONSTITUTION:The image of the vehicle 2 which passes by on a road 1 is detected by an image pickup camera 3 so as to be stored in an image memory 4. The vehicles of the respective image are segmented from consecutive image frames by a vehicle segmenting part 5 and the correspondance of the same vehicle is executed among the consecutive image frames in a vehicle corresponding part 6. A mobile vector is calculated in a mobile vector from a correlation value obtained by the correspondance by a mobile vector calculation part 7. Angle theta made by the calculated mobile vector with a road advancing direction vector which is previously obtained is calculated, the angle is compared with a previously decided threshold value so that presence or absence of abnormality is judged by an abnormality presence or absence judging part 8.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an apparatus for detecting whether or not a traffic flow is abnormal by image processing.

[0002]

2. Description of the Related Art The main purpose of a traffic flow abnormality detecting device is to detect a situation where an lane that should be able to pass through is blocked and cannot be passed due to an accident, breakdown, falling object, etc. To inform. The conventional traffic flow abnormality detection device (1) calculates the speed of the vehicle extracted by taking the difference between consecutive image frames, and detects traffic congestion from the speed, (2) tens of image frames There is proposed a method of integrating and deleting moving parts to detect obstacles on the road (accident vehicle, broken vehicle, falling object, etc.).

[0003]

However, since the device of the above (1) is configured to detect the traffic jam that occurs as a result of the lane being blocked by an obstacle, the traffic jam is detected after the actual failure occurs. There is a problem that there is a large time delay until it is done. Further, the device (2) also has a problem that it takes a long processing time because it is necessary to integrate several tens of image frames in order to detect an obstacle.

The present invention has been made in view of such a conventional problem, and detects abnormal behavior of a traveling vehicle by image processing to detect abnormality before an accident or the like occurs. It is an object of the present invention to provide a traffic flow abnormality detection device that enables the detection.

[0005]

SUMMARY OF THE INVENTION The present invention is directed to image detection means for detecting an image of a vehicle passing on a road, and movement of the vehicle within the image frame from consecutive image frames detected by the image detection means. Vehicle movement vector calculation means for calculating a vector, and abnormality presence / absence determination means for determining presence / absence of traffic flow abnormality based on an angle formed by a traveling direction vector at the vehicle position on the road and a movement vector obtained in advance , Is provided.

Further, the vehicle movement vector calculation means is
Calculate the circumscribed rectangle of the vehicle part by taking the difference between consecutive image frames, and then apply this circumscribed rectangle to the image frame to cut out the vehicle, and between the consecutive image frames with reference to the car cut out. And a movement vector calculation unit that calculates a movement vector of the vehicle based on the correspondence result of the vehicle in the vehicle correspondence unit.

The size of the vehicle cut out and extracted by the vehicle cutout unit, the certainty of the vehicle correspondence result obtained by the vehicle correspondence unit, and the movement vector of the vehicle calculated by the movement vector calculation unit Reliability calculation means for calculating the probability of image processing based on one or more pieces of predetermined information at the image processing stage including size, calculation result of the reliability calculation means, and the presence / absence determination It is characterized by further comprising a comprehensive determination means for comprehensively determining the presence or absence of an abnormality in the traffic flow from the determination result of the means.

Further, the reliability calculating means stores in advance fuzzy measure calculating means for calculating each fuzzy measure using the predetermined information as a feature quantity, and fuzzy measure value integration weights for each combination of feature quantities in advance. And a fuzzy integrating means for performing an integral operation of the fuzzy measure calculated by the fuzzy measure calculating means using the weights of the weight table.

Further, the comprehensive judging means is a tentative judging means for judging the presence or absence of a traffic flow abnormality on an image frame basis from the calculation result of the reliability calculating means and the judgment result of the abnormality judging means, and the tentative judging means. The provisional determination result storage means for storing the result of the determination in a plurality of frames, and the final determination means for finally determining the presence or absence of traffic flow abnormality from the determination results for the plurality of frames stored in the provisional determination result storage means. And are provided.

Further, the abnormality presence / absence determining means sets a plurality of reference lines drawn along the traveling direction of the road, such as white lines on the road in the image, on the XY coordinates, and then determines each in the Z direction. It is formed by moving each reference line in the Z direction so that the reference line interval has a predetermined length.
A means for calculating a three-dimensional curved surface z = f (x, y), and the traveling direction vector at a point (p, q) in the XY coordinates where the vehicle is located, as a curved surface z = f (x, y). Plane z =
and means for calculating the inclination of the intersection line of f (p, q).

[0011]

The image including the vehicle and the road passing through the road detected by the image pickup camera is input to the vehicle movement vector calculation means, and the movement vector of the vehicle is calculated here. The movement vector is calculated, for example, as follows. That is, the circumscribed rectangle including the vehicle portion is calculated by taking the difference between consecutive image frames, and the circumscribed rectangle is applied to the image frame to cut out the vehicle first. Then, by referring to the cut out vehicle, the same vehicle is made to correspond between consecutive image frames, and the movement vector of the vehicle is calculated based on the correspondence result of this vehicle. The same vehicle can be associated with each other by calculating a mutual correlation value. The movement vector of the vehicle calculated in this way,
The traveling direction vector at the vehicle position on the road, which is obtained in advance, is compared, and whether or not there is an abnormality in the traffic flow is determined based on the angle formed by these vectors. That is, if the angle between the vectors is equal to or greater than a certain value, it is determined that the diagonal line is blocked due to an obstacle on the road.

Further, in the above series of image processing steps, a reliability calculation means for calculating the certainty of the processing is provided, and the traffic flow is calculated based on the obtained abnormality presence / absence determination result and the calculated reliability result. Comprehensively determine whether there is an abnormality.
In this case, the certainty of the image processing includes the size of the vehicle extracted and extracted, the certainty of the correspondence result of the vehicle (that is, the correlation value), and the calculated magnitude of the movement vector of the vehicle. It is calculated based on one or a plurality of information from the predetermined information at the stage. When the certainty of the image processing calculated based on these pieces of information shows a low value, it is not determined that the traffic flow is in the abnormal state in the comprehensive determination.

Further, the reliability calculation means can be configured to calculate the reliability by a fuzzy measure. That is, each fuzzy measure using the predetermined information in the image processing stage as a feature amount is calculated, and further, the fuzzy velocity integral calculation is performed using a predetermined weight.

The presence / absence of traffic flow abnormality determined is stored as a tentative result for each image frame, and the presence / absence of traffic flow abnormality is finally determined from the tentative results of the plurality of image frames.

Further, when the road is curved, the traveling direction vector of the road is calculated. This calculation creates a three-dimensional curved road surface on the XYZ coordinates, and moves the XY coordinate plane parallel to the Z direction so that the XY coordinate point where the vehicle is located intersects this curved surface. With this method, the traveling direction vector at any point on the road can be easily calculated.

[0016]

1 is a block diagram of a traffic flow abnormality detecting apparatus according to an embodiment of the present invention.

The output of the image pickup camera 3 in which a predetermined range of the road 1 is set as the image pickup range is input to the image memory 4, and the image frame data stored in the image memory 4 is input to the vehicle cutout unit 5. To be done. In the figure, it is assumed that the vehicle 2 is moving in the upper direction of the figure.

2 and 3 are views for explaining a vehicle cutting method of the vehicle cutting section 5. Now, the imaging camera 3 captures continuous images as shown in FIGS. 2A and 2B, and the respective image frames are stored in the image memory 4.
Suppose it was stored in. When the vehicle cutout unit 5 obtains these two image frames, first, the difference between them is calculated. By taking the difference, the part that is not moving, that is, the road or the background part is removed. Subsequently, after the difference result is projected in the horizontal direction and only a portion equal to or larger than a predetermined threshold value (shown by a dotted line) is cut out, this time it is projected in the vertical direction to perform the same processing (see FIG. 3). As a result, rectangles 5a and 5b circumscribing the vehicle portion are calculated.

The circumscribed rectangles 5a and 5b are shown in FIG.
The vehicle is cut out by applying it to the input image frame shown in (A) (see FIG. 4), and the vehicle is cut out by applying it to the input image shown in FIG. The cut out vehicle is input to the vehicle association unit 6.

The vehicle associating unit 6 performs a normalized cross-correlation operation between the cut-out images using the image of the circumscribed rectangle of FIG. 4 including the vehicle portion of FIG. 2B corresponds to the vehicle portion of FIG. 2A. After associating the vehicles, the movement vector calculation unit 7 calculates the movement vector of the vehicle based on the displacement of the corresponding vehicles in FIGS. 2A and 2B (FIG. 5), and the abnormality presence / absence determination unit 8 The cos θ of the angle θ formed by the calculated vehicle movement vector and the previously obtained traveling direction vector of the road is calculated, and if the value is smaller than a predetermined threshold value, it is determined to be abnormal.

Since the abnormality detection apparatus described above can determine whether or not there is an abnormality in the traffic flow simply by capturing two image frames and performing image processing, the processing time can be short, and if an abnormal state is determined. In case, it is possible to promptly notify the following vehicle.

FIG. 6 shows a block diagram of another embodiment of the present invention. The difference from the configuration shown in FIG. 1 is that a reliability calculation unit 9 and a comprehensive determination unit 10 are provided. Reliability calculation unit 9
Is an input of the output of the vehicle cutout unit 5, the output of the vehicle association unit 6, and the output of the movement vector calculation unit 7, and the information (a) representing the size of the vehicle that is cut out and extracted, and the correspondence result of the vehicle Probability of image processing based on three pieces of information (b) indicating the certainty (that is, the correlation value) and information (c) indicating the magnitude of the movement vector of the vehicle,
That is, the reliability of the image processing is calculated, and this result is guided to the comprehensive determination unit 10 together with the determination result of the abnormality presence / absence determination unit 8. The comprehensive determination unit 10 refers to not only the determination result from the abnormality presence / absence determination unit 8 but also the result output from the reliability calculation unit 9 to perform comprehensive abnormality presence / absence determination.

In the reliability calculating section 9, if any one of the above three pieces of information is below a predetermined threshold value, "N"
If "G" is all equal to or greater than the threshold value, "OK" is output to the comprehensive determination unit 10. In the comprehensive determination unit 10, the final determination result is set to "abnormal" only when the determination result of the abnormality presence / absence determination unit 8 is "abnormal" and the result of the reliability calculation unit 9 is "OK", and otherwise "normal". Output as. By adding the reliability calculation unit 9 in this way, it is possible to prevent erroneous determination as “abnormal” due to an error in image processing such as extraction of noise other than the vehicle.
In this embodiment, all three pieces of information (a), (b), and (c) above are viewed to calculate the reliability, but only one piece of the information may be used. .

The reliability calculating section 9 can obtain the reliability result by calculating each fuzzy measure using the input information as a feature amount. FIG. 7 shows a configuration diagram in this case. In order to obtain the reliability by using the fuzzy measure, the reliability calculating unit 9 includes a fuzzy speed calculating unit 9a for calculating each fuzzy measure having the above-mentioned respective three pieces of information as feature amounts, and a fuzzy for each combination of feature amounts. It is composed of a weight table 9b that stores weights for measure value integration, and a fuzzy integrator 9c that performs integral calculation of the fuzzy speed calculated by the fuzzy measure calculating means using the weights obtained from the weight table. Here, for the fuzzy integral, the Choquet integral (Introduction to the fuzzy measure theory “1”: Kanno, Murofushi: Journal of Japan Society for Fuzzy Studies 90/5 V
ol. 2 No. 2 174-181 and the like. ) Is used. To perform this Choquet integration,
First, the fuzzy measure value (membership value) is obtained.

FIG. 8 is a diagram for explaining a method for obtaining the evaluation value V by Choquet integration.

First, as input information, a feature value ω, a membership value f (ω) and μ are prepared. The feature amount and the membership value are sorted by the sorting means in descending order of membership value. After that, integration is performed by the equation (1) shown in FIG. Here, μ represents the degree to which the reliability depends on the feature values when they have a certain value at the same time, and the closer the value is to 1, the closer the value is to 1. It shows that the reliability is high. The value of μ is given in advance in a table.
In this embodiment, the characteristic amount is obtained by the area (size of the vehicle extracted and extracted), the speed (size of the movement vector calculated by the movement vector calculation unit 7), and the correlation value (vehicle association unit 6). 9) and the weight μ for the combination of the feature amounts is shown in FIG.
Is set as follows. Further, FIG. 10 shows the membership function of each feature amount. The fuzzy measure calculation unit 9a uses this membership function to calculate a membership value (fuzzy measure).

Now, for example, three images P1, P2, P
Each feature quantity (area, speed, correlation value) of 3 is P1
(500,110,0.85), P2 (6500,1)
0, 0.9) and P3 (180, 150, 0.7), the fuzzy measures (Af, Bf, C
f) is P1 (1.0, 0.9, 0.7), P2 (0.
7, 0.33, 0.8), P3 (0.2, 0.5, 0.
4).

In the fuzzy integrator 9c, the integration shown in FIG. 8 is performed based on the fuzzy measures (Af, Bf, Cf) obtained as described above by the fuzzy measure calculator 9a and the weight μ obtained from the weight table 9b. And the evaluation value V is calculated. When the evaluation value V for each of the images P1, P2, P3 is calculated by the same equation, the result is as follows.

V (P1) = (Af−Bf) × μ ({Af}) + (Bf−Cf) × μ ({Af, Bf}) + Cf × μ ({Af, Bf, Cf}) = ( 1.0-0.9) * 0.6 + (0.9-0.7) * 0.5 + 0.7 * 1.0 = 0.86 Similarly, about V (P2) and V (P3) Respectively C
Since f>Af> Bf and Bf>Cf> Af, V (P2) = (Cf−Af) × μ ({Cf}) + (Af−Bf) × μ ({Af, Cf}) + Bf × μ ({Af, Bf, Cf}) = (0.8−0.7) × 0.5 + (0.7−0.33) × 0.6 + 0.33 × 1.0 = 0.602 V (P3) = (Bf−Cf) × μ ({Bf}) + (Cf−Af) × μ ({Bf, Cf}) + Af × μ ({Af, Bf, Cf}) = (0.5 −0.4) × 0.4 + (0.4−0.2) × 0.4 + 0.2 × 1.0 = 0.32 Whether the image processing is correct for the above evaluation value (integrated value) Reliability information. For example, using two thresholds,
When the reliability result is 0.8 or more, it is OK, when it is less than 0.4, it is NG, and when it is in the middle, it is GRAY.
1 is OK, P2 is GRAY, and P3 is NG.

As described above, a more reliable evaluation value can be obtained by using the fuzzy measure integration based on each feature quantity in the reliability calculating section 9.

FIG. 11 shows the construction of a comprehensive judging section according to another embodiment of the present invention. In this embodiment, the comprehensive determination unit 10 of FIG. 6 is a provisional determination unit 10a that determines the presence / absence of traffic flow abnormality in image frame units from the determination result of the abnormality determination unit 8 and the calculation result of the reliability calculation unit 9. , This provisional judgment unit 1
A temporary determination result storage unit 10b that stores a plurality of frames of the determination result of 0a, and a final determination unit 10c that finally determines the presence or absence of traffic flow abnormality from the stored multiple frames of the temporary determination results. I have it. In this apparatus, even if the provisional determination unit 10a determines once as "abnormal", the final determination is not immediately made, and the provisional determination result is stored in the provisional determination result storage unit 10b for a plurality of image frames and then based on the storage result. And make a final decision. Therefore, the final determination unit 10c outputs "abnormal" when the number of consecutive abnormal determination results is a threshold value or more.
Thus, by using the provisional determination results for a plurality of frames, it is possible to make a determination with higher reliability. Even if the provisional determination result of about 10 frames is used, the delay of the processing time due to that is not a serious problem, and there is no practical problem.

In the above embodiments, the road is straight and the traveling direction vector is given in advance. However, when the road is curved, the traveling direction vector changes depending on the position of the vehicle. . in this case,
A reference line drawn along the traveling direction of the road, for example, a white line is used to determine the traveling direction. The first conceivable way to calculate the direction of travel is to calculate the direction in which the ratio of the distance from the white line is constant and use that direction as the direction of travel, but in that method, calculate the correct direction of travel. Is difficult. Therefore, the white lines on both sides of the road are placed at different Z coordinates, the curved surface corresponding to the road virtually placed in the three-dimensional space is obtained by spline interpolation, and the traveling direction vector is calculated using the curved surface. FIG. 12A is a road image, and FIG. 12B is a diagram illustrating a method of calculating a traveling direction vector on the road. In addition, FIG.
FIG. 4 is a configuration diagram of a portion for calculating a traveling direction vector. In FIG. 13, the traveling direction vector calculation unit creates a part 11 for calculating the current vehicle position (p, q) and a three-dimensional curved surface z = f (x, y) based on the output of the vehicle cutout unit 5. The three-dimensional curved surface creating unit 12 and the portion 13 that calculates the traveling direction vector from these pieces of information.

FIG. 12 shows a road image without a vehicle.
When prepared as shown in (A), the coordinates of the white lines of the white lines 1a, 1b, 1c that delimit the lanes of the road are obtained and input to the cubic curved surface creation unit 12. In this creation unit 12,
The horizontal axis of the image in FIG. 12A is the X axis, and the vertical axis is the Y axis.
It is converted (mapped) to the Y coordinate plane (z = 0), and the white line on the XY coordinate plane is translated in parallel with the Z axis to create a curved surface.
That is, each white line is replaced by an X line as shown in FIG.
In YZ three-dimensional space, the leftmost white line is z = k, the center is z
= 2k, the rightmost white line moves in parallel on the respective planes of z = 3k, and the surface z = f (x, y) stretched by three curves in the obtained XYZ space is spline-interpolated. Ask by. On the other hand, based on the information from the vehicle cutout unit 5, the vehicle position calculation unit 11 obtains the position (p, q) of the cutout vehicle on the XY two-dimensional coordinates as soon as the vehicle appears in the image. The position (p, q) is preferably the position where the center of gravity in the image of the vehicle is obtained,
In order to simplify the calculation, the center of the circumscribed rectangle (intersection of diagonal lines) in the input image (1) of FIG. 2A is used here. Then, in the traveling direction vector calculation unit 13, the traveling direction vector at (p, q) of this two-dimensional coordinate is calculated from the curved surface z = f (x, y) created in advance by the three-dimensional curved surface creation unit 12. To do. Specifically, the traveling direction vector at the point (p, q) is XYZ.
In space, curved surface z = f (x, y) and plane z = f
It is obtained as the direction of the tangent at the point (p, q, f (p, q)) of the line of intersection with (p, q). In this way, by virtually replacing a two-dimensional road with a three-dimensional space,
By simply obtaining the coordinate data of the white line on the road, it becomes possible to easily and accurately obtain the traveling direction vector of an arbitrary point on the road in the image.

[0034]

According to the present invention, a movement vector of a vehicle can be calculated from consecutive image frames, and an abnormality can be detected when the movement direction of the vehicle greatly differs from the traveling direction of a road. That is, rather than detecting from a phenomenon after a traffic accident or the like occurs, early detection is possible because the phenomenon immediately before the accident and the situation that is about to occur are detected. Early detection of traffic flow abnormalities has a great effect on preventing secondary collisions by warning subsequent vehicles and speeding up emergency activities.

Further, the reliability of the image processing is calculated by using the size of the extracted vehicle portion, the certainty when the correspondence of the vehicles between the images is obtained, the size of the movement vector, and the like. It reduces erroneous determinations due to image processing mistakes and enables highly accurate determinations. Then, by calculating this reliability by fuzzy measure integration, it is possible to further reduce erroneous determination.

Further, the abnormality is not output by only one judgment, and the temporary judgment result is stored several times in succession, and the comprehensive judgment is made only when the plural temporary judgment results are judged to be abnormal. Since it is abnormal, the accuracy of the determination can be further increased.

Since the road is virtually curved in a three-dimensional space and the traveling direction vector is calculated using the curved surface, the traveling direction vector at any point on the road can be calculated correctly. Become.

[Brief description of drawings]

FIG. 1 is a diagram showing a block diagram of a traffic flow abnormality detection device according to an embodiment of the present invention.

2A and 2B are diagrams showing continuous input image frames.

FIG. 3 is a diagram illustrating a difference image and a vehicle cutout method.

FIG. 4 is a diagram showing a cut-out vehicle image.

FIG. 5 is a diagram showing a method of associating vehicles and calculating a movement vector.

FIG. 6 is a diagram showing the configuration of another embodiment of the present invention.

FIG. 7 is a diagram showing the configuration of a reliability calculation unit in yet another embodiment of the present invention.

FIG. 8 is a diagram for explaining the Choquet integration method used for fuzzy measure integration.

FIG. 9 is a diagram showing an example of a weight table.

FIG. 10 is a diagram showing a membership function of each feature amount.

FIG. 11 is a diagram showing a configuration of a comprehensive determination unit according to still another embodiment of the present invention.

12A and 12B are diagrams for explaining a road image and a method of calculating a traveling direction vector on the road, respectively.

FIG. 13 is a diagram showing a configuration of a portion for calculating a traveling direction vector according to still another embodiment of the present invention.

Claims (6)

[Claims] 1. An image detection means for detecting an image of a vehicle passing on a road, and a vehicle movement vector calculation for calculating a movement vector of a vehicle in the image frame from consecutive image frames detected by the image detection means. Traffic flow abnormality comprising means and abnormality presence / absence determining means for determining whether or not there is an abnormality in the traffic flow based on an angle formed between the traveling direction vector at the vehicle position on the road and the movement vector obtained in advance. Detection device. 2. The vehicle movement vector calculation means calculates a circumscribing rectangle of a vehicle portion by taking a difference between consecutive image frames, and further applies the circumscribing rectangle to the image frame to cut out the vehicle. , A vehicle associating unit that refers to the cut-out vehicle and associates the same vehicle between consecutive image frames, and a movement that calculates a vehicle movement vector based on the vehicle association result by the vehicle associating unit The traffic flow abnormality detection device according to claim 1, further comprising a vector calculation unit. 3. The size of the vehicle cut out and extracted by the vehicle cutout unit, the certainty of the vehicle correspondence result obtained by the vehicle association unit, and the movement vector of the vehicle calculated by the movement vector calculation unit. Reliability calculation means for calculating the reliability of image processing based on one or more of predetermined information in the image processing stage including size, calculation result of the reliability calculation means and the presence / absence determination The traffic flow abnormality detection device according to claim 2, further comprising: comprehensive determination means for comprehensively determining whether or not there is an abnormality in the traffic flow based on the determination result of the means. 4. The reliability calculation means stores in advance fuzzy measure calculation means for calculating each fuzzy measure using the predetermined information as a feature quantity, and fuzzy measure value integration weights for each combination of feature quantities in advance. 4. The traffic flow abnormality detection device according to claim 3, further comprising: a weight table for performing the fuzzy measure calculation using the weight of the weight table, and fuzzy integration means for performing an integration operation of the fuzzy measure calculated by the fuzzy measure calculation means. 5. The provisional determination means for determining whether or not there is an abnormality in a traffic flow in image frame units based on the calculation result of the reliability calculation means and the determination result of the abnormality presence / absence determination means, and the provisional determination means. And a final determination means for finally determining whether or not there is an abnormality in the traffic flow from the determination results for a plurality of frames stored in the temporary determination result storage means. The traffic flow abnormality detection device according to claim 3, further comprising: 6. The abnormality presence / absence determining means determines a plurality of reference lines, such as a white line on the road in the image, drawn along the traveling direction of the road from the state of being respectively placed on the XY coordinates, in each of the Z directions. Z each reference line so that the reference line interval is a predetermined length.
Means for calculating a three-dimensional curved surface z = f (x, y) formed by moving in the direction, and the traveling direction vector at a point (p, q) in the XY coordinates where the vehicle is located, The traffic flow abnormality detection device according to claim 1, further comprising: means for calculating a slope of an intersection of a curved surface z = f (x, y) and a plane z = f (p, q).