JP2864620B2 - Moving object detection device - Google Patents

Description

Description: BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a moving object detecting apparatus, and more particularly to a moving object detecting apparatus using an optical flow obtained from two temporally different images taken by a television camera or the like. The present invention relates to an apparatus for detecting a moving object.

[Prior Art] Conventionally, as a method of detecting a moving object from two temporally different images, a method based on a difference between the two images has been used. FIG. 7 is an explanatory diagram showing the method shown in, for example, page 375 of the Image Processing Handbook (1987, published by Shokodo). In the figure, (30) and (31) are two input images different in time, and (72) is an output image obtained by subtracting image 2 (31) from image 1 (30). In this method 2
If a moving object is present in the images (30) and (31), the obtained difference image (72) has an area where the density level is positive (A).
Then, an area (a) having a density level of 0 and an area (c) having a negative density level are obtained. The region (a) where the density level is 0 can be considered as the background portion.

8 and 9 are explanatory diagrams of a conventional method using a difference image. In the figure, (20) and (25) show the one-dimensional signal of the image 1 (30) shown in FIG. 7 along the line VIIIa-VIIIa, and (21) and (26) show the image 1 (30). VI of image 2 (31)
The image on the line IIb-VIIIb is represented by a one-dimensional signal. In each signal, the horizontal direction indicates the position of the image, and the vertical direction indicates the density level. In order to detect a moving object by the conventional difference method, two threshold values T1 and T2 are set in an image (82) obtained by subtracting two images (20) and (21), and a density change area (83 ) To extract. The moving direction is determined by the positive and negative positions of the area (83). That is, assuming that the direction in which the density change region changes from positive to negative is the movement direction, the arrow D is the movement direction in FIG.

However, since the density differs depending on the image, two thresholds
Since T1 and T2 are not constant, it is necessary to determine an optimum threshold value for each image to be processed, and it is practically difficult to handle various images. Further, there remains a problem that it is not easy to determine an optimum threshold even when the density difference between the object and the background is small. Further, if the density difference between the background and the object is reversed as in the two images (25) and (26) shown in FIG. 9, the density change area (97) extracted from the difference image (97) 98) is as shown in the figure. Therefore, if the moving direction is determined based on the positive or negative of the change area, the moving direction is determined as indicated by the arrow o in the moving direction.

Also, as another conventional moving object detection device, a magazine ｛Be
rthold KP Horn and Brian G. Schunck, `` Determining
Optical Flow '', Artificial Intelligence, Vol. 17, No.
1-3 (1981), pp. 185-203. As a technique for obtaining a motion vector of an object from two images that are different in time, a moving direction of the object is calculated by an optical flow calculated using a density gradient in a spatial direction and a density gradient in a time direction. Has been proposed. In this method, when the density of a certain coordinate (x, y) in an image at time t is represented by E (x, y, t), assuming that the density of an object is invariant with time, the following approximation is performed. Equation (1)
Holds.

E _x · u + E _y · v + E _t = 0 (1) E _x , E _y : density gradient in space direction (x direction, y direction) E _t : density gradient in time direction (t direction) u, v: Velocity component in x and y directions To equation (1), the assumption that the local velocity changes smoothly is added, and the velocity component of the optical flow obtained by u and v that minimizes the error function (equation (2)) It is assumed that

∫∫ (e _b ² + e _c ² ) dxdy (2) where e _b = E _x xu + E _y・ v + E _t e _c ² = (∂u / ∂x) ² + (∂u / ∂y ) ² + (∂v / ∂x) ² + (∂v / ∂y) ² FIGS. 10 and 11 are explanatory diagrams showing this method. now,
Consider a case where the object at the position of the broken line (110) in FIG. 10 moves to the position of the entity (111). Figure 11 shows the 10th
The density of the image on the line ΧI-ΧI in the figure is represented by a one-dimensional signal, and a signal (100) indicates a signal before movement and a signal (101) indicates a signal after movement. The optical flow obtained from such an image is a vector sum of a spatial density gradient (23) and a temporal density gradient (24) in a portion having a density difference from the background (edge portion of the object) (22 ). Therefore, if the optical flow is checked for the entire image, the moving direction (11
2) can be detected.

In the method based on the optical flow, a relatively large object in a screen is shared between different screens, such as a case where an object at a position indicated by a broken line (110) in FIG. 10 moves to a position indicated by a solid line (111). This is a method used to detect the entire movement direction (112) when performing a parallel movement or a rotational movement while holding an area. Therefore, it is impossible to detect the movement of a minute object composed of, for example, about several pixels, and a processing device for the movement of a minute object of about several pixels in an image has not been established.

[Problems to be Solved by the Invention] The conventional moving object detection device is configured as described above. In the method using the difference image, the two threshold values T1 and T2 are not constant because the density differs depending on the processing image. In addition, it is necessary to determine an optimum threshold value for each image to be processed, and it is practically difficult to handle various images. Further, there remains a problem that it is not easy to determine an optimum threshold even when the density difference between the object and the background is small. Further, when the density difference between the background and the object is reversed, there is a problem that an erroneous determination is made. The method based on optical flow is a method used when detecting a whole moving direction when a relatively large object in a screen is performing a parallel movement or a rotational movement while having a common area between different screens. In addition, there is a problem that a processing device for movement of a minute object of about several pixels in an image has not been established.

The present invention has been made in order to solve the above-described problems, and there is no need to set two thresholds T1 and T2 for each image, and the moving direction is correctly set even when the density difference is reversed. It is an object of the present invention to obtain a moving object detection device capable of making a determination and detecting a movement of a minute object of about several pixels in an image.

[Means for Solving the Problems] A moving object detecting apparatus according to the present invention is an optical flow calculating means for obtaining an optical flow having a size and a direction indicating a velocity distribution on two images which are temporally different from each other. Normalizing means for normalizing the size of the optical flow obtained by the optical flow calculating means, and outputting the direction of the optical flow having a size equal to or larger than a preset value; A flow pattern extracting means for extracting an optical flow pattern in the vicinity of the pixel for each image constituting the image from the direction; a shrinkage in which the flow direction is concentrated from the vicinity based on the flow pattern obtained by the flow pattern extracting means Judgment of diffusion point where the direction of the point and the flow has spread to the vicinity And moving object detecting means for detecting moving information of the object from the characteristic points extracted by the characteristic point extracting means.

[Function] In the feature point extracting means according to the present invention, from a flow pattern in the vicinity of an image of interest, a contraction point where a flow representing before movement is inward, and a flow representing after movement is outward. The diffusion point is extracted, and the moving object detection means detects the moving direction by determining the pair of contraction points and the diffusion point as the positions of the object before and after the movement.

Embodiment An embodiment of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram showing a moving object detection device according to one embodiment of the present invention. (10) and (11) are two input images that are temporally different, (12) is an arithmetic unit that calculates an optical flow having a size and direction indicating a velocity distribution from the images (10) and (11), and (13) ) Normalizing means for normalizing the magnitude of the optical flow obtained by the optical flow calculating means (12) and outputting the direction of the optical flow having a magnitude equal to or larger than a preset value;
4) is a flow pattern extracting means for extracting a flow pattern in the vicinity of 8 of the pixel of interest, (15) is a feature point extracting means for extracting feature points such as a contraction point and a diffusion point from the extracted flow pattern, and (16) ) Is a moving object detecting means for detecting the moving information of the object from the extracted feature points.

Next, the operation will be described. This embodiment utilizes the property of optical flow schematically represented by a one-dimensional signal as shown in FIG. FIG. 2 (a),
In (b), (20) and (25) are signals indicating the density level of the image 1 in the horizontal direction at a certain vertical position, for example, (2)
1) and (26) are signals indicating the horizontal density level of image 2 at the same vertical position as image 1 (20), solid arrow (22) is optical flow, dotted arrow (23) is spatial density gradient, The dash-dotted arrow (24) indicates the concentration gradient over time. That is, as shown in FIG. 2A, the optical flow near the background portion of the image 1 and the point (f) having the density difference is such a point that the point disappears in the image 2 (21) at the next time. Therefore, the flow becomes inward (22). In the image 2 (21), the optical flow in the vicinity of the point (g) having the density difference from the background portion is the point where the point newly appears. ). FIG. 2 (b)
As shown in (1), the flow direction does not change even in the case of the image 1 (25) and the image 2 (26) in which the density difference between the background and the object is reversed. This property is the third in 2D images.
It is obtained as an optical flow as shown in the figure. That is, temporally different images 1 (30) and 2 (31)
An optical flow (32) is obtained from these two images. The present invention uses such an optical flow property to set a point where a flow in the vicinity of a certain pixel of interest is inward (a point like FIG. A point whose direction is outward (a point as shown in FIG. 3 (g)) is defined as a diffusion point, a method for determining those characteristic points is proposed, and a moving object is detected.

The output from the optical flow calculation means (12) includes a subtle movement of the background due to camera shake and the like, but the flow is relatively small as compared with the flow obtained from the true moving object by the normalization means (13). Is regarded as the background and there is no direction data. If the flow is larger than a certain size,
Eight directions (e.g., 0 for the rightward flow, 1 for the lower rightward flow, 2 for the downward flow, 3 for the lower leftward flow,
The flow in the left direction is 4, the flow in the upper left direction is 5, the flow in the upper direction is 6, and the flow in the upper right direction is 7. The flows in other directions are the closest directions among the above 8 directions. And the output of the normalizing means (13). Using the flow thus obtained, a flow pattern extraction unit (14) extracts a flow pattern in the vicinity of a pixel of interest, and a feature point extraction unit (15) determines a contraction point and a diffusion point. The means for extracting feature points will be described below. First, as shown in FIG. 4, the contraction point (FIG. 4 (a) and the diffusion point (FIG. 4 (b)
Consider the flow patterns (40) and (41) for A point having a pattern as close as possible to such a flow pattern is determined as a contraction point or a diffusion point. For this reason, numbering according to the direction of the flow is performed for each flow pattern corresponding to the ideal contraction point and diffusion point shown in FIGS. 4 (a) and 4 (b). That is, for example, the flow in the right direction is 0, the flow in the lower right direction is 1, the flow in the lower direction is 2, the flow in the lower left direction is 3, the flow in the left direction is 4, the flow in the upper left direction is 5, and the flow in the upper direction is 5. Numbers are assigned from 0 to 7 according to the direction of each flow pattern so that the flow is 6 and the flow in the upper right direction is 7. When numbering is performed in this manner, the ideal contraction points shown in FIG. 4 (a) are numbered as shown in FIG. 5 (a).
The ideal diffusion point shown in FIG.
Numbering is performed as shown in FIG. As described above, the idealized contraction points and diffusion points shown in FIGS. 4 (a) and 4 (b) are numbered for the contraction point quantization mask (FIG. 5 ( a)) and the quantization mask of the diffusion point (FIG. 5 (b)). In order to determine whether the pixel of interest is a contraction point or a diffusion point, the direction data of the ideal feature point is set to the direction data (60) in FIG. The direction data is given as direction data (61) in FIG. 6 (b). Here, the direction data (60) for the contraction point is Fi = i + 4 (i = 1,2,3,) Fi = i-4 (i = 4,5,6, 7) For the diffusion point, from FIG. 5 (b), Fi = i (i = 0 to 7). Next, the direction data in the vicinity of the pixel of interest actually extracted by the flow pattern extraction means (14) is stored in the sixth direction.
The direction data (61) in FIG. That is, the flow at the right center near the pixel of interest is G0, and the flow at the lower right is G0.
G1 and the flow in the upper center are assigned to G2 and hereinafter to Gi (i = 0,..., 7) as shown in FIG. 6 (b). That is, for example, at the contraction point of the h shown in FIG. 3, a flow similar to the flow shown in FIG.
Numbering similar to that shown in FIG.
G0 = 4, G1 = 5, G2 = 6, G3 = 7, G4 = 0, G5 = 1, G6
= 2 and G7 = 3, and at the diffusion point of the key shown in FIG. 3, a flow similar to the flow shown in FIG. 4 (b) is obtained. Numbering is performed in the same manner as shown, G0 = 0, G1 = 1, G2 = 2, G3 = 3, G4
= 4, G5 = 5, G6 = 6, G7 = 7. At this time, the direction difference from the ideal feature point is given by the following equation.

When | Gi−Fi | ≦ 4, d (Gi) = | Gi−Fi | When | Gi−Fi | ≧ 5, d (Gi) = 8− | Gi−Fi |. Also, the average M of the direction differences between neighboring pixels where the flow appears is Is defined by The target flow is normalized by the normalizing means (13), and pixels whose flow size is smaller than a certain value do not have direction data. Therefore, Ω is a set of pixels in which the flow appears among the neighboring pixels, and N is the number of pixels included in the set Ω. Further, the average of the direction difference for the contraction point and the diffusion point is calculated as MC, MD respectively.
And When the flow pattern near a certain pixel matches the quantization mask at the contraction point, MC = 0 and MD = 4. From this, it is considered that a pixel may be regarded as a contraction point if the average MC of the difference between the flow pattern and the quantization mask at the contraction point near a certain pixel is close to 0. Conversely, if MC is close to 4, it may be determined that it is a diffusion point. The average MD of the direction difference between the diffusion point and the quantization mask
Is calculated, it is determined that it is a diffusion point if it is close to 0, and it is a contraction point if it is close to 4. In other words, when the quantization mask for the contraction point is used, G0 = 4, G1 = 5, G2 = 6, G3 = 7, G4 = 0, G5 for the points shown in FIG. =
1, G6 = 2, G7 = 3, and the quantization mask of the contraction point is F0 =
4, F1 = 5, F2 = 6, F3 = 7, F4 = 0, F5 = 1, F6 =
2, since F7 = 3, d (Gi) = 0, i = 0, 1,
…, 7 Becomes Conversely, when a diffusion point quantization mask is used, the diffusion point quantization mask is F0 = 0, F1 = 1, F2 = 2,
Considering that F3 = 3, F4 = 4, F5 = 5, F6 = 6, and F7 = 7, the point d shown in FIG.
i) = 4, i = 0, 1,..., 7, Becomes Thus, since MC is a value close to 0 (MD is a value close to 4), this point is determined to be a contraction point.

Similarly, for the point shown in FIG. 3, G0 = 4,
G1 = 5, G2 = 6, G3 = 7, G4 = 0, G5 = 1, G6 = 2, G7
= 3 and using a shrink point quantization mask, d
(Gi) = 4, i = 0, 1,..., 7, Becomes Conversely, when a diffusion point quantization mask is used, d (Gi) = 0, i = 0, 1,... Becomes Thus, since MC is a value close to 4 (MD is a value close to 0), this point is determined as a diffusion point. In (16), the moving object detection means determines whether or not there is a contraction point in the image 1 (10) and a diffusion point in the image 2 (11). The contraction point is a point that occurs when a point existing in the image 1 (10) moves and as a result, the corresponding position in the image 2 (11) becomes a background and apparently disappears. In 10), this is a point that is caused by the appearance of a new object due to the movement of the object to the position that was the background. Therefore, the movement direction of the object can be detected by detecting a pair of contraction points and a diffusion point. .

In the above embodiment, the contraction point based on 3/3 pixels,
The diffusion point extraction has been described. For pixels in which the flows of the neighboring pixels are all in the same direction, MC = MD = 2
By using the fact that the moving object can be used to determine the presence of the moving object when the object is moving with a common area between different images. That is, as shown in FIG. 10, in the case of a moving object that moves while having a common area between two images, since the neighboring pixels are all in the same direction, the value of each neighboring pixel is Are assigned to the same value. For example, in the case shown in FIG. 10, since the flow directions are all rightward, Gi = 0, i = 0, 1,.
.., 7 and MC = MD = 2. As described above, since MC (MD) is 2, it can be determined that there is a moving object that is moving while having a common area between the two images. At this time, a flow of 3/3 pixels or more may be checked.

In the above embodiment, the contraction point based on the optical flow,
Since the system is configured to extract diffusion points and detect moving objects, it is difficult to detect with a conventional device when the target object is very small, when the density difference between the background is small, or when the density difference between the background and the object is small. Is reversed, the position and the moving direction of the object can be detected relatively accurately. Further, unlike the related art, it is not necessary to set the two thresholds T1 and T2 for each image.

[Effects of the Invention] As described above, the present invention provides an optical flow calculation means for obtaining an optical flow having a size and a direction indicating a velocity distribution on two images different in time, and this optical flow calculation means Normalizing means for normalizing the size of the optical flow obtained by the above, and outputting the direction of the optical flow having a size equal to or larger than a preset value, and converting the image from the direction of the optical flow obtained by the normalizing means. A flow pattern extracting means for extracting an optical flow pattern in the vicinity of the pixel for each constituent pixel; a contraction point and a flow where the flow direction is concentrated from the vicinity based on the flow pattern obtained by the flow pattern extracting means; Feature point extracting means for determining a diffusion point whose direction is spreading to the vicinity,
In addition, the provision of the moving object detecting means for detecting the moving information of the object from the characteristic points extracted by the characteristic point extracting means eliminates the need for setting the threshold value required for the conventional device for each image, and reduces the density difference. Even when the direction is reversed, the moving direction can be correctly determined, and a moving object detection device capable of detecting the movement of a minute object of about several pixels in an image can be obtained.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a moving object detecting apparatus according to one embodiment of the present invention, FIG. 2 is an explanatory diagram for explaining the characteristics of an optical flow schematically represented by a one-dimensional signal, and FIG. FIG. 4 is an explanatory diagram showing an optical flow pattern obtained when moving, FIG. 4 is an explanatory diagram showing a flow pattern near a contraction point and a diffusion point, and FIG. 5 is a diagram showing a flow pattern near a contraction point and a diffusion point. FIG. 6 is an explanatory diagram showing quantization masks numbered in each direction, FIG. 6 is an explanatory diagram showing direction data of a pixel near an ideal feature point and direction data of a pixel near a point of interest, and FIG. FIG. 8 is an explanatory diagram for explaining the concept of a moving object extraction method using a conventional difference method, and FIG.
FIG. 9 is an explanatory view showing a method of extracting a moving object from a difference image obtained by a difference method from a one-dimensional signal which is a density signal at a line VIIIa-VIIIa and a line VIIIb-VIIIb, and FIG. FIG. 10 is an explanatory diagram showing a conventional example in which a moving direction is determined by extracting a feature point in the case where the background density is reversed, using a one-dimensional signal. FIG. 10 is a diagram showing an example of using a conventional optical flow. FIG. 11 is an explanatory diagram showing a density signal and an optical flow corresponding to the line I-I of FIG. (10), (11) ... image, (12) ... optical flow calculation means, (13) ... normalization means, (14) ... flow pattern extraction means, (15) ... feature point extraction means, (16)
... Moving object detection means, (22) optical flow, (23) spatial density gradient, (24) temporal density gradient. In the drawings, the same reference numerals indicate the same or corresponding parts.

──────────────────────────────────────────────────続 き Continued on front page (58) Field surveyed (Int.Cl. ⁶ , DB name) G01P 13/00 G01P 5/20

Claims (1)

(57) [Claims] 1. An optical flow calculating means for obtaining an optical flow having a magnitude and a direction indicating a velocity distribution on two images which are temporally different from each other, and the size of the optical flow obtained by the optical flow calculating means. Normalizing means for outputting the direction of an optical flow having a magnitude equal to or larger than a predetermined value, and for each pixel constituting the image from the direction of the optical flow obtained by the normalizing means, the vicinity of the pixel Flow pattern extraction means for extracting the optical flow pattern in the above, a contraction point where the flow direction is concentrated from the vicinity based on the flow pattern obtained by the flow pattern extraction means, and a diffusion point where the flow direction is spread to the vicinity Feature point extracting means for determining Moving object detection apparatus provided with a moving object detection means for detecting an object moving information from the extracted feature point Ri.