JPH06259561A - Device for calculating target in moving picture and target tracking device - Google Patents

Abstract

PURPOSE:To highly accurately and quickly calculate the moving speed and moving direction of a target in a moving picture by detecting the direction of an edge from a specific local picture out of still pictures constituting the moving picture and calculating a local speed by using the value of edge passage time in the direction vertical to the detected edge direction. CONSTITUTION:A filtered picture is inputted to a frame memory 3 at first and an edge direction detector 9 detects an edge from a local picture read out from the memory 3. The detector 9 includes an edge detecting unit for outputting a maximum value in the direction of a specific shadow edge to detect the direction of a local edge. Then an edge passage time calculating device 10 calculates the passage time of the edges of respective picture elements arranged in the direction vertical to the detected edge direction. A local speed calculating device 11 calculates a local speed based upon the calculated edge direction and edge passage time. An overall speed calculating device 12 integrates the local speeds and obtains the total speed of the target.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an apparatus for calculating the moving speed of an object being photographed on an image, from temporally continuous images (eg, television images).

[0002]

2. Description of the Related Art Image processing for capturing image information, distinguishing and identifying an object contained therein, and recognizing an external state has been performed by various methods. However, conventional image processing cannot perform flexible processing, and it is difficult to identify individual objects from general complex images. Only limited identification is possible when the target is limited. . The problem with this conventional engineering method is that
Especially in the image processing of a moving object, it appears remarkably, and there is a strong demand for a novel approach. On the other hand, in visual information processing of living things such as humans and monkeys, flexible processing is possible, each object can be identified under arbitrary circumstances, and three-dimensional stereoscopic recognition and motion detection are also possible.

Journal as an image processing model capable of calculating the velocity of a moving target based on the visual information processing of living things
of optical society of Am
Erica (1990), vol. 7, No. 2, pp
264-278. A. There is a method by Perrone. A flowchart of this method is shown in FIG.

First, a moving image is captured as a still image at a constant time interval, and six consecutive still images are taken as an input image (step 17), and a Gaussian filter is applied to this to obtain a high frequency component of image noise. Remove (Step 1
8). Next, the speed sensor calculates the local speed from the bright and dark edges of the local image (step 19). Next, the velocity component in a specific direction is calculated from the set of local velocities (step 2
0), the direction and speed at which the target moves are calculated from the velocity component (step 21).

FIG. 11 shows an example in which the spatial second derivative operation is applied to the one-dimensional brightness change of an image. (A) shows a spatial brightness change of an image. The result of applying a Gaussian filter for removing high frequency noise components to this is (b). The result of the spatial second-order differential calculation for this (b) is (c). Comparing the positional relationship between the light-dark change in (a) and the spatial second-order differential value in (c), it can be seen that the zero crossing point in (c) corresponds to the light-dark edge in (a).

FIG. 12 shows each pixel at time t1, t2, t3.
It shows the relationship between the bright and dark edges that pass by and the pixel passing time. (A) shows the relationship between the bright and dark edges and the pixel position, and the edge passage time of the central pixel (indicated by a double square mark) is t2. (B) shows a process in which the spread of the spatial second-order differential value corresponding to the light-dark edge moves to the right. At the position of the central pixel, the sign of the second-order differential value changes from negative to positive before and after time t2. is doing.
In general, the sign of the spatial second derivative of the light / dark change when the light / dark edge passes through a particular pixel is reversed (negative to positive, or positive to negative). (C) shows the temporal change of the second-order differential value in the central pixel, and it can be seen that the time t2 when the sign changes from negative to positive is the passing time of the bright and dark edges.

FIG. 13 is a view showing the speed calculation sensor for the bright and dark edges in step 19 of FIG. This sensor is composed of 13 pixels indicated by black circles in the figure, and the difference in edge passage time between the central pixel and the 12 pixels on the circumference is calculated. The edge passage time at a pixel is calculated by the method shown in FIG. In FIG. 13, when a bright / dark edge passes through the sensor at a speed v, the difference in edge passing time is calculated in 12 directions as viewed from the central pixel. Among these, the direction with the largest time difference is the bright / dark edge. Is the moving direction (0 degree direction in the figure), and the speed at the central pixel can be calculated from this time difference and the distance d between the central pixel and peripheral pixels, and the local speed vector can be obtained together with the direction.

FIG. 14 shows a method of estimating the moving direction and speed of the entire visual target by integrating the obtained local velocity vectors in the entire visual field. (A) is an orthogonal projection of the obtained local velocity vector in each direction (only 0, 45, 90, and 135 degrees are shown in the figure). Vectors A, B, and C are obtained by calculating the average of the orthogonal projection components of the local velocity vector (in the example of the figure, the average becomes approximately 0 in the 90 ° direction), and Can be seen. (B) shows three cosine curves in which the norm of these three vectors is the amplitude and the direction of the vector is the phase angle. (C) is a curve obtained from the sum of these three curves, and the speed and movement direction of the target are estimated from the extreme value of this curve and the corresponding angle.

[0009]

Since the conventional speed calculation method of Perrone calculates the speed while the inclination of the bright and dark edges is unknown, the speed must be evaluated in all 12 directions of the peripheral pixels. In addition, since the speed calculation is performed only from the information of two adjacent pixels, there is a problem that the calculation can be performed only in the case of the speed within a very limited range, and the calculation error becomes large when the speed exceeds the range. The present invention has been made to solve the above problems, and an object thereof is to perform a highly accurate and speedy calculation of speed. Against the background of these problems, we have been engaged in monkey physiological research for a long time. As a result, experimental results suggesting that the detection of the edge direction and the calculation of the edge movement speed are separately performed in the monkey were obtained, and the speed calculation device according to the present invention was proposed.

[0010]

A velocity calculation device according to the present invention first determines a direction of a light-dark edge from a local light-dark change, and limits a direction of velocity evaluation to a direction perpendicular to this edge, The speed of the local edge is calculated from the values of the edge passage times of a number of pixels lined up in the same direction, and the speed and direction of the entire target is calculated from the speed and direction of the multiple edges of the target calculated in the same manner. is there.

[0011]

According to this local velocity calculation method, the direction of an edge is detected from a specific one local image among the still images that form a moving image, and the edge passage time at a large number of pixels arranged in a direction perpendicular to that direction is detected. Since the local velocity is calculated using the value of, the local velocity can be calculated with high precision, and thus the velocity of the moving target can be calculated with high precision and speed.

[0012]

【Example】

Example 1. FIG. 1 is a device for calculating the speed of a target in a moving image, 1 is an image input device for capturing an input image that is repeatedly input, 2 is an image preprocessing filter device, and 3 to 8 are moving images. A frame memory for storing still image data to be processed, 9 is an edge direction detection device, 10 is an edge passage time calculation device, 11 is a local velocity calculation device, 12
Is an overall speed calculator.

The image input to the input device 1 at a constant time interval is processed by the image preprocessing filter device 2 to remove high frequency noise and perform preprocessing for the next data processing. The characteristic of the filter is a low pass filter such as a Gaussian filter. The filtered images are sequentially stored in the frame memories 3-8. The latest image enters the frame memory 3, the oldest image is held in the frame memory 8, and the image shifts in the order of 3 → 4 → 5 → 6 → 7 → 8 each time the time is updated.

Next, the edge direction detection device 9 detects edges from the local image in the frame memory 3. The edge direction detection device has an edge detection unit that outputs the maximum value in the direction of a specific bright and dark edge, and the direction of the local edge can be detected depending on which unit has the maximum output. FIG. 2A shows the filters of the four types of edge detection units used in this embodiment and the outputs of each unit for the four types of images. It can be seen that each edge detection unit gives the maximum output value in the case of an image including bright and dark edges in the direction to be detected. The same effect can be obtained by using the first-order differential type shown in FIG. 2B instead of the second-order differential type shown in FIG. 2A as the edge detection unit. In this case, there is also an effect that the SN ratio is good for an image with a lot of noise.

Next, the edge passage time calculation device 10 calculates the passage time of the edge of each pixel arranged in the direction perpendicular to the edge direction obtained by the edge direction detection device. The time series data required for this calculation is taken in from the frame memories 3-8. The edge passage time is the time at which the brightness changes abruptly at that pixel position, the response of the second-order differential filter ∇ ² G is calculated, and the time at which its sign is reversed is taken as the edge passage time.

In general, the spatial lightness / darkness change of an image is represented by I (x,
y), the Gaussian filter is G (σ, x, y) (σ is the standard deviation), and the spatial second-order differential operation is ▽ ² , the image processing process is ▽ ² (G * I) or (▽ ² G) *. Represented by I.
* Indicates convolution. By changing the parameter σ of G, it is possible to perform image processing corresponding to various brightness changes with respect to I. This spatial second-order differential calculation uses D as a parameter f (x−D) −2 · f (x) + f (x + D) (where f (x) = G * I (x)) (1) And ask. Changing the value of D has a similar effect to changing the value of σ, and by this, the same image processing as when changing the parameter σ of G is possible. In this embodiment, for example, D = 4 is used.

The local velocity calculation device 11 includes the device 9
This is a device for calculating the local velocity based on the edge direction calculated by 10 and the edge passage time. Pixels arranged in the direction perpendicular to the edge direction are searched, the relative distance from the pixel that is the reference position and the edge passage time are obtained, the slope between the distance and the passage time is obtained by the least square method, and the inverse of the slope is calculated. The local velocity at a specific pixel. FIG. 3 shows the estimated value of the local velocity calculated by the device 11 and its error, and it can be seen that highly accurate velocity calculation is possible.

The local velocity is integrated by the overall velocity calculation device 12 to obtain the velocity of the entire target. When the movement of the target generally includes movement in the depth direction and rotational movement, it is not easy to integrate the local velocities, but the movement is limited to translational movement on a two-dimensional plane and the contour of the object does not change. Then, for example, Nature (vol. 300, No. 9, pp5
23-525, 1982). H. A
delson and J.D. A. IOC by Movshong et al.
(Intersection Of Constrai
nt) algorithm, the velocity of the entire target can be calculated without iterative calculation by limiting the motion of the target to translational motion on a two-dimensional plane and assuming that the shape of the target does not change.

FIG. 4 is an explanation of the IOC algorithm. When a triangle moving in a two-dimensional plane is viewed from a local visual field, due to the fact that it is called the peephole problem, the motion of the side passing through the visual field can be evaluated only in the direction perpendicular to the side, and in the direction parallel to the side. The velocity component of cannot be calculated within the local visual field. Vectors A, B, and C in (a) are velocity vectors detected locally, and each represents a component perpendicular to three sides. The velocity vector that represents the true motion of the triangle is
As shown in (b), it can be obtained as a vector whose end points are the intersections of the three constraint lines defined by the vectors A, B, and C with the start points combined. The restraint lines are A, B,
C A perpendicular line that passes through the end of each vector, and the end of the vector that represents the overall velocity is limited to this constraint line. In the translational motion in the two-dimensional plane, the intersections formed by the constraint lines theoretically intersect at one point, so that the velocity vector of the translational motion of the entire target can be determined.

In the above embodiment, the still image held in the frame memory 3 is fetched into the edge inclination detecting device to detect the inclination of the bright and dark edges in each pixel, but the same operation can be performed in other frame memories.

In the above embodiment, the method of calculating the slope of the distance and the passage time by the approximation calculation using the least square method is used, but the approximation calculation may be performed by using another optimization method.

In the above embodiment, an example using four types of edge detection units is shown, but five or more types of edge detection units may be used.

In the above embodiment, the edge passage time is a discrete value corresponding to the discrete time of the image input, but when the sign of the second differential value is converted between two discrete times in a certain pixel, the respective times are changed. It is also possible to interpolate based on the second-order differential value in and take a continuous value as the edge passage time. For example, when the second-order differential values at times t1 and t2 are −v1 (negative value) and v2 (positive value), the edge passing time t by interpolation is, for example, t = (t1 · t2 + v2 · t1) / (v1 + v2 ) ... Can be calculated by (2). In the above-described embodiment, the image input device 1 can also process the output of a video tape recorder or the like as an input.

Example 2. Although the number of frame memories is six in the first embodiment, other numbers may be used. Fifth
The figure shows the relationship between the number of frame memories (4 to 9) and the calculation speed error. Target speed 1-9 pixels /
In the range of the screen, it is possible to keep the calculation error within 3% with 6 or more frame memories, but it is necessary to have 9 or more frame memories to keep the calculation error within 1%.

Example 3. Fig. 6 shows the second derivative filter ▽ ²
FIG. 7 is a diagram showing a relationship between image processing by G and edge moving speed calculation. (A) is a case of a slow speed, and the times t1 and t
The sign changes at 2 and the edge passage time can be calculated.
(B) is a case of high speed, and the values at the times t1 and t2 are both 0, and the edge passage time cannot be calculated.
Although the edge moving speed in (c) is the same as that in (b), the parameter D of the second-order differential calculation of the above-described equation (1) is used.
Image processed with a larger value than in the case of (b),
The slope near the zero crossing is gentle. From this, it can be understood that if D is taken large, it is possible to cope with the case where the moving speed of the light and dark edges is relatively fast. However, with respect to low-speed movement, the smaller the gradient near the zero crossing point, the smaller the change in the value. Therefore, at low speed, an error is likely to occur in the interpolation for calculating the edge passage time.

In FIG. 7, the parameter D of the second derivative calculation is set to 1
The allowable range of the evaluation error is changed to 3 from 12 to 12, for example.
This is an examination of the possible range of high-accuracy speed calculation when it is set to%. The vertical axis represents the speed (unit: pixel / screen), the horizontal axis represents the parameter D, and the square mark indicates a specific parameter with an error of 3
It indicates that the velocity can be calculated within%, and x indicates that the velocity can be calculated but the error is 3% or more.

From the figure, when D = 4, the movement of the target is 1
In the case of ~ 9 pixels / screen, the speed can be calculated within an error of 3%. When the speed is 10 pixels / screen or more, the speed can be calculated within an error of 3% when D is in the range of 7 to 10, and the maximum speed evaluation is the largest in D.
= 10. From the above, by preparing two kinds of edge passage time calculation devices, one having the second-order differential calculation parameter having D = 4 and the other having D = 10, the movement of the target is 1 to 21 by switching depending on the speed of the target. In the case of the pixel / screen range, the speed can be calculated within an error of 3%.

In the above embodiment, an example of a combination of D = 4 and D = 10 is shown, but in the case of a low speed, D = 1 instead of D = 4.
It is also possible to use an edge passage time calculation device in which any one of the above items is selected, and in the case of a high speed, D = 8 to 12 is selected and combined instead of D = 10.

The above embodiment relates to a system in which two or more kinds of second-order differential operations are used with the parameter σ (standard deviation) of the Gaussian filter being constant. For example, one kind of second-order differential operation is used and the parameter σ of the Gaussian filter is used. Even if two or more types are prepared and calculated, the same effect as this embodiment can be obtained. Further, D and σ may be changed proportionally.

Example 4. FIG. 8 shows a target tracking system including the target speed calculating device according to the first embodiment. Reference numeral 13 is an image input device that includes a movable camera, captures an image of a target and supplies it as two-dimensional image information, and 14 is a target velocity calculation device.
The velocity calculation method shown in FIG. 1 is used. Reference numeral 15 is a camera control signal conversion device, and 16 is a camera control device. In this embodiment, in order to follow the movement of the target, the speed evaluation device 1
The camera is controlled by the camera control signal conversion device 15 using the speed calculated in step 4. Such a system makes it possible to follow the target. Further, by actually following the target in this manner, the speed of the target can be calculated more accurately.

FIG. 9 shows the relationship between the error in the speed calculated by the target speed calculating apparatus of the first embodiment and the control of the image input apparatus. (A) shows that the target is moving at the speed v ₁ within the visual field as the image processing range. (B)
FIG. 6 is a diagram showing a case where there is an error from the actual speed v ₁ when the target tracking is performed with the target speed calculated by the target speed calculator as v ₂ . When you follow the target based on the calculated velocity v _2, it is again calculated by the target speed calculating device error v ₁ -v ₂ calculation speed as the speed of the apparent. In (c), the apparent speed is added to the follow-up speed to feed back, and the calculated speed is corrected and followed to reduce the calculation speed error, so that the calculated speed converges to the actual speed. We show that it is possible to follow a moving target with high accuracy.

[0032]

As described above, according to the present invention, the direction of an edge is detected from a specific one local image among the still images that make up a moving image, and a large number of pixels are arranged in a direction perpendicular to the direction. Since the local velocity is calculated using the value of the edge passage time at the pixel, the local velocity can be calculated with high accuracy, and thus the velocity of the moving target can be calculated with high precision and speed.

[Brief description of drawings]

FIG. 1 is a diagram showing a target velocity calculation device according to an embodiment of the present invention.

FIG. 2 is a diagram showing an application example of an edge detection filter according to an embodiment of the present invention.

FIG. 3 is a table showing accuracy of calculation speed according to an embodiment of the present invention.

FIG. 4 is a diagram for explaining an IOC algorithm.

FIG. 5 is a diagram showing a relationship between the number of frame memories and a calculation speed error according to an embodiment of the present invention.

FIG. 6 is a diagram showing a method of calculating pixel passing times of bright and dark edges according to an embodiment of the present invention.

FIG. 7 is a diagram showing a relationship with velocity calculation when a second-order differential parameter is changed according to an embodiment of the present invention.

FIG. 8 is a diagram showing a target tracking system including a visual velocity calculation device according to an embodiment of the present invention.

FIG. 9 is a diagram showing the relationship between the calculation speed error and the control of the image input device according to the embodiment of the present invention.

FIG. 10 is a diagram showing a flowchart of a conventional algorithm.

FIG. 11 is a diagram showing an example in which a spatial second derivative operation is applied to a one-dimensional brightness change of an image in the conventional method.

FIG. 12 is a diagram showing a relationship between a bright / dark edge and a pixel passing time in the conventional method.

FIG. 13 is a diagram showing a configuration of a speed sensor in a conventional method.

FIG. 14 is a diagram showing a method of estimating a local velocity and a method of estimating a velocity vector in a conventional method.

Claims (7)

[Claims] 1. An image input device for repeatedly inputting two-dimensional image information of an image including a moving target, a plurality of frame memories for shifting and storing the image information in an input order, and bright and dark at each point on the image. An edge direction detection device that detects the direction of an edge, an edge passage time calculation device that calculates a passage time of a light-dark edge in a direction perpendicular to the direction of the light-dark edge, and a direction perpendicular to the direction of the light-dark edge from the passage time of the light-dark edge. Local velocity calculation device for calculating the moving speed of the bright and dark edges in different directions, and overall velocity calculation for calculating the moving velocity and moving direction of the entire target from a plurality of sets of the same target of the direction of the bright and dark edges and the moving velocity And a device for calculating a target velocity in a moving image. 2. The edge passage time calculation device calculates the time at which the sign of the spatial second derivative of the image information is reversed at a certain pixel as the edge passage time at that pixel. The target velocity calculation device described in the paragraph. 3. The local velocity calculation device uses a passing time of the bright and dark edges in a plurality of pixels arranged in a direction perpendicular to the direction of the bright and dark edges to determine a moving velocity of the bright and dark edges in a direction perpendicular to the bright and dark edges. The target velocity calculating device according to claim 1 or 2, wherein the target velocity calculating device calculates. 4. The edge passage time calculation device is provided with a plurality of values of a parameter D of a spatial second derivative calculation, a low value of D is used for a low speed target, and a high value of D is used for a high speed target. The target velocity calculation device according to claim 2, which is used. 5. The target velocity according to claim 1, further comprising an image pre-processing filter having a low-pass filter in a preceding stage of the frame memory. Computing device. 6. The target velocity calculating apparatus according to claim 5, wherein the low pass filter of the image preprocessing filter is a Gaussian filter. 7. An image input device, comprising a movable camera, for capturing an image including a moving target and repeatedly inputting two-dimensional image information, and a plurality of frame memories for shifting and storing the image information in an input order. An edge direction detection device that detects the direction of a light-dark edge at each point on the image, an edge passage time calculation device that calculates a passage time of the light-dark edge in a direction perpendicular to the direction of the light-dark edge, and the light-dark edge A local velocity calculation device that calculates the moving speed of the light-dark edge in the direction perpendicular to the direction of the light-dark edge from the passing time, and the moving speed of the entire target from a plurality of sets of the same direction of the light-dark edge and the moving speed for the same target. And a target velocity calculation device including an overall velocity calculation device that calculates the moving direction, and based on information about the moving velocity and moving direction of the entire target from the previous target velocity calculation device. And a camera control device for driving and controlling the camera of the image input device so as to follow the optotype.