JPH0973540A - Motion vector calculation device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To calculate a motion vector which appropriately expresses the motion of an object by obtaining the motion vector of respective picture points on a picture through the use of a local correlation method, estimating the motion vector of a part where a luminance change is little and executing interpolation. SOLUTION: Picture data obtained from a TV camera and the like are time-sequentially accumulated in a frame memory 3. A central arithmetic processing part 1 divides respective picture data at respective time into plural picture areas and calculates the motion vector of the respective picture points for the whole picture by using the local correlation method. A correlation arithmetic part 4 calculates the correlation value of picture data at time t1 and t2 after little time passes. The central arithmetic processing part 1 calculates the motion vector of the respective picture points of the picture and the reliability based on the correlation value and substitutes the motion vector of the picture point whose reliability does not satisfy a prescribed value for an interpolation vector for the respective divided picture areas. The interpolation vector is synthesized by giving weight depending on the distance of the picture points to the motion vector of the picture point whose reliability satisfies the prescribed value.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a motion vector calculation device using a local correlation method, and more particularly to a motion vector calculation device for estimating and interpolating a motion vector of a portion having a small luminance change.

[0002]

2. Description of the Related Art Conventionally, in the field of image processing such as image recognition, as a method for tracking a moving object input from a visual sensor such as a TV camera, a distribution of motion vectors (optical flow) at each point on the moving object is used. ) Is taken. As a method of obtaining this motion vector, Berthold KPHorn et al., “Determining Optical F
low '' (ARTIFICAL INTELIGENCE, Vol-17, pp185-203, 1981)
The gradient method proposed in [1] is known. This technique is
The equation (1) is calculated based on the assumption that the image density of each image point on the moving object does not change with time, and the motion vector V (u, v) does not change spatially rapidly. By adding a condition, the motion vector V can be calculated from the equation (1).
(U, v) is obtained.

I _x u + I _y v + I _t = 0 (1) where I _{x is} the density gradient in the spatial x direction I _{y is} the density gradient in the spatial y direction u is the x direction component of the motion vector V v is: Y-direction component of motion vector V

According to this method, the motion vector V (u,
Due to the constraint that v) does not undergo abrupt spatial changes, the background may be clear if the parts that are not moving and the parts that are moving are clear, such as when an object moves in front of a stationary background. There is a problem that the boundary part of the moving object is blurred.

In contrast to the above-mentioned gradient method, as a method for obtaining another motion vector, Tachikawa et. (November 27th to 29th)
The local correlation method proposed in [1] is known. The principle of the local correlation method will be described below. FIG. 6 is a diagram for explaining a method of obtaining a motion vector using the local correlation method. First, attention is paid to an image point P ₁ on the moving object image (not shown) in the image F at time t ₁ . The image of the neighboring area centered on this image point P ₁ is the reference image R ₁
And Next, at time t ₂ when the minute time Δt has elapsed from time t _1, the area S ₁ larger than the reference image R ₁ area as the search area around the image point P _1, the reference image R ₁
The image area r ₁ most corresponding (similar to) is searched. The center point P _1dt of the image region r _{1 thus searched} is the point P 1.
_The image point P _1dt is recognized as the position after a minute time Δt of _1.
A vector whose end point is the image point P _{1 and} whose start point is the image point P ₁ is obtained as the motion vector V ₁ of the point P ₁ . Similarly, the motion vector V _n can be obtained for other image points P _n (n is a positive integer), and the optical flow can be obtained from this distribution.

As described above, in the local correlation method, the reference image R _n
By searching the image area r _n most corresponding (similar) to the area S _n , the destination of the image point P _n is recognized and the motion vector of the image point P _n is calculated. The degree of correspondence (similarity) is quantitatively grasped by introducing an evaluation amount called a correlation value.

By the way, the area S process of searching for best corresponding image region r _n and the reference image R _n from the _n, each image region r _n of the internal area S _n correlation values by the gray value of the image
Is calculated and the image area r _n showing the “highest correlation (similarity)” is found from this correlation value. Here, it should be noted that even if the correlation degree itself is high, it is difficult to specify the image area r _n having the “highest correlation degree” when the search area S is uniform. . In such a case, the motion vector V _n of the image point P _n does not always properly represent the motion of the object. Therefore, in the above-described local correlation method, it is necessary to evaluate how well the calculated motion vector V _n of the image point P _{n represents} the motion of the object.

A method of evaluating a motion vector calculated by using the local correlation method will be described below. In the local correlation method, each image region r inside the region S _n with respect to the reference image R _n
Since the correlation value of _n is calculated, the distribution in which the relative position (relative coordinates) with respect to the image point P _n is the domain and the correlation value is the domain is determined. This distribution is called a local correlation distribution. FIGS. 7A to 7C are two-dimensional local correlation value distribution diagrams showing an example of the distribution of correlation values inside the region S _n with respect to the reference image R _n . Originally, since the image F has a two-dimensional spread, the distribution of the correlation values is expressed three-dimensionally, but here, for convenience of description, the local two-dimensionally expressed. This will be explained using a correlation value distribution chart.

FIG. 7 shows that the smaller the correlation value, the higher the degree of correlation. As shown in FIG. 10A, in the area S _n having a clear local correlation value distribution where the high and low correlation values are high, the image area r _n exhibiting the “highest degree of correlation” is easily specified. be able to. Therefore, the motion vector V _n obtained by searching the region S _n having such a local correlation value distribution can be considered to appropriately represent the motion of the image point P _n , and the obtained motion It can be said that the reliability of the vector V _n is high. On the contrary, when the degree of correlation is uniform as in the example shown in FIG.
It becomes difficult to specify the image region r _n showing the “highest degree of correlation”, and the motion vector V _n obtained by searching the region S _n having such a local correlation value distribution is the motion of the image point P _n . Cannot be said to be properly expressed, and the reliability of the obtained motion vector V _n is low. Further, in the case of the example shown in FIG. 7B, the intermediate reliability is shown.

As described above, how well the motion vector V _n of the image point P _{n represents} the motion of the object can be evaluated by the shape of the local correlation value distribution in the region S _n , The degree to which the correlation value of the image region showing the “highest correlation degree” is more prominent than the average value can be used as a measure for evaluating the motion vector. Therefore, in order to quantitatively evaluate the reliability of the motion vector of the image point P _n , the following function rel is defined and called the reliability evaluation function.

Rel = (C _mean -C _min ) / C _mean (2) where C _mean : average value of local correlation value distribution C _min : minimum value of local correlation value distribution

By using this reliability evaluation function rel, it is possible to quantitatively grasp how much the correlation value C _{min in the} region having a high degree of correlation protrudes from the average value C _mean of the local correlation value distribution. As the value obtained from the reliability evaluation function rel is larger (closer to 1), the calculated motion vector V _n can be evaluated as appropriately expressing the motion of the object.

[0013]

By the way, according to the motion vector calculating apparatus using the above-described local correlation method, when the brightness of the internal image of the area S _n is uniform (the change is small), the reference image It becomes difficult to search the image region r _n that exhibits the “highest degree of correlation” with respect to R _n , and it becomes difficult to specify the point P _ndt that is the position after the minute time Δt of the point P _n . As a result, low reliability of the motion vector V _n of the calculated image point P _n, the motion vector V _n has a problem that not a motion of the image point P _n for appropriately represented.

[0014] Hereinafter, this problem will be described using an example of obtaining the motion vector V _n of each image point P _n from the image F having an area less brightness change. FIG. 8 is a diagram showing an image displaying the state of movement of the object 10, and FIGS. 8A and 8B show the state of the object 10 at time t ₁ and time t _2, respectively. . Here, the time t ₂ is the time when the minute time Δt has elapsed from the time t ₁ . The inner region of the object 10 is uniformly black, and there is almost no change in the brightness in the inner region of the object 10 before and after the elapse of the minute time Δt. In addition, the background is white, so that there is a large change in luminance in the contour portion of the object 10. This object 10
Is a rotational movement in a clockwise direction about the lower right corner.

A motion vector is calculated for such an object 10. FIG. 9 is a diagram showing a motion vector obtained from the two images shown in FIG. 8 using the local correlation method. As shown in FIG. 9, in the contour portion of the object 10 in which the luminance change is remarkable, the movement destination of the image point of the contour portion has been successfully searched, and the motion vector along the rotational motion of the object 10 has been succeeded. Is properly generated. On the contrary,
Inside the object 10 with almost no change in brightness, the search for the destination of each image point inside the object has failed, and therefore,
The motion vector inside the object 10 is disturbed, and does not properly represent the motion of the internal region of the object 10.

As described above, according to the local correlation method, the motion vector of a region where the luminance change is remarkable such as the contour of the object can be obtained with high reliability, and this region (the luminance change is remarkable). While it has the advantage of being able to clearly express the motion of the image in a small area), the reliability of the motion vector in the area where the luminance change is small, such as inside the object, decreases, and this area (the area where the luminance change is small) There is a problem that it is not possible to obtain a motion vector that appropriately represents the motion of the object.

The present invention has been made in view of the above problems, and in a motion vector calculation apparatus using the local correlation method, by estimating and interpolating a motion vector of a portion having a small luminance change, An object of the present invention is to provide a motion vector calculation device that can calculate a motion vector that appropriately represents a motion.

[0018]

In order to solve the above-mentioned problems, the present invention has the following arrangement. A motion vector calculation device according to a first aspect of the invention corresponds to a first time controlled by a storage unit that stores image data corresponding to each time, a central processing unit, and the central processing unit. And a correlation value calculator that calculates a correlation value between the image data and the image data corresponding to the second time, wherein the central processing unit (a) calculates each of the images based on the correlation value. The motion vector of the image point and the reliability of the motion vector are calculated, (b) the image is divided into one or more image regions, and (c) the reliability does not satisfy a predetermined value for each image region. The interpolation vector has a function of replacing the motion vector of the first image point with an interpolation vector, and the interpolation vector is the motion vector of one or more second image points whose reliability satisfies the predetermined value. Distance between image point and the second image point A synthesized vector and the dependent weighting, the second image point is constructed as belongs to an image region where the first image point belongs.

According to a second aspect of the present invention, there is provided a motion vector calculating device according to the first aspect, wherein the first color system color image data is converted into a second color system image. Equipped with color system conversion means for converting to data,
A motion vector is calculated using the converted image data of the second color system.

According to the motion vector calculating apparatus of the present invention, the central processing unit is arranged in the first storage unit.
The image data corresponding to the time and the image data corresponding to the second time are read, and these image data are passed to the correlation value calculator. The correlation value calculator calculates the correlation value between these image data. The central processing unit receives the calculation result of the correlation value from the correlation value calculation unit, and calculates the motion vector of each image point and its reliability based on the calculation result. And
The central processing unit divides the image into one or two or more image areas, evaluates whether or not the reliability of the motion vector of each image point satisfies a predetermined value for each image area, and the reliability is predetermined. The motion vector of the first image point that does not satisfy the value is replaced with the interpolation vector. This interpolation vector belongs to the image area to which the motion vector of the first image point to be replaced belongs, and the motion vector of one or more second image points whose reliability satisfies a predetermined value is set to the first This is a vector that is weighted and combined according to the distance between the image point and the second image point.

According to the motion vector calculating device of the second aspect, the color system conversion means converts the color image data of the first color system into the image data of the second color system. Then, the converted image data of the second color system is used to operate similarly to the motion vector calculation apparatus according to the first aspect of the invention.

[0022]

Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram showing the configuration of a motion vector calculation device according to an embodiment of the present invention. The motion vector calculation device according to the present embodiment is
Central processing unit 1 for controlling the overall operation of this device
A memory 2 for storing program data of the central processing unit 1, a frame memory 3 for storing image data from a visual sensor such as a TV camera, and a local correlation from the memory 2 or the data stored in the frame memory 3. The memory 2 and the frame memory 3 constitute a storage unit (no code). The correlation value calculation unit 4 generates a value distribution and calculates a motion vector.

The operation of the motion vector calculation apparatus according to the embodiment of the present invention configured as above will be described. This motion vector calculation device obtains the motion vector of each image point on the image by using the local correlation method, and replaces the motion vector of low reliability with the interpolation vector to obtain a motion vector that appropriately expresses the motion of the object. (Optical flow) is calculated.

The operation will be described in detail below with reference to the flowchart for explaining the operation of the central processing unit 1 shown in FIG. First, image data obtained from a visual sensor such as a TV camera is stored in the frame memory 3 in time series. The central processing unit 1 divides each of the image data at each time into a plurality of image areas A _i (i is a positive integer) (step S1). FIG. 3 shows a background image area A ₁ and an area A ₂ occupied by the object 10 in the image F shown in FIG.
It is a diagram showing the area | region divided into and. As shown in FIG. 3, the image F is divided into a plurality of image areas A _i based on the image of the image.

Next, the central processing unit 1 uses the local correlation method.
To the image point P over the entire image F using_nMovement of vector
Le V_nIs calculated (step S2). That is, the central performance
The arithmetic processing device 1 has a time t.₁Image data D corresponding to_R1To
It is read from the frame memory 3 and stored in the memory 2.
Here, the image data D_R1Is an image point P on the image₁Centered around
Reference image R of the neighborhood area₁Image data corresponding to
is there. Next, the central processing unit 1 sets the time t₁From minute
Time t when time Δt has passed₂Area S on the image₁Corresponding to
Image data D_S2From the frame memory 3
Data D_R1Is read from the memory 2 and these image data D
_R1And D_S2To the correlation value calculation unit 4. Correlation value calculator 4
Is the data D_R1And data D_S2Calculate the correlation value with
Area S ₁Reference image R₁Is most correlated with
High (small correlation value) image area r₁Explore this picture
Image area r₁The center coordinate value of is calculated. And the central operation
The processing unit 1 uses the image area calculated by the correlation value calculation unit 4.
r₁Enter the center coordinate value of₁Heart of
Standard value and image point P₁Image point P from the coordinate values of₁Movement
Tor V₁To calculate.

Next, the same processing by focusing on the image point P ₂ on the image F at time t ₁ stored in the frame memory 3 is executed, to produce similarly the motion vector V ₂ in the image point P ₂ . Further, similarly, a motion vector is generated for other image points P _n (n is a positive integer) over the entire image F (step S2), and an optical flow is obtained. The optical flow thus obtained contains a motion vector with low reliability, as described in the conventional example.

Next, the reliability of the motion vector V _n of each image point P _n is calculated from the optical flow obtained by the above operation (step S3). Then, the reliability of the motion vector V _n obtained by calculation is compared with the reliability evaluation value a, and if smaller than this value a (step S4, YE
S), _assuming that the reliability is low, this motion vector V _n
Is executed (steps S6 to S12). When the reliability of the motion vector V _n is equal to or larger than the value a (step S4, NO), the reliability of the motion vector at another image point is tested without executing the interpolation process (step S4). S5, NO-step S4).
In this way, the reliability is evaluated for the motion vectors V _n of all the image points P _n , and the interpolation processing of all the motion vectors V _n whose reliability is less than the value a is executed.

Interpolation processing (steps S6 to S1)
The contents of 2) will be described. The central processing unit 1, when the reliability of the motion vector V _n in the image point P _n does not satisfy the value a (step S4, YES), among the divided areas A _i, the point P _n belongs The divided area A _n to be searched is searched and specified (step S6), and the interpolation process is executed for the specified divided area A _n . First, the vector variable v is initialized by setting zero as an initial value (step S7). Next, focusing on the image point Q _j (Q _j εA _n ) (j is a positive integer) included in the specified divided area A _n (step S8), the motion vector V _qj of this image point Q _j is _calculated . The reliability is compared with the reliability evaluation value b (step S9). As a result of this comparison, the motion vector V _qj whose reliability is less than the value b
Is ignored (step S9, NO), and the reliability is similarly evaluated for other motion vectors V _qj (steps S11 to S8 to S9).

As a result of evaluation of the reliability of the motion vector V _{qj, when} the reliability is equal to or larger than the range b (step S9, YES), the motion vector V _{qj is set} to the image point P _n.
A dividing operation by the distance d _nj between the image point Q _j, by adding the vector variable v in the division operation result, performs vector synthesis processing, and stores the result of this process again vector variable v (step S10). Then, the reliability test is performed on the motion vector V _qj of the other image point Q _{j, and} the above-described vector synthesis operation process is repeatedly performed on the motion vector V _qj whose reliability value satisfies the value b to obtain the operation result as a vector. It is cumulatively stored in the variable v, and it is determined whether the same processing has been executed for all the image points Q _j of the specified divided area A _n (step S11, YES / NO). Motion vectors V _{qj of} all image points Q _j whose reliability satisfies the value b
When the vector synthesizing operation is executed and the operation result is accumulated and stored in the vector variable v (step S11, YE
S), the combined vector V _H stored in the vector variable v is defined as the motion vector V _{n at the} image point P _n (step S12). With the above operation, the reliability is the value a.
The motion vector V _n of image points P _n of less than is interpolated by the vector V _H obtained by synthesizing processing using all of the motion vector V _qj satisfying the value b of the specified divided regions A _n (substituted ) Is performed, and the motion vector interpolation process ends.

Here, the above-mentioned vector composition calculation processing (step S10) will be described. FIG. 4 is a diagram for explaining the vector composition calculation processing. In the figure, it is assumed that the reliability of the vectors V _q1 and V _q2 satisfies the value b, and the vector V _q1 and V _q2 is used to interpolate the motion vector V ₁ of the point P _1. A case will be described below. First, a vector V _q11 obtained by dividing the vector V _q1 at a distance d ₁ between the point P ₁ and the point Q _1.
Next, determine the vector V _q21 obtained by dividing the vector V _q2 at a distance d ₂ between the point P ₁ and point Q _2. Then, these vectors V _q11 and V _q21 are combined to obtain a vector V _H1 . The vector V _H1 is a composite vector having a size weighted by the distances d ₁ and d ₂ , and this vector V _H1 is a vector that interpolates (replaces) the motion vector V ₁ of the point P ₁ .

FIG. 5 is a diagram showing an optical flow generated by being interpolated by the interpolation vector V _H. As shown in the figure, also inside the object 10, a motion vector that appropriately expresses the motion of this object is generated. The process of step S10 of the flowchart of FIG. 2 will be supplementarily described using the vector V _q11 and the vector V _q21 described above. First, the above-mentioned vector V _q11 is obtained and stored in the vector variable v, and then the above-mentioned vector V _v11 is stored.
_{By finding q21} and adding it to the vector variable v,
The vectors V _q11 and V _q21 are vector-synthesized, and the synthesized vector obtained by this vector synthesis is stored again in the vector variable v. Therefore, when a large number of vectors satisfying the reliability value b exist inside the specified divided area A _n , the vector obtained by weighting these vectors is added to the vector variable v, and the vector variable v is added again. By repeating the process of storing in, a vector that is synthesized by using all the vectors whose reliability satisfies the value b is generated in the variable v. Then, the content of this variable v is defined as the interpolated vector V _H1 of the image point P _1, the motion vector V ₁ of the image point P ₁ interpolation vector V
It is replaced by _H1 . By performing such interpolation processing, the object 1 with almost no change in brightness as shown in FIG.
With respect to the internal region of 0, it is possible to obtain an optical flow having a distribution of motion vectors that appropriately expresses motion.

As described above, the motion vector calculation apparatus according to the embodiment of the present invention calculates and processes the correlation value from the lightness and darkness of an image. It is also possible to perform a process, for example, to calculate a correlation value from the hue and process it.
The essence of the present invention is not limited by the method of calculating the correlation value. For example, a color system conversion unit 5 is added to the motion vector calculation device shown in FIG. 1 to configure the device as shown in FIG. 10, and the color system conversion unit 5 converts RGB color image data into Munsell HVC (H : Hue, V: Luminance, C: Saturation) Convert to color system. In the correlation calculation, if the calculation based on the hue H is performed, it is possible to obtain the optical flow of the movement of the person, and for example, it is possible to easily recognize whether the person shakes the head vertically or horizontally. .

[0033]

According to the motion vector calculation apparatus of the present invention, since the interpolation processing is performed on the motion vector with low reliability generated in the area where the brightness change is small, even if the brightness change is small, the object A motion vector that appropriately represents the motion can be obtained. Further, since the motion vector is calculated using the local correlation method, it is possible to obtain a motion vector in which the contour portion of the object having a remarkable change in brightness is not blurred. Further, by providing the means for converting the color system, the motion vector can be calculated from the hue or the like even if there is no change in luminance.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of a motion vector calculation device according to an embodiment of the present invention.

FIG. 2 is a flowchart for explaining the operation of the central processing unit 1.

FIG. 3 is a background image area A ₁ and an area A ₂ occupied by the object 10.
It is a diagram showing the divided area of.

FIG. 4 is a diagram for explaining a vector synthesis calculation process.

FIG. 5 is a diagram showing an optical flow generated by being interpolated by an interpolation vector V _H.

FIG. 6 is a diagram for explaining a method of obtaining a motion vector using a local correlation method.

7A to 7C are two-dimensional local correlation value distribution charts showing an example of a distribution of correlation values inside the region S _n with respect to the reference image R _n .

8A and 8B are diagrams showing the states of the object at time t ₁ and time t ₂ , respectively.

FIG. 9 is a diagram showing an optical flow obtained by using a local correlation method.

FIG. 10 is a block diagram showing a configuration of a motion vector calculation device according to an embodiment of the present invention, which includes a color system conversion unit.

[Explanation of symbols]

 1 Central processing unit 2 Memory 3 Frame memory 4 Correlation value calculation unit 5 Color system conversion unit 10 Object

Claims (2)

[Claims] 1. A storage unit for storing image data corresponding to each time, a central processing unit, and the image data corresponding to a first time and a second time under the control of the central processing unit. A correlation value calculation unit that calculates a correlation value between the corresponding image data, and the central calculation processing unit includes (a) a motion vector of each image point of the image based on the correlation value and the motion vector. And (b) dividing the image into one or more image regions, and (c) calculating a motion vector of a first image point whose reliability does not satisfy a predetermined value for each image region. The interpolation vector has a function of replacing the motion vector of one or more second image points whose reliability satisfies the predetermined value with the first image point and the second image point. Vector with weighting depending on the distance to Le, as described above, and said motion vector calculation apparatus in which the second image point is characterized in that belongs to an image region where the first image point belongs. 2. The color image data of the first color system is converted into the second color image data.
The color vector conversion means for converting into the color coordinate system image data of (1), and the motion vector is calculated using the converted image data of the second color system. Motion vector calculation device.