JP2983354B2 - Exercise parameter extraction method - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a motion parameter extraction method for extracting a translation speed and a rotation speed of an object as three-dimensional motion parameters based on the coordinates and speed of a moving image projected on an image plane.

[0002]

2. Description of the Related Art Conventionally, a three-dimensionally moving object is imaged by a TV camera or the like and projected on an image plane. A motion parameter extraction method for extracting a translation speed and a rotation speed is known.

FIG. 7 is a diagram showing the principle of a conventional motion parameter extraction method of this kind. In FIG. 7, X, Y, and Z indicate the coordinates of a point on a moving object, and the coordinates of the point where the point is projected on the image plane are indicated by x and y. Here, the origin of the coordinate system is set at the center of the lens of the TV camera. It is assumed that the focal length of the lens is 1. At this time, the coordinates X, Y, Z and the coordinates x, y satisfy the following relational expression.

X = X / Z, y = Y / Z (1) Further, the translation speed T _X ,
When moving at T _Y , T _Z and rotation speeds W _X , W _Y , W _Z ,
Points x and y on the image plane are speeds V _X and V _Y calculated by the following equations.
Having. _{V X = -W X xy-W} Y (l + x 2) -W Z y + (T X -T Z x) / Z V Y = W Y xy-W X (l + y 2) + W Z x + (T Y -T Z y) / Z (2) On the other hand, the brightness of each point on the image plane at time t is expressed by I
(X, y, t). Each point on the image plane is V _X , V _Y
At the time t, the point at the position of the coordinates x, y at time t + dt is x + V _X dt,
Move to the position of y + V _Y dt. The brightness of each point does not change during the movement, that is, the image at time t and the time t
If the brightness of the corresponding points of the image at + dt is equal, the following equation is established.

I (x, y, t) = I (x + V _X dt, y + V _Y dt, t + dt) (3) Accordingly, when the equation (3) is partially differentiated, the following equation is established. ∂I / ∂xV _X + ∂I / ∂yV _Y + ∂I / ∂t = 0 (4) Here, when the velocities V _X and V _Y of equation (2) are substituted into equation (4), the image Brightness I (x, y, z) and motion parameter T
_The following equation representing the relationship between _X , T _Y , T _Z , W _X , W _Y , and W _Z and the depth coordinate Z for each point of the image is obtained.

[0006] _{∂I / ∂x {-W X xy +} W Y (l-x 2) -W Z y + (T X -T Z x) / Z} + ∂I / ∂y {W Y xy-W X (l + y _{^{2) + W Z x + (}} T Y -T Z y) / Z} + ∂I / ∂t = 0 ·· (5) the equation (5) is satisfied at each point of the image. By solving equation (5), it is expected that the motion parameters and the depth coordinates can be obtained.

However, while the equations exist as many as the number of pixels in the image, the depth coordinates Z
There are (x, y) as many as the number of pixels in the image, and there are as many as six motion parameters. For this reason, since the number of unknowns is larger than the number of equations, a solution cannot be obtained by equation (5).

Therefore, conventionally, a motion parameter and a depth coordinate have been calculated by adding a constraint condition such that the depth coordinate changes smoothly. For example, the result of adding the square of the left side of equation (5) to all the pixels,
A solution is obtained so as to minimize the sum of the sum of the square of the derivative of the depth coordinate Z (x, y) in the x direction and the square of the derivative in the y direction for all pixels. I was

[0009]

However, in order to obtain a solution using the above-mentioned conventional method, the solution is repeatedly updated so as to minimize the sum after giving an initial value of the solution. , The problem is that the amount of calculation becomes enormous. In addition, iterative calculations did not always converge on a correct solution.

An object of the present invention is to provide a motion parameter extraction method capable of reducing the amount of calculation and obtaining an accurate motion parameter.

[0011]

In order to solve the above-mentioned problems and achieve the object, the present invention has a structure as shown in FIG. 1 as a first invention. By imaging an object that moves in a three-dimensional coordinate system (X, Y, Z), a two-dimensional coordinate system (x,
image input means 1 for projecting a moving image on the image plane of y)
A spatial filter processing means 3 for applying at least two types of spatial filter processing to a moving image output from the image input means 1; a time differential and an x-direction for each output from the spatial filter processing means 3 Differential processing means 5 and 6 for performing the differential processing and the y-direction differential processing,
And a speed calculation means for calculating a translation speed and a rotation speed as three-dimensional motion parameters of the object by eliminating the coordinate Z component included in each output by using each output of 6.

FIG. 2 shows a second invention. Image input means 1 for projecting a moving image on an image plane of a two-dimensional coordinate system x, y by imaging an object moving in a three-dimensional coordinate system X, Y, Z, and output from the image input means 1. Filter processing means 11 for extracting at least two kinds of spectral bands from a moving image
And differential processing means 5 and 6 for performing time differentiation, x-direction differentiation and y-direction differentiation processing on each output of the color filter processing means 11, and using the outputs of the differentiation processing means 5 and 6. A speed calculating means 7 is provided for calculating a translation speed and a rotation speed as three-dimensional motion parameters of the object by eliminating the coordinate Z component included in each output.

Further, the following is more preferable. That is, the speed calculating means 7 is provided with the differential processing means 5, 6
The coordinate Z component is calculated based on each of the outputs.

[0014]

According to the present invention, the following functions are exhibited. The moving image from the image input means is subjected to two types of filter processing by the spatial filter processing means, the obtained outputs are subjected to time differentiation and spatial differentiation by the differentiation processing means, and the speed calculation means is obtained by using the obtained outputs. Then, the coordinate Z component included in each output is deleted, and the translation speed and the rotation speed are calculated as three-dimensional motion parameters of the object.

Further, the moving image from the image input means is subjected to filter processing by a color filter processing means, and the obtained outputs are subjected to time differentiation and spatial differentiation by a differentiation processing means. The calculation means eliminates the coordinate Z component included in each output, and calculates the translation speed and the rotation speed as three-dimensional motion parameters of the object.

If necessary, the depth coordinate Z can be obtained by solving each output of the differential processing means.

[0017]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, specific embodiments of the present invention will be described. <Embodiment 1> FIG. 3 is a block diagram showing the configuration of Embodiment 1 for realizing a motion parameter extraction method according to the present invention.
FIG. 4 is a flowchart for explaining the exercise parameter extraction method. Example 1 will be described with reference to FIGS. 3 and 4.

In FIG. 3, the TV camera 1 picks up an image of a moving object in a three-dimensional coordinate system (X, Y, Z) as shown in FIG. A / D converter 2 for projecting a moving image on a plane (step 101) and converting its output into a digital signal
Output to

The digital signal output from the A / D converter 2 is taken into the spatial filter circuits 3a and 3b. The spatial filter circuits 3a and 3b apply the following equation (6) to the digitized moving image from the A / D converter 2.
According to equation (7), at least two types of spatial filter processing are performed (step 102). Here, the spatial filter circuits 3a and 3b determine a weighted average value of image densities in a square (that is, n × n (n is an integer)) or a circular visual field with a certain point as a center, and return that value to that It is the density of a point. For example, a weighted average value of the density of each point in a 3 × 3 square area with respect to a certain point (x, y) is obtained as a new density of the image after the filter processing. The value of each element is a weight coefficient applied to the density of each point in the 3 × 3 pixel area.

That is, the spatial filter circuits 3a and 3b
In the two-dimensional digital image, noise is removed by performing image smoothing, that is, integration operation. Therefore, the spatial filter circuits 3a and 3b output the input image I
Two types of spatial filters fl for (x, y, t)
(X, y) and fl (x, y) are applied to obtain the following equation.

I ₁ (x, y, t) = {f ₁ (xy ′, yy ′) I ₁ (x ′, y ′, t) dx′dy ′ (6) I ₂ ( x, y, t) = {f ₂ (xy ′, yy ′) I ₂ (x ′, y ′, t) dx′dy ′ (7) The delay circuits 4 a and 4 b are spatial filters. The moving image subjected to the filtering process from the circuits 3a and 3b is held for a predetermined time with a delay, and is configured by an image memory or the like.

The time differentiating circuits 5a and 5b include a delay circuit 4
Time differential processing is performed by taking the difference between the filtered image (the image one time before) delayed by a fixed time in a and 4b and the immediately preceding filtered image from the spatial filter circuits 3a and 3b (step 103). ).

The x-direction differentiators 6a-1 and 6b-1 apply x-direction differentiation to the outputs from the spatial filter circuits 3a and 3b, and the y-direction differentiators 6a-2 and 6b-2.
Represents y with respect to the outputs from the spatial filter circuits 3a and 3b.
A differentiation process in the direction is performed (step 104).

That is, equations corresponding to equation (5) are established for the two images shown in equations (6) and (7). _{∂I 1 / ∂x {-W X xy} + W Y (l-x 2) -W Z y + (T X -T Z x) / Z} + ∂I 1 / ∂y {W Y xy-W X (l + y 2 _{) + W Z x + (T} Y -T Z y) / Z} + ∂I 1 / ∂t = 0 ·· (8) ∂I 2 / ∂x {-W X xy + W Y (l-x 2) -W Z _{y + (T X -T Z x} ) / Z} + ∂I 2 / ∂y {W Y xy-W X (l + y 2) + W J x + (T Y -T Z y) / Z} + ∂I 2 / ∂ t = 0 Next, the outputs from the time differentiating circuits 5a and 5b, the x-direction differentiating circuits 6a-1 and 6b-1, and the y-direction differentiating circuits 6a-2 and 6b-2 are used as the motion parameter calculating circuit 7. It is taken in. <Description of Operation of Motion Parameter Calculation Circuit> The motion parameter calculation circuit 7 includes x-direction differentiation circuits 6a-1, 6b-1, and y.
By using each output of the direction differentiating circuits 6a-2 and 6b-2 to eliminate the coordinate Z component contained in each output, the following translational and rotational speeds are calculated as three-dimensional motion parameters of the object. Perform such calculations.

First, the coordinate component Z is deleted from the equations (8) and (9), and the following equation is obtained. _{U XX A XX + U YY A} YY + U ZZ A ZZ + U XY A XY + U ZX A ZX + F X T X + F Y T Y + F Z T Z = 0 ·· (10) wherein, U _XX = {(∂I ₁ / ∂x) (∂I ₂ / ∂y) − (∂I ₂ / ∂x) (∂I ₁ / ∂y)｝ (1 + y ² ) / 2 U _YY = ｛(∂I ₁ / ∂x) (∂ I ₂ / ∂y) − (∂I ₂ / ∂x) (∂I ₁ / ∂y)｝ (1 + x ² ) / 2 U _ZZ = ｛(∂I ₁ / ∂x) (∂I ₂ / ∂y) − (∂I ₂ / ∂x) (∂I ₁ / ∂y)｝ (x ² + y ² ) / 2 U _XY = − ｛(∂I ₁ / ∂x) (∂I ₂ / ∂y) − (∂ I ₂ / ∂x) (∂I ₁ / ∂y)｝ xy U _YZ = − ｛(∂I ₁ / ∂x) (∂I ₂ / ∂y) − (∂I ₂ / ∂x) (∂I ₁ / ∂y)｝ y U _ZX = − ｛(∂I ₁ / ∂x) (∂I ₂ / ∂y) − (∂I ₂ / ∂x) (∂I ₁ / ∂y)｝ x ·· (11 ) F _X = + ( _{I 2 / ∂x) (∂I 1} / ∂t) - (∂I 1 / ∂x) (∂I 2 / ∂t) F Y = + (∂I 2 / ∂y) (∂I 1 / ∂t ) − (∂I ₁ / ∂y) (∂I ₂ / ∂t) F _Z = −y (∂I ₂ / ∂y) (∂I ₁ / ∂t) + y (∂I ₁ / ∂y) (∂ I ₂ / Δt) −x (ΔI ₂ / Δx) (ΔI ₁ / Δt) + x− (ΔI ₁ / Δx) (ΔI ₂ / Δt) (12) A _XX = _{2W X T X A YY = 2W} Y T Y A ZZ = 2W Z T Z A XY = W X T Y + W Y T X A YZ = W Y T Z + W Z T Y A ZX = W Z T X + W X T _Z · (13) Since the equation (10) holds at each point of the image, an equation for the number of pixels in the image is obtained. On the other hand, since the unknown is only six motion parameters, a sufficient expression for obtaining a solution was obtained. <Solution of Equation (10)> Next, an example of how to solve Equation (10) will be described. Since equation (10) holds at each point of the image, (1)
The sum of the square of the left side of equation (0) and the points in all the regions occupied by the object on the image is also equal to zero.

∫ (U_XXA_XX+ U_YYA_YY+ U_ZZA_ZZ+ E_XYA_XY+ U_ZXA_ZX+ F_XT_X+ F_YT_Y+ F_ZT_Z)^Twodxdy = 0 (14) However, since the image actually contains noise, (1)
Equation 4) does not always hold. Therefore, the left of equation (14)
Find the solution that minimizes the edge. Where A_XX, A _YY,
A_ZZ, A_XY, A_YZ, A_ZX, T_X, T_Y, T_ZThe independent unknown
Assuming that it is a variable, the solution that minimizes the left side of equation (14) is
The fixed value corresponding to the minimum eigenvalue of the observation matrix S defined by
Equivalent to a vector.

S = ∫UUdxdy (15) where U = (U _XX , U _YY , U _ZZ , U _XY , U _YZ , U _ZX , F _X ,
F _Y , F _Z ).

The observation matrix S is calculated by the above equation (15) (step 105), and the eigenvector corresponding to the minimum eigenvalue of the observation matrix S is calculated, whereby A _XX and A _YY defined by the equation (13) are obtained. _{, a ZZ, a XY, a} YZ, a ZX and translational velocity T _X, T _Y, obtaining the T _Z (step 106). Further, the rotational speed can also be obtained by solving the equation (13) for W _X , W _Y , and W _Z (step 107).

Thus, since the parameters of the three-dimensional motion of the object can be calculated from the moving image without repeatedly performing the calculation as in the related art, the amount of calculation can be reduced and convergence to an erroneous solution can be avoided. If the motion parameters are obtained, the equation (8) or (9) is solved assuming that the equation is an equation for Z, and the depth Z is obtained for each pixel (step 108).

Further, the three-dimensional motion parameter data obtained by the motion parameter calculation circuit 7 is stored in a parameter memory 8.
And these parameter data can be used later as position coordinate data of the industrial robot. <Embodiment 2> Next, Embodiment 2 of the present invention will be described.
FIG. 5 is a configuration block diagram of the second embodiment, and FIG. 6 is a flowchart for explaining a motion parameter extraction method of the second embodiment.

The feature of the second embodiment over the first embodiment is that two types of spatial filter circuits 3a and 3
b, a color filter circuit 11a for applying a color filter for extracting at least two types of spectral bands from a moving image, such as when a color image is input,
11b. Color filter circuits 11a, 11
It is assumed that b is built in the color TV camera. The other configuration of the second embodiment is the same as the configuration of the first embodiment.

For example, a moving image is input by the TV camera 1 (step 101), converted to a digital image by the A / D converter 2, and a red filter is applied to the digital image by the color filter circuit 11a (step 101). Step 202), an image Ir (x, y, z) is obtained. Further, a green filter is applied to the digital image by the color filter circuit 11b to obtain an image Ig (x, y, t).

Then, the image Ir (x, y, z) and the image I
g (x, y, t) and the first embodiment described above.
(Steps 103 to 108). As described above, even when the color filter circuits 11a and 11b are used, the translation speed, the rotation speed, and the depth coordinates can be obtained as the motion parameters without repeating the calculation. Note that the color filter is the same as the above-mentioned 2
Other colors may be used.

In the above-described embodiment, the rectangular coordinate system has been described. However, the present invention is applicable to a cylindrical coordinate system and a polar coordinate system. In the case of cylindrical coordinates, cylindrical coordinates (r, θ) are introduced by x = rcos θ y = rsinθ. At this time, ∂ / ∂x = cos θ∂ / ∂r- (sin θ∂ / ∂θ) / r∂ / ∂y = sin θ∂ / ∂r + (cos θ∂ / ∂θ) / r U _XX , U _YY , U in equations (11) and (12)
_ZZ is as follows. U _XX = (1 + r ² sin ² θ) ｛(∂I ₁ / ∂r) (∂I ₂ /
_{∂θ) - (∂I 2 / ∂r} ) (∂I 1 / ∂θ)} / 2r U YY = (1 + r 2 cos 2 θ) {(∂I 1 / ∂r) (∂I 2 /
∂θ)-(∂I ₂ / ∂r) (∂I ₁ / ∂θ)｝ / 2r U _ZZ = r ｛(∂I ₁ / ∂r) (∂I ₂ / ∂θ)-(∂I ₂
/ ∂r) (∂I ₁ / ∂θ)｝ / 2 U _XY = −r cos θ sin θ ｛(∂I ₁ / ∂r) (∂I ₂ / ∂
θ) − (∂I ₂ / ∂r) (∂I ₁ / ∂θ)｝ U _YZ = −sin θ ｛(∂I ₁ / ∂r) (∂I ₂ / ∂θ) −
(∂I ₂ / ∂r) (∂I ₁ / ∂θ)｝ U _ZX = −cos θ ｛(∂I ₁ / ∂r) (∂I ₂ / ∂θ) −
(∂I ₂ / ∂r) (∂I ₁ / ∂θ)｝ F _X = ｛cos θ (∂I ₂ / ∂r) −sin θ (∂I ₂ / ∂θ)
/ R｝ (∂I ₁ / ∂t)-｛cosθ (∂I ₁ / ∂r) -sin
θ (∂I ₁ / ∂θ) / r｝ (∂I ₂ / ∂t) F _Y = ｛sin θ (∂I ₂ / ∂r) + cos θ (∂I ₂ / ∂θ)
/ R｝ (∂I ₁ / ∂t) − ｛sin θ (∂I ₁ / ∂r) + cos
θ (∂I ₁ / ∂θ) / r｝ (∂I ₂ / ∂t) F _Z = −r (∂I ₂ / ∂r) (∂I ₁ / ∂t) + r (∂I ₁
/ ∂r) (∂I ₂ / ∂t) Others are the same as in the above-described rectangular coordinate system.

Next, polar coordinates will be described. x = tanφcosθ y = tanφsinθ Introduce polar coordinates (φ, θ). At this time, therefore, U _XX , U _YY , U _ZZ.
・ ・ Is as follows. U _XX = cos ³ φ (1 + r ² sin ² θ) ｛(∂I ₁ / ∂φ) (∂
I ₂ / ∂θ)-(∂I ₂ / ∂φ) (∂I ₁ / ∂θ)｝ / 2s
^{_{inφ U YY = cos 3 φ (}} 1 + r 2 cos 2 θ) {(∂I 1 / ∂φ) (∂
I ₂ / ∂θ)-(∂I ₂ / ∂φ) (∂I ₁ / ∂θ)｝ / 2s
_{inφ U ZZ = sinφcosφ {(∂I} 1 / ∂φ) (∂I 2 / ∂θ) -
(∂I ₂ / ∂φ) (∂I ₁ / ∂θ)｝ / 2 U _XY = −sinφcosφsinθcosθ ｛(∂I ₁ / ∂φ) (∂
I ₂ / ∂θ)-(∂I ₂ / ∂φ) (∂I ₁ / ∂θ)｝ U _YZ = −cos ² φsin θ ｛(∂I ₁ / ∂φ) (∂I ₂ / ∂
θ) − (∂I ₂ / ∂φ) (∂I ₁ / ∂θ)｝ U _ZX = −cos ² φcos θ ｛(∂I ₁ / ∂φ) (∂I ₂ / ∂
θ) − (∂I ₂ / ∂φ) (∂I ₁ / ∂θ)｝ F _X = ｛cos ² φcos θ (∂I ₂ / ∂φ) −sin θ (∂I ₂ /
∂θ) / tanφ｝ (∂I ₁ / ∂t) − ｛cos ² φcosθ (I ₁
∂ / ∂φ) −sin θ (I ₁ ∂ / ∂θ) / tanφ｝ (∂I ₂ /
^{_{∂t) F Y = {cos 2}} φsinθ (∂I 2 / ∂φ) + cosθ (∂I 2 /
∂θ) / tanφ｝ (∂I ₁ / ∂t) − ｛cos ² φsinθ (I ₁
∂ / ∂φ) + cos θ (I ₁ ∂ / ∂θ) / tanφ｝ (∂I ₂ /
_{∂t) F Z = -sinφcosφ (∂I} 2 / ∂φ) (∂I 1 / ∂t) + s
inφcosφ (∂I ₁ / ∂φ) (∂I ₂ / ∂t) Others are the same as those of the above-described orthogonal coordinate system. In addition, it goes without saying that various modifications can be made without departing from the spirit of the present invention.

[0036]

According to the present invention, since the parameters of the three-dimensional motion of the object can be calculated from the moving image without repeating the calculation, the amount of calculation can be reduced and convergence to an erroneous solution can be avoided.

[Brief description of the drawings]

FIG. 1 is a principle diagram of the first invention.

FIG. 2 is a principle view of the second invention.

FIG. 3 is a configuration block diagram of a first embodiment of the present invention.

FIG. 4 is a flowchart for explaining a motion parameter extraction method according to the first embodiment.

FIG. 5 is a configuration block diagram of a second embodiment of the present invention.

FIG. 6 is a flowchart illustrating a motion parameter extraction method according to a second embodiment.

FIG. 7 is a principle diagram of a conventional motion parameter extraction method.

[Explanation of symbols]

 1. TV camera 2. A / D converter 3. Spatial filter circuit 4. Delay circuit 5. Time differentiation circuit 6a..x direction differentiation circuit 6b..y direction differentiation circuit 7..Motion parameter calculation Circuit 8. Parameter memory 11. Color filter circuit

────────────────────────────────────────────────── ─── Continuation of the front page (56) References JP-A-3-6780 (JP, A) JP-A-4-276553 (JP, A) JP-A-5-151354 (JP, A) Toshio Endo, " Calculation of Optical Flow Using Spatio-Temporal Smoothness Constraint ”, 1991, IEICE Transactions D- ▲ II ▼, Vol. J74-D-II, No. 12, p. 1678-1685 (58) Field surveyed (Int. Cl. ⁶ , DB name) G06T 7/20 JICST file (JOIS)

Claims (3)

(57) [Claims] 1. An image of a moving object in a three-dimensional coordinate system (X, Y, Z) is taken, so that a two-dimensional coordinate system (x,
y) an image input means (1) for projecting a moving image on the image plane; and a spatial filter processing means for performing at least two types of spatial filter processing on the moving image output from the image input means (1). 3), differentiation processing means (5, 6) for performing time differentiation, x-direction differentiation, and y-direction differentiation processing on each output from the spatial filter processing means (3); 6) speed calculation means (7) for calculating a translation speed and a rotation speed as three-dimensional motion parameters of the object by eliminating a coordinate Z component included in each output by using each output of 6); The moving image from the image input means (1) is subjected to filter processing by the spatial filter processing means (3), and the obtained outputs are subjected to time differentiation and spatial differentiation by the differentiation processing means (5, 6). Speed calculation using each output A motion parameter extraction method, wherein the output means (7) deletes the coordinate Z component included in each output, and calculates a translation speed and a rotation speed as three-dimensional motion parameters of the object. 2. An image of a moving object in a three-dimensional coordinate system (X, Y, Z) is taken, so that a two-dimensional coordinate system (x,
y) an image input unit (1) for projecting a moving image on the image plane; and a color filter processing unit (1) for extracting at least two types of spectral bands from the moving image output from the image input unit (1). 11), differential processing means (5, 6) for performing time differentiation, x-direction differentiation, and y-direction differentiation processing on each output of the color filter processing means (11); 6) speed calculation means (7) for calculating a translation speed and a rotation speed as three-dimensional motion parameters of the object by eliminating a coordinate Z component included in each output by using each output of 6); The moving image from the image input means (1) is subjected to filter processing by the color filter processing means (11), and the obtained outputs are subjected to time differentiation and spatial differentiation by the differentiation processing means (5, 6). Speed using each output A motion parameter extraction method, wherein a calculating means (7) deletes a coordinate Z component included in each output, and calculates a translation speed and a rotation speed as three-dimensional motion parameters of the object. 3. The motion parameter according to claim 1, wherein said velocity calculating means calculates a coordinate Z component based on each output of said differential processing means. Extraction method.