JPH1141500A - Device and method for detecting motion vector and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To find a motion vector without depending on an image signal and to surely detect it so as not to generate erroneous detection caused by the pattern of a picked-up image by detecting the motion vector of a moving image, which is provided by operating an image pickup means having a lens, based on only the detection value of a parameter before and after this operation. SOLUTION: The initial coordinate value of a point Q for projecting a point P on an object onto a projection plane S is applied to a straight projection line generating part 5 and a difference detection part 8. An affin parameter as the moving amount of the projection plane is provided from the detection value from a camera parameter detection part 1 before and after camera operation. At an affin transforming part 4, affin transformation is performed to the projection plane S by this affin parameter and a projection plane S' (2nd image pickup plane) is found before and after the camera operation. Next, a straight projection line L through an origin and the point Q is found by the straight projection line generating part 5 and the coordinate of a cross section Q' between the straight projection line L and the projection plane S' is found by a moved projection point generating part 6. The initial value of the point Q is subtracted from the transformed coordinate of the point Q' found by the affin transforming part 7 to find the motion vector.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an apparatus and a method for detecting a motion vector and a storage medium, and more particularly to a method for detecting a motion vector of a moving image necessary for performing a motion compensation process or a special effect of a video in a moving image compression. The present invention relates to a motion vector detection device and method for performing detection, and a storage medium.

[0002]

2. Description of the Related Art With the advancement of inter-frame coding and moving picture signal processing, it has become important to accurately detect a motion vector.

A conventional motion vector detection device detects a motion vector based only on the signal values of pixels of a moving image.
For example, the block matching method that has been mainly used so far is an image correlation method using a cross-correlation coefficient. In the motion vector estimation method according to this method, a position where the sum of differences between pixels in a block is minimized is used. , And this is estimated as a motion vector representing the block.

[0004]

However, in the above-described conventional motion vector estimation method, since the motion vector is detected from the signal value of the pixel of the moving image, a change in the signal value such as a picture area is small. Then, there was a problem that erroneous detection of a motion vector occurred. In the block matching method, a motion vector is estimated relatively stably in an area having a high density, but becomes unstable in an area having a low density. For example, a building street is the former pattern, and a clear sky is the latter pattern.

Accordingly, the present invention has been made in view of the above points, and provides a motion vector detecting apparatus and method and a storage medium which solve the above-mentioned problems by taking into account the movement of a camera caused by a camera operation. The purpose is to do.

[0006]

In order to achieve the above object, in the apparatus according to the present invention, a motion vector of a moving image obtained by operating an imaging means having a lens is not moved. A motion vector detection device that detects based on detection values of various parameters of movement of the imaging unit that changes a captured image of the subject by the imaging unit,
A motion vector detecting unit that detects the motion vector corresponding to the motion of the imaging unit due to the operation based on only the detected values of the parameters before and after the operation.

In this case, in the apparatus according to the present invention, the motion vector detecting means may perform the same operation as the first coordinate of the subject on the first imaging surface of the captured image before the operation. Initial coordinate setting means for setting initial coordinates on a second imaging surface after the operation, post-operation coordinate detection means for obtaining second coordinates of the subject on a second imaging surface of the post-operation captured image, And a difference detecting means for obtaining a difference obtained by subtracting the initial coordinates from the second coordinates, and the difference vector may be used as the motion vector.

Here, in the apparatus according to the third aspect of the present invention, the post-operation coordinate detecting means includes an initial projection plane generating means for specifying the position coordinates of the first imaging plane based on the parameters. Means for determining a parameter of the amount of movement of the imaging surface, and a first space having an axis perpendicular to the first imaging surface by geometrically transforming the position coordinates of the first imaging surface with the parameter of the amount of movement. Means for determining the position coordinates of the second imaging surface in the above, and the intersection of the principal point of the lens and a straight line passing through the first coordinates of the subject on the first imaging surface with the second imaging surface. Means for obtaining coordinates in the first space, and geometrically transforming the coordinates of the intersection in a second space having an axis perpendicular to the second imaging surface by using the parameter of the movement amount, the second coordinates Means for seeking And it can also be.

In order to achieve the above object, according to the method of the present invention, a motion vector of a moving image obtained by operating an image pickup means having a lens is obtained by the image pickup means of an object which does not move. A motion vector detection method for detecting based on detection values of various parameters of movement of the imaging unit that changes a captured image, based on only the detection values of the parameters before and after the operation, A motion vector detecting step of detecting the motion vector corresponding to the motion.

Here, in the method of the present invention described in claim 5, the motion vector detecting step is the same as the operation of the first coordinate of the subject on a first imaging surface of the captured image before the operation. An initial coordinate setting step of setting initial coordinates on a second imaging plane after the operation; a post-operation coordinate detection step of obtaining a second coordinate of the subject on a second imaging plane of the captured image after the operation; A difference detecting step of obtaining a difference obtained by subtracting the initial coordinates from the second coordinates, and the difference vector may be used as the motion vector.

Here, in the method according to the present invention, the post-operation coordinate detecting step includes an initial projection plane generating step of specifying the position coordinates of the first imaging plane based on the parameters. Obtaining a parameter of a movement amount of the imaging surface; and a first space having an axis perpendicular to the first imaging surface by geometrically transforming the position coordinates of the first imaging surface with the parameter of the movement amount. The second in
Obtaining the position coordinates of the imaging surface of the object; and determining the principal point of the lens and the first position of the subject on the first imaging surface.
Determining the coordinates in the first space of the intersection of a straight line passing through the coordinates of the second imaging surface and the coordinates of the intersection in a second space having an axis perpendicular to the second imaging surface. Determining the second coordinates by performing a geometrical transformation using the parameter of the movement amount.

Here, the method of the present invention according to claim 7 may further include a step of detecting various parameters of a movement of the imaging means for changing an image of the immovable subject captured by the imaging means. .

In order to achieve the above object, in the storage medium according to the present invention, a motion vector of a moving image obtained by operating an image pickup means having a lens can be used as an image pickup means for an immovable subject. A storage medium storing a program of a motion vector detection method for detecting based on the detection values of various parameters of the movement of the imaging unit that changes the captured image according to the above, based on only the detection values of the parameters before and after the operation And a motion vector detecting step of detecting the motion vector corresponding to the motion of the imaging means by the operation.

Here, in the storage medium according to the ninth aspect of the present invention, the motion vector detecting step is the same as the first coordinate of the subject on a first imaging surface of the captured image before the operation. An initial coordinate setting step of setting initial coordinates on the second imaging plane after the operation, and a post-operation coordinate detection step of obtaining second coordinates of the subject on a second imaging plane of the captured image after the operation; A difference detection step of obtaining a difference obtained by subtracting the initial coordinates from the second coordinates, and a program for a motion vector detection method using the difference vector as the motion vector may be stored.

Here, in the storage medium according to the present invention, the post-operation coordinate detection step includes an initial projection plane generation step of specifying the position coordinates of the first imaging plane based on the parameters. Obtaining a parameter of a movement amount of the imaging surface, and geometrically transforming the position coordinates of the first imaging surface by the parameter of the movement amount, the first having an axis perpendicular to the first imaging surface. Obtaining position coordinates of the second imaging surface in space; and an intersection of a straight line passing through a principal point of the lens and the first coordinates of the subject on the first imaging surface and the second imaging surface. Calculating coordinates in the first space, and geometrically transforming the coordinates of the intersection in a second space having an axis perpendicular to the second imaging surface by using the parameter of the movement amount. It is also possible to store the program of the motion vector detecting method comprising determining a target.

Here, in the storage medium according to the present invention, the motion vector further includes a step of detecting various parameters of a motion of the imaging means for changing an image taken by the imaging means of the immovable subject. A detection method program can also be stored.

[0017]

Embodiments of the present invention will be described below in detail with reference to the drawings.

FIG. 1 is a block diagram of a main part of an embodiment of the present invention.

Each component in FIG. 1 may be realized only by hardware, or a component other than the camera parameter detecting unit 1 may be configured by using a well-known computer system having a CPU, a ROM, a RAM, and the like. ROM, CD-ROM,
Floppy disk, hard disk and other JAZ
(Registered trademark), ZIP (registered trademark), MO disk, etc., may be realized by software processing based on programs stored in various known storage media.

The camera parameter detection section 1 detects various parameters of camera movement that changes a captured image when a television camera (hereinafter, referred to as a camera) that captures an image of a subject is operated, and outputs a detection signal.

Here, with reference to FIG. 2, each of the above camera parameters and the camera parameter detecting section 1 will be described.

In FIG. 2, 20 is a camera body, 22 is a camera platform, and 24 is a truck. The carriage 24 can change the dolly position in a linear direction by moving on a rail (not shown) or the like. The camera shown in FIG. 2 is provided with levers (not shown) for arbitrarily adjusting a pan angle, a tilt angle, a pitch angle, and a camera height. The direction of the system can be changed. Further, the angle of view can be arbitrarily changed from wide to tele by moving the lens of the imaging system in a linear direction by operating a zoom switch or the like. The principal point position of the lens of the imaging system can be changed in a straight line direction by an operation such as a zoom operation or a change of a focal length (focus). The camera parameters include the structure and dimensions of the camera body 20.

A camera parameter detector 1 (FIG. 1) of the present embodiment for detecting each of the illustrated camera parameters includes a linear position sensor such as a potentiometer for detecting a linear position change, and a rotational position change. Has a rotation position sensor such as a rotary encoder for detecting the rotational position. The potentiometer is provided in the camera body 20 as is well known, and the rotary encoder is provided on the camera platform 22.
The linear position sensor for detecting the dolly position is a cart 24.
Or with rails.

Although the sensors are not limited to those described above, the camera parameter detecting section 1 shown in FIG. 1 outputs the detection values of the respective sensors for each field in synchronization with the synchronizing signal of the video signal. The configuration of the initial projection plane generation unit 2, the affine parameter generation unit 3, the affine transformation units 4, 7, the projection line generation unit 5, the post-operation projection point generation unit 6, and the difference detection unit 8 allows the camera operation of these sensors. The motion vector of the captured image is detected using only the initial detection value at the time and the post-operation detection value when the captured image is changed by the operation using the perspective projection method as described later.

Here, the application of the perspective projection method to the present invention will be described.

As shown in the simplified model diagram of the imaging in FIG.
The image captured by the camera can be simply represented as a model in which the principal point Pp of the lens of the camera is the center of projection and the image capturing surface is the projection surface S.

The projection plane S of this model is defined as the principal point Pp of the lens.
Assuming that the lens is positioned point-symmetrically with respect to the center, this is equivalent to perspective projection in which the principal point Pp of the lens is the center of projection and the imaging plane is the projection plane S, as shown in the model diagram of perspective projection in FIG. At this time, the point P on the subject in the three-dimensional space is
Projected to the upper point Q.

Assuming that the subject is stationary without moving, the captured image changes when the camera is operated so that the camera parameters change, and the projection plane S moves in a three-dimensional space. The position of the projection plane S after the movement by the operation is obtained by calculation from the detection values of the aforementioned sensors provided in the camera, as described later.

FIG. 5 is an explanatory diagram showing the relationship between the projection plane, the projection point, and the motion vector before and after the movement by the camera operation.

As shown in the drawing, the projection plane before the movement is S, and the projection plane after the movement is S '. Here, it is assumed that the projection plane S is rotated clockwise about the principal point by the angle θ. At this time, the stationary point P is projected onto the point Q 'on the projection plane S'. However, the pixel of the captured image at the position of the point Q on the projection plane S corresponds to the position of the point R on the projection plane S ′. Here, considering only on the projection plane S ′ after the movement, the object at the position of the point R appears to have moved to the position of the point Q ′. That is, the motion vector corresponding to the motion by the camera operation at the pixel at the position of the point R on the projection plane S 'is RQ' from the point R to the point Q '.

The method of obtaining the motion vector RQ 'will be described below with reference to FIGS. However, the coordinates of the point R on the projection plane S ′ after the movement are used to obtain a motion vector at the point R on the projection plane S ′.
It is given as an initial value.

Before the operation, first, the initial projection plane generation unit 2 makes the projection plane S (first imaging plane) perpendicular to the Z axis in the three-dimensional coordinate space with the principal point as the origin O, that is, the projection plane S
Is set to a constant value. This value is the distance between the origin O and the projection plane S. The initial detection value of the angle of view from the camera parameter detection unit 1 and the number of pixels on the imaging surface in the direction in which the angle of view was measured (or the number of vertical pixels if the angle of view is vertical) , The number of horizontal pixels if the angle of view is horizontal). This three-dimensional coordinate space is referred to as a space A (first space). Also, a point P on the subject
Are projected on the projection plane S, the coordinates of the point Q are equal to the coordinates of the point R on the projection plane S ′. Therefore, the coordinates of the point Q of the pixel for which a motion vector is to be obtained are given to the projection straight line generator 5 and the difference detector 8.

Assume that the camera is operated in the rotation direction, and the imaging system is moved clockwise by θ around the Y axis. When the captured image changes, the affine parameter generation unit 3
Thus, an affine transformation parameter (affine parameter), which is the movement amount of the projection plane, is obtained from the detection values before and after the operation from the camera parameter detection unit 1. In coordinate transformation by affine transformation, a straight line is generally transformed into a straight line, and the ratio of the distance between points on the straight line and the parallelism of the line segment are maintained as is well known.

Next, the affine transformation unit 4 affine-transforms the projection plane S using the affine parameters to obtain a projection plane S ′ (second imaging plane) after the camera operation.

Next, a projection straight line L passing through the origin O and the point Q is determined by the projection straight line generator 5, and coordinates of the intersection of the projection straight line L and the projection plane S 'are determined by the post-operation projection point generator 6. Thus, the coordinates of the point Q ′ in the space A are obtained. These coordinates are projection point coordinates after the camera operation.

Next, consider a three-dimensional coordinate space A '(second space) in which the Z axis is perpendicular to the projection plane S'. This space A '
The affine parameters generated by the affine parameter generation unit 3 are affine-transformed in the space A, and the affine transformation unit 7 affine-transforms the coordinates of the point Q ′ in the space A in the direction opposite to the moving direction of the projection plane. By performing the conversion, the coordinates in the space A 'of the point Q' after the movement of the pixel whose motion vector is to be obtained are obtained.

Then, in the difference detection unit 8, the point Q (point R) before the operation is calculated from the coordinates of the point Q 'obtained by the affine transformation unit 7.
Is subtracted, and the vector RQ ′ represented by the difference obtained by the subtraction is estimated to be the motion vector to be obtained.

The initial value of the coordinates of the point Q (R) is sequentially reset to each pixel point for which a motion vector is to be obtained on the projection plane S (S '), and the above-described motion vector detection processing is repeated. I do.

As described above, according to the present embodiment, since a motion vector is detected by using a geometrical transformation from a camera movement by a camera operation without depending on a moving image signal, a moving image signal such as a shading of an image is detected. Erroneous detection due to a pattern such as an area with little change does not occur, and the motion vectors are regularly arranged in the captured image, so that a motion vector that reflects the motion of the camera can be reliably detected.

[0040]

As described above, according to the present invention, when detecting a motion vector of a moving image by operating an imaging means having a lens, a parameter for changing a captured image of the subject by the imaging means is detected. Then, a motion vector corresponding to the motion of the imaging unit is detected based only on the value of this parameter at one time and the value at another time, and the imaging unit operates by the imaging unit without depending on the moving image signal. Since the motion vector is detected from the motion of the image, it is possible to surely detect the motion vector reflecting the motion of the imaging unit regardless of the pattern of the captured image, and to obtain the effect of preventing erroneous detection due to the pattern. .

[Brief description of the drawings]

FIG. 1 is a block diagram showing an embodiment of the present invention.

FIG. 2 is an explanatory diagram of camera parameters.

FIG. 3 is a simplified model diagram of imaging by a camera.

FIG. 4 is a model diagram based on perspective projection.

FIG. 5 is an explanatory diagram showing a relationship between a projection plane, a projection point, and a motion vector before and after movement by a camera operation.

FIG. 6 is an operation explanatory diagram of one embodiment of the present invention.

[Explanation of symbols]

 Reference Signs List 1 camera parameter detection unit 2 initial projection plane generation unit 3 affine parameter generation unit 4, 7 affine transformation unit 5 projection straight line generation unit 6 post-operation projection point generation unit 8 difference detection unit

 ──────────────────────────────────────────────────続 き Continued on the front page (72) Inventor Hideki Sumiyoshi 1-10-11 Kinuta, Setagaya-ku, Tokyo Japan Broadcasting Corporation Broadcasting Research Institute

Claims (11)

[Claims] 1. A motion vector of a moving image obtained by operating an imaging unit having a lens is determined based on detection values of various parameters of movement of the imaging unit for changing an image of a stationary subject captured by the imaging unit. A motion vector detecting device that detects the motion vector corresponding to the motion of the imaging device by the operation based on only the detected value of the parameter before and after the operation. Characteristic motion vector detection device. 2. The motion vector detecting means sets initial coordinates on the second imaging plane after the operation that are the same as the first coordinates of the subject on the first imaging plane of the imaging image before the operation. Initial coordinate setting means, and post-operation coordinate detecting means for obtaining second coordinates of the subject on a second imaging surface of the post-operation captured image, and subtracting the initial coordinates from the second coordinates The motion vector detecting device according to claim 1, further comprising: a difference detecting means for obtaining the difference vector, wherein the difference vector is used as the motion vector. 3. The post-operation coordinate detecting means, an initial projection plane generating means for specifying the position coordinates of the first imaging plane based on the parameter, a means for obtaining a parameter of a moving amount of the imaging plane, Means for geometrically transforming the position coordinates of the first imaging surface with a parameter of a movement amount to obtain position coordinates of the second imaging surface in a first space having an axis perpendicular to the first imaging surface. Means for calculating coordinates in the first space of an intersection of a principal point of the lens and a straight line passing through the first coordinates of the subject on the first imaging surface and the second imaging surface; Means for geometrically transforming the coordinates of the intersection in a second space having an axis perpendicular to the second imaging plane by using the parameter of the movement amount to obtain the second coordinates. The motion vector described in 2 Torque detection device. 4. A motion vector of a moving image obtained by operating an image pickup device having a lens, based on detection values of various parameters of a motion of the image pickup device for changing an image of a non-moving subject taken by the image pickup device. A motion vector detecting step of detecting the motion vector corresponding to the motion of the imaging unit by the operation based on only the detected value of the parameter before and after the operation. Characteristic motion vector detection method. 5. The motion vector detecting step sets initial coordinates on the second imaging plane after the operation that are the same as the first coordinates of the subject on the first imaging plane of the captured image before the operation. An initial coordinate setting step, a post-operation coordinate detection step of obtaining a second coordinate of the subject on a second imaging surface of the post-operation image, and a difference obtained by subtracting the initial coordinate from the second coordinate. The method for detecting a motion vector according to claim 4, further comprising a step of detecting a difference, wherein the difference vector is used as the motion vector. 6. The post-operation coordinate detection step, comprising: an initial projection plane generation step of specifying position coordinates of the first imaging plane based on the parameters; and a step of obtaining a parameter of a movement amount of the imaging plane; Geometrically transforming the position coordinates of the first imaging surface with a parameter of a movement amount to obtain position coordinates of the second imaging surface in a first space having an axis perpendicular to the first imaging surface; Obtaining a coordinate in the first space of an intersection of a principal point of the lens and a straight line passing through the first coordinates of the subject on the first imaging surface and the second imaging surface; Geometrically transforming the coordinates of the intersection in a second space having an axis perpendicular to the second imaging surface with the parameter of the movement amount to obtain the second coordinates. Motion vector detecting method according to Motomeko 5. 7. The apparatus according to claim 4, further comprising a step of detecting various parameters of a movement of the imaging unit that changes a captured image of the immovable subject by the imaging unit. Motion vector detection method. 8. A motion vector of a moving image obtained by operating an imaging unit having a lens is determined based on detection values of various parameters of a movement of the imaging unit that changes an image of a non-moving subject captured by the imaging unit. A storage medium storing a program of a motion vector detection method for detecting the motion vector, wherein the motion detecting the motion vector according to the motion of the imaging unit by the operation based on only the detected value of the parameter before and after the operation. A storage medium storing a program for a motion vector detection method, comprising a vector detection step. 9. The motion vector detecting step sets initial coordinates on the second imaging plane after the operation that are the same as the first coordinates of the subject on the first imaging plane of the captured image before the operation. An initial coordinate setting step, a post-operation coordinate detection step of obtaining a second coordinate of the subject on a second imaging surface of the post-operation image, and a difference obtained by subtracting the initial coordinate from the second coordinate. 9. A storage medium storing a program for a motion vector detection method according to claim 8, comprising a step of detecting a difference, wherein the difference vector is used as the motion vector. 10. The post-operation coordinate detection step, comprising: an initial projection plane generation step of specifying position coordinates of the first imaging plane based on the parameters; and a step of obtaining a parameter of a movement amount of the imaging plane. Geometrically transforming the position coordinates of the first imaging surface with a parameter of a movement amount to obtain position coordinates of the second imaging surface in a first space having an axis perpendicular to the first imaging surface; Obtaining a coordinate in the first space of an intersection of a principal point of the lens and a straight line passing through the first coordinates of the subject on the first imaging surface and the second imaging surface; Geometrically transforming the coordinates of the intersection in a second space having an axis perpendicular to the second imaging surface with the parameter of the movement amount to obtain the second coordinates. The storage medium storing the program of the motion vector detecting method according to claim 9. 11. The apparatus according to claim 8, further comprising a step of detecting various parameters of a movement of the imaging unit that changes an image of the immovable subject captured by the imaging unit. A storage medium storing a program for a motion vector detection method.