KR100574227B1 - Apparatus and method for separating object motion from camera motion - Google Patents

Abstract

1. TECHNICAL FIELD OF THE INVENTION

The present invention relates to an object motion extraction apparatus and method for compensating for camera motion.

2. The technical problem to be solved by the invention

An object of the present invention is to provide an object motion extraction apparatus and method capable of separating pure motion (three-dimensional motion) of each object compensating for camera movement in a dynamic environment in which camera and object motions are different.

3. Summary of the Solution of the Invention

The present invention provides a device for extracting motion of a video object in a dynamic environment in which a motion of a camera and an object is different, wherein an object and a background area are separated by using motion information of an image in at least two video frames, and the epipolar constraint is removed. Motion dividing means for separating the background region represented by the movement of the camera only and the object region represented by the movement of the camera and the object; Camera calibration means for obtaining an internal camera parameter (camera coordinate system) in the background area for each image frame; Camera motion estimation means for obtaining a "three-dimensional movement of the camera" between the camera coordinate systems for each image frame; Camera / object coordinate system conversion means for obtaining a conversion between the camera coordinate system and each object coordinate system for each image frame; And using the camera / object coordinate transformation and the camera coordinate transformation between the image frames to remove the 'three-dimensional movement of the camera' from the object region including the movement of the object and the camera for each image frame, thereby eliminating the 'pure of the object. Object motion estimation means for obtaining a three-dimensional motion.

4. Important uses of the invention

The present invention is used in the apparatus for separating the object movement from the camera movement.

Dynamic Environment, Object Motion Separation, Calibration, Coordinate System Transformation, Object Motion Estimation

Description

Apparatus and method for separating object motion from camera motion}             

1 is a block diagram of an embodiment of an object motion extraction apparatus for compensating for camera movement according to the present invention.

2 is a flowchart illustrating an object motion extraction method for compensating for camera motion according to the present invention.

FIG. 3 is a diagram illustrating an embodiment of a coordinate system transformation relationship between an object and a camera appearing in two images in a dynamic environment used in the present invention. FIG.

* Explanation of symbols on the main parts of the drawing

11: motion divider 12: camera calibration unit

13: camera motion estimation unit 14: coordinate system conversion unit

15: object motion estimation unit

The present invention relates to the field of object motion estimation technology. In particular, in a dynamic environment in which camera and object motions are different, an object motion extraction apparatus capable of separating pure motion (three-dimensional motion) of each object that compensates for camera motion And to a method thereof.

The problem of reconstructing three-dimensional space from an image occupies a large proportion in the field of computer vision, and various methods for this are currently being studied. However, these methods usually only limit the environment in which one of the cameras and objects are fixed, or only suggest ways to track objects in environments where both cameras and objects are moving. It is not being proposed.

Here, a method of restoring a three-dimensional environment by moving a camera from a fixed environment has been studied for a long time as a structure from motion (SFM). The SFM is representative of two methods of restoring the three-dimensional structure after performing the camera calibration and two methods of restoring the three-dimensional structure from the uncalibrated camera without the camera calibration information.

First, the method of restoring a three-dimensional structure after performing camera calibration assumes that the camera calibration parameters are known in advance. To this end, camera calibration is performed by using a sensor to measure in advance or to find the camera position from a marker that knows the three-dimensional absolute coordinates. If the three-dimensional position of the camera is known as described above, three-dimensional information may be extracted using images acquired at different positions in space. That is, a camera model is constructed between three-dimensional space coordinates and a pixel coordinate system on an image using calibration parameters of the camera, and then the intrinsic parameter and extrinsic ( After the pose) parameter) is extracted, three-dimensional information is extracted.

Second, a method of restoring a three-dimensional structure from an uncalibrated camera without camera calibration information is a method of using the information under the projection projection existing between two images by three-dimensional reconstruction with only an image without correction information. This method obtains 3D information after obtaining a fundamental matrix representing the geometric relationship between two images. This utilizes geometric epipolar constraints on the camera model in projection. The mathematical representation of the epipolar constraint corresponds to a fundamental matrix, which represents a rigid constraint between two images under a projection projection. However, the information obtained here is not information about a metric, but information under a projective transform.

As described above, although various methods exist for restoring a 3D environment from an image in the past, either a camera or an object is a fixed environment, and neither attempt has been made to restore the 3D environment under a moving dynamic environment. . Therefore, when the conventional method is used in a dynamic environment, the motions of the camera and the object are expressed in a mixed form, which is different from the motion of the actual object. Therefore, there is an urgent need for a method that can separate the pure movement of each object in a dynamic environment in which the movements of the camera and the object are different.

The present invention has been proposed to meet the above demands, and in a dynamic environment in which the motions of the camera and the object are different, an object capable of separating only the pure motion (three-dimensional motion) of each object that compensates for the camera motion It is an object of the present invention to provide a motion extraction device and a method thereof.

In order to achieve the above object, the present invention provides a device for extracting motion of an image object in a dynamic environment in which a motion of a camera and an object is respectively provided, wherein the object and the background area are separated using motion information of the image in at least two image frames. Motion dividing means for separating the background region represented by the movement of the camera only and the object region represented by the movement of the camera and the object using the epipolar constraint; Camera calibration means for obtaining an internal camera parameter (camera coordinate system) in the background area for each image frame; Camera motion estimation means for obtaining a "three-dimensional movement of the camera" between the camera coordinate systems for each image frame; Camera / object coordinate system conversion means for obtaining a conversion between the camera coordinate system and each object coordinate system for each image frame; And using the camera / object coordinate transformation and the camera coordinate transformation between the image frames to remove the 'three-dimensional movement of the camera' from the object region including the movement of the object and the camera for each image frame, thereby eliminating the 'pure of the object. And object movement estimating means for obtaining a three-dimensional motion.

Meanwhile, the present invention provides a method of extracting a motion of an object in a dynamic environment in which a camera and an object are respectively moved, wherein the object and the background area are separated from each other by using motion information of the image in at least two image frames. A motion segmentation step of separating a background region represented by camera movements and an object region represented by camera and object movements using polar constraints; A camera calibration step of obtaining an internal camera parameter (camera coordinate system) in the background area for each image frame; A camera motion estimation step of obtaining a 'three-dimensional movement of the camera' between the camera coordinate systems for each image frame; A camera / object coordinate system conversion step of obtaining a transformation between the camera coordinate system and each object coordinate system for each image frame; And using the camera / object coordinate transformation and the camera coordinate transformation between the image frames to remove the 'three-dimensional movement of the camera' from the object region including the movement of the object and the camera for each image frame, thereby eliminating the 'pure of the object. And an object motion estimation step of obtaining a three-dimensional motion.

In addition, the present invention is characterized in that it further comprises an object motion display step of displaying the movement of the separated object in the absolute coordinate system.

The present invention is to separate the movement of the camera and the object in the image sequence reflecting the movement of the camera and the object in a dynamic environment in which both the camera and the object is moving. At this time, in order to separate the movement of only the object, first, the three-dimensional movement of the camera must be restored from the two-dimensional movement projected on the image. Then, the movement of the object can be obtained by separating the movement of the camera from the movement on the projected image reflecting both the movement of the camera and the object.

In the present invention, to restore the three-dimensional motion of the moving object in such a dynamic environment, the preprocessing process for dividing the object and the background, and to separate the camera and the object movement to apply to the method of restoring the three-dimensional motion in the existing static environment It is easily generalized to dynamic environment through post-processing.

The above objects, features and advantages will become more apparent from the following detailed description taken in conjunction with the accompanying drawings. Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

To help understand, the technical principles of the present invention as a whole will be described.

In general, when using the information under the projection projection existing between two images by three-dimensional reconstruction with only the image without correction information, the information obtained is not information about a metric, but information under a projective transform.

Thus, for metric reconstruction, you must have a scene constraint (i.e., a point or line on a plane, or a vertical or horizontal condition on a line or plane). Generalizing these scene constraints, if there is a projection from a parallelepiped, a geometrical object that has all the constraints, then the camera internal parameters can be obtained, allowing metric reconstruction. However, if the scene constraints cannot be extracted from the image, only reconstruction under projective transformation is possible. In addition, it is possible to obtain by restraining the vanishing point, which is the intersection point of the parallel lines, and the internal parameters of the camera, without necessarily using a parallelepiped for metric restoration. Recently released cameras satisfy the conditions of square pixel or zero skewness, so if you know these conditions, you can perform metric reconstruction with only partial scene constraints.

Therefore, in the present invention, in order to satisfy the complete scene constraint, camera internal parameters are obtained through projection to a marker image that knows a three-dimensional structure such as a parallelepiped, or a partial scene constraint is known in advance. The camera internal parameter information is used to obtain the remaining camera internal parameters (focal length, principal point position) and perform metric restoration. In addition, once the camera internal parameters are obtained, a conversion between the object and the camera coordinate system can be obtained when the coordinate system of the moving object is known. Thus, the present invention utilizes at least four points each of which includes feature points that know a three-dimensional metric structure thereon, or which can be parameterized into one plane point on a two-dimensional metric structure. Get each object and camera coordinate system transformation.

However, if the objects in the environment are also moving (i.e. in a dynamic environment), the three-dimensional reconstruction cannot be done with the above method, because triangulation cannot be established to determine the three-dimensional position from the two views. to be. If the epipolar constraint obtained from the background region is used as it is, the object and camera movement are reflected as one. In other words, epipolar constraints appear differently for background areas that are only related to camera movement and for object areas that are related to both camera and object movement.

Therefore, in the present invention, when the object and the camera movement is separated, it is divided into the process of obtaining the camera movement largely (202, 203 of FIG. 2), and the process of separating the object movement using the movement of the camera (204, 205 of FIG. 2). . To this end, a process 201 of dividing the background of the camera-only movement and the area of the object represented by the movement of the camera and the object is essential.

Assuming such division, the divided background region can be restored by a metric using an existing SFM method, and the camera movement can be detected during the restoration (203). In addition, in the case of an object, a rigid body transformation between the object and the camera coordinate system can be obtained using at least four points on a plane and internal camera parameters obtained from the background (204). Can be obtained (205). The best case for this division is to have a marker like a parallelepiped that knows the three-dimensional structure, and to derive the camera calibration and object coordinate system from it. However, by mitigating the condition of the marker, self-calibration that finds the internal parameters of the camera through the background that knows the parallel and vertical relationship of points and lines is possible, and only the corresponding points of at least four points on the plane are on the object. Even if it is known, it is possible to use the present invention.

On the other hand, if there are no markers that can give these coordinates on objects or in the background and features that give partial scene constraints, Euclidean motion cannot be obtained and only motion under projective transformation is known. I can't. Therefore, in the present invention, a camera is provided through the constraints in the case where there is a marker that can give a coordinate system to each object and the background where the most information is given, and in the environment where the self-calibrable constraints are alleviated. The internal parameters and movements are identified, and the movement of objects is determined through the coordinate system transformation between the camera and the object. In addition, even when the image and the object cannot be distinguished from each other, manually specifying a minimum local correspondence area obtains the best matching point or line in the area and obtains an epipolar constraint or a scene constraint. . Manually assigning a matching point allows the user to interactively modify the matching point interactively if the matching point obtained by the algorithm is different from the actual one, and to easily obtain accurate constraints in case the automatic algorithm fails.

Lastly, for the case of missing or lacking features that satisfy the scene constraints, object constraints can be achieved by giving camera constraints instead of scenes. Various camera constraints may come out like scene constraints, but the present invention uses a fixed pair of stereo cameras. In this case, the fixed means that the relative orientation of the two cameras does not change when the camera moves, and the internal parameters of the camera do not change. In this case, the movement of the stereo camera is to determine the plane at infinity, which is equivalent to giving three vanishing points in a single camera, and even affine restoration in the existing SFM. This can be achieved by using self-calibration with the constraint of "image of absolute conic". This results in finding the relative external orientation and internal parameters of the two cameras. Next, for all the pixels classified as the background, the corresponding points are obtained by triangulation through stereo matching, and when the three-dimensional coordinates are obtained, the corresponding points on the object perform one three-dimensional coordinate transformation between two frames. Expressed by the rigid transformation of, the motion of the object can be easily obtained.

1 is a configuration diagram of an object motion extraction apparatus for compensating for camera motion according to an exemplary embodiment of the present invention, illustrating a motion separation structure of a camera and an object.

FIG. 3 shows the coordinate system transformation relationship between an object and a camera appearing in two images in a dynamic environment. The object motion is represented by a transformation of two object coordinate systems. The transformation is obtained by a linear composite transformation after obtaining the remaining three transformations. Can be. In Figure 3, C ₁ is the position of the camera center (camera coordinate system origin) of the first image, C ₂ is the position of the camera center (camera coordinate system origin) of the second image, O ₁ is the position of the object coordinate system origin at the first image acquisition The position and O ₂ represent the position of the object coordinate system origin at the second image acquisition, respectively.

In the present invention, in a dynamic environment in which both the camera and the object move, the camera and the object are separated from the image sequence reflecting the movement of the camera and the object.

To this end, first, camera motion information is extracted from the still background. Then, the object movement is separated using the extracted camera movement. This can restore the camera motion and the three-dimensional structure of the environment in the stationary environment using the existing method. In addition, if the object has a marker that can give a coordinate system, the rigid body transformation between the object and the camera can also be obtained by conventional methods. However, in the conventional method, in order to obtain the motion of an object, the motion of the object coordinate system must be found from the stationary background. In this case, the motion of the object coordinate system can be obtained every frame or directly relative to the absolute coordinate system. The movement of the camera coordinate system and the transformation of the object coordinate system to the camera coordinate system should be obtained and indirectly from these linear transformations to the absolute coordinate system to the object coordinate system. For this to be possible, a marker representing the absolute coordinate system must be visible every frame. To satisfy this, the camera movement must be limited so that the marker is always visible. However, even if the camera is fixed, if the object obscures the marker, it becomes impossible to obtain the object movement.

However, in the present invention, even if the background marker is not visible by focusing on the object for which the motion is to be found or the marker is not present, the movement of the object coordinate system is not known from the stationary absolute coordinate system. The transformation between the object and the camera can be obtained, and the linear motion transformation can be used to obtain the motion of the object. This means that even if there is no calibration target that gives the absolute coordinate system, it can be arbitrarily given to the camera position of the first image and the object's movement can be obtained. It can be saved. It only needs to satisfy the constraint that there is a marker on the object that can grasp the unique object coordinate system without ambiguity. The easiest markers presented in the present invention are four points with no mirror symmetry on the plane. In addition, even if the surface of the object is not flat and curved, if four or more points on the cross section of the virtually cut plane are designated as markers, this means that the movement of the object can be obtained. In addition, if the object does not even have such a marker, the camera is constrained to obtain the object's movement.

As a result, the conventional technology that can obtain only the relative movement with respect to the camera frame for the object moving independently of the camera movement can separate the movement of the object and output it to the screen. Enable 3D reconstruction

)을 구하기 위한 카메라 움직임 추정부(13)와, 각 영상프레임마다 카메라 좌표계와 각 객체 좌표계 사이의 변환(즉, 각 영상에서 카메라/객체간 강체 변환

)을 구하기 위한 좌표계 변환부(14)와, 카메라/객체간 좌표변환(즉, 각 영상에서 카메라/객체간 강체 변환

)과 영상프레임간의 카메라 좌표변환(즉, I₁에서 I₂로의 카메라 움직임을 나타내는 강체 변환

)으로부터 객체의 움직임(즉, I₁에서 I₂로의 객체 움직임을 나타내는 강체 변환

)을 구하기 위한 객체 움직임 추정부(15)를 포함한다. Referring to FIGS. 1 and 3, a configuration of an apparatus for extracting a motion of an image object compensating for camera movement according to the present invention having the above-described features may include: movement of an image in at least two image frames I ₁ and I ₂ . A motion dividing unit 11 for dividing an object and a background region using information, and a camera calibrating unit 12 for obtaining an internal camera parameter (camera coordinate system) K ₁ and K ₂ in the background region for each image frame. And a rigid body transform representing the camera movement between the camera coordinate systems for each image frame (i.e., the camera movement from I ₁ to I ₂ ).

A camera motion estimator 13 for obtaining the image data, and a transformation between the camera coordinate system and each object coordinate system for each image frame (i.e., a rigid body-to-camera transformation in each image).

) And a coordinate transformation between the camera and the object (i.e., a rigid body transformation between the camera and the object in each image).

) Camera coordinate transformation (i.e. rigid body transformation representing camera movement from I ₁ to I ₂ )

Rigid body representing the object's movement (i.e., I ₁ to I ₂ )

It includes the object motion estimation unit 15 for obtaining ().

Here, the motion of the object separated by the object motion estimation unit 15 can be displayed in the absolute coordinate system. Therefore, in the dynamic environment in which the movements of the camera and the object are respectively, pure movement of each object may be separated, and only the movement of the separated object may be displayed in the absolute coordinate system.

The motion divider 11 classifies the area of the moving object independently of the fixed background. When the background and the object have a marker that distinguishes it, the epipolar constraint of the markers of the background is epipolar. Epipoles are obtained by constraints, and outlier pixels that do not converge to the epipoles are classified to classify the background and the rest. In addition, for the remaining areas, the markers of the respective objects are classified in the same manner as the background classification process to classify the regions of the corresponding objects.

The motion divider 11 classifies the area of the moving object independently of the fixed background. When there is no marker in the background, the motion divider 11 manually sets the user to obtain the epipolar constraint. When the corresponding region is designated in the frame image, the corresponding point having the best local matching in the region is obtained (feature point matching), and the pixels converged by the epipole obtained as the feature point are classified into the background. Then, the area of each object is classified by performing the same process as the background classification process on the outlier pixels that are not the background.

In extracting camera internal parameters, the camera calibration unit 12 obtains a 3D camera coordinate system and a camera internal variable for the absolute coordinate system (camera calibration), and when there is no information about the 3D absolute coordinate system, a scene constraint or two Only the internal parameters of the camera are obtained using the correspondence of feature points in more than one image (self-calibration).

When the camera motion estimator 13 calculates camera movement between camera coordinate systems for each image frame, when there is information about a three-dimensional absolute coordinate system, the camera motion estimator 13 obtains the rotation and translational movement of the camera from an external parameter of the camera, If there is no related information, the rotation and translational movement of the camera are obtained from the camera internal parameters obtained from the camera calibrator 12 and the epipolar constraint of the background region.

The coordinate system converting unit 14 calculates the pose of the camera coordinate system with respect to the object coordinate system using the minimum points through the conversion between the camera and each object coordinate system. At this time, if there is no three-dimensional marker that can know the absolute coordinate system, using the result of the camera calibration unit 12 to obtain the internal variables of the camera using three or more image frames using only pixels in the fixed background area ( The conversion between the camera and the object is obtained by using an object coordinate system having self-calibration) and camera parameters obtained from self-calibration and corresponding points of at least six points on the object (four on the plane).

The object motion estimator 15 calculates a three-dimensional motion of the object by linear transformation of camera motion and coordinate system transformation. That is, the motion of the object is obtained from the combined transformation between the camera coordinate system obtained by the camera motion estimation unit 13 and the coordinate system transformation between the camera and the object obtained by the coordinate system conversion unit 14.

An operation of the image object motion extraction apparatus for compensating for camera movement according to the present invention having the above configuration will be described with reference to FIG. 2.

First, the motion division unit 11 divides an object and a background area (201).

If there are known markers, the two chapters classify them as pairs of pairs that satisfy the epipolar constraint of markers' corresponding points.

On the other hand, if there are no markers, manual / automatically determines minimum pairs of matching points that satisfy the epipolar constraint and determines whether a background / object is present.

In addition, more accurate motion boundaries can be compensated for by combining other features (eg contour, color).

After that, the divided background and the area of the object are processed as follows.

First, the camera calibration unit 12 obtains camera internal parameters in the background (202).

If a marker is given, it is calibrated from this to obtain camera internal / external parameters, immediately obtain camera movement from external parameters, and internal parameters to the object's pose estimation (rigid transformation between object and camera). I use it.

However, if no marker is given, self calibration should be performed automatically from the background area. This method uses the SFM method from an existing uncalibrated camera, which requires constraints to restore the metric from projective restoration. At this time, the constraints include scene constraints such as the information of the Euclidean space (vertical, parallel condition, etc.), camera external variable constraints such as camera motion information, and finally camera internal variable constraints. Normally, considering the camera CCD as a square pixel and limiting to zero skew, aspect ratio = 1, the parameters to be obtained are two principal points and one focal length. Normally, the principal point is placed at or near the image center and approximates only the focal length. At this time, the self-calibration is performed by optimizing by giving an appropriate cost function to approximate.

In this way, when the camera internal variable (internal parameter) is obtained (202), the metric is restored, and when matrix decomposition is performed from the obtained Euclidean projection matrix, the camera is divided into an internal parameter matrix and an external parameter matrix. The motion estimation unit 13 calculates the position of the camera from this (203).

In addition to this method, if the camera internal parameters are obtained, the internal parameter matrix of the two cameras is divided from the fundamental matrix representing the epipolar constraint to obtain a matrix representing only external parameters. Orthogonality can be used to calculate rotational and translational motion components that represent the position of the camera using methods such as Singular Value Decomposition (SVD). Then, the camera motion is obtained by linear transformation of matrices representing the positions of the two cameras.

Now, the remaining part is the movement of the object. First, the coordinate system converting unit 14 converts between the camera and the object coordinate system using the camera projection model obtained from the camera internal parameters obtained by the camera calibrating unit 12, that is, the camera with respect to the object coordinate system. The pose estimation of each frame is performed (204).

Finally, the object motion estimation unit 15 calculates the motion of the object using two previously obtained transformations between camera objects and a linear transformation of the camera motion (205).

In the conventional method, the position of the object (coordinate system) was obtained by the linear transformation of the camera coordinate system and the transformation of the coordinate system between the camera and the object with respect to the absolute coordinate system, but the origin of the absolute coordinate system must always be visible. However, in the dynamic environment, the camera moves away from the origin, or the object moves freely, obscuring the origin frequently. Therefore, the present invention compensates for the limitations of the existing method. We presented a way to obtain.

3 shows that the movement of the object (coordinate system transformation) can be obtained from the linear movement of the camera movement and coordinate transformations between the object and the camera. They are rigidly transformed and mathematically belong to groups, and because of the nature of the group that is unique to the inverse of one transform, the movement of the object is determined solely by the remaining three transforms.

In summary, the present invention is divided into three cases and proposed a method of separating the movement of the object for each.

First, when there are markers on the object and the background, the first process of performing the camera calibration from the background marker to obtain the coordinate system and camera movement between the camera and the background, the camera from the object marker The second process of determining the coordinate system transformation between objects, the transformation of the coordinate system between the camera and the background of the first process and the linear transformation of the coordinate system transformation between the camera objects of the second process, the coordinate transformation between the object and the background, that is, the movement of the object I can be divided into third courses.

Second, if the object has at least four points on a plane (six points in a typical three-dimensional space) and scene constraints in the background, the camera self-calibrates to determine camera internal parameters from the scene constraints in the background. Step 1, a second step of detecting the camera movement in the epipolar constraint obtained as the feature point of the background using the internal parameters obtained in the first step, the camera from the camera internal parameters and at least four points on the plane It may be divided into a third process of finding a coordinate system between objects and a fourth process of finding a motion of an object by using the results of the second and third processes.

Third, when there are no markers in the object and the background, a method of restoring the movement of the object by using a stereo camera having a fixed orientation and internal parameters is fixed. This can be done by using a stereo camera with fixed relative viewpoints and internal parameters to separate pure movements of objects even in environments where calibration targets or scene constraints cannot be given.

That is, a first process of projective reconstruction for each stereo image pair, a second process of obtaining a transformation matrix between two projective restorations for each object, and an infinite process from the transformation matrix obtained in the second process A third process of calculating plane at infinity to affine reconstruction, a fourth process of self-calibrating for metric reconstruction in the background area and obtaining camera movement therefrom, each It can be divided into a fifth process of performing metric reconstruction for each object and directly obtaining the object movement.

As described above, the present invention can restore the three-dimensional movement of the moving object existing in the three-dimensional space through the analysis of a continuous image using a (or a pair) camera. This ultimately means to analyze the dynamic image by the motion of the camera and to detect the object moving relative to the camera and to restore the motion information in the three-dimensional space of the object.

Existing image-based modeling techniques can restore a three-dimensional structure only in a static environment, but the present invention means that the three-dimensional structure can be restored to a general dynamic environment.

The existing method of restoring the three-dimensional structure is to use special equipment for sophisticated restoration, and there is typically a structured light using a laser. Such methods usually require expensive equipment and have disadvantages such as limited application environment indoors, while methods using images are relatively free of such restrictions. However, the method using such an image also has to restrict the position of the camera to restore the structure of the moving object. However, there are a lot of dynamic environments in the real environment that cannot impose this limitation. For example, collision detection with other cars in a car on the road, images of tracking a player or ball in athletics, detection and tracking of a moving person for surveillance in an unmanned store.

In the image obtained in such a dynamic environment, not only a moving object is detected but also useful information can be provided by performing the three-dimensional reconstruction every frame as in the present invention. For example, since the distance between the camera and the object on the road can be known for each frame, the speed of the car can be controlled in conjunction with the autonomous driving system of the vehicle. In addition, when used in an active camera system, the camera system can be usefully used in various fields, such as providing a motion or zooming feedback to the camera system according to the movement of a target object, so that the target can be tracked without missing a target. have.

The method of the present invention as described above may be implemented as a program and stored in a computer-readable recording medium (CD-ROM, RAM, ROM, floppy disk, hard disk, magneto-optical disk, etc.).

The present invention described above is not limited to the above-described embodiments and the accompanying drawings, and various substitutions, modifications, and changes are possible in the art without departing from the technical spirit of the present invention. It will be clear to those of ordinary knowledge.

As described above, the present invention has an effect of restoring three-dimensional movement of a moving object existing in three-dimensional space through analysis of a continuous image using one (or a pair) camera.

By doing so, the present invention can be applied in an environment without limitations, such as fixing a camera position, or in an environment in which limitations cannot be applied, for example, collision checking between cars on a road, video tracking a player or a ball in an athletic game, and an unmanned store. It can be used for the detection and tracking of a moving person for surveillance, and when used in an active camera system, the object can be missed by giving a movement or zooming feedback to the camera system according to the movement of the object. There is a traceable effect.

Claims (12)

In the video object motion extraction apparatus in a dynamic environment where the motion of the camera and the object is respectively, In order to separate the object and the background area using the motion information of the image in at least two video frames, and to separate the background area represented by the camera movement and the object area represented by the camera and the object movement using the epipolar constraint. Motion dividing means; Camera calibration means for obtaining an internal camera parameter (camera coordinate system) in the background area for each image frame; Camera motion estimation means for obtaining a "three-dimensional movement of the camera" between the camera coordinate systems for each image frame; Camera / object coordinate system conversion means for obtaining a conversion between the camera coordinate system and each object coordinate system for each image frame; And By using the camera-to-object coordinate transformation and the camera coordinate transformation between the image frames, the 'three-dimensional movement of the camera' is removed from the object region including the movement of the object and the camera for each image frame. Object motion estimation means to find dimensional motion Apparatus for extracting object movements that compensates for camera movements. The method of claim 1, The movement dividing means, In classifying the area of an object that moves independently of the fixed background, if the background and the object have markers that distinguish it, the epipole is affected by epipolar constraints of the markers in the background. (epipole), classify outlier pixels that do not converge to the epipole, classify the background and the rest, and classify the background for each object's markers, for the rest of the area. An apparatus for extracting motion of a camera, which compensates for camera motion, comprising performing a process identical to the process to classify a region of a corresponding object. The method of claim 1, The movement dividing means, In classifying the area of an object that moves independently of the fixed background, if there is no marker in the background, the user can manually specify the corresponding area in the two-frame image to obtain the epipolar constraint. Find the matching point that has the best local matching in the area (matching feature), classify the pixels that converge to the epipole obtained as the feature point, and classify the background for outlier pixels rather than the background. Compensating for the camera movement, the object motion extraction apparatus characterized in that to classify the area of each object by performing the same process. The method of claim 1, The camera calibration means, In extracting the camera internal parameters, the 3D camera coordinate system and the camera internal variables for the absolute coordinate system are obtained (camera calibration), and in the absence of information on the 3D absolute coordinate system, the feature constraints or the mapping of feature points in two or more images An apparatus for extracting object motions that compensates for camera movements, wherein only the camera internal parameters are obtained using a relationship (self-calibration). The method of claim 1, The camera motion estimation means, In calculating camera movements between camera coordinate systems for each image frame, when there is information about a three-dimensional absolute coordinate system, the rotation and translational movement of the camera are obtained from external parameters of the camera, and when there is no information about the absolute coordinate system, the camera calibration is performed. An object motion extraction apparatus for compensating for camera motion, comprising obtaining a rotation and a translation motion of a camera from an internal parameter obtained from means and an epipolar constraint of a background area. The method of claim 1, The camera / object coordinate system conversion means, Calculate the pose of the camera coordinate system with respect to the object coordinate system using the minimum points through the conversion between the camera and each object coordinate system. If there is no three-dimensional marker that can know the absolute coordinate system, the result of the camera calibration means is used. Using only pixels in a fixed background area, camera internal variables are obtained using three or more image frames (self-calibration), camera parameters obtained from self-calibration, and at least six points on the object (four on a plane) An object motion extraction apparatus for compensating for camera movement, comprising: obtaining a transformation between a camera and an object using an object coordinate system having a corresponding point of. The method according to any one of claims 1 to 6, The object motion estimation means, The three-dimensional motion of the object is calculated by linear transformation of camera motion and coordinate system transformation, and the object is obtained from the combined transformation of the transformation between the camera coordinate system obtained by the camera motion estimation means and the coordinate system transformation between the camera and object obtained by the camera / object coordinate system conversion means. An object motion extraction apparatus for compensating for camera movements, the motions of which are obtained. In the method for extracting the movement of an object in a dynamic environment where the movement of the camera and the object is respectively, A motion that separates the object and the background area using the motion information of the image in at least two video frames, but separates the background area represented by the camera movement and the object area represented by the camera and the object movement using the epipolar constraint. Partitioning step; A camera calibration step of obtaining an internal camera parameter (camera coordinate system) in the background area for each image frame; A camera motion estimation step of obtaining a 'three-dimensional movement of the camera' between the camera coordinate systems for each image frame; A camera / object coordinate system conversion step of obtaining a transformation between the camera coordinate system and each object coordinate system for each image frame; And By using the camera-to-object coordinate transformation and the camera coordinate transformation between the image frames, the 'three-dimensional movement of the camera' is removed from the object region including the movement of the object and the camera for each image frame. Object motion estimation step to find 'dimensional motion' Object motion extraction method that compensates for the camera movement comprising a. The method of claim 8, An object motion display step of displaying a motion of the separated object in an absolute coordinate system Object movement extraction method that compensates for the camera movement further comprising. The method according to claim 8 or 9, When we find the movement of the object, If there are markers on the object and in the background, A camera calibration step of performing a camera calibration from a marker of a background to obtain a coordinate system transformation between the camera and the background and to obtain a camera movement; A coordinate system transformation step of finding a coordinate system transformation between camera objects from a marker of the object; And An object motion estimation step of determining the object-background coordinate system transformation, that is, the motion of the object, by the linear transformation of the camera-background coordinate system transformation in the camera calibration step and the camera object coordinate transformation in the coordinate system transformation step. Object motion extraction method that compensates for the camera movement comprising a. The method according to claim 8 or 9, When we find the movement of the object, If an object has at least 4 points on a plane (6 points in a typical 3D space) and a scene constraint in the background, Self-calibrating the camera to calibrate the camera internal parameters from the scene constraints in the background; A camera movement estimating step of knowing camera movement in an epipolar constraint obtained as a background feature using the internal parameters obtained in the self-calibration step; A coordinate system transformation step of finding a coordinate system between a camera and an object from at least four points on a plane and a camera internal parameter; And Object motion estimation step of finding the motion of the object by using the results of the camera motion estimation step and the coordinate system conversion step Object motion extraction method that compensates for the camera movement comprising a. The method according to claim 8 or 9, When we find the movement of the object, If there are no markers on the object and background, restore the object's movement using a stereo camera with fixed relative orientation and internal parameters. Projective reconstruction for each stereo image pair (projective reconstruction), A transformation matrix extraction step of obtaining a transformation matrix between two projective restorations for each of the background and objects; An affine restoration step of calculating a plane at infinity from the transformation matrix obtained in the transformation matrix extraction step and affine reconstruction; A camera motion estimation step of self-calibration for metric reconstruction in the background area and obtaining camera movement therefrom; And Object motion estimation step of metric recovery for each object and direct object motion from it Object motion extraction method that compensates for the camera movement comprising a.

Images