KR100447778B1 - Apparatus for Embodying Stereo/Multiview Realistic Mixed Reality using Pose Estimation and Method Thereof - Google Patents

Abstract

The present invention relates to a stereo / multi-view realistic mixed reality implementation apparatus using pose estimation and a method thereof and a computer-readable recording medium recording a program for realizing the method. The mixed reality implementation apparatus of the present invention comprises: calibration means for receiving images from the plurality of cameras, calculating internal / external parameters for each camera, and determining a search range of a reference camera and a feature point; Tracking means for tracking a feature point based on the search range with respect to an image of a reference camera determined by the calibration means; Pose estimation means for estimating a pose of the virtual image using the feature point; And matching means for matching the real image with the virtual image in real time using the pose estimated by the pose estimating means.

Description

Apparatus for Embodying Stereo / Multiview Realistic Mixed Reality using Pose Estimation and Method Thereof}

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a stereo / multi-view realistic mixed reality implementation apparatus using pose estimation and a method thereof, and to a computer readable recording medium recording a program for realizing the method. The present invention relates to a stereo / multi-view realistic mixed reality realization apparatus and a method thereof, and a computer-readable recording medium recording a program for realizing the method.

With the development of imaging technology, the demand for sensory media information for mixing virtual objects based on images as a means of delivering images more realistically is increasing rapidly.

The sensational media refers to various element information for overcoming space and time constraints in a virtual environment.

In order to realize such a sensational media, various information such as stereoscopic digital video, graphic composite object, character, animation, etc. should be able to be combined without any unnaturalness from the correlation with each other, and should be able to be fused with the user through various interfaces.

As a related art, AR (Augmented Reality) technology, which combines another world artificially synthesized by a computer and additional information artificially synthesized based on reality, has been studied. come.

In such a system, a virtual object is combined with a location and orientation obtained by analyzing and tracking correlation with an image for the interaction of a converged environment, and using a head mounted display (HMD) or a HTD ( It outputs through a spectacle display device such as a head tracked display, and by using the principle that a human can see the object with both eyes to determine the distance between the objects and recognize the object realistically, the spectacle type such as HMD Two cameras are attached to the display device, and each image is divided and shown to each eye to realize stereoscopic vision.

However, the conventional technology as described above is limited to a single stereoscopic viewpoint, and there is a problem that a difference occurs in the perceived distance according to the difference between the camera setting and the observer's human body structure (distance difference between eyes, etc.). In some cases, there was a problem in that it is impossible to recognize the distance or feel severe fatigue.

In order to solve the above problems, multi-view image technology has been introduced that can provide an appropriate image to an observer while expanding the viewpoint by using an image (multi-view image) acquired by a multi-view camera.

However, since the amount of information increases in proportion to the number of viewpoints compared to the single viewpoint image, the feature extraction, tracking, and registration position of each viewpoint image for analyzing and tracking such images to match each virtual object is increased. There is a serious problem that the amount of computation for estimation also increases proportionally.

The present invention has been proposed to solve the problems of the prior art as described above, by performing a pose estimation for four points on the same plane, stereo / using a pose estimation to efficiently reduce the amount of calculation between multiple points It is an object of the present invention to provide a multi-view realistic mixed reality implementation apparatus.

In addition, another object of the present invention is to provide a stereo / multi-view mixed reality implementation method using a pose estimation to efficiently reduce the amount of computation between multi-points by performing pose estimation for four points on the same plane. .

Another object of the present invention is to provide a computer-readable recording medium having recorded thereon a program for efficiently reducing the amount of computation between multi-points by performing pose estimation for four points on the same plane.

1 is a structural diagram of an embodiment of a stereo / multi-view mixed reality implementation apparatus using a pose estimation according to the present invention;

2 is a detailed structural diagram of an embodiment of the posing unit of FIG. 1;

3 is an exemplary view for explaining a plane and a direction component estimated by the estimator of FIG. 2;

4 is a flowchart illustrating a method of implementing a stereo / multi-view realistic mixed reality using a pose estimation according to the present invention;

5a and 5b is an exemplary view for showing a mixed reality implemented according to the stereo / multi-view realistic mixed reality implementation method using the pose estimation in accordance with the present invention.

* Explanation of symbols for the main parts of the drawings

110: camera 120: calibration unit

130: tracking unit 140: posing

150: matching unit 160: virtual image processing unit

141: selector 143: compensator

145: estimator

In order to achieve the above object, the present invention provides a stereo / multi-view realistic mixed reality implementation apparatus using pose estimation for real-time synthesis of real images received from a plurality of cameras and pre-modeled virtual images, from the plurality of cameras. Calibration means for receiving an image, calculating internal / external parameters for each camera, and determining a search range of the reference camera and the feature point; Tracking means for tracking a feature point based on the search range with respect to an image of a reference camera determined by the calibration means; Pose estimation means for estimating a pose of the virtual image using the feature point; And matching means for matching the real image with the virtual image in real time using the pose estimated by the pose estimating means.

The present invention also provides a stereo / multi-view mixed reality implementation method using pose estimation for real-time synthesis of real images received from a plurality of cameras and pre-modeled virtual images. A first step of calculating internal / external parameters for each camera and determining a search range of the reference camera and the feature point; A second step of performing feature point tracking on an image of a reference camera determined by the calibration means based on the search range; Performing pose estimation of the virtual image using the feature point; And a fourth step of performing real-time registration of the real image and the virtual image using the pose.

In addition, the present invention is a mixed reality implementation apparatus having a microprocessor to provide a stereo / multi-view realistic mixed reality implementation method using a pose estimation for real-time synthesis of the real image and the pre-modeled virtual image received from a plurality of cameras A first function of calculating internal / external parameters for each camera with respect to images received from the plurality of cameras, and determining a search range of a reference camera and a feature point; A second function of performing feature point tracking on an image of a reference camera determined by the calibration means based on the search range; A third function of performing pose estimation of the virtual image using the feature point; And a computer-readable recording medium having recorded thereon a program for realizing a fourth function of performing real-time matching between the real image and the virtual image using the pose.

The present invention overcomes the spectacle limitations of augmented reality systems, which are evolving into a form of collaboration with a small number of users or in a mixed environment. We will use the terminology that implies the overall scope of mixed reality (MR), a method for creating mixed environments.

The apparatus for implementing mixed reality according to the present invention is a synthesis system through natural combining with an existing image medium, and may have a structure compatible with a multiview codec capable of encoding additional information when extended to a multiview.

The above objects, features and advantages will become more apparent from the following detailed description taken in conjunction with the accompanying drawings. First of all, in adding reference numerals to the components of each drawing, it should be noted that the same components have the same number as much as possible even if displayed on different drawings. Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a structural diagram of an embodiment of a stereo / multi-view mixed reality implementation apparatus using a pose estimation in accordance with the present invention.

As shown in the figure, the mixed reality implementation apparatus of the present invention, a plurality of cameras 110, the calibration unit 120, the tracking unit 130, the pose estimation unit 140, the matching unit 150 and the virtual image The processing unit 160 is included.

The plurality of cameras 110 is responsible for obtaining the same image at various points in time.

The calibration unit 120 is responsible for calculating internal and external parameters of the plurality of cameras 110 and determining a search range of a reference camera and a feature point to perform tracking.

The feature point is generally used in computer vision, and refers to a point for processing a particular image, particularly a geometric process. In the present invention, the feature point is used to calculate a position to raise a virtual object. I mean it.

The internal parameter includes a focal length associated with the projection matrix P, a distortion factor, a camera center, and the like, and corresponds to distortions generated by the plurality of cameras 110 itself. Parameter.

By calculating the internal parameters of the camera, it is possible to remove all distortion elements, thereby reducing the error of the pose estimation.

On the other hand, an external parameter is a parameter related to relative movement and rotation between the plurality of cameras 110 with respect to a reference camera, and is mainly a parameter used for calculating a pose estimation and an internal parameter. However, since more than eight spatial points are required for the calculation, the calculation is very complicated, and thus there is a problem that the calculation is impossible in real time.

Therefore, the calculation of the external parameter implemented in the present invention is performed for accurate calculation of the internal parameter, and thus the pose estimation of the present invention may be independent of calibration.

The tracking unit 130 is responsible for tracking a feature point based on a predetermined search range of the image of the reference camera by the calibration unit 120.

The posing unit 140 will be described in more detail with reference to FIG. 2.

FIG. 2 is a detailed structural diagram of an embodiment of the posing unit of FIG.

As shown in the figure, the pose estimator of the present invention includes a selector 141, a corrector 143, and an estimator 145.

The selector 141 is responsible for selecting four points on the same plane (Coplanar) among the feature points tracked by the tracking unit 130.

The corrector 143 is responsible for correcting the position of a feature point in an image of a camera other than the reference camera by correcting the distortion of the projection by using the internal parameter calculated by the calibration unit 120.

In addition, the corrector 143 is responsible for correcting a pose of an image of a camera not tracked by using a relationship between an external parameter and a multi-view camera.

The present invention can reduce the error of the pose estimation by correcting the position of the feature point by the corrector 143.

The estimator 145 is responsible for estimating the pose of four points on the same plane.

In the present invention, since the calibration method for complex external parameters is not used for pose estimation, real-time matching can be performed based on a small number of feature points.

Hereinafter, the pose estimation will be described in detail with reference to FIG. 3.

In the present invention, by obtaining a direction vector in the similar space (Affine Space), it estimates the rotation and movement of the rectangular plane marker with four corners as a feature point, and performs the pose estimation of the rectangular plane marker with respect to the reference camera using the same do.

FIG. 3 is an exemplary diagram for describing a plane and a direction component estimated by the estimator of FIG. 2.

B ₁ , B ₂ and B ₃ shown in the drawings indicate the moving direction of the plane S ₁ constituting the four points tracked above, and can be seen that they are perpendicular to each other.

In the present invention, B ₁ , B _2, and B ₃ , which are basic vectors of the estimated surface, can be obtained by Equation 1 below.

Where a _mn (m, n is an integer) represents a reference point, λ _L , λ _W , and λ _H represent a scale, respectively, and b _mn (m, n is an integer) represents a normal vector. .

From the rotation and the movement of the quadrangle direction component vector forming four points, a pose having the rotation and movement of the rectangular marker from the corner of one plane at the viewpoint of the reference camera can be estimated.

The virtual image processor 160 is responsible for modeling a virtual image that will constitute a mixed reality.

The matching unit 150 converts the virtual image received from the virtual image processing unit 160 into a pose received from the pose estimating unit 140 and performs a function of matching the real image and the virtual image in real time.

In this case, the image received from the reference camera, the image received from the other camera and the virtual image may be matched using Equation 2 below.

In this case, T _{CC '} is a transformation from the coordinate system of the reference camera C to a coordinate system of another camera C', T _CP is a transformation from the coordinate system of the reference camera C to the coordinate system of the real image object P, T _C'P means the transformation from the coordinate system of another camera C 'to the coordinate system of the object P, R _CP is the rotation of the object P from the coordinate system of the reference camera C, R _{C' P} means rotation of the object P from the coordinate system of another camera C '. In addition, t _CP means the movement of the object P from the coordinate system of the reference camera (C), t _C'P represents the movement of the object (P) from the coordinate system of the other camera (C ') it means.

4 is a flowchart illustrating a method of implementing a stereo / multi-view realistic mixed reality using a pose estimation according to the present invention.

As shown in the drawings, the mixed reality implementation method of the present invention receives images of the same object from the plurality of cameras 110 (S401), calculates internal and external parameters thereof, and sets a reference camera. Then, calibration such as determining a search range of the feature point is performed (S403).

Thereafter, the feature points are searched according to the search range determined above (S405). When the feature points are traced, four points on the same plane are selected from the searched feature points (S411). If it fails to track the feature point (S407), the feature point is tracked using external parameters (S409), and four points on the same plane are selected (S411).

Subsequently, the projection distortion of the camera is corrected (S413) to correct the position of the feature point, and a pose estimation is performed (S415).

Finally, matching of the pre-modeled virtual image and the real image is performed using the estimated pose at step S417.

5a and 5b is an exemplary view for showing the mixed reality implemented according to the stereo / multi-view realistic mixed reality implementation method using the pose estimation in accordance with the present invention, Figure 5a is implemented for the image of the reference camera FIG. 5B illustrates a pose estimation using calibration parameters for the reference camera for images of cameras other than the reference camera.

The image of the matching result according to the present invention using the multi-view relationship obtained through the calibration parameters for the reference camera in the pose estimation may have little difference from the image estimated in the pose from the respective cameras.

The method of the present invention as described above may be implemented as a program and stored in a computer-readable recording medium (CD-ROM, RAM, ROM, floppy disk, hard disk, magneto-optical disk, etc.).

The present invention described above is capable of various substitutions, modifications, and changes without departing from the spirit of the present invention for those skilled in the art to which the present invention pertains. It is not limited to the drawing.

As described above, the present invention obtains the relative movement and rotation between cameras in the pose estimation from the calibration parameters in advance, and does not perform the pose estimation operation when matching to a multiview, thereby enabling fast real-time multiview mixing reality. It has the effect of making it work.

In addition, since the present invention estimates and restores feature points that disappear during tracking by using the relationship between cameras, there is an effect of maintaining a robust tracking between time points unless all feature points are lost.

In addition, the present invention by using the four-point pose estimation on the same plane, it is possible to perform a fast operation, and to be able to match if the four feature points in the case of a single marker or any object forming the same plane There is.

In addition, the present invention has the effect of preventing the false estimation of the pose and reducing the error by correcting the error by using the internal parameters of the pre-calculated calibration at the time of the four point pose estimation on the same plane.

Claims (9)

In the apparatus for implementing stereo / multi-view realistic mixed reality using pose estimation for real-time synthesis of real images received from multiple cameras and pre-modeled virtual images, Calibration means for receiving images from the plurality of cameras, calculating internal / external parameters for each camera, and determining a search range of a reference camera and a feature point; Tracking means for tracking a feature point based on the search range with respect to an image of a reference camera determined by the calibration means; Pose estimation means for estimating a pose of the virtual image using the feature point; And Matching means for matching the real image and the virtual image in real time using the pose estimated by the pose estimating means. Stereo / multi-view realistic mixed reality implementation apparatus using a pose estimation including a. The method of claim 1, The internal parameter is An apparatus for implementing stereo / multi-view realistic mixed reality using pose estimation, comprising a focal length, a distortion element, and a camera center associated with a projection matrix. The method of claim 1, The external parameter is, And stereo / multi-view realistic mixed reality implementation apparatus using pose estimation, wherein the parameter is a parameter relating to relative movement and rotation of the other camera with respect to the reference camera. The method of claim 1, The posing unit, Selection means for selecting four points on the same plane among the feature points selected by the tracking means; Correction means for correcting the position of a feature point in an image of a camera other than a reference camera using the internal parameter; And Estimating means for estimating a pose with respect to four points on the same plane Stereo / multi-view realistic mixed reality implementation apparatus using a pose estimation, characterized in that it comprises a. The method of claim 4, wherein The estimating means, An apparatus for implementing stereo / multi-view realistic mixed reality using a pose estimation, wherein the direction vector of the estimated plane is determined by the following equation. Where a _mn (m, n is an integer) represents a reference point, λ _L , λ _W , and λ _H each represent a scale, and b _mn (m, n is an integer) represents a normal vector. Indicates) The method of claim 1, The matching means, And an image received from the reference camera or an image received from another camera other than the reference camera and the virtual image according to the following equation. 3. (At this time, T _{CC '} is a transformation from the coordinate system of the reference camera (C) to the coordinate system of another camera (C'), T _CP is a transformation from the coordinate system of the reference camera (C) to the coordinate system of the real image object (P), T _C'P means the transformation from the coordinate system of another camera (C ') to the coordinate system of the object (P), R _CP is the rotation of the object (P) from the coordinate system of the reference camera (C), R _{C 'P} ' refers to the rotation of the object P from the coordinate system of another camera C. In addition, t _CP means the movement of the object P from the coordinate system of the reference camera C. , t _C'P refers to the movement of the object P from the coordinate system of another camera C ') A stereo / multi-view mixed reality implementation method using pose estimation for real-time synthesis of real images received from multiple cameras and pre-modeled virtual images, A first step of calculating an internal / external parameter for each camera based on images received from the plurality of cameras, and determining a search range of a reference camera and a feature point; A second step of performing feature point tracking on an image of a reference camera determined by the calibration means based on the search range; Performing pose estimation of the virtual image using the feature point; And A fourth step of performing real-time matching between the real image and the virtual image using the pose Stereo / multi-view realistic mixed reality implementation method using a pose estimation comprising a. The method of claim 7, wherein If the feature point tracking fails, the fifth step of tracking the feature point using the external parameter Stereo / multi-view realistic mixed reality implementation method using a pose estimation further comprising. In a mixed reality implementation apparatus having a microprocessor to provide a stereo / multi-view realistic mixed reality implementation method using a pose estimation for real-time synthesis of real images received from a plurality of cameras and a pre-modeled virtual image, A first function of calculating internal / external parameters for each camera based on images received from the plurality of cameras, and determining a search range of a reference camera and a feature point; A second function of performing feature point tracking on an image of a reference camera determined by the calibration means based on the search range; A third function of performing pose estimation of the virtual image using the feature point; And A fourth function of performing real-time matching between the real image and the virtual image using the pose A computer-readable recording medium that records a program for realizing the problem.